[Yanel-dev] Fwd: Cluster documentation

[Balz Schreier]

you got it :-) that's what I tried to explain: read, modify and update must
be a single atomic operation.
cheers
balz

On Wed, Feb 2, 2011 at 7:30 PM, Cedric Staub <cedric.staub at wyona.com> wrote:

> On Wed, Feb 02, 2011 at 06:41:26PM +0100, Balz Schreier wrote:
> > 1) Regarding "Build a locking mechanism into Yarep": How would that
> really
> > help?
> > Talking of a cluster, we mean multiple JVMs, each of them are running
> Yarep,
> > all accessing a shared file in the filesystem.
> >
> > If you want to solve it with writing a lock file (like Lucene does), then
> > you need to be sure that the lock file gets written synchronously and in
> a
> > single atomic operation (form the JVM down to the harddrive, no buffers,
> > etc.).
> > NFS can not guarantee that.
>
> I was thinking of building the locking into Yarep, e.g. the Yanel nodes
> could communicate with each other in the network in order to implement a
> form of distributed locking. I didn't say it's an easy task though, but
> it would allow the locking to be independent from the backend.
>
> > What I read in the Lucene Book (Lucene in Action, 2nd Edition 2010) is
> the
> > following approach:
> > - The JVMs (and therefore Yarep too) do not write to the local file
> system
> > (reading is ok though), instead they call a central "write service". Of
> > course you have a single point of failure (for writing, reading continues
> to
> > work fine). But you don't have to deal with thinking about locking etc.
> >
> > 2) Regarding "Simply use a backend that can do concurrent writing": How
> do
> > you solve this then:
> >
> > assume you have two JVMs running your app, both with the ability to write
> > nodes.
> > assume that at the same time, both JVMs read a file with a content
> element A
> > in it.
> > The first JVM adds B into the file (A,B), the second JVM adds C into the
> > file (A,C).
> > But "adding" means: read from yarep, modify, write back.
> > So even if you have proper locking in place, what do you end up with on
> the
> > filesystem? A,C or A,B ? What should be in the file is of course A,B,C or
> > A,C,B... and I think that this can only be done with a single "write"
> > service (equal to writing into the same database).
>
> How can you achieve this with a simple writing service?
>
> You'll still have the same problem:
> * Process A reads contents
> * Process B reads contents
> * Process A modifies and sends to write service
> * process B modifies and sends to write service
>
> You will still end up with a file that only contains the modifications from
> process B and the modifications from process A will be lost. You would
> have to make the whole sequence - read, modify, write - atomic. I don't
> see how the writing service does that, but maybe I just don't understand
> what you mean by writing service exactly. Can you elaborate on the idea?
>
> Cheers
> Cedric
> --
> Yanel-development mailing list Yanel-development at wyona.com
> http://lists.wyona.org/cgi-bin/mailman/listinfo/yanel-development
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wyona.org/pipermail/yanel-development/attachments/20110202/55651548/attachment.html>