<meta charset="utf-8">1) Regarding "Build a locking mechanism into Yarep": How would that really help?<div>Talking of a cluster, we mean multiple JVMs, each of them are running Yarep, all accessing a shared file in the filesystem.</div>
<div><br></div><div>If you want to solve it with writing a lock file (like Lucene does), then you need to be sure that the lock file gets written synchronously and in a single atomic operation (form the JVM down to the harddrive, no buffers, etc.).</div>
<div>NFS can not guarantee that.</div><div><br></div><div>What I read in the Lucene Book (Lucene in Action, 2nd Edition 2010) is the following approach:</div><div>- The JVMs (and therefore Yarep too) do not write to the local file system (reading is ok though), instead they call a central "write service". Of course you have a single point of failure (for writing, reading continues to work fine). But you don't have to deal with thinking about locking etc.</div>
<div><br></div><div>2) Regarding "<meta charset="utf-8"><span class="Apple-style-span" style="font-family: arial, sans-serif; font-size: 13px; border-collapse: collapse; ">Simply use a backend that can do concurrent writing": How do you solve this then:</span></div>
<div><font class="Apple-style-span" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse: collapse;"><br></span></font></div><div><font class="Apple-style-span" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse: collapse;">assume you have two JVMs running your app, both with the ability to write nodes.</span></font></div>
<div><font class="Apple-style-span" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse: collapse;">assume that at the same time, both JVMs read a file with a content element A in it. </span></font></div>
<div><font class="Apple-style-span" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse: collapse;">The first JVM adds B into the file (A,B), the second JVM adds C into the file (A,C).</span></font></div>
<div><font class="Apple-style-span" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse: collapse;">But "adding" means: read from yarep, modify, write back.</span></font></div><div><font class="Apple-style-span" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse: collapse;">So even if you have proper locking in place, what do you end up with on the filesystem? A,C or A,B ? What should be in the file is of course A,B,C or A,C,B... and I think that this can only be done with a single "write" service (equal to writing into the same database).</span></font></div>
<div><font class="Apple-style-span" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse: collapse;"><br></span></font></div><div><font class="Apple-style-span" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse: collapse;">Cheers<br>
</span></font><br><div class="gmail_quote">On Wed, Feb 2, 2011 at 6:27 PM, Cedric Staub <span dir="ltr"><<a href="mailto:cedric.staub@wyona.com">cedric.staub@wyona.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">On Wed, Feb 02, 2011 at 06:21:33PM +0100, Michael Wechner wrote:<br>
> Hi Balz<br>
><br>
> On 2/2/11 5:55 PM, Balz Schreier wrote:<br>
> > Hi Michael,<br>
> > interesting point is mentioned in the restrictions:<br>
> ><br>
> > * Only one cluster node writes to the repository. This means that<br>
> > the authoring environment is only accessible on one dedicated<br>
> > cluster node, which is called the /master/ node.<br>
> ><br>
> ><br>
> > For the project I'm working for this is not a solution as each node<br>
> > must have write access to the repository as well as to the Lucene index.<br>
><br>
> agreed and I don't think you need to worry<br>
><br>
> > This is what I wanted to highlight in today's workshop that<br>
> > Yanel/Yarep should come up with a concept of a single "write" instance<br>
> > which gets used by all members in the cluster so that writing is also<br>
> > possible for other nodes... to be continued....<br>
><br>
> <a href="http://wyona.com" target="_blank">wyona.com</a> is running within a cluster and so far we didn't have any<br>
> problems.<br>
> Also please see Cedric's tests, whereas to be fair these tests were done<br>
> only with concurrent read access.<br>
> But we will also do concurrent write/read.<br>
<br>
</div>I don't think doing concurrent writing with NFS will work though.<br>
We have to make sure only one process can write to a file at once.<br>
<br>
Some of my thoughts about possible solutions:<br>
<br>
* Build a locking mechanism into Yarep. This would solve the problem,<br>
but I think it will be very difficult to build a good, solid locking<br>
mechanism.<br>
<br>
* Simply use a backend that can do concurrent writing, e.g. a good<br>
distributed filesystem or a database. It's easier, but not always<br>
practical, e.g. what if someone has already deployed a given<br>
filesystem somewhere and we can't change it etc.<br>
<br>
Just my thoughts,<br>
Cedric<br>
<font color="#888888">--<br>
</font><div><div></div><div class="h5">Yanel-development mailing list <a href="mailto:Yanel-development@wyona.com">Yanel-development@wyona.com</a><br>
<a href="http://lists.wyona.org/cgi-bin/mailman/listinfo/yanel-development" target="_blank">http://lists.wyona.org/cgi-bin/mailman/listinfo/yanel-development</a><br>
</div></div></blockquote></div><br></div>