[Yanel-dev] too many open files problem on VMWare Ubuntu

[Michael Wechner]

Hi

I recently experienced this problem again with some custom code and
noticed eventually that the search index files get opened with every
search, but not closed

lsof -p PROCESS_ID_OF_TOMCAT | grep fulltext | wc -l

The reason for this was that the IndexSearcher

org.apache.lucene.search.Searcher searcher = .....;

was not closed as it should be

searcher.close();

After this line was added, all was fine again :-)

Thanks

Michael



Am 08.08.11 um 10:52 schrieb Michael Wechner:
> Hi
>
> In order to improve this it helps to find out which files are actually
> open, e.g.
>
> Get Tomcat process ID: ps ax | grep tom
> Display all open files: lsof -p PROCESS_ID_OF_TOMCAT
> Display number of open files: lsof -p PROCESS_ID_OF_TOMCAT | wc -l
>
> For example on my Mac I have approx. 173 open files currently under
> the tomcat process ID, whereas
> most of them are the jar files.
>
> Cheers
>
> Michael
>
>
>
> Am 23.07.11 11:05, schrieb Michael Wechner:
>> On 4/1/11 1:06 PM, Michael Wechner wrote:
>>> Hi
>>>
>>> We are experiencing a "too many open files" problem on VMWare Ubuntu
>>> when running the lucene re-indexing, whereas this is done on a mac
>>> and if running the lucene re-indexing on Mac OS X everything is fine.
>>
>> It's odd that on Mac OS X it's fine although according to "ulimit -a"
>> one has
>>
>> open files                      (-n) 256
>>
>> whereas on "our" Linux we have
>>
>> open files                      (-n) 1024
>>
>> Well, one might argue that on the Linux system there might be other
>> processes "eating" these, but
>> I couldn't see anything which would be much different than Mac OS X
>>>
>>> I could image that the VMWare Ubuntu is slow on the IO (because of
>>> the various layers) and hence a "queue" of open files is established
>>> which are not being closed fast enough and hence when the limit of
>>> Ubuntu is reached it will "crash".
>>>
>>> I guess we could increase this limit
>>>
>>> #su root
>>> #ulmit -n 1000000
>>
>> An alternative might be to increase it for  a particular user:
>>
>> - sudo su root
>> - vi /etc/security/limits.conf
>> - Add the following lines (whereas 'wyona' should be replaced by the
>> users you are using):
>> wyona           soft    nofile          10240
>> wyona           hard    nofile          20480
>> - exit
>> - start a new shell for the particular user (e.g. 'wyona')
>> - ulimit -a (should tell you that you now: open
>> files                      (-n) 10240
>>
>> This worked for us :-) (but still it would be nice if we wouldn't
>> have to do this in the first place)
>>
>> HTH
>>
>> Michael
>>
>>>
>>> but I don't think that's really solving the problem.
>>>
>>> Anyway it's good to keep this in mind when using virtualization.
>>>
>>> Cheers
>>>
>>> Michael
>>>
>>>
>>>
>>
>