Highlighted
Super Contributor.. Super Contributor..
Super Contributor..
501 views

Too many open files issue

We have an issue with "too many open files"....  increasing slowly up to the limit and forces us to restart the application.

As the user running ucmdb application (on the ucmdb application server),

 # ulimit -a | grep -i "open files"
open files (-n) 20480

The ucmdb_server process:

# lsof -p 56360 | wc -l
19569

# lsof -p 56360 | grep -i FIN_WAIT2 | wc -l
17164

80% of this is between the ucmdb-server and the probes like this

TCP ucmdbserver.domain.com:pcsync-http->Probeserver1.domain.com:11784 (FIN_WAIT2)

 

Does anyone have any ideas what to do? Apart from restarting the application... 😉 

 

Wbr / Fredrik

 

0 Likes
7 Replies
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Re: Too many open files issue

In the deployment guide is clearly stated thay you have to raise the ulimit:

ulimit.PNG

 Petko

 

 

Likes are appreciated!
0 Likes
Highlighted
Super Contributor.. Super Contributor..
Super Contributor..

Re: Too many open files issue

Well, I guess you have 2 options there and we have increased the open files to 20480  in /etc/security/limits.conf as described in option 1.

It eats slowly up the "open files" and after 1 month you need to restart the application...  and as mention there are a lot of FIN_WAIT2 against the probe servers... 

Wbr / Fredrik

0 Likes
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Re: Too many open files issue

I think that's more important why we have so many open files not how to increase that limit.
Jetty connections, log files on disk, other files opened by other applications, etc.

Kind regards,
Bogdan Mureșan

EMEA Technical Success
0 Likes
Highlighted
Super Contributor.. Super Contributor..
Super Contributor..

Re: Too many open files issue

The open files described here are the ones for the ucmdb_server process, and as mention 80% is FIN_WAIT2 and 8444 communication with the probes... 

TCP ucmdbserver.domain.com:pcsync-http->Probeserver1.domain.com:11784 (FIN_WAIT2)

 

Wbr / Fredrik

0 Likes
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Re: Too many open files issue

We had a regression on this side.
What is the UCMDB version and CP level?

Kind regards,
Bogdan Mureșan

EMEA Technical Success
0 Likes
Highlighted
Super Contributor.. Super Contributor..
Super Contributor..

Re: Too many open files issue

Hi Bogdan!

Our version is HP Universal CMDB 10.30.305 with content pack 21.00.75.

Wbr / Fredrik

0 Likes
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Re: Too many open files issue

I was expecting a version betweenm 10.30-10.33

It was fixed in 10.33 CUP1
https://softwaresupport.softwaregrp.com/group/softwaresupport/search-result/-/facetsearch/document/LID/QCCR1H119632
What you can do on short term is to disable Solr if you don't use the indexing for Browser searches.

Kind regards,
Bogdan Mureșan

EMEA Technical Success
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.