GWIA, IMAP, CLOSE_WAIT

[mguldner]

This question is related to the problem described here: https://community.microfocus.com/t5/GroupWise-User-Discussions/GWIA-quot-link-recovery-quot-messages/m-p/2702822/highlight/true#M84203

The problem still pops up from time to time, while it does not occur in between for some weeks or month.

I found another symptom and I wonder if this could help finding an explanation.

GROUPWISE_MAX_OPEN_FILE_HANDLES="200000" did not solve the problem. But I found that indeed the GWIA process has a lot of open files when the problem happens. For example currently I see this:

gwia:~ # ll /proc/9234/fd | wc -l
3658

This will go up to about 10000.  This is the soft limit even with GROUPWISE_MAX_OPEN_FILE_HANDLES="200000"

gwia:~ # cat /proc/9234/limits | grep "open files"
Max open files            10240                200000               files

The interesting part: most of these "open files" do not seem to be files but network connections between GWIA and POA; almost all of them are in a state "CLOSE_WAIT" and will never disappear.

gwia:~ # netstat -n | grep "tcp        1      0 141.79.11.18" | grep ":167[78]" | grep "CLOSE_WAIT" | wc -l
2326

Open files and "CLOSE_WAIT" connections will rise simultaneously and will disappear when GWIA is killed. Normaly, when there are no problems, CLOSE_WAIT connections vary between 100 and 200; then for some unknown reason they start to accumulate.

To rule out network problems I installed a GWIA on the same server as one POA - same problem. Network problem without network? Any ideas what the reason for this might be? Why would connections between GWIA and POA hang forever in that state.

This seems to use up resources so that the MTA (!) eventually reports: "Insufficient resources to accept inbound connection"

Thanks,

Mirko

[KBOYLE]

---

@mguldner wrote:

To rule out network problems I installed a GWIA on the same server as one POA - same problem. Network problem without network? Any ideas what the reason for this might be? Why would connections between GWIA and POA hang forever in that state.

---

 Have you checked your GWIA logs? You may want to set logging to Debug to get the details.

It is possible that someone is trying to access your system by repeatedly guessing IMAP passwords or using AUTH LOGIN via SMTP. This could open a lot of connections. Your GWIA log should provide the details.

[mguldner]

Just to be sure: do you mean "diag" setting for logging? Or is there an additional undocumented "debug" setting?

Indeed I see a lot of failed logins. But I find it very difficult to make sense of these, because there is no username and no IP in the same line. Sometimes there is a username in the line before and an IP in the line after - but not always. There also seems be a difference between LOGIN and AUTHENTICATE. Is there an easy way to bring together these data to get an idea were these failed logins come from?

But anyway - would this explain the symptoms? I found this for CLOSE_WAIT:

"CLOSE_WAIT - Indicates that the server has received the first FIN signal from the client and the connection is in the process of being closed. This means the socket is waiting for the application to execute close(). A socket can be in CLOSE_WAIT state indefinitely until the application closes it. Faulty scenarios would be like a file descriptor leak: server not executing close() on sockets leading to pile up of CLOSE_WAIT sockets."

Does this possibly mean, the POA should close the connection, but doesn't? But then I guess I should probably see the same CLOSE_WAIT connections on the server with POA - which I don't.

Thanks,

Mirko

[laurabuckley]

Hi Mirko,

What IMAP client is being used against this GWIA?

What is the exact version and build GroupWise on the server?

Cheers,

Laura

[mguldner]

Hi Laura,
we have many IMAP-users, so I don't know which Clients all of them are using. Clients I know about are Thunderbird, Mac Mail and Outlook. But certainly there are more, for example phones.

Until today I had the release build of GW 18.3. Today I updated to 18.3.0 -137474 (because of another issue, SR#101321412931). After the update I observed a somewhat different behavior: again the CLOSE_WAIT connections went up - and then the GWIA crashed. This did not happen before - I will wait and see if this happens again.

Thanks,
Mirko

[mguldner]

Yes, most probably there are password attacks going on. I see roughly the same number of failed logins as successful logins (numbers from 17th Jan to today):

AUTHENTICATE: OK: 181689, FAILED: 170283
LOGIN: OK: 210575, FAILED: 79060
Sum: OK: 392264, FAILED: 249343

But I still have no idea how this (or something else) causes the CLOSE_WAIT sockets. Currently my workaround is to check for the number of "open files" of the GWIA's process and kill -9 it if there are more than 10000. This happens several times per day. Quite unsatisfactory.

Any more ideas what to look for?

Thanks,

Mirko

[davidkrotil]

Do you have some sort of firewall between rough world and GroupWise ? There are even free products like pfSense, which can block these attacks on network level before they reach GroupWise.