Highlighted
Micro Focus Expert
Micro Focus Expert
1317 views

How to monitor C/S Requests Pending on a POA

Hello Members,

I would like to monitor the "C/S Requests Pending" attribute of 60 POAs every minute without GroupWise Monitor.
It happens from time to time, that C/S Handler Threads are maxed out (99) on random POAs and users complain about login problems, slow client responses. After 10-15 minutes the threads are freed up without any intervention. I would like to investigate this problem when it is in progress, so I need almost real time data and notifying about this. There is a working Icinga system with a well proven action plan and I would like to have a service with the C/S Requests Pending or C/S Handler Threads in it.
I do have GW Monitor but the notifying capabilities are limited compared to that if Icinga.
I can download the web interface (port 7181 )with wget and grep for the required strings but this looks like an overkill for me. I need only one bit of information but I download the whole web page. And on the other side I suspect that the web interface is serviced with the C/S Handler Threads, so if all threads are busy, I cannot reach the web interface either. GW Monitor is not affected by this, it can read the numbers.
I wish I could use the same method as GW Monitor, but all I've found is that it is polling the POAs over XML. Whatever this means.
How does GW Monitor polls the GW agents, and how can I use this method to have my own monitoring in Icinga? I assume it is proprietary and I cannot use it, but if I don't ask, I don't get answers 🙂

Regards,
Gellért
Labels (1)
Tags (1)
0 Likes
9 Replies
Highlighted
Knowledge Partner
Knowledge Partner

Re: How to monitor C/S Requests Pending on a POA

On 21.01.2019 11:54, gehorvath wrote:
>
> Hello Members,
>
> I would like to monitor the "C/S Requests Pending" attribute of 60 POAs
> every minute without GroupWise Monitor.
> It happens from time to time, that C/S Handler Threads are maxed out
> (99) on random POAs and users complain about login problems, slow client
> responses. After 10-15 minutes the threads are freed up without any
> intervention. I would like to investigate this problem when it is in
> progress, so I need almost real time data and notifying about this.
> There is a working Icinga system with a well proven action plan and I
> would like to have a service with the C/S Requests Pending or C/S
> Handler Threads in it.
> I do have GW Monitor but the notifying capabilities are limited compared
> to that if Icinga.
> I can download the web interface (port 7181 )with wget and grep for the
> required strings but this looks like an overkill for me. I need only one
> bit of information but I download the whole web page. And on the other
> side I suspect that the web interface is serviced with the C/S Handler
> Threads, so if all threads are busy, I cannot reach the web interface
> either. GW Monitor is not affected by this, it can read the numbers.
> I wish I could use the same method as GW Monitor, but all I've found is
> that it is polling the POAs over XML. Whatever this means.
> How does GW Monitor polls the GW agents, and how can I use this method
> to have my own monitoring in Icinga? I assume it is proprietary and I
> cannot use it, but if I don't ask, I don't get answers 🙂


You should relatively easy get what you want using SNMP:

https://www.novell.com/documentation/groupwise2014r2/gw2014_guide_admin/data/adm_poa_mon_snmp.html

CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: How to monitor C/S Requests Pending on a POA

Hello Massimo,

Thank you for the idea. I went ahead and set up snmp as described in the documentation.
Unfortunately it doesn't work for me.
When I do a
snmpwalk -v2c -c mysecret myserverip .1.3.6.1.4.1.23.2.38.1

I get only:

End of MIB

I see this in the log files:

GWSNMP: POAhandleGetNextRequest failed for MODE_GETNEXT request - 2
Connection from UDP: [IP1]:52997->[IP2]:161
GWSNMP/XPLAT: Cannot poll any GW agent
GWSNMP: GetMibValue retured error 1
GWSNMP: ProcessSnmpRequests failed for MODE_GET request

I've found this:

https://www.novell.com/documentation/groupwise18/gw18_readme/data/gw18_readme.html#b1g1gwfg

4.2 SNMP doesn’t work on SLES 12 SP2#
SNMP does not currently work on SLES 12 SP2. Use a different OS if you need to use SNMP

Is this still a problem on SLES 12 SP3? I am running on GW on SLES 12 SP3 64 bit.

Regards,
Gellért
0 Likes
Highlighted
Knowledge Partner
Knowledge Partner

Re: How to monitor C/S Requests Pending on a POA

On 21.01.2019 16:44, gehorvath wrote:
>
> Hello Massimo,
>
> Thank you for the idea. I went ahead and set up snmp as described in the
> documentation.
> Unfortunately it doesn't work for me.
> When I do a
> snmpwalk -v2c -c mysecret myserverip .1.3.6.1.4.1.23.2.38.1
>
> I get only:
>
> End of MIB
>
> I see this in the log files:
>
> GWSNMP: POAhandleGetNextRequest failed for MODE_GETNEXT request - 2
> Connection from UDP: [IP1]:52997->[IP2]:161
> GWSNMP/XPLAT: Cannot poll any GW agent
> GWSNMP: GetMibValue retured error 1
> GWSNMP: ProcessSnmpRequests failed for MODE_GET request
>
> I've found this:
>
> https://www.novell.com/documentation/groupwise18/gw18_readme/data/gw18_readme.html#b1g1gwfg
>
> 4.2 SNMP doesn�t work on SLES 12 SP2#
> SNMP does not currently work on SLES 12 SP2. Use a different OS if you
> need to use SNMP
>
> Is this still a problem on SLES 12 SP3? I am running on GW on SLES 12
> SP3 64 bit.


Let me ask around.

CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: How to monitor C/S Requests Pending on a POA

Thank you!

Gellért
0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: How to monitor C/S Requests Pending on a POA

Until I figure out the SNMP monitoring on SLES12, I've set up GW Monitor, and I was able to catch the event where all the GW tcp threads of POA were used. Unfortunately I don't know how to proceed. All I saw was that the GWTCP-Servername-Handler_XX threads were in the State of "GettingInfo" Almost every thread except three. What is this state? What info is GW "getting"? Could this be network related? I know for sure that there are many routers, firewalls, switches between the POA and the windows clients.
0 Likes
Highlighted
Knowledge Partner
Knowledge Partner

Re: How to monitor C/S Requests Pending on a POA

On 25.01.2019 15:24, gehorvath wrote:
>
> Until I figure out the SNMP monitoring on SLES12, I've set up GW
> Monitor, and I was able to catch the event where all the GW tcp threads
> of POA were used. Unfortunately I don't know how to proceed. All I saw
> was that the GWTCP-Servername-Handler_XX threads were in the State of
> "GettingInfo" Almost every thread except three. What is this state? What
> info is GW "getting"? Could this be network related? I know for sure
> that there are many routers, firewalls, switches between the POA and the
> windows clients.
>
>


Have you done any basic troubleshooting yet? My educated guess would
rather be storage aka I/O performance vs. Network issues. Just watching
top when the issue occurs and checking the wait collumn could give a hint.

CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: How to monitor C/S Requests Pending on a POA

The CPU usage is normal, it varies from 0 to 150%, load is under 1, I/O wait is around 0%. netstat -ntp shows that there are about 800 open tcp session with clients for gwpoa process. There are about 600 connected users and 1000 application connections. The abnormal is however, that the Recv-Q and Send-Q is not 0 of these connections. For example

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 68 0 a.b.c.d:1677 internalIP:54748 ESTABLISHED 30042/gwpoa
tcp 69 0 a.b.c.d:1677 externalIP:61992 CLOSE_WAIT 30042/gwpoa
tcp 192 0 a.b.c.d:1677 externalIP:49633 ESTABLISHED 30042/gwpoa
tcp 68 0 a.b.c.d:1677 internalIP:54332 ESTABLISHED 30042/gwpoa
tcp 68 0 a.b.c.d:1677 internalIP:63764 ESTABLISHED 30042/gwpoa
tcp 165 0 a.b.c.d:1677 internalIP:53302 CLOSE_WAIT 30042/gwpoa

After a while I had to stop investigating because users complained about GW being slow, so I restarted the gwpoa. Another annoying thing is that I usually do a "systemctl stop xinetd" before I stop GW, so the GW Monitor won't restart the process automatically. But since the servers were upgraded from SLES11 to SLES12 sometimes "systemctl stop xinetd" stops the gwpoa process also. Even when I don't want to stop gw at all.

Gellért
0 Likes
Highlighted
Knowledge Partner
Knowledge Partner

Re: How to monitor C/S Requests Pending on a POA

Hi.

Back to the original question first: There is a fix available for the
SNMP issue, but you'll have to open a SR and ask for the fix for BUG #968328

As for the question why your POAs get into that situation, I'm afraid
I'm out of ideas for the moment, sorry. If nobody else here has some,
maybe it's another SR...

CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: How to monitor C/S Requests Pending on a POA

Hello Massimo,

thank you for the reply, I will open an SR for the SNMP monitoring.
Yesterday I've opened an SR for the original issue. Will report back when I have results.

Thank you for the help!

Gellért
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.