Micro Focus Expert
Micro Focus Expert
4254 views

Another High utilization on POA question

Hello,

I have a GW server which behaves strange in a negative meaning. Its a GW 2014 R2 HP1 system on SLES11 SP4. The situation is, that the load on the server goes up to 30ish, gwpoa is using a max cpu. The result is slow client connections and users complains. I have turned on Performance Snapshots on the POA and the results are strange. Everything is per minute!
CPU usage sometimes goes over 1000% yes, 1000%. I don't know what this mean, as the system has only 2 cores.
Disk Read (KB) is 2116830.
Disk write (KB) 143478.
Disk Read Count 1050980
Disk Write Count 21611
App connection is about 60, everything else is low. Users are mainly using WebAccess and IMAP.
Of course there is nothing spectacular in the logs. IMAP logins, message distributions, and so on.

This in only 1 server from the total of 45. The others does not have this situation or at least I have not recognised it yet.
My question is that how can I find out what the poa agent is doing? Why is it reading 2 GB / minute from the disk?

There is no GWcheck in progress, no quickfinder indexing.
After users go home, the load goes back to normal.

Regards,
Gellert
Labels (1)
0 Likes
22 Replies
Micro Focus Expert
Micro Focus Expert

Re: Another High utilization on POA question

Hi Gellert,

Just a quick question - did you ever adjust the imap read limit for the POA or is it at default?

Look in your POA startup file for the switch. If commented out it's still at default, but if not, let us know what number it is set to.

Please let us know.

Cheers,
Laura Buckley

Views/comments expressed here are entirely my own.
If you find this post helpful, please show your appreciation and click on "Like" below...
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Another High utilization on POA question

Hello Laura,

imap read limit is at max, 65K
imapreadfirst is also enabled.

I had to max out imap read limit, because users were complaining, that they cannot download all emails in the inbox folder, and they get a warning from Thunderbird that there are more emails but it it unable to download it. I've checked the mailboxes and there are indeed a few thousand emails in the inbox.

TIA,
Gellert
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Another High utilization on POA question

Hi Gellert,

I think that's your answer for why your POA is maxing out.

You could try the --udpslap switch to attempt improving performance.

Cheers,
Laura Buckley

Views/comments expressed here are entirely my own.
If you find this post helpful, please show your appreciation and click on "Like" below...
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Another High utilization on POA question

Hello Laura,

If I lower the imapreadlimit then users with a lots of emails in one folder cannot download them. If I max out imapreadlimit, then the POA gets a high load and everybody on the same PO suffers. 🙂
This high imapreadlimit setting is the main reason that the poa reads GBs from the disk per minute?
I will try the udpslap switch.

Look, what I have found : https://www.novell.com/support/kb/doc.php?id=7014986

I will report back when I have the results.

Regards,
Gellert
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Another High utilization on POA question

Hi Gellert,

I can only imagine the amount of strain on a POA with people pulling so much mail via IMAP with such a high IMAP read limit.

Besides udpslap, perhaps increase the memory - not too sure if that will help.

Cheers,
Laura Buckley

Views/comments expressed here are entirely my own.
If you find this post helpful, please show your appreciation and click on "Like" below...
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Another High utilization on POA question

Hello Laura,

I have to say, that udpslap switch didn't helped at all. I also lowered imapreadlimit to 40k. After 8 hours of testing, the load still goes up sometimes to 30ish, usually it is around 5. The load goes back to under 1 only after work hours. cpu spends 40-50% in io wait and poa reads hundreds of mb, or more every minute. Tomorrow I will compare the performance data of two gw servers. This one and a server that does not shows the symptoms. Maybe it helps.
Oh, and one more thing. GMS is complaining, that "It is taking a long time to receive events from GroupWise". Atm the value is over 1h.

Gellert
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Another High utilization on POA question

Hi Gellert,

With regards to GMS - I can only imagine that your POA is simply too busy to respond via SOAP. Please let us know your findings when comparing the two servers. You could have a disk I/O problem whereby your storage subsystem is just not fast enough to cope with what GroupWise is expecting it to do.

Cheers,
Laura Buckley

Views/comments expressed here are entirely my own.
If you find this post helpful, please show your appreciation and click on "Like" below...
0 Likes
Knowledge Partner
Knowledge Partner

Re: Another High utilization on POA question

i'd check, x-check, double- and triple-check that there are no errors of any kind on indexing. especially on imap access without read limit missing / corrupt indexes are a major performance killer as the agent has to "manually" walk the box every few seconds. i've once been at a site where indexing was failing on a box after at somewhat 5000 out of 300000 (and counting) items, it was failing for months without anyone taking notice as it has been a service mailbox which was polled every 2 seconds by some daemon, effectively maxing out the entire server, especially on I/O. once index files for this single mailbox had been properly created all went back to normal.
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Another High utilization on POA question

Hello Laura,

New observation:
When the load goes high (30), gwpoa is using 200% cpu (2 cpus available) then io wait is 0, although poa reads over 1.5 GB every minute according to the GW Performance Snapshots. No indexing, no gwcheck in progress.
Then after a while, poa utilization goes back to "normal" and io wait goes up to 40-70%.

I've let GW collect Performance Data for a day, then I tried to find something suspicious in the data, but I've found nothing. The only correlation I found is that when cpu usage goes high then the Data Read goes high also. IMAP, SOAP, C/S are independent from everything.

Quickfinder is suspicious and not the same time. Quickfinder is set up to run every 4 hours from 20 hours. So it should run at 20:00, 00:00, 04:00, 08:00, 12:00, 16:00. But I see only 1 quickfinder process starts at 20:00. There is nothing in the logs from the other runs. But the process that starts at 20:00 does its job. I see log entries like "Updating QuickFinder index: usereh8.db (13) ... at the end QuickFinder indexing thread finished" and it runs for 33 minutes. The last modified date of the index files are: 500 files are up to date, 1200 is outdated. Now I try to reindex the whole PO. This will take a while.

How to rebuild a user's quickfinder index
https://www.novell.com/support/kb/doc.php?id=7012997

This was a big problem for me after the upgrade to R2 and IMHO is still not solved 100%:
POA is almost constantly coring in DVA process after upgrade to R2.
https://www.novell.com/support/kb/doc.php?id=7017203
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Another High utilization on POA question

Hi,

A little known, and often misunderstood, fact about the Quick Finder Indexing process is that it will run from it's offset from Midnight. It will run until midnight and then won't run again until the time specified in the offset is reached. So you will need to adjust your offset from midnight to an hour early in the morning in order for your indexing process to run at every interval expected by you. You could very well be in a position where your indexing is behind due to only running once a day. Also, if not adjusted in the POA startup file, only the first 500 messages per database are indexed. To adjust the QF index to index absolutely everything you need to add the --qflevel 999 switch to your POA startup file.

I would recommend that you adjust your midnight offset. Then on a Friday add the qflevel switch to your POA startup file. Stop/Start your POA and just let the index process run for the weekend. That should give it sufficient time to catch up, but of course that depends on how outdated your indexes are.

Cheers
Laura Buckley

Views/comments expressed here are entirely my own.
If you find this post helpful, please show your appreciation and click on "Like" below...
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Another High utilization on POA question

Hello Laura,

As far as I see, the reindexing is done, but I don't know if it was a "absolutely everything" reindexing, because it took only 1 hour.

I used this method: https://www.novell.com/support/kb/doc.php?id=7012997
Maybe I should use this instead (on Friday): https://www.novell.com/support/kb/doc.php?id=D7017730

This is in the logs for a user. What does the number in brackets mean? Is it the number of items in the mailbox?

17:13:38 E04E Compressing QuickFinder index: userzld.db (128)
17:13:48 E04E Updating QuickFinder index: userzld.db (128)
17:13:49 E04E QuickFinder: 10 items indexed
17:13:49 E04E QuickFinder: 20 items indexed
17:13:50 E04E QuickFinder: 30 items indexed
17:13:51 E04E QuickFinder: 40 items indexed
17:13:51 E04E QuickFinder: 50 items indexed
17:13:52 E04E Compressing QuickFinder index: uservod.db (0)

Regards,
Gellert
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Another High utilization on POA question

Hi Gellert,

The number in brackets indicates the number of items that were indexed for that user.

I don't really have an opinion on the second TID as I tend to do it as I described in my previous post. Possibly because I'm too lazy to do it one user at a time. I don't believe it is necessary for you to delete your existing indexes as that just creates more work for you and the poor POA.

Cheers,
Laura Buckley

Views/comments expressed here are entirely my own.
If you find this post helpful, please show your appreciation and click on "Like" below...
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Another High utilization on POA question

Hello Laura,

I've rescheduled the indexing job for every PO. Offset is 1 hour, interval is 4 hours. I've checked the result of the last run, and it has indexed less then 10 mails (most mailboxes had 0) for every mailbox, so I assume, that indexes are up-to-date. But I still see outdated files in the index folder of the PO which makes me concerns. Those are idx and inc files and some dates back to 2015. There are 148 inactive mailboxes in this PO. Could it be, that those indexes belongs to the inactive mailboxes If yes, than this is not a problem I assume.
I've also doubled the memory and cpu for this PO. It has now 8 GB ram and 4 cpus. Unfortunately the symptoms stayed the same, gwpoa just uses 400% cpu instead of 200%.
How can I otherwise check if indexes are corrupt or not?
Based on your suggestion I have not done the delete and reindex for every mailbox. I just tried the process for 1 mailbox to see how it works.

Regards,
Gellert
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Another High utilization on POA question

Hi Gellert,

If you are concerned about old/outdated index files accumulating on your system I would suggest that you run a GWCheck against the Post Office, only tick Structure, no content as that would take too long, tick User and untick Message database as that's not required for the information we are going to gather. In the resultant log file you will see something like the following:

===============================================================================
Checking user = LauraB (72i / 237) 602112 bytes, 06/30/16 14:00 (Laura Buckley [])
- Current time 2:00:26 pm
STRUCTURAL VERIFICATION of database /media/nss/GWVOL/po1/ofuser/user72i.db
Verifying data
- Database is structurally consistent
Reducing database /media/nss/GWVOL/po1/ofuser/user72i.db
6144 bytes reclaimed
Quickfinder: Pending jobs = 0 Files = 576ec6b0.idx, 577525f9.inc


Note the last line - that identifies what index files belong to the user in question. Perhaps use this information to cross reference your inactive mailboxes with the outdated index files?

As for your constant high utilization - do you have a DVA running on that server?

Please let us know.

Cheers,
Laura Buckley

Views/comments expressed here are entirely my own.
If you find this post helpful, please show your appreciation and click on "Like" below...
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.