Nightly maintenance Groupwise Mobility Service and email delays
Groupwise Mobility Service Running 18.0.1 Build 285 Sles 12
Has anyone experiences issues with mail is getting delayed sending to devices while nightly maintenance is set to run/running?
We have been noticing this to be the case. Nightly maintenance starts at midnight and runs it looks like it ends approximately 2:00-215 am most evenings . I've tested sending myself a few emails at midnight (client receives them) but there are long delays when receiving it to the device (granted I didn't stay up to watch my device for incoming messages, but it was well over an hour wait. I've had a few users test this and tell me that it seems to get messages to devices again around 2 am). I have checked the dashboard between midnight and 1 am and it shows everything is 'green'.
I check the agent alerts and it shows me these alerts from about 2:15 am - 4:06 am daily for a majority of users:
Time to Process GroupWise Events: It is taking a long time to receive events from GroupWise
Any reason why this lag would occur? Does the nightly maintenance routine hold up the processing of mail to devices?
I haven't seen this behavior
A thought on why this might be happening is a resource constant/mismatch for the POA.
- How many CPUs/cores does the POA have available vs how many GWCheck Worker Threads.
- Is your server stuck waiting for storage while the GWChecks are running? perhaps reduce then some.
- how hard is swap being hit? perhaps we are tight on RAM. I've seen many systems grow their RAM needs just by the users dealing increasing volumes of email.
What is your nightly maintenance? just Structural Analyze/Fix? or if more, what is it doing?
have you checked the GWCheck logs and the POA logs for any errors?
Thanks for the response. Truly I didn't think of checking POA logs as I was just looking at the nightly maintenance run on the GMS server since the times this is happening (email delays between 12-2ish) corresponds to when the maintenance on that server is running.
So the only thing in this time window that is happening on the POA side is a Check fix Structure/Index at 2 am every evening.
On the POA, the gwcheck typically end between 2:15-2:30. There are 2 CPUS/1 core per socket and there are 4 GW Worker threads.There's also 8 GB Ram on the servers (all PO servers are configured the same). It doesn't look like the SWAP is being affected at all.
So i don't know if I'm dealing with two different issues here - with emails being delayed to the device from 12am-2am, then the alerts indicating long time to process events from GroupWise from approx 2 am - 4 am every day.
how have you checked how long the GMS nightly maintenance is taking?
mobility-agent.log grep for MaintenanceMonitor to see when you get the "successful for N out of N users" is what I normally consider.
have you run a dsapp General Health Check? while some of the things don't work anymore on newer GW & SLES versions, it can still give you an idea of any hot spots. Does anyone want to take on updating dsapp now that the last guy maintaining it moved on?
When was the last time you Vacuumed or reindexed the databases?
a thought for your POA is to drop the GW Worker threads to 2 and see how that affects things. note that this will also affect how long all GWChecks run
for your POA, what is it running on? if OES then you have ganglia to see what your server was upto CPU and RAM wise at night, http://yourserverIP/gweb
re: 8GB of RAM, I have a client with just such of 100 users needing over 30GB of RAM to not hit memory issues, but then they do a really strong volume of mail with multiple mailboxes past 100GB in size.