jgosney
New Member.
2694 views

One PO running high threads

On our GW8.0.1 system, we have four primary post offices. All four PO's are configured identical -- same hardware, same software settings -- and the load across all four is fairly balanced, approximately 250 users per post office.

But one of our post offices is consistently running slower than the other three -- I mean in response time. At just about any time of day, if I go out to the PO console, the Busy Threads is sitting up in the 20's while the other 3 post offices rarely go above 5 and they are usually averaging 0.

Since all four PO's settings are the same, I'm looking for guidance as to how to troubleshoot why this PO's Busy Threads are continually running high.
Labels (2)
0 Likes
13 Replies
Anonymous_User Absent Member.
Absent Member.

Re: One PO running high threads

Is the MTA local? Is it a routing domain? What other things could be
causing problems? What about you switch or switch port or patch cable? Did
you recently have the indexer restart from scratch?

>>> On 12/9/2009 at 10:06 AM, jgosney<jgosney@no-mx.forums.novell.com>

wrote:

> On our GW8.0.1 system, we have four primary post offices. All four PO's
> are configured identical ‑‑ same hardware, same software settings

‑‑ and
> the load across all four is fairly balanced, approximately 250 users per
> post office.
>
> But one of our post offices is consistently running slower than the
> other three ‑‑ I mean in response time. At just about any time of

day,
> if I go out to the PO console, the Busy Threads is sitting up in the
> 20's while the other 3 post offices rarely go above 5 and they are
> usually averaging 0.
>
> Since all four PO's settings are the same, I'm looking for guidance as
> to how to troubleshoot why this PO's Busy Threads are continually
> running high.

0 Likes
jgosney
New Member.

Re: One PO running high threads

The MTA is on a separate box and it is a routing domain. But the PO's are all stand-alone machines. "Control" server houses the MTA and GWIA and other stuff.

Have checked the lan ports and between all four boxes, all numbers are comparable. No extreme highs on this problem box for utilization or errors.
0 Likes
Highlighted
jgosney
New Member.

Re: One PO running high threads

I originally started this thread back in December but it didn't generate much response so I'll try again since we are still seeing the same behavior.

Our four post offices are all identically configured. And the user community load is evenly balanced across all four. But we have two POA's that are constantly running high threads. Several times a week, I'll starting getting calls about GW being slow and when I look at the POA's, the threads will constantly be from 80% to 100% of max. And unless I reboot the server, it will stay like that.

We started seeing this problem when we upgraded to GW 8.0.0 and I was hoping a subsequent patch would fix it. But we've gone to 8.0.0hp1, then 8.0.1, and now 8.0.1hp1 and the problem still persists.

I would love if anyone could offer some ideas as to what we are seeing here any ways to resolve it.
0 Likes
Knowledge Partner
Knowledge Partner

Re: One PO running high threads

Are you on NetWare? You may be able to check NRM to see what's cranking up the item (which thread exactly)

Otherwise I think you can use the web console for the POA to see which threads are running

(ie, gwcheck, or the dca or something)

Otherwise have you shut off the dca (document conversion agent). It's very problematic (IMO) on NetWare and can even cause issues on SLES.

If on NetWare, add a:

/nodca

to the .poa file

If on Linux:

--nodca
0 Likes
jgosney
New Member.

Re: One PO running high threads

kjhurni,

thanks for the reply. Yes, we are running on netware. I can use the web-based POA console to see who has the threads open and, using their IP address to correlate, I can then use TCPCON from the console to kill threads. But neither of those two utilities tell me WHICH thread is the one that is hung up or causing issues. And when there are 40 - 50 threads open, guessing isn't really an option.

As for the DCA, we have already disabled indexing of PDF's because of that known issue but turning off DCA entirely is not really an option for us since people do a lot of searching for text in documents. Sadly, our user community uses Groupwise as a file storage system. Do you think it's the GWDCA that is locking the threads open?
0 Likes
Knowledge Partner
Knowledge Partner

Re: One PO running high threads

It's a distinct possibility, IMO.

Your only other choice may be to wait until it happens, break into the debugger and send the coredump to Novell.
0 Likes
isoni Absent Member.
Absent Member.

Re: One PO running high threads

I also have 4 Post Offices in a cluster out of which one of them constantly has issues no matter which Server it is on. I have also turned off the DCA. They load in protected memory and I also get multiple abends and restarts. It seems to be happening on busier days and mostly in the morning. The only difference with this Post Office are the number of Blackberry users.

I also discovered recently that I have users with corrupt appts that show up as 4gb in size. I don't know if this is a factor in all the issues. Also seems worse when the ALL distribution lists are being used or maybe that puts the threads over the edge. I am still trying to resolve the slowness issues that started right after the gw 8.0 upgrade and only on one Post Office out of 7 others.
0 Likes
hspeirs Absent Member.
Absent Member.

Re: One PO running high threads

jgosney,

> thanks for the reply. Yes, we are running on netware. I can use the
> web-based POA console to see who has the threads open and, using their
> IP address to correlate, I can then use TCPCON from the console to kill
> threads. But neither of those two utilities tell me WHICH thread is the
> one that is hung up or causing issues. And when there are 40 - 50
> threads open, guessing isn't really an option.


When the server is showing high utilization, go into NRM ->
Profile/Debug -> Additional Debug Options -> Profile Processor
Execution, then put a 0 (zero) in the Symbol or Address field - hit
execute. It'll show you what thread(s) are stressing the CPU - may help
you narrow down the issue.

H.

0 Likes
MRL Absent Member.
Absent Member.

Re: One PO running high threads

I have another post about quickfinder slowing down the groupwise post office to a standstill. I switched hard drives and do not have my news account setup yet, so I am just putting up this quick post.

Quickfinder running, /nodcs, /qflevels-999,
From POA screen actions, quickfinder, update
On NW65SP8 - in a cluster, in protected memory with all post SP8 patches applied.


I have seen that it is the GWTCP-MRLP Handler that seems to get stuck (blocked on a semaphore) . One client was still connected according to the http interface for the postoffice agent. It showed that the client was in online mode, current transaction = read records, current operation = purge (0) objects and time elapsed was 6000 seconds and counting.

Had to unload the POA and then reload to get back service which stops quickfinder.
Did a gwcheck on the user account - no problems with account.

Joe
0 Likes
Knowledge Partner
Knowledge Partner

Re: One PO running high threads

MRL;1947015 wrote:
I have another post about quickfinder slowing down the groupwise post office to a standstill. I switched hard drives and do not have my news account setup yet, so I am just putting up this quick post.

Quickfinder running, /nodcs, /qflevels-999,
From POA screen actions, quickfinder, update
On NW65SP8 - in a cluster, in protected memory with all post SP8 patches applied.


I have seen that it is the GWTCP-MRLP Handler that seems to get stuck (blocked on a semaphore) . One client was still connected according to the http interface for the postoffice agent. It showed that the client was in online mode, current transaction = read records, current operation = purge (0) objects and time elapsed was 6000 seconds and counting.

Had to unload the POA and then reload to get back service which stops quickfinder.
Did a gwcheck on the user account - no problems with account.

Joe


Hmm, something's definitely not right. I mean normally I don't like to run the qf indexing during production hours, but I have on occasion had to do that (like a complete rebuild of the indexes) and while things may be a little slower than normal, I've never had it hang or anything. Have you actually done a delete and regenerate vs. an update? (I can't remember if you did that already or not).

Again, short of getting a coredump when that happens to send to Novell for analysis. (you'd have to make some changes on the NetWare server or NOT run the POA in protected memory to get an accurate one).
0 Likes
MRL Absent Member.
Absent Member.

Re: One PO running high threads

kjhurni;1947350 wrote:
Hmm, something's definitely not right. I mean normally I don't like to run the qf indexing during production hours, but I have on occasion had to do that (like a complete rebuild of the indexes) and while things may be a little slower than normal, I've never had it hang or anything. Have you actually done a delete and regenerate vs. an update? (I can't remember if you did that already or not).

Again, short of getting a coredump when that happens to send to Novell for analysis. (you'd have to make some changes on the NetWare server or NOT run the POA in protected memory to get an accurate one).

----------------

Thanks for the help

Yes I've rebuilt the indexs a couple times (or at least tried to) from scratch
Quickfinder is disabled from running at the moment. I did not let it start at all or over the weekend. I restarted the POA by itself on Saturday evening

Currently the POA is running on the server with high utilization 50%-80%. If I stop the POA and restart it it will be down in the 3%-5% ranges without restarting the server. Just stopping the POA from running in protected memory and the restarting it. There is an MTA that runs fine and doesn't get touched, also in protected memory.

I've set the profile processor execution to processor 0 in Norm and taken some screen shots as well to attempt to see what is going on. Did the same for Processor 1

server and load on processor 0 take up 20%, 15% - 35%, 23% etc respectfully

on processor 1 Gwenn5 and Libc- 13%, 12% - 11%, 10% respectfully
Sometimes loaded gets in there with a 6%-8% load
Plus a whole bunch of other gwenn5 in the 1-3% range
0 Likes
wrossing Absent Member.
Absent Member.

Re: One PO running high threads

jgosney;1942245 wrote:
I originally started this thread back in December but it didn't generate much response so I'll try again since we are still seeing the same behavior.

Our four post offices are all identically configured. And the user community load is evenly balanced across all four. But we have two POA's that are constantly running high threads. Several times a week, I'll starting getting calls about GW being slow and when I look at the POA's, the threads will constantly be from 80% to 100% of max. And unless I reboot the server, it will stay like that.

We started seeing this problem when we upgraded to GW 8.0.0 and I was hoping a subsequent patch would fix it. But we've gone to 8.0.0hp1, then 8.0.1, and now 8.0.1hp1 and the problem still persists.

I would love if anyone could offer some ideas as to what we are seeing here any ways to resolve it.
0 Likes
wrossing Absent Member.
Absent Member.

Re: One PO running high threads

We are having this exact same issue, all started with the 8.0.1 version, I did start a new thread. Hopefully we can get this resolved.
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.