Highlighted
Acclaimed Contributor.
Acclaimed Contributor.
2111 views

Can someone please explain to me if _queue files in agentdata is supposed to be there?

Jump to solution

Hey everyone!

I have looked at many threads posted on the forums, and there has been several information posts about pre and post processing cache. Most say that .cache files is post, and _queue files in pre, and that pre is quite normal if they empty out in the end.

Though i found quite a disturbing pattern on my infrastructure. We have 15+ loggers, and even more connectors. And i found out that for all my connectors, there is always a _queue file for the Logger destination.

This never gets empty, and you will see in the connector logs that the old fileread thread is stopped, and a new one is started to continue, so i get insane numbers like so:

 3nAb2emEBABCABH1OoH3BVw==_queue.syslogd.75315

On the Loggers i see the logs coming in, but i also always see this for every connector:

 [smartmessage01[HttpsReceiverThread]] Receiver timed out while waiting for packet, flushed chunk on port 0 for storage group 648518346341351424 using flush wait time of 10000

I get a bit worried when this is something that happens on all our connectors + loggers without fail.

Setting multithreading on pre-parsing is tried and does not help, increasing heap size has been tried, multithreading towards the logger has been configured as well without any help.

So i am guessing that the receiver timeout is where i need to debug? But there is no significant load on the Logger, and other logs are not showing me anything.

Has anyone seen this before? Is it normal?

 UPDATE:

For each logger ET=Up but HT=Down (so events are accepted, but heartbeat does not work). Is this anyway related?

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
1 Solution

Accepted Solutions
Highlighted
Honored Contributor.
Honored Contributor.

1.  There's always going to be one queue file.  If you see more than one tho, there is likely an issue.  Also files in agentdata dir can become stale and not used (say if you change the configuration like a destination, the previous destination cache files will remain and never be processed).  If you think this may apply to you, stop the conn, clear the agentdata dir and start fresh to see if issue reoccurs.

2.  Queue files are not specific to destinations (IE it's not your Loggers, it's the connectors, they can't keep up).  The ArcSight Load Balancer is a full proof way to fix this.

3.  That Logger error may or may not be related, but the following property may fix (add to agent.properties on problem connector): 

transport.loggersecure.connection.persistent=true


4.  Are the connectors using TCP?  Does switching to UDP help?  Or if TCP, you can try tuning the tcp params in agent.properties (or even at the OS level).

5.  Some additional performance enhancements that help in our environment (not specific to queuing, but may help overall)

http.transport.threadcount=4
http.transport.queuesize=2000
transport.loggersecure.threads=4
syslog.parser.multithreading.enabled=true
syslog.parser.threadcount=4
agents[0].usecustomsubagentlist=true
agents[0].customsubagentlist=list|only|the|subagents|you|need

Be sure to change batch size to 600 for each destination via runagentsetup.   Also check how often conns are Full GC in the logs, and update heap accordingly

View solution in original post

3 Replies
Highlighted
Honored Contributor.
Honored Contributor.

1.  There's always going to be one queue file.  If you see more than one tho, there is likely an issue.  Also files in agentdata dir can become stale and not used (say if you change the configuration like a destination, the previous destination cache files will remain and never be processed).  If you think this may apply to you, stop the conn, clear the agentdata dir and start fresh to see if issue reoccurs.

2.  Queue files are not specific to destinations (IE it's not your Loggers, it's the connectors, they can't keep up).  The ArcSight Load Balancer is a full proof way to fix this.

3.  That Logger error may or may not be related, but the following property may fix (add to agent.properties on problem connector): 

transport.loggersecure.connection.persistent=true


4.  Are the connectors using TCP?  Does switching to UDP help?  Or if TCP, you can try tuning the tcp params in agent.properties (or even at the OS level).

5.  Some additional performance enhancements that help in our environment (not specific to queuing, but may help overall)

http.transport.threadcount=4
http.transport.queuesize=2000
transport.loggersecure.threads=4
syslog.parser.multithreading.enabled=true
syslog.parser.threadcount=4
agents[0].usecustomsubagentlist=true
agents[0].customsubagentlist=list|only|the|subagents|you|need

Be sure to change batch size to 600 for each destination via runagentsetup.   Also check how often conns are Full GC in the logs, and update heap accordingly

View solution in original post

Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Ah, if it's just 1 queue file then everything is fine! It always bothered me a bit, and together with that Logger error i thought it might have had something to do with it, as the queue file is tagged with the ID of the logger.

I can check out persistent on some, but other logger clusters are peered, so there i cannot use it.

Another thing i noticed is that Logger destinations always comes with "E=Up, H=Down", but heartbeat is supposed to run on the same port right? We use non standard port for Logger destinations (9000).

I did enable some more threading on certain machines yesterday to check if it helped, and on most it does, except 2 which are always filled with m1.cache (which is supposed to be internal events?)

FullGC is fine, does not happen often, we normally run these with 512MB heap space.

The queuesize and only specific parser i have tried yet, il note those down thanks!

I also noticed int he older connector tweaking guides referring to "tcppeerclosedchecktimeout", though i did not see any entry for that in the agent.default.properties. It might help to run a check and close it, so the logger does not timeout and keep the threads for too long.

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
Highlighted
Honored Contributor.
Honored Contributor.

Regarding the IDs, every connector is assigned an ID:

agents[0].id=<ID>

And that same ID is used on the *first* destination, but will change on each subsequent dests:

agents[0].destination[0].agentid=<same as agent ID>
agents[0].destination[1].agentid=<new ID>


Queue files use the ID of the connector.

All of our Loggers are peers and we use that setting, but I may be misunderstanding what you're saying.

Also for that TCP setting I recall back in the day it was never in agent.properties and you had to specifically add it.  But that was on old versions and I'm pretty sure it's there by default now.  And I just checked a new 7.5.0 connector of ours and it's there (granted it's "-1").  Maybe upgrade software versions?  In any case, might be worth a shot to add/tweak it.

The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.