Highlighted
Acclaimed Contributor.. Acclaimed Contributor..
Acclaimed Contributor..
587 views

Urgent: Invalid sysUpTime messages in statepoller-logs

Jump to solution

Hi

We have recently upgraded to NNMi 10.10 on Linux in both a test- and a prod-environment. We have installed the latest patch 2 (NNM1010L_00002) on both servers.

I have today discovered that the statepoller.trace.log.<n> files get FLOODED with messages like this:

2016-12-06 08:04:53,249 INFO [com.hp.ov.nms.statepoller.logger.StatePollerTraceLogger] (Collector_PollerResultProcessor0) For the poll of policy SNMP Agent Availability Monitoring for measurement target SNMPMeasurementTarget 10.1.2.3 (146029099312) timeout (5000) retries (2) version (2) the following non-repeater data is invalid: sysUpTime: Invalid sysUpTime value. The current sysUpTime value of 2,755,322,992 is less than or equal to zero. sysUpTime:.1.3.6.1.2.1.1.3.0

From the prod-server it is not easy to see when this started, as the files get flooded and all files only cover a time span of a few hours.

From the test server, however, it becomes clear that this started to happen as soon as I had installed patch NNM1010L_00002 !!!

I have found the knowledge document QCCR1B146589 (https://softwaresupport.hpe.com/group/softwaresupport/search-result/-/facetsearch/document/KM02487041) mentioning messages like that. Apparently there is a fix for 10.20 that makes sure that no rediscovery is made due to these messages. For 10.10 the status is 'Closed-No Change', however.

On our testserver we have experienced problems the last few days regarding stale collections etc. I *fear* that this is related to this issue, but I have no evidence of this. So far we have not had that problem on the production server, but it is very important for us to get some more information regarding these worrying log entries in the statepoller logfiles.

Is there a hotfix for 10.10? Should we uninstall patch NNM1010L_00002 again?

I eagerly await your replies!

BR,
Frank Mortensen
Managon AB

0 Likes
1 Solution

Accepted Solutions
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: Urgent: Invalid sysUpTime messages in statepoller-logs

Jump to solution

Hi Frank,

Let me share the end of the story here.

After applying NNMi 10.10 Patch3 (NNM1010_00003), there were  only 31 instances of  such “Invalid sysUpTime value” messages left in the statepoller.trace.log file, all related to devices that had been up for more than 248.5 days.

The root cause is an improper variable being used for the sysUpTime comparison, causing sysUpTime to be improperly interpreted as a negative value.

Defect QCCR1B152969  got created and exported to SSO.

All the best

Marie-Noelle

Micro Focus Customer Support

If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.

If you are satisfied with anyone’s response please remember to give them KUDOS and show your appreciation.

View solution in original post

0 Likes
21 Replies
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: Urgent: Invalid sysUpTime messages in statepoller-logs

Jump to solution

Hi Frank,

For 10.10, a Hotfix ia available for this issue on top of both 10.10 and 10.10 Path1:

HOTFIX-NNMI-10.10-DISCOVERY-20161031
HOTFIX-NNMI-10.1XP1-DISCOVERY-20161031

I would recommend you to install Patch1 and the HotFix on top of Patch1.

All the best

Marie-Noelle

Micro Focus Customer Support

If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.

If you are satisfied with anyone’s response please remember to give them KUDOS and show your appreciation.
0 Likes
Highlighted
Acclaimed Contributor.. Acclaimed Contributor..
Acclaimed Contributor..

Re: Urgent: Invalid sysUpTime messages in statepoller-logs

Jump to solution

Thanks Marie, but like I said we already have patch 2 installed (NNM1010L_00002).

I strongly suspect that this issue was introduced with this patch. What do you suggest that we do, based on this scenario?

0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: Urgent: Invalid sysUpTime messages in statepoller-logs

Jump to solution

Hi Frank,

The right Hotfix that had been created on top of 10.10 Patch 1 for this issue was a statepoller one:
HF-NNMI-10.1X-STATEPOLLER-20160922.

It is possible that the fix was not in Patch2, which means a new HotFix would need to be built on top of Patch2 then.

Could it be that you had already the StatePoller Hotfix installed on you 10.10 system prior to installing Patch2, and the Patch2 installation has overriden the fix for that issue?

Let me check this further and come back to you.

All the best

Marie-Noelle

Micro Focus Customer Support

If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.

If you are satisfied with anyone’s response please remember to give them KUDOS and show your appreciation.
Highlighted
Acclaimed Contributor.. Acclaimed Contributor..
Acclaimed Contributor..

Re: Urgent: Invalid sysUpTime messages in statepoller-logs

Jump to solution

Thanks for investigating this further!

These were the patches and hotfixes we had *before* installing patch 2 the other day:

Patch Independent Hotfix List

=============================

    Hotfix-NNMI-10.1X-Security-20160921                          : installed  : Fri Nov 18 08:22:15 2016

    Hotfix-NNMI-10.1X-NmsJdk-Linux-20160218                      : installed  : Fri Nov 18 08:32:42 2016

 

 

Patch Dependent Hotfix List

===========================

  NNM1010_Base

    Hotfix-NNMI-10.1XPx-MON-CONFIG-20160405                      : installed  : Thu Nov 17 15:49:04 2016

    Hotfix-NNMI-10.10-DISCOVERY-20160928                         : installed  : Thu Nov 17 15:49:39 2016

 

So we did in other words *not* have that statepoller hotfix installed. 

Cheers,
Frank

0 Likes
Highlighted
Acclaimed Contributor.. Acclaimed Contributor..
Acclaimed Contributor..

Re: Urgent: Invalid sysUpTime messages in statepoller-logs

Jump to solution

Hi again

While waiting for an update in this thread, I have received a copy of theHF-NNMI-10.1X-STATEPOLLER-20160922 hotfix from HPE Support. Just to have it available 🙂

Reading its readme-file it claims to deal with the following issue (as well as one more, not relevant right now):

"QCCR1B146589 - Invalid sysUpTime values (negative or constant) cause rediscovery to be requested when it should not."

I do not know whether we actually suffer from a problem where these negative or constant values do indeed cause rediscovery. What I know, however, is that NNMi in those statepoller-logs throw LOADS of messages regarding the polled sysUpTime-values being negative, although they are not at all so. They are all high and positive.

Furthermore, I have looked through all the .jar and .ear files contained in the hotfix. They alle have the EXACT same file size as the files that are currently installed on our NNMi 10.10 P2 server, so I RECKON that this hotfix is indeed contained in Patch 2?

So I am now afraid that we do not suffer from the exact problem that the hotfix addresses AND that we nonetheless have that hotfix installed since we have Patch 2 installed....

I'd appreciate any comments on this.

Cheers,
Frank 

0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: Urgent: Invalid sysUpTime messages in statepoller-logs

Jump to solution

Hi Frank,

I am trying to sort this out in the back scene but if the issue is urgent I would recommend  a case to be opened to Support in parallel. 

Could you please check the value of the following properties in the statepoller properties file:

/var/opt/OV/shared/nnm/conf/props/nms-statepoller.properties

com.hp.ov.nms.statepoller.statemapper.snmpAgentRestartConfigCheckEnabled
com.hp.ov.nms.statepoller.statemapper.snmpNoSuchObjectRediscoveryEnabled
com.hp.ov.nms.statepoller.statemapper.snmpAgentUnresponsiveTriggersDiscovery

Setting these properties to false would limit the number of asynchronous rediscovery triggered by statepoller upon state changes.

This could at least be tried as a workaround until the root cause of these false "negative" sysUpTime values notifications could be narrowed down further and fixed.

All the  best

Marie-Noelle

Micro Focus Customer Support

If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.

If you are satisfied with anyone’s response please remember to give them KUDOS and show your appreciation.
Highlighted
Acclaimed Contributor.. Acclaimed Contributor..
Acclaimed Contributor..

Re: Urgent: Invalid sysUpTime messages in statepoller-logs

Jump to solution

Hi again. I appreciate your help!

The light is currently amber, but not red, as the issue does not seem to affect the functionality of the production server at the moment (the test server has more severe issues with statepoller stale collections, but I do not know whether that is caused by this issue or something else). I am somewhat nervous regarding the prod server, though, due to the state of the test server...

I appreciate your workaround tip, but the nms-statepoller.properties file does not at all exist on any of our servers (!) How come?

It does not even exist on the old 10.0 server that my customer used before. The following files reside in the /var/opt/OV/shared/nnm/conf/props/ directory:

-rw-r--r--. 1 root nmsgrp 7301 Dec 1 06:59 nms-apa.properties
-rw-r--r--. 1 root nmsgrp 10597 Dec 1 06:59 nms-cluster.properties
-rw-r--r--. 1 root nmsgrp 4591 Dec 1 06:59 nms-communication.properties
-rw-r--r--. 1 root nmsgrp 2782 Dec 1 06:59 nms-custompoller.properties
-rw-r--r--. 1 root nmsgrp 12456 Dec 1 06:59 nms-disco.properties
-rw-r--r--. 1 root nmsgrp 12632 Dec 1 06:59 nms-jboss.properties
-rw-r--r--. 1 root nmsgrp 1254 Dec 1 06:59 nms-mon-config.properties
-rw-r--r--. 1 root nmsgrp 5623 Dec 1 06:59 nms-support.properties
-rw-r--r--. 1 root nmsgrp 1889 Dec 1 06:59 nms-topology.properties
-rw-r--r--. 1 root nmsgrp 13805 Dec 1 06:59 nms-ui.properties
-rw-r--r--. 1 root nmsgrp 2333 Dec 1 06:59 nnmaction.properties
-rw-r--r--. 1 root nmsgrp 1663 Dec 1 06:59 nnm-logging.properties
-rw-r--r--. 1 root nmsgrp 2657 Dec 1 06:59 nnm-server.properties
-rw-r--r--. 1 root nmsgrp 10268 Dec 1 06:59 nnmtrapserver.properties
-rw-r--r--. 1 root nmsgrp 3173 Sep 14 11:30 ovjboss.jvmargs
-rw-r--r--. 1 root nmsgrp 2914 Sep 13 12:28 ovjboss.jvmargs-original

On this server patch 2 was installed on Dec 1, so it seems that most of the files were updated by that patch.

Cheers,
Frank

 

 

0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: Urgent: Invalid sysUpTime messages in statepoller-logs

Jump to solution

Hi Frank,

You are right, the nms-statepoller.properties file does not exist by default because these statepoller properties are internal ones that are not documented by default for now.
However, you can create the file and set any of them to false if you want to limit the number of asynchronous rediscoveries triggered by the StatePoller upon state changes.

If the jmx-console has been enabled using the steps of KM02054160, you can check the current value assigned to these properties from CLI using the following nnmtwiddle.ovpl commands:

/opt/OV/support/nnmtwiddle.ovpl get com.hp.ov.nms.statepoller:name=StateMapper StateMapperSNMPAgentRestartConfigCheckEnabled

/opt/OV/support/nnmtwiddle.ovpl get com.hp.ov.nms.statepoller:name=StateMapper StateMapperNoSuchObjectRediscoveryEnabled

I could not find the equivalent of the 3rd property available as RW boulean attribute from the jmx-console on a local test system so I am not sure if the 3rd one is still a valid one.

Now regarding the sysUpTime values returned by some Device Agents that are wrongly interpreted as negative value on the NNMi side, it could be that NNMi is not using an unsigned integer for the comparison right now hence if the corresponding devices have not been rebooted for a very long time, it would result in the value to be wrongly interpreted as a negative value. If you have test devices on the management domain of your devlelopment server, you may want to reboot them to see if the sysUpTime messages disappear for these devices afterwards. If this is the case, it would be definitively a Defect on our code, so I suggest that you open a case to Support to get this fixed.

Cheers

Marie-Noelle

Micro Focus Customer Support

If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.

If you are satisfied with anyone’s response please remember to give them KUDOS and show your appreciation.
0 Likes
Highlighted
Acclaimed Contributor.. Acclaimed Contributor..
Acclaimed Contributor..

Re: Urgent: Invalid sysUpTime messages in statepoller-logs

Jump to solution

Hm, I have now created that nms-statepoller.properties on the test server and restarted the procs. It now has the following content:

com.hp.ov.nms.statepoller.statemapper.snmpAgentRestartConfigCheckEnabled=false com.hp.ov.nms.statepoller.statemapper.snmpNoSuchObjectRediscoveryEnabled=false com.hp.ov.nms.statepoller.statemapper.snmpAgentUnresponsiveTriggersDiscovery=false

The JMX still returns them (the first two) as true, though:

/opt/OV/support/nnmtwiddle.ovpl get com.hp.ov.nms.statepoller:name=StateMapper StateMapperSNMPAgentRestartConfigCheckEnabled
StateMapperSNMPAgentRestartConfigCheckEnabled=true

opt/OV/support/nnmtwiddle.ovpl get com.hp.ov.nms.statepoller:name=StateMapper StateMapperNoSuchObjectRediscoveryEnabled StateMapperNoSuchObjectRediscoveryEnabled=true

How come...?

Ok, I will file a regular support case on this. I strongly suspect that this is a bug introduced in patch 2 (!!) and that it relates to the use of a wrong variable or similar, as you mention too. I will see if I have time to test this on the test server at some stage, but for now I will file a case and push them to involve your lab in the question ASAP, rather than asking me to spend a lot of time providing them with a lot of extra info from my customer's system 😉

BTW, I have observed that all reported negative sysUpTime values are roughly in the range 2 148<mmmmmm> to 4279<nnnnnn>, where <mmmmmm> and <nnnnnn> are any (?) 6-digit numbers.

Cheers,
Frank

0 Likes
Highlighted
Acclaimed Contributor.. Acclaimed Contributor..
Acclaimed Contributor..

Re: Urgent: Invalid sysUpTime messages in statepoller-logs

Jump to solution

Just  a little update regarding the nms-statepoller.properties file.

I have tried to modified the parameters a few times to reflect the exact Java Class and Attribute names shown in the JMX-console, i.e.:

com.hp.ov.nms.statepoller.statemapper.StatePollerStateMapper.StateMapperSNMPAgentRestartConfigCheckEnabled=false
com.hp.ov.nms.statepoller.statemapper.StatePollerStateMapper.StateMapperNoSuchObjectRediscoveryEnabled=false

But it does not hit.. After restarting the processes these are still set to true.

So I have now completely deleted the nms-statepoller.properties file and set them to false through the JMX-console instead. I guess this setting will be valid until the next restart of the procs.

I have opened a support case on the flooding-the-log issue.

Cheers,
Frank

0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: Urgent: Invalid sysUpTime messages in statepoller-logs

Jump to solution

Hello Frank,

There is a similar discovery hotfix for NNMi 10.10 as well: HF-NNMI-10.1X-STATEPOLLER-20160922 . It makes sense for support to verify first if that hotfix was included in the Patch 2. The patch 2 does not list one. So, may be it was not included.

The message looks to me suspicious. The current  value of sysUpTime  2,755,322,992 does not exceed 32bit. I would recommend to log a support case to investigate providing NNMi logs.

Best regards,

Sergey Pankratov
NNMi Support

The views expressed in my contributions are my own and do not necessarily reflect the views and strategy of HPE
If you find this or any post resolves your issue, please be sure to mark it as an accepted solution.
If you are satisfied with anyone’s response please remember
to give them a KUDOS and show your appreciation
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.