Highlighted
Honored Contributor.
Honored Contributor.
447 views

Last Access Time not being updated by vCenter Topology job

Jump to solution

Hello,

We're currently experiencing which i think is an unusual behavior regarding the "last access time" attribute on CIs.

Since one week a lot of CIs are not being updated (i mean this attribute and also the "last discovered time" attribute since there is no change on these CIs) , despite still existing on our vCenter for example, therefore the aging mechanism starts to kick in (candidate / deletion time set to 3 / 7) and we're losing lots of CIs.

I've tried restarting multiple times vCenter Topology by VIM jobs on our vCenters, i can see all CIs being added to the vector/results in communication logs but nothing is updated on the CI,

I've also tried reducing the "touch time" to 12H according to this post but no luck either : https://community.microfocus.com/t5/CMS-Discovery-CMDB-User/UCMDB-Support-Tip-Discovery-not-updating-CI-attributes-Last/td-p/2700829

Also concerned nodes are not being accessed through host connection jobs, so we need them to be updated by the vCenter at least.

We're on 2019.11.

Thanks for your help,
Best regards,
Yann Pingot

1 Solution

Accepted Solutions
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

They are on 2019.11. There is nowhere to upgrade to. 

However in the release notes of 2019.05 it says:

 

UCMDB Server - DatabaseQCCR1H125137

After synchronizing CIs through the CSV Import adapter and DBEnhancedAdapter, the CIs are not touched by the probe anymore. The "failed to insertTouchResultQueue" error seen in the probe-error.log is caused by the local PostgreSQL on the probe failing to insert a record in the touchqueue table. If the CIs are not touched, they start to be subject to aging and subsequently will be deleted from UCMDB unless a full synchronization is performed again.

The issue is now fixed by applying a code change to the installation script. This fix is applicable to new install of CMS 2019.05 only.

For CMS environment upgraded from an older version, you need to execute the alter table query in UCMDB database as follows:

ALTER TABLE ddm_discovery_touch_results_queue ALTER COLUMN JOBID DROP NOT NULL;

 

That is what I am trying to verify - whether the issue matches this case.

Petko

Likes are appreciated!

View solution in original post

10 Replies
Highlighted
Valued Contributor.. Valued Contributor..
Valued Contributor..

I am having the same problem for NNMI integration. We run the integration per a week. But doesn't change Last Access Time of attribute. Then we lose the relevant CI's, because of the aging mechanism.

0 Likes
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

There is a touching mechanism, which exists on the probe side. It is enabled by default. Basically what it does is to check if there is a change in the CIs from the last synchronization period and if the CI is not changed, it is added to a touch queue, which is pushed to UCMDB once per day. The touching is basically update of the Last Access Time. 

Unfortunately I have found several bugs with the touching mechanism, specifically when the CIs come from Integrations. Opening cases resolved them, but patches have to be installed. All should be running fine in version 2019.11. 

Likes are appreciated!
0 Likes
Highlighted
Honored Contributor.
Honored Contributor.

This is what i thought, it seems that reducing the touch time to 12H helped.

Should it be working on all discovery/integrations jobs ? Running manually a host ressource by shell job on a single target for example directly updates the last access time when the job is done.
On the other hand the attribute is only updated on nodes when the touch is triggered for a vCenter Topology discovery.
Maybe this is false analysis on my side and this is because there is almost always something being updated on a node (nodeElements are taken into account I presume for a node update).

So if i resume correctly :

- If something is changed for a node its last access time is directly updated.
- If nothing is changed but the node is still existing it's added to the "touch queue" and thhe touch is triggered every 24H (if not modified).

Best regards,
Yann Pingot

0 Likes
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Yes, your resume is correct. What you can do is to connect to the postgresql on the probe and see if the CI of vcenter is in the touch queue table. To make the test easier, you can also clean the DFP db with the tools script - cleandataflowprobe. If the record exists, then you have to check if it fails when being sent to ucmdb. If it doesn't exist, you have to check why it fails inserting it.

Petko

Likes are appreciated!
Highlighted
Valued Contributor.. Valued Contributor..
Valued Contributor..

The problem has improved since we upgraded, upgrading may be the solution to you.

0 Likes
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

They are on 2019.11. There is nowhere to upgrade to. 

However in the release notes of 2019.05 it says:

 

UCMDB Server - DatabaseQCCR1H125137

After synchronizing CIs through the CSV Import adapter and DBEnhancedAdapter, the CIs are not touched by the probe anymore. The "failed to insertTouchResultQueue" error seen in the probe-error.log is caused by the local PostgreSQL on the probe failing to insert a record in the touchqueue table. If the CIs are not touched, they start to be subject to aging and subsequently will be deleted from UCMDB unless a full synchronization is performed again.

The issue is now fixed by applying a code change to the installation script. This fix is applicable to new install of CMS 2019.05 only.

For CMS environment upgraded from an older version, you need to execute the alter table query in UCMDB database as follows:

ALTER TABLE ddm_discovery_touch_results_queue ALTER COLUMN JOBID DROP NOT NULL;

 

That is what I am trying to verify - whether the issue matches this case.

Petko

Likes are appreciated!

View solution in original post

Highlighted
Valued Contributor.. Valued Contributor..
Valued Contributor..
Hi,
Apparently it keep going on my problem. I had the same problem again. But after upgrading, the problem seemed to be resolved, so i take it back.
By the way we're on 10.33.
Thanks
0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert
Hello Yann,

there may be also some performance issues for this job as it can bring a lot of CIs.
Can you paste here the <parameter> section of the CommLog?

Kind regards,
Bogdan Mureșan
EMEA CMS Technical Success
0 Likes
Highlighted
Honored Contributor.
Honored Contributor.

Hi Bogdan,

Here you go :

<params>
<param param_name="reportLUN" param_value="false" />
<param param_name="ignoreHostsWithoutHostNames" param_value="false" />
<param param_name="JOB_ID" param_value="VMware VirtualCenter Topology by VIM" />
<param param_name="reportBasicTopology" param_value="false" />
<param param_name="reportDiscoveredOsName" param_value="false" />
<param param_name="discoverUnknownIPs" param_value="true" />
<param param_name="runInSeparateProcess" param_value="true" />
<param param_name="remoteJVMArgs" param_value="" />
<param param_name="reportPoweredOffVMs" param_value="true" />
<param param_name="taskType" param_value="regular" />
<param param_name="reportLayer2connection" param_value="false" />
<param param_name="maxThreadRuntime" param_value="900000" />
<param param_name="remoteJVMClasspath" param_value="%minimal_classpath%;../lib/axis.jar;../lib/axis-jaxrpc.jar;../lib/axis-wsdl4j.jar;../runtime/probeManager/discoveryResources/vmware/vim.jar;../runtime/probeManager/discoveryResources/vmware/vim25.jar" />
</params>

This vCenter contains less than 3000 VMs, it's usually in warning with a message like below but i don't think it's a big deal, however it fails sometimes (quite rarely nowadays) with some reconciliation timeouts (bulk is too big even with tweaks to the fuse / rates) :

Les CI suivants ont été ignorés : From Class [usage] 5 CIs were ignored due to Link with ignored end; From Class [interface] 5 CIs were ignored due to Multiple Match;

Best regards,
Yann Pingot

0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Hello Yann,

 

this is what I run on my lab

 

<params>
<param param_name="reportFCHBA" param_value="false" />
<param param_name="reportLUN" param_value="false" />
<param param_name="ignoreHostsWithoutHostNames" param_value="true" />
<param param_name="JOB_ID" param_value="VMware VirtualCenter Topology by VIM" />
<param param_name="reportBasicTopology" param_value="false" />
<param param_name="reportDiscoveredOsName" param_value="true" />
<param param_name="discoverUnknownIPs" param_value="true" />
<param param_name="runInSeparateProcess" param_value="true" />
<param param_name="remoteJVMArgs" param_value="-Xms64m -Xmx8192m -XX:MaxMetaspaceSize=1024m" />
<param param_name="reportPoweredOffVMs" param_value="false" />
<param param_name="taskType" param_value="regular" />
<param param_name="reportLogicalVolume" param_value="false" />
<param param_name="reportLayer2connection" param_value="true" />
<param param_name="maxThreadRuntime" param_value="2800000" />
<param param_name="remoteJVMClasspath" param_value="%minimal_classpath%;../lib/axis.jar;../lib/axis-jaxrpc.jar;../lib/axis-wsdl4j.jar;../runtime/probeManager/discoveryResources/vmware/vim.jar;../runtime/probeManager/discoveryResources/vmware/vim25.jar" />
</params>

 

The bolded part is relevant as it will set strict limits for the remote process JVM that will handle the actual discovery. For my scenario the 8GB max memory is tailored for a 4k VM env.

If you have multiple tigger CIs (vmware_virtual_center) try to limit the job at only 1 MaxThread from the adapter. Check also if you have duplicate vCenter CIs based on their InstanceUUID or related IP. This can cause duplicate discovery which will overwhelm the reconciliation engine.

I will truy to post this week an interesting scenario with the Docker veth interfaces. If you ahve Interfaces which are like veth%@% then please delete them as they are short live interfaces linked to the docker pods and they are poisining the reconciliation engine. I am testing a new fix to handle them differently but it's not yet ready.

 

Kind regards,
Bogdan Mureșan
EMEA CMS Technical Success
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.