Welcome Serena Central users! CLICK HERE
The migration of the Serena Central community is currently underway. Be sure to read THIS MESSAGE to get your new login set up to access your account.
Fred McGhee Respected Contributor.
Respected Contributor.
1313 views

Devices by Device type Starburst no longer populates after upgrading to Smart connector version 7.9.

Devices by Device type Starburst no longer populates after upgrading to Smart connector version 7.9.x

We recently upgraded all the conectors to version 7.9..0.8084. After the upgrade the starburst on the dash board screen of ArcMC show  zero. We have upgraded the ArcMC to version 2.8.1 as instructed by support, but no change. 

 

Has anyone else seen this issue and have an idea of what is causing the error in devices by device type.? 2018-07-23_15-42-49.png

Labels (1)
0 Likes
22 Replies
Trusted Contributor.. mederic.hurier Trusted Contributor..
Trusted Contributor..

Re: Devices by Device type Starburst no longer populates after upgrading to Smart connector version

I have the same issue with ArcMC 2.8.1 and connectors 7.7, 7.8 and 7.9.

No issue with ArcMC 2.8.0 but according to support it's not possible to rollback from ArcMC 2.8.1 to 2.8.0.

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Devices by Device type Starburst no longer populates after upgrading to Smart connector version

@mederic.hurier and @Fred McGhee

It might be that device tracking files has gone bad after the upgrade.

Try to stop the connectors and ArcMC, and delete the device tracking file, it should be located under:

\current\user\agent\agentdata\ps.adptracking* on both the connector and the ArcMC, might be a slightly different path, but it's both in agendata with that name.

Turn on the connectors and ArcMC again, and see if that resolved it.

If not, check in the same agentdata folder on ArcMC, and see if it is building up cache, or if it is just old files there, we had several issues with graphing and monitoring stopping to work, cleaning up the ./agentdata folder completely and restart ArcMC would then resolve it. A longer time ago i managed to track that down to ArcMC having trouble handling corrupt cache files, and it get's stuck trying to finalize it instead of removing it.

If that either does not resolve it, i would need you to upload a copy of some of your ArcMC log files, so we can see if we can pinpoint the issue 🙂

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
Trusted Contributor.. mederic.hurier Trusted Contributor..
Trusted Contributor..

Re: Devices by Device type Starburst no longer populates after upgrading to Smart connector version

Deleting the cache on the connectors does not solve the issue.

I can't find any folder called agentdata on the ArcMC and no file called ps.adptracking, could youp please give the exact path?

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Devices by Device type Starburst no longer populates after upgrading to Smart connector version

Yeah it has to be done together with ArcMC, in most cases ArcMC cache alone normally solves it. Seems like i was a bit quick when i wrote the last post. I documented it last time i debugged our issues, so here was my findings:

1. Setup a tail on this logfile:

/opt/arcsight/userdata/logs/arcmc/arcmc_web.log

2. Restart ArcMC, and look for errors, the one i found might be different than yours, so keep a look out! 🙂 An example of what i saw was:

Skipping update heavy cache as available memory XYZ.

In theory ArcMC does not need much memory, though if the cache or data files was to big or corrupt, it has issues loading it into memory.

3. Stop ArcMC and delete or backup all files in this directory, after backup delete the files: 

/opt/arcsight/userdata/arcmc/data/

These files are used for statistics, so removing them might cause your graphs to have some small gaps, but does not cause any other issues.

4. Find the tracking files, the easiest way to do this is to just run this command, it can be done on both connector and ArcMC:

find /opt/arcsight/ -name *ps.adp*

5. Remove those files and start things up again. Remember to keep tailing your arcmc log

Hopefully that solves it!

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
Trusted Contributor.. mederic.hurier Trusted Contributor..
Trusted Contributor..

Re: Devices by Device type Starburst no longer populates after upgrading to Smart connector version

Thanks for the details.

So I have tailed the log file and found also errors like "Skipping update heavy cache" and also :

 

[2018-07-25 14:27:22,652][ERROR][DeviceTracker][scanAndUpdateDeviceCache][task-scheduler-1]
java.lang.NullPointerException
        at com.arcsight.arcmc.device.DeviceBreachNotifier.notifyDeviceBreaches(DeviceBreachNotifier.java:46)
        at com.arcsight.arcmc.device.DeviceTracker.scanAndUpdateDeviceCache(DeviceTracker.java:188)
        at com.arcsight.arcmc.device.DeviceTracker.scanDevices(DeviceTracker.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:64)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

 

So stopped the ArcMC services and deleted the files in /opt/arcsight/userdata/arcmc/data/.

But I still could not find any file with *ps.adp*, even tried from /

After restarting the ArcMC, still the same errors.

Regards

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Devices by Device type Starburst no longer populates after upgrading to Smart connector version

HMm i wonder if it's a permission issue then.

Can you post some of the newest logs in /opt/arcsight/current/logs/agent.out.wrapper.log ?

Also see if you find anything in  /opt/arcsight/current/arcsight/aps/logs/ ? Should be named catalina.

Is arcmcagent running when you list a status?

Last but not least, let's check if its a permission issue, this command should not return anything, if it doesnt work, you might need to escape the ! like so \!

find /opt/arcsight/ ! -uid "$(id -u)"

 It runs find in your folder and subfolders, and using ! with find says just anything NOT, and then uid = your arcsight user, so you have to run this as your arcsight user.

One of these has to come up with a solution 🙂

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
Trusted Contributor.. mederic.hurier Trusted Contributor..
Trusted Contributor..

Re: Devices by Device type Starburst no longer populates after upgrading to Smart connector version

Hi,

All services are running:

Process 'apache'                    running
Process 'aps'                       running
Process 'arcmcagent'                running
Process 'postgresql'                running
Process 'web'                       running

 

I cannot find any files called agent.out.wrapper.log in my ArcMC, or do you mean the connector one?

The catalina log files are full of

Jul 26, 2018 6:24:21 AM com.sun.jersey.spi.container.servlet.WebComponent filterFormParameters
WARNING: A servlet request, to the URI https://127.0.0.1/platform-service/rest/PlatformService/getSensors, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.

I've attached the full log to this message.

 

The command regarding the permission check returns a lot of line. I ran it using arcsight user.

Regards

0 Likes
Highlighted
Trusted Contributor.. mederic.hurier Trusted Contributor..
Trusted Contributor..

Re: Devices by Device type Starburst no longer populates after upgrading to Smart connector version

Hi,

 

Still can't get the ArcMC to report the devices.

Any update regarding the issue or a possible fix?

 

Regards

0 Likes
Marijo Mandic Acclaimed Contributor.
Acclaimed Contributor.

Re: Devices by Device type Starburst no longer populates after upgrading to Smart connector version

Hello,

1) Please try to restart "web" service on ArcMC if this helps.
2) If this does not help please create Service Request with Snapshot attached and screenshot of the issue for better understanding of the issue.
3) Also please note if it was upgrade, what was the version when it worked and if it is HW or SW.

Regards,

Marijo

0 Likes
Trusted Contributor.. mederic.hurier Trusted Contributor..
Trusted Contributor..

Re: Devices by Device type Starburst no longer populates after upgrading to Smart connector version

Hello Marijo,

Thanks for your answer.

1) I already restart web service and even reboot the host, still the same issue.

2) Already done but not helping. SD02239924

3) Upgrade from 2.8.0 which was working well. This is a the software version runing on RHEL 7.3.

Regards

 

0 Likes
Marijo Mandic Acclaimed Contributor.
Acclaimed Contributor.

Re: Devices by Device type Starburst no longer populates after upgrading to Smart connector version

Hello,

I checked your situation and Development Team is working on this reported issue.

Please have patience as it seems not easy to reproduce/not happening in all scenarios.

I am sure that all relevant information will be communicated to you via your Service Request.

Regards,

Marijo

0 Likes
Fred McGhee Respected Contributor.
Respected Contributor.

Re: Devices by Device type Starburst no longer populates after upgrading to Smart connector version

Hello 

 

Thank you, I can imagine the recreation has been difficult. I look forward to the response. 

0 Likes
Fred McGhee Respected Contributor.
Respected Contributor.

Re: Devices by Device type Starburst no longer populates after upgrading to Smart connector version

Hello,

I cleared the data and restarted, but the issue remains. 

I also was unable to find any file named ps.adptracing or ps (anything) 

Thoughts?

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Devices by Device type Starburst no longer populates after upgrading to Smart connector version

Hello @Fred McGhee

Please see my last post, maybe you could try out those parts as well.

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.