Highlighted
Super Contributor.
Super Contributor.
456 views

iSPI Metrics - dbproxy crash after a few queries from NNMi

Jump to solution

Hi all

I have a new installation of iSPi Perf Metrics. In general it works and I see performance data on the NNMi panes.  But after some queries from NNMi the dbproxy suddently crashes (statusdb.ovpl shows HPE NNM iSPI Performance Database and DB SQL Rewrite Proxy processes are NOT running) After doing a startdb.ovpl it works again for a while.

 NNMi and NPS(with all roles) running on two separate servers both on windows and with the latest V10.20.

The prspi.log shows following:

Sep 09, 2016 11:20:53 AM [PID:-10816] FATAL C:\Program Files (x86)\HP\HP BTO Software\/nonOV/perl/a/lib/site_perl/NPS/nps.pm[1848] Utility.dbProxyRun Stacktrace: 0. C:\Program Files (x86)\HP\HP BTO Software\/nonOV/perl/a/lib/site_perl/NPS/nps.pm:1848 [NPS::npslog::FATAL()]

Sep 09, 2016 11:20:53 AM [PID:-10816] FATAL C:\Program Files (x86)\HP\HP BTO Software\/nonOV/perl/a/lib/site_perl/NPS/nps.pm[1848] Utility.dbProxyRun Stacktrace: 1. C:\Program Files (x86)\HP\HP BTO Software\NNMPerformanceSPI\bin\Win32\..\dbProxyRun.ovpl:29 [main::system_or_die()]

Sep 09, 2016 11:20:53 AM [PID:-10816] FATAL C:\Program Files (x86)\HP\HP BTO Software\/nonOV/perl/a/lib/site_perl/NPS/nps.pm[1848] Utility.dbProxyRun Failed command system("C:/Program Files (x86)/HP/HP BTO Software/NNMPerformanceSPI/bin/win32/dbproxy.exe" -d localhost -p 9303 -l 9306 -v 4 > "D:/ProgramData/HP/HP BTO Software/NNMPerformanceSPI/logs/dbproxy.log" 2>&1): 65280

 

In the dbproxy.log I see this pattern repeatingly even when the system seems to work….

[09/09-2016 | 15:41:28]   INFO:   Incoming:  10.124.  4.170   Routing to localhost:9303

[09/09-2016 | 15:41:28]   Incoming Query: Does not reference fact or dimension table or security. Passthrough without rewrite.

[09/09-2016 | 15:41:28]   Incoming Query: Does not reference fact or dimension table or security. Passthrough without rewrite.

[09/09-2016 | 15:41:29]   Incoming Query: Does not reference fact or dimension table or security. Passthrough without rewrite.

 I did  a reset of the NPS (resetSPI.ovpl option 5 ) but it didn't help.

Any Idea ?

Thanks Thomas

0 Likes
1 Solution

Accepted Solutions
Highlighted
Super Contributor.
Super Contributor.

Hi got this information from HP Support, I tweaked  my one to -m40 , ever since I never had a crash again and I didn't see a negative side effect so far.

Cheers Thomas

***************************************************************************************

You can try to use the workaround that will avoid dbproxy to crash.

1. Take a backup copy of $NPSInstallDir/bin/dbproxyRun.ovpl
2. Edit file of $NPSInstallDir/bin/dbproxyRun.ovpl

Modify the line
my $args = "-d localhost -p $cfg{PRSPI_DB_PORT} -l $cfg{PRSPI_DB_PROXY_PORT} -v 4"
to
my $args = "-d localhost -p $cfg{PRSPI_DB_PORT} -l $cfg{PRSPI_DB_PROXY_PORT} -v 4 -m 30"

3. Restart the NPS using:

stopNPS.ovpl
startNPS.ovpl

That it is just a workaround for now, and the number to specify behind the -m parameters depends on the actual server memory.
If the dbproxy still crash after restart increase the value “-m” by 5 (eg: -m 35) and restart NPS.
If needed increase by 5 again (-m 40)

Important: there is a side effect applying this workaround cause
The Dashbord may need to be refreshed several times.

View solution in original post

0 Likes
10 Replies
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Hello Thomas

Thanks for posting

For troubleshooting this matter I would suggest you to open a support case with us in order to find the root cause of this issue.

Regards,

Vincent Montenegro Mena
Customer Support Engineer

If you find that this or any other post resolves your issue, please be sure to mark it as an accepted solution.
If you are satisfied with anyone’s response please remember to give them a KUDOS by clicking on the STAR at the bottom left of the post and show your appreciation.
0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Hi Thomas,

Is it NNMi 10.20?

I have just installed it on a Windows server, and seems to be reproducing the same kind of errors in the logs.
Also the number of dbrproxy threads seem to be growing upon dashboards access from NNMi console.
I am checking with the labs now but would encourage you to raise a case straight forward to HPE Support to have the issue troubleshooted further and fixed.

Once you'll have open the case to Support, in order to close this thread, please make sure to mark any appropriate answer as Accepted Solution.

All the best

Marie-Noelle

Micro Focus Customer Support

If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.

If you are satisfied with anyone’s response please remember to give them KUDOS and show your appreciation.
0 Likes
Highlighted
Super Contributor.
Super Contributor.

Hi All

Thanks for your relies.

It is Version 10.20 on Windows and I've raised a case adding some NNMi logs and the output of collectDiagnostics.ovpl from the NPS Server. 

I will post the outcome on this thread

Thomas

0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Excellent. Many thanks Thomas.


Would you mind proving me the case number so that I could point it out while engaging the labs in that issue?

Many thanks

Marie-Noelle

Micro Focus Customer Support

If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.

If you are satisfied with anyone’s response please remember to give them KUDOS and show your appreciation.
0 Likes
Highlighted
Super Contributor.
Super Contributor.

Service Request 5311795402

Thx Thomas

0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Thanks Thomas.

Micro Focus Customer Support

If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.

If you are satisfied with anyone’s response please remember to give them KUDOS and show your appreciation.
0 Likes
Highlighted
Absent Member.. Absent Member..
Absent Member..

Hi

DId you find a solution to your DBproxy crash?

I experienced the same problem and worked with HPE software support to find a fix.

Let me know.

Personal info Erased

-LB

0 Likes
Highlighted
Super Contributor.
Super Contributor.

Hi got this information from HP Support, I tweaked  my one to -m40 , ever since I never had a crash again and I didn't see a negative side effect so far.

Cheers Thomas

***************************************************************************************

You can try to use the workaround that will avoid dbproxy to crash.

1. Take a backup copy of $NPSInstallDir/bin/dbproxyRun.ovpl
2. Edit file of $NPSInstallDir/bin/dbproxyRun.ovpl

Modify the line
my $args = "-d localhost -p $cfg{PRSPI_DB_PORT} -l $cfg{PRSPI_DB_PROXY_PORT} -v 4"
to
my $args = "-d localhost -p $cfg{PRSPI_DB_PORT} -l $cfg{PRSPI_DB_PROXY_PORT} -v 4 -m 30"

3. Restart the NPS using:

stopNPS.ovpl
startNPS.ovpl

That it is just a workaround for now, and the number to specify behind the -m parameters depends on the actual server memory.
If the dbproxy still crash after restart increase the value “-m” by 5 (eg: -m 35) and restart NPS.
If needed increase by 5 again (-m 40)

Important: there is a side effect applying this workaround cause
The Dashbord may need to be refreshed several times.

View solution in original post

0 Likes
Highlighted
Absent Member.. Absent Member..
Absent Member..

Hi

The problemwe identified with my dbproxy crash was a bug in the NNMi dashboard db connection pool code. I haven't had an issue with the dbproxy crashing since applying the fix.

Cheers
-LB

0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Hi Thomas and Labuss,

Yes, this issue has now been narrowed out as being a Defect and an official Hotfix should be released very soon for it (a temp fix got already released that got successfully tested by a few customers).

You can follow the Defect Life Cycle at:

KM02597194: Database DB SQL Rewrite Proxy (dbproxy) crashes after some time after restarting the NPS 

Note that limiting the number of threads via the -m parameter in $NPSInstallDir/bin/dbproxyRun.ovpl has been a temporarily workaround that got proposed a few time to avoid the dbproxy crash until the root cause of this Defect could be fully isolated and fixed.

However, that workaround  can have a side effect on the Dashboards Performance.

Knowing that the root cause of the Defect has now been fully isolated and fixed, the best is to install the official Hotfix or patch for this Defect as soon as it will be released and restore the original $NPSInstallDir/bin/dbproxyRun.ovpl file. 

Thomas, would you mind if I unmark the workaround you had pointed out earlier as Accepted solution and let you mark this one as Accepted solution instead? 

This is just so that the workaround won't be used by other community members as a definitive solution for this issue knowing that it can have side effects on performance and a definitive solution for this Defect can now be delivered as a Hotfix. 

If you agree with this plan, I will unmark the previously accepted solution and let you mark this one instead.

Cheers

Marie-Noelle

Micro Focus Customer Support

If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.

If you are satisfied with anyone’s response please remember to give them KUDOS and show your appreciation.
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.