Highlighted
New Member..
New Member..
791 views

Record being added contains a NULL key error in HS environment

Hi everyone

We've been experiencing some inconsistent issues whereby some of the nodes in our Horizontally-scaled SM environment would not allow to submit/update Incidents or Changes due to "Unrecoverable error in application:  schedule.add on panel add.schedule". Our environment is:

Application version 7.11.000

RTE version 9.33.0045

Mostly every Monday starting at about 8.40 and lasting for about 30minutes, any users logged on to SM via the Windows client would get thei sessions frozen where the sand clock would just spin and would not allow to do them anything. After that, they would get logged off and a full SM environment restart would be needed. Sometimes (but not always) sm.log would display at the time of the crash:

11468( 12052) 11/11/2015 14:36:14  RTE E Error: SQL State: 08S01-11    Message: [Microsoft][ODBC SQL Server Driver][DBNETLIB]General network error. Check your network documentation.

On other days users would get the following errors when updating/Inserting/viewing IM or Change records. At the same time, sm.exe process would spike to 100% CPU (bofore I do any actions).

On display/init of IM record:
The record being added contains a NULL key (message,add.schedule)
file:(schedule) key:(schedule.id=) (message,add.schedule)
Key #7 is empty. (message,add.schedule)
The record being added contains a NULL key (schedule.add,add.schedule)
file:(schedule) key:(schedule.id=) (schedule.add,add.schedule)
Key #7 is empty. (schedule.add,add.schedule)
Unrecoverable error in application:  schedule.add on panel add.schedule

On update of IM record:
The record being added contains a NULL key (message,add.schedule)
file:(schedule) key:(schedule.id=) (message,add.schedule)
Key #7 is empty. (message,add.schedule)
The record being added contains a NULL key (schedule.add,add.schedule)
file:(schedule) key:(schedule.id=) (schedule.add,add.schedule)
Key #7 is empty. (schedule.add,add.schedule)
Update cancelled by trigger (apm.save.problem,update.go)
file:(probsummary) key:(number=INC2922302) (apm.save.problem,update.go)
The record being added contains a NULL key (message,add.schedule)
file:(schedule) key:(schedule.id=) (message,add.schedule)
Key #7 is empty. (message,add.schedule)
The record being added contains a NULL key (apm.problem.change.state,sched.add)
file:(schedule) key:(schedule.id=) (apm.problem.change.state,sched.add)
Key #7 is empty. (apm.problem.change.state,sched.add)
The record being added contains a NULL key (message,add.schedule)
file:(schedule) key:(schedule.id=) (message,add.schedule)
Key #7 is empty. (message,add.schedule)
The record being added contains a NULL key (schedule.add,add.schedule)
file:(schedule) key:(schedule.id=) (schedule.add,add.schedule)
Key #7 is empty. (schedule.add,add.schedule)
Unrecoverable error in application:  schedule.add on panel add.schedule
Unrecoverable error in application:  apm.problem.change.state on panel sched.add
Unrecoverable error in application:  schedule.add on panel add.schedule

Until I restart the node I am getting the error on, this error would always occur in IM/Change update or insert. I can create an interaction just fine (first time fix).

On the same environment I would log on directly to another node, update/insert a new IM/Change record, and it would work just fine. Seemingly some nodes have this issue, and some do not. When all the problematic nodes are restarted, there are no errors such as above anymore.

I've looked at all the articles in this forum related to this issue, but none seem to have a proper solution that we might use. We've checked if there are any tasks/reports running agains the SM DB, but there seem to be none.

I'd like to start investigating with sm.exe at 100% issue - is there anyway to know in SM what RAD app/JS script or anything else is consuming these resources?

Any ideas?

0 Likes
3 Replies
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: Record being added contains a NULL key error in HS environment

Check your sm logs files for this message:

RTE E Error in fetching sequence number for file tablename.

If you find that error you have encountered a known RTE defect that can cause records to have NULL key values for those tables that rely on the counters or numbers files.

This issue is fixed in 9.33 HF3, 9.33 P2, 9.34 and 9.40.

This might explain why it happens on one node and not the other, a full restart would fix this until it happens again.

 

 

0 Likes
Highlighted
New Member..
New Member..

Re: Record being added contains a NULL key error in HS environment

Thanks swhirschfeld. I did check the logs, but seen no such message in any of the nodes. Could there by any other error messages that could be caused by the same RTE known error? It does lead me to believe that something is happening at the point when a  new counter value is being generated for the schedule record (I assume the schedule record is being initialized when displaying/initializing the IM or Change record?). Is there a way to track the sequence from start of the IM record display until it is fully displayed? I know of 't on' option in the Win client, maybe there is something else?

0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: Record being added contains a NULL key error in HS environment

 

The defect would only produce that one error message, just to be clear in the message, tablename is a place holder for the real table name like incidents, activity, etc.

Windows ( Microsoft ) provides a tool called Process Explorer that will allow you to view the CPU use of each thread in a process.  When you have identified the pid that is at 100% you could examime the pid with Process Explorer, on the tools Thread tab is a list of each TID ( thread id ) and the current cpu utilization.  That and the SM system status display should help you isolate the session that is using all of the CPU.

Also in the release notes for 9.31 there is a new feature called - Dynamic Debugging of User Sessions or Schedulers - that allows you to enable tracing on an already running session, so you could turn on rtm:3 debugdbquery:999 for the TID that you have identified as using all/most of the CPU.  That may allow you to see the problematic RAD or JS.

 

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.