Shared memory required for IR is not according HP recommended calculation

Regarding shared memory, we have read that an estimate value for shared memory can be calculated by doing:

48 MB 1MB per 10 users Shared Memory for IR

We have calculated the following value:134217728 (128MB). With this configuration we have a 32% of free space of shared memory.

We are expimenting problems with IR regen. It is generating errors like:

23  RTE E Caught SCFramework::SCEHException: Win32 Exception: 0xC0000005

7884( 11164) 10/17/2016 14:45:23  RTE I ===> Signal 11: User=IRQUEUE, Application=Unknown, Panel=Unknown

If we increase the shared memory to 256Mb IR regen works fine. But this value is very far from the calculation recommended from HP.  Is there any other solution without increasing shared memory?

Our IR is working in asynchronous mode.

  • Hello,

     

    I'm not sure, if the error during IR regen that you reported really is related to shared memory. Actually, you'd see an reportshm output in the log file, if shared memory shortage is the cause.

    So, if you can send more of the error, and what happens before that in the log file it would help to understand the issue.

     

    The background is that before SM 9.32 (Patch ?) the IR regen was always executed synchronously and therefore best in a non-productive system. After that, the IR regen is always executed asynchronously, which means that the IRQUEUE background process is just running over all records in the target dbdict again and updates the IR index.

     

    Regarding IR cache size in shared memory, please also be aware to exclude it when calculating the Free Space percentage (the reportshm does unfortunately include it): The IR cache depends on the size of IR index - so it can be large, but is not really related to the continues allocation and deallocations during production. This means to stick with rule that 25-75 percentage of Free Space should exist anytime while system is up, calculate (free space) by (shm size-IR cache size).

     

    Best regards,

    Armin

  • Hi Armin,

     

    This is the complete log of the error:

     
    
    7884( 11164) 10/17/2016 14:45:21  RTE D Dump of Adding term
    
      7884( 11164) 10/17/2016 14:45:21  RTE D 0x09A44948: 73656E74                            [sent            ]
    
      7884( 11164) 10/17/2016 14:45:23  RTE I [Address ] Symbol(Parm1,Parm2,Parm3,Parm4)  Offset                                          Source module
    
      7884( 11164) 10/17/2016 14:45:23  RTE I  1: [730AAE7A] memcpy(0x85079008,0x99F0DA0,0x4A937,0x2EAD9B0)  0x5A                             C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.6161_none_50934f2ebcb7eb57\MSVCR90.dll-
    
      7884( 11164) 10/17/2016 14:45:23  RTE I  2: [101AC7DC] dl_read(0x800AA1A8,0x9A4A3A0,0x110DEE96,0x30010)  0xFC                           E:\HP\Service Manager\Server\RUN\SM.dll-ir.cpp-5056
    
      7884( 11164) 10/17/2016 14:45:23  RTE I  3: [101AC8F6] tind_add(0x2EAD9B0,0x800AA1A8,0x11154E52,0x10F94C26)  0xB6                       E:\HP\Service Manager\Server\RUN\SM.dll-ir.cpp-5199
    
      7884( 11164) 10/17/2016 14:45:23  RTE I  4: [101AED08] tl_process(0x2EAD9B0,0x800AA1A8,0x85A1819C,0x99F9DF0)  0xB8                      E:\HP\Service Manager\Server\RUN\SM.dll-ir.cpp-4300
    
      7884( 11164) 10/17/2016 14:45:23  RTE I  5: [101AF258] ir_add(0x800AA190,0x0,0x99EF0D0,0xAC)  0x2C8                                     E:\HP\Service Manager\Server\RUN\SM.dll-ir.cpp-7000
    
      7884( 11164) 10/17/2016 14:45:23  RTE I  6: [1003D59D] dbIRAdd(0x99FD5C0,0x0,0xC,0x2EAD9B0)  0xFD                                       E:\HP\Service Manager\Server\RUN\SM.dll-dbi.cpp-7592
    
      7884( 11164) 10/17/2016 14:45:23  RTE I  7: [101B0330] irProcessIrQueue(0x5F05800,0x22794118,0x1,0x0)  0x5F0                            E:\HP\Service Manager\Server\RUN\SM.dll-ir.cpp-6628
    
      7884( 11164) 10/17/2016 14:45:23  RTE I  8: [10122DCA] symanContinue(0x99DF904,0x5F05800,0x22794118,0x22794120)  0xDA                   E:\HP\Service Manager\Server\RUN\SM.dll-syman.cpp-525
    
      7884( 11164) 10/17/2016 14:45:23  RTE I  9: [1006C7E8] Java_com_hp_ov_sm_server_utility_GenericServerThread_symanContinue(0x5F05930,0x99DF9D0,0x99DF9CC,0x53AA0D4)  0x58 E:\HP\Service Manager\Server\RUN\SM.dll-genericserverthread.cpp-87
    
      7884( 11164) 10/17/2016 14:45:23  RTE I 10: [0317AABD] <not found>(0x0,0x1F80,0x0,0x3170372)
    
      7884( 11164) 10/17/2016 14:45:23  RTE I 11: [70040732] JVM_Clone(0x99DFA60,0x99DFBF8,0xA,0x22793F20)  0x42CE2                           E:\HP\Service Manager\Server\RUN\jre\bin\client\jvm.dll-
    
      7884( 11164) 10/17/2016 14:45:23  RTE I 12: [70040E6A] JVM_Clone(0x99DFBF0,0x9DFB54,0x99DFB60,0x3179890)  0x4341A                       E:\HP\Service Manager\Server\RUN\jre\bin\client\jvm.dll-
    
      7884( 11164) 10/17/2016 14:45:23  RTE I 13: [7010529E] JVM_FindSignal(0x70040D00,0x99DFBF0,0x99DFB54,0x99DFB60)  0x62E1E                E:\HP\Service Manager\Server\RUN\jre\bin\client\jvm.dll-
    
     7884( 11164) 10/17/2016 14:45:23  RTE I 14: [70041035] JVM_Clone(0x99DFBF0,0x53AA0B4,0x2FBD198,0x2FBE710)  0x435E5                      E:\HP\Service Manager\Server\RUN\jre\bin\client\jvm.dll-
    
      7884( 11164) 10/17/2016 14:45:23  RTE I 15: [70041097] JVM_Clone(0x99DFBF0,0x53AA0B0,0x53AA0B4,0x2FBD198)  0x43647                      E:\HP\Service Manager\Server\RUN\jre\bin\client\jvm.dll-
    
      7884( 11164) 10/17/2016 14:45:23  RTE I 16: [6FFEC49F] jio_printf(0x53AA0B4,0x5F05800,0x5F05800,0x0)  0xAF                              E:\HP\Service Manager\Server\RUN\jre\bin\client\jvm.dll-
    
      7884( 11164) 10/17/2016 14:45:23  RTE I 17: [700622DC] JVM_Clone(0x5F05800,0x0,0x0,0x1)  0x6488C                                        E:\HP\Service Manager\Server\RUN\jre\bin\client\jvm.dll-
    
      7884( 11164) 10/17/2016 14:45:23  RTE I 18: [70062D37] JVM_Clone(0x728D1200,0xFFFDE000,0x77B20214,0x99DFC2C)  0x652E7                   E:\HP\Service Manager\Server\RUN\jre\bin\client\jvm.dll-
    
      7884( 11164) 10/17/2016 14:45:23  RTE I 19: [700A71E9] JVM_FindSignal(0x5F05800,0xE16835A8,0x0,0x0)  0x4D69                             E:\HP\Service Manager\Server\RUN\jre\bin\client\jvm.dll-
    
      7884( 11164) 10/17/2016 14:45:23  RTE I 20: [674AC6DE] endthreadex(0x0,0x99DFF94,0x75A4336A,0x5EDC2C8)  0x3A                            E:\HP\Service Manager\Server\RUN\MSVCR100.dll-
    
      7884( 11164) 10/17/2016 14:45:23  RTE I 21: [674AC788] endthreadex(0x5EDC2C8,0x99DFFD4,0x77A59902,0x5EDC2C8)  0xE4                      E:\HP\Service Manager\Server\RUN\MSVCR100.dll-
    
      7884( 11164) 10/17/2016 14:45:23  RTE I 22: [75A4336A] BaseThreadInitThunk(0x5EDC2C8,0xEA4372AA,0x0,0x0)  0x12                          C:\Windows\syswow64\kernel32.dll-
    
      7884( 11164) 10/17/2016 14:45:23  RTE I 23: [77A59902] RtlInitializeExceptionChain(0x674AC724,0x5EDC2C8,0xFFFFFFFF,0x77AE7514)  0x63    C:\Windows\SysWOW64\ntdll.dll-
    
      7884( 11164) 10/17/2016 14:45:23  RTE I 24: [77A598D5] RtlInitializeExceptionChain(0x674AC724,0x5EDC2C8,0x0,0x365BA988)  0x36           C:\Windows\SysWOW64\ntdll.dll-
    
      7884( 11164) 10/17/2016 14:45:23  RTE I !!! Core Generation is disabled.  Ignoring Generate Core Dump Request. 
    
      7884( 11164) 10/17/2016 14:45:23  RTE E Caught SCFramework::SCEHException: Win32 Exception: 0xC0000005 (instruction 0x730AAE7A while reading address 0x09A2F000)
    
      7884( 11164) 10/17/2016 14:45:23 JRTE I thread: -que:ir_thread_1 id: 11164 has terminated!
    
      7884( 11164) 10/17/2016 14:45:23  RTE I -Memory : S(641832) O(240004) MAX(1398572) - MALLOC's Total(261717)
    
      7884( 11164) 10/17/2016 14:45:23  RTE I ===> Signal 11: User=IRQUEUE, Application=Unknown, Panel=Unknown
    
      7884( 11164) 10/17/2016 14:45:23  RTE I  RAD thread id 0, last option key 0
    
      7884( 11164) 10/17/2016 14:45:23  RTE I Parameters passed on the command line:
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      que:ir
    
      7884( 11164) 10/17/2016 14:45:23  RTE I Parameters passed from the ini file:
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      shared_memory:134217728
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      allowunsupporteddbversion:1
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      alertfilters:Mapping*;Performance*
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      log:../logs/sm.log
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      system:13080
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      httpPort:13080
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      httpsPort:13443
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      queryhashcode:plsmtest.grifols.com:13080
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      sqldictionary:oracle10
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      [oracle10]
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      sqldb:TORADB01
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      sqllogin - Value not displayed
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      sqllibrary:sqoracle.oci10.DLL
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      plugin0:kmplugin.dll
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      threadsperprocess:60
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      ldapauthenticateonly
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      ldapnostrictlogin:1
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      ldapbinddn:
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      ldapbindpass - Value not displayed
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      ldapserver2:p2dc2.grifols.com
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      ldapdisable:1
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      ldapstats:0
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      ldapdisconnect:1
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      numberoflogfiles:7
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      maxlogsize:51200000
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      agstackl:800
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      enablecoredump:0
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      ir_disable:0
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      ir_asynchronous:1
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      ir_autostop:1
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      ir_max_relevant_answers:40
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      ir_max_shared:40265318
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      ir_timelimit:60
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      ir_trace:801
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      sessiontimeout:1200
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      trustedsignon:1
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      keystoreFile:server.keystore
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      keystorePass - Value not displayed
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      truststoreFile:cacerts
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      truststorePass - Value not displayed
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      ssl_trustedClientsJKS:trustedclients.keystore
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      ssl_trustedClientsPwd - Value not displayed
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      cache_slots:19973
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      cache_locks:19963
    
      7884( 11164) 10/17/2016 14:45:23  RTE I      ir_languagefiles_path:..\irlang
    
      7884( 11164) 10/17/2016 14:45:23  RTE I End of parameters
    
      7884( 11164) 10/17/2016 14:45:23  RTE W kill on process with must complete lock

     

    I have extracted the free space from the sm reportshm, its the value indicated in Free Space item.

  • R&D has recommended that 256-512MB (depending on size of IR) is a good number to use when IR is enabled, especially if you don't want to calculate the actual value needed. If you want to get a better idea on the actual number that will be needed, run sm -util -> vrir -> ir.probsummary

    PLEASE NOTE:

    This command can take a very long time and it should be done on a test or development environment that is a copy of production to get an accurate reading. This should show how much shared memory is going to be used by a particular IR file (in this instance probsummary). You would need to run the command for all the ir files and add them together to get a more accurate idea of IR memory consumption.

     For example:

    vrir report in sm.log(pid, thread id and time stamp are omitted)---

    RTE I *** 512 bytes allocated for IR header

    RTE I *** 320072 bytes allocated for hashtables

    RTE I *** 5764/ 5633 bytes allocated/used by DOCs

    RTE I *** 10708/ 10360 bytes allocated/used by TERMs

    RTE I *** 16694/ 12529 bytes allocated/used by DOCLISTs

    RTE I *** 0/ 0 bytes allocated/used by ADL DOCLISTs

    RTE I *** 17648/ 15884 bytes allocated/used by TERMLISTSs

    RTE I *** 0/ 0 bytes allocated/used by ADL TERMLISTSs

    RTE I *** 0 bytes free total according to IR header

    RTE I *** 320584 bytes used by physical file according to IR header

    The ir_max_shared parameter value should be a sum of the IR header, hashtables, DOCs, TERMs, DOCLISTs, ADL DOCLISTs, TERMLISTSs, and ADL TERMLISTSs for all files.

    If you find that this or any other post resolves your issue, please be sure to mark it as an accepted solution.
    If you are satisfied with anyone’s response please remember to give them a KUDOS by clicking on the "thumbs up" at the bottom left of the post and show your appreciation.

  • Hello Scuenca,

    I can now see what you are referring to: The stack trace shows error in memcpy() function - hence the issue is related to this function call. When looking into our database, I couldn't find a similiar stack trace during IR processing.

    As here R&D analysis is required, to resolve it requires a support ticket so we can elevate.

    Best regards,

    Armin

  • Hi Mike,

    When you say: "R&D has recommended that 256-512MB ". Did you mean, shared_memory isn't it?

    I will do the calculation of these values to have a more accurate value.

    Thank you very much for your help!

  • Verified Answer

    Hello Scuenca,

    As far as I know, the formular you were using is correct. The estimation of SHM required for IR is not easy - Mike described this.

    The 32bit servlet has 4GB limit for its virtual address space. As it contains shared memory, java heap and native heap, each pool must not be oversized - I think this is the idea behind the recommendation below. In my opinion SHM should really not exceed 512M for this reason.

    Best regards,

    Armin