Internal error: DbaXXXX functions after Update to DP 10.70

After our update from DP to version 10.70, I keep getting the message Internal error: DbaXXXX functions during the backups.
In the event log of the Windows OS there is also a suitable entry regarding a crash of dbsm.exe.

Faulting application name: dbsm.exe, version: 10.70.210.0, time stamp: 0x5eb1a796
Faulting module name: ucrtbase.DLL, version: 10.0.14393.2990, time stamp: 0x5caeb96f
Exception code: 0xc0000409
Fault offset: 0x000000000006e00e
Faulting process id: 0x2ab0
Faulting application start time: 0x01d660b22724ec98
Faulting application path: E:\HPE_DP_BIN\OmniBack\bin\dbsm.exe
Faulting module path: C:\Windows\SYSTEM32\ucrtbase.DLL
Report Id: 65274ed1-cca5-11ea-80ee-0050568f4073
Faulting package full name:
Faulting package-relative application ID:

Parents
  • Hello , 

    Try this: 

    1. Rename DebugLogger.librarytype and LVDataMoverComponent.librarytype from C:\ProgramData\OmniBack\Config\client\components\registry
    2. Restart Services
    3. Rerun command "omnidbutil -encrypt_user_pass -debug 1-300  userpass.txt" successful

    Regards, 

  • Andres,

    i have same problem after upgrading to 10.70, but no one file (DebugLogger.librarytype or LVDataMoverComponent.librarytype) in the folder C:\ProgramData\OmniBack\Config\client\components\registry.

    What we can do more for solving problem?

     

     

  • Hello ,

    Check if the C:\ProgramData\OmniBack\Config\client\components\registry folder contains any *.librarytype files that has no valid reference to DLLs in C:\Program Files\OmniBack\bin\components folder.  Remove any *.librarytype file that has no corresponding *.dll.

    Regards,
    Sebastian Koehler

  • Sebastian, hello.

    I have not found any lone files in the folder "D:\ProgramData\OmniBack\Config\client\components\registry"

     

    C:\>dir "D:\Program Files\OmniBack\bin\components\*.dll"
    Volume in drive D is Новый том
    Volume Serial Number is E679-E0FF

    Directory of D:\Program Files\OmniBack\bin\components

    06/04/2020 06:25 PM 4,785,416 ArrayProvider3PAR.dll
    06/04/2020 06:23 PM 3,836,680 CasDiskComponent.dll
    06/04/2020 06:26 PM 5,635,760 CDpDataMoverComponent.dll
    06/04/2020 06:29 PM 7,821,064 dellemcunityarrayprovider.dll
    06/04/2020 06:23 PM 4,311,304 DpBackendService.dll
    06/04/2020 06:28 PM 6,803,632 DPHostingEnvironmentComponent.dll
    06/04/2020 06:19 PM 3,426,480 DpSessionLogger.dll
    06/04/2020 06:26 PM 5,543,088 DpSplitMirrorComponent.dll
    06/04/2020 06:29 PM 7,762,096 netapparrayprovider.dll
    06/04/2020 06:30 PM 3,534,512 ormapper.dll
    06/04/2020 06:19 PM 3,044,616 PocoLogger.dll
    06/04/2020 06:18 PM 2,786,568 ScaContainer.dll
    06/04/2020 06:15 PM 2,313,648 SerializerBasic.dll
    06/04/2020 06:13 PM 2,130,184 SerializerFwsTypes.dll
    06/04/2020 06:19 PM 2,811,656 ServiceFactory.dll
    06/04/2020 06:29 PM 9,682,184 vepalib_h3ccas.dll
    06/04/2020 06:28 PM 6,772,488 vepalib_hyperv.dll
    06/04/2020 06:25 PM 6,469,384 vepalib_vcd.dll
    06/04/2020 06:30 PM 65,457,416 vepalib_vmware.dll
    06/04/2020 06:30 PM 49,595,656 VmdkMountComponent.dll
    20 File(s) 204,523,832 bytes
    0 Dir(s) 959,466,872,832 bytes free

    C:\>dir "D:\ProgramData\OmniBack\Config\client\components\registry"
    Volume in drive D is Новый том
    Volume Serial Number is E679-E0FF

    Directory of D:\ProgramData\OmniBack\Config\client\components\registry

    08/26/2020 11:12 AM <DIR> .
    08/26/2020 11:12 AM <DIR> ..
    08/10/2020 04:39 PM 342 ArrayProvider3PAR.librarytype
    08/10/2020 04:39 PM 346 ArrayProviderNetApp.librarytype
    08/10/2020 04:39 PM 351 ArrayProviderUnity.librarytype
    08/10/2020 04:39 PM 339 CasDiskComponent.librarytype
    08/10/2020 04:39 PM 346 CDpDataMoverComponent.librarytype
    08/10/2020 04:39 PM 366 DpBackendService.librarytype
    08/10/2020 04:39 PM 309 DPHostingEnvironmentComponent.librarytype
    08/10/2020 04:39 PM 322 DpSessionLogger.librarytype
    08/10/2020 04:39 PM 374 DpSplitMirrorComponent.librarytype
    08/10/2020 04:39 PM 371 H3CCASVirtualEnvironmentProtector.librarytype
    08/10/2020 04:39 PM 392 HyperVVirtualEnvironmentProtector.librarytype
    08/10/2020 04:39 PM 314 ORMapper.librarytype
    08/10/2020 04:39 PM 299 PocoLogger.librarytype
    08/10/2020 04:39 PM 303 ScaContainer.librarytype
    08/10/2020 04:39 PM 2,030 SerializerBasic.librarytype
    08/10/2020 04:39 PM 3,840 SerializerFwsTypes.librarytype
    08/10/2020 04:39 PM 315 ServiceFactory.librarytype
    08/10/2020 04:39 PM 382 VCDVirtualEnvironmentProtector.librarytype
    08/10/2020 04:39 PM 344 VmdkMountComponent.librarytype
    08/10/2020 04:39 PM 371 VMwareVirtualEnvironmentProtector.librarytype
    20 File(s) 12,056 bytes
    2 Dir(s) 959,466,868,736 bytes free

  • Hello ,

    Do you see this kind of errors when running a command like omnidb -session -latest, too? Are there recent hard errors in D:\ProgramData\OmniBack\server\db80\pg\log?

    Regards,
    Sebastian Koehler

  • Sebastian, 

    in the D:\ProgramData\OmniBack\server\db80\pg\log no errors - only "connection" and "disconnection".

    Command "omnidb -session -latest"  working fine, no new errors in the OS system log.

     

    In the debug.log exist messages:

    2020-08-26 12:23:23.743 DBSM.1124.7712 ["/lib/cmn/safemem.c $Rev$ $Date:: $:":112] A.10.70 b210
    MALLOC failed Size=18446744072703442944

    2020-08-26 12:23:23.743 DBSM.1124.7712 ["/lib/cmn/safemem.c $Rev$ $Date:: $:":120] A.10.70 b210
    MALLOC failed Size=18446744072703442944, errcode=8

     

    I think the size of the requested segment is a little too big -  Size=18446744072703442944.

     

    Currently used 72% of total memory on the CS - 14TB of 20TB.

     

     

  • Hello ,

    Have you restarted the Cell Manager already? From which versions did you perform the upgrade?

    Regards,
    Sebastian Koehler

  • Sebastian,

    CS not restarted yet - some backups running.

    Upgrade was from 10.50 to 10.70.

     

    Some findings - "MALLOC failed" from debug.log match "Faulting application name: dbsm.exe, version: 10.70.210.0, time stamp: 0x5ed9637f" from Application Log (Windows)

    debug.log

    2020-08-26 12:23:23.743 DBSM.1124.7712 ["/lib/cmn/safemem.c $Rev$ $Date:: $:":112] A.10.70 b210
    MALLOC failed Size=18446744072703442944

    2020-08-26 12:23:23.743 DBSM.1124.7712 ["/lib/cmn/safemem.c $Rev$ $Date:: $:":120] A.10.70 b210
    MALLOC failed Size=18446744072703442944, errcode=8

    2020-08-26 12:30:29.295 OMNICC.7792.9332 ["/lib/ipc/ipc.c $Rev$ $Date:: $:":3934] A.10.70 b210
    gethostbyaddr(omnis.corp.te.ru) .Network performance warning: took 0,843000 sec

    2020-08-26 12:30:30.127 OMNICC.7792.9332 ["/lib/ipc/ipc.c $Rev$ $Date:: $:":3934] A.10.70 b210
    gethostbyaddr(slm-bkp.corp.te.ru) .Network performance warning: took 0,826000 sec

    2020-08-26 12:30:30.949 OMNICC.7792.9332 ["/lib/ipc/ipc.c $Rev$ $Date:: $:":3934] A.10.70 b210
    gethostbyaddr(te-rz.corp.te.ru) .Network performance warning: took 0,820000 sec

    2020-08-26 12:48:09.216 INET.1076.8468 ["/inet/inet_util.c $Rev$ $Date:: $:":2020] A.10.70 b210
    [RxSmallFtp] peerHost:id-srk-02.corp.te.ru, peerProg:DBSM, event:Rejected:/barlists/veagent/id vm id-pyramid-01..04 (temporary)
    [KeycloakRESTApiConnector::restMethodExecute] Response: 400 body:

    {"error":"invalid_grant","error_description":"Invalid refresh token"}

    2020-08-26 12:54:59.775 DBSM.9276.10212 ["/lib/cmn/safemem.c $Rev$ $Date:: $:":112] A.10.70 b210
    MALLOC failed Size=18446744072703442944

    2020-08-26 12:54:59.775 DBSM.9276.10212 ["/lib/cmn/safemem.c $Rev$ $Date:: $:":120] A.10.70 b210
    MALLOC failed Size=18446744072703442944, errcode=8

     

    Application log - see picture debug&Application.log.jpeg.

     

     

     

     

Reply
  • Sebastian,

    CS not restarted yet - some backups running.

    Upgrade was from 10.50 to 10.70.

     

    Some findings - "MALLOC failed" from debug.log match "Faulting application name: dbsm.exe, version: 10.70.210.0, time stamp: 0x5ed9637f" from Application Log (Windows)

    debug.log

    2020-08-26 12:23:23.743 DBSM.1124.7712 ["/lib/cmn/safemem.c $Rev$ $Date:: $:":112] A.10.70 b210
    MALLOC failed Size=18446744072703442944

    2020-08-26 12:23:23.743 DBSM.1124.7712 ["/lib/cmn/safemem.c $Rev$ $Date:: $:":120] A.10.70 b210
    MALLOC failed Size=18446744072703442944, errcode=8

    2020-08-26 12:30:29.295 OMNICC.7792.9332 ["/lib/ipc/ipc.c $Rev$ $Date:: $:":3934] A.10.70 b210
    gethostbyaddr(omnis.corp.te.ru) .Network performance warning: took 0,843000 sec

    2020-08-26 12:30:30.127 OMNICC.7792.9332 ["/lib/ipc/ipc.c $Rev$ $Date:: $:":3934] A.10.70 b210
    gethostbyaddr(slm-bkp.corp.te.ru) .Network performance warning: took 0,826000 sec

    2020-08-26 12:30:30.949 OMNICC.7792.9332 ["/lib/ipc/ipc.c $Rev$ $Date:: $:":3934] A.10.70 b210
    gethostbyaddr(te-rz.corp.te.ru) .Network performance warning: took 0,820000 sec

    2020-08-26 12:48:09.216 INET.1076.8468 ["/inet/inet_util.c $Rev$ $Date:: $:":2020] A.10.70 b210
    [RxSmallFtp] peerHost:id-srk-02.corp.te.ru, peerProg:DBSM, event:Rejected:/barlists/veagent/id vm id-pyramid-01..04 (temporary)
    [KeycloakRESTApiConnector::restMethodExecute] Response: 400 body:

    {"error":"invalid_grant","error_description":"Invalid refresh token"}

    2020-08-26 12:54:59.775 DBSM.9276.10212 ["/lib/cmn/safemem.c $Rev$ $Date:: $:":112] A.10.70 b210
    MALLOC failed Size=18446744072703442944

    2020-08-26 12:54:59.775 DBSM.9276.10212 ["/lib/cmn/safemem.c $Rev$ $Date:: $:":120] A.10.70 b210
    MALLOC failed Size=18446744072703442944, errcode=8

     

    Application log - see picture debug&Application.log.jpeg.

     

     

     

     

Children
No Data