ZDB : [ 236:52] Failed to resolve a storage volume on the host

Hi,

did anyone face this error during zdb backup {snapshot/split mirror =>storage plugin} on vmware environment? Please suggest.

Setup:

cell manager {virtual machine} 9.07

backup host [RHEL 6.8]

backup host [RHEL 6.8]

Datastore lun is presented to backup host

------------------------------------------------------------------------------------------------

[Normal] From: BSM@VM_CELL_MANAGER "New2_1"  Time: 8/18/2016 2:39:37 PM
 Backup session 2016/08/18-20 started.

[Normal] From: BSM@VM_CELL_MANAGER "New2_1"  Time: 8/18/2016 2:39:37 PM
 OB2BAR application on "Physical_Backup_Host" successfully started.

[Normal] From: VEPALIB_VMWARE@Physical_Backup_Host ""  Time: 08/18/2016 02:39:38 PM
 Resolving objects for backup on vCenter 'bkc099vic072in.rdc-infra.local' ...

[Normal] From: VEPALIB_VMWARE@Physical_Backup_Host ""  Time: 08/18/2016 02:42:50 PM
 Add Virtual Machine to the backup ...
  Name: Virtual_Machine_DPclient
  Path: /Data_center/vm/INFRA/Network Security/Virtual_Machine_DPclient
  InstanceUUID: 503fd356-9dea-316c-a3d4-45f34bcd04b7

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:42:50 PM
 Starting agent on Physical_Backup_Host.

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:42:50 PM
 Starting agent on Physical_Backup_Host.

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:42:50 PM
 Starting backup session with session ID 2016/08/18-20

  Snapshot source: Original volume
  Snapshot type: VSnap
  Number of replicas rotated: 0
  VRaid level: Same as source
  Use the same mount points as on the application system.

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:42:50 PM
 Resolving backup objects on the application system.

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:42:50 PM
 Resolving of backup objects on the application system completed.

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:42:51 PM
 Checking the HPE 3PAR provider using this
 connection data:
  Host:  10.10.10.10
  User:  3paradm
  Namespace: root/tpd
  Port:  5988
  SSL mode: FALSE

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:42:51 PM
 This HPE 3PAR provider has access to the following unit:
   Name:  3par_storage
   WWN:  2FF70002AC01A3B1
   Description: HP_3PAR 8440, ID: 107000, Serial number: Serial, InForm OS version: 3.2.2 (MU2)

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:42:54 PM
 Presentations were successfully located for this storage volume:
 Storage volume : 3PAR_VOLUME.61
 VVName : 3PAR_VOLUME.61
 Host   : TEMPESXi2
 Type   : HOST_SEES
 LUNID  : 61
 Port   : -

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:42:54 PM
 Presentations were successfully located for this storage volume:
 Storage volume : 3PAR_VOLUME.61
 VVName : 3PAR_VOLUME.61
 Host   : TEMPESXi1
 Type   : HOST_SEES
 LUNID  : 61
 Port   : -

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:42:54 PM
 Presentations were successfully located for this storage volume:
 Storage volume : 3PAR_VOLUME.61
 VVName : 3PAR_VOLUME.61
 Host   : PHYSICAL_BACKUP_HOST2
 Type   : HOST_SEES
 LUNID  : 61
 Port   : -

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:42:54 PM
 Presentations were successfully located for this storage volume:
 Storage volume : 3PAR_VOLUME.61
 VVName : 3PAR_VOLUME.61
 Host   : ESXHOST_24
 Type   : HOST_SEES
 LUNID  : 61
 Port   : -

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:42:54 PM
 Presentations were successfully located for this storage volume:
 Storage volume : 3PAR_VOLUME.61
 VVName : 3PAR_VOLUME.61
 Host   : ESXHOST_20
 Type   : HOST_SEES
 LUNID  : 61
 Port   : -

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:42:54 PM
 Presentations were successfully located for this storage volume:
 Storage volume : 3PAR_VOLUME.61
 VVName : 3PAR_VOLUME.61
 Host   : Physical_Backup_Host
 Type   : HOST_SEES
 LUNID  : 61
 Port   : -

[Normal] From: VEPALIB_VMWARE@Physical_Backup_Host ""  Time: 08/18/2016 02:42:56 PM
 Virtual Machine 'Virtual_Machine_DPclient': Locking vMotion ...

[Normal] From: VEPALIB_VMWARE@Physical_Backup_Host ""  Time: 08/18/2016 02:42:58 PM
 Creating folder /var/opt/omni/tmp/da9376a5-5dc9-4265-a43b-5ad083580aad ...

[Normal] From: VEPALIB_VMWARE@Physical_Backup_Host ""  Time: 08/18/2016 02:42:58 PM
 Virtual Machine 'Virtual_Machine_DPclient': Backing up configuration file Virtual_Machine_DPclient.vmx ...

[Normal] From: VEPALIB_VMWARE@Physical_Backup_Host ""  Time: 08/18/2016 02:43:00 PM
 Virtual Machine 'Virtual_Machine_DPclient': Creating snapshot ...

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:43:03 PM
 Creation of replica started on host Physical_Backup_Host.

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:43:03 PM
 Creating a replica as user "3paradm" for the following volume:
  Source volume: 3PAR_VOLUME.61
  Replica name: DP-2016.08.18-20-057B59F47
  Replica type: Demand allocated snapshot
  Raid level:  RAID5
  Target CPG:  R5_CPG_CS

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:43:04 PM
 Creation of replica on host Physical_Backup_Host completed.

[Normal] From: VEPALIB_VMWARE@Physical_Backup_Host ""  Time: 08/18/2016 02:43:07 PM
 Virtual Machine 'Virtual_Machine_DPclient': Removing snapshot ...

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:43:09 PM
 Creating a presentation as user "3paradm" for the following storage volume:
  Volume:   DP-2016.08.18-20-057B59F47
  Host:    ESXHOST_15.FQDN
  Presentation type: Host Sees
  Array port:   -

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:43:11 PM
 Creating a presentation as user "3paradm" for the following storage volume:
  Volume:   DP-2016.08.18-20-057B59F47
  Host:    Physical_Backup_Host
  Presentation type: Host Sees
  Array port:   -

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:43:13 PM
 Starting drive scan.

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:43:53 PM
 Drive scan has completed.

[Major] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:43:53 PM
[236:52]  Failed to resolve a storage volume on the host.
 Storage volume: DP-2016.08.18-20-057B59F47

[Normal] From: VEPALIB_VMWARE@Physical_Backup_Host ""  Time: 08/18/2016 02:44:12 PM
 Mounting datastores to host 'ESXHOST_15.FQDN' ...

[Critical] From: VEPALIB_VMWARE@Physical_Backup_Host ""  Time: 08/18/2016 02:44:12 PM
 Error mounting datastores ...

[Normal] From: VEPALIB_VMWARE@Physical_Backup_Host ""  Time: 08/18/2016 02:44:12 PM
 Virtual Machine 'Virtual_Machine_DPclient': Unlocking vMotion ...

[Normal] From: BSM@VM_CELL_MANAGER "New2_1"  Time: 8/18/2016 2:44:15 PM
 OB2BAR application on "Physical_Backup_Host" disconnected.

[Critical] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:44:15 PM
 ABORT request received from session manager. SMIS-Backup agent will start with clean up.

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:44:16 PM
 Removing a presentation as user "3paradm" for the following storage volume:
  Volume: DP-2016.08.18-20-057B59F47
  Host:  ESXHOST_15.FQDN

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:44:17 PM
 Removing a presentation as user "3paradm" for the following storage volume:
  Volume: DP-2016.08.18-20-057B59F47
  Host:  Physical_Backup_Host

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:44:19 PM
 Starting drive scan.

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:44:59 PM
 Drive scan has completed.

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:44:59 PM
 Next disks will be deleted:
 DP-2016.08.18-20-057B59F47

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:44:59 PM
 Deleting the following volume as user "3paradm":
  Storage volume: DP-2016.08.18-20-057B59F47

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:44:59 PM
 ABORTED SMIS-Backup agent on Physical_Backup_Host.

[Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:44:59 PM
 ABORTED SMIS-Application agent on Physical_Backup_Host.

[Critical] From: BSM@VM_CELL_MANAGER "New2_1"  Time: 8/18/2016 2:44:59 PM
 None of the Disk Agents completed successfully.
 Session has failed.

[Normal] From: BSM@VM_CELL_MANAGER "New2_1"  Time: 8/18/2016 2:44:59 PM

 Backup Statistics:
         
  Session Queuing Time (hours)         0.09       
  -------------------------------------------     
  Completed Disk Agents ........          0         
  Failed Disk Agents ...........          0         
  Aborted Disk Agents ..........          0         
  -------------------------------------------     
  Disk Agents Total  ...........          0         
  ===========================================     
  Completed Media Agents .......          0         
  Failed Media Agents ..........          0         
  Aborted Media Agents .........          0         
  -------------------------------------------     
  Media Agents Total  ..........          0         
  ===========================================     
  Throttled Gateways ...........          0         
  Gateways Total ...............          1         
  ===========================================     
  Overall Deduplication Ratio ..          -           
  ===========================================     
  Mbytes Total .................       0 MB       
  Used Media Total .............          0         
  Disk Agent Errors Total ......          0   


============================================================================
                      Session failed!
============================================================================

 

 

 

Parents
  • Hi,

    To be certain i would need to see debug logs of SMIS on backup host. But my assumption is that problem is here:

    [Normal] From: SMISA@Physical_Backup_Host "SMISA"  Time: 8/18/2016 2:43:09 PM
     Creating a presentation as user "3paradm" for the following storage volume:
      Volume:   DP-2016.08.18-20-057B59F47
      Host:    ESXHOST_15.FQDN
      Presentation type: Host Sees
      Array port:  

    Please check on 3PAR that host "ESXHOST_15.FQDN" has the same name as it is seen in DNS. Also note the case sensitivity of hostname -> this was introduced with 9.07.

    Regards AlesKol

  • And please do the same check for "Physical_Backup_Host"

  • Hi Aleskol,

     many thanks for suggestion. i've checked DNS resolution for ESX, 3par storage, backup hosts. all are OK - names are resolvable how they should be in small letters.
     
    i've checked debug file and noticed that he is trying to resolve volume to device files at backup host [marked in red color].
     
    but still i couldn't find any solution of this problem.
     
    i see some warning at /var/log/message [backup host] but not sure if it is the reason of failure.
     
    kernel: sd 3:0:6:56: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
     
    #### debug logs
     
     [199] 2016-08-18 23:14:02.147 ("/integ/smis/Communication/Connections/IpcConnection.cpp $Rev: 9836 $ $Date:: 2010-06-15 07:30:58         ":317)
      23468 [199] <<=== (5) }  /* IpcConnection::send */
      23469 [199]
      23470 [100] *************** SKPrintf [START] ***************
      23471 [100] Tokenize the string...
      23472 [100] token [0] = ^G[%1$s] From: %3$s@%4$s "%6$s"  Time: %2$s
      23473 %5$s
      23474
      23475 [100] token [1] = Major
      23476 [100] token [2] = 08/18/2016 11:14:02 PM
      23477 [100] token [3] = SMISA
      23478 [100] token [4] = backup_host.fqdn
      23479 [100] token [5] = ^G[236:52]    Failed to resolve a storage volume on the host.
      23480         Storage volume: %s
      23481
      23482 [100] token [6] = DP-2016.08.18-44-057B616AE
      23483 [100] token [7] = SMISA
      23484 [100] *************** SKPrintf [END] ***************
      23485
      23486 [  4] 2016-08-18 23:14:02.147 ("/integ/smis/Resolver/ResolverUxImpl.cpp $Rev: 52654 $ $Date:: 2016-03-23 06:11:33 ":1016)
      23487 [  4] [REPORT 2] in ResolverUxImpl::ResStorageVolumes2DeviceFiles
      23488     Marked: ("":Line:0 in '')
      23489     Code is:0
      23490 [Major] From: SMISA@backup_host,fqdn"SMISA"  Time: 08/18/2016 11:14:02 PM
      23491 [236:52]        Failed to resolve a storage volume on the host.
      23492         Storage volume: DP-2016.08.18-44-057B616AE
     
    Thanks,
    DPX
  • Hi,

     

    If you have debug logs, can you share them? Only SMIS from backup host is needed.

    Or if you can check for any of the following entries:

    • Search for “[ -2]” and provide output -10 lines
    • Search for “error” and if in - 10 lines (somewhere near) you can find also “exec3PARCliCommand” or “CDpArrayProviderObj” or “WebServiceAPI3ParMgmt” provide those lines also.

     

    Regards

    AlesKol

  • Many thanks ales for reply.

    I've attached the logs. Please change the file extention to .7z

    thanks,

    DPX

     

  • Hi,

    I am sorry, can’t help you here. Seems that communication with array is OK, but smis fail to find disk on backup host.

    There could be multiple reasons for this. Maybe if you would share entire log, I might be able to help.

    Otherwise I suggest that you open a call with HPE.

     

    Regards Aleskol

  • Many thanks , i have also the same impression that backup host is not able to read device information.

    I've attached the logs. Please rename extension of the file to .7z.

    DPX

     

  • Many thanks , i have also the same impression that backup host is not able to read device information.

    I've attached the logs. Please rename extension of the file to .7z.

    DPX

     

  • Many thanks , i have also the same impression that backup host is not able to read device information.

    I've attached the logs. Please rename extension of the file to .7z.

    DPX

     

  • Verified Answer

    Hi,

     

    From logs I was able to determine:

    • Replica is successfully created
    • Replica is successfully presented
    • After presentation rescan is initiated on backup host
    • Seems that after rescan replica is still not seen on backup host.

     

    Would suggest following:

    • On backup host set following 2 omnirc variables:

    ZDB_DELAY_BEFORE_RESCAN=120

    ZDB_DELAY_AFTER_RESCAN=120

    • With these settings you will get 4 minutes to determine if volume/replica is visible on backup host after presentation is made.

    Would suggest that you connect to backup host, and perform rescan of FC manually (rescan-scsi-bus.sh)

    And determine that replica is visible on backup host.

     

    Could be that delay itself will give enough time to backup host to recognize the new volume.

     

    If you still cant find volume on the backup host then I suggest you check your settings on backup host.

    • Create replica manually on 3PAR host
    • Present it manually to backup host
    • Rescan and try to find volume on backup host
    • Make sure that is visible with “multipath -ll” command

    /etc/multipath.conf is where I would start looking.

    I have seen cases where “find_multipaths = yes” setting was preventing backup host to see volume.

     

    Please let me know the findings & regards

    Ales

  • Many thanks Ales. It was issue with multipath configuration on the backup host.

    Regards,

    DPx

Reply Children
No Data