With an implementation of eDirectory and Identity Manager on a multi-node LinuxHA cluster, each server in the cluster can potentially host any of several configured instances. In order to back up all of the instances, I set off looking for a good way to put a script in cron that could be used to back up the DIBs to files, then the regular system backup can spool those off to tapes. I found an older Cool Solutions article (Scheduled Backup of Clustered Multiple eDirectory Instances with dsbk) that had some good ideas, but it did not take full advantage of what was available, and it did not handle cases where the instance may be configured, but is not currently hosted on this particular node.

So I set off to improve it. The results are listed in the complete script below. But, code without explanation and documentation is rarely useful after an upgrade or two or other changes, so here is a longer explanation of what it does, how it does it, and most important, why it does it.

#!/bin/bash

# Backup Options:
#
#      'backup'    Perform an eDirectory backup
#      -b             Performs a full backup of the eDirectory database.
#      -f             Specifies the path and file to back up to
#      -l             Specifies the log file for the back up
#      -t             Includes the stream files when backing up the eDirectory database.
#      -w             Overwrites the backup file specified with the -f switch if a file of the same name already exists.
#      -e <pwd>       Includes the NICI encryption keys using password <pwd>

TODO=backup
BACKUPTYPE=-b
BACKUPTO=-f
BACKUPLOG=-l
BACKUPSTREAMS=-t
OVERWRITE=-w
BACKUPNICI=-e
NICIPWD=secret
BACKUPPATH=/opt/novell/edir.backups

In this first block of code, we have the invocation of the correct script interpretter (bash), some documentation as to what the various options are to the backup agent, and default settings. These are the options I wanted for my backups. You may want other options, especially if you're interested in using incremental backups. You will want to specify a better NICI backup password, as what is being backed up here is the security keys for the encryption used in the database. These need to be protected.

if [ -z "$1" ]; then
 if [ "$1" != "-q" ]; then
  echo "Running in verbose mode. Use -q for fewer messages."
  VERBOSE="yes"
 fi
else
 VERBOSE="no"
fi

Most of the time when I am developping and testing code, whether shell scripts, Javascript, or C source, I litter the code with output of some sort intended to help show what is going on and where in the code it is happening. This aids developping and debugging, but quickly gets to be annoying levels of cruft produced when the code is tested and in production. So, here I am setting up the option of turning that stuff all off, but defaulting it to on. Normally, you will see a lot of output from this script. If you want to see less, use the "-q" command line parameter to make it go away.

# For each configured eDirectory instance, get the instance configuration file
for i in $(cat /etc/opt/novell/eDirectory/conf/.edir/instances.0); do
 if [ "$VERBOSE" == "yes" ]; then
  echo "Parsing config file $i"
 fi
 DIBDIR=`grep "n4u.nds.dibdir" $i | cut -d '=' -f 2`
 TREENAME=`grep "n4u.base.tree-name" $i | cut -d '=' -f 2`

 if [ "$VERBOSE" == "yes" ]; then
  echo "  DIBDIR is $DIBDIR"
  echo "  TREENAME is $TREENAME"
 fi

The first task to be accomplished is to find what needs to be backed up. Since I have many instances configured, but many servers to host them on (using LinuxHA clustering to move the instances from node to node in the cluster), I did not want to have to change this backup script every time a new instance is added to the cluster or an old one is removed.

eDirectory stores the list of configured instances in a plain text file (/etc/opt/novell/eDirectory/conf/.edir/instances.0). When adding a new instance to the cluster, this file has to be replicated to all cluster nodes. This file lists the configuration files for the configured instances. So it might look something like:

/etc/nds-1.conf
/etc/nds-2.conf
/etc/nds-3.conf

The actual names of the configuration files are set during the eDirectory instance creation and could be anything, and could be in any path. The only thing we know for sure is that they will all be listed in the instances.0 file. (This is for 'root' eDirectory installations. 'Non-root' installations have a similar setup.)

In order to find what we need to know, the for() loop goes through the lines in the instances.0 file, and each one is checked. The DIBDIR variable is set to the path to the database, which is obtained from the configuration file. Then the TREENAME variable is set to the name of the tree, also obtained from the configuration file.

You can see more detail of this being done if you run the script in verbose mode. Normally we don't care about watching this happen, so long as it does.
As each configured instance is found and its configuration file is parsed, the next thing to check for is the existance of the instance's database (DIB) directory. As this is a cluster, where the DIB is actually stored on a SAN and moved between nodes, it may be here, or it may be elsewhere on another node right now.

If the database directory exists, then we want to back it up on this node.

# eDirectory 8.8.5 version
# Check to see if the instance is currently hosted on this server
 if [ -d "$DIBDIR" ]; then
  echo "    Tree $TREENAME is hosted on this server. Backup starting: `date`."
  echo "/tmp/dsbk.config" > /etc/dsbk.conf
  echo "$TODO $BACKUPTYPE $BACKUPTO $BACKUPPATH/$TREENAME.backup $BACKUPLOG $BACKUPPATH/$TREENAME-backup.log $BACKUPSTREAMS $OVERWRITE $BACKUPNICI $NICIPWD" > /tmp/dsbk.config
  /opt/novell/eDirectory/bin/ndstrace --config-file $i -c "load dsbk"

Note: In eDirectory 8.8.5, the 'dsbk' backup agent has one strange requirement. It must find a file /etc/dsbk.conf containing a path and file (/tmp/dsbk.config) that contains the command line options you want it to use. It then removes this file. I do not know why it needs this.

# eDirectory 8.8.6 version
# Check to see if the instance is currently hosted on this server
 if [ -d "$DIBDIR" ]; then
  echo "    Tree $TREENAME is hosted on this server. Backup starting: `date`."
  /opt/novell/eDirectory/bin/ndstrace -c "load dsbk $TODO $BACKUPTYPE $BACKUPTO $BACKUPPATH/$TREENAME.backup $BACKUPLOG $BACKUPPATH/$TREENAME-backup.log $BACKUPSTREAMS $OVERWRITE $BACKUPNICI $NICIPWD --config-file $i"

Note: In eDirectory 8.8.6, this use of /etc/dsbk.conf and /tmp/dsbk.config is no longer required, and has been replaced by passing the entire command line to ndstrace.

After taking care of the /etc/dsbk.conf and /tmp/dsbk.config files, the actual backup is kicked off using dstrace to load the dsbk module. This seems a convoluted way to start a backup, but it works. The problem is that it runs the backup asynchronously, in the background, and there is no indication that it completed.

# Wait for backup to complete
  OSIZE=0
  CSIZE=1

  while [ $OSIZE -ne $CSIZE ];
  do
    if [ "$VERBOSE" == "yes" ]; then
      echo "     Waiting for backup of $TREENAME to complete."
    fi
    sleep 10
    OSIZE=$CSIZE
    CSIZE=`stat --format="%s" $BACKUPPATH/$TREENAME.backup`
  done
  if [ "$VERBOSE" == "yes" ]; then
    echo
  fi
  echo "    Backup of $TREENAME done: `date`."
  sleep 10

I did not want to have multiple instance backups running simultaneously, and I wanted to know that the backup had started, and most importantly I wanted to know that it had completed. So another loop in the script takes care of this. The backup file is checked every 10 seconds to see if it is changing size. If the size is changing, then the backup is not done yet.

Once the loop is finally satisfied, an additional 10 second wait is used to ensure that it is really completely done. This was added after testing where I found that the backup itself may be done, and the backup file stops changing in size, but eDirectory hasn't completely written out the log file yet. This pause gives it a chance to finish that, too, before the script continues.

  if [ "$VERBOSE" == "yes" ]; then
    echo "    Checking status of backup."
  fi
  tail -1 $BACKUPPATH/$TREENAME-backup.log | grep "success" > /dev/null
  if [ "$?" -ne 0 ]; then
    echo "     Backup failed."
    cat $BACKUPPATH/$TREENAME-backup.log
    exit 1
  else
    echo "     Backup completed successfully."
  fi

After the backup completes, the backup log file is checked to see if the most recent entry contains the word "success" or not and an appropriate message is produced. This is important to monitor, so that when a disaster happens you are not surprised to find out that your backups are no good.

Here's the complete script:

#!/bin/bash

# eDirectory 8.8.5 version
# Backup Options:
#
#      'backup'    Perform an eDirectory backup
#      -b             Performs a full backup of the eDirectory database.
#      -f             Specifies the path and file to back up to
#      -l             Specifies the log file for the back up
#      -t             Includes the stream files when backing up the eDirectory database.
#      -w             Overwrites the backup file specified with the -f switch if a file of the same name already exists.
#      -e <pwd>       Includes the NICI encryption keys using password <pwd>

TODO=backup
BACKUPTYPE=-b
BACKUPTO=-f
BACKUPLOG=-l
BACKUPSTREAMS=-t
OVERWRITE=-w
BACKUPNICI=-e
NICIPWD=secret
BACKUPPATH=/opt/novell/edir.backups

if [ -z "$1" ]; then
 if [ "$1" != "-q" ]; then
  echo "Running in verbose mode. Use -q for fewer messages."
  VERBOSE="yes"
 fi
else
 VERBOSE="no"
fi


# For each configured eDirectory instance, get the instance configuration file
for i in $(cat /etc/opt/novell/eDirectory/conf/.edir/instances.0); do
 if [ "$VERBOSE" == "yes" ]; then
  echo "Parsing config file $i"
 fi
 DIBDIR=`grep "n4u.nds.dibdir" $i | cut -d '=' -f 2`
 TREENAME=`grep "n4u.base.tree-name" $i | cut -d '=' -f 2`

 if [ "$VERBOSE" == "yes" ]; then
  echo "  DIBDIR is $DIBDIR"
  echo "  TREENAME is $TREENAME"
 fi

# Check to see if the instance is currently hosted on this server
 if [ -d "$DIBDIR" ]; then
  echo "    Tree $TREENAME is hosted on this server. Backup starting: `date`."
  echo "/tmp/dsbk.config" > /etc/dsbk.conf
  echo "$TODO $BACKUPTYPE $BACKUPTO $BACKUPPATH/$TREENAME.backup $BACKUPLOG $BACKUPPATH/$TREENAME-backup.log $BACKUPSTREAMS $OVERWRITE $BACKUPNICI $NICIPWD" > /tmp/dsbk.config
  /opt/novell/eDirectory/bin/ndstrace --config-file $i -c "load dsbk"

# Wait for backup to complete
  OSIZE=0
  CSIZE=1

  while [ $OSIZE -ne $CSIZE ];
  do
    if [ "$VERBOSE" == "yes" ]; then
      echo "     Waiting for backup of $TREENAME to complete."
    fi
    sleep 10
    OSIZE=$CSIZE
    CSIZE=`stat --format="%s" $BACKUPPATH/$TREENAME.backup`
  done
  if [ "$VERBOSE" == "yes" ]; then
    echo
  fi
  echo "    Backup of $TREENAME done: `date`."
  sleep 10
  if [ "$VERBOSE" == "yes" ]; then
    echo "    Checking status of backup."
  fi
  tail -1 $BACKUPPATH/$TREENAME-backup.log | grep "success" > /dev/null
  if [ "$?" -ne 0 ]; then
    echo "     Backup failed."
    cat $BACKUPPATH/$TREENAME-backup.log
    exit 1
  else
    echo "     Backup completed successfully."
  fi
 else
  echo "    Tree $TREENAME is not currently hosted on this server. Skipping..."
 fi
 echo

done

#!/bin/bash

# eDirectory 8.8.6 version
# Backup Options:
#
#      'backup'    Perform an eDirectory backup
#      -b             Performs a full backup of the eDirectory database.
#      -f             Specifies the path and file to back up to
#      -l             Specifies the log file for the back up
#      -t             Includes the stream files when backing up the eDirectory database.
#      -w             Overwrites the backup file specified with the -f switch if a file of the same name already exists.
#      -e <pwd>       Includes the NICI encryption keys using password <pwd>

TODO=backup
BACKUPTYPE=-b
BACKUPTO=-f
BACKUPLOG=-l
BACKUPSTREAMS=-t
OVERWRITE=-w
BACKUPNICI=-e
NICIPWD=secret
BACKUPPATH=/opt/novell/edir.backups

if [ -z "$1" ]; then
 if [ "$1" != "-q" ]; then
  echo "Running in verbose mode. Use -q for fewer messages."
  VERBOSE="yes"
 fi
else
 VERBOSE="no"
fi


# For each configured eDirectory instance, get the instance configuration file
for i in $(cat /etc/opt/novell/eDirectory/conf/.edir/instances.0); do
 if [ "$VERBOSE" == "yes" ]; then
  echo "Parsing config file $i"
 fi
 DIBDIR=`grep "n4u.nds.dibdir" $i | cut -d '=' -f 2`
 TREENAME=`grep "n4u.base.tree-name" $i | cut -d '=' -f 2`

 if [ "$VERBOSE" == "yes" ]; then
  echo "  DIBDIR is $DIBDIR"
  echo "  TREENAME is $TREENAME"
 fi

# Check to see if the instance is currently hosted on this server
 if [ -d "$DIBDIR" ]; then
  echo "    Tree $TREENAME is hosted on this server. Backup starting: `date`."
  /opt/novell/eDirectory/bin/ndstrace -c "load dsbk $TODO $BACKUPTYPE $BACKUPTO $BACKUPPATH/$TREENAME.backup $BACKUPLOG $BACKUPPATH/$TREENAME-backup.log $BACKUPSTREAMS $OVERWRITE $BACKUPNICI $NICIPWD --config-file $i"

# Wait for backup to complete
  OSIZE=0
  CSIZE=1

  while [ $OSIZE -ne $CSIZE ];
  do
    if [ "$VERBOSE" == "yes" ]; then
      echo "     Waiting for backup of $TREENAME to complete."
    fi
    sleep 10
    OSIZE=$CSIZE
    CSIZE=`stat --format="%s" $BACKUPPATH/$TREENAME.backup`
  done
  if [ "$VERBOSE" == "yes" ]; then
    echo
  fi
  echo "    Backup of $TREENAME done: `date`."
  sleep 10
  if [ "$VERBOSE" == "yes" ]; then
    echo "    Checking status of backup."
  fi
  tail -1 $BACKUPPATH/$TREENAME-backup.log | grep "success" > /dev/null
  if [ "$?" -ne 0 ]; then
    echo "     Backup failed."
    cat $BACKUPPATH/$TREENAME-backup.log
    exit 1
  else
    echo "     Backup completed successfully."
  fi
 else
  echo "    Tree $TREENAME is not currently hosted on this server. Skipping..."
 fi
 echo

done