Cybersecurity
DevOps Cloud (ADM)
IT Operations Cloud
There have been more documents related to advanced Patch management with ZENworks Linux Management, for example, this one in the wiki.
However, these articles are all based upon YUP, which works well, but YUP itself is not a supported solution and the yup and yup2zlm scripts are fairly complex and thus not easy to adapt for many people.
Note: One of the benefits of YUP though, is the generation of a yum tree - an rpm-md repository, which can be used as installation source for e.g. yum, zen-updater, rug or the zypper client. The same benefit can be achieved using zlmmiror in combination with:
This is, however, beyond the scope of this document.
In this AppNote I will describe how to patch both SLES 10 and OES2 servers using ZENworks Linux Management. We will be using a TEST – Pre-Production – Production structure for save deployment, using the default zlmmirror options introduced with the latest Support pack and hot patches.
I have also tried to keep the scripts simple and easy to adapt to your specific scenario.
Probably the most important task for the ZENworks Linux Management server in a data centre will be maintaining patches for the Linux servers. Given the amount of patches that come out, it is crucial that an automated procedure is in place. Obviously, patches will first need to be tested, but after that, the deployment should be handled automatically.
The following strategy will be used in this document.
Patches will be placed inside different categories or groups:
The optional, recommended and security categories will be used for test and pre-production. In the production environment, we will group them together as there is less requirement for differentiation.
Using zlmmirror, patches will be pulled in and placed inside a zlmmirror folder. Per OS we will have a catalog for optional, recommended and security, pointing to the actual patch bundles.
Using scripts we will copy new patches to the test groups.
After testing, a script can be used to move the tested patches to the pre-production environment.
Once patches have been successfully deployed in a pre-production environment, they can be copied to the production group.
As an example, the testing procedure can be as follows:
Note: Please note that by no means will this procedure ensure that patching will always be flawless. However, it makes the most efficient use of resources and manpower while still providing an adequate level of security for many deployments.
To distribute and install RPMs and patches to managed devices, you need to ensure that all packages the RPMs depend on, and all packages dependent on the RPM are also imported to the ZLM server. For example, to distribute updates to the SLES10 devices mirrored from https://nu.novell.com, you need to ensure that all the packages from the SLES10 media are imported to the ZLM server and assigned to a catalogue. You need to assign the catalogue containing the installation source to the managed devices.
In our environment we have created several installation sources and made available the following catalogs:
The catalogues can be created by importing all the rpm's from the different architecture directories in your repository using the command:
zlman bap /path/bundle-name os-target *.rpm
ZLMMirror
With the new --category switch, it is now possible to download only specific categories of patches, these categories are:
Based on the category, you will be able to decide what patches to give priority in testing.
Since zlmmirror will be used on the ZENworks Linux Management server to pull in patches for all platforms (SLES10 SP1, OES2, different architectures), it is best to use the mirrored credentials set available from the Novell Customer Center. This allows us to get all patches for all operating systems at once.
To automate the file retrieval and storage processes, please edit the /etc/crontab file and put in a few scripts to get patches each morning. For example:
## Getting updates from nu.novell.com and placing them inside ZLM
00 8 * * * root /var/zlmdata/scripts/mirror-SLES10-SP1.sh
55 8 * * * root /var/zlmdata/scripts/mirror-OES2.sh
After the patches have been downloaded, the following structure is present in ZENworks Linux Management:
These folders will not be modified. Several scripts have been created to copy the contents to a working structure.
The following zlmmirror confguration files are available in Appendix A:
At the base level of /Bundles, the following structure must be created for all test, pre-production and production systems:
Note: For this example, we did not differentiate between the different categories for OES patches. If you think this is required, it is easy to change according to the examples provided for SLES 10 SP1.
According to this example, server assignment should be handled as follows:
And, since any OES2 server is also a SLES10 SP1 server:
Several scripts have been created inside the /var/zlmdata/scripts folder to make the task of moving bundles and running zlmmirror easier. The scripts are available in Appendix A.
The following scripts are available for either SLES10 or OES2:
These scripts can be safely run by Cron once or twice a day to pull in new patches.
These scripts should be run manually after each accompanying stage of the testing and patching process has been completed.
Note: Please note that the zlmSLESmoveTesttoPreProd.sh accepts the different categories as a parameter to move only those specific patch bundles to the Pre-Production group.
The two settings files also hold the Administrator name and password for ZENworks Linux Management. If you do not want to place them in the settings file, you can add them in a separate file, and remove the -U and -P parameters in all scripts.
To create the separate file, create a .zlmanrc file (notice the leading period!) in your home directory:
As an example, the patching procedure for OES2 servers is described below. The process is similar for SLES10 SP1 servers.
After the zlmmirror process completes, the zlmOES2copyNewtoTest.sh script needs to be run to copy all new patches (it searches for patches not yet available in the production environment) to the test folder.
After all patches have been tested and verified, the zlmOES2moveTesttoPreProd.sh script needs to be run to move all patches in the Test folder to the Pre-Production folder.
Two days later, after the patches have been deployed to a few production servers, the zlmOES2movePreProdtoProd.sh needs to be run to move all patches to their final production folder.
Note: Please note that every OES2 server is also a SLES10 SP1 server and thus needs those patches as well.
All scripts used in this document can be found below.
#! /bin/bash
#******** Purpose **********
# This script retrieves all SLES10-SP1 patches based on category.
# ********* History *********
# 21-11-2007 – V1.0 – Novell Consulting
LIST=(recommended security optional)
for Category in ${LIST[*]} ; do
zlmmirror m -p --force-nevra -c sles10_$Category.xml --category=$Category
done
# Still pull in all patches
zlmmirror m -p -c sles10_all.xml
# Essentially these commands are executed:
#zlmmirror m -p --force-nevra -c sles10_recommended.xml --category=recommended
#zlmmirror m -p --force-nevra -c sles10_optional.xml --category=optional
#zlmmirror m -p --force-nevra -c sles10_security.xml --category=security
#zlmmirror m -p --force-nevra -c sles10_all.xml
#! /bin/bash
#******** Purpose **********
# This script retrieves all OES2 patches.
# ********* History *********
# 21-11-2007 – V1.0 – Novell Consulting
zlmmirror m -p --force-nevra -c oes2.xml
<ZLMMirrorConf>
<Session>
<RemoteServer>
<Base>https://nu.novell.com/repo</Base>
<Proxy>http://10.0.0.1:8080</Proxy>
<Type>nu</Type>
<User>xxxxxx</User>
<Password>yyyyyyyyyy</Password>
</RemoteServer>
<LocalServer>
<Type>zlm</Type>
<User>administrator</User>
<Password>novell</Password>
</LocalServer>
<Catalog>
<Name>OES2-Updates</Name>
<LocalName></LocalName>
<Folder>zlmmirror/OES2</Folder>
<Target></Target>
<ExcludeTarget></ExcludeTarget>
<Bundle></Bundle>
<ExcludeBundle></ExcludeBundle>
<Package></Package>
<ExcludePackage></ExcludePackage>
</Catalog>
</Session>
</ZLMMirrorConf>
<ZLMMirrorConf>
<Session>
<RemoteServer>
<Base>https://nu.novell.com/repo</Base>
<Proxy>http://10.0.0.1:8080</Proxy>
<Type>nu</Type>
<User>xxxxxxxxx</User>
<Password>yyyyyyyyyyyy</Password>
</RemoteServer>
<LocalServer>
<Type>zlm</Type>
<User>administrator</User>
<Password>novell</Password>
</LocalServer>
<Catalog>
<Name>SLES10-SP1-Updates</Name>
<LocalName>SLES10-SP1-Updates-all</LocalName>
<Folder>zlmmirror/SLES10-SP1-Updates/</Folder>
<Target>sles-10-i586</Target>
<Target>sles-10-x86_64</Target>
<ExcludeTarget></ExcludeTarget>
<Bundle></Bundle>
<ExcludeBundle></ExcludeBundle>
<Package></Package>
<ExcludePackage></ExcludePackage>
</Catalog>
</Session>
</ZLMMirrorConf>
<ZLMMirrorConf>
<Session>
<RemoteServer>
<Base>https://nu.novell.com/repo</Base>
<Proxy>http://10.0.0.1:8080</Proxy>
<Type>nu</Type>
<User>xxxxxxxx</User>
<Password>yyyyyyyyyy</Password>
</RemoteServer>
<LocalServer>
<Type>zlm</Type>
<User>administrator</User>
<Password>novell</Password>
</LocalServer>
<Catalog>
<Name>SLES10-SP1-Updates</Name>
<LocalName>SLES10-SP1-Updates-recommended</LocalName>
<Folder>zlmmirror/SLES10-SP1-Updates/</Folder>
<Target>sles-10-i586</Target>
<Target>sles-10-x86_64</Target>
<ExcludeTarget></ExcludeTarget>
<Bundle></Bundle>
<ExcludeBundle></ExcludeBundle>
<Package></Package>
<ExcludePackage></ExcludePackage>
</Catalog>
</Session>
</ZLMMirrorConf>
<ZLMMirrorConf>
<Session>
<RemoteServer>
<Base>https://nu.novell.com/repo</Base>
<Proxy>http://10.0.0.1:8080</Proxy>
<Type>nu</Type>
<User>xxxxxxxx</User>
<Password>yyyyyyyyyy</Password>
</RemoteServer>
<LocalServer>
<Type>zlm</Type>
<User>administrator</User>
<Password>novell</Password>
</LocalServer>
<Catalog>
<Name>SLES10-SP1-Updates</Name>
<LocalName>SLES10-SP1-Updates-optional</LocalName>
<Folder>zlmmirror/SLES10-SP1-Updates/</Folder>
<Target>sles-10-i586</Target>
<Target>sles-10-x86_64</Target>
<ExcludeTarget></ExcludeTarget>
<Bundle></Bundle>
<ExcludeBundle></ExcludeBundle>
<Package></Package>
<ExcludePackage></ExcludePackage>
</Catalog>
</Session>
</ZLMMirrorConf>
<ZLMMirrorConf>
<Session>
<RemoteServer>
<Base>https://nu.novell.com/repo</Base>
<Proxy>http://10.0.0.1:8080</Proxy>
<Type>nu</Type>
<User>xxxxxxxx</User>
<Password>yyyyyyyyyy</Password>
</RemoteServer>
<LocalServer>
<Type>zlm</Type>
<User>administrator</User>
<Password>novell</Password>
</LocalServer>
<Catalog>
<Name>SLES10-SP1-Updates</Name>
<LocalName>SLES10-SP1-Updates-security</LocalName>
<Folder>zlmmirror/SLES10-SP1-Updates/</Folder>
<Target>sles-10-i586</Target>
<Target>sles-10-x86_64</Target>
<ExcludeTarget></ExcludeTarget>
<Bundle></Bundle>
<ExcludeBundle></ExcludeBundle>
<Package></Package>
<ExcludePackage></ExcludePackage>
</Catalog>
</Session>
</ZLMMirrorConf>
# General Settings
ZLMpass=novell
ZLMuser=administrator
# SLESLIST holds all valid catagories
SLESLIST=(recommended security optional)
# Prefixes for production, pre-production and test
ProdPref=Prod
PreProdPref=Pre-Prod
TestPref=Test
# SLESCatName = name of the catalogs without the SLESLIST extention.
SLESCatName=SLES10-SP1-Updates
# SLESSource = the folder that contains all catalogs created by zlmirror
SLESSource=/zlmmirror/SLES10-SP1-Updates
# SLESfold is the folder that contains all Bundle groups for the SLES10 test environment
SLESTest=Patches/Servers/sles10-sp1
# SLESProduction is the folder that contains all Bundle groups for the SLES10 Pre production environment
SLESProduction=Patches/Servers/sles10-sp1
# SLESPreProduction is the folder that contains all Bundle groups for the SLES10 production environment
SLESPreProduction=Patches/Servers/sles10-sp1
# SLESSourceLog is a temporary file listing the contents of the SLES10-SP1-Updates-patches folder
SLESSourceLog=/tmp/zlm-mirror-sles10-patches
# SLESProductionLog is een temporary file listing the contents of the production bundle group.
SLESProductionLog=/tmp/zlm-prod-sles10-patches
# Algemene Instellingen
ZLMpass=novell
ZLMuser=administrator
# OES2Source = The folder where all OES2 patches retrieved by zlmmirror are at
OES2Source=/zlmmirror/OES2/OES2-Updates-patches
# OES2Test is the Bundle group where all to be tested patches for OES2 should be added
OES2Test=Patches/Servers/oes2/Test
# OES2Production is the Bundle Group where all OES2 production patches should be
OES2Production=Patches/Servers/oes2/prod
# OES2Production is the Bundle Group where all OES2 pre-production patches should be
OES2PreProduction=Patches/Servers/oes2/Pre-Prod
# OES2SourceLog is a temporary file listing the contents of the OES2-Updates-patches folder
OES2SourceLog=/tmp/zlm-all-oes2-patches
# OES2ProductionLog is a temporary file listing the contents of the OES2 Productie Bundle Group
OES2ProductionLog=/tmp/zlm-prod-oes2-patches
#! /bin/bash
#******** Purpose **********
# This script will compare the production and zlmmirror patches and copy all “new” patches to the test group.
# ********* History *********
# 21-11-2007 – V1.0 – Novell Consulting
# Source in all settings
. /var/zlmdata/scripts/zlmSLESsettings
# Clean up
if [ -e $SLESProductionLog ] ; then
rm $SLESProductionLog
fi
if [ -e $SLESSourceLog ] ; then
rm $SLESSourceLog
fi
# Read bundle and folder and write to temp files for zlmmirror and production
for Category in ${SLESLIST[*]} ; do
zlman clb -U $ZLMuser -P $ZLMpass -V $SLESSource/$SLESCatName-$Category |grep 'Software Bundle'|cut --fields=1 --delimiter=' ' |sort >$SLESSourceLog
zlman bgm -U $ZLMuser -P $ZLMpass -V $SLESProduction |grep 'Software Bundle'|cut --fields=1 --delimiter=' ' | sort >$SLESProductionLog
# Should not happen. This only occurs if patches exist in the
# production group that are not present in the “all” bundle.
if [ -n "`comm -1 -3 $SLESSourceLog $SLESProductionLog`" ] ; then
echo "WARNING-ZLM: the following patches should probably not be in the production group: "$SLESProduction": "
echo ""
echo " `comm -1 -3 $SLESSourceLog $SLESProductionLog` "
fi
# Defines list of "newest" patches
for NEW in `comm -3 -2 $SLESSourceLog $SLESProductionLog` ; do
zlman bga -U $ZLMuser -P $ZLMpass -V $SLESTest/$TestPref-$Category $SLESSource/$SLESCatName-patches/"$NEW"
done
# Clean up after ourselves
rm $SLESProductionLog
rm $SLESSourceLog
done
#! /bin/bash
#******** Purpose **********
# This script moves all patches from the Test groups to the pre-production groups.
# ********* History *********
# 21-11-2007 – V1.0 – Novell Consulting
# Source in all settings
. /var/zlmdata/scripts/zlmSLESsettings
# Accept category as input value
if [ -z "$1" ] ; then
# No command line entered
# Defines list of "newest" patches
for Category in ${SLESLIST[*]} ; do
for PATCHES in `zlman bgm -U $ZLMuser -P $ZLMpass -V $SLESTest/$TestPref-$Category |grep 'Software Bundle'|cut --fields=1 --delimiter=' ' | sort` ; do
# copy patches to pre-production group
zlman bga -U $ZLMuser -P $ZLMpass -V $SLESPreProduction/$PreProdPref-$Category $PATCHES
# remove patches from test group
zlman bgr -U $ZLMuser -P $ZLMpass -V $SLESTest/$TestPref-$Category $PATCHES
done
done
else
# Defines list of "newest" patches
for PATCHES in `zlman bgm -U $ZLMuser -P $ZLMpass -V $SLESTest/$TestPref-$1 |grep 'Software Bundle'|cut --fields=1 --delimiter=' ' | sort` ; do
# copy patches to pre-production group
zlman bga -U $ZLMuser -P $ZLMpass -V $SLESPreProduction/$PreProdPref-$1 $PATCHES
# remove patches from test group
zlman bgr -U $ZLMuser -P $ZLMpass -V $SLESTest/$TestPref-$1 $PATCHES
done
fi
#! /bin/bash
#******** Purpose **********
# This script moves all SLES10-SP1 patches from the various pre-prod groups to the production group
# ********* History *********
# 21-11-2007 – V1.0 – Novell Consulting
# Source in all settings
. /var/zlmdata/scripts/zlmSLESsettings
for Category in ${SLESLIST[*]} ; do
# Defines list of current "pre-prod" patches
for PATCHES in `zlman bgm -U $ZLMuser -P $ZLMpass -V $SLESPreProduction/$PreProdPref-$Category |grep 'Software Bundle'|cut --fields=1 --delimiter=' ' | sort` ; do
# copy patches to production group
zlman bga -U $ZLMuser -P $ZLMpass -V $SLESProduction $PATCHES
# remove patches from pre-production group
zlman bgr -U $ZLMuser -P $ZLMpass -V $SLESPreProduction/$PreProdPref-$Category $PATCHES
done
done
#! /bin/bash
#******** Purpose **********
# This script will compare all production patches to all zlmmirror patches and copies the “new” patches to the test group.
# ********* History *********
# 21-11-2007 – V1.0 – Novell Consulting
# Source in all settings
. /var/zlmdata/scripts/zlmOES2settings
# Clean up
if [ -e $OES2ProductionLog ] ; then
rm $OES2ProductionLog
fi
if [ -e $OES2SourceLog ] ; then
rm $OES2SourceLog
fi
# Read bundle and folder and write to temp files
zlman bl -U $ZLMuser -P $ZLMpass -V $OES2Source |grep 'Software Bundle'|cut --fields=1 --delimiter=' ' |sort >$OES2SourceLog
zlman bgm -U $ZLMuser -P $ZLMpass -V $OES2Production |grep 'Software Bundle'|cut --fields=1 --delimiter=' ' | sort >$OES2ProductionLog
# Should not happen.
if [ -n "`comm -1 -3 $OES2SourceLog $OES2ProductionLog`" ] ; then
echo "WARNING-ZLM: The following patches should not be in the production bundle group "$OES2Production": "
echo ""
echo " `comm -1 -3 $OES2SourceLog $OES2ProductionLog` "
fi
# Defines list of "newest" patches
for NEW in `comm -3 -2 $OES2SourceLog $OES2ProductionLog` ; do
zlman bga -U $ZLMuser -P $ZLMpass -V $OES2Test $OES2Source/"$NEW"
done
# Clean up after ourselves
rm $OES2ProductionLog
rm $OES2SourceLog
#! /bin/bash
#******** Purpose **********
# This script will move all patches in the test group to the pre-production group.
# ********* History *********
# 21-11-2007 – V1.0 – Novell Consulting
# Source in all settings
. /var/zlmdata/scripts/zlmsettings
# Read bundle group and copy all patches to pre-production groups
# Defines list of "newest" patches
for PATCHES in `zlman bgm -U $ZLMuser -P $ZLMpass -V $OES2Test |grep 'Software Bundle'|cut --fields=1 --delimiter=' ' | sort` ; do
# copy patches to pre-production group
zlman bga -U $ZLMuser -P $ZLMpass -V $OES2PreProduction $PATCHES
# remove patches from test group
zlman bgr -U $ZLMuser -P $ZLMpass -V $OES2Test $PATCHES
done
#! /bin/bash
#******** Purpose **********
# This script will move all patches in the pre-production group to the production group.
# ********* History *********
# 21-11-2007 – V1.0 – Novell Consulting
# Source in all settings
. /var/zlmdata/scripts/zlmsettings
# Read bundle group and copy all patches to pre-production groups
# Defines list of current "pre-prod" patches
for PATCHES in `zlman bgm -U $ZLMuser -P $ZLMpass -V $OES2PreProduction |grep 'Software Bundle'|cut --fields=1 --delimiter=' ' | sort` ; do
# copy patches to production group
zlman bga -U $ZLMuser -P $ZLMpass -V $OES2Production $PATCHES
# remove patches from pre-production group
zlman bgr -U $ZLMuser -P $ZLMpass -V $OES2PreProduction $PATCHES
done