So, in our previous blog, The Importance of Updating ArcSight, we established why you should upgrade your ArcSight systems to the latest models, but how would you go about that? Bear with me, this is going to be a long one! This part of our blog series will go into a bit more detail on how we prepared to upgrade our customers’ systems so everything would run smoothly when we went live. This isn’t going to be a step-by-step instructional blog about how to run the upgrade but will give a general overview of our process, if you want to know more about how to upgrade or want a hand sorting everything out, you can drop us an email on firstname.lastname@example.org and we’d be happy to help! Unfortunately, due to the nature of upgrades and the ArcSight platform in particular, most of the upgrade steps need to be implemented through the command line interface. Though this is not a huge problem it can mean some of the more involved steps can be a bit fiddley when compared to completing installations through dedicated wizards which is what most systems have implemented now.
This is why step one of our upgrade preparations was to set up our own virtual production environment on RHEL 7.7 and install legacy versions of ArcSight ESM, ArcMC and ArcSight SmartConnectors in line with the customer environment that we would be upgrading. This allowed us to simulate the upgrade in an environment that would be safe from live data loss or system outages if something were to go awry during the upgrade process. It also gave us the opportunity to feel out any quirks that may cause issues down the line, such as upgrading to the specific version of the OS needed to support each product at each upgrade hop, rather than just jumping straight to the latest OS available.
Now that we had a simulated environment set up and ready for upgrade it was time to download the software required for each upgrade hop for ESM, ArcMC and the SmartConnectors from the Micro Focus Software & Licences portal and transfer them to our simulated ESM and ArcMC servers respectively. We also created backups of the opt/arcsight/ directories into opt/backup directories at this stage so we had somewhere to revert to if needed. These backups created the first problem to overcome. Depending on how the systems are utilised, these backups could be extremely large and creating copies of them could completely fill up storage space which would be required to upgrade the software, but they are essential! Though this was not an issue for our simulation environment it could be for any live upgrades, which is why we requested sight of the customers’ system storage and one of the reasons we advocate running regular backups of your systems, regardless of whether you have any changes planned.
With backups complete we began in earnest! The ArcSight manager was stopped, we ran system validation to check for errors, and restarted the manager before accessing the root user and un-taring the first ESM download file. From there it was a quick change of user to non-root before installing the binary, reverting to root, and restarting the services. We then repeated the previous steps for the next ESM hop, with the addition of upgrading the OS from RHEL 7.7 to 7.8 to 7.9 and checking for the dejavu, fontconfig, ncurses-compat-libs, libnsl, libaio, and numactl libraries. Without these libraries we found that ESM would not run reports so a vital final step which is easily overlooked!
Next up in the simulation upgrade was ArcMC! The first step for this one was checking the logind.conf file and downloading the ArcMC binaries for each hop as root, before swapping to a non-root user to run the binary and complete the setup, then repeating all the previous steps for the final hop. When completing these upgrades, it is important to remember that the latest CentOS is not supported by ArcMC and the highest supported is CentOS 8.2. From ArcMC we then bulk upgraded the SmartConnectors which was the most straightforward part of the process as there is a user-friendly process within ArcMC to manage connected devices!
The OS version upgrades were the most significant issue we found when completing these simulated upgrades. Not all OS versions are supported by the ArcSight products which can lead to problems with compatibility and the support of the services. There have been rumours that Micro Focus are removing the rigidity around supported OS versions as they block users from having the latest release security patches but for the moment this is something you would need to keep in mind when performing upgrades.
With all upgrades complete on our virtual environment we were happy with the upgrade process, so we set a date with our customer to run the full upgrade on their systems. The final post in this blog series will explore how the upgrade went and the issues we faced in the live environment!
About the Author
Melissa Hartley-Brighton is a Junior Security Consultant at Titan Labs Ltd. This is a three part series of blogs on ArcSight. The first blog discussed The Importance of Updating ArcSight. The second part of her blog series, Preparing for an ArcSight Upgrade, goes into detail on how customers can prepare their systems to make everything run smoothly. The third and final blog post, The Final Step in Upgrading ArcSight, describes how to upgrade your instance of ArcSight.