Setting Sentinel Main (Reporting) Server Bind Addresses

over 6 years ago
There are always configuration options that you find the first time you setup a new system, and then there are some that you find when you are asked for some odd business reason that may not have been as obvious on day one. One of these options for services is the ability to restrict which IP address(es) is/are bound by a service for external access. For example, which IP address, of the half-dozen on a big server, are bound for web services by Apache httpd? Which ones are bound by PostgreSQL for database use? There are a lot of reasons for opening ports, and just as many for leaving them only accessible to localhost (or not at all via TCP/IP, but rather using sockets when possible).

Today's question was how to limit ports bound for Sentinel. Sentinel is meant to be scalable via many systems to support high event rates from disparate systems around the globe, and as a result there are several components to configure. A few of those include:

Jetty web service - Used for the main Sentinel UI, the REST interface, and distributing the thick clients via Java WebStart.
PostgreSQL - The internal database primarily for configuration data, but also used for a relational datastore for those queries requiring that kind of thing
ActiveMQ - Used to accept connections from Collector Managers (CM) sending data received and parsed on the CMs.

Of the ten or so (by default) listening sockets on the default Sentinel 7.2 reporting server, some of them are listening only on the localhost ( IP address, such as MongoDB (powers some advanced features such as the Security Intelligence Dashboard). PostgreSQL is another that listens locally only since, normally, there is no reason for remote access to be provided to the internal database and preventing PostgreSQL from listening prevents misconfiguration by those who disable host firewalls or use weak passwords for the system.

Some other services can be configured, and by default listen on all addresses in order to be reachable no matter where the clients are. For example, a Collector Manager, or Sentinel Control Center (SCC) client, or user in the Web UI may come from any network to which the server is connected; certainly some networks are more likely than others (the network hosting back services only probably does not, or should not, have client connections originating there) but during the install it is possible that not all IP addresses are known. Perhaps the "listen on all addresses" option was chosen for convenience, and now somebody wants to change. Perhaps one IP was used a couple of years ago and now somebody has re-addressed the network, so a change is necessary.

Assuming a default install, the file to choose is the following (alternate base directories may apply if you used the option to relocate Sentinel elsewhere, in which case you should prepend this directory with that alternate base directory):


The output from my test system, just as an example, follows:
# Name:
# Contains values that are set as Java system properties, and are also made
# available by as shell variables.

baselining.sidb.password=asdf1vSWantnqwer5GpzWajVgFzbaasdfAHovA0asdfZ LviZgpKMasdfrLlqwer
baselining.sidb.dbpassword=asdf1vSWantnqwer5GpzWajVgFzbaasdfAHovA0asdfZ LviZgpKMasdfrLlqwer

Look for lines like activemq.ip.collectormanager, activemq.ip.server (which you probably should not change), and baselining.sidb.bindip (which you also probably should not change).

Note that at the top of this file it mentions, which is a script you can run if you are a non-root user to basically setup your user's environment (typically the 'novell' user's environment) to as it would be just before Sentinel starts. This is really useful if you want to do any testing of components with certain settings, or if you want to create some scripts that go in and behave like the (by default) 'novell' user to do things like backup components, examine files, monitor connections, etc. Ever wonder if your change to the file or file actually worked? Run as the 'novell' user, like this:
#First, become root:
sudo -i
#Next, become 'novell'
su - 'novell'
#Next, source (meaning run it in the current shell, vs kicking off a new shell via a normal script call) the script:
source /opt/novell/sentinel/bin/
#Finally, look at the current user's environment variables to see what is there now, which may not have been before:
env | less
#Just for record-keeping purposes, make a copy of the current environment, sorted, so that everything is in one place:
env | sort > ~/all-set.environment
#Not sure what was there before? Start a new shell as 'novell' and run the following command:
env | sort > ~/not-set.environment
#Now compare the two files generated in the last couple of steps to see what was there after running the script, vs. before:
vimdiff ~/not-set.environment ~/all-set.environment
#or the less-graphical way
diff ~/not-set.environment ~/all-set.environment

It turns out that this method of setting up the environment is used by the Sentinel plugin for supportconfig in order to reliably gather as much data as possible about the Sentinel setup for troubleshooting purposes:

TID# 7010586

To change PostgreSQL's listening port check out postgresql.conf, and this is even documented with a tool provided. The command to do it since the reason to do this is to allow external reporting against the internal database, thus the script:

From the postgresql.conf file itself (located under /var/opt/novell/sentinel/3rdparty/postgresql/data for the record) is the following snippet showing both the IP address and port number configuration options:
#listen_addresses = 'localhost'         # what IP address(es) to listen on;
# comma-separated list of addresses;
# defaults to 'localhost', '*' = all
# (change requires restart)
#port = 5432 # (change requires restart)

Keep in mind that if you really plan on accessing the database from outside of the Sentinel server that you also need to allow users to connect from other systems, so please use the script above. Hacking the other PostgreSQL configuration file isn't hard, but it's less-obvious what the options do, and the script is there and documented, so why not use it? For the record, here is the uncommented part of pg_hba.conf which also must be changed in order to authenticate using passwords from remote systems:

# "local" is for Unix domain socket connections only
host all all md5
local all all peer map=siemuser
# IPv4 local connections:
#host all all md5
host all all md5
# IPv6 local connecl neetions:
host all all ::1/128 md5

Changing Jetty's listener for the web interface and REST API is another story altogether, and if you want to maintain support for your system you should skip this paragraph. More on why is covered down below when it talks about packaging and configuration files. For now, the way to tell Jetty to listen on certain IP addresses rather than all bound IP addresses (showing up as or just :::8443 (IPv6)) is to manually make (significant) changes to the jetty-ssl.xml file under /etc/opt/novell/sentinel/3rdparty/jetty which is not defined as a configuration file. If you do a bit of Googling for things " multiple" you'll probably find this solution online: What this solution states is that the 'host' setting within a 'call' to SelectChannelConnector is needed to specify a specific IP address to be bound for listening, which is basically what we see in the commented-out example within jetty.xml. Since there is no 'host' specified in the jetty-ssl.xml file, the application listens on all IP addresses; so far so good. If we add a line like the following, though, we can tell the system to listen only on (in this case)
<Set name="host"></Set>

Note that this sets the system to listen on the public IP address properly, but those paying close attention to the original system should have noticed that the application talks to itself using the loopback/localhost/ IP address, which we have now excluded. The end result is that if add just the line above to the existing <Cal/> section in jetty-ssl.xml you will get a login page, and then never be able to login. Sentinel is now broken and you're left to revert your change and restart again to get things back to normal. What you can see from the StackOverflow article above, though, is that you can have multiple blocks to bind multiple IP addresses, and a quick test of that seems to fix the problem in my case (disclaimer: I've done very little testing so far, so while I feel it is working I haven't verified every component by any stretch of the imagination). To do this my total set of steps include adding a single line to one of the <Call/> blocks to tell it to listen to the desired public IP address, and then duplicate the entire <Call/> block (most of the file) and then change the 'host' line to be, at which time I restart Sentinel in the usual way. Looking at the list of bound sockets, I now see the following:
/usr/sbin/ss -pln | grep :8443
0 100 ::ffff: :::* users:(("java",7996,1043))
0 100 ::ffff: :::* users:(("java",7996,941))

Before doing this I made a backup of my jetty-ssl.xml file, which I would recommend doing anytime anything in Sentinel or another application is changed, just as a good practice: A sample command to do that follows:
cp -a /etc/opt/novell/sentinel/3rdparty/jetty/jetty-ssl.xml  /etc/opt/novell/sentinel/3rdparty/jetty/jetty-ssl.xml-`date  %s`

If this (binding specific IP addresses with Jetty) is an important option for you for some reason, try submitting an enhancement request, or perhaps open a Service Request with NetIQ. As mentioned above and discussed below, this jetty-ssl.xml file is NOT marked in a package as a configuration file, so when you patch the owning RPM (named 'novell-Sentinelwebserver') your changes will be lost, reverting the system to a still-functioning state of listening on all IP addresses, which shouldn't hurt but is counter to this exercise.

It probably makes sense at this point to briefly mention why hacking the XML/sh/etc. files directly is not preferred. After all, this is where the values, including, are in use, right? Hacking them there would seem to provide a bit less work in determining what the values are a year later when forgotten; certainly it is easier to read the value in the file directly than to become 'root', become 'novell', run a script, then compare the user's environment with a pre-script version of the environment, right? To all of these, I agree, but there is always a tradeoff in anything between abstraction and having the real configuration files right there, between sticking with some default configuration options in a large system and moving them to a certain properties file to be used by all components, between defining entire classes for every conceivable purpose and using principles of inheritance to allow some classes to be derived from others. The point is that all of these former options are there for maintenance purposes. Imagine, if you will, that you work in Sentinel engineering; your job is to provide a system that is configurable by end users and, if possible, also fairly simple to use. Because you're smart, you use a lot of reliable code from various open source projects (Jetty, PostgreSQL, MongoDB, Jasper Server) because it lets you get to market with something solid and scalable in something less than astronomical time. Unfortunately, all of these other products do their own thing regarding configuration files, even though most of them are Java-based and are very well-documented online. If your customers need to do something simple, like set a specific IP address to be used for all listening, they'll need to learn the half-dozen products that you are learning simply to properly implement options. All they really need is a single place that lets them set things that are oft-configured. As a result, you the NetIQ developer add a bit more work on your plate to setup a file for all of the obvious options, so that your customers can twiddle bits in a single place.

Let's take the scenario one step further; a year passes and you find you would like to release a new version, but requiring your existing customers to create new environments is a good way to make people very upset, so of course you want to provide an upgrade option. All of those open source products have also had some releases in the meantime, so their defaults may have changed, requiring you to somehow merge in arbitrary changes from the upstream sources as well as customers, and if you mess it up you risk losing customers' data, trust, and future business. No pressure. Thankfully, because you were smart in the previous paragraph, you're not that concerned, and neither are your customers. As an RPM-based installation, Sentinel has the ability to mark some files as being configuration files, while others are just regular files to be managed exclusively by the RPM. For example, the shells scripts that your customers may have wanted to modify directly are likely controlled by the RPM; they are not configuration files in the sense that the file is. The PostgreSQL files may be configuration files, but again init scripts, libraries, sample databases probably are not. Various xml files used to configure the many component of a system like Sentinel can all be treated as non-configuration files because all of the really interesting bits have been abstracted out into a single file (well, actually a few other files too) that are preserved by the install system naturally, taking advantage of decades of install experience built into the RPM packaging system. The only thing you as the developer had to do for all of this to work was mark, when building the RPMs, the file as a config file, which is trivial.

Of course, this is a very simple example for a product that is very large, and just one example of ways that the underlying operating system and packaging systems can be used to simplify life. I mentioned earlier that it is possible to relocate an installation of Sentinel from a location other than the root of the filesystem, for example to a dedicated /apps mountpoint. This means that all kinds of things may differ for a system which introduces risk and complexity for a vendor. The packaging system helps with this too by having a 'relocate' option to maintain where an RPM is installed. As an administrator you can determine where a package was relocated using simple RPM commands, which may come in handy for running the script above since it is hard to run what you cannot find:
rpm -ql novell-Sentineljre

The RPM command's output lets you know quickly that the base directory for this install was the default /opt/novell/sentinel, but if this was relocated we'd see something else entirely:
rpm -ql novell-Sentineljre

That's all for today.
Comment List
Related Discussions