Developing and Testing Anti-virus Solutions for OES-Linux
3. Understanding OES
3.1 OES-Linux vs. OES-NetWare
3.2 SLES Base
3.3 OES Services
3.3.1 NetWare Core Protocol (NCP)
3.3.2 Novell Storage Services (NSS)
3.3.3 eDirectory and LUM
3.3.4 Novell Cluster Services (NCS)
4. Adding OES Support
4.1 Adding NCP Support
4.2 Adding NSS Support
4.3 Adding eDirectory Support
4.4 Adding NCS Support
5. Verifying OES Support
5.1 Testing Philosophy
5.2 No Access Impedance Test
5.3 Virus Detection Test
5.4 Metadata Preservation Test
Appendix A – NSS Kernel Modules
Appendix B – NCS Kernel Modules
Appendix C – Downloadable Test Files
Novell's Open Enterprise Server (OES) products extend the capabilities of the SUSE Linux Enterprise Server (SLES) platform by adding advanced storage, networking, security, and management services. Enterprise features such as eDirectory, Novell Storage Services (NSS), and Novell Cluster Services (NCS) provide end-users and administrators with rich and robust services for thousands of concurrent users. By moving these key NetWare services to a Linux server platform, Open Enterprise Server gives administrators all the strength and breadth of Linux while preserving the NetWare client experience that administrators and end-users know and trust.
As Novell migrates its significant base of NetWare customers to Linux, the market for third-party OES security products is growing. Having anti-virus product support for OES is particularly important, since OES systems often function as file servers to large numbers of vulnerable Windows client systems.
This document provides:
- an overview of OES-Linux – what it is and how it differs from SLES
- a discussion of the OES-Linux components that are pertinent to anti-virus product developers
- suggested approaches for providing OES-Linux anti-virus support
- suggested approaches and scripts for testing OES-Linux anti-virus support
This document does not provide:
- Full documentation on OES-Linux (for this, see http://www.novell.com/documentation/oes2 )
- Full anti-virus product test suites (this is considered to be the responsibility of the anti-virus developer)
- Discussion of anti-virus support for virtual machines (including NetWare) on OES-Linux
This document is intended for:
- Developers of OES-Linux anti-virus solutions
- End-users of OES-Linux anti-virus solutions
Novell's OES product actually includes two separate sub-products: OES-Linux and OES-NetWare. OES-Linux is an augmented SLES system: it is made up of the underlying SLES operating system and a set of services which operate on top of that system. I.e., OES = SLES + OES Services. This document does not cover OES-NetWare.
The SLES base in OES-Linux is unmodified: it consists of all the same packages and versions provided in Novell's standalone SLES product. This situation is illustrated by the fact that OES customers who receive automatic updates from Novell are registered for 2 update sources: the standard SLES update repository and an OES update repository.
OES includes a SLES 9 base and OES 2 includes SLES 10.
The OES 2 installation process further underlines the “SLES + OES Services” nature of OES: To install OES 2, a user installs the stock SLES 10 product and includes the OES services as an “Add-on Product”.
OES-Linux is differentiated from SLES by the “OES Services”: applications/middleware that provide the same functionality as a native NetWare 6.5 server. Some of the OES services have been ported directly from NetWare, while others have been significantly re-architected/modified in order to integrate them cleanly with SLES. Regardless of how the services have been implemented, in most cases, Novell has been able to preserve the services' user and administrator interfaces. The goal is to ensure that administrators can replace their NetWare servers with OES-Linux systems without impacting the client experience.
The OES services are numerous, but this document only covers those most pertinent to anti-virus product developers: the Novell Communication Protocol (NCP), Novell Storage Services (NSS), eDirectory and Linux User Management (LUM), and Novell Cluster Services (NCS). Depending on an anti-virus product's design, it may or may not need to be modified to function correctly with each of these services. Even if modifications are minimal, Novell suggests that developers implement full testing to ensure that their products operate as expected on OES-Linux.
The NetWare Core Protocol (NCP) is an application, presentation and session layer protocol used to communicate between NetWare servers and non-NetWare (generally Windows) clients; it is the communication method used by many of the traditional NetWare services. (See http://www.novell.com/documentation/oes2/novell-client-access.html#novell-client-access .) While NCP was originally designed to function over the IPX/SPX protocol, NCP now uses TCP/IP. NCP client support is available (from the OS vendor or from Novell) for most desktop operating systems.
The OES-Linux NCP server implementation is not “wired-in” to any particular file system. It is possible to configure the NCP server to provide client access to files on standard Linux file systems as well as NSS volumes.
Impact on anti-virus products:
File access via NCP should not affect an anti-virus product's on-demand scanning, nor should it affect any on-access scanning implemented in the Linux kernel via syscall or VFS-layer hooks. On-access scanning using other (non-kernel) methods may or may not provide protection for NCP file access. As a simplistic example, an anti-virus product which implements its on-access protection by intercepting SMB functions will provide no protection for NCP file access.
As the file system for the NetWare operating system, NSS provides enterprise-level file services including (but not limited to) a trustee access control model, quotas, rich file attributes, event file lists, and a file salvage subsystem. (See http://www.novell.com/documentation/oes2/storage.html#nss .) By providing NSS on Linux, Novell gives administrators the ability to use a multi-purpose Linux server platform without relinquishing the file system features and client interface they require.
One of the best ways for Linux developers to gain a basic understanding of NSS on Linux is to contrast and compare NSS with other standard Linux file systems (Reiser, ext2, ext3, etc.). A complete comparison table is provided in the NSS documentation, however, this document highlights some specific points.
Similarities between NSS on Linux and standard Linux file systems:
- NSS volumes are mounted on a directory (default is /media/nss/<volname>).
- The local root user has full access to NSS files and directories.
- NSS is implemented using the Linux VFS kernel functions.
- By default, NSS uses the UNIX (case-sensitive) namespace.
Differences between NSS and standard Linux file systems:
- NSS volumes are created using iManager or the nssmu curses-based utility (instead of the Linux partition management utilities).
- Aside from the local root user, only eDirectory users with appropriate rights can access NSS files and directories. Local non-root Linux users cannot access NSS files and directories (even though the files' POSIX permissions might indicate otherwise.)
- NSS files and directories have additional metadata associated with them. On Linux, this metadata is stored in the extended attributes associated with the pertinent inode.
- There are interfaces to NSS which do NOT “go through” the kernel VFS layer. For instance, Novell provides “zAPIs” for performing file functions. On OES-Linux, the zAPIs access a zlss kernel module which communicates directly with the NSS kernel modules (without going through VFS). Apple File Protocol (AFP) access of NSS files uses the zAPIs.
- In the Linux kernel, all access of NSS files is done as root; any access determinations are made by user-space NSS processes before passing the requests (as root) to the kernel.
In general, it is important for Linux developers to understand that NSS is “not just another Linux file system.” The differences between the NSS trustee-based model and the POSIX permission model are significant and structural in nature (e.g., NSS supports a directory-inheritance rights model whereas Linux does not.) It is also important for Linux developers to understand that the primary NSS file access method is via NCP requests from remote client systems (usually Windows).
Impact on anti-virus products:
- If the anti-virus product runs as root user, it should have no trouble scanning NSS files. If the anti-virus product runs as a non-root user, it will be unable to scan NSS files unless the user has been added to eDirectory and given appropriate rights to the NSS files (see the “eDirectory and LUM” section below).
- NSS files have a “delete inhibit” attribute. If this attribute is set on a file, the expectation is that the file should not be able to be deleted (or moved) until the attribute is cleared. This may be of concern to anti-virus products needing to quarantine files.
- If an anti-virus product moves an NSS file to a non-NSS file system location (e.g., during a quarantine procedure), the NSS file's metadata will be lost unless the anti-virus product takes specific steps to preserve it.
- Anti-virus products that provide on-access scanning via kernel syscall or VFS-layer hooks will be bypassed by AFP access of NSS files.
- NSS includes kernel modules, so the possibility exists for conflicts between the NSS kernel modules and any anti-virus on-access scanning modules. Conflicts may be most likely (or most evident) during product startup.
Novell eDirectory (formerly called Novell Directory Services, or “NDS”) is a directory service software product for centrally managing access to network resources. (See http://www.novell.com/documentation/oes2/directory.html#directory .) Novell eDirectory is available for a number of different platforms.
Because almost all OES services (including NSS) require eDirectory, it is reasonable to assume that every OES-Linux system has eDirectory installed and enabled. The OES-Linux system may be functioning as the primary eDirectory server or as a replica server in a larger eDirectory tree. The eDirectory tree is generally used to manage users and groups as well as other network services.
While eDirectory is integral to NetWare's user authentication model, Linux uses files (in /etc) as its default user and group “databases”. Thus, on OES-Linux, we must determine how to use eDirectory in addition to (or in place of) the standard Linux file-based user and group model. Fortunately, this is a well-understood problem and has been addressed via the concept of “Pluggable Authentication Modules” (PAM). The PAM framework provides administrators with the ability to choose a system's authentication model. Developers create PAM modules to perform or redirect user/group authentication services, then administrators install these PAM modules and configure the system's authentication process to use the modules instead of the /etc user and group files.
OES-Linux eDirectory includes PAM modules, and the eDirectory installation configures OES-Linux such that user authentication is directed through the PAM modules to eDirectory instead of the /etc/passwd and /etc/shadow files. This redirection (as reflected in the the OES-Linux /etc/nsswitch.conf file) is the first step in establishing eDirectory users as local OES-Linux users.
Given that eDirectory trees often contain large numbers of users, it would be problematic (at least from a security standpoint) if having an OES-Linux system configured to use eDirectory implied that all eDirectory users would have local rights to the OES-Linux system. To address this issue, eDirectory includes “Linux User Management” (LUM) technology. LUM gives administrators the ability to control, from eDirectory, the rights that each user has on the local OES-Linux system.
By default, when a user is added to eDirectory (e.g., via iManager), the user is NOT enabled as a local user on OES-Linux. I.e., default eDirectory users have no rights on OES-Linux. To give an existing eDirectory user local OES-Linux rights, an administrator can “LUM-enable” the user via iManager or the various OES-Linux “nam” (man nam) command-line utilities.
Impact on anti-virus products:
As noted in the “NSS” section above, if an anti-virus product runs as a non-root user, it will be unable to access NSS files unless the user has been LUM-enabled and given appropriate rights to the NSS files.
On an OES-Linux system, it is highly likely that non-root users and groups are stored in eDirectory rather than in the /etc user and group files. Therefore, any Linux anti-virus product (or other software product) that performs user authentication may not function as expected in the OES-Linux eDirectory environment.
OES-Linux includes Novell Cluster Services (NCS), a server clustering system that ensures high availability and manageability of storage resources, applications, and services. (See http://www.novell.com/documentation/oes2/clus_admin_lx/index.html?page=/documentation/oes2/clus_admin_lx/data/h4r4bw6c.html#h4r4bw6c .) NCS requires eDirectory.
In terms of hardware, an NCS cluster is generally comprised of a number of servers attached to a shared disk subsystem containing cluster-enabled storage resources. An NCS cluster-enabled storage resource may be an NSS pool or an EVMS volume containing a traditional Linux filesystem.
NCS includes user-space and kernel-space functionality.
Impact on anti-virus products:
NCS operates “above” the filesystem level, so it should not impact anti-virus on-access or on-demand scanning functionality. The possibility does exist for startup conflicts between the NCS kernel modules and any anti-virus on-access scanning modules.
By moving from NetWare to a Linux base, Novell has instantly given its customers access to a broad range of third-party applications. Anti-virus products are no exception: if an anti-virus product already supports SLES, it will likely function well on OES-Linux. However, there is a difference between “running on OES” and “fully supporting OES”. Fully supporting OES implies understanding the OES services, augmenting the anti-virus product as necessary to support the OES services, testing the anti-virus product on the OES services, and officially claiming support for OES as a platform. The previous section explained the OES-Linux functionality that might cause problems for a Linux anti-virus product; this section explains how to modify an anti-virus product to address these issues.
Most Linux anti-virus products do not need to be modified in order to support NCP access of files on standard Linux filesystems. Information on supporting NCP access of NSS files is provided in the “Adding NSS Support” section below.
However, because NCP communication is not commonly used in Linux-only networks, it is important for anti-virus developers to explicitly test in an NCP (OES-Linux/Windows Client) environment. Anti-virus developers may wish to differentiate between their support for NCP access of files on standard Linux filesystems vs. NCP access of NSS files.
The following details should be considered when adding anti-virus support for NSS on Linux:
- Because OES-Linux does not require NSS, the anti-virus product should be able to function on OES-Linux regardless of whether or when NSS is installed. The anti-virus product's startup scripts should not contain any dependencies on or interactions with NSS: it is preferable to embed any NSS checks and/or functionality in the anti-virus product's actual scanning routines.
- The anti-virus product should ensure that its user has appropriate rights to scan NSS files. See further discussion on this subject in the “Adding eDirectory Support” section below.
- The anti-virus product should preserve metadata for any NSS files it moves to quarantine. As noted above, NSS file metadata is stored in the Linux extended attributes. Therefore, if the anti-virus software already supports and preserves Linux extended attribute data, there should be no need to add further features to preserve NSS metadata. If the anti-virus software does not support Linux extended attributes, NSS metadata may be preserved using the xattr (man xattr) functions. The following code gets and sets an NSS file's metadata:
* get and set NetWare metadata
int main(int argc, char** argv)
const char* path = argv;
ssize_t bufSize, result;
/* get size of NetWare metadata */
bufSize = getxattr(path, "netware.metadata", metadata, 0))
printf(“Size of netware.metadata is %d bytes.\n”, bufSize);
/* get NetWare metadata */
if (metadata = malloc(bufSize))
result = getxattr(path, "netware.metadata", metadata, bufSize);
/* set NetWare metadata */
result = setxattr(path, "netware.metadata", metadata, bufSize, 0);
- The anti-virus software should ensure that its startup does not conflict with the NSS startup. If possible, the anti-virus products should load before NSS. If the anti-virus software cannot cleanly complete a startup before NSS starts, or if the anti-virus software needs to load after NSS for some other reason, the anti-virus software should delay its startup until the NSS admin volume is functioning. The startup state of the NSS admin volume can be determined by parsing the output of the “/etc/init.d/novell-nss” script.
- When deleting or moving an NSS file, the anti-virus software should consider the state of the file's “delete-inhibit” attribute before taking action. In order to preserve the function of the “delete-inhibit” attribute, Novell encourages the anti-virus software to prompt the user for authorization before deleting or moving a “delete-inhibit” file.
If the anti-virus software runs as root, and does not interact with the system's user database, there is no need to add specific support for eDirectory.
If the anti-virus software runs as a non-root user, this user should be 1) added to eDirectory, 2) LUM-enabled, 3) given security equal to the eDirectory admin user, and 4) given appropriate rights to the NSS files.
The anti-virus software can create and LUM-enable the user (and group) via the OES-Linux command-line “namgroupadd” and “namuseradd” utilities:
namgroupadd -a <eDirectory admin FDN> -x <group context> <group name>
namuseradd -a <eDirectory admin FDN> -x <user context> -g <group FDN> <user name>
Providing the anti-virus user with eDirectory admin equivalence should only be done with the full knowledge of the system administrator. For this reason, the anti-virus software should not set the security equivalence automatically, but instead include installation instructions directing the end-user/administrator to set the equivalence manually via iManager.
The OES “rights” command can be used to give an eDirectory user appropriate rights to NSS files:
rights -f <path-to-NSS-volume> -r s trustee <eDirectory FDN of user>
If the anti-virus product interacts with the system user database (such as by authenticating users), then the product should use PAM APIs to ensure that it will function with eDirectory (and/or other directories). E.g., assume that an anti-virus product includes the ability to assign various administration levels to non-root users. To successfully authenticate these non-root users against eDirectory, the software should use PAM APIs (instead of the getpwent()/getspent() functions). More information on using PAM APIs is available in a number of on-line documents (see the Linux Journal article, “Securing Applications on Linux with PAM, by S. Fernandes and KLM Reddy.)
Most anti-virus products will not need to be modified in order to support NCS. However, because NCS does include a number of kernel modules (see Appendix A), it is important for anti-virus developers to fully test their products' startup and operation in an NCS environment. If an anti-virus product's startup conflicts with NCS, the product should be modified to delay its startup until after the output of the “/etc/init.d/novell-ncs status” command indicates that the NCS startup has completed.
Novell understands and expects that anti-virus product vendors perform comprehensive testing of their products on the various platforms they support. Anti-virus product vendors are ultimately responsible for the quality of their products. However, in order to simplify the testing of anti-virus products on OES-Linux, Novell has created several basic tests that vendors (and customers) can run and/or extend for their own purposes. These tests are not sufficient to ensure that a Linux anti-virus product fully supports OES-Linux, but they do provide quick “litmus” checks to indicate common problems.
The “test-access” tests ensure that, in the absence of infection, an anti-virus product does not interfere with normal NSS file access. These tests are applicable to any OES anti-virus product.
The “test-access” tests write and modify NSS files, as different types of users, from both the local OES-Linux system and from a Windows client. Depending on user access, the writes and modifications are expected to succeed or fail.
The “test-access” test files and READMEs (test instructions) are provided in the test-access-linux.tgz and test-access-windows.zip downloadable files (see Appendix C).
Testing an anti-virus product's ability to detect infection requires introducing real viruses into the test environment. While this scenario may be reasonable in a managed laboratory, casual testers and end users cannot be expected to use real viruses in order to determine whether or not an anti-virus product is functioning correctly. It is for this reason that the “eicar” test file was created. The eicar test file is not a virus, but a non-malicious executable designed to test the integrity of anti-virus software (see http://en.wikipedia.org/wiki/Eicar .)
The following information details how to use eicar to ensure that an anti-virus product detects infection in NSS files.
Test Prerequisites: OES-Linux system with:
- NSS volume
- Anti-virus software installed and enabled
Windows client system with:
- Drive mapped to OES-Linux NSS volume
- No anti-virus software installed
- Ensure that the anti-virus product under test supports detection of eicar.
- Download the eicar test files from http://www.eicar.org/anti_virus_test_file.htm to a local directory on the Windows system.
- Attempt to copy each eicar test file to the drive mapped to the OES-Linux NSS volume.
- If the anti-virus product includes on-access scanning, ensure that the anti-virus product's on-access scanning detects the attempted write.
- If the anti-virus product does not include on-access scanning, perform a manual scan of the NSS volume and ensure that the scan detects the eicar test file.
The “test-metadata-linux” test ensures that an anti-virus product preserves an NSS file's metadata when moving the file to a quarantine location. This test is applicable to any anti-virus product that is able to recognize and quarantine the eicar test file.
The “test-metadata-linux” test: 1) waits for the user to download the eicar.com file to the NSS volume, 2) records the file's metadata, 3) waits while the anti-virus product scans and quarantines the file, 4) waits while the user restores the file from quarantine, then 5) compares the restored file's metadata to the original file's metadata.
The “test-metadata-linux” test files and instructions (README) are provided in the downloadable file test-metadata-linux.tgz (see Appendix C).
Because OES-Linux is comprised of OES services running on a SLES base, anti-virus products that support SLES will function well on OES-Linux. But as noted throughout this document, SLES anti-virus developers who wish to fully support OES-Linux do need to take some additional steps: they must ensure that their products support the OES services (as well as the SLES base), they must fully test their products in OES environments, and they must state official support for their products on the OES platform.
While addressing the above issues requires some investment on the part of the anti-virus vendor, the payoff is significant. By supporting OES, anti-virus product vendors immediately expand their potential customer base to include existing OES-Linux users and the large number of NetWare users who will be moving to OES-Linux in the future.