Using an Active Directory User Source with ZCM: Optimization and Issue Resolutions
Summary Introduction about Active Directory LDAP
Active Directory (AD) works extremely well as a "User Source" with ZENworks when optimally configured. The trick is a combination of understanding how data is stored in AD, how LDAP queries behave when issued against AD, and how to configure the AD user source within ZCM based upon how AD may be set up in a particular organization.
While this article does not deal exclusively with single AD Domain setups, these configurations often cause the most issues due to two common fallacies that are good to keep in mind when reading this article.
- If my Active Directory setup has only a single domain I do not have to worry about LDAP generating referrals, since all Domain Controllers contain all objects. (Incorrect - While all Domain Controllers DO contain all objects, referrals may still be generated.)
- If LDAP referrals are being generated, this must indicate that the Domain has been partitioned and some AD Controllers do not contain all of the objects for which LDAP is searching. (Incorrect - The "Domain" is the small possible "partition" of an AD tree. The referrals are not due to the AD controllers lacking objects but how AD is being queried by LDAP.)
Note: In layman's terms, a "Referral" happens in LDAP when an LDAP server issues a request to have a different server complete an LDAP query. As illogical as the supposed fallacies above appear to be, make sure to read the article carefully and learn why they are fallacies and how to ensure your AD LDAP source is configured to handle these intricacies of AD LDAP.
Active Directory may consist of one or more domains. Even if there is only a single domain in Active Directory, it is still a Forest which will be an important concept to keep in mind. Each "Domain Controller" (DC) in a Domain will contain all of the details about all of the objects in the domain. By default, the very first DC installed will also be a "Global Catalog Server" (GCS). The GCS not only has all of the objects for its own domain, but it will also contain partial details about ALL objects in the entire forest including those in other domains.
Connecting to a Domain Controller using the standard LDAP ports of 389(Clear Text) or 636 (SSL) will not contain any results from the Global Catalog and be restricted to Domain Information.
Connecting to a Domain Controller using the Global LDAP ports of 3268(Clear Text) or 3269 (SSL) will contain results from the entire Forest.
To Determine if a DC is also a GCS:
Planning GCS configuration:
In a single-domain forest, configure all domain controllers as global catalog servers. Because every domain controller stores the only domain directory partition in the forest, configuring each domain controller as a global catalog server does not require any additional disk space usage, CPU usage, or replication traffic. In a single-domain forest, all domain controllers act as virtual global catalog servers; that is, they can all respond to any authentication or service request.
(Note: Following Microsoft's Recommendation is not Required and I would not suggest changing your AD configuration based upon a single document. The primary object of this article is about how to configure ZCM based upon your AD setup and not how to setup AD for ZCM.)
Technical Details on Using Active Directory as an LDAP Source
There can be important differences in how a GCS accessed on ports 3268/3269 responds to an LDAP query versus a DC (or GCS) that is accessed on ports 389/636. The key difference revolves around referrals.
How referrals function in Active Directory are quite different than with eDirectory.
In eDirectory, if an LDAP Server does not hold sufficient details to resolve an LDAP Query, the LDAP Server itself will specifically query another eDirectory server that it knows holds the details and return all the details to the client that issued the LDAP Request.
A Referral from an Active Directory LDAP Server is not necessarily an indication that the server lacks any of the information necessary to complete the query. An AD LDAP Server may generate Referrals even if it is the only AD Controller in an entire Forest. In many cases, the referrals issued by an AD LDAP server may be back to themselves! As a reminder, in any single domain forest, every AD Controller contains every object. The issue is that on the default LDAP port of 636, AD controllers are not AWARE of the fact they are the only domain and they hold all of the objects.
Thus any LDAP query set to the top to the "DC Level" such: " --baseDN dc=zenral,dc=com" will result in the referrals to the following addresses: zenral.com, DomainDnsZones.zenral.com, and ForestDnsZones.zenral.com.
The following example is from an Active Directory Forest that has only a single domain and a single AD Controller.
In the example above, we can see that an LDAP Query against LDAP Port 389 to the LDAP Server on address 192.168.1.201 resulted in 3 referrals to ForestDNSZones.zenral.com, DomainDNSZones.zenral.com, and zenral.com even though they all resolve back to 192.168.1.201.
In the example above, these added referrals create additional LDAP overhead, since these LDAP referrals need to be chased even though in essence the LDAP server is requesting to ask itself. The referral overhead can be significant if these DNS addresses to different servers are over the WAN. In some cases, these addresses may Round-Robin among a large number of Domain Controllers resulting in the referral sometimes being local and sometimes across a WAN.
If the query is set to a lower location such as "--baseDN OU=NewYork,DC=zenral,DC=com" then the AD LDAP server would generate zero referrals and thus have significantly lower overhead in the query. The primary drawback of using a lower baseDN is that depending upon the AD configuration you may need to define many such as for "Seattle, Boston, Dallas, Miami, etc...". The issue with searching multiple lower-level --baseDNs to avoid referrals is that an index query of a single lower-level --baseDN is not more efficient than the same query at a higher level since the data is stored in the same index. Since multiple lower-level baseDN queries would need to be performed the overhead will increase linearly with each additional --baseDN added.
Below are two examples of configuring the User Source. The first points to the top of the AD Tree and may result in referrals, which may cause issues if care is not taken. The second shows how to define multiple lower-level OUs, but this configuration may result in additional overhead due to the need to query the same index ten times instead of one to find a particular object.
Above will have Referrals, which can add overhead and redirect LDAP queries to unexpected AD controllers. Note: Referrals with LDAPS are an extra concern because the LDAP Certificate for each LDAP server used by ZCM must be imported into ZCM. This can cause many LDAPS queries to fail if the certificates to those additional servers are not imported into ZCM. In some cases, simply adding a Domain Controller to the Domain can add that server to the referral round-robin leading to failures in the referral process. While LDAPS is strongly recommended over LDAP for security reasons, when making the switch from LDAP to LDAPS it is critical to fully understand all LDAP referrals in your setup.
The above will not generate referrals, but searching for an object will have additional overhead since many different LDAP queries will be generated against the same Domain Index. The example above could be adjusted to only reference "/zenral.com/Humans", which would be noticeably more efficient by only hitting the Domain Indexes three times instead of often looking for the same object.
Minimize the Impact of Referral Overhead
#1 - Use Global Catalog Servers and the Global Catalog LDAP Ports. This eliminates referrals. There should be no downside in a single domain setup other than convincing your AD Admins to read the Microsoft AD documentation above and configure the AD Controllers used by ZCM to be GCs. This works very well with multi-domain setups, except that not all of your servers can be GCSs unless there are a limited number of DCs in the disparate domains. (Note: In Multi-Domain Configurations, I recommend customers use standard LDAP browsers to examine objects, such as groups, that contain objects from multiple domains. Be sure to use both Catalog and Non-Catalog ports to understand the difference in the data returned on the different ports.)
#2 - Define a limited number of Lower Level OUs instead of pointing to the DC level when the use of Global Catalog Servers is not possible. Queries at the DC level could POTENTIALLY include additional domains, which is why non-Global Catalog servers send out queries to find out if they exist since they do not inherently know they are actually the only domain.
#3 - If your AD User Source IS configured to user referrals, be fully aware of how each LDAP Server defined in ZCM will resolve the referral addresses of:
- YourCompany.Com (YourCompany.com is not a literal value.)
- DomainDnsZones.YourCompany.com (YourCompany.com is not a literal value.)
- ForestDnsZones.YourCompany.com (YourCompany.com is not a literal value.)
Try to avoid sending referrals over WAN Links but if a WAN Link must be traversed be sure to pick AD controllers that traverse that must traverse the fewest links.
#4 - Use Super Secret Setting on a "Configuration Primary" so that referrals are ignored when performing "configuration" related LDAP Queries. Drawbacks include the fact it's a manual configuration on each server. This will likely NOT work well in a Multi-Domain configuration where referrals MAY be valid. If referrals exist they could also import authentication which is not covered by this setting. This setting is not documented because it may be misused by customers who do not understand referrals. I would only recommend this if there is a single domain and your AD administrators do not want to allow ZCM to use Global Catalog ports.
In the "AuthSourceConfig.xml" file on your configuration primary, one can change "IgnoreADReferrals" to True.
(Important: "IgnoreADReferrals" should only be used if all other recommendations are not possible. I have never used this setting for a customer, since there are better options but I include it for sake of completeness.)
SSL Considerations with Active Directory
Using SSL with Active Directory can be a special challenge for two reasons.
- Referrals - When creating an LDAP Connection in the ZCC for an LDAP Server, a trust for that LDAP Server's certificate is created. However, referrals can create two issues.
- The referral may occur to a server for which we may not have a trust to that server's certificate.
- The referral may occur to an LDAP server that is configured for LDAPS
- Active Directory LDAP Server Certificates are often configured to renew themselves automatically. This may create an issue where the SSL Certificate is no longer trusted by ZCM until the server's LDAP certificate is updated in the ZCC. This update may happen expectedly and during inconvenient times.
The simplest way to avoid the issue regarding server certificate trust caused by referrals or the automatic update of LDAP server certificates is to create a trust for Active Directory's CA certificate. This way if LDAP refers to an unexpected LDAPS server or an LDAPS server's certificate is automatically renewed, connections will still be allowed to the servers since the CA Certificate itself will be trusted.
ZCM can be configured to trust a CA cert in addition to the server certs by using the "user-source-trustedcert-add (usta)" ZMAN command. Manually added trusts can be seen by using the "user-source-trustedcert-list (ustl)" server command. These manually added trusts will not be visible in the ZCC itself.
1. From the Root Certificate Authority Server and run this command: "Certutil -ca.cert ca.cer" to export the CA certificate.
2. Copy the "ca.cer" to a temporary working location on your primary such as /tmp.
3. Run "zman usta" to add the trust. (Note: Do not use spaces or special characters for the Alias name to avoid potential issues in creation.)
4. Run "zman ustl" to verify the trust was created successfully.
Note: Since LDAP Referrals are often based upon simple DNS names versus actual Active Directory Objects, depending on DNS resolution those referrals can be to AD controllers that do not have LDAPS configured. This is another reason it is important to configure LDAP so that referrals are not generated or to be very aware of DNS configuration so that DNS referrals always refer to an appropriate LDAPS server.
Ensure all needed Objects are configured to be read by ZCM
When configuring the ZCM User Source to point to lower-level containers instead of the DC container, it is important to include all OUs that may contain all Users and Groups which ZCM may need to reference.
Example: If all users are in OU=Employees,DC=Company,DC=COM, and all groups are in OU=UserGroups,DC=Company,DC=COM but only OU=Employees is referenced in the user source setup, then it will not be possible to use group assignments since ZCM will never examine the UserGroups OU.
Furthermore, while it should not be a concern with ZCM 2020 or later, it is possible that on some older versions of ZENworks issues may occur if a user is a member of any group that is not in one of the OUs ZCM is configured to use. Issues may also occur if one of those groups is included for ZCM to read but contains users which are not within the scope configured for ZENworks to use.
Example - A Company may have a single domain but Offices in London and Berlin. Perhaps only one of those sites uses ZENworks, but groups in either the London or Berlin OU may contain users from the other for non-ZENworks reasons.
Again, this should NOT be an issue with ZCM 2020 or even later versions of ZCM 2017, but in some cases, earlier versions did not fully handle the LDAP errors that could occur parsing the missing information. All my recent testing indicates ZCM should handle those cases correctly now, but do keep in mind in the event one does see an anomaly and some side cases I missed in testing may still exist.