LDAP Virtual List View (VLV) Control

Knowledge Partner
Knowledge Partner
0 0 5,122
0 Likes

Once in a while an issue comes up where application A gives some odd errors in eDirectory but those errors seem to be cosmetic as application A is still fine and works (unless you count the messages in ndstrace). What does this mean when you see a -714 from eDirectory while tracing LDAP that is talking about some "iterator"? What does an LDAP error 51 mean when everything still seems to return properly? Today I am going to try to share some information on Virtual List Views (VLV) by explaining what they are, how you can use them, and a few cases where they do not work with eDirectory.



First it may be useful to cover a little information about LDAP "controls" which are ways to request additional functionality from an LDAP server. eDirectory does not have an LDAP backend but is an X.500 directory which then has a Lightweight DAP (LDAP) interface because that is the standard directory interface of choice for the industry. Because eDirectory is not LDAP on its backend, it currently does not fully support the additional controls that are made for LDAP like Server-Side Sort (SS) and Virtual List Views, though those controls work in most cases. As they are not supported they do not show up on the RootDSE when queried by an LDAP application but if the application will still try to use them in most cases all will be well and the return will be as expected.



With some LDAP controls testing is possible with the standard ldapsearch command that comes with most computers. SSS is this way where there is a switch you add to the command allowing you to specify the attribute(s) on which to sort and how to sort based on them (ascending, descending). With VLV I have yet to find a graceful way to implement this particular control with ldapsearch which is why most of the remainder of this article will be using Perl to do the same task. With the perl-ldap module from the Comprehensive Perl Archive Network (CPAN) we can implement various controls easily and with examples directly from CPAN as shown here: http://search.cpan.org/~gbarr/perl-ldap-0.39/lib/Net/LDAP/Control/VLV.pm



So let's dive in. Going back to what VLV is, it is a way to get data returned in a certain order or in certain subsets (windows or views). Normally when using ldapsearch you specify the base context for the search, the scope of the search, and then apply a filter that limits the results based on attribute values of the desired objects in the directory. For example I could run ldapsearch to only return objects from o=novell,dc=org, then set the scope to a 'subtree' search (to recursively get all objects within that context), and finally limit my responses based on the filter 'cn=ab' which would only return objects with a 'cn' attribute having a value of 'ab'. While this limits the number of results returned it does not necessarily mean I will only get one result. Removing the filter and base (context) parameters it is likely I will receive many results indeed. VLV lets us put a limit on those objects returned so the client application is not overwhelmed with data, much of which it may not need or even be able to handle at this time. If there are one million objects beneath o=novell,dc=org then I may not want all of those at once but would instead prefer to have those in batches. I may also want, eventually, to go to a certain batch where I know an entry exists. I may want even to jump to the end of the list and only get those values (think of a webpage that shows you everybody listed on a site, but lets you page through all of the results without showing one million entries at a time; it is likely that the backend is returning subsets to increase application performance).



There are a few things implied with the scenarios listed above. First, an order to the results is implied. By default LDAP does NOT order its results; doing so is often a waste of the server's time since there may not be a need for ordering. With that said ordering on the server side can be done if the server implements SSS and that control is requested. VLV implies that SSS is available and used so whenever a control is used to implement VLV the SSS control must also be used. If this were not the case the first request for the first twenty objects and the subsequent request for the next twenty objects could have overlapping values or completely random values, etc. In eDirectory's case SSS is another control that works in most cases but also is not officially supported currently.



Now that we have these defined we can start looking at the Perl used to make a connection and implement these controls. Perl is used as it is a flexible, powerful, cross-platform and open language that should work for anybody reading this; for many of you, in fact, Perl is installed on your computer by default as the majority of operating systems we use do this out of the box. To start out we need to include some Perl modules that are on my system in the 'perl-ldap' RPM package.



use Net::LDAP;
use Net::LDAP::Control::VLV;
use Net::LDAP::Control::Sort;
use Net::LDAP::Constant qw( LDAP_CONTROL_VLVRESPONSE );
use Net::LDAP::Constant qw(LDAP_CONTROL_SORTRESULT);



The lines above include the standard LDAP libraries, the VLV and Sort controls, and also define two constants which are initialized to the OID values for VLV and SSS, respectively. These constants come in handy later on as we'll see. With the first step done let's actually skip the code that establishes a connection and binds via LDAP. Some of the variables are not defined in this article though they are in the script; I believe they make sense by their names and context in any case:



#Connect to the server and bind as the user specified.
my($ldap) = Net::LDAP->new($bindServer) or die "Can't bind to ldap: $!\n";

my($mesg) = $ldap->bind(
dn => $bindDN,
password => $bindPassword
);



That's all probably fairly straight forward, especially with the comments, but essentially we create a new $ldap object which makes the connection the LDAP server (my eDirectory 8.8.5 system on SLES 10 SP2 currently as $bindServer is the IP address of that same system) followed by the creation of a $mesg variable which is the result of my bind attempt passing in my application user (cn=admin,dc=user,dc=system) and that user's password as parameters to the bind method. Normally if I was not interested in VLV I could define arguments for my query, run the query, and print output and indeed that may be a good place to start just so we have a basic starting script:




#Query Arguments
my(@args) = ( base => 'o=novell,dc=org',
scope => 'subtree',
attrs => ['dn'],
filter => '(objectClass=inetOrgPerson)'
);

#Run the search
$mesg = $ldap->search( @args );

#Dump the results.
foreach $entry ($mesg->entries) { $entry->dump; }




In this case I am requesting all of the objects under 'o=novell,dc=org' which have an objectClass of inetOrgPerson (a User object in eDirectory terms, by default). Currently I'm not doing anything with the results except printing them to the screen. As a sidenote I am explicitly requesting just the 'dn' of the objects which may not be normal but it makes the output a bit shorter. Just to point it out as well the 'args' array (@args) is an associative array (or hash) with keys set to values that are either scalars (regular variables) or nested arrays themselves ('attrs' is an array, for example). The main 'args' array is then passed to the 'search' function which carries out the search and returns the results to $mesg from where they are dumped to the screen via the 'dump' method for each entry. The result of this query could be the following, though longer due to the thousands of objects within the containers:





------------------------------------------------------------------------
dn:cn=testintruder00,ou=intruder,o=novell,dc=org

------------------------------------------------------------------------
dn:cn=testLongPW00,ou=finance,o=novell,dc=org



The 'dump' method formats the data in a way that is fairly simple for humans to parse and while not as verbose and useful as it could be the data are there to determine if the code is working in most cases. Now in order to implement VLV we need to do a couple things. First, we must define the SSS and VLV controls in Perl. Once completed we must pass those controls in as arguments to the 'search' method via @args. The latter change is simple; compare it with the previous definition of @args to see the differences:





#Query Arguments
my(@args) = ( base => 'ou=minion,o=novell,dc=org',
scope => 'subtree',
attrs => ['dn'],
filter => '(objectClass=inetOrgPerson)',
control => [ $sort, $vlv ],
);



As you can see I now have a 'control' array within @args and this array has two values, specifically $sort and $vlv. As you have not seen these defined yet we'll now go create those and explain what the options within them mean and which options are allowed. Their definition must precede that of @args or they will not be valid in time for their inclusion as values for the 'control' index so the following code and explanation should come before the code that was just shown. Let's start with the example from the VLV control page from CPAN modified somewhat:




# Get the first 20 entries
my($vlv) = Net::LDAP::Control::VLV->new(
before => 0, # No entries from before target entry
after => 1, # 0 entries after target entry
content => 0, # List size unknown
offset => 1, # Target entry is the first
);

my($sort) = Net::LDAP::Control::Sort->new( order => 'cn' );




The code above defines our two variables. The SSS component ('sort') is straight-forward and basically creates a new object telling the LDAP server to sort returned results by the 'cn' attribute. The 'vlv' section is a bit more complex and to understand it we need to understand what each option does. First we have the 'before' parameter which tells how many entries before the 'target' we are requesting. Remember this VLV technology is about getting pages of data from a directory so this lets us request a given target by an index and then request entries before that as well, all of them ordered by 'cn' in this particular example. The next setting is 'after' which is fairly straight forward as well. The third entry is 'content' which specifies how many entries to return once the target is found. The CPAN docs state this should be zero to begin with and then is set later on by server responses from the initial query. The last option is 'offset' which specifies the offset from the target for which options will be returned. So now let's run a few queries. First I'll use the default query but only want '2' in my 'after' parameter and I'll run that a couple times in a row to see what we get back:




ab@mybox0:~/Desktop> ./ldapvlvctrl.pl edirserver.novell.com cn=admin,dc=user,dc=system
Please enter a password:
------------------------------------------------------------------------
dn:cn=test00,ou=minion,o=novell,dc=org

NEW QUERY
------------------------------------------------------------------------
dn:cn=test00,ou=minion,o=novell,dc=org

NEW QUERY
------------------------------------------------------------------------
dn:cn=test00,ou=minion,o=novell,dc=org




So in my current setup I run the query several times but never ask for subsequent pages of data. See the following:





#Run the search
$mesg = $ldap->search( @args );

# Get VLV response control
($ctrlResp) = $mesg->control( LDAP_CONTROL_VLVRESPONSE ) or die('Unable to set control response: ' . $!);
$vlv->response( $ctrlResp );

#Dump the results.
foreach $entry ($mesg->entries) { $entry->dump; }

print 'NEW QUERY' . "\n";

#Query again.
$mesg = $ldap->search( @args );

# Get VLV response control
($ctrlResp) = $mesg->control( LDAP_CONTROL_VLVRESPONSE ) or die('Unable to set control response: ' . $!);
$vlv->response( $ctrlResp );

#Dump the results.
foreach $entry ($mesg->entries) { $entry->dump; }

print 'NEW QUERY' . "\n";

#Query again.
$mesg = $ldap->search( @args );

# Get VLV response control
($ctrlResp) = $mesg->control( LDAP_CONTROL_VLVRESPONSE ) or die('Unable to set control response: ' . $!);
$vlv->response( $ctrlResp );

#Dump the results.
foreach $entry ($mesg->entries) { $entry->dump; }




Unless I actually tell the system to keep on going with its queries (to move to the next set of values) it gets the same responses over and over. This is nice because, as an application programmer, if I desired to get the same results over and over reliably I should be able to. I should also be able request the same start and get anything that has changed since my last query. In the code above it is important to note that the server's response with regard to VLV is captured each time and written to the $resp variable. This is happening because I have selected to only get the target, nothing before or after it, and no additional content. So now what if I wanted to go ahead and get the next two each time along with my target? Let's modify the code a little. Every time (after the first time) that we currently have '$mesg = $ldap->search( @args );' precede that statement with '$vlv->scroll_page(1);' which basically tells the server to go to the next "page" of responses. For example the following:





$mesg = $ldap->search( @args );




should become:





$vlv->scroll_page(1);
$mesg = $ldap->search( @args );





Updating $vlv with the code above and then running the same queries over and over I get new output, and as you can see we are now advancing one object at a time because we are scrolling to the next page repeatedly.





------------------------------------------------------------------------
dn:cn=test00,ou=minion,o=novell,dc=org

NEW QUERY
------------------------------------------------------------------------
dn:cn=test01,ou=minion,o=novell,dc=org

NEW QUERY
------------------------------------------------------------------------
dn:cn=test02,ou=minion,o=novell,dc=org

NEW QUERY
------------------------------------------------------------------------
dn:cn=test03,ou=minion,o=novell,dc=org

NEW QUERY
------------------------------------------------------------------------
dn:cn=test04,ou=minion,o=novell,dc=org





Now how, exactly, did eDirectory know where it was from one query to the next? That is where the line updating '$ctrlResp' comes into play. This variable is storing the output from the LDAP query that is specific to the VLV control. For example:





$mesg = $ldap->search( @args );

# Get VLV response control
($ctrlResp) = $mesg->control( LDAP_CONTROL_VLVRESPONSE ) or die('Unable to set control response: ' . $!);
$vlv->response( $ctrlResp );





In the code above the $mesg variables holds the output from the LDAP search method and is then called to set $ctrlResp with the part of that message that pertains to the control defined by the constant LDAP_CONTROL_VLVRESPONSE. As a result when $mesg is given output specific to that control it will return it and set $ctrlResp with it when requested. This response (which includes offsets, pages, etc.) is then passed back to the $vlv object which has a method to take the response directly. Subsequent calls that use $vlv, then, will be ready to get the NEXT batch of data instead of the same one over and over, assuming we requested it. Each time the system runs an LDAP query it is passing the VLV information updated from the previous query (assuming there was a previous query) which lets the server respond with subsequent pages of data each time. In our case the pages are just single objects but obviously that can easily be expanded upon.





Besides our example showing scrolling entire pages it is also possible to scroll by individual entries or a set of entries:





ab@mybox0:~/Desktop> ../ifolder/aburgemeister/scripts/ldapvlvctrl.pl edirserver.novell.com cn=admin,dc=user,dc=system
Please enter a password:
------------------------------------------------------------------------
dn:cn=test00,ou=minion,o=novell,dc=org

------------------------------------------------------------------------
dn:cn=test01,ou=minion,o=novell,dc=org

NEW QUERY
------------------------------------------------------------------------
dn:cn=test00,ou=minion,o=novell,dc=org

------------------------------------------------------------------------
dn:cn=test01,ou=minion,o=novell,dc=org

------------------------------------------------------------------------
dn:cn=test02,ou=minion,o=novell,dc=org

NEW QUERY
------------------------------------------------------------------------
dn:cn=test01,ou=minion,o=novell,dc=org

------------------------------------------------------------------------
dn:cn=test02,ou=minion,o=novell,dc=org

------------------------------------------------------------------------
dn:cn=test03,ou=minion,o=novell,dc=org




This case is a bit interesting because we actually get duplicate values currently as we are returning a target and what is before/after it and then only incrementing by one. If we were to increment by three at a time (since we are getting three at a time) it would be a bit less redundant.



So with a little Perl and an LDAP server demonstrating the usefulness and possible business cases of VLV is fairly straight forward. Tinkering to work out all of the technology's options and making them work perfectly may take a little time but at least now the functionality can be verified. The attached script should be a starting place from which to continue testing what is needed and what works in your environment.

The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.