eDir Transaction ID Finder

1 Likes

Avoiding eDirectory Meltdowns With the 'eDirectory Transaction ID Finder



This program finds Transaction IDs embedded in eDirectory 8.7.3.x / 8.8x roll-forward log files (as found in the nds.rfl directory).



If repairs are not run regularly on your eDirectory servers, it is important to know the range of transaction IDs that are currently being issued on each server, so that you can take the necessary preventative action when required.



eDirectory is based on the FLAIM database engine, which has a maximum transaction value of FFFFE000 (hex) and as such it is possible to 'run out' of transaction IDs if a repair is not run (which will reduce the last transaction ID counter).



If a server does run out of transaction IDs, only Novell Technical Services has the ability to correct the problem.



See: eDirectory "Transaction ID" Warning / error -618 database stops working and won't open.
http://www.novell.com/support/viewContent.do?externalId=7002658&sliceId=1



By using this program and comparing results, it is also possible to approximate how long it will be before a particular server runs out of transaction IDs, allowing you to plan accordingly if you have a 24x7 eDirectory server.



Running The Program



The program uses a copy of the roll-forward log from any 8.7.3.x or 8.8x database.



The roll-forward log files are normally located in:




  • NetWare - SYS:\_netware\nds.rfl

  • Linux - /var/opt/novell/eDirectory/data/nds.rfl

  • Windows - C:\Novell\NDS\DIBFiles\nds.rfl



In most cases, the roll-forward log will be called 00000001.LOG and should have a fairly recent timestamp.



In some cases, there may be no log files, the log file may be only 512 bytes in size (no transactions) or there may be multiple log files.



Generally, only one log file is in use at any time by one eDirectory instance.



There are multiple ways to identify the current roll-forward log file. If there is only one log file in the nds.rfl directory then this is the file 'in use'. If there are multiple files, identifying the file with the latest time stamp should indicate which file is being used. The dsbk command can also be used:




  • On NetWare: dsbk getconfig

    Depending on the eDir version, the results are displayed on the logger screen or on a separate 'dsbk' console screen.


  • On Linux: dsbk getconfig (if previously configured - if no /etc/dsbk.conf, will not work)

    The output from the command is sent to the default log file (ndsd.log).


  • On Windows: From control panel, run the 'Novell eDirectory services' applet. Highlight the dsbk.dlm line, enter the word 'getconfig' in the 'Startup Parameters' box, and then click on start. Nothing will appear to happen, but the config info should have been written to the output log file.


    The output is written to C:\Novell\NDS\backup.out



Via iManager - eDirectory Maintenance - Backup Configuration options.



If there are no log files, you may need to enable roll-forward logging before you can make use of this program (see eDirectory documentation).



NOTE: In most cases roll-forward log files appear to be created and updated even if roll forward logging is set to OFF.


NOTE: Although it can be OK to delete log files which are no longer used, do not do this unless you need the space. Be sure you do not delete or modify the 'current' roll-forward log file as it could have severe consequences.


Program Requirements:



The program has been tested on SLED10 and Windows XP, but should run on any OS that supports Perl 5.5.8 or later.



The program was tested with Perl 5.5.8 on Linux and Activeperl 5.10.1 on MS Windows.



Installation:



Make a new directory and place the roll.pl and allroll.pl programs in the directory. Acquire a copy of the the roll-forward log file from the server to be checked and place in the directory with the perl programs.



Execute the program using: perl roll.pl



Enter the name of the log file to be processed (case sensitive on Linux/Unix).




NOTES:

The program allroll.pl is included as a diagnostic utility which will extract ALL transaction IDs from a roll-forward log file regardless of age. The roll.pl program only references what it has identified as recent transactions and will not include transactions considered as 'old' data. Log files are actually re-used from the start, so the ordering of transaction IDs can appear to be inconsistent when viewing transactions from the whole file. The output file for the allroll.pl program is called 'All_eDir_Transactions.txt'.



When running dsrepair/ndsrepair in order to reduce the current transaction ID counter, please make sure you ONLY set the 2 options to 'yes' as indicated in TID#7002658. If any other combination of yes/no switches is used, the transaction ID will probably not be reduced.




Example Program Run / Output:



Name of roll-forward log file [Default=00000001.log]: 00000001.LOG

Processing!


Last Transaction ID identified: Hex [6392b4e5] : HexMax [FFFFe000]
Dec [1670558949] : DecMax [4294959104]

Number of Transactions left : Dec [2624400155] 61.10% remaining

Labels:

Collateral
Comment List
Anonymous
Parents
  • Ok, so Novell's stance seems to be to only run dsrepair if you're having a problem. Yet, with this transaction ID limit, you would think that periodic dsrepairs would be recommended by Novell (?).
  • I think the problem with making any recommendation for a product like eDirectory and the use of ndsrepair/dsrepair is that there is really no general case.

    For example, in a previous comment I noted that a multi million object DIB, was only at the 200 million mark, but no logins were happening.

    I had a test system with 15K objects that I have deleted and recreated 10K of those objects several times, and write on average 15 attrs to each object on each test pass. So I did a LOT of writes, and I am only at 3 million or so.

    Others have reported servers at the 3+ billion mark. Others in the real world have actually gone over the 4 billion, and needed NTS to fix it, prior to 885.

    Additionally, what a single transaction is, seems to depend on how it was generated. I.e. Atomic operation, like a single LDIF op, a user create, etc seem to count as one, whereas modifies out of IDM to multiple attributes, still needs more testing.

    Furthermore, we are still unclear on how logins affect the count? 1? 3? 4 per login? Dunno. Need more testing.

    So for a small DIB, say 10K users, and normal login patterns, it seems quite likely that in the lifetime of the DIB set, you will never hit 4 billion.

    I.e. You will more likely replace the server/disk farm, and recreate the DIB, before you hit 4 biillion.

    For an astonishingly large tree (millions of objects) with low login counts but lots of creates/deletes , it looks likely that a decade or more will be needed to hit 4 billion.

    Therefore it is pretty safe to say, this is important to know about. It is important to think about. But it seems like it will be pretty darn rare! But it could really happen, if you are in one of those specail cases.

    I suspect that the easiest way to trigger this, would be to have an LDAP auth tree, with a few thousand users, and hundreds of thousands, if not millions of login events a day. That will add up very fast. Might still need a year or three to get into trouble, but it could happen.

    I guess my point is, for probably 95% of eDirectory users, this is a total non issue. For that last 5% well, that is why you pay people to support software.



Comment
  • I think the problem with making any recommendation for a product like eDirectory and the use of ndsrepair/dsrepair is that there is really no general case.

    For example, in a previous comment I noted that a multi million object DIB, was only at the 200 million mark, but no logins were happening.

    I had a test system with 15K objects that I have deleted and recreated 10K of those objects several times, and write on average 15 attrs to each object on each test pass. So I did a LOT of writes, and I am only at 3 million or so.

    Others have reported servers at the 3+ billion mark. Others in the real world have actually gone over the 4 billion, and needed NTS to fix it, prior to 885.

    Additionally, what a single transaction is, seems to depend on how it was generated. I.e. Atomic operation, like a single LDIF op, a user create, etc seem to count as one, whereas modifies out of IDM to multiple attributes, still needs more testing.

    Furthermore, we are still unclear on how logins affect the count? 1? 3? 4 per login? Dunno. Need more testing.

    So for a small DIB, say 10K users, and normal login patterns, it seems quite likely that in the lifetime of the DIB set, you will never hit 4 billion.

    I.e. You will more likely replace the server/disk farm, and recreate the DIB, before you hit 4 biillion.

    For an astonishingly large tree (millions of objects) with low login counts but lots of creates/deletes , it looks likely that a decade or more will be needed to hit 4 billion.

    Therefore it is pretty safe to say, this is important to know about. It is important to think about. But it seems like it will be pretty darn rare! But it could really happen, if you are in one of those specail cases.

    I suspect that the easiest way to trigger this, would be to have an LDAP auth tree, with a few thousand users, and hundreds of thousands, if not millions of login events a day. That will add up very fast. Might still need a year or three to get into trouble, but it could happen.

    I guess my point is, for probably 95% of eDirectory users, this is a total non issue. For that last 5% well, that is why you pay people to support software.



Children
No Data
Related Discussions
Recommended