thought in blue

Can you please check the transitive sync and see if there are transitive vectors in the tree that no longer have references or perhaps do not have a timestamp update?

Then also perform the usual nds health check. If you have servers in the tree that do not have a replica, rebuild the backlinks. (Keyword xk3 in the old Novell world), what does the replica check and the agent check in iMonitor say.

Finally, look for collision objects, maybe there is something there too. Do all servers see each other on 524 and what does slp say?

George

Thank you Georg, I'll do what I can.

Something from my experience:

In the past, I have also worked in environments with several hundred servers and replicas. It is actually quite normal for -625 or -626 to be present (keyword referrals).

One lesson I have learned from large environments is that there are design specifications for certain numbers of servers and replicas in an NDS. Then NDS tuning is a very important thing. (keyword cache behavior). Bandwidth management is also very important. Even if the transitive sync is very helpful to get all objects through a tree in sync, it also reaches its limits. It may be that a sync for a partion is actually stuck and you have to react.

It is therefore always important to monitor the sync on every master server using of an replica with an ndstrace. Here it can be helpful to trigger a dstrace with the help of scripts and then to search for errors via logfile pharser and if any errors are detected in the pharser to report them directly to the helpdesk Please also keep an eye on the schema sync and on the schema itself.

A simple nmap is also helpful for ndsanalysis. Write scripts here that regularly scan all servers from top to bottom with nmap and the most important ports top down, also detect errors here via pharsen and report them to the helpdesk, basically errors in the NDS communication mesh can also be found in this way.

Another important thing is a security strategy for the NDS itself, for the certificates and the tree CA and how to deal with the  trustees. Finally, we need to keep an eye on security, but that is certainly a separate issue


One more thing at the end, if a server is shut down for maintenance purposes, there is a notification in the NDS during a gracefull shutdown as far as I remember. Normally the limber process will look up the replica pointer table and compare it so that the server is reported up again in the NDS of the corresponding partition, in the field I have seen that a -625 or -625 is reported in the communication anyway. Please check if this is cosmetic or real, if real there are also sync problem

My post just came out of my head. Normally, when I have such large NDSs, I write an action plan with every step for an entire diagnosis. Without such any instruction, a technician is poking around in the fog without.

Perhaps I have just carried owls to Athens. My job was often to play fireman when there was a major breakdown. The motto is, where others run out, we run in.

So warm up your fingers slowly, put down a pad and pencil and write down every step you have taken so that you can check where the error may have occurred if something goes wrong. Good luck and a good hand in such a big NDS

George

This sounds like a lot of work, because of the size of the tree, number of servers and partitions. I know that this tree is near/has reached the limits of eDirectory regarding some of its aspects. eDir servers are monitored constantly, obituaries, open ports, slp, number of servers in slp, certificates, CA, transaction ID, memory consumption, nam. CA has been replaced last year, certificates have been recreated on all servers. There are 5 busy replica servers, where the transaction ID has to be reset every year. Unfortunately there is not enough staff and knowledge here to carry out such investigations and right now there is no fire that needs to be put out. I am thankful for your help, I'll see what I can implement of it.

I have seen nds test trees with more than 5 million objects live in labs of a university because a backup product that was supposed to back up the NDS got a hiccup with the number of objects. So just silence.