Highlighted
Not applicable
458 views

NDS Imonitor : script to display max ring delta as a alert.

Hello ,

We have some unknown objects getting created daily on our all the edir replicas , some of them sometime gives us a problem like 609 error.

So on daily basis we check the max ring delta in I monitor and if it is high we then try to remove the unknown objects post which max ring delta becomes stable.

I wanted a script which can display max ring delta value as a alert every time it crosses the threshold of 30 min.

Can someone please help.

Best Regards,
Ravi
Labels (1)
0 Likes
4 Replies
Knowledge Partner
Knowledge Partner

Re: NDS Imonitor : script to display max ring delta as a alert.

On 03/13/2019 01:46 AM, CAPVCC SUPPORT wrote:
>
> We have some unknown objects getting created daily on our all the edir
> replicas , some of them sometime gives us a problem like 609 error.


How are they getting created? It seems like a better use of time to
figure this out, and fix it, so the problem does not show up anymore.

> So on daily basis we check the max ring delta in I monitor and if it is
> high we then try to remove the unknown objects post which max ring delta
> becomes stable.


Happening daily implies something is really wrong in your tree; unknown
objects happen because of some odd corruption typically, like the removal
of a mandatory attribute (normally not possible).

> I wanted a script which can display max ring delta value as a alert
> every time it crosses the threshold of 30 min.


How about searching for the unknown objects, if they are the cause of the
sync delta, and act on that directly, rather than waiting for a secondary
symptom of the max ring delta? If that's an option, you could just use
ldapsearch to look for the objects, and maybe even delete them
automatically if you are feeling brave:


ldapsearch -x -LLL -H ldaps://server.goes.here:636 -b '' -s sub
objectClass=Unknown dn



--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
Knowledge Partner
Knowledge Partner

Re: NDS Imonitor : script to display max ring delta as a ale

CAPVCC_SUPPORT;2496685 wrote:
Hello ,

We have some unknown objects getting created daily on our all the edir replicas , some of them sometime gives us a problem like 609 error.

So on daily basis we check the max ring delta in I monitor and if it is high we then try to remove the unknown objects post which max ring delta becomes stable.

I wanted a script which can display max ring delta value as a alert every time it crosses the threshold of 30 min.

Can someone please help.

Best Regards,
Ravi


Something important is missing from this story. Unknown objects replicate just fine, having one or more won't cause max ring delta to do anything interesting.

Are you getting collision objects? Unknown objects? Neither should be happening, but they're very different problems.

I'd focus on finding the problem, then fix the cause of that.
0 Likes
jwilleke Trusted Contributor.
Trusted Contributor.

Re: NDS Imonitor : script to display max ring delta as a ale

I have worked with a client for several years and this issue still happens, perhaps not daily, but often.
We have had open SR with no resolutions.

What appears to happen is repaid deletion and renames or creation (same named entry) on more than one server happening within the synchronization time for the replica.

The ndstrace files shows:
3370448640 SKLK: [2019/03/19 4:02:52.158] DCRequest failed, missing mandatory (-609).


From iMonitor we will find two entries with some of the same attributes (like cn or UID) one is okay and the other is unknown.


And sure we all know this cannot happen but it does.
-jim
0 Likes
Knowledge Partner
Knowledge Partner

Re: NDS Imonitor : script to display max ring delta as a alert.

On 03/19/2019 05:26 AM, jwilleke wrote:
>
> I have worked with a client for several years and this issue still
> happens, perhaps not daily, but often.
> We have had open SR with no resolutions.


I think you should start a thread on it, and we should go after it.

> What appears to happen is repaid deletion and renames or creation (same
> named entry) on more than one server happening within the
> synchronization time for the replica.


Just a few thoughts that I'm sure you've already had.

1. Time is synchronized I"m sure, but is it? I do not think that should
matter a ton, but it's good to have verified. For others who may not
know, "in sync" (to avoid an error) means within two (2) seconds of
eachother, preferably within a fraction of a second of world clocks often
synchronized via NTP, but all that really matters is consistency among the
eDirectory boxes.

2. Have you done manual verification of schema among replica holders?
I've never seen (not that I've seen everything) eDirectory replicas
perform a create or modification which resulted in a missing mandatory,
but the message below and the symptoms makes it sound like a create may be
done on a replica that doesn't have the attribute as a mandatory, and then
when replicating to other replica-holders the sync fails of course.

3. Is it possible any old replicas exist where they have been forgotten?
Is a disaster recovery (DR) box out there somewhere with a copy of a real
system, able to cause unhealthy traffic among the noes without being
easily seen because it is actually using some other server object?
Sometimes these can be seen by looking at the Network Address attribute on
the NCP Server object over time, or the Replica attributes on the
Partition root record. Incorrect cleanup of a server has caused this for
me in the past.

4. Do you have a load balancer in the mix? The intermittent nature of
this kind of thing makes me wonder if pointing to a bad replica might
happen when the load balancer or DNS round robin setup causes connections
to one of many boxes every once in a while.

> The ndstrace files shows:
> 3370448640 SKLK: [2019/03/19 4:02:52.158] DCRequest failed, missing
> mandatory (-609).


If this shows as coming from a particular server, then comparing that
server's schema with this server's schema might be valuable. iMonitor has
a schema comparison function, but LDAP is usually easy for me (and knowing
your LDAP experience Jim, probably you too).

> From iMonitor we will find two entries with some of the same attributes
> (like cn or UID) one is okay and the other is unknown.


Are these collision renames, or #_# objects? I'm guessing no, as those
are not necessarily Unknown or lacking mandatory attributes, but it might
be useful to know. For others, collision renames are objects with the
same DN but different creation timestamps (eDirectory has second
granularity). This has happened in the past when a client tried a create,
thought it did not work, and then tried again on another replica. When
they are both created on different replicas all is well until replication
converges the differences and the duplicate DNs are noticed, so one of
them is renamed to #_# (replicaNumber_eventNumber) and then both objects
have unique DNs and will replicate properly, but one of them should be
cleaned up, and ultimately they are a sign of a client behaving poorly, so
that root cause should be fixed.

I've also seen odd things when backup DIBs are restored (e.g. the DR
server situation mentioned above), but that would probably be a one-time
thing not an ongoing issue.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.