joe_fortier Absent Member.
Absent Member.
1264 views

Error -659 troubleshooting

We've seen errors while using iManager to update attribute:

(Error -659) Time synchronization services has detected a problem with an operation attempting to modify replica information

My edir-phu is weak.
Initial googling (and the nature of the error) pointed to NTP issues.
I've cleaned up a few minor issues, but even from the beginning
ndsrepair -T
showed only a 1 second divergence (and "Yes" for time is in sync column). Now all servers report 0. The problem persists.

Some details
1) Not all imanager instances in the replica ring exhibit this problem
2) On a related note, LDAP updates seem to be going fine
3) We have 9 ring members (I know, too many)
4) ndsrepair -E output looks good (as near as I can tell) with all partitions reporting being up to date

I've run ndsrepair -R against all (or most all) of the replica members.

I suspect the next step is to run ndstrace on one of the servers that seems to have issues and attempt an attribute update.

But I'll be rapidly approaching the limits of my knowledge, and would love some more expert advice 🙂

Thanks
Labels (1)
0 Likes
18 Replies
joe_fortier Absent Member.
Absent Member.

Re: Error -659 troubleshooting

I've run
ndsrepair -C -Ad -A
Against all the members report "Total Errors: 0"
But one reports a series of obituaries in the EXTERNAL REFERENCES CHECK
Almost all are flagged OK_TO_PURGE
With one flagged PURGEABLE
All concern one account and are timestamped in the past
0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: Error -659 troubleshooting

Obituaries should never cause a -659, though time sync problems may lead
to stuck obituaries in some odd case. Still, the problem reported is -659
which is pretty rare since most servers are pretty good at maintaining
time now.

You may want to look at the object in eDirectory, specifically via
iMonitor, to see if the value timestamps on attributes you are modifying
are perhaps really weird, e.g. far into the future. For example if you
are trying to set a Full Name value if 'new full name' and that attribute
already has values that are in the future then maybe that would lead to a
-659, but that would probably be most-indicative of a problem in the past,
maybe the distant past.

Ideally we need to see which change to which attribute of which object is
causing the -659 since that lets us drill down, and you already have the
steps for that it would seem (ndstrace). Being able to duplicate, or not,
with certain versions of iManager is really odd if everything is equal
(meaning they talk to the same eDirectory servers on the backend
regardless of thee iManager server chosen) so it seems likely that each
iManager box finds a different eDirectory box for operations, so we are
back to eDirectory troubleshooting. If you can ensure (via a lot of
force) that iManager talks to one eDirectory box only, and then test that
way, and do the same with another, differently-performing, instance of
iManager, then that would be interesting if results differed. The tricky
thing is that while iManager will take an IP address (or DNS address) in
the Tree field, it will still walk the tree when it feels like it must,
for example if you put in a box that has no replicas, or does not have
replicas of the whole tree, or whatever.


--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
joe_fortier Absent Member.
Absent Member.

Re: Error -659 troubleshooting

Thanks 🙂
I've been using a test user to replicate. Searching iMonitor on one of the effected servers, I see a Modification timestamp a (little over a) year in the future (which seems a problem).
"Purge time" looks to be current (although that seems most likely not useful).
Looking a the attribute I've been testing I see a modification time of 08/14/99 (which I assume is 1999).
(it's cool seeing just how much is visible in iMonitor 🙂
0 Likes
joe_fortier Absent Member.
Absent Member.

Re: Error -659 troubleshooting

I'm guessing I need to repair timestamps and declare a new epoch.
I've picked a stack of entries and see the same issue (modification over a year in the future).
0 Likes
joe_fortier Absent Member.
Absent Member.

Re: Error -659 troubleshooting

FWIW, some more info about ndsrepair -R efforts
The summary shows (on one sample/representative replica)
NOTICE: 2259 more illegal timestamps found in current partition

There are numerous timestamp errors of the form

ERROR: Illegal timestamps were found in this replica.
You may need to run 'Repair Timestamps'
Value: 5bbf65dd, ID: 00008090, DN: OU=Students.O=Augsburg.T=AUGSBURG
Time stamp: October 11, 2018 03:01:49 PM
; rep # = 0004; event = f6ed

The summary shows
total errors = 33

This is fairly consistent, which suggests that the error's are not being resolved.
0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: Error -659 troubleshooting

On 10/03/2017 11:26 AM, joe fortier wrote:
>
> FWIW, some more info about ndsrepair -R efforts
> The summary shows (on one sample/representative replica)
> NOTICE: 2259 more illegal timestamps found in current partition
>
> There are numerous timestamp errors of the form
>
> ERROR: Illegal timestamps were found in this replica.
> You may need to run 'Repair Timestamps'
> Value: 5bbf65dd, ID: 00008090, DN: OU=Students.O=Augsburg.T=AUGSBURG
> Time stamp: October 11, 2018 03:01:49 PM
> ; rep # = 0004; event = f6ed
>
> The summary shows
> total errors = 33
>
> This is fairly consistent, which suggests that the error's are not being
> resolved.


Sure; the system will not change timestamps until the values change, and
that's a problem when the changes will not come until the future,
particularly when the "future" may be months/years out. You can wait, or
you can delete/recreate the objects with problems (can be difficult), or
you can declare a partition epoch which is a big deal, and should usually
only be done when you are sure you want to, usually with Support guiding
you. It is also good to be sure that the root cause is handled so this
does not happen again long before you look into declaring epochs.

Declaring an epoch with a lot of replicas is a bigger deal than with only
a few. You can try to repair timestamps, but I am not convinced that will
help; I do not ever remember using that a lot, so either I used it very
little, or it is new (unlikely).

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
joe_fortier Absent Member.
Absent Member.

Re: Error -659 troubleshooting

How can I just repair timestamps?
Everything I've seen so far suggests that I'll need to do the epoch declaration (I'd be happy to just modify record timestamps as needed).

Some related questions:
1) Is there a way to query for objects with bad timestamps?
I suppose ndsrepair -R logs will sort of do that (with a window).

2) What are the consequences of removal and recreation?
In particular I'd expect there's a GUID change.
Is there a recommended way to do this?
0 Likes
joe_fortier Absent Member.
Absent Member.

Re: Error -659 troubleshooting

I've got at least part of that, iMonitor advanced options allows re-timestamping

From https://www.novell.com/support/kb/doc.php?id=7004722

Using iMonitor with Advanced Mode enabled (click on NDS iMonitor in the upper left hand corner to enable advanced mode). On the server with the object that HAS all the attributes, or most of them, find the object and under Advanced Options, select Timestamp Entry. This will timestamp the object and all attributes and send them out. You will then have to repeat the operation for the object on other replicas if they have attributes this server did not have.

This works.

is there a way to do this directly with ndsrepair?
0 Likes
joe_fortier Absent Member.
Absent Member.

Re: Error -659 troubleshooting

Arrghhh... it only works on that replica
0 Likes
joe_fortier Absent Member.
Absent Member.

Re: Error -659 troubleshooting

It looks like it might be working if I then go back to advanced options and choose the sendobj option.
I also found this interesting tidbit, basically using wget (or curl) against iMonitor to automate the process.
It indicates that I'd need the EID of the object. It's not clear I can easily get this.
https://groups.google.com/forum/#!topic/novell.support.edirectory.unix/2x07YR_Sujs
0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: Error -659 troubleshooting

The EID is very easy to get, thankfully, particularly if you are scripting
at the command line interface (CLI). Using LDAP (ldapsearch) you can get
it as the 'localEntryID' property, and that is a decimal value I think
(iMonitor may want the hexadecimal version, but do not quote me on that).

Hopefully doing the iMonitor/curl/timestamp stuff will get you there, and
automating/scripting it should make that fairly painless compared to the
alternative.


/usr/bin/ldapsearch -x -LLL -h ldap://localhost:389 -D
cn=admin,dc=sa,dc=system -W -b 'cn=target,ou=object,o=here' localentryid


Change out your admin DN and the target object to be retrieved above as
applicable and then the rest will probably work for you.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
joe_fortier Absent Member.
Absent Member.

Re: Error -659 troubleshooting

Cool! Apparently I need to explicitly request localentryid. But once I do, it comes 🙂
Is there a list of other attributes like this that are only visible when explicitly requested?
On a related note, is there a way (attribute) to request timestamp?
Thanks for all your help
0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: Error -659 troubleshooting

On 10/04/2017 12:34 PM, joe fortier wrote:
>
> Cool! Apparently I need to explicitly request localentryid. But once I
> do, it comes 🙂
> Is there a list of other attributes like this that are only visible when
> explicitly requested?
> On a related note, is there a way (attribute) to request timestamp?
> Thanks for all your help


loaclEntryID, createTimestamp, modifyStamp, etc. are properties, and you
can request them all with a '+' like you get all attributes with a '*', or
you can request them specifically with their name.

Keep in mind that those timestamps mentioned above are object, not value,
and there is no standard way to get value timestamps via LDAP. You could
probably combine curl and iMonitor to get something there, but it would be
a lot of brute force.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
Knowledge Partner
Knowledge Partner

Re: Error -659 troubleshooting

joe fortier wrote:

> Cool! Apparently I need to explicitly request localentryid. But once I
> do, it comes 🙂
> Is there a list of other attributes like this that are only visible when
> explicitly requested?


Operational attributes are not returned by default. You need to request them
explicitly or use the "+" wildcard.
Other options are "*" to request all user attributes and "1.1" to request no
attributes at all.

--
http://www.is4it.de/en/solution/identity-access-management/

(If you find this post helpful, please click on the star below.)
______________________________________________
https://www.is4it.de/identity-access-management
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.