Anonymous_User Absent Member.
Absent Member.
197 views

Extending schema stopped or broke LDAP access


eDir 8.8 SP8, SLES 11 SP2

I made a schema extension yesterday - I defined a new attribute and then
added that attribute to an existing class definition. Immediately after
making that change, we lost all ability to make LDAP queries to any
replica in the system. We didn't have time to trouble shoot the problem
in detail - this happened on a live production system, and we needed to
restore operations ASAP. I applied another schema change to remove the
new attribute from the class definition. That restored operations.

We've been making schema changes in this way several times a year for
over a decade. This was the first time we've run into a problem like
this. We still need to make this change, but now we're concerned about
system stability in case the problem happens again. We'd like to
understand what possibly could have gone wrong here. This exact same
schema change had already been made in a different data center with no
problems.

Any ideas?

Thanks,

Steve Lewis


--
sedentaryz
------------------------------------------------------------------------
sedentaryz's Profile: https://forums.netiq.com/member.php?userid=10365
View this thread: https://forums.netiq.com/showthread.php?t=54196

Labels (1)
0 Likes
8 Replies
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: Extending schema stopped or broke LDAP access

It'd help if you provided the LDIF so we could see the change. For
example, if you create a new class called 'ldapServer' that conflicts with
something like 'LDAP Server' (when converted to LDAP-compatible syntax)
then perhaps that is related. Providing your LDIF means we can test it.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Extending schema stopped or broke LDAP access


The LDIF file is below. It defines the attribute and then adds that
attribute to the existing class 'nxBillOrg'.

We've looked at the SLES messages log files and various eDir log files,
but there were no entries from the time frame where we made this change.
To get things working again, I modified the LDIF file: I removed the
"add: attributetypes" block and then switched the contents of the
"delete: objectClasses" and "add: objectClasses" blocks. After applying
that change, the system started working again.


Code:
--------------------
version: 1

dn: cn=schema
changetype: modify
add: attributetypes
attributeTypes: ( 2.16.840.1.113719.2.139.12.175
NAME 'ndPrivateHSMs'
SYNTAX 1.3.6.1.4.1.1466.115.121.1.15{64512}
X-NDS_NOT_SCHED_SYNC_IMMEDIATE '1')
-
delete: objectClasses
objectClasses: ( 2.16.840.1.113719.2.139.11.3 NAME 'ndBillOrg' AUXILIARY MAY (
ndService $ ndReseller $ ndMaxUsers $ ndMembers $ ndCreator $ ndCreationDate
Time $ ndBillingCode $ ndDisabled $ ndCoBrandPartner $ ndDisabledDate $ ndTot
alStorage $ ndTotalEncryptedStorage $ ndEchoing $ ndOptions $ ndGUID $ ndDocA
ttrDefs $ ndLDSAddresses $ ndLDSCredentials $ ndEmailDomains $ ndMaxExternalU
sers $ ndDefaultListView $ ndStopWords $ ndMaxND2InterfaceUsers $ ndND2Interf
aceUsersGroup $ ndExternalSearchLink $ ndExternalSearchLinkText $ ndBESServer
Address $ ndBESListenPort $ ndSingleSignonHashSecret $ ndCabinetDelegateGroup
$ ndAffiliation $ ndOptions2 $ ndDocRetentionPolicies $ ndDocRetentionPolici
esID $ ndDownloadThreshold $ ndFedIdentUrl $ ndFedIdentExceptionGroup $ ndFed
IdentSigningKey $ ndLastDocRetentionDate $ ndSalesForceIntegrationInfo $ ndAu
thRequirements $ ndLookupUploadDelegateGroup $ ndLitigationHolds $ ndMobileAp
pAccess $ ndSyncNewDeviceNotificationAddress $ ndSyncAddresses $ ndAdvSearchF
ormat $ ndPrivateStorageLocations $ ndSearchDefaultBillOrg ) X-NDS_NOT_CONTAI
NER '1' )
-
add: objectClasses
objectClasses: ( 2.16.840.1.113719.2.139.11.3 NAME 'ndBillOrg' AUXILIARY MAY (
ndService $ ndReseller $ ndMaxUsers $ ndMembers $ ndCreator $ ndCreationDate
Time $ ndBillingCode $ ndDisabled $ ndCoBrandPartner $ ndDisabledDate $ ndTot
alStorage $ ndTotalEncryptedStorage $ ndEchoing $ ndOptions $ ndGUID $ ndDocA
ttrDefs $ ndLDSAddresses $ ndLDSCredentials $ ndEmailDomains $ ndMaxExternalU
sers $ ndDefaultListView $ ndStopWords $ ndMaxND2InterfaceUsers $ ndND2Interf
aceUsersGroup $ ndExternalSearchLink $ ndExternalSearchLinkText $ ndBESServer
Address $ ndBESListenPort $ ndSingleSignonHashSecret $ ndCabinetDelegateGroup
$ ndAffiliation $ ndOptions2 $ ndDocRetentionPolicies $ ndDocRetentionPolici
esID $ ndDownloadThreshold $ ndFedIdentUrl $ ndFedIdentExceptionGroup $ ndFed
IdentSigningKey $ ndLastDocRetentionDate $ ndSalesForceIntegrationInfo $ ndAu
thRequirements $ ndLookupUploadDelegateGroup $ ndLitigationHolds $ ndMobileAp
pAccess $ ndSyncNewDeviceNotificationAddress $ ndSyncAddresses $ ndAdvSearchF
ormat $ ndPrivateStorageLocations $ ndSearchDefaultBillOrg $ ndPrivateHSMs ) X-NDS_NOT_CONTAI
NER '1' )

--------------------


--
sedentaryz
------------------------------------------------------------------------
sedentaryz's Profile: https://forums.netiq.com/member.php?userid=10365
View this thread: https://forums.netiq.com/showthread.php?t=54196

0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: Extending schema stopped or broke LDAP access

I was not able to duplicate your symptom using a very-modified LDIF (I
lack the definitions for most of those attributes) but I tried with the
following and did not have any issues with eDirectory 8.8 SP8 Patch 5:


dn: cn=schema
changetype: modify
add: attributetypes
attributeTypes: ( 2.16.840.1.113719.2.139.12.175 NAME 'ndPrivateHSMs'
SYNTAX 1.3.6.1.4.1.1466.115.121.1.15{64512} X-NDS_NOT_SCHED_SYNC_IMMEDIATE
'1')
-
add: objectClasses
objectClasses: ( 2.16.840.1.113719.2.139.11.3 NAME 'ndBillOrg' AUXILIARY MAY (
ndPrivateHSMs ) X-NDS_NOT_CONTAI
NER '1' )



--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Extending schema stopped or broke LDAP access


I appreciate both of you taking time to look at this. This is the first
time in many, many years of making schema extensions to our eDirectory
implementation that we've ever run into any problem. We're scheduling
some maintenance time this weekend on late Saturday to attempt this
change again. Because this will be a scheduled maintenance, we'll have a
little more flexibility and time to diagnose the problem if it happens
again.

We have two main concerns if we see a similar error again when extending
the schema.
1) If the problem happens again, how can we diagnose the problem so that
we can find the cause and make the appropriate fix?
2) When this happened last week we were able to clear the issue by
reverting the change - I removed the new attribute from the class
definition. We're concerned that if we have an error again that this
rollback step may not work, leaving us in a broken state. We need to do
all that we can in preparation to be able to recover from any problem.

What actions should we take during this next week to prepare for another
attempt at the schema extension? We follow the documented guidelines for
periodic health checks on the tree, and we'll repeat those. Are there
other specific actions that we can take to check the health of the
schema on all replicas?

When extending the schema, does it matter which replica is used for the
change (master / non-master)?

What preparations can we make to be able to recover our system if the
worse case occurs - the schema extension breaks the system again and
we're unable to rollback the change? I need to be able to propose a
recovery plan that can restore the system to a working state.

Thanks again for your assistance!


--
sedentaryz
------------------------------------------------------------------------
sedentaryz's Profile: https://forums.netiq.com/member.php?userid=10365
View this thread: https://forums.netiq.com/showthread.php?t=54196

0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: Extending schema stopped or broke LDAP access

On 08/31/2015 02:24 PM, sedentaryz wrote:
>
> I appreciate both of you taking time to look at this. This is the first
> time in many, many years of making schema extensions to our eDirectory
> implementation that we've ever run into any problem. We're scheduling
> some maintenance time this weekend on late Saturday to attempt this
> change again. Because this will be a scheduled maintenance, we'll have a
> little more flexibility and time to diagnose the problem if it happens
> again.


That helps. Edward's note about using +LDAP (and I'd add +INIT as well)
for troubleshooting a problem loading the nldap module is valid for if you
get to that point.

> We have two main concerns if we see a similar error again when extending
> the schema.
> 1) If the problem happens again, how can we diagnose the problem so that
> we can find the cause and make the appropriate fix?


ndstrace with +LDAP +INIT at the least. When you try to start LDAP ('load
nldap' from within ndstrace) see what shows up.

> 2) When this happened last week we were able to clear the issue by
> reverting the change - I removed the new attribute from the class
> definition. We're concerned that if we have an error again that this
> rollback step may not work, leaving us in a broken state. We need to do
> all that we can in preparation to be able to recover from any problem.


Unless you use the attribute on an object extended with the aux class,
there's no reason you should not be able to remove the attribute again.
Even if you do use it, removing the attribute and waiting for the values
to be purged from objects should allow removal of the attribute from the
class's definition again.

> What actions should we take during this next week to prepare for another
> attempt at the schema extension? We follow the documented guidelines for
> periodic health checks on the tree, and we'll repeat those. Are there
> other specific actions that we can take to check the health of the
> schema on all replicas?


Having a DIB backup is always prudent. You can do this the supported way
with dsbk or the unsupported way with something like ndsrc.pl
(CoolSolution written to stop eDir, grab NICI and DIB information, then
restart eDir). The ndsrc.pl works really well, but cannot really be used
for a restore the same way dsbk can, so the dsbk way is better/supported/etc.

> When extending the schema, does it matter which replica is used for the
> change (master / non-master)?


As long as you extend it in the [root] partition, all should be well, but
technically schema flows "out and down", meaning out from the Master
replica and down from the current partition to child partitions (if
applicable). As a result, it's best if you extend directly on the box
holding the Master replica of [root].

> What preparations can we make to be able to recover our system if the
> worse case occurs - the schema extension breaks the system again and
> we're unable to rollback the change? I need to be able to propose a
> recovery plan that can restore the system to a working state.


Worst-case scenario, call support. They can back out any schema change if
they have access to the system (remotely) and a tool for doing that kind
of thing (ndsdump).

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...
0 Likes
Knowledge Partner
Knowledge Partner

Re: Extending schema stopped or broke LDAP access

On Mon, 31 Aug 2015 20:24:01 +0000, sedentaryz wrote:

> I appreciate both of you taking time to look at this. This is the first
> time in many, many years of making schema extensions to our eDirectory
> implementation that we've ever run into any problem.


I'm curious about this one, because it makes no sense. I've also made
many schema extensions over the years and never have I seen it do
anything like what you're describing.


> 1) If the problem happens again, how can we diagnose the problem so that
> we can find the cause and make the appropriate fix?


I'd start by asking exactly what problem you have (had)? Is nldap
unloading? Is it loaded but complaining about something? Are there error
messages? What leads you to believe that there is a problem?


> What preparations can we make to be able to recover our system if the
> worse case occurs - the schema extension breaks the system again and
> we're unable to rollback the change?


Call NTS and open an SR. There's nothing they can't fix, one way or
another.


--
--------------------------------------------------------------------------
David Gersic dgersic_@_niu.edu
Knowledge Partner http://forums.microfocus.com

Please post questions in the forums. No support provided via email.
If you find this post helpful, please click on the star below.
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Extending schema stopped or broke LDAP access


>> I appreciate both of you taking time to look at this. This is the

first
>> time in many, many years of making schema extensions to our

eDirectory
>> implementation that we've ever run into any problem.


> I'm curious about this one, because it makes no sense. I've also made
> many schema extensions over the years and never have I seen it do
> anything like what you're describing.


We just attempted to make this schema change again; this time the update
succeeded without any problem. I too would like to know what happened
the last time. Time constraints didn't allow us any opportunity to
diagnose the problem last time, which is unfortunate for understanding
what happened. My guess is either a once in a lifetime "glitch" caused a
problem with the schema update that caused all ldap communications to
the eDir server to fail, or the initial schema update and subsequent
change to revert that update coincided almost exactly with some other
network related problem we had.

This episode did cause us to tighten up our procedures and policies for
making schema modifications, so that if it happens again we'll have the
time and resources available to address the problem a little better.


--
sedentaryz
------------------------------------------------------------------------
sedentaryz's Profile: https://forums.netiq.com/member.php?userid=10365
View this thread: https://forums.netiq.com/showthread.php?t=54196

0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Extending schema stopped or broke LDAP access

sedentaryz wrote:

>
> eDir 8.8 SP8, SLES 11 SP2
>
> I made a schema extension yesterday - I defined a new attribute and
> then added that attribute to an existing class definition.
> Immediately after making that change, we lost all ability to make
> LDAP queries to any replica in the system. We didn't have time to
> trouble shoot the problem in detail - this happened on a live
> production system, and we needed to restore operations ASAP. I
> applied another schema change to remove the new attribute from the
> class definition. That restored operations.


Any errors you got back from LDAP other then "lost the ability to make
LDAP queries"?

Maybe next time when you attempt it run dstrace with +LDAP and when
LDAP breaks look at dstrace and see if you can get any additional info.


--
Cheers,
Edward
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.