Knowledge Partner
Knowledge Partner
621 views

FLAIM Attribute Containerization vs regular indexes

Does anyone have any real-world experience where Attribute Containerization is
better or worse than a regular index (or no index at all).

Thinking primarily in an IDM context, and with regards to dynamic group
membership (LDAP searches).
Not talking about the initial building of the index, but the longer term value
(for search and update performance)

In particular that an attribute has values larger than 2k, are these actually
indexed in the container? Normally value indexes are truncated at a far smaller
length.

--
If you find this post helpful, and are viewing this using the web, please show
your appreciation by clicking on the star below
Alex McHugh - Knowledge Partner - Stavanger, Norway
Who are the Knowledge Partners
If you appreciate my comments, please click the Like button.
If I have resolved your issue, please click the Accept as Solution button.
Labels (1)
0 Likes
12 Replies
Knowledge Partner
Knowledge Partner

Re: FLAIM Attribute Containerization vs regular indexes

Alex McHugh wrote:

> Normally value indexes are truncated at a far smaller length.


Care to elaborate on this?

--
http://www.is4it.de/en/solution/identity-access-management/

(If you find this post helpful, please click on the star below.)
0 Likes
Knowledge Partner
Knowledge Partner

Re: FLAIM Attribute Containerization vs regular indexes

Lothar Haeger wrote:

> > Normally value indexes are truncated at a far smaller length.

>
> Care to elaborate on this?



https://support.novell.com/techcenter/articles/ana20000705.html

"Value indexes on large string or octet string attributes may not provide the
desired performance improvement. NDS truncates indexed string values at 32
bytes and indexed octet string values at 49 bytes. When a query includes a
value that is larger than the truncation value, the index can only be used to
generate a possible result set. Each object in the possible set must then be
read and evaluated to make sure it fits the criteria."

Don't know if this has been improved in later eDir versions. Couldn't find
anything.

--
If you find this post helpful, and are viewing this using the web, please show
your appreciation by clicking on the star below
Alex McHugh - Knowledge Partner - Stavanger, Norway
Who are the Knowledge Partners
If you appreciate my comments, please click the Like button.
If I have resolved your issue, please click the Accept as Solution button.
0 Likes
Knowledge Partner
Knowledge Partner

Re: FLAIM Attribute Containerization vs regular indexes

Alex McHugh wrote:

> https://support.novell.com/techcenter/articles/ana20000705.html
>
> "Value indexes on large string or octet string attributes may not provide the
> desired performance improvement. NDS truncates indexed string values at 32
> bytes and indexed octet string values at 49 bytes. When a query includes a
> value that is larger than the truncation value, the index can only be used to
> generate a possible result set. Each object in the possible set must then be
> read and evaluated to make sure it fits the criteria."
>
> Don't know if this has been improved in later eDir versions. Couldn't find
> anything.


Interesting, I did not know that. And good to see that you were referring to
large individual values and not to large sets of values (which I had
misunderstood first). Thanks a lot!

--
http://www.is4it.de/en/solution/identity-access-management/

(If you find this post helpful, please click on the star below.)
0 Likes
Knowledge Partner
Knowledge Partner

Re: FLAIM Attribute Containerization vs regular indexes

Lothar Haeger wrote:

> Interesting, I did not know that. And good to see that you were referring to
> large individual values and not to large sets of values (which I had
> misunderstood first). Thanks a lot!


We have some "blob" attributes that contain structured data (normally XML) from
a source system, a business logic driver selectively unpacks these.
eDirectory likes to move them to their own container, wanted to know if there
is any reason to block that.

Sometimes, we resorted to using medial searches (blah=*somevalue*) on these
blobs, wanted to know if indexing or containerization would make these faster
(suspect not).

--
If you find this post helpful, and are viewing this using the web, please show
your appreciation by clicking on the star below
Alex McHugh - Knowledge Partner - Stavanger, Norway
Who are the Knowledge Partners
If you appreciate my comments, please click the Like button.
If I have resolved your issue, please click the Accept as Solution button.
0 Likes
Knowledge Partner
Knowledge Partner

Re: FLAIM Attribute Containerization vs regular indexes

Substring indexes are the same with regard to length, I believe, with
thirty-two character limits most of the time. This matters due to things
in RRSD and was made configurable because of a need to make it a bit
longer in some cases. Whether or not it is a good idea to do this,
particularly because of substring indexes, is probably very questionable.

Whether or not auto-movement to containers is good is probably also a good
question depending on use of the attributes. It may be notable that in
eDirectory 9.x they disabled that automatic movement to containers, making
it something you are notified about and then able to do, or not.

Keep in mind that indexes are still primarily used for searching. If you
are not searching based on an attribute, it probably does not make sense
to create an index for it. With those larger values, especially the 2k+
byte ones, I have never seen anybody actually do a search for something 2k
bytes long, since that would be a little crazy.


--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
Knowledge Partner
Knowledge Partner

Re: FLAIM Attribute Containerization vs regular indexes

ab wrote:

> Whether or not auto-movement to containers is good is probably also a good
> question depending on use of the attributes.


Can you elaborate?
All i see is that the containerization can help avoid bloating the entry cache.
What I am asking is: why the mandatory indexing? Is it something that is
near-zero cost as a consequence of the containerization

> It may be notable that in
> eDirectory 9.x they disabled that automatic movement to containers, making
> it something you are notified about and then able to do, or not.
>


The way I understood this TID was that it the automatic movement is disabled
only for very early 9.0.x releases and upgrades.
https://www.novell.com/support/kb/doc.php?id=7017234

See the note way at the end:

"NOTE: A new installation of eDirectory 9.0.2 or later will use automatic
indexing."

--
If you find this post helpful, and are viewing this using the web, please show
your appreciation by clicking on the star below
Alex McHugh - Knowledge Partner - Stavanger, Norway
Who are the Knowledge Partners
If you appreciate my comments, please click the Like button.
If I have resolved your issue, please click the Accept as Solution button.
0 Likes
jwilleke Trusted Contributor.
Trusted Contributor.

Re: FLAIM Attribute Containerization vs regular indexes

In large trees >5MM we have seen issues when we had indexes and the attribute fell into the "rules" for automatic movement.
Apparently the existing index is dropped and then the new index is built. When this happens, there is NO index for LDAP searches and they time-out.

-jim
0 Likes
Knowledge Partner
Knowledge Partner

Re: FLAIM Attribute Containerization vs regular indexes

jwilleke wrote:

> Apparently the existing index is dropped and then the new index is
> built. When this happens, there is NO index for LDAP searches and they
> time-out.


so the time-out occurs until the new index is available? With that many records
- could take a while.

--
If you find this post helpful, and are viewing this using the web, please show
your appreciation by clicking on the star below
Alex McHugh - Knowledge Partner - Stavanger, Norway
Who are the Knowledge Partners
If you appreciate my comments, please click the Like button.
If I have resolved your issue, please click the Accept as Solution button.
0 Likes
Knowledge Partner
Knowledge Partner

Re: FLAIM Attribute Containerization vs regular indexes


> Keep in mind that indexes are still primarily used for searching. If you
> are not searching based on an attribute, it probably does not make sense
> to create an index for it. With those larger values, especially the 2k+
> byte ones, I have never seen anybody actually do a search for something 2k
> bytes long, since that would be a little crazy.


I once asked an eDir guy about the System indexes, for attributes with
more than 10 or 25 (I forget the value now) values.

He explained that to add a new value to an attribute, it first scans all
the existing attribute values to ensure it is not identical. This scan
uses the index, and this is why they auto add an index.


0 Likes
Knowledge Partner
Knowledge Partner

Re: FLAIM Attribute Containerization vs regular indexes

On 06/10/2018 03:52 PM, Geoffrey Carman wrote:
>
> I once asked an eDir guy about the System indexes, for attributes with
> more than 10 or 25 (I forget the value now) values.


It is twenty-five (25) values, or 2k bytes in length (thus the reason IDM
XML content ends up getting indexed so often).

> He explained that to add a new value to an attribute, it first scans all
> the existing attribute values to ensure it is not identical. This scan
> uses the index, and this is why they auto add an index.


That's... interesting. I'd normally take your word for it, but something
about that is fishy to me. In ndsdump when you look at attributes values,
whether indexed or not, they are all in a nice little chain, basically an
array, and it is fast/responsive regardless of whether there is one value
or there are tens of thousands, so I do not think there is much
performance hit in going from one to the next. Also, an index covers all
entries in the DIB, meaning across many objects; when validating a unique
value for a singleobject, you only validate it on the current object
(where you have that nice array thing of values), not among every object
ever. Finally, twenty-five values seems like a pretty small number to
auto-index for value adds

I suppose if you combine using the internal AncestorID index, along with a
value index of this type, perhaps that allows the system to somehow only
search part of the index, but that's getting into internals of the
environment than can go based on experience. I wonder how much of it is
valid in today's FLAIM (vs. recman) environment, or on today's hardware,
or with today's environments.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
Knowledge Partner
Knowledge Partner

Re: FLAIM Attribute Containerization vs regular indexes

On 6/10/2018 11:13 PM, ab wrote:
> On 06/10/2018 03:52 PM, Geoffrey Carman wrote:
>>
>> I once asked an eDir guy about the System indexes, for attributes with
>> more than 10 or 25 (I forget the value now) values.

>
> It is twenty-five (25) values, or 2k bytes in length (thus the reason IDM
> XML content ends up getting indexed so often).


After I wrote this, I saw the TID you posted earlier in the thread that
specifically answered that point. Thanks interesting TID!

>> He explained that to add a new value to an attribute, it first scans all
>> the existing attribute values to ensure it is not identical. This scan
>> uses the index, and this is why they auto add an index.

>
> That's... interesting. I'd normally take your word for it, but something
> about that is fishy to me. In ndsdump when you look at attributes values,


I hear your point. I may be misremembering. And I see your point. Be
fun to find someone who can confirm/deny this point.


> whether indexed or not, they are all in a nice little chain, basically an
> array, and it is fast/responsive regardless of whether there is one value
> or there are tens of thousands, so I do not think there is much
> performance hit in going from one to the next. Also, an index covers all
> entries in the DIB, meaning across many objects; when validating a unique
> value for a singleobject, you only validate it on the current object
> (where you have that nice array thing of values), not among every object
> ever. Finally, twenty-five values seems like a pretty small number to
> auto-index for value adds
>
> I suppose if you combine using the internal AncestorID index, along with a
> value index of this type, perhaps that allows the system to somehow only
> search part of the index, but that's getting into internals of the
> environment than can go based on experience. I wonder how much of it is
> valid in today's FLAIM (vs. recman) environment, or on today's hardware,
> or with today's environments.
>


0 Likes
Sabhay1 Absent Member.
Absent Member.

Re: FLAIM Attribute Containerization vs regular indexes

geoffc;2482295 wrote:

I once asked an eDir guy about the System indexes, for attributes with
more than 10 or 25 (I forget the value now) values.

It is 25.

geoffc;2482295 wrote:

He explained that to add a new value to an attribute, it first scans all
the existing attribute values to ensure it is not identical. This scan
uses the index, and this is why they auto add an index.


Correct. and to add more to this point, it is not only adding of a new value but all types of modifications(update, sync, etc.) where it will use this new index to search for the values. AFAIK, the actual character limit still stays whether you move an attribute to a separate container or not.
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.