Anonymous_User Absent Member.
Absent Member.
1162 views

acquireWriteSemaphore hang / Null Pointer exception in Reader thread


I'm seeing LDAPConnection methods hang sometimes when run against Fedora
Core DS 1.0.4. I'm running the latest jldap from CVS. Doing kill -3 to
get the stack trace:

com.novell.ldap.Connection.acquireWriteSemaphore(Connection.java:285)
- locked <0x26ad1c18> (a java.lang.Object)
at com.novell.ldap.Connection.shutdown(Connection.java:948)
at com.novell.ldap.Connection.destroyClone(Connection.java:590)
- locked <0x26ad1ab8> (a com.novell.ldap.Connection)
at com.novell.ldap.LDAPConnection.disconnect(LDAPConnection.java:2344)
at com.novell.ldap.LDAPConnection.disconnect(LDAPConnection.java:2304)

Sometimes this stack trace is hung in a bind instead of a disconnect.
Either way, this happens right after the following null pointer exception
is printed to standard error.

Exception in thread "Thread-7845" java.lang.NullPointerException
at com.novell.ldap.Message.putReply(Message.java:328)
at com.novell.ldap.Connection$ReaderThread.run(Connection.java:1295)
at java.lang.Thread.run(Thread.java:613)


Basically, the null pointer in the Connection's reader thread results in
the semaphore for the Connection class not getting freed. As a result, the
next call to acquireWriteSemaphore just hangs.

If you follow the Null Pointer, you'll see it is because the conn member in
Message.class is null. The only way this can happen is if a null
Connection is passed in during construction of the Message object *highly
unlikely*, or if Message::cleanup had already been called for the class.

There does appear to be a timing problem in the code. In lines 504-506 of
Message::abandon we have:

LDAPMessage msg = new LDAPAbandonRequest( msgId, cont);
// Send abandon message to server
conn.writeMessage( msg);

Later in the method in line 537 we have:
cleanup();

cleanup is the only method in the Connection class which sets the conn
member to null. It is also only called in the finalizer and from
Message::abandon.

So it would seem that the following scenario would cause the null pointer
exception:

1) Message::abandon is called
2) LDAPAbandonRequest is sent to server
3) cleanup is called, setting conn to null
4) Reply to LDAPAbandonRequest is received from server, calling
Message::putReply()
5) Message::putReply null pointers since conn object is null and doesn't
free the semaphore

It seems that the code assumes that 4 happens before 3. However, if the
LDAP server is under enough load, 3 could happen before 4.

Indeed, in my environment, if I simply comment out setting conn to null in
Message::cleanup() everything works. I am not sure that this is an
acceptable solution though. Is there a better way to fix this issue?

I'd appreciate any help or suggestions. If it is useful, I can post a test
program which causes the problem. Basically it just fires up 30 threads
each of which do the following in a continuous loop: new LDAPConnection,
connect, bind, disconnect. One thread inevitably hangs after a few
minutes. Thanks.

Labels (1)
0 Likes
3 Replies
Anonymous_User Absent Member.
Absent Member.

Re: acquireWriteSemaphore hang / Null Pointer exception in Reader thread


This actually still seems to be an issue, did anyone look at this for a
code fix?

Cheers,
Tomas


--
primetomas
------------------------------------------------------------------------
primetomas's Profile: http://forums.novell.com/member.php?userid=102151
View this thread: http://forums.novell.com/showthread.php?t=364974

0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: acquireWriteSemaphore hang / Null Pointer exception in Reader thread


I also believe I have seen this problem. I've seen threads block if the
system is busy and timeouts trigger an abandon to be called. Is the
change described above a legitimate fix? If so why was it not included
in the 2008 March release? I will try this, but would appreciate if
anyone was able to confirm it works independently. thanks


--
bcampbell88
------------------------------------------------------------------------
bcampbell88's Profile: http://forums.novell.com/member.php?userid=104522
View this thread: http://forums.novell.com/showthread.php?t=364974

0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: acquireWriteSemaphore hang / Null Pointer exception in Reader thread


Hello,

we have the same problem.
Our server uses the LDAP-API.
The system runs without problems, suddenly the thread count goes up (it
varys from 3000 to 16000).
After several minutes the NullPointerException occurs and the vm freezes
(the thread-dump shows that a thread is in WAITING state at the LDAP
code)

"RMI TCP Connection(14917)-172.29.48.81" - Thread t@216581
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <702dcb15> (a java.lang.Object)
at java.lang.Object.wait(Object.java:485)
at com.novell.ldap.Connection.acquireWriteSemaphore(Unknown Source)
at com.novell.ldap.Connection.writeMessage(Unknown Source)
at com.novell.ldap.Connection.writeMessage(Unknown Source)
at com.novell.ldap.Message.sendMessage(Unknown Source)
at com.novell.ldap.MessageAgent.sendMessage(Unknown Source)
at com.novell.ldap.LDAPConnection.search(Unknown Source)
at com.novell.ldap.LDAPConnection.search(Unknown Source)

I have tried to repdroduce the scenario but i wasn't successful.
Can you please post your testcode so that i can compare it with my code
?

With kind regards

Dirk


--
dciese
------------------------------------------------------------------------
dciese's Profile: https://forums.netiq.com/member.php?userid=3505
View this thread: https://forums.netiq.com/showthread.php?t=2579

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.