Subject | How to debug pool issues |
---|---|
Author | Steffen Heil |
Post date | 2011-05-22T16:02:05Z |
Hi
I am using jaybird since several years and within the last weeks I ran 3
times into a situation where I would not get any more connections to the
firebird pool.
My problem is that I cannot reproduce that issue, the heap dumps I created
don't actually help me much and I don't know where to start.
In any case the system in question was a heavily loaded webserver, but only
using database connections for very few requests.
That lead me to the idea that the issue was caused by some severe server
load that triggered some raise condition.
I took thread dumps and the problematic thread dump is listed below. That
call blocked several minutes unit it failed.
However as I always acquire new connection though some helper object that is
itself blocking on getConnection(), the next waiting thread run into the
very same problem just after the first timed out starting the timeout itself
again.
Where could I start searching for this issure.
(BTW: I could only get the webserver running again by restarting the whole
JVM. As I cannot reproduce it directly, I have no good way to test
solutions; IF the problem occurs again, I might take short tests, but I need
to get the webserver up again. Therefor I would prefer to add as much
logging as possible, but I don't know how.)
Regards,
Steffen
"osiris4-exec-1792" daemon prio=10 tid=0x0000000040207800 nid=0x4e95 in
Object.wait() [0x0000000047aaa000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000077c595320> (a java.lang.Object)
at org.firebirdsql.pool.BlockingStack.pop(BlockingStack.java:197)
- locked <0x000000077c595320> (a java.lang.Object)
at
org.firebirdsql.pool.PooledConnectionQueue.take(PooledConnectionQueue.java:4
12)
- locked <0x000000077c5576d0> (a java.lang.Object)
at
org.firebirdsql.pool.AbstractConnectionPool.getPooledConnection(AbstractConn
ectionPool.java:230)
- locked <0x000000077c53b848> (a
org.firebirdsql.pool.FBConnectionPoolDataSource)
at
org.firebirdsql.pool.AbstractFBConnectionPoolDataSource.getPooledConnection(
AbstractFBConnectionPoolDataSource.java:359)
- locked <0x000000077c53b848> (a
org.firebirdsql.pool.FBConnectionPoolDataSource)
at
org.firebirdsql.pool.AbstractFBConnectionPoolDataSource.getPooledConnection(
AbstractFBConnectionPoolDataSource.java:379)
- locked <0x000000077c53b848> (a
org.firebirdsql.pool.FBConnectionPoolDataSource)
at
org.firebirdsql.pool.FBWrappingDataSource.getConnection(FBWrappingDataSource
.java:216)
at
com.osiris4.core.persistence.Database.getSafeConnection(Database.java:300)
- locked <0x000000077c5211d0> (a
com.osiris4.core.persistence.Database)
......
I am using jaybird since several years and within the last weeks I ran 3
times into a situation where I would not get any more connections to the
firebird pool.
My problem is that I cannot reproduce that issue, the heap dumps I created
don't actually help me much and I don't know where to start.
In any case the system in question was a heavily loaded webserver, but only
using database connections for very few requests.
That lead me to the idea that the issue was caused by some severe server
load that triggered some raise condition.
I took thread dumps and the problematic thread dump is listed below. That
call blocked several minutes unit it failed.
However as I always acquire new connection though some helper object that is
itself blocking on getConnection(), the next waiting thread run into the
very same problem just after the first timed out starting the timeout itself
again.
Where could I start searching for this issure.
(BTW: I could only get the webserver running again by restarting the whole
JVM. As I cannot reproduce it directly, I have no good way to test
solutions; IF the problem occurs again, I might take short tests, but I need
to get the webserver up again. Therefor I would prefer to add as much
logging as possible, but I don't know how.)
Regards,
Steffen
"osiris4-exec-1792" daemon prio=10 tid=0x0000000040207800 nid=0x4e95 in
Object.wait() [0x0000000047aaa000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000077c595320> (a java.lang.Object)
at org.firebirdsql.pool.BlockingStack.pop(BlockingStack.java:197)
- locked <0x000000077c595320> (a java.lang.Object)
at
org.firebirdsql.pool.PooledConnectionQueue.take(PooledConnectionQueue.java:4
12)
- locked <0x000000077c5576d0> (a java.lang.Object)
at
org.firebirdsql.pool.AbstractConnectionPool.getPooledConnection(AbstractConn
ectionPool.java:230)
- locked <0x000000077c53b848> (a
org.firebirdsql.pool.FBConnectionPoolDataSource)
at
org.firebirdsql.pool.AbstractFBConnectionPoolDataSource.getPooledConnection(
AbstractFBConnectionPoolDataSource.java:359)
- locked <0x000000077c53b848> (a
org.firebirdsql.pool.FBConnectionPoolDataSource)
at
org.firebirdsql.pool.AbstractFBConnectionPoolDataSource.getPooledConnection(
AbstractFBConnectionPoolDataSource.java:379)
- locked <0x000000077c53b848> (a
org.firebirdsql.pool.FBConnectionPoolDataSource)
at
org.firebirdsql.pool.FBWrappingDataSource.getConnection(FBWrappingDataSource
.java:216)
at
com.osiris4.core.persistence.Database.getSafeConnection(Database.java:300)
- locked <0x000000077c5211d0> (a
com.osiris4.core.persistence.Database)
......