| Subject | Re: [firebird-support] Insert ot update statements locks a table and all other instances that try to run same statement are lcoked to |
|---|---|
| Author | Mark Rotteveel |
| Post date | 2014-09-21T16:46:42Z |
On 18-9-2014 10:14, doychin@... [firebird-support] wrote:
stacktrace. All Firebird connections are waiting for a response from the
server.
I don't know enough about reading below The Firebird stacktraces, but it
looks like apart for process 14830 that they are all waiting for a lock.
Mark Rotteveel
> here is a stack trace from Java app server. After that you will findWell, the 'good' news is that I don't see anything strange in the Java
> back trace from some of fb_inet processes and a list of all live
> processes. The mssing back traces are the same like for PID Attaching to
> process 16275.
>
> After that is the output of fb_lock_print and then is the output from
> monitoring tables for all running queries.
stacktrace. All Firebird connections are waiting for a response from the
server.
I don't know enough about reading below The Firebird stacktraces, but it
looks like apart for process 14830 that they are all waiting for a lock.
> Attaching to process 14830--
>
> #0 0x00007fb6da7905b3 in __select_nocancel () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007fb6db7eace1 in packet_receive (port=0x7fb6dba93080,
> buffer=0x7fb6dba93222 "", buffer_length=8192, length=0x7fff4400f60e) at
> ../src/remote/inet.cpp:3689
> #2 0x00007fb6db7eaac1 in inet_read (xdrs=0x7fb6dba93148) at
> ../src/remote/inet.cpp:3369
> #3 0x00007fb6db7ea856 in inet_getbytes (xdrs=0x7fb6dba93148,
> buff=0x7fff4400f674 "", count=0) at ../src/remote/inet.cpp:3112
> #4 0x00007fb6db7ea89f in inet_getlong (xdrs=0x1, lp=0x7fff4400f694) at
> ../src/remote/inet.cpp:3134
> #5 0x00007fb6db7fff0e in xdr_enum (xdrs=0x1, ip=0x7fff4400f760) at
> ../src/remote/xdr.cpp:433
> #6 0x00007fb6db7fd277 in xdr_protocol (xdrs=0x7fb6dba93148,
> p=0x7fff4400f760) at ../src/remote/protocol.cpp:299
> #7 0x00007fb6db7ede1d in receive (main_port=0x7fb6dba93080,
> packet=0x7fff4400f760) at ../src/remote/inet.cpp:2436
> #8 0x00007fb6db7ff90a in rem_port::receive (this=0xfffffffffffffdfe,
> pckt=0x7fff4400f0c0) at ../src/remote/remote.cpp:780
> #9 0x00007fb6db80f820 in SRVR_main (main_port=0x7fb6dba93080,
> flags=61632) at ../src/remote/server.cpp:290
> #10 0x00007fb6db80f2ee in server_main (argc=1, argv=0x7fff44010380) at
> ../src/remote/inet_server.cpp:450
> #11 0x0000000000400709 in main (argc=1, argv=0x7fff4400f0c0) at
> ../src/remote/server_stub.cpp:12
>
>
> Attaching to process 16275
>
> #0 0x00007f1c3c7ec187 in semop () at ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f1c3d678dee in ISC_event_wait (count=1,
> events=0x7fffe206c690, values=0x7fffe206c69c, micro_seconds=10000000,
> timeout_handler=
> 0x7f1c3d83a510 <lock_alarm_handler(void*)>,
> handler_arg=0x7f1c3bd19e04) at ../src/jrd/isc_sync.cpp:4583
> #2 0x00007f1c3d83bcad in wait_for_request (request=0x7f1c3b587d78,
> lck_wait=1, status_vector=0x7f1c3bd30000) at ../src/lock/lock.cpp:5059
> #3 0x00007f1c3d83a28e in grant_or_que (request=0x7f1c3b587d78,
> lock=0x43fd78, lck_wait=1) at ../src/lock/lock.cpp:3006
> #4 0x00007f1c3d83775a in LOCK_enq (prior_request=995655032,
> parent_request=994547264, series=4, value=0x7fffe206c8f0 "r\267\b\001",
> length=4, type=3 '\003',
> ast_routine=0xfffffffffffffffc, ast_argument=0xfffffffffffffffc,
> data=0, lck_wait=1, status_vector=0x7fffe206ca50,
> owner_offset=991199232) at ../src/lock/lock.cpp:567
> #5 0x00007f1c3d773801 in LCK_lock (tdbb=0x0, lock=0x7fffe206c860,
> level=3, wait=1) at ../src/jrd/lck.cpp:1084
> #6 0x00007f1c3d773900 in LCK_lock_non_blocking (tdbb=0x218005,
> lock=0x7fffe206c860, level=3, wait=1) at ../src/jrd/lck.cpp:810
> #7 0x00007f1c3d7c3605 in TRA_wait (tdbb=0x7fffe206f320,
> trans=0x7f1c3b98ec48, number=17348466, wait=jrd_tra::tra_wait) at
> ../src/jrd/tra.cpp:2099
> #8 0x00007f1c3d7cc071 in VIO_get_current (tdbb=0x7fffe206f320,
> old_rpb=0x7fffe206c3b0, rpb=0x7fffe206cce0, transaction=0x7f1c3b98ec48,
> pool=0x7f1c3b98e1a0,
> foreign_key=114, has_old_values=@0xfffffffffffffffc) at
> ../src/jrd/vio.cpp:1941
> #9 0x00007f1c3d74a041 in check_duplicates (tdbb=0x7fffe206f320,
> record=0x7f1c3b98c8c0, record_idx=0x7fffe206df70,
> insertion=0x7fffe206df20, relation_2=0x0)
> at ../src/jrd/jrd.h:905
> #10 0x00007f1c3d749cbe in insert_key (tdbb=0x7fffe206f320,
> relation=0x7f1c3ba95b58, record=0x7f1c3b98c8c0,
> transaction=0x7f1c3b98ec48, window_ptr=0x7fffe206def0,
> insertion=0x7fffe206df20, bad_relation=0x7fffe206dea0,
> bad_index=0xfffffffffffffffc) at ../src/jrd/idx.cpp:1650
> #11 0x00007f1c3d74907a in IDX_store (tdbb=0x7fffe206f320,
> rpb=0x7f1c3b98d6a8, transaction=0x7f1c3b98ec48) at ../src/jrd/idx.cpp:1062
> #12 0x00007f1c3d73a498 in store (tdbb=0x7fffe206f320,
> node=0x7f1c3b9a7328, which_trig=0) at ../src/jrd/exe.cpp:3685
> #13 0x00007f1c3d738295 in looper (tdbb=0x7fffe206f320,
> request=0x7f1c3b98d278, in_node=0x7f1c3d738268) at ../src/jrd/exe.cpp:2635
> #14 0x00007f1c3d737016 in execute_looper (tdbb=0x7fffe206f320,
> request=0x7f1c3b98d278, transaction=0x7f1c3b98ec48,
> next_state=jrd_req::req_proceed)
> at ../src/jrd/exe.cpp:1443
> #15 0x00007f1c3d7367a0 in EXE_send (tdbb=0x7fffe206f320,
> request=0x7f1c3b98d278, msg=6, length=80, buffer=0x7f1c3b9a1f78 "\t") at
> ../src/jrd/exe.cpp:993
> #16 0x00007f1c3d76ad04 in jrd8_start_and_send
> (user_status=0x7fffe206fa40, req_handle=0x7f1c3b98d278,
> tra_handle=0x7f1c3b98ec48, msg_type=0, msg_length=40,
> msg=0x7f1c3b7a1f98 "", level=0) at ../src/jrd/jrd.cpp:3789
> #17 0x00007f1c3d68c153 in isc_start_and_send (user_status=0x218005,
> req_handle=0x7f1c3bd3ca18, tra_handle=0x7fffe206fa40, msg_type=0,
> msg_length=40,
> msg=0x7f1c3b7a1f98 "", level=0) at ../src/jrd/why.cpp:450
> #18 0x00007f1c3d80414f in execute_request (request=0x7f1c3b7a2da8,
> trans_handle=0x7f1c3b7a28c8, in_blr_length=20, in_blr=0x7f1c3bd3bd00
> "\005\002\004", in_msg_length=34,
> in_msg=0x7f1c3bd3d4a8 "", out_blr_length=0, out_blr=0x0,
> out_msg_length=0, out_msg=0x0, singleton=false) at ../src/dsql/dsql.cpp:3397
> #19 0x00007f1c3d801f7f in GDS_DSQL_EXECUTE_CPP (user_status=0x218005,
> trans_handle=0x7fffe206fa3c, req_handle=0x20d8b, in_blr_length=20,
> in_blr=0x7f1c3bd3bd00 "\005\002\004", in_msg_type=0,
> in_msg_length=34, in_msg=0x7f1c3bd3d4a8 "", out_blr_length=0,
> out_blr=0xfffffffffffffffc <Address 0xfffffffffffffffc out of
> bounds>, out_msg_type=0, out_msg_length=0,
> out_msg=0xfffffffffffffffc <Address 0xfffffffffffffffc out of
> bounds>) at ../src/dsql/dsql.cpp:561
> #20 0x00007f1c3d685688 in isc_dsql_execute2_m (user_status=0x218005,
> tra_handle=0x7fffe206fa3c, stmt_handle=0x7f1c3bd3cb30, in_blr_length=20,
> in_blr=0x7f1c3bd3bd00 "\005\002\004", in_msg_type=0,
> in_msg_length=34, in_msg=0x7f1c3bd3d4a8 "", out_blr_length=34,
> out_blr=0x0, out_msg_type=54440,
> out_msg_length=0, out_msg=0x0) at ../src/jrd/why.cpp:450
> #21 0x00007f1c3d8632b3 in rem_port::execute_statement
> (this=0x7f1c3dae5080, op=op_execute, sqldata=0x218005,
> sendL=0x7fffe20702f0) at ../src/remote/server.cpp:2180
> #22 0x00007f1c3d8656b4 in process_packet2 (port=0x7f1c3dae5080,
> sendL=0x7fffe20702f0, receive=0x7fffe206fe90, result=0x7fffe206fe88) at
> ../src/remote/server.cpp:3644
> #23 0x00007f1c3d8650d2 in process_packet (port=0x7f1c3dae5080,
> sendL=0x7fffe206c3b0, receive=0x1, result=0x7fffe206fe88) at
> ../src/remote/server.cpp:3393
> #24 0x00007f1c3d861841 in SRVR_main (main_port=0x7f1c3dae5080,
> flags=50096) at ../src/remote/server.cpp:294
> #25 0x00007f1c3d8612ee in server_main (argc=2195461,
> argv=0x7fffe2070ab0) at ../src/remote/inet_server.cpp:450
> #26 0x0000000000400709 in main (argc=2195461, argv=0x7fffe206c3b0) at
> ../src/remote/server_stub.cpp:12
>
>
> Attaching to process 16276
> Reading symbols from /opt/firebird/bin/fb_inet_server...Reading symbols
> from /opt/firebird/bin/.debug/fb_inet_server.debug...done.
> done.
> Reading symbols from /opt/firebird/lib/libfbembed.so.2.1...Reading
> symbols from /opt/firebird/lib/.debug/libfbembed.so.2.1.6.debug...done.
> done.
> Loaded symbols for /opt/firebird/lib/libfbembed.so.2.1
> Reading symbols from /lib64/libdl.so.2...Reading symbols from
> /usr/lib/debug/lib64/libdl-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libdl.so.2
> Reading symbols from /lib64/libncurses.so.5...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libncurses.so.5
> Reading symbols from /usr/lib64/libstdc++.so.5...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libstdc++.so.5
> Reading symbols from /lib64/libm.so.6...Reading symbols from
> /usr/lib/debug/lib64/libm-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libm.so.6
> Reading symbols from /lib64/libgcc_s.so.1...Reading symbols from
> /usr/lib/debug/lib64/libgcc_s-4.4.7-20120601.so.1.debug...done.
> done.
> Loaded symbols for /lib64/libgcc_s.so.1
> Reading symbols from /lib64/libc.so.6...Reading symbols from
> /usr/lib/debug/lib64/libc-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /opt/firebird/lib/libicuuc.so.30...Reading symbols
> from /opt/firebird/lib/.debug/libicuuc.so.30.0.debug...done.
> done.
> Loaded symbols for /opt/firebird/lib/libicuuc.so.30
> Reading symbols from /opt/firebird/lib/libicudata.so.30...Reading
> symbols from /opt/firebird/lib/.debug/libicudata.so.30.0.debug...done.
> done.
> Loaded symbols for /opt/firebird/lib/libicudata.so.30
> Reading symbols from /opt/firebird/lib/libicui18n.so.30...Reading
> symbols from /opt/firebird/lib/.debug/libicui18n.so.30.0.debug...done.
> done.
> Loaded symbols for /opt/firebird/lib/libicui18n.so.30
> Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from
> /usr/lib/debug/lib64/ld-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libtinfo.so.5
> Reading symbols from /lib64/libpthread.so.0...Reading symbols from
> /usr/lib/debug/lib64/libpthread-2.12.so.debug...done.
> [Thread debugging using libthread_db enabled]
> done.
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from /opt/firebird/intl/fbintl...Reading symbols from
> /opt/firebird/intl/.debug/fbintl.debug...done.
> done.
> Loaded symbols for /opt/firebird/intl/fbintl
> 0x00007fda6896b187 in semop () at ../sysdeps/unix/syscall-template.S:82
> 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> Missing separate debuginfos, use: debuginfo-install
> compat-libstdc++-33-3.2.3-69.el6.x86_64
> ncurses-libs-5.7-3.20090208.el6.x86_64
> (gdb) bt
> #0 0x00007fda6896b187 in semop () at ../sysdeps/unix/syscall-template.S:82
> #1 0x00007fda697f7dee in ISC_event_wait (count=1,
> events=0x7fff9e2f8800, values=0x7fff9e2f880c, micro_seconds=10000000,
> timeout_handler=
> 0x7fda699b9510 <lock_alarm_handler(void*)>,
> handler_arg=0x7fda67e98fa4) at ../src/jrd/isc_sync.cpp:4583
> #2 0x00007fda699bacad in wait_for_request (request=0x7fda677c96c8,
> lck_wait=1, status_vector=0x0) at ../src/lock/lock.cpp:5059
> #3 0x00007fda699b928e in grant_or_que (request=0x7fda677c96c8,
> lock=0x5026c8, lck_wait=1) at ../src/lock/lock.cpp:3006
> #4 0x00007fda699b675a in LOCK_enq (prior_request=1736218312,
> parent_request=1734313536, series=4, value=0x7fff9e2f8a60 "r\267\b\001",
> length=4, type=3 '\003',
> ast_routine=0xfffffffffffffffc, ast_argument=0xfffffffffffffffc,
> data=0, lck_wait=1, status_vector=0x7fff9e2f8bc0,
> owner_offset=1730965504) at ../src/lock/lock.cpp:567
> #5 0x00007fda698f2801 in LCK_lock (tdbb=0x0, lock=0x7fff9e2f89d0,
> level=3, wait=1) at ../src/jrd/lck.cpp:1084
> #6 0x00007fda698f2900 in LCK_lock_non_blocking (tdbb=0x218005,
> lock=0x7fff9e2f89d0, level=3, wait=1) at ../src/jrd/lck.cpp:810
> #7 0x00007fda69942605 in TRA_wait (tdbb=0x7fff9e2fb490,
> trans=0x7fda67949258, number=17348466, wait=jrd_tra::tra_wait) at
> ../src/jrd/tra.cpp:2099
> #8 0x00007fda6994b071 in VIO_get_current (tdbb=0x7fff9e2fb490,
> old_rpb=0x7fff9e2f8520, rpb=0x7fff9e2f8e50, transaction=0x7fda67949258,
> pool=0x7fda67ac15e8, foreign_key=114,
> has_old_values=@0xfffffffffffffffc) at ../src/jrd/vio.cpp:1941
> #9 0x00007fda698c9041 in check_duplicates (tdbb=0x7fff9e2fb490,
> record=0x7fda67ac1448, record_idx=0x7fff9e2fa0e0,
> insertion=0x7fff9e2fa090, relation_2=0x0)
> at ../src/jrd/jrd.h:905
> #10 0x00007fda698c8cbe in insert_key (tdbb=0x7fff9e2fb490,
> relation=0x7fda67c14b58, record=0x7fda67ac1448,
> transaction=0x7fda67949258, window_ptr=0x7fff9e2fa060,
> insertion=0x7fff9e2fa090, bad_relation=0x7fff9e2fa010,
> bad_index=0xfffffffffffffffc) at ../src/jrd/idx.cpp:1650
> #11 0x00007fda698c807a in IDX_store (tdbb=0x7fff9e2fb490,
> rpb=0x7fda67ac32b0, transaction=0x7fda67949258) at ../src/jrd/idx.cpp:1062
> #12 0x00007fda698b9498 in store (tdbb=0x7fff9e2fb490,
> node=0x7fda67944960, which_trig=0) at ../src/jrd/exe.cpp:3685
> #13 0x00007fda698b7295 in looper (tdbb=0x7fff9e2fb490,
> request=0x7fda67ac2e80, in_node=0x7fda698b7268) at ../src/jrd/exe.cpp:2635
> #14 0x00007fda698b6016 in execute_looper (tdbb=0x7fff9e2fb490,
> request=0x7fda67ac2e80, transaction=0x7fda67949258,
> next_state=jrd_req::req_proceed) at ../src/jrd/exe.cpp:1443
> #15 0x00007fda698b57a0 in EXE_send (tdbb=0x7fff9e2fb490,
> request=0x7fda67ac2e80, msg=6, length=80, buffer=0x7fda6794f180 "\t") at
> ../src/jrd/exe.cpp:993
> #16 0x00007fda698e9d04 in jrd8_start_and_send
> (user_status=0x7fff9e2fbbb0, req_handle=0x7fda67ac2e80,
> tra_handle=0x7fda67949258, msg_type=0, msg_length=40,
> msg=0x7fda66291f98 "", level=0) at ../src/jrd/jrd.cpp:3789
> #17 0x00007fda6980b153 in isc_start_and_send (user_status=0x218005,
> req_handle=0x7fda67ebc5a8, tra_handle=0x7fff9e2fbbb0, msg_type=0,
> msg_length=40, msg=0x7fda66291f98 "",
> level=0) at ../src/jrd/why.cpp:450
> #18 0x00007fda6998314f in execute_request (request=0x7fda66292da8,
> trans_handle=0x7fda662928c8, in_blr_length=20, in_blr=0x7fda67eba1e0
> "\005\002\004", in_msg_length=34,
> in_msg=0x7fda67ebc0b0 "", out_blr_length=0, out_blr=0x0,
> out_msg_length=0, out_msg=0x0, singleton=false) at ../src/dsql/dsql.cpp:3397
> #19 0x00007fda69980f7f in GDS_DSQL_EXECUTE_CPP (user_status=0x218005,
> trans_handle=0x7fff9e2fbbac, req_handle=0x16a43, in_blr_length=20,
> in_blr=0x7fda67eba1e0 "\005\002\004", in_msg_type=0,
> in_msg_length=34, in_msg=0x7fda67ebc0b0 "", out_blr_length=0,
> out_blr=0xfffffffffffffffc <Address 0xfffffffffffffffc out of
> bounds>, out_msg_type=0, out_msg_length=0,
> out_msg=0xfffffffffffffffc <Address 0xfffffffffffffffc out of
> bounds>) at ../src/dsql/dsql.cpp:561
> #20 0x00007fda69804688 in isc_dsql_execute2_m (user_status=0x218005,
> tra_handle=0x7fff9e2fbbac, stmt_handle=0x7fda67ebc198, in_blr_length=20,
> in_blr=0x7fda67eba1e0 "\005\002\004", in_msg_type=0,
> in_msg_length=34, in_msg=0x7fda67ebc0b0 "", out_blr_length=34,
> out_blr=0x0, out_msg_type=49328, out_msg_length=0,
> out_msg=0x0) at ../src/jrd/why.cpp:450
> #21 0x00007fda699e22b3 in rem_port::execute_statement
> (this=0x7fda69c64080, op=op_execute, sqldata=0x218005,
> sendL=0x7fff9e2fc460) at ../src/remote/server.cpp:2180
> #22 0x00007fda699e46b4 in process_packet2 (port=0x7fda69c64080,
> sendL=0x7fff9e2fc460, receive=0x7fff9e2fc000, result=0x7fff9e2fbff8) at
> ../src/remote/server.cpp:3644
> #23 0x00007fda699e40d2 in process_packet (port=0x7fda69c64080,
> sendL=0x7fff9e2f8520, receive=0x1, result=0x7fff9e2fbff8) at
> ../src/remote/server.cpp:3393
> #24 0x00007fda699e0841 in SRVR_main (main_port=0x7fda69c64080,
> flags=34080) at ../src/remote/server.cpp:294
> #25 0x00007fda699e02ee in server_main (argc=2195461,
> argv=0x7fff9e2fcc20) at ../src/remote/inet_server.cpp:450
> #26 0x0000000000400709 in main (argc=2195461, argv=0x7fff9e2f8520) at
> ../src/remote/server_stub.cpp:12
>
>
>
> Attaching to process 17159
> Reading symbols from /opt/firebird/bin/fb_inet_server...Reading symbols
> from /opt/firebird/bin/.debug/fb_inet_server.debug...done.
> done.
> Reading symbols from /opt/firebird/lib/libfbembed.so.2.1...Reading
> symbols from /opt/firebird/lib/.debug/libfbembed.so.2.1.6.debug...done.
> done.
> Loaded symbols for /opt/firebird/lib/libfbembed.so.2.1
> Reading symbols from /lib64/libdl.so.2...Reading symbols from
> /usr/lib/debug/lib64/libdl-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libdl.so.2
> Reading symbols from /lib64/libncurses.so.5...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libncurses.so.5
> Reading symbols from /usr/lib64/libstdc++.so.5...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libstdc++.so.5
> Reading symbols from /lib64/libm.so.6...Reading symbols from
> /usr/lib/debug/lib64/libm-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libm.so.6
> Reading symbols from /lib64/libgcc_s.so.1...Reading symbols from
> /usr/lib/debug/lib64/libgcc_s-4.4.7-20120601.so.1.debug...done.
> done.
> Loaded symbols for /lib64/libgcc_s.so.1
> Reading symbols from /lib64/libc.so.6...Reading symbols from
> /usr/lib/debug/lib64/libc-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /opt/firebird/lib/libicuuc.so.30...Reading symbols
> from /opt/firebird/lib/.debug/libicuuc.so.30.0.debug...done.
> done.
> Loaded symbols for /opt/firebird/lib/libicuuc.so.30
> Reading symbols from /opt/firebird/lib/libicudata.so.30...Reading
> symbols from /opt/firebird/lib/.debug/libicudata.so.30.0.debug...done.
> done.
> Loaded symbols for /opt/firebird/lib/libicudata.so.30
> Reading symbols from /opt/firebird/lib/libicui18n.so.30...Reading
> symbols from /opt/firebird/lib/.debug/libicui18n.so.30.0.debug...done.
> done.
> Loaded symbols for /opt/firebird/lib/libicui18n.so.30
> Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from
> /usr/lib/debug/lib64/ld-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libtinfo.so.5
> Reading symbols from /lib64/libpthread.so.0...Reading symbols from
> /usr/lib/debug/lib64/libpthread-2.12.so.debug...done.
> [Thread debugging using libthread_db enabled]
> done.
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from /opt/firebird/intl/fbintl...Reading symbols from
> /opt/firebird/intl/.debug/fbintl.debug...done.
> done.
> Loaded symbols for /opt/firebird/intl/fbintl
> 0x00007f04c1018187 in semop () at ../sysdeps/unix/syscall-template.S:82
> 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> Missing separate debuginfos, use: debuginfo-install
> compat-libstdc++-33-3.2.3-69.el6.x86_64
> ncurses-libs-5.7-3.20090208.el6.x86_64
> (gdb) bt
> #0 0x00007f04c1018187 in semop () at ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f04c1ea4dee in ISC_event_wait (count=1,
> events=0x7fff0754dfd0, values=0x7fff0754dfdc, micro_seconds=10000000,
> timeout_handler=
> 0x7f04c2066510 <lock_alarm_handler(void*)>,
> handler_arg=0x7f04c054538c) at ../src/jrd/isc_sync.cpp:4583
> #2 0x00007f04c2067cad in wait_for_request (request=0x7f04bff0f2f0,
> lck_wait=1, status_vector=0x0) at ../src/lock/lock.cpp:5059
> #3 0x00007f04c206628e in grant_or_que (request=0x7f04bff0f2f0,
> lock=0x59b2f0, lck_wait=1) at ../src/lock/lock.cpp:3006
> #4 0x00007f04c206375a in LOCK_enq (prior_request=-1074728208,
> parent_request=-1077258688, series=4, value=0x7fff0754e230
> "r\267\b\001", length=4, type=3 '\003',
> ast_routine=0xfffffffffffffffc, ast_argument=0xfffffffffffffffc,
> data=0, lck_wait=1, status_vector=0x7fff0754e390,
> owner_offset=-1080606720) at ../src/lock/lock.cpp:567
> #5 0x00007f04c1f9f801 in LCK_lock (tdbb=0x0, lock=0x7fff0754e1a0,
> level=3, wait=1) at ../src/jrd/lck.cpp:1084
> #6 0x00007f04c1f9f900 in LCK_lock_non_blocking (tdbb=0x218005,
> lock=0x7fff0754e1a0, level=3, wait=1) at ../src/jrd/lck.cpp:810
> #7 0x00007f04c1fef605 in TRA_wait (tdbb=0x7fff07550c60,
> trans=0x7f04bffa2978, number=17348466, wait=jrd_tra::tra_wait) at
> ../src/jrd/tra.cpp:2099
> #8 0x00007f04c1ff8071 in VIO_get_current (tdbb=0x7fff07550c60,
> old_rpb=0x7fff0754dcf0, rpb=0x7fff0754e620, transaction=0x7f04bffa2978,
> pool=0x7f04c01f02b0, foreign_key=114,
> has_old_values=@0xfffffffffffffffc) at ../src/jrd/vio.cpp:1941
> #9 0x00007f04c1f76041 in check_duplicates (tdbb=0x7fff07550c60,
> record=0x7f04bfffeec8, record_idx=0x7fff0754f8b0,
> insertion=0x7fff0754f860, relation_2=0x0)
> at ../src/jrd/jrd.h:905
> #10 0x00007f04c1f75cbe in insert_key (tdbb=0x7fff07550c60,
> relation=0x7f04c02c1b58, record=0x7f04bfffeec8,
> transaction=0x7f04bffa2978, window_ptr=0x7fff0754f830,
> insertion=0x7fff0754f860, bad_relation=0x7fff0754f7e0,
> bad_index=0xfffffffffffffffc) at ../src/jrd/idx.cpp:1650
> #11 0x00007f04c1f7507a in IDX_store (tdbb=0x7fff07550c60,
> rpb=0x7f04bfffdb38, transaction=0x7f04bffa2978) at ../src/jrd/idx.cpp:1062
> #12 0x00007f04c1f66498 in store (tdbb=0x7fff07550c60,
> node=0x7f04bffab758, which_trig=0) at ../src/jrd/exe.cpp:3685
> #13 0x00007f04c1f64295 in looper (tdbb=0x7fff07550c60,
> request=0x7f04bfffd708, in_node=0x7f04c1f64268) at ../src/jrd/exe.cpp:2635
> #14 0x00007f04c1f63016 in execute_looper (tdbb=0x7fff07550c60,
> request=0x7f04bfffd708, transaction=0x7f04bffa2978,
> next_state=jrd_req::req_proceed) at ../src/jrd/exe.cpp:1443
> #15 0x00007f04c1f627a0 in EXE_send (tdbb=0x7fff07550c60,
> request=0x7f04bfffd708, msg=6, length=80, buffer=0x7f04bffa6558 "\t") at
> ../src/jrd/exe.cpp:993
> #16 0x00007f04c1f96d04 in jrd8_start_and_send
> (user_status=0x7fff07551380, req_handle=0x7f04bfffd708,
> tra_handle=0x7f04bffa2978, msg_type=0, msg_length=40,
> msg=0x7f04c007df98 "", level=0) at ../src/jrd/jrd.cpp:3789
> #17 0x00007f04c1eb8153 in isc_start_and_send (user_status=0x218005,
> req_handle=0x7f04c05695a8, tra_handle=0x7fff07551380, msg_type=0,
> msg_length=40, msg=0x7f04c007df98 "",
> level=0) at ../src/jrd/why.cpp:450
> #18 0x00007f04c203014f in execute_request (request=0x7f04c007eda8,
> trans_handle=0x7f04c007e8c8, in_blr_length=20, in_blr=0x7f04c0567aa0
> "\005\002\004", in_msg_length=34,
> in_msg=0x7f04c0569368 "", out_blr_length=0, out_blr=0x0,
> out_msg_length=0, out_msg=0x0, singleton=false) at ../src/dsql/dsql.cpp:3397
> #19 0x00007f04c202df7f in GDS_DSQL_EXECUTE_CPP (user_status=0x218005,
> trans_handle=0x7fff0755137c, req_handle=0x6c0a, in_blr_length=20,
> in_blr=0x7f04c0567aa0 "\005\002\004",
> in_msg_type=0, in_msg_length=34, in_msg=0x7f04c0569368 "",
> out_blr_length=0, out_blr=0xfffffffffffffffc <Address 0xfffffffffffffffc
> out of bounds>, out_msg_type=0,
> out_msg_length=0, out_msg=0xfffffffffffffffc <Address
> 0xfffffffffffffffc out of bounds>) at ../src/dsql/dsql.cpp:561
> #20 0x00007f04c1eb1688 in isc_dsql_execute2_m (user_status=0x218005,
> tra_handle=0x7fff0755137c, stmt_handle=0x7f04c05693d8, in_blr_length=20,
> in_blr=0x7f04c0567aa0 "\005\002\004", in_msg_type=0,
> in_msg_length=34, in_msg=0x7f04c0569368 "", out_blr_length=34,
> out_blr=0x0, out_msg_type=37736, out_msg_length=0,
> out_msg=0x0) at ../src/jrd/why.cpp:450
> #21 0x00007f04c208f2b3 in rem_port::execute_statement
> (this=0x7f04c2311080, op=op_execute, sqldata=0x218005,
> sendL=0x7fff07551c30) at ../src/remote/server.cpp:2180
> #22 0x00007f04c20916b4 in process_packet2 (port=0x7f04c2311080,
> sendL=0x7fff07551c30, receive=0x7fff075517d0, result=0x7fff075517c8) at
> ../src/remote/server.cpp:3644
> #23 0x00007f04c20910d2 in process_packet (port=0x7f04c2311080,
> sendL=0x7fff0754dcf0, receive=0x1, result=0x7fff075517c8) at
> ../src/remote/server.cpp:3393
> #24 0x00007f04c208d841 in SRVR_main (main_port=0x7f04c2311080,
> flags=56560) at ../src/remote/server.cpp:294
> #25 0x00007f04c208d2ee in server_main (argc=2195461,
> argv=0x7fff075523f0) at ../src/remote/inet_server.cpp:450
> #26 0x0000000000400709 in main (argc=2195461, argv=0x7fff0754dcf0) at
> ../src/remote/server_stub.cpp:12
>
>
>
> Attaching to process 17483
> Reading symbols from /opt/firebird/bin/fb_inet_server...Reading symbols
> from /opt/firebird/bin/.debug/fb_inet_server.debug...done.
> done.
> Reading symbols from /opt/firebird/lib/libfbembed.so.2.1...Reading
> symbols from /opt/firebird/lib/.debug/libfbembed.so.2.1.6.debug...done.
> done.
> Loaded symbols for /opt/firebird/lib/libfbembed.so.2.1
> Reading symbols from /lib64/libdl.so.2...Reading symbols from
> /usr/lib/debug/lib64/libdl-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libdl.so.2
> Reading symbols from /lib64/libncurses.so.5...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libncurses.so.5
> Reading symbols from /usr/lib64/libstdc++.so.5...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libstdc++.so.5
> Reading symbols from /lib64/libm.so.6...Reading symbols from
> /usr/lib/debug/lib64/libm-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libm.so.6
> Reading symbols from /lib64/libgcc_s.so.1...Reading symbols from
> /usr/lib/debug/lib64/libgcc_s-4.4.7-20120601.so.1.debug...done.
> done.
> Loaded symbols for /lib64/libgcc_s.so.1
> Reading symbols from /lib64/libc.so.6...Reading symbols from
> /usr/lib/debug/lib64/libc-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /opt/firebird/lib/libicuuc.so.30...Reading symbols
> from /opt/firebird/lib/.debug/libicuuc.so.30.0.debug...done.
> done.
> Loaded symbols for /opt/firebird/lib/libicuuc.so.30
> Reading symbols from /opt/firebird/lib/libicudata.so.30...Reading
> symbols from /opt/firebird/lib/.debug/libicudata.so.30.0.debug...done.
> done.
> Loaded symbols for /opt/firebird/lib/libicudata.so.30
> Reading symbols from /opt/firebird/lib/libicui18n.so.30...Reading
> symbols from /opt/firebird/lib/.debug/libicui18n.so.30.0.debug...done.
> done.
> Loaded symbols for /opt/firebird/lib/libicui18n.so.30
> Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from
> /usr/lib/debug/lib64/ld-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libtinfo.so.5
> Reading symbols from /lib64/libpthread.so.0...Reading symbols from
> /usr/lib/debug/lib64/libpthread-2.12.so.debug...done.
> [Thread debugging using libthread_db enabled]
> done.
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from /opt/firebird/intl/fbintl...Reading symbols from
> /opt/firebird/intl/.debug/fbintl.debug...done.
> done.
> Loaded symbols for /opt/firebird/intl/fbintl
> 0x00007f1b479a3187 in semop () at ../sysdeps/unix/syscall-template.S:82
> 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> Missing separate debuginfos, use: debuginfo-install
> compat-libstdc++-33-3.2.3-69.el6.x86_64
> ncurses-libs-5.7-3.20090208.el6.x86_64
> (gdb) bt
> #0 0x00007f1b479a3187 in semop () at ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f1b4882fdee in ISC_event_wait (count=1,
> events=0x7fff989956b0, values=0x7fff989956bc, micro_seconds=10000000,
> timeout_handler=
> 0x7f1b489f1510 <lock_alarm_handler(void*)>,
> handler_arg=0x7f1b46ed0e04) at ../src/jrd/isc_sync.cpp:4583
> #2 0x00007f1b489f2cad in wait_for_request (request=0x7f1b467223f8,
> lck_wait=1, status_vector=0x0) at ../src/lock/lock.cpp:5059
> #3 0x00007f1b489f128e in grant_or_que (request=0x7f1b467223f8,
> lock=0x4233f8, lck_wait=1) at ../src/lock/lock.cpp:3006
> #4 0x00007f1b489ee75a in LOCK_enq (prior_request=1181885432,
> parent_request=1180894784, series=4, value=0x7fff98995910 "r\267\b\001",
> length=4, type=3 '\003',
> ast_routine=0xfffffffffffffffc, ast_argument=0xfffffffffffffffc,
> data=0, lck_wait=1, status_vector=0x7fff98995a70,
> owner_offset=1177546752) at ../src/lock/lock.cpp:567
> #5 0x00007f1b4892a801 in LCK_lock (tdbb=0x0, lock=0x7fff98995880,
> level=3, wait=1) at ../src/jrd/lck.cpp:1084
> #6 0x00007f1b4892a900 in LCK_lock_non_blocking (tdbb=0x218005,
> lock=0x7fff98995880, level=3, wait=1) at ../src/jrd/lck.cpp:810
> #7 0x00007f1b4897a605 in TRA_wait (tdbb=0x7fff98998340,
> trans=0x7f1b46a8dcd8, number=17348466, wait=jrd_tra::tra_wait) at
> ../src/jrd/tra.cpp:2099
> #8 0x00007f1b48983071 in VIO_get_current (tdbb=0x7fff98998340,
> old_rpb=0x7fff989953d0, rpb=0x7fff98995d00, transaction=0x7f1b46a8dcd8,
> pool=0x7f1b46a8df90, foreign_key=114,
> has_old_values=@0xfffffffffffffffc) at ../src/jrd/vio.cpp:1941
> #9 0x00007f1b48901041 in check_duplicates (tdbb=0x7fff98998340,
> record=0x7f1b46ad6a90, record_idx=0x7fff98996f90,
> insertion=0x7fff98996f40, relation_2=0x0)
> at ../src/jrd/jrd.h:905
> #10 0x00007f1b48900cbe in insert_key (tdbb=0x7fff98998340,
> relation=0x7f1b46c4cb58, record=0x7f1b46ad6a90,
> transaction=0x7f1b46a8dcd8, window_ptr=0x7fff98996f10,
> insertion=0x7fff98996f40, bad_relation=0x7fff98996ec0,
> bad_index=0xfffffffffffffffc) at ../src/jrd/idx.cpp:1650
> #11 0x00007f1b4890007a in IDX_store (tdbb=0x7fff98998340,
> rpb=0x7f1b46a94998, transaction=0x7f1b46a8dcd8) at ../src/jrd/idx.cpp:1062
> #12 0x00007f1b488f1498 in store (tdbb=0x7fff98998340,
> node=0x7f1b46add278, which_trig=0) at ../src/jrd/exe.cpp:3685
> #13 0x00007f1b488ef295 in looper (tdbb=0x7fff98998340,
> request=0x7f1b46a94568, in_node=0x7f1b488ef268) at ../src/jrd/exe.cpp:2635
> #14 0x00007f1b488ee016 in execute_looper (tdbb=0x7fff98998340,
> request=0x7f1b46a94568, transaction=0x7f1b46a8dcd8,
> next_state=jrd_req::req_proceed) at ../src/jrd/exe.cpp:1443
> #15 0x00007f1b488ed7a0 in EXE_send (tdbb=0x7fff98998340,
> request=0x7f1b46a94568, msg=6, length=80, buffer=0x7f1b46a8d170 "\t") at
> ../src/jrd/exe.cpp:993
> #16 0x00007f1b48921d04 in jrd8_start_and_send
> (user_status=0x7fff98998a60, req_handle=0x7f1b46a94568,
> tra_handle=0x7f1b46a8dcd8, msg_type=0, msg_length=40,
> msg=0x7f1b46928f98 "", level=0) at ../src/jrd/jrd.cpp:3789
> #17 0x00007f1b48843153 in isc_start_and_send (user_status=0x218005,
> req_handle=0x7f1b46ef45a8, tra_handle=0x7fff98998a60, msg_type=0,
> msg_length=40, msg=0x7f1b46928f98 "",
> level=0) at ../src/jrd/why.cpp:450
> #18 0x00007f1b489bb14f in execute_request (request=0x7f1b46929da8,
> trans_handle=0x7f1b469298c8, in_blr_length=20, in_blr=0x7f1b46ef2a50
> "\005\002\004", in_msg_length=34,
> in_msg=0x7f1b46ef4068 "", out_blr_length=0, out_blr=0x0,
> out_msg_length=0, out_msg=0x0, singleton=false) at ../src/dsql/dsql.cpp:3397
> #19 0x00007f1b489b8f7f in GDS_DSQL_EXECUTE_CPP (user_status=0x218005,
> trans_handle=0x7fff98998a5c, req_handle=0xad18, in_blr_length=20,
> in_blr=0x7f1b46ef2a50 "\005\002\004",
> in_msg_type=0, in_msg_length=34, in_msg=0x7f1b46ef4068 "",
> out_blr_length=0, out_blr=0xfffffffffffffffc <Address 0xfffffffffffffffc
> out of bounds>, out_msg_type=0,
> out_msg_length=0, out_msg=0xfffffffffffffffc <Address
> 0xfffffffffffffffc out of bounds>) at ../src/dsql/dsql.cpp:561
> #20 0x00007f1b4883c688 in isc_dsql_execute2_m (user_status=0x218005,
> tra_handle=0x7fff98998a5c, stmt_handle=0x7f1b46ef4508, in_blr_length=20,
> in_blr=0x7f1b46ef2a50 "\005\002\004", in_msg_type=0,
> in_msg_length=34, in_msg=0x7f1b46ef4068 "", out_blr_length=34,
> out_blr=0x0, out_msg_type=16488, out_msg_length=0,
> out_msg=0x0) at ../src/jrd/why.cpp:450
> #21 0x00007f1b48a1a2b3 in rem_port::execute_statement
> (this=0x7f1b48c9c080, op=op_execute, sqldata=0x218005,
> sendL=0x7fff98999310) at ../src/remote/server.cpp:2180
> #22 0x00007f1b48a1c6b4 in process_packet2 (port=0x7f1b48c9c080,
> sendL=0x7fff98999310, receive=0x7fff98998eb0, result=0x7fff98998ea8) at
> ../src/remote/server.cpp:3644
> #23 0x00007f1b48a1c0d2 in process_packet (port=0x7f1b48c9c080,
> sendL=0x7fff989953d0, receive=0x1, result=0x7fff98998ea8) at
> ../src/remote/server.cpp:3393
> #24 0x00007f1b48a18841 in SRVR_main (main_port=0x7f1b48c9c080,
> flags=21456) at ../src/remote/server.cpp:294
> #25 0x00007f1b48a182ee in server_main (argc=2195461,
> argv=0x7fff98999ad0) at ../src/remote/inet_server.cpp:450
> #26 0x0000000000400709 in main (argc=2195461, argv=0x7fff989953d0) at
> ../src/remote/server_stub.cpp:12
>
>
> [root@provisioning ~]# ps ax |grep fb_in
> 14830 ? Ss 0:46 fb_inet_server
> 16275 ? Ss 0:21 fb_inet_server
> 16276 ? Ss 0:17 fb_inet_server
> 17159 ? Ss 0:05 fb_inet_server
> 17483 ? Ss 0:07 fb_inet_server
> 17484 ? Ss 0:20 fb_inet_server
> 17950 ? Ss 0:01 fb_inet_server
> 18564 ? Ss 0:02 fb_inet_server
> 20447 pts/1 S+ 0:00 grep fb_in
> [root@provisioning ~]#
>
>
> LOCK_HEADER BLOCK
> Version: 144, Active owner: 0, Length: 6343488, Used: 6320424
> Lock manager pid: 435
> Semmask: 0x2A320, Flags: 0x0001
> Enqs: 178045374, Converts: 2877880, Rejects: 2021295, Blocks:
> 10077157
> Deadlock scans: 7, Deadlocks: 0, Scan interval: 10
> Acquires: 263029525, Acquire blocks: 2825140, Spin count: 0
> Mutex wait: 1.1%
> Hash slots: 20011, Hash lengths (min/avg/max): 0/ 1/ 6
> Remove node: 0, Insert queue: 0, Insert prior: 0
> Owners (10): forward: 173984, backward: 202488
> Free owners (39): forward: 4557136, backward: 6320056
> Free locks (12468): forward: 654352, backward: 1970488
> Free requests (27976): forward: 5451080, backward: 4703800
> Lock Ordering: Enabled
>
> OWNER BLOCK 173984
> Owner id: 435, type: 1, flags: 0x04, pending: 0,
> semid: 1
> Process id: 435, UID: 0x0 Alive
> Flags: 0x04
> Requests: *empty*
> Blocks: *empty*
>
> OWNER BLOCK 263312
> Owner id: 14830, type: 3, flags: 0x20, pending: 0,
> semid: 9 (available)
> Process id: 14830, UID: 0x54 Alive
> Flags: 0x20 wake
> Requests (4368): forward: 3069384, backward: 6187376
> Blocks: *empty*
>
> OWNER BLOCK 6319536
> Owner id: 16275, type: 3, flags: 0x02, pending: 4455800,
> semid: 10
> Process id: 16275, UID: 0x54 Alive
> Flags: 0x02 scan
> Requests (3687): forward: 4236408, backward: 4455800
> Blocks: *empty*
>
> OWNER BLOCK 6319952
> Owner id: 16276, type: 3, flags: 0x02, pending: 5252808,
> semid: 8
> Process id: 16276, UID: 0x54 Alive
> Flags: 0x02 scan
> Requests (4239): forward: 2304312, backward: 5252808
> Blocks: *empty*
>
> OWNER BLOCK 172856
> Owner id: 17159, type: 3, flags: 0x02, pending: 5878512,
> semid: 4
> Process id: 17159, UID: 0x54 Alive
> Flags: 0x02 scan
> Requests (3646): forward: 5991664, backward: 5878512
> Blocks: *empty*
>
> OWNER BLOCK 298416
> Owner id: 17483, type: 3, flags: 0x02, pending: 4338680,
> semid: 15
> Process id: 17483, UID: 0x54 Alive
> Flags: 0x02 scan
> Requests (4233): forward: 5393480, backward: 4338680
> Blocks: *empty*
>
> OWNER BLOCK 3289272
> Owner id: 17484, type: 3, flags: 0x02, pending: 5772232,
> semid: 2
> Process id: 17484, UID: 0x54 Alive
> Flags: 0x02 scan
> Requests (3950): forward: 2876824, backward: 5772232
> Blocks: *empty*
>
> OWNER BLOCK 5860872
> Owner id: 17950, type: 3, flags: 0x02, pending: 5143560,
> semid: 5
> Process id: 17950, UID: 0x54 Alive
> Flags: 0x02 scan
> Requests (4016): forward: 2903064, backward: 5143560
> Blocks: *empty*
>
> OWNER BLOCK 338592
> Owner id: 18564, type: 3, flags: 0x02, pending: 4223480,
> semid: 25
> Process id: 18564, UID: 0x54 Alive
> Flags: 0x02 scan
> Requests (4390): forward: 1977400, backward: 4223480
> Blocks: *empty*
>
> OWNER BLOCK 202488
> Owner id: 19980, type: 3, flags: 0x00, pending: 0,
> semid: 3 (available)
> Process id: 19980, UID: 0x54 Alive
> Flags: 0x00
> Requests (140): forward: 2053880, backward: 1340344
> Blocks: *empty*
>
> Event log:
> DEL_OWNER: owner = 4557032, lock = 4557032, request = 0
> DEL_OWNER: owner = 217312, lock = 217312, request = 0
> DEL_OWNER: owner = 280800, lock = 280800, request = 0
> DEL_OWNER: owner = 356352, lock = 356352, request = 0
> DEL_OWNER: owner = 274872, lock = 274872, request = 0
> DEL_OWNER: owner = 310792, lock = 310792, request = 0
> DEL_OWNER: owner = 6319536, lock = 6319536, request = 0
> DEL_OWNER: owner = 6319952, lock = 6319952, request = 0
> DEL_OWNER: owner = 3289272, lock = 3289272, request = 0
> DEL_OWNER: owner = 4557136, lock = 4557136, request = 0
> DEL_OWNER: owner = 6318808, lock = 6318808, request = 0
> DEL_OWNER: owner = 234624, lock = 234624, request = 0
> DEL_OWNER: owner = 309976, lock = 309976, request = 0
> DEL_OWNER: owner = 6318704, lock = 6318704, request = 0
> DEL_OWNER: owner = 174912, lock = 174912, request = 0
> DEL_OWNER: owner = 338592, lock = 338592, request = 0
> DEL_OWNER: owner = 6319744, lock = 6319744, request = 0
> DEL_OWNER: owner = 6319848, lock = 6319848, request = 0
> DEL_OWNER: owner = 241064, lock = 241064, request = 0
> DEL_OWNER: owner = 6319640, lock = 6319640, request = 0
> DEL_OWNER: owner = 6319432, lock = 6319432, request = 0
> DEL_OWNER: owner = 172856, lock = 172856, request = 0
> DEL_OWNER: owner = 4557240, lock = 4557240, request = 0
> DEL_OWNER: owner = 6319224, lock = 6319224, request = 0
> DEL_OWNER: owner = 247056, lock = 247056, request = 0
> DEL_OWNER: owner = 344688, lock = 344688, request = 0
> DEL_OWNER: owner = 217312, lock = 217312, request = 0
> DEL_OWNER: owner = 356352, lock = 356352, request = 0
> DEL_OWNER: owner = 310792, lock = 310792, request = 0
> DEL_OWNER: owner = 6319952, lock = 6319952, request = 0
> DEL_OWNER: owner = 4557136, lock = 4557136, request = 0
> DEL_OWNER: owner = 6319328, lock = 6319328, request = 0
> DEL_OWNER: owner = 350488, lock = 350488, request = 0
> DEL_OWNER: owner = 6318912, lock = 6318912, request = 0
> DEL_OWNER: owner = 341640, lock = 341640, request = 0
> DEL_OWNER: owner = 369080, lock = 369080, request = 0
> DEL_OWNER: owner = 298416, lock = 298416, request = 0
> DEL_OWNER: owner = 263312, lock = 263312, request = 0
> DEL_OWNER: owner = 234624, lock = 234624, request = 0
> DEL_OWNER: owner = 208368, lock = 208368, request = 0
> DEL_OWNER: owner = 5860872, lock = 5860872, request = 0
> DEL_OWNER: owner = 311160, lock = 311160, request = 0
> DEL_OWNER: owner = 292424, lock = 292424, request = 0
> DEL_OWNER: owner = 6318704, lock = 6318704, request = 0
> DEL_OWNER: owner = 338592, lock = 338592, request = 0
> DEL_OWNER: owner = 6319848, lock = 6319848, request = 0
> DEL_OWNER: owner = 6319640, lock = 6319640, request = 0
> DEL_OWNER: owner = 172856, lock = 172856, request = 0
> DEL_OWNER: owner = 6319224, lock = 6319224, request = 0
> DEL_OWNER: owner = 344688, lock = 344688, request = 0
> DEL_OWNER: owner = 356352, lock = 356352, request = 0
> DEL_OWNER: owner = 6319952, lock = 6319952, request = 0
> DEL_OWNER: owner = 6319328, lock = 6319328, request = 0
> DEL_OWNER: owner = 218512, lock = 218512, request = 0
> DEL_OWNER: owner = 326048, lock = 326048, request = 0
> DEL_OWNER: owner = 211160, lock = 211160, request = 0
> DEL_OWNER: owner = 202488, lock = 202488, request = 0
> DEL_OWNER: owner = 4557032, lock = 4557032, request = 0
> DEL_OWNER: owner = 6318912, lock = 6318912, request = 0
> DEL_OWNER: owner = 369080, lock = 369080, request = 0
> DEL_OWNER: owner = 263312, lock = 263312, request = 0
> DEL_OWNER: owner = 208368, lock = 208368, request = 0
> DEL_OWNER: owner = 311160, lock = 311160, request = 0
> DEL_OWNER: owner = 6319016, lock = 6319016, request = 0
> DEL_OWNER: owner = 6318600, lock = 6318600, request = 0
> DEL_OWNER: owner = 274872, lock = 274872, request = 0
> DEL_OWNER: owner = 280800, lock = 280800, request = 0
> DEL_OWNER: owner = 3289272, lock = 3289272, request = 0
> DEL_OWNER: owner = 6318704, lock = 6318704, request = 0
> DEL_OWNER: owner = 6319848, lock = 6319848, request = 0
> DEL_OWNER: owner = 172856, lock = 172856, request = 0
> DEL_OWNER: owner = 344688, lock = 344688, request = 0
> DEL_OWNER: owner = 6319952, lock = 6319952, request = 0
> DEL_OWNER: owner = 218512, lock = 218512, request = 0
> DEL_OWNER: owner = 326048, lock = 326048, request = 0
> DEL_OWNER: owner = 211160, lock = 211160, request = 0
> DEL_OWNER: owner = 174912, lock = 174912, request = 0
> DEL_OWNER: owner = 362280, lock = 362280, request = 0
> DEL_OWNER: owner = 6319744, lock = 6319744, request = 0
> DEL_OWNER: owner = 6319120, lock = 6319120, request = 0
> DEL_OWNER: owner = 6319432, lock = 6319432, request = 0
> DEL_OWNER: owner = 4557240, lock = 4557240, request = 0
> DEL_OWNER: owner = 247056, lock = 247056, request = 0
> DEL_OWNER: owner = 217312, lock = 217312, request = 0
> DEL_OWNER: owner = 310792, lock = 310792, request = 0
> DEL_OWNER: owner = 6318808, lock = 6318808, request = 0
> DEL_OWNER: owner = 4557136, lock = 4557136, request = 0
> DEL_OWNER: owner = 350488, lock = 350488, request = 0
> DEL_OWNER: owner = 4557032, lock = 4557032, request = 0
> DEL_OWNER: owner = 369080, lock = 369080, request = 0
> DEL_OWNER: owner = 208368, lock = 208368, request = 0
> DEL_OWNER: owner = 6319016, lock = 6319016, request = 0
> DEL_OWNER: owner = 274872, lock = 274872, request = 0
> DEL_OWNER: owner = 3289272, lock = 3289272, request = 0
> DEL_OWNER: owner = 6319848, lock = 6319848, request = 0
> DEL_OWNER: owner = 344688, lock = 344688, request = 0
> DEL_OWNER: owner = 218512, lock = 218512, request = 0
> DEL_OWNER: owner = 211160, lock = 211160, request = 0
> DEL_OWNER: owner = 362280, lock = 362280, request = 0
> DEL_OWNER: owner = 6319120, lock = 6319120, request = 0
> DEL_OWNER: owner = 202488, lock = 202488, request = 0
> DEL_OWNER: owner = 341640, lock = 341640, request = 0
> DEL_OWNER: owner = 309976, lock = 309976, request = 0
> DEL_OWNER: owner = 6319536, lock = 6319536, request = 0
> DEL_OWNER: owner = 6320056, lock = 6320056, request = 0
> DEL_OWNER: owner = 241064, lock = 241064, request = 0
> DEL_OWNER: owner = 5860872, lock = 5860872, request = 0
> DEL_OWNER: owner = 4557240, lock = 4557240, request = 0
> DEL_OWNER: owner = 217312, lock = 217312, request = 0
> DEL_OWNER: owner = 6318808, lock = 6318808, request = 0
> DEL_OWNER: owner = 350488, lock = 350488, request = 0
> DEL_OWNER: owner = 369080, lock = 369080, request = 0
> DEL_OWNER: owner = 6319016, lock = 6319016, request = 0
> DEL_OWNER: owner = 338592, lock = 338592, request = 0
> DEL_OWNER: owner = 298416, lock = 298416, request = 0
> DEL_OWNER: owner = 356352, lock = 356352, request = 0
> DEL_OWNER: owner = 6319328, lock = 6319328, request = 0
> DEL_OWNER: owner = 292424, lock = 292424, request = 0
> DEL_OWNER: owner = 3289272, lock = 3289272, request = 0
> DEL_OWNER: owner = 344688, lock = 344688, request = 0
> DEL_OWNER: owner = 211160, lock = 211160, request = 0
> DEL_OWNER: owner = 6319120, lock = 6319120, request = 0
> DEL_OWNER: owner = 341640, lock = 341640, request = 0
> DEL_OWNER: owner = 6318912, lock = 6318912, request = 0
> DEL_OWNER: owner = 6319640, lock = 6319640, request = 0
> DEL_OWNER: owner = 6318600, lock = 6318600, request = 0
> DEL_OWNER: owner = 280800, lock = 280800, request = 0
> DEL_OWNER: owner = 6318704, lock = 6318704, request = 0
> DEL_OWNER: owner = 6319536, lock = 6319536, request = 0
> DEL_OWNER: owner = 172856, lock = 172856, request = 0
> DEL_OWNER: owner = 6319952, lock = 6319952, request = 0
> DEL_OWNER: owner = 326048, lock = 326048, request = 0
> DEL_OWNER: owner = 174912, lock = 174912, request = 0
> DEL_OWNER: owner = 6319432, lock = 6319432, request = 0
> DEL_OWNER: owner = 241064, lock = 241064, request = 0
> DEL_OWNER: owner = 4557240, lock = 4557240, request = 0
> DEL_OWNER: owner = 6318808, lock = 6318808, request = 0
> DEL_OWNER: owner = 369080, lock = 369080, request = 0
> DEL_OWNER: owner = 338592, lock = 338592, request = 0
> DEL_OWNER: owner = 356352, lock = 356352, request = 0
> DEL_OWNER: owner = 292424, lock = 292424, request = 0
> DEL_OWNER: owner = 344688, lock = 344688, request = 0
> DEL_OWNER: owner = 6319120, lock = 6319120, request = 0
> DEL_OWNER: owner = 247056, lock = 247056, request = 0
> DEL_OWNER: owner = 310792, lock = 310792, request = 0
> DEL_OWNER: owner = 311160, lock = 311160, request = 0
> DEL_OWNER: owner = 4557136, lock = 4557136, request = 0
> DEL_OWNER: owner = 234624, lock = 234624, request = 0
> DEL_OWNER: owner = 208368, lock = 208368, request = 0
> DEL_OWNER: owner = 6319744, lock = 6319744, request = 0
> DEL_OWNER: owner = 274872, lock = 274872, request = 0
> DEL_OWNER: owner = 6318912, lock = 6318912, request = 0
> DEL_OWNER: owner = 6318600, lock = 6318600, request = 0
> DEL_OWNER: owner = 6318704, lock = 6318704, request = 0
> DEL_OWNER: owner = 172856, lock = 172856, request = 0
> DEL_OWNER: owner = 326048, lock = 326048, request = 0
> DEL_OWNER: owner = 6319432, lock = 6319432, request = 0
> DEL_OWNER: owner = 4557240, lock = 4557240, request = 0
> DEL_OWNER: owner = 369080, lock = 369080, request = 0
> DEL_OWNER: owner = 6319848, lock = 6319848, request = 0
> DEL_OWNER: owner = 6320056, lock = 6320056, request = 0
> DEL_OWNER: owner = 4557032, lock = 4557032, request = 0
> DEL_OWNER: owner = 202488, lock = 202488, request = 0
> DEL_OWNER: owner = 356352, lock = 356352, request = 0
> DEL_OWNER: owner = 344688, lock = 344688, request = 0
> DEL_OWNER: owner = 247056, lock = 247056, request = 0
> DEL_OWNER: owner = 311160, lock = 311160, request = 0
> DEL_OWNER: owner = 5860872, lock = 5860872, request = 0
> DEL_OWNER: owner = 362280, lock = 362280, request = 0
> DEL_OWNER: owner = 218512, lock = 218512, request = 0
> DEL_OWNER: owner = 6319224, lock = 6319224, request = 0
> DEL_OWNER: owner = 350488, lock = 350488, request = 0
> DEL_OWNER: owner = 6319016, lock = 6319016, request = 0
> DEL_OWNER: owner = 234624, lock = 234624, request = 0
> DEL_OWNER: owner = 298416, lock = 298416, request = 0
> DEL_OWNER: owner = 6319328, lock = 6319328, request = 0
> DEL_OWNER: owner = 3289272, lock = 3289272, request = 0
> DEL_OWNER: owner = 309976, lock = 309976, request = 0
> DEL_OWNER: owner = 341640, lock = 341640, request = 0
> DEL_OWNER: owner = 6319744, lock = 6319744, request = 0
> DEL_OWNER: owner = 6318912, lock = 6318912, request = 0
> DEL_OWNER: owner = 6318704, lock = 6318704, request = 0
> DEL_OWNER: owner = 326048, lock = 326048, request = 0
> DEL_OWNER: owner = 4557240, lock = 4557240, request = 0
> DEL_OWNER: owner = 6319848, lock = 6319848, request = 0
> DEL_OWNER: owner = 4557032, lock = 4557032, request = 0
> DEL_OWNER: owner = 356352, lock = 356352, request = 0
> DEL_OWNER: owner = 247056, lock = 247056, request = 0
> DEL_OWNER: owner = 5860872, lock = 5860872, request = 0
> DEL_OWNER: owner = 280800, lock = 280800, request = 0
> DEL_OWNER: owner = 6319640, lock = 6319640, request = 0
> DEL_OWNER: owner = 241064, lock = 241064, request = 0
> DEL_OWNER: owner = 217312, lock = 217312, request = 0
> DEL_OWNER: owner = 211160, lock = 211160, request = 0
> DEL_OWNER: owner = 338592, lock = 338592, request = 0
> DEL_OWNER: owner = 218512, lock = 218512, request = 0
> DEL_OWNER: owner = 350488, lock = 350488, request = 0
> DEL_OWNER: owner = 234624, lock = 234624, request = 0
> DEL_OWNER: owner = 6319328, lock = 6319328, request = 0
> DEL_OWNER: owner = 309976, lock = 309976, request = 0
> DEL_OWNER: owner = 6319744, lock = 6319744, request = 0
> DEL_OWNER: owner = 6318704, lock = 6318704, request = 0
> DEL_OWNER: owner = 326048, lock = 326048, request = 0
> DEL_OWNER: owner = 4557240, lock = 4557240, request = 0
> DEL_OWNER: owner = 310792, lock = 310792, request = 0
> DEL_OWNER: owner = 174912, lock = 174912, request = 0
> DEL_OWNER: owner = 6318808, lock = 6318808, request = 0
> DEL_OWNER: owner = 4557032, lock = 4557032, request = 0
> DEL_OWNER: owner = 247056, lock = 247056, request = 0
> DEL_OWNER: owner = 280800, lock = 280800, request = 0
> DEL_OWNER: owner = 6319848, lock = 6319848, request = 0
> DEL_OWNER: owner = 241064, lock = 241064, request = 0
> DEL_OWNER: owner = 211160, lock = 211160, request = 0
> DEL_OWNER: owner = 6319432, lock = 6319432, request = 0
> DEL_OWNER: owner = 369080, lock = 369080, request = 0
> DEL_OWNER: owner = 6320056, lock = 6320056, request = 0
> DEL_OWNER: owner = 202488, lock = 202488, request = 0
> DEL_OWNER: owner = 4557136, lock = 4557136, request = 0
> DEL_OWNER: owner = 311160, lock = 311160, request = 0
> DEL_OWNER: owner = 362280, lock = 362280, request = 0
> DEL_OWNER: owner = 218512, lock = 218512, request = 0
> DEL_OWNER: owner = 234624, lock = 234624, request = 0
> DEL_OWNER: owner = 309976, lock = 309976, request = 0
> DEL_OWNER: owner = 6318704, lock = 6318704, request = 0
> DEL_OWNER: owner = 4557240, lock = 4557240, request = 0
> DEL_OWNER: owner = 174912, lock = 174912, request = 0
> DEL_OWNER: owner = 4557032, lock = 4557032, request = 0
> DEL_OWNER: owner = 280800, lock = 280800, request = 0
> DEL_OWNER: owner = 344688, lock = 344688, request = 0
> DEL_OWNER: owner = 341640, lock = 341640, request = 0
> DEL_OWNER: owner = 241064, lock = 241064, request = 0
> DEL_OWNER: owner = 6319432, lock = 6319432, request = 0
> DEL_OWNER: owner = 6319848, lock = 6319848, request = 0
> DEL_OWNER: owner = 274872, lock = 274872, request = 0
> DEL_OWNER: owner = 217312, lock = 217312, request = 0
> DEL_OWNER: owner = 356352, lock = 356352, request = 0
> DEL_OWNER: owner = 6318600, lock = 6318600, request = 0
> DEL_OWNER: owner = 247056, lock = 247056, request = 0
> DEL_OWNER: owner = 377088, lock = 377088, request = 0
> DEL_OWNER: owner = 368272, lock = 368272, request = 0
> DEL_OWNER: owner = 350488, lock = 350488, request = 0
> DEL_OWNER: owner = 6319328, lock = 6319328, request = 0
> DEL_OWNER: owner = 6319744, lock = 6319744, request = 0
> DEL_OWNER: owner = 326048, lock = 326048, request = 0
> DEL_OWNER: owner = 211160, lock = 211160, request = 0
> DEL_OWNER: owner = 6319016, lock = 6319016, request = 0
> DEL_OWNER: owner = 6319120, lock = 6319120, request = 0
> DEL_OWNER: owner = 6319224, lock = 6319224, request = 0
> DEL_OWNER: owner = 208368, lock = 208368, request = 0
> DEL_OWNER: owner = 292424, lock = 292424, request = 0
> DEL_OWNER: owner = 6318912, lock = 6318912, request = 0
> DEL_OWNER: owner = 6319640, lock = 6319640, request = 0
> DEL_OWNER: owner = 6318808, lock = 6318808, request = 0
> DEL_OWNER: owner = 310792, lock = 310792, request = 0
> DEL_OWNER: owner = 369080, lock = 369080, request = 0
> DEL_OWNER: owner = 6320056, lock = 6320056, request = 0
>
>
> MON$ATTACHMENT_ID MON$SERVER_PID MON$STATE MON$TIMESTAMP
> MON$STAT_ID MON$TIMESTAMP_1 MON$SQL_TEXT
> ----------------------------------------------------------------------------------------------------------
> 14786 16275 1 18.9.2014 г. 07:35:39
> 113 18.9.2014 г. 08:05:00 INSERT INTO CASH_CASHBOX_DAY_AMMOUNT
> (CASHBOXID, DATE_TIME, AMMOUNT) VALUES (?, ?, ?)
> 14798 17483 1 18.9.2014 г. 07:58:09
> 71 18.9.2014 г. 08:05:31 INSERT INTO CASH_CASHBOX_DAY_AMMOUNT
> (CASHBOXID, DATE_TIME, AMMOUNT) VALUES (?, ?, ?)
> 14804 17950 1 18.9.2014 г. 08:05:40
> 29 18.9.2014 г. 08:09:02 INSERT INTO CASH_CASHBOX_DAY_AMMOUNT
> (CASHBOXID, DATE_TIME, AMMOUNT) VALUES (?, ?, ?)
> 14796 17159 1 18.9.2014 г. 07:50:39
> 57 18.9.2014 г. 08:12:25 INSERT INTO CASH_CASHBOX_DAY_AMMOUNT
> (CASHBOXID, DATE_TIME, AMMOUNT) VALUES (?, ?, ?)
> 14799 17484 1 18.9.2014 г. 07:58:09
> 85 18.9.2014 г. 08:19:17 INSERT INTO CASH_CASHBOX_DAY_AMMOUNT
> (CASHBOXID, DATE_TIME, AMMOUNT) VALUES (?, ?, ?)
> 14787 16276 1 18.9.2014 г. 07:35:39
> 99 18.9.2014 г. 08:20:34 INSERT INTO CASH_CASHBOX_DAY_AMMOUNT
> (CASHBOXID, DATE_TIME, AMMOUNT) VALUES (?, ?, ?)
> 14807 18564 1 18.9.2014 г. 08:12:29
> 43 18.9.2014 г. 08:21:43 INSERT INTO CASH_CASHBOX_DAY_AMMOUNT
> (CASHBOXID, DATE_TIME, AMMOUNT) VALUES (?, ?, ?)
> 14814 20475 1 18.9.2014 г. 08:37:38
> 2 18.9.2014 г. 08:37:39 select att.MON$ATTACHMENT_ID,
> att.MON$SERVER_PID, att.MON$STATE, att.MON$TIMESTAMP, att.MON$STAT_ID,
> st.MON$TIMESTAMP, MON$SQL_TEXT
> from MON$ATTACHMENTS att
> join mon$STATEMENTS st on st.MON$ATTACHMENT_ID = att.MON$ATTACHMENT_ID
> where mon$SQL_TEXT is not null
> order by st.MON$TIMESTAMP
>
> I hope this can give you an idea what is happening here. The problem now
> shows every 2-3 days on only one installation from 20.
>
> I'll try to upgrade installation to the most recent kernel available for
> CentOS 6 later today.
Mark Rotteveel