Subject | Firebird crash and dump |
---|---|
Author | Michael Vilhelmsen |
Post date | 2008-12-23T08:51:44Z |
Hi
Yesterday our firebird server died.
Its running on Debian and its Firebird 2.1.1.17910.
Its Firebird 64 bit.
The server has 12 Gb of RAM.
It happens at a time of a little above average usage.
Some 3Gb of RAM was used.
Some 50% CPU usage.
At this point CPU usage dropped to almost 0.
Nobody could connect to the server, disconnect from the server and
those running something was all frozen.
My colleague stopped firebird at this point and started it up again.
It gave a core dump file of 880 Mb.
He has ran an analyze with gdb.
This is the result.
According to him someone migth be able to tell something from this.
Dec 22 14:39:33 localhost kernel: fbserver[17402]: segfault at
00002aaab20770ea rip 00000000005148e1 rsp 00000000630447d0 error 4
En analyse med gdb siger dette:
DBSRV07:~# gdb
/root/FirebirdSS-debuginfo-2.1.1.17910-0.amd64/opt/firebird/bin/.debug/fbserver.debug
/tmp/core.11015
GNU gdb 6.4.90-debian
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "x86_64-linux-gnu"...Using host
libthread_db library "/lib/libthread_db.so.1".
warning: core file may not match specified executable file.
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libncurses.so.5...done.
Loaded symbols for /lib/libncurses.so.5
Reading symbols from /opt/firebird/lib/libicuuc.so.30...done.
Loaded symbols for /opt/firebird/lib/libicuuc.so.30
Reading symbols from /opt/firebird/lib/libicudata.so.30...done.
Loaded symbols for /opt/firebird/lib/libicudata.so.30
Reading symbols from /opt/firebird/lib/libicui18n.so.30...done.
Loaded symbols for /opt/firebird/lib/libicui18n.so.30
Reading symbols from /usr/lib/libstdc++.so.5...done.
Loaded symbols for /usr/lib/libstdc++.so.5
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libnss_compat.so.2...done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libnss_nis.so.2...done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /opt/firebird/intl/fbintl...done.
Loaded symbols for /opt/firebird/intl/fbintl
Core was generated by `/opt/firebird/bin/fbserver'.
Program terminated with signal 11, Segmentation fault.
#0 jrd8_unwind_request (user_status=0x63044a80,
req_handle=0x2aaab23eaa68, level=0) at ../src/common/classes/alloc.h:83
83 ../src/common/classes/alloc.h: No such file or directory.
in ../src/common/classes/alloc.h
(gdb) bt
#0 jrd8_unwind_request (user_status=0x63044a80,
req_handle=0x2aaab23eaa68, level=0) at ../src/common/classes/alloc.h:83
#1 0x000000000043cb23 in isc_unwind_request (user_status=0x630447d0,
req_handle=0x2aaaaf18de58, level=0) at ../src/jrd/why.cpp:446
#2 0x00000000005aae9c in close_cursor (request=0x2aaaaf18dda8) at
../src/dsql/dsql.cpp:3067
#3 0x00000000005a9806 in GDS_DSQL_FREE_CPP (user_status=0x630447d0,
req_handle=0x2aaaacc49848, option=1) at ../src/dsql/dsql.cpp:1188
#4 0x0000000000436dd0 in isc_dsql_free_statement
(user_status=0x630447d0, stmt_handle=0x2aaaab6121b8, option=1) at
../src/jrd/why.cpp:446
#5 0x0000000000411ab3 in rem_port::end_statement
(this=0x2aaaaceff740, free_stmt=0x2aaab7dc0b58, sendL=0x2aaab7dc0338)
at ../src/remote/server.cpp:1852
#6 0x000000000041498f in process_packet2 (port=0x2aaaaceff740,
sendL=0x2aaab7dc0338, receive=0x2aaab7dc0790, result=0x63045128)
at ../src/remote/server.cpp:3625
#7 0x0000000000414452 in process_packet (port=0x2aaaaceff740,
sendL=0x2aaab23eaa68, receive=0x2aaab20770f8, result=0x63045128)
at ../src/remote/server.cpp:3363
#8 0x0000000000416e25 in loopThread (flags=0x630447d0) at
../src/remote/server.cpp:5381
#9 0x000000000042d634 in threadStart (arg=0x2aaab0366bd8) at
../src/jrd/ThreadData.cpp:272
#10 0x00002ac1505e3f1a in start_thread () from /lib/libpthread.so.0
#11 0x00002ac1507bc5d2 in clone () from /lib/libc.so.6
#12 0x0000000000000000 in ?? ()
(gdb)
Is there someone who can read something from the above dump?
Michael
Yesterday our firebird server died.
Its running on Debian and its Firebird 2.1.1.17910.
Its Firebird 64 bit.
The server has 12 Gb of RAM.
It happens at a time of a little above average usage.
Some 3Gb of RAM was used.
Some 50% CPU usage.
At this point CPU usage dropped to almost 0.
Nobody could connect to the server, disconnect from the server and
those running something was all frozen.
My colleague stopped firebird at this point and started it up again.
It gave a core dump file of 880 Mb.
He has ran an analyze with gdb.
This is the result.
According to him someone migth be able to tell something from this.
Dec 22 14:39:33 localhost kernel: fbserver[17402]: segfault at
00002aaab20770ea rip 00000000005148e1 rsp 00000000630447d0 error 4
En analyse med gdb siger dette:
DBSRV07:~# gdb
/root/FirebirdSS-debuginfo-2.1.1.17910-0.amd64/opt/firebird/bin/.debug/fbserver.debug
/tmp/core.11015
GNU gdb 6.4.90-debian
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "x86_64-linux-gnu"...Using host
libthread_db library "/lib/libthread_db.so.1".
warning: core file may not match specified executable file.
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libncurses.so.5...done.
Loaded symbols for /lib/libncurses.so.5
Reading symbols from /opt/firebird/lib/libicuuc.so.30...done.
Loaded symbols for /opt/firebird/lib/libicuuc.so.30
Reading symbols from /opt/firebird/lib/libicudata.so.30...done.
Loaded symbols for /opt/firebird/lib/libicudata.so.30
Reading symbols from /opt/firebird/lib/libicui18n.so.30...done.
Loaded symbols for /opt/firebird/lib/libicui18n.so.30
Reading symbols from /usr/lib/libstdc++.so.5...done.
Loaded symbols for /usr/lib/libstdc++.so.5
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libnss_compat.so.2...done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libnss_nis.so.2...done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /opt/firebird/intl/fbintl...done.
Loaded symbols for /opt/firebird/intl/fbintl
Core was generated by `/opt/firebird/bin/fbserver'.
Program terminated with signal 11, Segmentation fault.
#0 jrd8_unwind_request (user_status=0x63044a80,
req_handle=0x2aaab23eaa68, level=0) at ../src/common/classes/alloc.h:83
83 ../src/common/classes/alloc.h: No such file or directory.
in ../src/common/classes/alloc.h
(gdb) bt
#0 jrd8_unwind_request (user_status=0x63044a80,
req_handle=0x2aaab23eaa68, level=0) at ../src/common/classes/alloc.h:83
#1 0x000000000043cb23 in isc_unwind_request (user_status=0x630447d0,
req_handle=0x2aaaaf18de58, level=0) at ../src/jrd/why.cpp:446
#2 0x00000000005aae9c in close_cursor (request=0x2aaaaf18dda8) at
../src/dsql/dsql.cpp:3067
#3 0x00000000005a9806 in GDS_DSQL_FREE_CPP (user_status=0x630447d0,
req_handle=0x2aaaacc49848, option=1) at ../src/dsql/dsql.cpp:1188
#4 0x0000000000436dd0 in isc_dsql_free_statement
(user_status=0x630447d0, stmt_handle=0x2aaaab6121b8, option=1) at
../src/jrd/why.cpp:446
#5 0x0000000000411ab3 in rem_port::end_statement
(this=0x2aaaaceff740, free_stmt=0x2aaab7dc0b58, sendL=0x2aaab7dc0338)
at ../src/remote/server.cpp:1852
#6 0x000000000041498f in process_packet2 (port=0x2aaaaceff740,
sendL=0x2aaab7dc0338, receive=0x2aaab7dc0790, result=0x63045128)
at ../src/remote/server.cpp:3625
#7 0x0000000000414452 in process_packet (port=0x2aaaaceff740,
sendL=0x2aaab23eaa68, receive=0x2aaab20770f8, result=0x63045128)
at ../src/remote/server.cpp:3363
#8 0x0000000000416e25 in loopThread (flags=0x630447d0) at
../src/remote/server.cpp:5381
#9 0x000000000042d634 in threadStart (arg=0x2aaab0366bd8) at
../src/jrd/ThreadData.cpp:272
#10 0x00002ac1505e3f1a in start_thread () from /lib/libpthread.so.0
#11 0x00002ac1507bc5d2 in clone () from /lib/libc.so.6
#12 0x0000000000000000 in ?? ()
(gdb)
Is there someone who can read something from the above dump?
Michael