Well - before we go hire someone to help us solve our problem, I will ask once more if someone has a solution to our problem.

Firebird 2.1.3 SS running on a Debian Linux 64 bits 12 Gb RAM.
Database is some 25 Gb.
There are up to 100 active connections.

Every once in a while Firebirds stops.
It does not die - it just stops responding. Already connected users can do nothing. And noone can connect to the DB.
users connect from Windows Terminal Servers (other users on the same servers connect to other DBs on other servers. They work fine)

The server itself runs fine. We can log in to the server and see whatever is running on the server.
Firebird will not work until it has been killed and restarted.
After that everything works fine.

What have we tried so far:

- Backup/restore cycle
- Disabling 3 of the 4 cores og the CPU.
- Upgrading to newest Firebird.
- Checked cables and NICs

It really seems as if Firebird is waiting for itself to do something.
As an endless loop.


Last time it happend was just an hour ago.
My colleque did some backtrace.
15 threads where running.

They end with one of these states:

[Switching to thread 3 (Thread 1174452576 (LWP 25503))]#0 0x00002ba09b41ceeb in __lll_mutex_lock_wait () from /lib/libpthread.so.0


[Switching to thread 2 (Thread 1191237984 (LWP 25504))]#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0


[Switching to thread 15 (Thread 1090525536 (LWP 9497))]#0 0x00002ba09b41c76f in sem_wait () from /lib/libpthread.so.0


pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
(gdb) bt



We switched from a windows platform a year ago to this linux.
We did that because of performance and because we had some simular problems on the windows platform.

We have a lot of installations out. And this is the only one that gives ud problems.


Michael


Last Backtrace:

GNU gdb 6.4.90-debian
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Attaching to process 9494
Reading symbols from /opt/firebird/bin/fbserver...(no debugging symbols found)...done.
Using host libthread_db library "/lib/libthread_db.so.1".
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libncurses.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib/libncurses.so.5
Reading symbols from /opt/firebird/lib/libicuuc.so.30...(no debugging symbols found)...done.
Loaded symbols for /opt/firebird/lib/libicuuc.so.30
Reading symbols from /opt/firebird/lib/libicudata.so.30...(no debugging symbols found)...done.
Loaded symbols for /opt/firebird/lib/libicudata.so.30
Reading symbols from /opt/firebird/lib/libicui18n.so.30...
(no debugging symbols found)...done.
Loaded symbols for /opt/firebird/lib/libicui18n.so.30
Reading symbols from /usr/lib/libstdc++.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libstdc++.so.5
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 47968802984336 (LWP 9494)]
[New Thread 1191237984 (LWP 25504)]
[New Thread 1174452576 (LWP 25503)]
[New Thread 1166059872 (LWP 25414)]
[New Thread 1182845280 (LWP 25382)]
[New Thread 1157667168 (LWP 25065)]
[New Thread 1124096352 (LWP 23210)]
[New Thread 1115703648 (LWP 21382)]
[New Thread 1140881760 (LWP 17435)]
[New Thread 1149274464 (LWP 17434)]
[New Thread 1098918240 (LWP 17433)]
[New Thread 1107310944 (LWP 17432)]
[New Thread 1132489056 (LWP 14931)]
[New Thread 1082132832 (LWP 9498)]
[New Thread 1090525536 (LWP 9497)]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libc.so.6...
(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libnss_compat.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libnss_nis.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /opt/firebird/intl/fbintl...(no debugging symbols found)...done.
Loaded symbols for /opt/firebird/intl/fbintl
Reading symbols from /opt/firebird/UDF/ib_udf.so...(no debugging symbols found)...done.
Loaded symbols for /opt/firebird/UDF/ib_udf.so
0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0

(gdb) t 1
[Switching to thread 1 (Thread 47968802984336 (LWP 9494))]#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
(gdb) bt
#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00000000004241a0 in gds__default_printer ()
#2 0x00000000004264ac in gds__default_printer ()
#3 0x00000000004259f8 in gds__default_printer ()
#4 0x00000000005e4abb in ERRD_post ()
#5 0x00000000005e46df in ERRD_post ()
#6 0x00000000005f639d in ERRD_post ()
#7 0x000000000040b37c in putchar ()
#8 0x000000000040aa1f in ?? ()
#9 0x0000000000413739 in putchar ()
#10 0x00002ba09b5434ca in __libc_start_main () from /lib/libc.so.6
#11 0x000000000040a32a in ?? ()
#12 0x00007fffc84656f8 in ?? ()
#13 0x0000000000000000 in ?? ()

(gdb) t 2
[Switching to thread 2 (Thread 1191237984 (LWP 25504))]#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
(gdb) bt
#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00000000004241a0 in gds__default_printer ()
#2 0x00000000004264ac in gds__default_printer ()
#3 0x00000000004259f8 in gds__default_printer ()
#4 0x00000000005761aa in SVC_output ()
#5 0x00000000004287d4 in (anonymous namespace)::threadStart ()
#6 0x00002ba09b417f1a in start_thread () from /lib/libpthread.so.0
#7 0x00002ba09b5f05d2 in clone () from /lib/libc.so.6
#8 0x0000000000000000 in ?? ()

(gdb) t 3
[Switching to thread 3 (Thread 1174452576 (LWP 25503))]#0 0x00002ba09b41ceeb in __lll_mutex_lock_wait () from /lib/libpthread.so.0
(gdb) bt
#0 0x00002ba09b41ceeb in __lll_mutex_lock_wait () from /lib/libpthread.so.0
#1 0x0000000000000000 in ?? ()

(gdb) t 4
[Switching to thread 4 (Thread 1166059872 (LWP 25414))]#0 0x00002ba09b41ceeb in __lll_mutex_lock_wait () from /lib/libpthread.so.0
(gdb) bt
#0 0x00002ba09b41ceeb in __lll_mutex_lock_wait () from /lib/libpthread.so.0
#1 0x00000000008781c0 in (anonymous namespace)::pendingSignals ()
#2 0x00000000008782cc in global_mutex ()
#3 0x00002ba09b419d9f in pthread_mutex_lock () from /lib/libpthread.so.0
#4 0x0000000045809980 in ?? ()
#5 0x0000000000425c1d in gds__default_printer ()
#6 0x000000000050bc18 in Firebird::IntlUtil::cvtUtf16ToAscii ()
#7 0x000000000042f10a in (anonymous namespace)::threadStart ()
#8 0x000000000040c6c2 in putchar ()
#9 0x000000000040f868 in putchar ()
#10 0x000000000040f502 in putchar ()
#11 0x0000000000411eee in putchar ()
#12 0x00000000004287d4 in (anonymous namespace)::threadStart ()
#13 0x00002ba09b417f1a in start_thread () from /lib/libpthread.so.0
#14 0x00002ba09b5f05d2 in clone () from /lib/libc.so.6
#15 0x0000000000000000 in ?? ()

(gdb) t 5
[Switching to thread 5 (Thread 1182845280 (LWP 25382))]#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
(gdb) bt
#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00000000004241a0 in gds__default_printer ()
#2 0x00000000004264ac in gds__default_printer ()
#3 0x00000000004259f8 in gds__default_printer ()
#4 0x000000000041222f in putchar ()
#5 0x00000000004287d4 in (anonymous namespace)::threadStart ()
#6 0x00002ba09b417f1a in start_thread () from /lib/libpthread.so.0
#7 0x00002ba09b5f05d2 in clone () from /lib/libc.so.6
#8 0x0000000000000000 in ?? ()


(gdb) t 6
[Switching to thread 6 (Thread 1157667168 (LWP 25065))]#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
(gdb) bt
#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00000000004241a0 in gds__default_printer ()
#2 0x00000000004264ac in gds__default_printer ()
#3 0x00000000004259f8 in gds__default_printer ()
#4 0x0000000000570c24 in SVC_output ()
#5 0x0000000000505cc1 in Firebird::IntlUtil::cvtUtf16ToAscii ()
#6 0x000000000042c565 in (anonymous namespace)::threadStart ()
#7 0x0000000000412f88 in putchar ()
#8 0x000000000040f7ba in putchar ()
#9 0x000000000040f502 in putchar ()
#10 0x0000000000411eee in putchar ()
#11 0x00000000004287d4 in (anonymous namespace)::threadStart ()
#12 0x00002ba09b417f1a in start_thread () from /lib/libpthread.so.0
#13 0x00002ba09b5f05d2 in clone () from /lib/libc.so.6
#14 0x0000000000000000 in ?? ()

(gdb) t 7
[Switching to thread 7 (Thread 1124096352 (LWP 23210))]#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
(gdb) bt
#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00000000004241a0 in gds__default_printer ()
#2 0x00000000004264ac in gds__default_printer ()
#3 0x00000000004259f8 in gds__default_printer ()
#4 0x00000000004add6c in Jrd::GlobalRWLock::blocking_ast_cached_lock ()
#5 0x000000000057b307 in SVC_output ()
#6 0x00000000004af9b8 in CCH_down_grade_dbb ()
#7 0x00000000004b0625 in CCH_down_grade_dbb ()
#8 0x0000000000519f36 in THD_mutex_unlock ()
#9 0x00000000005468fb in DelayFailedLogin::~DelayFailedLogin ()
#10 0x0000000000546607 in DelayFailedLogin::~DelayFailedLogin ()
#11 0x0000000000545ad3 in DelayFailedLogin::~DelayFailedLogin ()
#12 0x00000000004d5514 in ERR_post ()
#13 0x00000000005468c2 in DelayFailedLogin::~DelayFailedLogin ()
#14 0x0000000000545ad3 in DelayFailedLogin::~DelayFailedLogin ()
#15 0x00000000004de0a4 in ERR_post ()
#16 0x00000000004dd2c6 in ERR_post ()
#17 0x00000000004dc953 in ERR_post ()
#18 0x000000000050e187 in Firebird::IntlUtil::cvtUtf16ToAscii ()
#19 0x0000000000434b6f in (anonymous namespace)::threadStart ()
#20 0x00000000005a6f18 in SVC_output ()
#21 0x0000000000431d19 in (anonymous namespace)::threadStart ()
#22 0x000000000040dbc1 in putchar ()
#23 0x000000000040fa25 in putchar ()
#24 0x000000000040f502 in putchar ()
#25 0x0000000000411eee in putchar ()
#26 0x00000000004287d4 in (anonymous namespace)::threadStart ()
#27 0x00002ba09b417f1a in start_thread () from /lib/libpthread.so.0
#28 0x00002ba09b5f05d2 in clone () from /lib/libc.so.6
#29 0x0000000000000000 in ?? ()
(gdb)

(gdb) t 8
[Switching to thread 8 (Thread 1115703648 (LWP 21382))]#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
(gdb) bt
#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00000000004241a0 in gds__default_printer ()
#2 0x00000000004264ac in gds__default_printer ()
#3 0x00000000004259f8 in gds__default_printer ()
#4 0x000000000041222f in putchar ()
#5 0x00000000004287d4 in (anonymous namespace)::threadStart ()
#6 0x00002ba09b417f1a in start_thread () from /lib/libpthread.so.0
#7 0x00002ba09b5f05d2 in clone () from /lib/libc.so.6
#8 0x0000000000000000 in ?? ()

(gdb) t 9
[Switching to thread 9 (Thread 1140881760 (LWP 17435))]#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
(gdb) bt
#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00000000004241a0 in gds__default_printer ()
#2 0x00000000004264ac in gds__default_printer ()
#3 0x00000000004259f8 in gds__default_printer ()
#4 0x00000000005769af in SVC_output ()
#5 0x00000000004287d4 in (anonymous namespace)::threadStart ()
#6 0x00002ba09b417f1a in start_thread () from /lib/libpthread.so.0
#7 0x00002ba09b5f05d2 in clone () from /lib/libc.so.6
#8 0x0000000000000000 in ?? ()

(gdb) t 10
[Switching to thread 10 (Thread 1149274464 (LWP 17434))]#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
(gdb) bt
#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00000000004241a0 in gds__default_printer ()
#2 0x00000000004264ac in gds__default_printer ()
#3 0x00000000004259f8 in gds__default_printer ()
#4 0x00000000004b243c in CCH_down_grade_dbb ()
#5 0x00000000004287d4 in (anonymous namespace)::threadStart ()
#6 0x00002ba09b417f1a in start_thread () from /lib/libpthread.so.0
#7 0x00002ba09b5f05d2 in clone () from /lib/libc.so.6
#8 0x0000000000000000 in ?? ()

(gdb) t 11
[Switching to thread 11 (Thread 1098918240 (LWP 17433))]#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
(gdb) bt
#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00000000004241a0 in gds__default_printer ()
#2 0x00000000004264ac in gds__default_printer ()
#3 0x00000000004259f8 in gds__default_printer ()
#4 0x00000000005769af in SVC_output ()
#5 0x00000000004287d4 in (anonymous namespace)::threadStart ()
#6 0x00002ba09b417f1a in start_thread () from /lib/libpthread.so.0
#7 0x00002ba09b5f05d2 in clone () from /lib/libc.so.6
#8 0x0000000000000000 in ?? ()

(gdb) t 12
[Switching to thread 12 (Thread 1107310944 (LWP 17432))]#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
(gdb) bt
#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00000000004241a0 in gds__default_printer ()
#2 0x00000000004264ac in gds__default_printer ()
#3 0x00000000004259f8 in gds__default_printer ()
#4 0x00000000004b243c in CCH_down_grade_dbb ()
#5 0x00000000004287d4 in (anonymous namespace)::threadStart ()
#6 0x00002ba09b417f1a in start_thread () from /lib/libpthread.so.0
#7 0x00002ba09b5f05d2 in clone () from /lib/libc.so.6
#8 0x0000000000000000 in ?? ()

(gdb) t 13
[Switching to thread 13 (Thread 1132489056 (LWP 14931))]#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
(gdb) bt
#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00000000004241a0 in gds__default_printer ()
#2 0x00000000004264ac in gds__default_printer ()
#3 0x00000000004259f8 in gds__default_printer ()
#4 0x000000000041222f in putchar ()
#5 0x00000000004287d4 in (anonymous namespace)::threadStart ()
#6 0x00002ba09b417f1a in start_thread () from /lib/libpthread.so.0
#7 0x00002ba09b5f05d2 in clone () from /lib/libc.so.6
#8 0x0000000000000000 in ?? ()

(gdb) t 14
[Switching to thread 14 (Thread 1082132832 (LWP 9498))]#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
(gdb) bt
#0 0x00002ba09b41ab3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00000000004241a0 in gds__default_printer ()
#2 0x00000000004264ac in gds__default_printer ()
#3 0x00000000004259f8 in gds__default_printer ()
#4 0x000000000041222f in putchar ()
#5 0x00000000004287d4 in (anonymous namespace)::threadStart ()
#6 0x00002ba09b417f1a in start_thread () from /lib/libpthread.so.0
#7 0x00002ba09b5f05d2 in clone () from /lib/libc.so.6
#8 0x0000000000000000 in ?? ()

(gdb) t 15
[Switching to thread 15 (Thread 1090525536 (LWP 9497))]#0 0x00002ba09b41c76f in sem_wait () from /lib/libpthread.so.0
(gdb) bt
#0 0x00002ba09b41c76f in sem_wait () from /lib/libpthread.so.0
#1 0x000000000040adfb in ?? ()
#2 0x00000000004287d4 in (anonymous namespace)::threadStart ()
#3 0x00002ba09b417f1a in start_thread () from /lib/libpthread.so.0
#4 0x00002ba09b5f05d2 in clone () from /lib/libc.so.6
#5 0x0000000000000000 in ?? ()
(gdb) t 16
Thread ID 16 not known.
(gdb)
Thread ID 16 not known.
(gdb) t 30
Thread ID 30 not known.