Subject FB classic 3.0.4 - cannot find tip page (165), file: tra.cpp line: 2331
Author Vladimir Zbodulja
Post date 2019-05-10T15:41:57Z
Hello,
has anyone experienced following error:
     internal Firebird consistency check (cannot find tip page (165), file: tra.cpp line: 2331)

We are running Firebird Classic server LI-V3.0.4.33054 (64bit) on RHEL 7.6 (3.10.0-957.5.1.el7.x86_64).
Database has 50GB (and configured shadow) and there are around 1000 concurrent connection to database with approximately  1M growth in transactions per day (server has 24 cores and 256GB RAM).

Error occurs (mostly) during peak loads (i.e. server load around 50%), 3-4 times a day. Application that is using database is 32bit, connects to database with legacy 32bit C API by using  libfbclient.so (3.0.4 - also 32bit).  Connection is remote (i.e. app is running on one server and Firebird is on another server).

Scenario goes as follows:
  • at some time this error occurs in firebird.log:
srv      Fri May 10 14:10:59 2019 
       Database: /DBS/interbase/database.gdb
       internal Firebird consistency check (cannot find tip page (165), file: tra.cpp line: 2331)

  • when this error occurs users can still connect to database and current connection are still working 
  • after some time (10-15 min.), another two errors appear:
srv      Fri May 10 14:23:33 2019
       I/O error during "read" operation for file "/DBS/interbase/database.gdb"
       Error while trying to read from file
       No such file or directory

srv      Fri May 10 14:23:33 2019
       I/O error during "read" operation for file "/DBS2/interbase/shadow.shd"
       Error while trying to read from file
       No such file or directory
  • after that all connection are blocked/stopped (and no new connection to database is possible - any connection attempt hangs indefinitely)
Only solution to end this is to kill all "firebird" processes after which connections are possible again and everything is working until same scenario repeats(3-5 hours later).

I've tried gfix (-v -full -ig) - it doesn't report any error. Also tried backup/restore, but problem persists.

I'm not sure if this could be connected, but here is a message from FBTRACE log around time when error happened:

2019-05-10T14:10:59.1630 (24834:0x7f340408d1c0) ERROR AT JResultSet::fetchNext
       /DBS/interbase/pisdb.gdb (ATT_49208, SYSDBA:NONE, NONE, TCPv6:::1/38886)
       /home/App/app_run:24606
335544333 : internal Firebird consistency check (cannot find tip page (165), file: tra.cpp line: 2331)

Only thing that I've found and seems loosely related to this is:
http://tracker.firebirdsql.org/browse/CORE-5935?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
but I'm not if this two issues are the same (or even similar).

Any help or suggestion is appreciated.

Settings are (firebird.conf):
AuditTraceConfigFile = /opt/firebird/fbtrace.conf
DefaultDbCachePages = 10240
TempBlockSize = 2M
AuthServer = Srp, Legacy_Auth
WireCrypt = Disabled
LockMemSize = 1M
LockHashSlots = 21001
ServerMode = Classic

Br,
    Vladimir.

--
Vladimir Zbodulja
t: +385 1 36 39 330
f: +385 1 36 91 471
m: +385 98 40 27 15

Vitezićeva 2
HR / 10110 Zagreb
www.pis.eu.com

Ova elektronicka poruka i/ili bilo koji privitak ovoj poruci mogu sadrzavati povjerljive informacije. Otkrivanje njihova sadrzaja drugim osobama moguce je samo uz prethodno odobrenje. Ova poruka je namijenjena samo osobi/osobama kojima je adresirana. Ako vi niste osoba kojoj je ova poruka namijenjena, molim vas da je odmah izbrisete. 

This email and/or any of its attachments, may contain confidential information. It must not be disclosed to any person(s) without authorization. This email is intended for the attention of the named addressee(s). If you are not the intended recipient, please, delete this message immediately.