| Subject | Re: [firebird-support] Performance Degrade - HELP |
|---|---|
| Author | Rajesh Punjabi |
| Post date | 2003-10-21T09:53:18Z |
Hi Everyone,
As committed I am posting the results of my study of what is causing
memory leaks on the server. Though I am nearly a week late ... just
wanted to be sure before posting.
We were using Mailscanner (www.mailscanner.co.uk) ver 4.14 which in turn
uses McAfee DAT files for virus scanning. Mailscanner got a major memory
leak problem with the last two dats (ver 4297 and one before that).
Since the dats were upgraded Mailscanner started leaking memory. (This
is an autoupdate of dats by mailscanner) Mailscanner site says that
there is a parameter to restart all child processes every n seconds
(default value 14400 secs) since child processes can cause memory leaks.
We have narrowed the source of the problem down to this.
Once Mailscanner started leaking memory, firebird somehow caught on
(when both were running simultaneously) and started leaking even more
memory. While Mailscanner alone would leak in the range of 2 MB per hour
, both put together would do 100 MB per hour. No other task was affected
except FB.
Somehow the process of leaking memory is irreversible with both these
running on our server. What I mean is once a portion of memory was used
up even restarting both processes together / seperately would not free
up the leaked memory. Something wrong with the garbage collection of
Linux with the specific kernel (2.4.18-14) with both these running. In
fact simply stopping mailscanner did not have any effect coz if
mailscanner had run even for a few seconds ... somehow fb was not able
to shield itself from the effects of that leak and would degrade
drastically over time. Only when we uninstalled Mailscanner and
restarted the server did the leak problem subside.
Other processes on our server include Sendmail and fetchmail which seem
to be fairly stable themselves and do not seem to be causing other
programs to behave erratically.
Right now the server has been running for last five days continuously
with the same db and no noticable user performance degrade at the
moment. However still there is one major cause for concern. In the last
five days the memory usage pattern of firebird that I have observed is
as under :
At startup (machine startup) with no user logged in FB takes about 2MB
memory footprint. Once users start logging in usage keeps going up and
stabilizes to a certain point (I guess this is a function of the kind of
queries being run - not sure ... can someone clarify on this). When all
users log out the FB Memory footprint was at apporx 24 MB on day 1 and
peak utilization in the day was at 104 MB. Today by day 5 the peak
utilization is at 204MB though the type of queries is not changed. Same
users .. same queries. When all users logged out for lunch now the FB
footprint was 126MB. Looks like FB is still eating a cool 24MB a day and
not releasing it. I have not restarted FB even once in these 5 days.
Is there an overall problem with some limited leak with FB or the way FB
1.0.2 uses memory and does not release it ? We are planning to make the
server run to the point that we are forced to do a restart (FB or Server
itself) and note the values thereof for memory usage. If problem
persists as of now we are planning to upgrade to FB 1.0.3 (Do not wish
to take a chance with FB 1.5 till it has been in production for a few
months atleast) Can anyone throw light on the memory usage pattern that
we noticed. Again is it just our db or is this a consistent thing across
OS and versions and users.
I wish to express gratitude to the entire FB community for the help and
suggestions given to try and help us solve this problem. As of now it
looks like we know what the problem was and have taken remedial steps to
correct it. Will keep you all posted should this happen again.
Warm regards,
RP
Paul Reeves wrote:
As committed I am posting the results of my study of what is causing
memory leaks on the server. Though I am nearly a week late ... just
wanted to be sure before posting.
We were using Mailscanner (www.mailscanner.co.uk) ver 4.14 which in turn
uses McAfee DAT files for virus scanning. Mailscanner got a major memory
leak problem with the last two dats (ver 4297 and one before that).
Since the dats were upgraded Mailscanner started leaking memory. (This
is an autoupdate of dats by mailscanner) Mailscanner site says that
there is a parameter to restart all child processes every n seconds
(default value 14400 secs) since child processes can cause memory leaks.
We have narrowed the source of the problem down to this.
Once Mailscanner started leaking memory, firebird somehow caught on
(when both were running simultaneously) and started leaking even more
memory. While Mailscanner alone would leak in the range of 2 MB per hour
, both put together would do 100 MB per hour. No other task was affected
except FB.
Somehow the process of leaking memory is irreversible with both these
running on our server. What I mean is once a portion of memory was used
up even restarting both processes together / seperately would not free
up the leaked memory. Something wrong with the garbage collection of
Linux with the specific kernel (2.4.18-14) with both these running. In
fact simply stopping mailscanner did not have any effect coz if
mailscanner had run even for a few seconds ... somehow fb was not able
to shield itself from the effects of that leak and would degrade
drastically over time. Only when we uninstalled Mailscanner and
restarted the server did the leak problem subside.
Other processes on our server include Sendmail and fetchmail which seem
to be fairly stable themselves and do not seem to be causing other
programs to behave erratically.
Right now the server has been running for last five days continuously
with the same db and no noticable user performance degrade at the
moment. However still there is one major cause for concern. In the last
five days the memory usage pattern of firebird that I have observed is
as under :
At startup (machine startup) with no user logged in FB takes about 2MB
memory footprint. Once users start logging in usage keeps going up and
stabilizes to a certain point (I guess this is a function of the kind of
queries being run - not sure ... can someone clarify on this). When all
users log out the FB Memory footprint was at apporx 24 MB on day 1 and
peak utilization in the day was at 104 MB. Today by day 5 the peak
utilization is at 204MB though the type of queries is not changed. Same
users .. same queries. When all users logged out for lunch now the FB
footprint was 126MB. Looks like FB is still eating a cool 24MB a day and
not releasing it. I have not restarted FB even once in these 5 days.
Is there an overall problem with some limited leak with FB or the way FB
1.0.2 uses memory and does not release it ? We are planning to make the
server run to the point that we are forced to do a restart (FB or Server
itself) and note the values thereof for memory usage. If problem
persists as of now we are planning to upgrade to FB 1.0.3 (Do not wish
to take a chance with FB 1.5 till it has been in production for a few
months atleast) Can anyone throw light on the memory usage pattern that
we noticed. Again is it just our db or is this a consistent thing across
OS and versions and users.
I wish to express gratitude to the entire FB community for the help and
suggestions given to try and help us solve this problem. As of now it
looks like we know what the problem was and have taken remedial steps to
correct it. Will keep you all posted should this happen again.
Warm regards,
RP
Paul Reeves wrote:
>On Friday 10 October 2003 15:08, Rajesh Punjabi wrote:
>
>
>
>>Clearly firebird is unable to free the memory for some reason ?
>>
>>
>
>Clearly Linux is behaving normally as far as I can see.
>
>
>
>>What can I do to improve this ?
>>
>>
>
>Look elsewhere for the cause of your problem. My guess is that you have a long
>running transaction, or that you are using commit retaining somewhere, which
>has the same effect as a long running transaction. Either way the engine has
>to keep track of all the intermediate versions created since the start of the
>oldest transaction.
>
>
>Paul
>
>