Subject Re: [firebird-support] large system slows over time
Author Steve Wiser
Post date 2015-05-08T13:55:50Z
Hi Nick,

My interpretation of those stats is that you are now out of memory as I always look at the Swap Used.  You aren't using any.  My experience with Linux is that the OS will take as much memory as it can/needs so it can look like you are maxed out on RAM usage or something, but you really need to pay attention to the swap used.

What I would be interested in seeing is that when you run your gbak backup go ahead and also run top.  Are you seeing a lot of CPU time being spent in IOWait?  If so that would mean that the CPU is being starved of data and is waiting around -- usually because the disk isn't fast enough.  We normally only see this on virtual hosts though.

-steve


--
Steve Wiser
President
Specialized Business Software
6325 Cochran Road, Unit 1
Solon, OH 44139

www.specializedbusinesssoftware.com
www.docunym.com
(440) 542-9145 - fax (440) 542-9143
Toll Free: (866) 328-4936


On Fri, May 8, 2015 at 6:32 AM, Nick Upson nu@... [firebird-support] <firebird-support@yahoogroups.com> wrote:
 

Your mention of memory & paging got me thinking. I expected the output below to show the memory was fully used such that more RAM would help
but unless I'm reading this wrong only 5.3Gb is being used and 2.4 GB is unused so more RAM will be no help, can anyone confirm or put me right

$ cat /proc/meminfo
MemTotal:      8309036 kB
MemFree:        307408 kB
Buffers:         59296 kB
Cached:        7433812 kB
SwapCached:         32 kB
Active:        5365212 kB
Inactive:      2470416 kB
HighTotal:     7471040 kB
HighFree:        18724 kB
LowTotal:       837996 kB
LowFree:        288684 kB
SwapTotal:     4095992 kB
SwapFree:      4095788 kB
Dirty:          186968 kB
Writeback:           0 kB
AnonPages:      342384 kB
Mapped:         140812 kB
Slab:           148064 kB
PageTables:       6824 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   8250508 kB
Committed_AS:  2795520 kB
VmallocTotal:   116728 kB
VmallocUsed:      5788 kB
VmallocChunk:   110588 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB


Nick Upson, Telensa Ltd, Senior Operations Network Engineer
direct +44 (0) 1799 533252, support hotline +44 (0) 1799 399200

On 5 May 2015 at 15:21, Ann Harrison aharrison@... [firebird-support] <firebird-support@yahoogroups.com> wrote:
 

Hi Nick,


I have a system that is slowing down the longer it stays running and I'd like to know why.
...

Is there any evidence I can gather before I reboot the system which I expect (from past experience) will return the system to the better performance


Hmmm....   If rebooting will solve the problem and you don't have a very long running transaction, then I doubt that garbage accumulation or collection is the source of the problem.  
Nor is there anything else wrong with your physical database - fragmentation or whatever.
I'd look at memory usage - using both Firebird and OS tools.  Look for paging before you reboot.
After you reboot, track memory usage daily or more often.   It doesn't take much of a leak - or unfortunate caching - to built up over month of 27 tps.

Accelerating the move to 2.5.4 would also be a good idea, both because it may correct the problem and because it probably offers better diagnostic tools for this sort of problem.  

Good luck,

Ann