Subject | Import of > 1 mio. records |
---|---|
Author | thomas_besand |
Post date | 2005-03-18T13:44:47Z |
Hi everybody,
I'm developing a database application, which will administer several
hundred tables with up to 1.5 mio records of text data each.
Right now I'm working on the import side of the beast.
I noticed that in a long running import routine the firebird
server(1.5.2.4731) is getting slower and slower until it virtually
grinds to a standstill.
Also Taskmanager (Win2k) shows it's eating up more and more memory.
Take a look at this excerpt from a log file, I'm writing during import
(the HEARTBEAT item is written every 10,000 records):
<snip>
SUCCESS Create us076881_nochmal 18.03.2005 00:24:33
START Import us076881_nochmal 18.03.2005 00:24:35
HEARTBEAT Import us076881_nochmal 18.03.2005 00:28:52
HEARTBEAT Import us076881_nochmal 18.03.2005 00:31:59
HEARTBEAT Import us076881_nochmal 18.03.2005 00:35:25
HEARTBEAT Import us076881_nochmal 18.03.2005 00:39:02
HEARTBEAT Import us076881_nochmal 18.03.2005 00:42:52
HEARTBEAT Import us076881_nochmal 18.03.2005 00:46:58
HEARTBEAT Import us076881_nochmal 18.03.2005 00:51:15
HEARTBEAT Import us076881_nochmal 18.03.2005 00:55:45
HEARTBEAT Import us076881_nochmal 18.03.2005 01:00:32
HEARTBEAT Import us076881_nochmal 18.03.2005 01:05:41
HEARTBEAT Import us076881_nochmal 18.03.2005 01:10:59
HEARTBEAT Import us076881_nochmal 18.03.2005 01:16:29
HEARTBEAT Import us076881_nochmal 18.03.2005 01:22:07
HEARTBEAT Import us076881_nochmal 18.03.2005 01:27:56
HEARTBEAT Import us076881_nochmal 18.03.2005 01:34:01
HEARTBEAT Import us076881_nochmal 18.03.2005 01:40:15
HEARTBEAT Import us076881_nochmal 18.03.2005 01:46:44
HEARTBEAT Import us076881_nochmal 18.03.2005 02:00:32
HEARTBEAT Import us076881_nochmal 18.03.2005 02:14:57
HEARTBEAT Import us076881_nochmal 18.03.2005 02:29:47
HEARTBEAT Import us076881_nochmal 18.03.2005 02:46:06
HEARTBEAT Import us076881_nochmal 18.03.2005 03:01:54
HEARTBEAT Import us076881_nochmal 18.03.2005 03:18:00
HEARTBEAT Import us076881_nochmal 18.03.2005 03:34:47
HEARTBEAT Import us076881_nochmal 18.03.2005 03:51:39
HEARTBEAT Import us076881_nochmal 18.03.2005 04:08:55
HEARTBEAT Import us076881_nochmal 18.03.2005 04:26:39
HEARTBEAT Import us076881_nochmal 18.03.2005 04:44:46
HEARTBEAT Import us076881_nochmal 18.03.2005 05:03:25
HEARTBEAT Import us076881_nochmal 18.03.2005 05:22:39
HEARTBEAT Import us076881_nochmal 18.03.2005 05:42:07
HEARTBEAT Import us076881_nochmal 18.03.2005 06:02:01
HEARTBEAT Import us076881_nochmal 18.03.2005 06:22:20
HEARTBEAT Import us076881_nochmal 18.03.2005 06:43:02
<snap>
I cancelled the import this morning with around 350,000 records
imported; too bad the job was only about 30% finished.
Can someone please point me to a technique I can use, to avoid this
uncomfortable behavior?
Kind regards,
Thomas Besand
Berlin, Germany
I'm developing a database application, which will administer several
hundred tables with up to 1.5 mio records of text data each.
Right now I'm working on the import side of the beast.
I noticed that in a long running import routine the firebird
server(1.5.2.4731) is getting slower and slower until it virtually
grinds to a standstill.
Also Taskmanager (Win2k) shows it's eating up more and more memory.
Take a look at this excerpt from a log file, I'm writing during import
(the HEARTBEAT item is written every 10,000 records):
<snip>
SUCCESS Create us076881_nochmal 18.03.2005 00:24:33
START Import us076881_nochmal 18.03.2005 00:24:35
HEARTBEAT Import us076881_nochmal 18.03.2005 00:28:52
HEARTBEAT Import us076881_nochmal 18.03.2005 00:31:59
HEARTBEAT Import us076881_nochmal 18.03.2005 00:35:25
HEARTBEAT Import us076881_nochmal 18.03.2005 00:39:02
HEARTBEAT Import us076881_nochmal 18.03.2005 00:42:52
HEARTBEAT Import us076881_nochmal 18.03.2005 00:46:58
HEARTBEAT Import us076881_nochmal 18.03.2005 00:51:15
HEARTBEAT Import us076881_nochmal 18.03.2005 00:55:45
HEARTBEAT Import us076881_nochmal 18.03.2005 01:00:32
HEARTBEAT Import us076881_nochmal 18.03.2005 01:05:41
HEARTBEAT Import us076881_nochmal 18.03.2005 01:10:59
HEARTBEAT Import us076881_nochmal 18.03.2005 01:16:29
HEARTBEAT Import us076881_nochmal 18.03.2005 01:22:07
HEARTBEAT Import us076881_nochmal 18.03.2005 01:27:56
HEARTBEAT Import us076881_nochmal 18.03.2005 01:34:01
HEARTBEAT Import us076881_nochmal 18.03.2005 01:40:15
HEARTBEAT Import us076881_nochmal 18.03.2005 01:46:44
HEARTBEAT Import us076881_nochmal 18.03.2005 02:00:32
HEARTBEAT Import us076881_nochmal 18.03.2005 02:14:57
HEARTBEAT Import us076881_nochmal 18.03.2005 02:29:47
HEARTBEAT Import us076881_nochmal 18.03.2005 02:46:06
HEARTBEAT Import us076881_nochmal 18.03.2005 03:01:54
HEARTBEAT Import us076881_nochmal 18.03.2005 03:18:00
HEARTBEAT Import us076881_nochmal 18.03.2005 03:34:47
HEARTBEAT Import us076881_nochmal 18.03.2005 03:51:39
HEARTBEAT Import us076881_nochmal 18.03.2005 04:08:55
HEARTBEAT Import us076881_nochmal 18.03.2005 04:26:39
HEARTBEAT Import us076881_nochmal 18.03.2005 04:44:46
HEARTBEAT Import us076881_nochmal 18.03.2005 05:03:25
HEARTBEAT Import us076881_nochmal 18.03.2005 05:22:39
HEARTBEAT Import us076881_nochmal 18.03.2005 05:42:07
HEARTBEAT Import us076881_nochmal 18.03.2005 06:02:01
HEARTBEAT Import us076881_nochmal 18.03.2005 06:22:20
HEARTBEAT Import us076881_nochmal 18.03.2005 06:43:02
<snap>
I cancelled the import this morning with around 350,000 records
imported; too bad the job was only about 30% finished.
Can someone please point me to a technique I can use, to avoid this
uncomfortable behavior?
Kind regards,
Thomas Besand
Berlin, Germany