Subject | Re: [firebird-support] Import of > 1 mio. records |
---|---|
Author | Pavel Menshchikov |
Post date | 2005-03-18T14:09:28Z |
Hello Thomas,
t> I'm developing a database application, which will administer several
t> hundred tables with up to 1.5 mio records of text data each.
t> Right now I'm working on the import side of the beast.
t> I noticed that in a long running import routine the firebird
t> server(1.5.2.4731) is getting slower and slower until it virtually
t> grinds to a standstill.
t> Also Taskmanager (Win2k) shows it's eating up more and more memory.
t> Take a look at this excerpt from a log file, I'm writing during import
t> (the HEARTBEAT item is written every 10,000 records):
Do you commit during your import process? If no, try to commit every
5000 records for example (this number may vary for the best
performance, it depends on the data structure and size). You also may
deactivate indices and/or triggers for a period of the import process,
if it's suitable in your situation (for example if triggers in fact
don't modify the data when importing).
HTH
--
Best regards,
Pavel Menshchikov
http://www.ls-software.com
t> I'm developing a database application, which will administer several
t> hundred tables with up to 1.5 mio records of text data each.
t> Right now I'm working on the import side of the beast.
t> I noticed that in a long running import routine the firebird
t> server(1.5.2.4731) is getting slower and slower until it virtually
t> grinds to a standstill.
t> Also Taskmanager (Win2k) shows it's eating up more and more memory.
t> Take a look at this excerpt from a log file, I'm writing during import
t> (the HEARTBEAT item is written every 10,000 records):
Do you commit during your import process? If no, try to commit every
5000 records for example (this number may vary for the best
performance, it depends on the data structure and size). You also may
deactivate indices and/or triggers for a period of the import process,
if it's suitable in your situation (for example if triggers in fact
don't modify the data when importing).
HTH
--
Best regards,
Pavel Menshchikov
http://www.ls-software.com