Subject | Re: [IBO] Pumping data uses a lot of memory |
---|---|
Author | Dmitry Beloshistov |
Post date | 2005-12-05T15:29:51Z |
Hello, williamvdw2005!
You wrote to <IBObjects@yahoogroups.com> on Mon, 05 Dec 2005 13:59:18 -0000:
w> I am pumping about a million records from one database to another
w> using IB_Datapump. This process is using a lot of memory causing my
w> server to run out of virtual memory.
I`m use external tools for data dump/pump - IBEScript from IBExpert suite.
One of solution - use own writed utility for dump and pump data by blocks
(1000-10000 records in one records portion (block) per one transaction
between start/commit.)
Simple algorithm:
1) Make on-disk based SQL file for pump.
2) Read portions of records for pumping from this file into IB_Script
3) Start transaction
4) Execute script (IB_Sctipt.Execute)
5) Commit transaction
6) Repeat step 2 (while not EOF in input file)
In this case
1) server commit only one portion of records in each transaction commit
time.
2) we have only one portion of records in memory
WBR,Dmitry Beloshistov AKA [-=BDS=-]
You wrote to <IBObjects@yahoogroups.com> on Mon, 05 Dec 2005 13:59:18 -0000:
w> I am pumping about a million records from one database to another
w> using IB_Datapump. This process is using a lot of memory causing my
w> server to run out of virtual memory.
I`m use external tools for data dump/pump - IBEScript from IBExpert suite.
One of solution - use own writed utility for dump and pump data by blocks
(1000-10000 records in one records portion (block) per one transaction
between start/commit.)
Simple algorithm:
1) Make on-disk based SQL file for pump.
2) Read portions of records for pumping from this file into IB_Script
3) Start transaction
4) Execute script (IB_Sctipt.Execute)
5) Commit transaction
6) Repeat step 2 (while not EOF in input file)
In this case
1) server commit only one portion of records in each transaction commit
time.
2) we have only one portion of records in memory
WBR,Dmitry Beloshistov AKA [-=BDS=-]