Subject | INSERT efficiency |
---|---|
Author | drstanl |
Post date | 2004-01-21T01:43:58Z |
Using Delphi 5 and Firebird 1.0.3:
I am reading a 16 million line text file,
parsing the data on each line, and storing the
resulting records into a table.
The key loop is as follows:
while not eof(Fi) do begin
readln(Fi,S);
Parse(S); { parse string "S" into structure "Rec" }
IB_Q.sql.clear;
IB_Q.sql.add(format('INSERT INTO T1 (ID1, ID2, D1, D2, V1) VALUES ('
+' ''%s'', ''%s'', ''%s'', ''%s'', %s )',
[Rec.ID1,Rec.ID2,Rec.D1,Rec.D2,Rec.V]));
IB_Q.execSQL;
end;
The table has a primary key consisting of the first 4
columns. Columns ID1, ID2, D1 and D2 are all VARCHARs of
certain lengths, and V1 is a double. The IB_Query is
connected to an IB_Connection. I haven't done anything
special with these other than set the minimally required
properties.
I'm sure this is the absolute slowest way to perform
this task. Nothing is "prepared", no tricks of the trade
are used.
I'd like to learn some of the more obvious and straight-
forward things I can do to make this process more efficient.
Thanks,
-- Stan
I am reading a 16 million line text file,
parsing the data on each line, and storing the
resulting records into a table.
The key loop is as follows:
while not eof(Fi) do begin
readln(Fi,S);
Parse(S); { parse string "S" into structure "Rec" }
IB_Q.sql.clear;
IB_Q.sql.add(format('INSERT INTO T1 (ID1, ID2, D1, D2, V1) VALUES ('
+' ''%s'', ''%s'', ''%s'', ''%s'', %s )',
[Rec.ID1,Rec.ID2,Rec.D1,Rec.D2,Rec.V]));
IB_Q.execSQL;
end;
The table has a primary key consisting of the first 4
columns. Columns ID1, ID2, D1 and D2 are all VARCHARs of
certain lengths, and V1 is a double. The IB_Query is
connected to an IB_Connection. I haven't done anything
special with these other than set the minimally required
properties.
I'm sure this is the absolute slowest way to perform
this task. Nothing is "prepared", no tricks of the trade
are used.
I'd like to learn some of the more obvious and straight-
forward things I can do to make this process more efficient.
Thanks,
-- Stan