Subject | Re: [firebird-support] INSERT efficiency |
---|---|
Author | Markus Ostenried |
Post date | 2004-01-21T04:01:52Z |
At 01:43 Wednesday, 21.01.2004 +0000, you wrote:
as Alexandre said, it would be best to use parameters in your statement.
Since it looks like your're using IBObjects I suggest you use a TIB_DSQL
instead of a TIB_Query. TIB_Query has a buffer which isn't needed if you
just want to insert. And with 16 million records it would be good to not
call ParamByName() in the loop.
Here's an example:
var
DSQL: TIB_DSQL;
Param1, Param2: TIB_Column;
RecCnt: Integer;
begin
DSQL := TIB_DSQL.Create( Self );
try
with DSQL do begin
IB_Connection := YourConnection;
SQL.Add( 'INSERT INTO T1 (ID1, ID2)' );
SQL.Add( ' VALUES ( :ID1, :ID2 )' );
Prepare;
Param1 := ParamByName( 'ID1' );
Param2 := ParamByName( 'ID2' );
IB_Transaction.StartTransaction;
end;
RecCnt := 0;
while not eof(Fi) do begin
readln(Fi,S);
Parse(S); { parse string "S" into structure "Rec" }
Param1.AsInteger := Rec.ID1;
Param2.AsInteger := Rec.ID2;
DSQL.Execute;
Inc( RecCnt );
if ((RecCnt mod 10000) = 0) and DSQL.IB_Transaction.Started then begin
DSQL.IB_Transaction.Commit;
DSQL.IB_Transaction.StartTransaction;
end;
end;
finally
DSQL.Free;
end;
end;
HTH,
Markus
> Using Delphi 5 and Firebird 1.0.3:...
>
> I am reading a 16 million line text file,
>parsing the data on each line, and storing the
>resulting records into a table.
> I'd like to learn some of the more obvious and straight-Hi,
>forward things I can do to make this process more efficient.
as Alexandre said, it would be best to use parameters in your statement.
Since it looks like your're using IBObjects I suggest you use a TIB_DSQL
instead of a TIB_Query. TIB_Query has a buffer which isn't needed if you
just want to insert. And with 16 million records it would be good to not
call ParamByName() in the loop.
Here's an example:
var
DSQL: TIB_DSQL;
Param1, Param2: TIB_Column;
RecCnt: Integer;
begin
DSQL := TIB_DSQL.Create( Self );
try
with DSQL do begin
IB_Connection := YourConnection;
SQL.Add( 'INSERT INTO T1 (ID1, ID2)' );
SQL.Add( ' VALUES ( :ID1, :ID2 )' );
Prepare;
Param1 := ParamByName( 'ID1' );
Param2 := ParamByName( 'ID2' );
IB_Transaction.StartTransaction;
end;
RecCnt := 0;
while not eof(Fi) do begin
readln(Fi,S);
Parse(S); { parse string "S" into structure "Rec" }
Param1.AsInteger := Rec.ID1;
Param2.AsInteger := Rec.ID2;
DSQL.Execute;
Inc( RecCnt );
if ((RecCnt mod 10000) = 0) and DSQL.IB_Transaction.Started then begin
DSQL.IB_Transaction.Commit;
DSQL.IB_Transaction.StartTransaction;
end;
end;
finally
DSQL.Free;
end;
end;
HTH,
Markus