Subject Re: bulk insert
Author Dorin Vasilescu
Hi
The best throughput I've achieved without external tables was by
constructing a blob type text consisting in lines with data,each ended
with line feed and then sending the blob to a stored procedure that
count lines , parses it (using UDFs-rfunc b_line()) and build insert
statements on the fly, next execute statements with 'EXECUTE STATEMENT'
I had 5300 rows/second as result in a DB with Force writes off.
The best result was by sending 50-100 rows blob one at a time (I
believe because algorithms in blob UDFs).

Here is the procedure
CREATE PROCEDURE BULK_INSERT (
DEST_TABLE VARCHAR(255),
COLUMNS_LIST VARCHAR(1000),
COLUMNS_VALUES BLOB SUB_TYPE 1)
RETURNS (
OP_LC INTEGER)
AS
DECLARE VARIABLE LINE_COUNT INTEGER = 0;
DECLARE VARIABLE COUNTER INTEGER = 0;
DECLARE VARIABLE SQL_STMT VARCHAR(1000) = '' ;
DECLARE VARIABLE CURRENT_LINE VARCHAR(1000) = '' ;
begin
LINE_COUNT = B_LINE_COUNT(COLUMNS_VALUES) ;
WHILE ( COUNTER < LINE_COUNT) DO BEGIN
COUNTER = COUNTER + 1 ;
CURRENT_LINE = B_LINE( :COLUMNS_VALUES , :COUNTER ) ;
SQL_STMT = 'INSERT INTO ' || :DEST_TABLE || ' (' ||
:COLUMNS_LIST || ') VALUES ( ' ||
:CURRENT_LINE || ')' ;
EXECUTE STATEMENT :SQL_STMT ;
END
op_lc = line_count ;
suspend ;
end


>
> currently i select the relevant data from the source databases into
> an ado.net dataset, open a connection to the firebird db, create a
> table in the local firdbird database based on the schema of the
> dataset, then step thru each of the records in that dataset and
> generate an insert statement to insert the data into the firebird
> database, and finally close the connection.
>
> if the dataset only contained a 100,000 records the time would be
> acceptable at about a second per 1000 records, but at a million or 2
> reconds this just takes too long. i'm hoping that there exists a
> solution that can insert the data into the firebird in close to the
> same time that it takes to get the data from the source databases.
> ex, a million or so records from the source databases takes about 3
> minutes.
>
> is it possible to select the data from the source databases directly
> into the firebird database?
>
> i'm using the 1.6.3 .net provider and the 1.5.2.4731 embedded
> firebird engine, and am not at all oposed to changing either if it
> will help.
>
> help with either of these questions is much appreciated.
> gabe