Subject | import data from external table (huge file) |
---|---|
Author | zimrilin2000 |
Post date | 2005-08-24T11:43:01Z |
Hi all!
I'm trying to import the data from a 2.6 Gb fixed length format
external file with 35 million records into a regular table inside the
database. Each line in the external table file is 76 characters long.
I'm using a stored procedure to do the import, because I need to
detect and store duplicates from the file in another table (violation
of table primary key) and continue with the processing
I execute the SP (EXECUTE PROCEDURE INSERT_EXTERNAL_TABLE) with EMS
QuickDesk and after 2 hour processing (the bdd file grows until it
gets 5 Gb size) I get the following error:
EXECUTE PROCEDURE INSERT_EXTERNAL_TABLE
fmSQLScript.Script:
Unsuccessful execution caused by a system error that precludes
successful execution of subsequent statements.
I/O error for file "D:\SRC\INSERTFILES\DATA.DAT".
Error while trying to open file.
El dispositivo no reconoce el comando.
.
I'm using Firebird Firebird-1.5.2.4731_win32
This is the stored procedure
CREATE PROCEDURE INSERT_EXTERNAL_TABLE
AS
DECLARE VARIABLE A VARCHAR(13);
DECLARE VARIABLE B VARCHAR(10);
DECLARE VARIABLE C VARCHAR(9);
DECLARE VARIABLE D VARCHAR(10);
DECLARE VARIABLE E VARCHAR(20);
BEGIN
/* Procedure body */
FOR SELECT A,B,C,D,E FROM EXTERNAL_TABLE
INTO :A, :B, :C, :D, :E
DO
BEGIN
INSERT INTO TABLE (A,B,C,D,E)
VALUES (:A, :B, :C, :D, :E);
WHEN SQLCODE -803 DO
BEGIN
INSERT INTO TABLE_TMP(A,B,C,D,E)
VALUES (:A, :B, :C, :D, :E);
END
END
EXIT;
END
Is the file too big to do the import?
Is there any other way of doing the import?
Thanks
Diego
I'm trying to import the data from a 2.6 Gb fixed length format
external file with 35 million records into a regular table inside the
database. Each line in the external table file is 76 characters long.
I'm using a stored procedure to do the import, because I need to
detect and store duplicates from the file in another table (violation
of table primary key) and continue with the processing
I execute the SP (EXECUTE PROCEDURE INSERT_EXTERNAL_TABLE) with EMS
QuickDesk and after 2 hour processing (the bdd file grows until it
gets 5 Gb size) I get the following error:
EXECUTE PROCEDURE INSERT_EXTERNAL_TABLE
fmSQLScript.Script:
Unsuccessful execution caused by a system error that precludes
successful execution of subsequent statements.
I/O error for file "D:\SRC\INSERTFILES\DATA.DAT".
Error while trying to open file.
El dispositivo no reconoce el comando.
.
I'm using Firebird Firebird-1.5.2.4731_win32
This is the stored procedure
CREATE PROCEDURE INSERT_EXTERNAL_TABLE
AS
DECLARE VARIABLE A VARCHAR(13);
DECLARE VARIABLE B VARCHAR(10);
DECLARE VARIABLE C VARCHAR(9);
DECLARE VARIABLE D VARCHAR(10);
DECLARE VARIABLE E VARCHAR(20);
BEGIN
/* Procedure body */
FOR SELECT A,B,C,D,E FROM EXTERNAL_TABLE
INTO :A, :B, :C, :D, :E
DO
BEGIN
INSERT INTO TABLE (A,B,C,D,E)
VALUES (:A, :B, :C, :D, :E);
WHEN SQLCODE -803 DO
BEGIN
INSERT INTO TABLE_TMP(A,B,C,D,E)
VALUES (:A, :B, :C, :D, :E);
END
END
EXIT;
END
Is the file too big to do the import?
Is there any other way of doing the import?
Thanks
Diego