Subject | Re: import data from external table (huge file) |
---|---|
Author | zimrilin2000 |
Post date | 2005-08-25T07:50:37Z |
Hi,
I have enough space in the hard disk (about 300 Gb), and I did
commit after executing the stored procedure.
Past nigth I have tested the insertion in other way: I have split
the file in 35 files with 1 million records in each file. Until file
number 26, files are imported in 2 minutes time (I insert a log
record with the times in a table between inserts, after commiting),
but file 27 is taking 20 minutes, and after that each file takes 5
more minutes than previous file. File 35 takes 1 hour to import.
The table definition is
CREATE TABLE TABLE1(
A VARCHAR(13),
B VARCHAR(10),
C VARCHAR(9),
D VARCHAR(32),
E SMALLINT,
DATE1 DATE,
FLAG SMALLINT,
);
The primary key is C and D. The selectivity of field D is poor, but
it is no foreign key of any table, only composite key of this table
Should I use a surrogate key for the table although is not used as a
foreign key?
Am I reaching any limit I don't know?
Any ideas/suggestions?
Diego
--- In firebird-support@yahoogroups.com, Helen Borrie <helebor@t...>
wrote:
I have enough space in the hard disk (about 300 Gb), and I did
commit after executing the stored procedure.
Past nigth I have tested the insertion in other way: I have split
the file in 35 files with 1 million records in each file. Until file
number 26, files are imported in 2 minutes time (I insert a log
record with the times in a table between inserts, after commiting),
but file 27 is taking 20 minutes, and after that each file takes 5
more minutes than previous file. File 35 takes 1 hour to import.
The table definition is
CREATE TABLE TABLE1(
A VARCHAR(13),
B VARCHAR(10),
C VARCHAR(9),
D VARCHAR(32),
E SMALLINT,
DATE1 DATE,
FLAG SMALLINT,
);
The primary key is C and D. The selectivity of field D is poor, but
it is no foreign key of any table, only composite key of this table
Should I use a surrogate key for the table although is not used as a
foreign key?
Am I reaching any limit I don't know?
Any ideas/suggestions?
Diego
--- In firebird-support@yahoogroups.com, Helen Borrie <helebor@t...>
wrote:
> At 03:58 PM 24/08/2005 +0000, you wrote:that red
> >So, which is the limit of the table size and why is failing the
> >import?
>
> Numbers as quoted - which is 2 BILLION not 2 million. Sorry about
> herring.The
>
>
> >I just have tried to split the file in two 17 million rows files.
> >first file is copied into internal file without problems in 45running
> >minutes, but the second one has failed (the process is still
> >after 3 hours processing)
>
> How do you know it has failed?
>
> Did you hard-commit the first batch before commencing the second?
>
> ./heLen