Subject | Re: Import of > 1 mio. records |
---|---|
Author | thomas_besand |
Post date | 2005-03-18T22:49:14Z |
Thanks everybody for your suggestions. Took me some time to change the
code according to your propositions; and after all- it's Friday
afternoon- and I was out on my motorbike with my son ;-)
Back to work:
throw that into the actual writing routine- sounds a bit more
complicated than necessary, but in fact I do some kind of analysing
and preprocessing in those steps.
I put an index on the newly created table only after importing.
Well anyway, I implemented the COMMITs on every 10000 writes, as Dan
and others proposed, so now Firebird grabs itself around 16k of memory
and stays stable at that. If I take a look at the logfile, I can see
it's taking roughly 2 min/10000 records steadily. So this side begins
to look good.
I thought at first that if AutoCommit was turned on for the
ZeosConnection, I wouldn't need to bother about committing explicitly.
This seems obviously not true.
Thanks again for your support
Greets from Berlin
Thomas Besand
code according to your propositions; and after all- it's Friday
afternoon- and I was out on my motorbike with my son ;-)
Back to work:
> > I noticed that in a long running import routine the firebirdI'm reading a text file line by line, split it up in a StringList and
> > server(1.5.2.4731) is getting slower and slower until it virtually
> > grinds to a standstill.
>
> Do you just call INSERTs, or do you perform some
> additional processing in stored procedures ?
throw that into the actual writing routine- sounds a bit more
complicated than necessary, but in fact I do some kind of analysing
and preprocessing in those steps.
I put an index on the newly created table only after importing.
> What tool/components do you use ?zeoslib, Delphi 5
Well anyway, I implemented the COMMITs on every 10000 writes, as Dan
and others proposed, so now Firebird grabs itself around 16k of memory
and stays stable at that. If I take a look at the logfile, I can see
it's taking roughly 2 min/10000 records steadily. So this side begins
to look good.
I thought at first that if AutoCommit was turned on for the
ZeosConnection, I wouldn't need to bother about committing explicitly.
This seems obviously not true.
Thanks again for your support
Greets from Berlin
Thomas Besand