Subject | RE: [IBO] How to improve bulk insert |
---|---|
Author | Jason Wharton |
Post date | 2006-01-24T18:47:23Z |
Keep in mind if you do a Commit it could cause your source cursor to close.
My understanding of Firebird/InterBase is that these incremental commits
should not be necessary. The only thing that would be painful is if you did
lots of them and then had to do a rollback. Otherwise, it is optimistic and
a commit is simply an adjustment to the transaction and all the record
versions stay just as they are. In which case, why do incremental commits
along the way?
Jason
My understanding of Firebird/InterBase is that these incremental commits
should not be necessary. The only thing that would be painful is if you did
lots of them and then had to do a rollback. Otherwise, it is optimistic and
a commit is simply an adjustment to the transaction and all the record
versions stay just as they are. In which case, why do incremental commits
along the way?
Jason
> -----Original Message-----
> From: IBObjects@yahoogroups.com [mailto:IBObjects@yahoogroups.com]On
> Behalf Of henry FRANQUET
> Sent: Monday, January 23, 2006 3:42 PM
> To: IBObjects@yahoogroups.com
> Subject: Re: [IBO] How to improve bulk insert
>
>
> --- In IBObjects@yahoogroups.com, Aage Johansen <aagjohan@o...> wrote:
> >
> > henry FRANQUET wrote:
> > > Hi,
> > > I use TIB_DataPump and IB_cursor to insert data into a
> database (a
> > > empty one created with backup and restore only metadata).
> > > As I have Tables with about one hundrer thousand
> records, I perform a
> > > commit retainning on after execute event.
> ---snip ---
> > > How can I improve inserting ? Is there a counter I could use, for
> > > doing a commit every 1000 inserts ?
> > >
> > > I have seen in forums, somebody's speaking of inserting
> thousand's
> > > insert a second, but even when doing insert without
> IB_DataPump and
> > > committing every 1000 insert, I can't go further 400
> inserts a second
> > > with 2 fields table. Should I use virtual disk for
> temporary files,
> > > disable forced write and enable it after inserts done ?
> > >
> >
> > It depends on several factors: hardware (cpu/disk).
>
> Yes, I have seen than the size of the cache of the disk is critical.
> And disabling Forced write, just increased the speed by a factor of 2
>
> > Aslo, are records
> > produced by a fast system?, do you prepare more than once?,
>
> No TIB_Datapump does the job
>
> > are there triggers?, any indexes to be maintained?
>
> None, I have disabled all triggers and indexes
>
> > I haven't used the Datapump, but TIB_DSQL is quite fast when you're
> doing
> > simple inserts (and otherwise, as well).
>
> I'll try to replace my TIB_Cursor with a TIB_DSQL
>
> > I wouldn't worry if commits were done every 10.000 records (or even
> less
> > frequently).
> It's done after inserting every table, and so about 50.000 to 100.000
> records. I have tried to commit on the event AfterExecuteItems, every
> 5000 records, but I get an error witch I can't explain
> > A few thousand inserts a second is not an ambitious goal. Even on
> > equipment that isn't "state of the art".
> Yes, but even with forcedWrite off, I hardly get one thousand.
> >
> > --
> > Aage J.
> Thanks for your answer
>