Subject Re: Massive INSERT test
Author arbitragex <arbitragex@yahoo.com>
> How selective are your indexes? I.e. will there be lots of
duplicates? If
> you were doing an insert like this, I would assume your best bet
would be
> to turn off the indexes during the insert and then turn them back
on
> afterwards.

Only one indice isn't unique. For the one which isn't I can't answer
the question right now.

However I need the data to be indexed at all times how would I know
if I am not creating duplicates otherwise?

> 10,000 records in 34 seconds sounds too slow - I'm not used to test
massive
> inserts, but selects can manage 10,000 records in 1 or 2 seconds.
How are
> you inserting? Parameterized dSQL I presume, but do you use
ParamByName or

This is my embedded SQL statement:

INSERT INTO myDB (myCols ... ) VALUES( ... )

Not sure if it answers your question.

> Params or whatever (I come from an IBO background)? And what about
> transactions? I assume commit every 10,000 or 20,000 would be OK.

The data is "fault-tolerant". It is a snapshot of certains processes
and I can afford a great deal of data loss. Data errors would be
more of a concern of course ...

> >Insert semi-random data benchmark:
> >
> >2,000,000 0:53 sec
> >2,200,000 1:08
> >2,300,000 1:19
> >2,400,000 1:48
> >2,500,000 1:51
> >2,600,000 2:07
> >2,700,000 2:07
> >2,800,000 2:19
> >2,860,000 4:51
> >2,900,000 4:47
> >
> >ibserver.exe used 40 MB memory at that point.
> >
> >I stopped the test and restarted the service.
> >
> >3,000,000 1:28
> >3,100,000 1:43
> >3,200,000 1:50
> >3,300,000 2:14
> >3,400,000 2:06
> >
> >ibserver.exe is up to 10 MB memory usage at this time.
>
> Sorry, I don't understand your benchmarks at all. Increasing
numbers from
> 2,000,000 up to 3,400,000 followed by seconds that mostly, but not
> exclusively increases as the other number increase??

It's the non cumulative time it takes to insert 10,000 rows. The
left figure is the current number of rows in the table.

Thanks