| Subject | Odp: [firebird-support] Proper way for copy data from big table |
|---|---|
| Author | liviuslivius@poczta.onet.pl |
| Post date | 2015-06-19T14:31:29Z |
Hi,
I am not surprised about skip slowdown
I am talking about generic.
Something like "universal migrator".
I can imagine some trick with order by rdb$db_key but this solution can work with Firebird/Interbase but is not avaiable for other servers.
If table key is not single field then this is not simple. And also if table have no key at all.
Regards,
Karol Bieniaszewski
it has to count all of the records you want to skip.
You need an index of some sort on the data so that there is at least
some way of finding a record in the middle of the list. A simple
generator based ID number, or a timestamp is all you need to reduce the
time taken to find the starting point of the block you want to use.
--
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk
I am not surprised about skip slowdown
I am talking about generic.
Something like "universal migrator".
I can imagine some trick with order by rdb$db_key but this solution can work with Firebird/Interbase but is not avaiable for other servers.
If table key is not single field then this is not simple. And also if table have no key at all.
Regards,
Karol Bieniaszewski
----- Reply message -----
Od: "Lester Caine lester@... [firebird-support]" <firebird-support@yahoogroups.com>
Do: <firebird-support@yahoogroups.com>
Temat: [firebird-support] Proper way for copy data from big table
Data: pt., cze 19, 2015 15:58
Od: "Lester Caine lester@... [firebird-support]" <firebird-support@yahoogroups.com>
Do: <firebird-support@yahoogroups.com>
Temat: [firebird-support] Proper way for copy data from big table
Data: pt., cze 19, 2015 15:58
On 19/06/15 14:20, liviuslivius liviuslivius@...
[firebird-support] wrote:
> SELECT FIRST 1000000 SKIP 0 * FROM SOURCE_TABLE <-we got 1000000 reads -> okWhy would you expect any better when you are reading the data raw, and
> SELECT FIRST 1000000 SKIP 1000000 * FROM SOURCE_TABLE <-we got 2000000
> reads -> not ok
> SELECT FIRST 1000000 SKIP 2000000 * FROM SOURCE_TABLE <-we got 3000000
> reads -> not ok worser and worser (slower and slower)
it has to count all of the records you want to skip.
You need an index of some sort on the data so that there is at least
some way of finding a record in the middle of the list. A simple
generator based ID number, or a timestamp is all you need to reduce the
time taken to find the starting point of the block you want to use.
--
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk