Subject | Re: [firebird-support] Bulk Loading for FB? |
---|---|
Author | David Johnson |
Post date | 2005-05-27T22:46:05Z |
Look at using Java/JDBC. Your solution will be cross-platform without
any extra coding, and over 90% of data type changes can be accomplished
without any coding on your part. If data model changes are required,
you can use XSLT to restructure the data in between. I currently push
over 1000 rows per second (with a full transform step) sustained for 50
minutes using this mechanism.
As Ann says, if you possibly can, run the migration app on the server
hardware that you are migrating the data to. Firebird's wire protocol
is not that efficient (yet ... just waiting for Vulcan).
Good luck.
any extra coding, and over 90% of data type changes can be accomplished
without any coding on your part. If data model changes are required,
you can use XSLT to restructure the data in between. I currently push
over 1000 rows per second (with a full transform step) sustained for 50
minutes using this mechanism.
As Ann says, if you possibly can, run the migration app on the server
hardware that you are migrating the data to. Firebird's wire protocol
is not that efficient (yet ... just waiting for Vulcan).
Good luck.
On Fri, 2005-05-27 at 15:03 -0400, Ann W. Harrison wrote:
> Hi Bill -
> >
> > I am looking for some guidance on bulk loading data into Firebird. What
> > I want to do is turn off integrity constraints and just load the data
> > as fast as possible.
>
> Well, one approach is to turn off integrity checks. If this is
> something you do a log, I'd create two DDL scripts for the database -
> one that defines only columns and tables and one that adds constraints,
> triggers, indexes, etc. Run the first, load the data, then run the
> second.
> >
> In terms of actually loading the data, the fastest approach is to format
> the data in files with fixed length columns, map those files to external
> tables, and load the internal tables with
>
> insert into <internal table> select from <external table>
>
> Of course, that gets messy if you've got nulls, but few non-database
> data sources represent nulls well.
>
> Failing that, write a preprocessed application that reads your data
> source and stores the data. It should run on the server - classic on
> Unix type systems, embedded for Windows.
>
> Regards,
>
>
> Ann
>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> Visit http://firebird.sourceforge.net and click the Resources item
> on the main (top) menu. Try Knowledgebase and FAQ links !
>
> Also search the knowledgebases at http://www.ibphoenix.com
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> Yahoo! Groups Links
>
>
>
>
>
>