Subject | Re: [Firebird-Architect] Feature Request |
---|---|
Author | Jim Starkey |
Post date | 2005-04-30T18:45:39Z |
Christian Stengel wrote:
enable the indexes. The indexes will be built using a "fast load"
scheme from a sorted data stream, which is much faster than incremental
index updates. This does not require additional database support.
the database.
algorithm loses big time for short items. A 6 character string, for
example, bloats in 118 bytes. The algorithm is space efficient only for
large strings.
There has been some thought and discussion given to smarter compression
schemes than run length encoding, but a good candidate has not yet emerged.
itself or you wish to write your own external file manager. External
files are useful to import or export data or to interact with an
application that uses RMS. It is not intended to every be an
alternative record storage format.
>The way to do a fast load is to load the data with indexes disabled then
>1. fast import utility
>Currently I have a project where I get 35.000.000 records per day from
>my biggest source. In Firebird 1.5 this takes 75 minutes to import with
>native c api (24 Minutes for processing the data and 61 Minutes for the
>import - on an PowerBook - the server will be faster :-)). (Currently
>it's not clear which database system we will take - it depends on the
>performance :-) ).
>
>
enable the indexes. The indexes will be built using a "fast load"
scheme from a sorted data stream, which is much faster than incremental
index updates. This does not require additional database support.
>2. Splitting tables internally into multiple tablesThis is properly done by a layered utility. It should not be part of
>This has already been requested in this list but was rejected as there
>was no example where it will be useful.
>
>
the database.
>3. metric calculationsThat's what UDFs are for. Use them.
>Ann sometimes asked whether there is a need for spartial extensions
>(like in GIS applications). It would also be great, if firebird could
>calculate with metrics or pysical data (e.g. the field is defined as
>square meters and I can cast it to sqare miles, kilometers or something
>else - like kilogram, pounds ...., calculate with it properly ...).
>
>
>4. compressed data fieldRecordds are already run-length compressed. The zip compression
>if I know, that I am not going to search for a field - it would be
>great, if the value of this field could be stored compressed (e.g. zip
>or bzip2 compressed) to save space :-).
>
>
algorithm loses big time for short items. A 6 character string, for
example, bloats in 118 bytes. The algorithm is space efficient only for
large strings.
There has been some thought and discussion given to smarter compression
schemes than run length encoding, but a good candidate has not yet emerged.
>5. Index file for external tablesNot a chance in the world unless the external file manager does it
>there are external tables in firebird - howabout an additional file
>with index information? The index gets updated when data is inserted by
>the engine or when recreated with alter index. So eveyone knows: when
>changing the data with an external program - the index has to be
>recreated :-)
>
>
itself or you wish to write your own external file manager. External
files are useful to import or export data or to interact with an
application that uses RMS. It is not intended to every be an
alternative record storage format.