Well that just took all the wind out my sails..

The varchar led me to beleive it might not be limited to fixed length after
all. I can live without blob data, but not without nulls. Also, padding
would increase my file sizes by a factor of 10-20 times ( Unused field space
+ a lot of 50 character nulls add up quickly). I need to be able to
preserve bandwidth here as well. So what would normally be a 100 meg file
blowing up into 2 gig file to transfer is unworkable.

I guess the API solution is it, or I am stuck at an unworkable ceiling of
300 tps from VB.

I would do just about anything right now for some VB6 Interbase components.

Edwin Epstein

> Continuity Partners Inc.
> 2753 S. Highland Dr.
> Suite 1010
> Las Vegas, Nevada 89109
> (702) 851-7850 Ext. 247
> eepstein@...
>

CONFIDENTIALITY NOTICE: The information contained in this message may be
privileged and confidential and protected from disclosure.
If the reader of this message is not the intended recipient, or an employee
or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any dissemination,
distribution or copying of this communication is
strictly prohibited. If you have received this communication in error,
please notify us immediately by replying to the message and
deleting it from your computer.




-----Original Message-----
From: Ivan Prenosil [mailto:Ivan.Prenosil@...]
Sent: Monday, February 09, 2004 11:38 AM
To: firebird-support@yahoogroups.com
Subject: Re: [firebird-support] External File Construction


The rules are really simple

* NULLs and BLOBs can't be stored in external tables.
* everything is stored as fixed length values.

That's all !

E.g.
INTEGER = 4 bytes
SMALLINT = 2 bytes
TIMESTAMP = 2 integers = 8 bytes
CHAR(10) = 10 bytes
VARCHAR(10) = 2+10 = 12 bytes

Fixed length means that even single character
stored in VARCHAR(10) will still require 12 bytes
(for single byte character sets).

If you expect that external file contains CRLF or LF,
you have to create appropriate column for it
(either CHAR(1) or CHAR(2) or SMALLINT).

Your declaration
(A VarChar(10),B VarChar(2),C VarChar(6));
means that one record is 2+10 + 2+2 + 2+6 = 24 bytes, no crlf expected.

Ivan
http://www.volny.cz/iprenosil/interbase/

> Double Precisions, Integers, Small Integers, and Time stamps are all
> represented with the appropriate number of bytes. These operate as fixed
> length values. I still dont have an exact idea of how these values are
> constructed properely, left-right, right-left, signed, etc. I will try to
> find an algorithm for constructing timestamp values.
>
> Text fields can use the Char(X) specification, but would require padding
> values and be very inefficient on disk usage and external file creation.
> Interestingly enough, you can use VarChar for a field specification.
> However, it operates with unexpected results. 2 bytes must precede the
> actual field value. You would think these 2 bytes indicate how many bytes
> to use as the field data, but it does not work that way. The number
> indicated in the file specifications overrides this. So although a

Varchar

> value can be used, its still operating as a fixed length field and STILL
> requires padding of values at the end. The 2 bytes are effectively

useless.

> I tested this by creating a file with 3 fields. 10,2, and 6 character

field

> lengths for each field per row. All field values had hex values of 000A,
> 0002, or a 0006 as the 2 bytes preceding the field value itself. The file
> specifications where (A VarChar(10),B VarChar(2),C VarChar(6));. These
> specifications worked perfectly. However, when ...(A VarChar(15),... was
> used, it would then grab B's 2 byte length values as well as B itself and
> would then screw up the rest of the file. If 0000 were used as a byte
> length value (my thoughts on how to indicate null), it actually stopped
> processing of the entire file itself as if EOF had been reached.
>
> I am now completely lost on how to properly represent a null value in an
> external file if now even possible at all. Also, a null value inside of a
> number is an interesting question since those values do not have bytes in
> front of them to represent their length and 00 00 00 00 would represent 0

as

> an integer which is still a value.
>
> If anybody has detailed knowledge on external files, their creation, and
> corresponding file specifications i would be very gratefull.

Yahoo! Groups Links