Subject | RE: [firebird-support] External File Construction |
---|---|
Author | Epstein, Ed |
Post date | 2004-02-10T01:24:11Z |
All data is in either CSV format or in some limited cases DBF v4-5 during
import and export.
There is only a single IB Server. Replication of data from a central server
to other servers is not possible in this instance, since the front-end user
is not part of our corporation and would not likely install it. I don't
know if its truly neccessary at this point since a firehose cursor is fairly
fast and I have not had any problem in retrieving hundreds of thousands of
records at one time remotely. A server next to IB server could be made to
construct the file and allow the remote client to download it.
As for the number of rows, different tables, and frequency there are on
average 3 to 4 major files a week being processed in house. Each of these
files is at least 500,000 rows in CSV format. There are also around 200-300
smaller files each week that in total do not exceed 10,000 rows. The major
files all get moved into a single table that has a stored procedure run on
it which processes every record and moves it to 1 of 11 different tables
based on the processing results. I would like to bypass that by running the
stored procedure directly from the front-end but that takes too much time
since the effective ceiling for that has been tested at <100 tps.
Edwin Epstein
privileged and confidential and protected from disclosure.
If the reader of this message is not the intended recipient, or an employee
or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any dissemination,
distribution or copying of this communication is
strictly prohibited. If you have received this communication in error,
please notify us immediately by replying to the message and
deleting it from your computer.
-----Original Message-----
From: Alan McDonald [mailto:alan@...]
Sent: Monday, February 09, 2004 4:58 PM
To: firebird-support@yahoogroups.com
Subject: RE: [firebird-support] External File Construction
different tables and how frequently do you dispatch it?
I take it these remote users/server networks are not visible to the internet
infrastructure? I take it that replication is not possible from a central
location to these servers?
Alan
Yahoo! Groups Links
import and export.
There is only a single IB Server. Replication of data from a central server
to other servers is not possible in this instance, since the front-end user
is not part of our corporation and would not likely install it. I don't
know if its truly neccessary at this point since a firehose cursor is fairly
fast and I have not had any problem in retrieving hundreds of thousands of
records at one time remotely. A server next to IB server could be made to
construct the file and allow the remote client to download it.
As for the number of rows, different tables, and frequency there are on
average 3 to 4 major files a week being processed in house. Each of these
files is at least 500,000 rows in CSV format. There are also around 200-300
smaller files each week that in total do not exceed 10,000 rows. The major
files all get moved into a single table that has a stored procedure run on
it which processes every record and moves it to 1 of 11 different tables
based on the processing results. I would like to bypass that by running the
stored procedure directly from the front-end but that takes too much time
since the effective ceiling for that has been tested at <100 tps.
Edwin Epstein
> Continuity Partners Inc.CONFIDENTIALITY NOTICE: The information contained in this message may be
> 2753 S. Highland Dr.
> Suite 1010
> Las Vegas, Nevada 89109
> (702) 851-7850 Ext. 247
> eepstein@...
>
privileged and confidential and protected from disclosure.
If the reader of this message is not the intended recipient, or an employee
or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any dissemination,
distribution or copying of this communication is
strictly prohibited. If you have received this communication in error,
please notify us immediately by replying to the message and
deleting it from your computer.
-----Original Message-----
From: Alan McDonald [mailto:alan@...]
Sent: Monday, February 09, 2004 4:58 PM
To: firebird-support@yahoogroups.com
Subject: RE: [firebird-support] External File Construction
> Its the end-users themselves. remote users would have the dataOK, then in what form is this original data? and how many rows, how many
> packaged up
> and sent to a server where it would it would then be inserted. So all
> inserts would be done on the same local network where the IB
> server resides.
> admin users or "staff" would also be on the local network
> performing inserts
> of data we receive through other channels.
>
> Edwin Epstein
different tables and how frequently do you dispatch it?
I take it these remote users/server networks are not visible to the internet
infrastructure? I take it that replication is not possible from a central
location to these servers?
Alan
Yahoo! Groups Links