|Subject||Bulk upload options|
I am looking for the fastest way to run a bulk insert.
I believe one way would be using isql.exe , I need to plan a solution to upload into a single table around 500 million records. Does anyone have experience with bulk updates via isql? Would it be X times faster than using a normal application inserting the records inside a begin transaction and commit at the end?
The issue is I have a remote database (not a FB database) and my options are reading the remote DB and writing into FB one record at the time with a loop, or asking the remote DB to dump the table into a txt file, then compressing the file with winrar or whatever compression, and then using that file to execute a bulk upload to FB," This is an ongoing issue, is not a one of, so I am trying to design the best performing solution, not a one of trick.
I know the volume of data will be around 250 Gb per run, hence that could be compressed at the server non-FB and sent to the FB server via internet, the compressed size would be probably just 5% of the uncompressed size as it is all text, so we are talking about 12 Gb aprox, the only question is "would then uploading the txt file into a table be quicker than a line by line insert into table?