On June 07, 2012 6:08 PM Doug Chamberlin wrote:
> Yes. On further thought I think I would make the whole ETL process a
> two-pass affair. First pass check for existence of all needed
entities
> and creates ones that are missing. Second pass should do the
inserting
> proper without errors.

That is actually an interesting idea that I hadn't thought of. We are
dealing with fairly large files - 250k to 900k lines in file, so I'm
not sure if the overhead of doing two-pass processing can be absorbed,
but it's an option to try.

As to your other idea of processing the FK entries in a seperate
transaction - that is the first idea that I had, but we wound up with
serious issues when we last tried that - it's been about a year and a
half or so, and I don't remember the specifics right now. Maybe I'll
go back and check that one more time...

Thanks.