Subject | Handling large imports while system is in production |
---|---|
Author | rudibrazil78 |
Post date | 2013-02-05T13:21:01Z |
Hi all
We've been having to deal with a bad situation for quite a few years now - how to handle large imports without taking the server down (in fact that data is meant to be acessible as soon as its ready in production).
This is a call center software which routinely imports lists of clients and possible clients to contact.
The import happens on 2 main tables, clients and calls, which are exactly the same tables the operators will use/update while working.
These are very large tables (over 200 fields each), with many fk's (and a lot of the fk indexes are reportedly bad ones).
Not only importing is taking very long, but while importing is happening, performance in production takes a severe hit.
I know theres no magic wand, but I seriously need to start making a checklist of practices I should be trying out to contour this situation.
Thanks in advance...please let me know if I can clarify the issue further.
We've been having to deal with a bad situation for quite a few years now - how to handle large imports without taking the server down (in fact that data is meant to be acessible as soon as its ready in production).
This is a call center software which routinely imports lists of clients and possible clients to contact.
The import happens on 2 main tables, clients and calls, which are exactly the same tables the operators will use/update while working.
These are very large tables (over 200 fields each), with many fk's (and a lot of the fk indexes are reportedly bad ones).
Not only importing is taking very long, but while importing is happening, performance in production takes a severe hit.
I know theres no magic wand, but I seriously need to start making a checklist of practices I should be trying out to contour this situation.
Thanks in advance...please let me know if I can clarify the issue further.