Subject | RE: [ib-support] Re: Out of memory |
---|---|
Author | Alan McDonald |
Post date | 2003-02-10T12:13:07Z |
The unidirectional setting certainly reduced memory requirements. That's
probably what made the difference for you.
Hopefully you have a hierarchy in your record schema, so that for so many
child records (228,000) you have a structure of parent records. e.g. rooms,
inside buildings, inside complexes, inside precincts.
You could start by selecting distinct precincts, then looping thru the
precincts, select all the complexes (for that precinct), then looping thru
the complexes, select all the buildings (for the complex), then looping thru
the buildings, select all the rooms (for that building)... that way a select
will only have the rooms of one building at a time. You'd be very surprised
what a speed increase you will get by processing in this way. You would be
even better off doing this processing on the server under SP control.
Alan
-----Original Message-----
From: Michael Vilhelmsen <Michael.Vilhelmsen@...>
[mailto:Michael.Vilhelmsen@...]
Sent: Monday, 10 February 2003 10:46 PM
To: ib-support@yahoogroups.com
Subject: [ib-support] Re: Out of memory
That means, that I should do my select in portions instead ?
Michael
probably what made the difference for you.
Hopefully you have a hierarchy in your record schema, so that for so many
child records (228,000) you have a structure of parent records. e.g. rooms,
inside buildings, inside complexes, inside precincts.
You could start by selecting distinct precincts, then looping thru the
precincts, select all the complexes (for that precinct), then looping thru
the complexes, select all the buildings (for the complex), then looping thru
the buildings, select all the rooms (for that building)... that way a select
will only have the rooms of one building at a time. You'd be very surprised
what a speed increase you will get by processing in this way. You would be
even better off doing this processing on the server under SP control.
Alan
-----Original Message-----
From: Michael Vilhelmsen <Michael.Vilhelmsen@...>
[mailto:Michael.Vilhelmsen@...]
Sent: Monday, 10 February 2003 10:46 PM
To: ib-support@yahoogroups.com
Subject: [ib-support] Re: Out of memory
That means, that I should do my select in portions instead ?
Michael
--- In ib-support@yahoogroups.com, "Alan McDonald" <alan@m...> wrote:
> Sounds to me like it was telling you something very sensible.
> Let's imagine for a moment that your database table grows over time
until it
> has 50 million records. Does your current design in any way cater
for this?
> To select 50 million records (or even 228,000 records) in one hit
is not a
> good idea. There are many ways to design your processing such that
only
> small numbers of records are snatched off the server at any one
time. I
> would seriously think of these ways and ways in which you can
normalise your
> data such that there is little or no need to see all records in a
table in
> one select. Your performance will improve a hundred or more fold
too. You
> might be very surprised.
>
> Alan
>
> -----Original Message-----
> From: Michael Vilhelmsen <Michael.Vilhelmsen@M...>
> [mailto:Michael.Vilhelmsen@M...]
> Sent: Monday, 10 February 2003 9:12 PM
> To: ib-support@yahoogroups.com
> Subject: [ib-support] Re: Out of memory
>
>
> To answer my own question.
>
> I set UniDirectional to true, and the problem was gone.
>
> But I still would like to know why I got the error in the first
place.
> And if I could do something to avoid it !
>
> Michael
>
> --- In ib-support@yahoogroups.com, "Michael Vilhelmsen
> <Michael.Vilhelmsen@M...>" <Michael.Vilhelmsen@M...> wrote:
> > Hi
> >
> > I have an appl. that contects to an Firebird DB (1.0).
> > In my appl. I have made a rutine, that select 228.000 records, and
> > parse through each of them.
> > For each record I call a stored procedure that either update
> another
> > record or insert a record.
> >
> > That'll say it will update / insert up to 228.000 records.
> >
> > Does two query both uses the same transaction.
> >
> > Before starting I do a
> >
> > if not MyTrans.active then
> > MyTrans.StartTransaction
> >
> > After the rutine has run through I do
> >
> > MyTrans.Commit
> >
> > On a smaller dataset it completes with no error.
> > But on this bigger dataset (which I doesn't find that big) I get
> this
> > error:
> >
> > Project xxx raised exception class EOutOfMemory with message "Out
> of
> > memory". Process stopped.
> >
> > I get it running both from inside D5 and running my appl. alone.
> >
> > I kept an eye on my Task manager and all the time I had at least
96
> > Mb of physical memory, and my swapfile wasn't used that much.
> >
> > I have even tried running the program without any other programs
> > running on my machine at the same time.
> > Same result.
> >
> > Is this caursed by my program, because it doesn't allocate enough
> > memory for it self ?
> > Can I change that ?
> >
> > I know in Turbo Pascal 7 I could set some compiler directives to
> > increase the amount of memory available to my program.
> >
> >
> > Regards
> > Michael
>
>
>
> To unsubscribe from this group, send an email to:
> ib-support-unsubscribe@egroups.com
>
>
>
> Your use of Yahoo! Groups is subject to
http://docs.yahoo.com/info/terms/
To unsubscribe from this group, send an email to:
ib-support-unsubscribe@egroups.com
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/