----- Original Message -----
From: "Jim Starkey" <jas@...>
>[snipped a bit]
> At 12:34 AM 9/18/02 +1000, Steven Shaw wrote:
> >
> >Are you saying that AT&T developers were dim to require that shared
memory
> >pages remain in physical memory? Should they be allowed to swap out? Does
it
> >matter anyway since, as you point out, memory is essentially free?
> >
>
> Memory is cheap, but address space is precious. If software were
> as wasteful with physical memory as they are with virtual memory,
> nothing would get done. Non-paged physical memory must be
> managed as a scarce resource.

You are saying that fixing shared memory in physical memory is a bad idea?
and that this is what the AT&T unix developers did?

>
> >
> >I know I need to go back to school but could you exludiate (just a
little)
> >on why intra-process locks are quicker than inter-process locks. Cheers.
> >
>
> First, I think we agree that any excursion into an OS kernel is
> expensive -- change in protection ring, virtual address space,
> flushing or invalidating the cache, flushing the instruction
> pipeline, validating and probing argument addresses, etc.

Yep.

>
> Inter-process synchronization requires kernel intervention. A
> kernel primitive is better that a process level wait/wakeup
> mechanism as it gives better information the scheduler, eliminating
> unnecessary process wakeups and context switches.

I hear you but on large unix box with 4, 8 and more processors it's quite
possible that the other processors are already live and no context switch
happens. People have told me that using spin-locks is much better than using
semaphores. I can see how thread synchronisation (throughput with threads)
could be even better considering the enlightened scheduler and reducing
context switches. I think spin-locks can work for threads too, though.

> Intra-process (inter-thread) synchronization has shared memory
> available to it permitting synchronization mechanisms that can
> handle a subset of contention modes without resorting to a kernel
> call.

Can't inter-process synchonisation work exactly the same way as
inter-thread synchronisation - through shared memory and test-and-set type
operations - and so not resort to a system call?

>An example is read/read contention without a writer, which
> happens to be the most common case when synchronizing data structures.
>
> One could do the same sort of inter-thread synchronization to
> do inter-process synchronization, but from an OS perspective, it
> violates the generally accepted standards for process isolation.

Ah. I see you agree that you can! This argument about violating
"generally accepted standards" is a pretty weak one...
Process isolation is explictly broken with shared memory - so why not take
advantage of all that has to offer?

>
> Linux pthreads operate without significant OS support other than
> allowing processes to share a single virtual address space. You
> might take a look at what the pthread mutex implementation -- very
> ughly, very inefficient. Although Linux context switches at twice
> the rate of NT, NT outperforms Linux on a heavily threaded load
> by at least 20% due to cheaper inter-thread synchronization
> (criticial section v. pthread mutexes).

I thought Linux had better/proper threads now.

>
> >
> >I have seen shared memory mapped to a fixed virtual location work across
> >AIX, HP-UX, Solaris, Linux and Windows-NT. You have to be careful of the
in
> >which external libraries are initialised in each process that much attach
to
> >the shared memory at the same virtual address. Working with "direct
> >pointers" in memory is pretty nice. I've seen it done with offsets, too.
> >It's pretty ugly because all code needs to have the magical global
knowledge
> >about the base-address.
> >
>
> Yuck.

:-). Only working with offsets is yucky. Working with direct pointers in
shared memory is wonderful.

> >
> >When using unsafe languages like C and C++, the threaded server case is
> >worse in that it allows one thread to corrupt not only shared data
> >structures but also "private" data of other threads.
> >
>
> C and C++ are no less safe than another other language. Non-
> synchronized data structures are just as dangerous in Java as
> C or C++. That said, C and C++ allow choice of synchronization
> mechanism, allowing a multi-state lock rather than a simple
> mutex, while Java has mutexes hard wired.

With multi-process model with shared memory, errant processes can only
scribble on their own private memory and pages of shared memory currently
mapped.

With multi-threaded model, all memory is shared between the threads. An
errant thread can scribble over not only it's private memory and shared
structures but "private" memory of other threads too.

That's part of the trade-off when you chose the multi-threaded model over
the multi-process model. You lose the isolation that you get with process
memory.


> [snipped a bit]
> >Is the Netfrastructure architecture single-process/multi-thread?
> >Is this the state-of-the-art architecture for a dbms?
> >
>
> Netfrastructure is single process/multi-thread server. Unlike
> most database engines, it also contains a Java virtual machines,
> a search engine, and an elaborate page generation engine.

Cool.

> Netfrastructure has synchronization objects on about two
> dozen classes for a total about a 30 types of things being
> synchronized. The synchronization object resolves read/read
> contention (sic!) with interlocked increment/decrement
> instructions, resorting to pthread and critical section
> synchronization of thread wait queues. The goal, usually
> achieved, is to satisfy a web request without a thread
> or context switch.

You kinda lost me hear, Jim.

I do think that a single-process, multi-threaded model is the way to go
(over multi-process with shared-memory) for ease of programming. The reason
is that it is easy to share objects. With shared-memory you have to be very
careful to map memory to the same location in each process and also to put
no C++ objects with vtables into shared-memory as the vtables will point to
process memory (and disaster ensues). You have to be careful to remember if
an "object" lives in shared-memory or process-memory and free it
appropriately. I just worry about the robustness of the system that relies
on just one process. It seems to me that the OS should be offering something
better...

The main reason that we use multi-process/shared-memory model at work is
that threads were not an option when the project(s) began. I've been
thinking of recommending single-process/multi-thread as the "architecture of
the future" at work.

>
> Netfrastructure uses a "in memory database" model using the
> disk as a persistent backfill. While it has a page cache,
> records rather than page images reside in memory. And,
> while it support multi-generational records, it does so
> in memory. The on-disk structure is not multi-generational.

Jim, I envy you. I always wanted to write a main memory database system :-)

Cheers, Steve.