Hi Jim,

> Ann and I were talking about the performance degradation when
> operating with forced write. It occurred to me that although
> Unix normally performs writes by copying data out of the process
> space to a global disk cache, perhaps it was doing page writes
> directly out of process space, which would necessarily require
> freezing the process for the duration of the operation.
>
> Does anybody know if this is the case?

Unix generally doesn't do I/O directly from user space unless the file
was opened with the O_DIRECT mode (available on SGI Irix and FreeBSD,
possibly others), or perhaps if an aio_* call was used. The FreeBSD
implementation (I believe) still goes through the buffer cache, but
"minimises the impact".

However, the on files opened with O_SYNC, the call to write(2) won't
return until the block is on disk. In this case, you must wait for the
I/O. The process you are seeing is (most likely): copy to the global
disk cache, perform I/O, return to user process.

To get performance in this case (ie: have multiple outstanding writes),
you either need multiple threads/processes, depending on whether your
threading library is kernel implemented (ie: threads are scheduled
entities) or user mode.

Assuming that files aren't opened with O_SYNC, and fsync(2) is used, you
can decide when you want to wait for the I/O, but you don't get to
decide the page order. mmap(2) and msync(2) can be used to decide page
order and flush consecutive pages in a single operation. You still need
to wait for it to complete to know what happened.

Aio* calls are the other possibility.

> The Interbase/Firebird page cache manager goes to some lengths
> to put page buffers on natural page boundaries to allow page
> IO by the operation system virtual memory manager (if it happens
> to support that optimization). On VMS, which didn't (doesn't?)
> have cached IO, this was a huge performance win. It may be
> possible that same code is a UNIX deoptimization.

No, it can still help in an optimised O_DIRECT or aio case. It
certainly doesn't hurt.

> If this is indeed the case, a simple dedicated page write thread
> may give a performance boost when forced write is enabled.

Yes, this will help on operating systems where there is kernel mode
threading by allowing other non-I/O work to proceed.

Regards,

Jan.