Jan,

To my mind, you are describing the MSMQ/MQ Series messaging platforms.
Since these products already exist, IB shouldn't try to duplicate them.

Based on Jim's outline of the API, I have conceived of an IB
Message/MSMQ gateway/broker program which could run on the IB server.
This message broker would accept all the IB messages and be able to feed
the messages into appropriate queue(s). These queues which would/could
be a secure persistent (if you want) message, with guaranteed delivery.
The MSMQ functions (I know about these) already provide a callback
function to provide immediate client notification of new message, as
well as Peek (read but don't delete from queue) and Read (read and
delete) functions/methods. Using the broker, the only IB client
receiving messages would be the message broker, all clients would
"register" with the broker and receive message via the queues.

What Jim has proposed seems a very reason and workable solution, which
"leverages" the existing event concepts. It provides basic (and pretty
complete) messaging requirements which, as I have outlined, be coupled
with other technology depending on the needs of the clients/application.
I am, though, working on a couple of small suggestions.

Jason, I know your listening, what you have proposed (so far) appears to
rely to heavily on a "twine and bailing wire" quilt of DB tables (with
unique constraint) and the existing Event mechanism. While, I'm sure
you could make it "work", to the "impartial" third-party it seems an
ugly, ugly, ugly mess. I look forward, though, to seeing your update
design.


Sean

-----Original Message-----
From: Jan Mikkelsen [mailto:janm@...]
Sent: Thursday, May 18, 2000 4:12 AM
To: IB-Architect@egroups.com
Subject: Re: [IB-Architect] Messaging API

While talking about APIs are client implementations issues might be
interesting to some, I think getting the semantics of the service right
might be useful.

Let's look at the requirements a bit.

I see multiple distinct and useful services:

- A transactional point to point queued message service. For extra
points, allow multiple producers and consumers, ensure that each only
one consumer sees each message, and ensure that multiple consumers can
process work simultaneously. Consumer failures must be handled by
putting the message back at the head of the queue.

- A reliable broadcast message service. Client applications get an
unbroken and ordered sequence of messages from a given point.

- An unreliable broadcast message service. Essentially UDP or IP
multicast, not very interesting for the purposes of this discussion.

People here seem most interested in the second service, which is
reasonable. There are interesting implications for such a service.

Network Traffic

There are network scalability issues when messages are sent individually
to each client. Network traffic goes up linearly with the number of
clients. For example, assume a message rate of 10/sec, average size 50
bytes and 1000 clients. This is roughly 5Mbit/sec of traffic after
allowing some protocol overhead. Where the underlying network supports
broadcasts, clearly there is a win.

Underlying Network Naming

Using a protocol like UDP lets you send a broadcast to all nodes on a
subnet listening on a given port number. Problems: Only a submit, need
to use a port number, which might clash with other applications.

IP multicast allows broadcasts to cross routers, and addressing can be
locally controlled.

Of course other protocols (eg. ATM) define broadcast services with
slightly different semantics; they can still be used.

Higher Level Naming

At the application level, I would expect a database object for the
message stream to be created. An application would subscribe to a
message stream and be given the listening details by the server.

If filtering on a stream is to be performed, it should be done at the
client, otherwise detecting gaps becomes problematic.

Message Structure

I have deliberately left the message structure undefined. I think the
message structure should either be defined in terms of native Interbase
types, or left as a blob. The actual representation should be machine
readable and small. Please, no mandated ASCII, and especially no XML!

Joining (or resyncing with) a Message Stream

If a client misses a message, it needs to get a copy of the lost message
before it can present anything to the application. It then needs to
request that the server resends a particular message. If a "small"
number of messages are missing, the server could either send them
directly to the client, or (as an optimisation if many clients missed a
message) rebroadcast the messages. Clients which did receive the
messages would drop them as a duplicates.

If the number of missed messages is "large", we assume that the server
has forgotten about some of the missed messages. Joining a message
stream is much the same as missing too many messages. The client needs
to resync to match the server's state so that it can start listening to
broadcasts again and interpret them correctly. How this is done will
vary with the application. I expect the range to be "don't care, just
start listening again" to "start listening, and remember everything
while
you receive the entire state as at sequence N." Of course, generating
the state as at sequence N is application dependent.

Of course, early detection of missed messages reduces the size of this
problem, so regular heartbeat messages are useful. The definitions of
"small" and "large" are probably related to the amount of traffic
between
some small number of heartbeats.

Security

Submitting messages to the queue would require a database connection
with
appropriate permissions. If you really care about who listens or about
message authentication, encrypt and/or sign the messages. To get
message
authentication, you need public key cryptography and to previously have
distributed the server's public key. If you just care about
confidentiality, the client and the server could do a Diffie-Hellman key
exchange to agree on a key for securing the broadcast key. No external
configuration required.


Anyway, there are some random thoughts.

Jan Mikkelsen
janm@...



------------------------------------------------------------------------
Best friends, most artistic, class clown Find 'em here:
http://click.egroups.com/1/4054/4/_/830676/_/958637512/
------------------------------------------------------------------------

To unsubscribe from this group, send an email to:
IB-Architect-unsubscribe@onelist.com