created: 1245140632|%e %B %Y, %H:%M
We now have basic APIs for queue, messages, and selectors, implemented in the respective classes. These are simple wrappers, e.g. this is the xump_queue_fetch method:
<method name = "fetch" return = "self">
<doc>
This public method fetches a queue from the store. It acts as a
constructor and returns a new queue object when successful. The
caller must unlink this queue object when finished using it.
</doc>
<argument name = "store" type = "xump_store_t *">Enclosing store</argument>
<argument name = "name" type = "char *">Queue name, if any</argument>
<declare name = "self" type = "$(selftype) *" />
//
xump_store_request_queue_fetch (store, &self, name);
</method>
It's basically just a wrapper around the portal method. Now, the problem with these basic APIs is that they don't add enough value. For example, when we fetch a queue, it would be nicer to not have to know the store in advance. Queue names need to be globally unique, so that selector destinations make sense. This uniqueness should be managed by the API, so the caller code cannot get it wrong. Thus, the API has to sit above all the stores.
We already implemented a simple queue name cache that lets us work with queues as global objects within an engine instance, and I'll use this as the basis for a new engine-level API. The general model will be:
- The create and fetch methods will be implemented at the engine level, in the xump class. This makes it possible to work at the engine global level. We'll put delete at the engine level as well, for consistency.
- Access to object properties will be via object methods.
- Other methods, such as adding credit to a selector, will be done at the object level.
Starting with stores, we want to be able to create stores by name. This should be idempotent, so that they do not depend on a precise agreed engine state. E.g. creating a database-backed store should work if the store already exists.
Next, queues. These are global within an engine instance. That is, the caller can work with queues without having to specify what store they are in. The engine must manage this. It's only when we create a queue that we need to specify a store, and it makes sense to allow a default here. Another option would be to use a naming convention (like a path prefix) but that means it is impossible to reconfigure a queue to use a different store, something that might be useful. A store name as prefix remains an option, which we may come back to later.
Then, selectors. As for queues, we already have an API that works but we'll move the methods into the xump engine class as explained. The previous implementation of the selector class referred to the source queue as an object, but it would be cleaner to refer to a queue name.
The natural way to get messages out of the network seems to be to define a selector that has no destination queue, or rather, a selector that has an external destination rather than an internal queue destination. The caller will know whether a selector is an exit node or not, when it creates it. We will call this the "terminal" property of the selector. That is, a selector is a terminal when it delivers messages to an external destination rather than an internal queue. When a selector is not a terminal, its destination name must refer to an existing queue.
Lastly, messages. It seems clear there are two sets of semantics for messages:
- Message flow through queues and selectors.
- Browsing historic data in queues.
For the first case, we can rely on the fact that the engine is a closed system. That is, all output is driven by input. If there are no inputs to the system, there will be no outputs either. In a synchronous model, the calling application thus gives an incoming message or acknowledgment to the engine, and in return gets a set (zero or more) deliverable messages, which it can process in a loop.
In an asynchronous model, the processing of messages can take an arbitrary time and the caller would not want to wait. So, we'd want to work with callbacks (e.g. asynchronous portal reply methods).
For now, we'll design both the engine and the calling application as synchronous and single-threaded, so the API will provide methods to accept incoming messages and acknowledgments, and will provide a list of outgoing messages that the caller can process. Each "publish" pushes messages through the network until all routing is complete, and zero or more "deliveries" are push to that outgoing queue.
For the case of historic data, there are two use cases I can see. One is to scroll back and forwards through the history of a queue, a page of messages at a time: "get me the 50 previous messages". It matches an end-user's browsing of a queue. The semantics are classic: oldest, latest, previous, next. The second use case is to catch up with historic data since a particular time: "give me everything since 9.00am". In both cases, it only makes sense insofar as the queue in question has historic data, which depends on how it was configured and what selectors have done to it.
In earlier sketches of Xump I used the notion of "cursors" but I'm discarding this concept, it appears to be both unnecessary and unhelpful. It seems easier to create a new selector when we want to read historic data on a queue.
The simplest form of a queue is an ordered list of messages stretching from tail (newest mesage) to head (oldest message). New published messages are attached before the tail. Messages are taken off the list from the head:
[[code]]
Tail Head
| |
[M][M][M][M]
<Unprocessed —>
[[code]]
Actual queue implementations depend on the storage layer. To implement COPY, we need to divide the queue into two sections:
[[code]]
Tail Current Head
| | |
[M][M][M][M][M][M][M][M]
<Unprocessed > < Historical —>
[[code]]
When we create a selector we can thus specify where the selector starts:
- At the current position, which is the normal and default choice;
- At the head, i.e. the oldest message
- At any position between the current and the head.
Rate this post:
