created: 1243947614|%e %B %Y, %H:%M
I'm analysing the storage layer, which holds queues and messages in different ways (RAM, file system, SQL database, and so on). My goal is to make a minimal storage layer using the iPR portal system, with two or three different implementations. Then, to test these and get performance measurements. The design for Xump is very roughly this:
+--------------------------------+
| Application layer |
+--------------------------------+
| Routing layer |
+--------------------------------+
| Storage layer |
+----------+----------+----------+
| Back-end | Back-end | Back-end |
+----------+----------+----------+
Making it simpler… what can we ignore for now? Our use case for Xump (which is a social messaging layer above Wikidot) calls for very high volumes of quite small text messages (80-160 characters). We would expect thousands of queues, each containing thousands of messages. We'd want these to be persistent and archived, with messages being held on their queues for as long as the queues existed.
We can ignore messages with multipart contents. This is an advanced use case and arguably it is not a good one, since it creates messages that are complex for applications to work with. If my application produces multipart messages, that makes it hard for others to talk to me. So a framework that allows this may actually be helping me to shoot myself in the foot.
For high volumes of small messages, copying is not a major concern. We avoid copying by manipulating references, but that is not free. So, we will for now copy data. Later, and for larger messages, we'll see about normalizing data so that copying is reduced when the same message is sent to many consumers.
Every single message has an address, which is used heavily, so we will make this a dedicated property of the class. A message also has content body, content length, and a table of name/value fields.
Let's build the storage system around queues. That is, a store holds a number of queues, which hold a number of messages. We can interrogate the store for queues and messages, and it will return us memory objects that we can work with.
The iPR portal system provides a virtual class interface. Our portal class will be xump_store.icl. This would have implementations (called "portal back-ends") like xump_store_fs.icl (file system), xump_store_ram.icl, xump_store_sql.icl, and so on. Each portal back-end instance is equal to a store instance, and implements the portal methods. Which we can break into:
- Creating and working with queues in a particular store.
- Creating and working with messages in a particular queue (in its store).
The queues and messages that a store manages are opaque, and invisible as such to the storage layer. To work with a queue, the caller asks the back-end to provide a queue object. It can execute methods on this object (some of which may affect the stored queue). It then destroys the queue object. Messages are handled in the same way.
We will use the create-retrieve-update-delete pattern for working with queues and messages. To some extent this mirrors the RESTful pattern for working with remote resources. In this case, the remote resources sit in a store that is opaque to the caller, which accesses it via the portal interface.
To repeat in a different way: when the caller does xump_queue_destroy(), this destroys a queue object but does not destroy the queue held in the store.
The queue and message objects that the store provides to the calling layer are private, not locked, and not shared (they do not support reference counting).
The simplest workable identifiers for queues are names. It makes sense that a store can auto-name private queues. For messages, we can use sequence numbers.
The xump_store.icl portal class needs these methods for working with queues:
- Create (named or unnamed) queue in store, with given properties.
- Retrieve queue from store by name.
- Update queue with given properties.
- Delete queue from store.
And for working with messages:
- Create new message in queue, with given content and address.
- Retrieve message from queue (there are various patterns here).
- Update a message with given properties.
- Delete message from queue.
And finally, to manipulate fields on messages:
- Create new field on message, with given name and value.
- Retrieve field from message, by name.
- Update field value.
- Delete field from message.
We then need classes to hold queue, message and field references:
- xump_queue.icl, which refers to a queue in a store.
- xump_message.icl, which refers to a message in a queue.
- xump_field.icl, which refers to a field in a message.
So our resource hierarchy looks like this:
Store
+-- Queue
+-- Message
+-- Field
Rate this post:
