An introduction to Pliant storage

I have a problem with writing this article, because as in many other areas, the rational explination of the path to the selected solution is in fact very artificial and so weak from the intellectual point of view: the serious one would be to test all reasonably simple solutions and report problems met with each, but it is not something possible in a decent amount of lines.
The real inventor effective process has already in fact both kind of troubles, since it's a mixture of intuition (artificial) and experiments (long), so, at report time, fully assuming the artificial weakness is an obvious choice since it mostly cannot be avoided in facts.

History of mainstream storage standards

The first level of storage has been led down by Unix in the seventies: permanent data are stored on disk as a tree of files, where each file is a flat set of bytes. This concept proved over the years to be good because both simple and quite flexible. (1)

Some kind of data fit nicely on the file notion, such as ASCII text files or images. In very few words, most data that you load at some point in the application, then process in main memory, and finally store back to disk.
On the other hand, database informations (tables containing large set of records that each contain several fields) don't fit that nicely to the disk as a tree of files concept. One of the reason is: what should we map a file to ? A whole database, a table, a record or a field ? If we map a database or even a table, then we have integrity issue because the complex in file layout required to efficiently map a whole table in a single file might get inconsistent in case of an application crash. On the other hand, if we decide to map each record or even field to a file, then most file systems will collapse both on speed and space usage.
This shows that two concepts are missing in the Unix file systems layout: transactions (granting that none or all operations will be applied but not some only, so that in the end the complex on disk layout be granted to remain consistent) and efficient handling of large sets of very small data.
These two extra concepts have been added at application level in relational database servers, so that in most modern computing systems, two storages mechanisms are used simultaneously: the Unix file system for application that do load and save, and a relational database server for handling large sets of small data.

Constrains and opportunities provided by hardware evolution

Back in 1984, a hard disk typical performances are

in 2008, they are:

so, in 25 years, the transfer (continuous read or write) rate has been multiplied by 500, but the access time has been divided by only 10.
As a result, a modern computer hard disk should be seen more as a tape than a device with direct access capabilities (also it might change with the emerging memory sticks).
The few remaining slowness of modern desktop computers (an application needing several seconds to start up, or the disk check lasting many many minutes) are direct result of this fact: the computer capabilities are growing mostly exponentially, but the access time (and network latency) does not.

In order to cope with disk access time slowness, all modern operating systems do disk caching: some of the on disk data are also stored in the main memory, so that they can be read, and more importantly, accessed much faster.
Now, if we focus more specifically on database storage, the direct consequence of disks technology evolution is that under any decent load, a database server will behave smoothly only if most of the frequently accessed data are stored in the operating system (or database server) disk cache.
Well, since a modern server memory can be counted in GB, it is no more a serious issue to have a copy of all the database files in the main memory for anything but very large databases.
As a result, the next question is: if we have all database data in the main memory, what is the disk useful for. The obvious answer is: it is only a backup of the main memory, in case of a crash.
So, the next next question is: since memory and disk are used to provide very different features (respectively process and restore), is it still a good idea to have the same data structure in memory and on disk, I mean organize in memory data the same way as on disk ? (2)
Well, I bet you will not be surprised if I now tel you that my answer is just no.

As a summary, at step 1, we see that disk slow access speed evolution with cheap huge amount of main memory now makes it mandatory to have most of the database not only on the disk, but also in the main memory in order to fully benefit from modern server capabilities.
Then, at step 2, we see that if we decide to have not most of the database but rather all of it, then we can get some extra freedom: the in memory and on disk data organization need no more to be the same so that each of them can be optimized accordingly to what it is really used for.

The computer is a caches pile: main memory organization

As soon as I've asserted that we are going to process the database in the main memory instead of on the disk, I bet many of you have eared a bell immediately ringing in their head: what if we run out of memory because the database growing.

I'll rather try now to provide a very high overview of what a modern computer truly is.
First of all, let's say that a modern computer is just a pile of caches:

So, if we want to keep the freedom of different organization on disk and in memory, yet have performances slowly getting down when memory is full instead of an abrupt stop, all we need is to split the data set in several pieces (objects) so that not all of them are used at a given time. As a result, should we need to free some memory, then we can just drop some objects not used at the moment, because we will be able to rebuild them from the on disk log when needed, at the price of some performance penalty.

There are mainly three different ways to organize main memory:

Pliant uses a mixture of explicit allocation, references count and a global cache.

At the beginning of the article, we have specified that in any modern operating system, on disk data are stored in a tree of files. In Pliant, each storage object loaded in memory is placed in a tree of objects called the global cache. I'm not aware of any other mainstream language providing an equivalent of the Pliant global cache, even if it could be implemented with more or less success in most of them.

Anyway, the view of the cache pile as the spinal column of a modern computer memory layout bring you the design reason for Pliant to use references count instead of a garbage collector. The most important part is the global cache, and reference count is a better way to support it because garbage collector would prevent to fill most of the memory without serious performances drop, and would bring constrains on the way to handle low level operations.

Pliant storage model

The data organization adopted by Pliant storage system is a set of objects, with each object stored:

Any implementation attempt shows that the only serious issue with this new model is to find a way to identify in the on disk log the in memory nodes implied with each change. In memory, a node is identified by it's address, but it's not a permanent identity since it is likely to change with each restart of the application. Several solutions exist, but it's beyond the scope of this introduction.

Let's finish this introduction to Pliant storage through listing a few properties of the Pliant storage model:

Providing infinite log is very easy and enables two very import things in open systems:

Synchronizing several distant servers is not too complex since it's mostly a matter of sending and applying the log of each of them on all others.

Desktop documents (XML trees), that don't fit nicely with the mainstream relational database model, do fit nicely on top of the Pliant storage model, so that we can get read of the load and save feature for desktop applications and don't need to reinvent a dirty little logging feature in each application (for copping with unexpected crashes).

Pliant database model

The Pliant database model sits on top of the Pliant storage model.
The data layout model adopted by Pliant database engine is not the mainstream relational model, but a tree model.

Let's start by explaining what the relational model is, what the Pliant model is, and properties attached with each of them.

The relational model says that a database is a set of tables, a table is a set of records, and each record in a table contain the same set of fields. Then relations 1 to 1, 1 to n and n to n are established through fields having the same value in different tables. As an example, in the shopping database we take as the example in the 'gentle introduction to using Pliant databases' article, an order contains n articles, and the connection between the order and it's articles (a subset of all articles records stored in the single articles table) is established using a field in the articles table that have to match the corresponding field value in the order record.

On the other hand, Pliant database model is more modeled on what people do when they handle data directly with their favorite computing language without the help of a specific storage engine. Back to the sample, an order is a record, and it has a field which is a set of articles (an embedded subtable).

From the theory point of view, the rational model is an arbitrary graph with all relations being equal, and the Pliant model forces you to extract a main tree from it, with two different classes or relations: the ones matching the main tree structure being easier and more efficient to use than the ones crossing the tree.

Both systems provide an immediate advantage:

Since the relational model keeps all relations equal (the database engine is automatically deciding to create indexes to speed up most often used 1 to n relations), then it adapts much better if the main usage of the database changes over time, so that the main tree get's different.
Moreover, n to n relations are much better handled because they fit naturally and efficiently on the model whereas with the Pliant database tree model, all n to n relations cannot be part of the main tree, so have to be handled the not easy not efficient way.
As a summary, the relational model is a much more general model so that in the end it handles complex relations more cleanly and more efficiently.

On the other hand, many database have a fairly evident main tree. In our sample database, an order with a set of records. In such a situation, the locality of the Pliant tree model can be a serious advantage. If we want to display a single form to a user showing an order, and the order is not yet loaded in the main memory, with Pliant model, we will do a single disk access to read the order record and it will load all the embedded subtables (a single articles one in our simple sample database) at once. On a relational database server, the order record is read, then for each subtable, one or many more disks access are required (remind that direct access on modern disks is painfully slow) to read related records in other tables involved in the 1 to n relations.
In other words, the tree model better matches the Pliant storage global cache notion.

Then, we have to study how the application deals with the data. Only afterwards, it will be to conclude what each system seams best suited for.

Relational model is nearly always associated with SQL access language. SQL is a logical programming language, and in facts the only logical programming language that succeeded to become mainstream.
Logical programming is opposed to procedural programming. In procedural programming, you describe to the computer how to build the result step by step. In logical programming, you describe the assertions the result must match, and the computer deduces a way to compute it. So, the question is: Why nearly all database servers ended with using logical programming whereas nearly everything else ended using procedural programming.
The answer is double: we have seen that the relational model treats all relation equal, so that the database engine is (at least in theory) responsible for deciding what indexes it creates and maintains to speed up operations. Using a logical programming language for queries maximizes the freedom of the database engine because it will also decide the way to compute the result to the queries.
Then, the logical programming main difference with procedural programming is that it's better because it enables more elegant solutions to some problems, but it's also worse because it requires the programmer to be more clever (because it's more abstract) so that it minimizes rather than amplify programmers capabilities. (4) The database queries is an area where most programs are either trivial or quite simple so that logical programming providing better freedom for the database engine in order to maximize the benefits of the relational model, at the expense of limiting programmers capabilities is something that makes sense in the end.

On the other end, relational SQL server have serious latency issues. The reason is that since the server is handling complex on disk data structures (variable length fields and indexes) the server has to run in it's own process or even own server because a crash can be a serious trouble (5). So operations that require many small requests as opposed to one big one, such as just displaying a form to the operator, tend to be poorly handled, both on the efficiency front and on the program simplicity front.On the other hand the Pliant storage model with storage objects that match standard languages organization and on disk infinite log makes it efficient (the storage server is the application server and works in main memory), safe (on disk format is so simple) and clean to program (the database database data are no different from other data).

So, my general conclusion is that Pliant storage and database models are better suited than relational model supported by SQL servers for the front-end, I mean displaying forms to the operators, keeping data safely on disk (including infinite log that can be rewind selectively), and over the Internet life synchronization among several servers.
On the other end, relational model with SQL queries on top is better suited for the back-end, I mean make statistics, due to the ability of the SQL engine to benefit from the relation model freedom associated with the logical programming freedom to automatically organize computations efficiently on batch requests handling large sets of data.
Now, the very simple log structure used by Pliant database servers makes it fairly easy to propagate on the fly any change to some SQL server, so that a mixed configuration to benefit from both words advantages is easy to setup provided the programs modifying data do it on the Pliant database server side.

I end this article explaining Pliant storage design with a management level advise: any ERP (home made or home specialized database system that closely matches company organization) is likely to fall to an such inconsistent level over time that it needs replacement (6). As a result, using a database system that has a logging system that can be easily parsed to live plug changes to another completely different system through filtering the simple log is a the feature that enables to run a mixed configuration at some point, so make the transition less scaring.
In very few words, at project startup time, checking the log your new system will provide might be a greater value for your time than checking the features it will provide.