Pliant database engine layout
The main design issue with Pliant database on top of Pliant storage is that there are too many layers stacked. As a result, the overall thing is fairly hard to understand.
Light datas, fat pointers
In 'The storage machinery layout' article, we have shown that a Pliant database will be stored on disk as a PML encoded set of instructions (or XML like ASCII file, but it does not matter). Executing all the instructions is the way to restore the database state in the main memory global cache when the Pliant process is restarted.
Now, once again, the main difficulty is to design the instructions set, and more precisely to translate on disk the in memory pointer notion. Let's get back to the sample shopping database we introduced in the 'A gentil intoduction to using Pliant databases'.
var Data:Article a :> ...
and we execute:
a count := 3
The 'count' field of the 'Article' record will be modified in the database object in the computer main memory, but we must also write immediately an instruction to the disk (either PML encoded file if the database engine is sitting of top of the storage machinery, or an XML like ASCII file) so that the change be not lost if the Pliant process crashes soon after.
We could store the information in each field ('count' in our example would not have type Int but a bigger one, let's say DbInt, that would contain the required references to produce the PML stream) but it would be terrificly expansive as far as memory consuming is concerned.
A better approche would be to store the record identifier at record level instead of field level, so the 'Article' data type would turn to:
This is what the relational database model does.
c :> a count
where 'foo' is a function that modifies the content of 'c', then the problem to determine the order and article 'c' refers to still holds.
The selected solution is to use fat pointers for accessing database datas. Basically, the pointer contains the path to the data it points. This explain why when handling database datas, 'c' does not have type Pointer:Int but rather Data:Int.
The same applies at object level: a Pliant database field is not a real Pliant object (it does not have a header with references count and type pointer) so generic methods cannot be called, but the database fat pointer contains an interface field that enable to call database generic methods.
Fat pointers detailed content
The Pliant database fat pointers data type support is defined in module /pliant/storage/database/prototype.pli and named 'Data_'
In our previous sample where 'c' was a pointer to 'count' field, the 'Data_' fat pointer support fields would have the following content:
The effective path of the database pointed field can be obtained at any time through calling 'path' method or 'dbpath' method (they are defined in the same module).
At application level, the fat pointers to database datas will have type Data:xxx where xxx is the expected format of the underlying data, so in our sample database, we would have:
var Data:Article a
Each time the application does something on such a pointer (in this example, we get the 'count' field), a generic method on the 'interface' object of the underlying 'Data_' is called.
So, you now know the overall high level machinery of the Pliant database engine: each time the application wants to do something on the database (get a subrecord, create a new subrecord, get a field, change it, etc), a generic method of the 'DataInterface_' object that is pointed by the 'interface' field of the 'Data_' underlying fat pointer data type will be called.
Still in /pliant/storage/database/prototype.pli, we define 'Database_' data type, which is the prototype for a Pliant database. I mean, it is the data stucture that connects the database to the Pliant storage layer (global cache, PML encoded file, etc). The data type is declared at the beginning of the module, but the generic methods it provides are defined at the end.
From high level application interface down to real code
Let's see how things truely happend on a very simple example:
var Data:Article a :> ...
First, the data type of 'c', I mean 'Data:Int' is defined in /pliant/storage/database/pointer.pli, in:
function Data t -> tt
and we have is_field = true (see code in order to understand what I mean by 'is_field').
As a result, with a bit of over simplification in order to provide a step to understand the real code, function Data will dynamically compile :
Implementation note: Where is the locking !!!
So, if we write in our application:
var Int i := c
then the Pliant language casting machinery will silently (because we have set 'implicit' flag in 'cast Int' function) call 'cast Int' function, so that 'get' method of the underlying fat pointer interface will be called.'
I will now explain where the 'interface' field of the fat pointer, that provides all the generic methods that do the true work, comes from, since it is what does the real work in facts, with everything else beeing mostly glue code.
We see (assuming you also read the code while reading the documentation) that if the type is a field ('c' in our example), the interface object will have type 'DataField', and if the field is a record ('a' in our example), the interface object will have type 'DataRecord'.
method df get d adr type -> status
So, if the requested type provided in 'type' argument matches the real type of the field stored in memory which is 'df:type', then we just copy it through calling 'copy_instance', else we cast the value stored in memory to a string, then cast the string to the requested type.
Let's continue with our initial example in this paragraph through explaining how 'c := 3' will be processed.
First of all, it will be compiled by:
meta ':=' e
defined in /pliant/storage/database/pointer.pli, so that function 'data_set' defined in the same modeule is called:
function data_set d v t
After obtaining the database semaphore, 'data_set' function issues:
d:interface set d addressof:v t
So, the 'set' generic method of the 'interface' field of the database big pointer is called, and we have just seen that since 'c' as type Data:Int, it will be 'set' method of 'DataField' data type, defined in /pliant/storage/database/inmemory.pli.
method df set d adr type -> status
It's basically the same as 'get', but the other way round, with a extra call to 'notify_set' at the end.
We start with the application definining:
(gvar Database:Shop shop_database) load "data:/my_corp/shop/shop.pdb"
so that function 'Database' defined in /pliant/storage/database/pointer.pli is called to build the real data type of 'shop_database'.
method df notify_set d adr type
This paragraph is probably fairly hard to understand because Pliant database glue code is a fairly big pile, so I'll try now to provide some overview of it:
Mapping database application level feature to interface generic methods
Let's get back to the sample database, and assume that we have:
var Data:Article a :> ...
First, let's read a field value. We have seen in the previous paragraph that:
var Int i := c
is handled through 'cast Int' function defined at the time Data:Int is defined, and ends to calling 'get' generic method of the interface field of the big pointer:
method di get d adr type -> status
Then, let's write a field value. We have also seen that:
c := 3
is handled by meta ':=' defined in /pliant/storage/database/pointer.pli and translates to calling 'data_set' function, which calls 'set' generic method of the interface field of the fat pointer:
method di set d adr type -> status
Let's access a field within a record:
c :> a count
is handled by meta '' defined in /pliant/storage/database/pointer.pli and translates to calling 'map_field' function, which ends in either calling 'apply' function defined in /pliant/storage/database/inmemory.pli or calling the 'search' generic method:
method di search d k -> d2
The 'apply' function is just a way to speed up things, through avoiding to look for the field in a dictionary at execution time.
We are assuming 'o' to be a database pointer to an 'Oder' record in our sample database:
var Data:Order o :> ...
First, create a new record.
o:article create "a1"
'create' meta is also defined in /pliant/storage/database/pointer.pli, and translates to calling 'data_create', which ends to calling 'create' generic method:
method di create d k -> status
Deleting is not very deferent:
o:article delete "a1"
is compiled by 'delete' meta defined in the same module, and translates to calling 'data_delete', which ends to calling 'delete' generic method:
method di delete d k -> status
Testing the existence of a record is performed at application level through 'exists':
it is compiled by 'exists' meta, sill in the same module, and translates to calling 'data_exists', which translate to ... just testing 'adr' field of the database big pointer to not be null.
We'll finish with the scanning application level instruction:
each a o:article
'each' is compiled by meta 'each', still in /pliant/storage/database/pointer.pli, and translates (when no extra option is provided) mostly to calling 'scan_first' and 'scan_next', which ends to calling 'first' and 'next' generic methods:
method di first d start stop buf -> d2
method di next d start stop buf -> d2
We have not introduced all generic methods of the 'DataInterface_' type ('interface' field the database big pointer) yet:
method di reset d -> status
method di type d -> t
method di address d -> a
method di count d start stop -> count
method di first_to_store d start stop buf -> d2
method di next_to_store d start stop buf -> d2
method di pre_delete d k
Well, if you reached that point, the code is the documentation is probably something you are now ready for, and if you get locked, then dropping me an email will probably make me happily resume trying to explain how it's currently led down.