Pliant compiler machinery

This article is assuming that you already red and understood Pliant meta programming basic concepts. It is intended to explain how the Pliant compiler works.

The parser (stage 1 to stage 2 transition)

This article is explaining how the Pliant parser internally works. If you want to just understand how to use it, or define simple extensions, rather refer to Pliant syntax article.

The outermost function is 'compile_text' at the end of /pliant/language/parser/engine.pli
What it does is create a 'ParserContext' data, then loop on 'ParserContext_parse_one_token'

What 'ParserContext_parse_one_token' does is:
Seach for 'pliant parser sections' in the Pliant definitions dictionary. It is expected to be a list that will provide token classes.
Then for each class, the name of the class is searched for in Pliant definitions dictionary, and for each instance found that has 'ParserFilter' data type and is visible from the current module (being visible is the way to have extensions that don't apply everywhere but rather only to modules that ask for), the function is called.
At the end of the loop, if the parser cursor has not been moved forward, then an error is set.

Parsing filters do three things:

Among the token filters, some play a special role.
'Parser_filter_newline' is opening and closing subexpressions according to indentation.
'Parser_filter_execute' is detecting that a bloc has been finished, so that 'ParserContext_execute' can be called.

'ParserContext_execute' first calls 'ParserContext_fold_operators',
then recursively call 'Expression_precompile_rewrite' on each node of the expression,
and finally calls 'Expression_execute' to perform stage 2 to stage 3 transition, stage 3 to stage 4 and finally execute the produced code.

'ParserContext_fold_operators' just calls each folding functions according to operators priority.

A sample is provided in /pliant/sample/parser.pli module that defines a new set of parsing rules from scratch, implementing a tiny subset of C syntax.

The compiler (stage 2 to stage 3 transition)

We have seen in the previous section that the parser will end through calling 'Expression_execute'.
then call 'Expression_compile' to perform the stage 2 (expressions tree) to stage 3 (instructions list) transition.

'Expression_compile' is defined in /pliant/language/compiler/expression/expression.c
All it does is call 'Expression_compile_step2', then generate an error message if compiling failed.

First, 'Expression_compile_step2' will call 'Expression_compile_step3' which is just a wrapper to 'Expression_compile_step4',
If compiling fails, 'Expression_compile_step2' will try to compile in two steps: first compile the root node with no argument, then try to compile the result with the arguments. Here is a sample:

function foo -> s
  arg Str s
  ...
console (foo len) eol

Compiling (foo len) directly fails because 'foo' function is expecting no argument.
If compiling fails again, 'Expression_compile_step2' will call 'Expression_failedtocompile_rewrite'  that will try to call all functions with name 'pliant failedtocompile rewrite'. Such a function is used to provide Pliant 'implicit' feature.
Finally, 'Expression_compile_step2' will add listing positions to instructions, for debugging purpose, and call 'Expression_postcompile_rewrite' set of hook functions.

'Expression_compile_step4' that is responsible to do effective compiling of the expression, that is attaching an instructions list and a result argument, does very few in facts. Beside setting a error handler and doing cleanups, it just calls 'call_active' function on the expression node value.

'call_active' is implemented in /pliant/language/compiler/active.c
It get's the type of the expression node value, and sees if an 'active_type' method is provided for that type.
If some is provided, it will be called.
If none is provided, the node value will be considered as a constant, mapped to an argument, and the argument active type method will be called.

As a summary, everything we have seen starting from 'Expression_compile' is mostly glue code. The effective code that will compile expressions are the active_type_xxx functions defined in /pliant/language/compiler/active.c

As an example, 'active_type_Ident' is responsible to compile a node with an identifier at it's root. What it will do is scan the main dictionary for available definitions, and try each of them. Then it will check that one of the definitions is clearly better because it casts less arguments, or 'weak' and 'strong' attributes have been used to provide guidelines.

'active_type_Function' will check that the number of arguments is correct, and that each argument can be casted to the type selected in the prototype.

'active_type_Meta' will just call the meta function and let do the job.

'active_type_Argument' will try to compile (a b c) as ('. b' a c), then ('' a b c)

Casting

In the introduction to meta programming article, I have explained that compiling a stage 2 expression means attaching to it a stage 3 instructions list and a result argument.
It is a bit more complicated in facts. When calling 'compile' method on the expression, what I have described is correct. But when calling 'cast' method on the expression (named 'Expression_cast' in the C part of the code), calling 'compile' (named 'Expression_compile' in the C part of the code) is just the first step. A second step is implemented in 'Expression_cast3'.

'Expression_cast3' which is the center of the Pliant casting machinery is looping on two functions to do the job:

'Expression_test_cast_to' returns a boolean specifying if it succeeded.
The important part of it is:

  ...
  Str_build(&name); Str_concat(&name,Z,"cast ",S,&wished_type->name,END);
  for(r=Dictionary_first(G.general_dictionary,&name); r!=G.null_constant; r=Dictionary_next(G.general_dictionary,&name,r)) {
    ...
    if(!Expression_test_cast_to(type,t,flags,module,excluded,nb_excluded+1,&test_function,&test_type,&test_length,&test_nb)) continue;
    ...

and it means that (assuming that we want to cast to something to 'Float' target data type) it's scanning the Pliant general definitions dictionary for 'cast Float' definitions, and assuming that we find a casting function from 'Int' to 'Float', it will recusively call itself with 'Int' as the target data type.
Then 'path_length' is the number of casting functions that have to be applied to reach the target path and 'path_nb' is the number of solutions using exactly that number of casting functions.
So, the last line of 'Expression_test_cast_to' says that casting succeeds if it's non ambigious, and non ambigious means that there is a single set of casting functions that provide the shortest casting path to the target data type.

Back to meta programming overview, the stage 3 instruction list and argument we attach to the stage 2 expression is just a bit mode complicated. The instructions list has to part. The head is computing the uncasted result, and the tail is casting to the expected data type. Now there is a result argument associated with the head of the instructions list that specify the uncasted result, and another one associated with the tail that specifies the final result.
So, when we call 'cast' method on the expression with the expected type as a parameter, the algorithm is the following:

The code generator (stage 3 to stage 4 transition)

'Expression_execute' will call 'GeneratorContext_optimize' to perform the stage 3 (instructions list) to stage 4 (processor native code) transition.

'GeneratorContext_optimize' is fairly simple.
It get the list named 'pliant optimizer sections',
then for each string it contains, it scans the Pliant general definitions dictionary for it and call the function.
The easiest way to see the effect of each filter before reading the code is to use the http://hc.fullpliant.org/demo/meta online demonstration tool. Futhermore, if you run Pliant locally, then you can connect to loopback:/pliant/site/demo/meta and enter any expression you like instead of beeing constrained to use one of the predefined ones.

↣ Meta programmation illustration

Feel free to ask questions if you have problem to understand how some existing code generator filter works.

Hooks

This section is intended to list all hooks available in Pliant compiler machinery, also I'm sure I have forgotten some at the moment:

'Expression_precompile_rewrite' calls all functions with name 'pliant precompile rewrite' at the end of the parser process, after nodes folding, before compiling the expression.

'Expression_failedtocompile_rewrite' calls all functions with name 'pliant failedtocompile rewrite' when compiling an expression the standard way failed.

'Expression_postcompile_rewrite' calls all functions with name 'pliant postcompile rewrite' at the end of the compiler process, after attaching the instructions list, before generating code.

'pliant_generator_context_begin_hooks' is a list of functions that are called before executing any code generator filter.
'pliant_generator_context_end_hooks' is a list of functions called after executing any code generator filter.