Pliant streams

A Pliant stream is used to transfer some data as a set of bytes. The main usage is storing or loading data from disk, and sending or receiving data from a network connection.

Pliant moreover provides a lot of filters making compression, encryption, and applying standard protocols such as FTP, HTTP or SMTP straight forward.
The name to use for dealing with various kind of streams is documented in 'Pliant stream names' article.

Basic operations available on a stream are:

For experimented users, what is special about Pliant stream is just that the 'query' and 'configure' method deal with ASCII parameters as opposed to Posix 'ioctl'.

Files handling is covered in another article.

Basic usage

writing data

module "/pliant/language/stream.pli"
var Stream s
var ExtendedStatus status := s open "file:/tmp/test.bin" out+safe
if status=failure
  console "Failed to open test file (" s:message ")" eol
var Int i := 5
s raw_write addressof:i Int:size
if s=failure
  console "Something went wrong" eol
var ExtendedStatus status := s close
if status=failure
  console "Failed to flush and close test file (" s:message ")" eol

In order to append at the end of the file instead of overwriting the file content, you could use:

var ExtendedStatus status := s open "file:/tmp/test.bin" append+safe

reading data

module "/pliant/language/stream.pli"
var Stream s
s open "file:/tmp/test.bin" in+safe
var Int i
s raw_read addressof:i Int:size
if s=failure
  console "Failed to read the content" eol
if not s:atend
  console "More data was available" eol
s close

Errors handling strategy

There are two possible policies:

When using the 'safe' policy, you can test at any time if some error occurred on the stream using '=failure' as in the very first provided sample. Moreover, when reading a stream where an error occurred, 'raw_read' will fill the buffer with zeros bytes, and 'readline' (see below) will return an empty line.
From a high level point of view, Pliant streams have been designed to make it safe to not test successful completion on any read or write operation, but rather only at key points in the program. If something goes wrong with the stream, the error is not reported only to the instruction raising the problem, but also to all subsequent ones.
As a result, when writing some data, the following final sequence is sufficient to test that everything went fine:

if s=success and s:close=success
  console "Bingo !"

Symmetrically, when reading a file, the single fact the data is non zero is sufficient to grant that everything went fine at open an previous readings steps.

Threading

A Pliant 'Stream' data type is not thread safe, so if you want two threads to work on the same stream, you have to protect the access using a semaphore.

There is one exception: if you open a stream with 'noautopost' flag set (see 'Advanced operations' section bellow for 'noautopost' semantic), then it is safe to have one thread reading from the stream, and another one writing to the stream.

Accessing a ASCII data stream

writing

s writechars "Some text"
s writeline "More text"

'writeline' will output the specified text, followed by a LF or CR+LF or CR end of line sequence.

The instruction:

s writeline "More text"

produces the same result as:

s writechars "More text"
s eol

reading

while not s:atend
  var Str l := s readline
  ...

It is possible to rewind one line through:

s unreadline "Some text"

but it is deprecated feature: rather use the more powerfull 'rewind' feature bellow.

selecting end of line convention

When reading some content, the kind of end of line (LF, CR+LF or CR) will be auto discovered from the content, unless you specify it explicitly at open time.
On the other hand, when writing only or writing first, and the expected end of line is not the Unix LF one, it has to be specified:

s open "file:/tmp/test.txt" out+safe+cr+lf

Please notice that Pliant will default to using LF end of line even when running under Windows where the standard is CR+LF.
Please also notice that if the end of line convention used is CR (Macintosh text files), and the connection is networked to a remote process that sends a single line then waits for the answer, Pliant auto discovery will dead lock waiting for a potential LF so that you have to specify the end of line convention explicitly.

If the end of line convention might be unconsistent (change from a line to another) in the stream, you might use 'anyeol' flag to ask Pliant to check all possibilities with line:

s open "file:/tmp/test.txt" in+safe+anyeol

Accessing PML encoded data stream

PML is a kind of binary XML introduced and used by Pliant for most native network protocols.
It is documented in a specific 'PML encoding' article.

File direct access

var Stream s
s open "file:/tmp/test.bin" in+out+safe
s configure "seek 1000"
var uInt16 v := 12 ; s raw_write addressof:v uInt16:size
if ((s query "seek") parse (var Intn offset))
  console "current offset is " offset eol

Advanced opening

'open' has several variants with more or less parameters. The most complete one is:

module "/pliant/language/stream/filesystembase.pli"
module "/pliant/language/stream/openmode.pli"
module "/pliant/language/stream.pli"
module "/pliant/language/stream/multi.pli"
var Stream s
var Link:Stream support
...
var ExtendedStatus status := s open "gzip:" "comment [dq]Some title[dq] level 1" out+safe pliant_default_file_system support

The first parameter is the name, and the second is an options string.
As in many Pliant functions, an option string is a way to pass a dictionary to a function (a set of keyword value pairs).
The third parameter provides a set of flags (from the theoretical point of view, it could have been provided as keywords in the options string).
The fourth parameter is the file system, and it is responsible for mapping the provided file name to the effective code handling operations for this kind of stream. See how it works section bellow.
Then the last parameter is providing an underlying stream for filtering streams.

We have already seen 'in' 'out' 'append' 'safe' and 'noautopost' open flags.
If 'mkdir' flag is provided and the file path is to some non existing directory, missing subdirectories will be automatically created.
'nocache' flag disables the read and write caches. 'raw_read' and 'raw_write' methods are granted to work on streams with no cache, but other methods might not. Use only if you really know what you do.
'linecache' will force a flush after each 'writeline' or 'eol'.
'bigcache' will force using 64 KB read and write caches instead of 4 KB.
'seek' flag specifies that seeking will be requested.
'seekmuch' flag specifies that seeking will be requested very frequently. This will tel the operating system not to do less read ahead.

About filtering streams naming convention, if you write:

var ExtendedStatus status := s open "gzip:" "comment [dq]Some title[dq] level 1" out+safe pliant_default_file_system support

the 's' stream will transparently apply compression and feed compressed data to the 'support' provided underlying stream. Please notice that the underlying stream must be real Pliant object, created  with:

var Link:Stream support :> new Stream

or:

ovar Stream support

as opposed to:

var Stream support

The alternative way to create the filtering stream is:

var ExtendedStatus status := s open "gzip:file:/tmp/test.gz" "comment [dq]Some title[dq] level 1" out+safe pliant_default_file_system (null map Stream)

so that the underlying stream will transparently be created.

A more frequently uses variant of open is:

var ExtendedStatus status := s open "gzip:file:/tmp/test.gz" "comment [dq]Some title[dq] level 1" out+safe

and it is equivalent to:

var ExtendedStatus status := s open "[dq]gzip:file:/tmp/test.gz[dq] comment [dq]Some title[dq] level 1" out+safe

In other words, if the first character of the name parameter provided to open is a double quote, then the name will be slit to a real name plus an options string.

Blobs

It is possible to store the content of the stream in memory, or pick it's content from memory:

module "/pliant/language/type/misc/blob.pli"
module "/pliant/language/stream/blob.pli"
var Blob b
var Stream s
s open b out
...

Please remind that a blob contains just a set of bytes, and it's content can be set to map any area in memory:

var Address adr
var Int size
var Blob b
var Stream s
...
b map adr size
s open b in
...

Temporary file

This is documented in the 'Files handling' article:

var Str temp
var Stream s
temp := file_temporary
s open temp out
...
s close
file_delete temp

Advanced operations

When writing

You can force data in the stream our buffer to be sent to the disk or the network using 'flush' method:

s flush anytime

s flush async

s flush sync

'anytime' means that the operating system is free to keep the data in it's cache for some finit time before sending it.
'async' means that the operating system should send the data immediately, but the 'flush' method should return before the data reached the target.
'sync' means that call to 'flush' method should bloc until the data reached the target.

If you do:

var Stream s
s open "tcp://server/25" in+out+cr+lf+safe
s write "220 smtp.fullpliant.org Simple Mail Transfer Service Ready"
s readline (var Str cmd)

a 'flush async' instruction will be transparently inserted between the 'write' and the 'read' operation. I mean, when a read operation (readline or raw_read) does not find enough bytes already available in the input buffer, it checks the output buffer, and if some data are pending there, it forces them out.
If you don't like this behaviour, just open the steam with 'noautopost' flag:

s open "tcp://server/25" in+out+cr+lf+safe+noautopost

When reading

'read_available' is frequently used to speed up reading through having the application read data directly from the stream buffer has opposed to using 'raw_read' that copies data from the stream buffer to the application buffer:

s read_available (var Address adr) (var Int size)

s read_available (var Address adr) (var Int size) 100

When returning, 'adr' will be a pointer to the data available in the read buffer, and 'size' receives the number of available bytes. 'read_available' consumes the data.
If a third parameter is provided, it specifies the maximum number of bytes to consume.

Copying

A high function is provided to copy some data from a stream to another:

var Stream s1 s2
var Int copied := raw_copy s1 s2 1 4*2^20

The third parameter is the minimal number of bytes to copy before returning, and the fourth the maximum. The actual number of bytes copied is returned, so either an error occured, or it will be in the specified range.

Rewinding

Pliant provides a mechanism to enable reading several time the same sequence in an input stream.

s rewind_open
... # read some content
s rewind
... # read the content again
s rewind
... # and again
s rewind_close

The rewindable data is stored, so consumes memory. So a 'rewind_limit' parameter exists to prevent consuming too much memory. The default value is 16MB.

s rewind_limit := 64*2^20

It would be a good idea to enable a two stages limit, so that when the first one is reached, data are dumped to a temporary file that can grow up to the second limit.

Testing the stream status

var CBool c := s is_open

Returns true if the last open operation succeeded and no close operation has been performed since.

var CBool c := s is_crashed

Returns true if some error happen in a read or a write operation on the stream.

s error "Some message"

Sets the crash bit of the stream. If the stream has not been open with 'safe' flag set, then a standard Pliant error with error ID error_id_io will be raised.

s recover

Clears the crash bit of the stream. Don't use it unless you reviewed the implied underlying Pliant code.

Please notice that:

var CBool c := s=success

is equivalent to:

var CBool c := s:is_open and not s:is_crashed

How streams machinery works

Buffering machinery

A Pliant stream uses an input and an output buffer.
The buffers are used to speed up things through reducing the overhead related to calling underlying drivers glue code or underlying operating system functions.
For C fluent users, it is the same as using Posix 'fread' and 'fwrite' as opposed to 'read' and 'write'.

Direct access to the read buffer

The read buffer is governed by the three fields 'read_buffer' 'read_cur' and 'read_stop'.
These are named 'stream_read_buffer'' 'stream_read_cur' and 'stream_read_stop' when used from an application.
The number of available bytes in the buffer is:

var Int avail := (cast s:stream_read_stop Int).-.(cast s:stream_read_cur Int)

Moeover, any application is allowed to change 'stream_read_cur' pointer, so if your applications know that 2 bytes are available, it can consume them through something like:

var uInt16 data := s:stream_read_cur map uInt16
s stream_read_cur := s:stream_read_cur translate Byte 2

Direct access to the write buffer

The write buffer is govered by the three fields 'write_buffer' 'write_cur' and 'write_stop'.
These are named 'stream_write_buffer'' 'stream_write_cur' and 'stream_write_stop' when used from an application.
The remaining room in the write buffer is:

var Int room := (cast s:stream_write_stop Int).-.(cast s:stream_write_cur Int)

Moeover, any application is allowed to change 'stream_write_cur' pointer, so if your applications know that 2 bytes are available, it can set them through something like:

s:stream_write_cur map uInt16 := 5
s stream_write_cur := s:stream_write_cur translate Byte 2

Pliant file systems

A Pliant file system inherits from 'Filesystem' data type defined in /pliant/language/stream/filesystembase.pli module, with an extra 'open' method defined at the beginning of /pliant/language/stream/stream.pli module.

When 'open' stream method is called, the file system 'open' method will be called, and it is responsible to attach a stream driver to the stream. If open was provided few parameters 'pliant_default_file_system' is used, and it contains an object with 'MultiFileSystem' data type, defined in /pliant/language/stream/multi.pli
As a summary, when 'open' method is called, most of the time (always) 'open' method defined in /pliant/language/stream/multi.pli is called.

Pliant 'MultiFileSystem' is just a namespace dispatching filesystem. Basically, all file systems are recording themself to it using 'mount' method and providing a namespace header that just says: I'm handling files with name starting with 'xxx:'

A stream driver is a class implementing the API defined for 'StreamDriver' data type at the beginning of /pliant/language/stream/stream.pli module.

So, when defining a new kind of Stream, one declare a new class inheriting from 'StreamDriver', and a new class inheriting from 'FileSystem', then create a filesystem object (through a simple 'gvar' instruction) and record it in the default multiple file system namespace using 'mount' method.
Very simple code implementing a new kind of stream is available in /pliant/language/stream/null.pli and a bit more complete one is available in /pliant/language/stream/count.pli
The main Pliant file system used to access disk files is defined in /pliant/language/stream/native.pli and the one for using TCP connections is defined in /pliant/language/stream/tcp.pli
Lastly, /pliant/protocol/http/chunk.pli is a good example of a not too complicated filtering filesystem implementing HTTP chunking, and /pliant/language/stream/zlib.pli is providing transparent interface with the well known Zlib compression library.