Facilities are provided by the AST library for performing input and output (I/O) with any kind of Object. This means it is possible to write any Object into various external representations for storage, and then to read these representations back in, so as to restore the original Object. Typically, an Object would be written by one program and read back in by another.
We refer to “external representations” in the plural because AST is designed to function independently of any particular data storage system. This means that Objects may need converting into a number of different external representations in order to be compatible with (say) the astronomical data storage system in which they will reside.
In this section, we discuss the basic I/O facilities which support external representations based on a textual format referred to as the AST “native format”. These are implemented using a new kind of Object—a Channel. We will examine later how to use other representations, based on an XML format or on the use of FITS headers, for storing Objects. These are implemented using more specialised forms of Channel called XmlChan (§18) and FitsChan (§16).
The best way to start thinking about a Channel is like a C file stream, and to think of the process of creating a Channel as that of opening a file and obtaining a FILE pointer. Subsequently, you can read and write Objects via the Channel.
This analogy is not quite perfect, however, because a Channel has, in principle, two “files” attached to it. One is used when reading, and the other when writing. These are termed the Channel’s source and sink respectively. In practice, the source and sink may both be the same, in which case the analogy with the C file stream is correct, but this need not always be so. It is not necessarily so with the basic Channel, as we will now see (§15.2).
The process of creating a Channel is straightforward. As you might expect, it uses the constructor function astChannel:
The first two arguments to astChannel specify the external source and sink that the Channel is to use. There arguments are pointers to C functions and we will examine their use in more detail later (§15.13 and §15.14).
In this very simple example we have supplied NULL pointers for both the source and sink functions. This requests the default behaviour, which means that textual input will be read from the program’s standard input stream (typically, this means your keyboard) while textual output will go to the standard output stream (typically appearing on your screen). On UNIX systems, of course, either of these streams can easily be redirected to files. This default behaviour can be changed by assigning values to the Channel’s SinkFile and/or SourceFile attributes. These attributes specify the paths to text files that are to be used in place of the standard input and output streams.
The process of saving Objects is very straightforward. You can simply write any Object to a Channel using the astWrite function, as follows:
The effect of this will be to produce a textual description of the Object which will appear, by default, on your program’s standard output stream. Any class of Object may be converted into text in this way.
astWrite returns a count of the number of Objects written. Usually, this will be one, unless the Object supplied cannot be represented. With a basic Channel all Objects can be represented, so a value of one will always be returned unless there has been an error. We will see later, however, that more specialised forms of Channel may impose restrictions on the kind of Object you can write (§17.2). In such cases, astWrite may return zero to indicate that the Object was not acceptable.
Before discussing the format of the output produced above (§15.3), let us consider how to read it back, so as to reconstruct the original Object. Naturally, we would first need to save the output in a file. We can do that either by using the SinkFile attribute, or (on UNIX systems), by redirecting standard output to a file using a shell command like:
Within a subsequent program, we can read this Object back in by using the astRead function, having first created a suitable Channel:
By default, this function will read from the standard input stream (the default source for a basic Channel), so we would need to ensure that our second program reads its input from the file in which the Object description is stored. On UNIX systems, we could again use a shell redirection command such as:
Alternatively, we could have assigned a value to the SinkFile attribute before invoking astRead.
I/O operations performed on a basic Channel are sequential. This means that if you write more than one Object to a Channel, each new Object’s textual description is simply appended to the previous one. You can store any number of Objects in this way, subject only to the storage space you have available.
After you read an Object back from a basic Channel, the Channel is “positioned” at the end of that Object’s textual description. If you then perform another read, you will read the next Object’s textual description and therefore retrieve the next Object. This process may be repeated to read each Object in turn. When there are no more Objects to be read, astRead will return the value AST__NULL to indicate an end-of-file.
The pointer returned by astRead (§15.4) could identify any class of Object—this is determined entirely by the external data being read. If it is necessary to test for a particular class (say a Frame), this may be done as follows using the appropriate member of the astIsAClass family of functions:
Note, however, that this will accept any Frame, so would be equally happy with a basic Frame or a SkyFrame. An alternative validation strategy would be to obtain the value of the Object’s Class attribute and then test this character string, as follows:
This would only accept a basic Frame and would reject a SkyFrame.
Occasionally, you may want to store a number of Objects and later retrieve them and use each for a different purpose. If the Objects are of the same class, you cannot use the Class attribute to distinguish them when you read them back (c.f. §15.6). Although relying on the order in which they are stored is a possible solution, this becomes complicated if some of the Objects are optional and may not always be present. It also makes extending your data format in future more difficult.
To help with this, every AST Object has an ID attribute and an Ident attribute, both of which allows you, in effect, to attach a textual identification label to it. You simply set the ID or Ident attribute before writing the Object:
You can then test its value after you read the Object back:
The only difference between the ID and Ident attributes is that the ID attribute is unique to a particular Object and is lost if, for example, you make a copy of the Object. The Ident attrubute, on the other hand, is transferred to the new Object when a copy is made. Consequently, it is safest to set the value of the ID attribute immediately before you perform the write.
Let us now examine the format of the textual output produced by writing an Object to a basic Channel (§15.3). To give a concrete example, suppose the Object in question is a SkyFrame, written out as follows:
The output should then look like the following:
You will notice that this output is designed both for a human reader, in that it is formatted, and also to be read back by a computer in order to reconstruct the SkyFrame. In fact, this is precisely the way that astShow works (§4.4), this function being roughly equivalent to the following use of a Channel:
Some lines of the output start with a “#” comment character, which turns the rest of the line into a comment. These lines will be ignored when read back in by astRead. They typically contain default values, or values that can be derived in some way from the other data present, so that they do not actually need to be stored in order to reconstruct the original Object. They are provided purely for human information. The same comment character is also used to append explanatory comments to most output lines.
It is not sensible to attempt a complete description of this output format because every class of Object is potentially different and each can define how its own data should be represented. However, there are some basic rules, which mean that the following common features will usually be present:
Beyond these general principles, the best guide to what a particular line of output represents will generally be the comment which accompanies it together with a general knowledge of the class of Object being described.
It is not always necessary for the output from astWrite (§15.3) to be human-readable, so a Channel has attributes that allow the amount of detail in the output to be controlled.
The first of these is the integer attribute Full, which controls the extent to which optional, commented out, output lines are produced. By default, Full is zero, and this results in the standard style of output (§15.8) where default values that may be helpful to humans are included. To suppress these optional lines, Full should be set to 1. This is most conveniently done when the Channel is created, so that:
would result in output containing only the essential information, such as:
In contrast, setting Full to 1 will result in additional output lines which will reveal every last detail of the Object’s construction. Often this will be rather more than you want, especially for more complex Objects, but it can sometimes help when debugging programs. This is how a SkyFrame appears at this level of detail:
Another way of controlling output from a Channel is via the boolean (integer) Comment attribute, which controls whether comments are appended to describe the purpose of each value. Comment has the value 1 by default but, if set to zero, will suppress these comments. This is normally appropriate only if you wish to minimise the amount of output, for example:
might result in the following more compact output:
The safest advice about editing the textual output from astWrite (or astShow) is “don’t!”—unless you know what you are doing.
Having given that warning, however, it is sometimes possible to make changes to the text, or even to write entire Object descriptions from scratch, and to read the results back in to construct new Objects. Normally, simple changes to numerical values are safest, but be aware that this is a back door method of creating Objects, so you are on your own! There are a number of potential pitfalls. In particular:
By default, when you use astRead to read from a basic Channel (§15.4), it is assumed that you are reading a stream of text containing only AST Objects, which follow each other end-to-end. If any extraneous input data are encountered which do not appear to form part of the textual description of an Object, then an error will result. In particular, the first input line must identify the start of an Object description, so you cannot start reading half way through an Object.
Sometimes, however, you may want to store AST Object descriptions intermixed with other textual data. You can do this by setting the Channel’s boolean (integer) Skip attribute to 1. This will cause every read to skip over extraneous data until the start of a new AST Object description, if any, is found. So long as your other data do not mimic the appearance of an AST Object description, the two sets of data can co-exist.
For example, by setting Skip to 1, the following complete C program will read all the AST Objects whose descriptions appear in the source of this document, ignoring the other text. astShow is used to display those found:
Thus far, we have only considered the default behaviour of a Channel in reading and writing Objects through a program’s standard input and output streams. We will now consider how to access Objects stored in files more directly.
The simple approach is to use the SinkFile and SourceFile attributes of the Channel. For instance, the following will read a pair of Objects from a text file called “fred.txt”:
Note, the act of clearing the attribute tells AST that no more Objects are to be read from the file and so the file is then closed. If the attribute is not cleared, the file will remain open and further Objects can be read from it. The file will always be closed when the Channel is deleted.
This simple approach will normally be sufficient. However, because the AST library is designed to be used from more than one language, it has to be a little careful about reading and writing to files. This is due to incompatibilities that may exist between the file I/O facilities provided by different languages. If such incompatibilities prevent the above simple system being used, we need to adopt a system that off-loads all file I/O to external code.
What this means in practice is that if the above simple approach cannot be used, you must instead provide some simple C functions that perform the actual transfer of data to and from files and similar external data stores. The functions you provide are supplied as the source and/or sink function arguments to astChannel when you create a Channel (§15.2). An example is the best way to illustrate this.
Consider the following simple function called Source. It reads a single line of text from a C input stream and returns a pointer to it, or NULL if there is no more input:
Note that the input stream is a static variable which we will also access from our main program. This might look something like this (omitting error checking for brevity):
Here, we first open the required input file, saving the resulting FILE pointer. We then pass a pointer to our Source function as the first argument to astChannel when creating a new Channel. When we read an Object from this Channel with astRead, the Source function will be called to obtain the textual data from the file, the end-of-file being detected when this function returns NULL.
Note, if a value is set for the SourceFile attribute, the astRead function will ignore any source function specified when the Channel was created.
As for reading, writing Objects to files can be done in two different ways. Again, the simple approach is to use the SinkFile attribute of the Channel. For instance, the following will write a pair of Objects to a text file called “fred.txt”:
Note, the act of clearing the attribute tells AST that no more output will be written to the file and so the file is then closed. If the attribute is not cleared, the file will remain open and further Objects can be written to it. The file will always be closed when the Channel is deleted.
If the details of the language’s I/O system on the computer you are using means that the above approach cannot be used, then we can write a Sink function, that writes a line of output text to a file, and use it in basically the same way as the Source function in the previous section (§15.13):
Note that we must supply the final newline character ourselves.
In this case, our main program would supply a pointer to this Sink function as the second argument to astChannel, as follows:
Note that we can specify a source and/or a sink function for the Channel, and that these may use either the same file, or different files according to whether we are reading or writing. AST has no knowledge of the underlying file system, nor of file positioning. It just reads and writes sequentially. If you wish, for example, to reposition a file at the beginning in between reads and writes, then this can be done directly (and completely independently of AST) using standard C functions.
If an error occurs in your source or sink function, you can communicate this to the AST library by setting its error status to any error value using astSetStatus (§4.15). This will immediately terminate the read or write operation.
Note, if a value is set for the SinkFile attribute, the astWrite function will ignore any sink function specified when the Channel was created.
It should be obvious from the above (§15.13 and §15.14) that a Channel’s source and sink functions provide a flexible means of intercepting textual data that describes AST Objects as it flows in and out of your program. In fact, you might like to regard a Channel simply as a filter for converting AST Objects to and from a stream of text which is then handled by your source and sink functions, where the real I/O occurs.
This gives you the ability to store AST Objects in virtually any data system, so long as you can convert a stream of text into something that can be stored (it need no longer be text) and retrieve it again. There is generally no need to retain comments. Other possibilities, such as inter-process and network communication, could also be implemented via source and sink functions in basically the same way.