Chapter 15
The Data System

In this chapter, a bottom-up approach is adopted — the low-level packages are described first, followed by the higher-level ones.

15.1 HDS — Hierarchical data system

15.1.1 Symbolic names and Include files

Symbolic names should be used for important constant values in HDS programs to make them clearer and to insulate them from possible future changes in their values. These symbolic names are defined by Fortran ‘include’ files as shown in the examples below. The following include files are available:

The symbolic names of these include files may be used directly on VAX/VMS systems, but on Unix systems an explicit directory specification for the file is normally also required, and the file name should appear in lower case. Thus, to include the DAT_PAR file on a VAX/VMS system, the following code would be used:

whereas on a Unix system, the following code is required:

(/star/include is a standard directory containing include files on Starlink machines running Unix.)

If it is necessary to test for specific error conditions, the appropriate include file and symbolic names should be used in your program. Here is an example of how to use these symbols:

15.1.2 Creating objects

To fix ideas, look at the example data structure in Fig 15.1. This is actually one form of NDF structure, but for the purposes of this chapter it will be treated as if it were simply an arbitrary HDS structure, i.e. we will use HDS routines rather than NDF routines to process it. The following notation is used to describe each object:

Note that scalar objects have no dimensions and that each level down the hierarchy is indented.

This example exhibits several of the most important properties of HDS data objects.

• Both structures and primitives are present in the structure.
• Scalar and non-scalar objects are present.
• You can have arrays of structures, e.g. the AXIS component is a vector structure (with two elements).

The following code will create the structure in Fig 15.1:

The following points should be borne in mind:

• The structure created is in no way static — new objects can be added or existing ones deleted at any level without disturbing what already exists. (Remember, however, that there are very strong rules about what can and cannot be put into an NDF structure. In general, the NDF routines of Section 15.3 should be used to manipulate NDFs.)
• No primitive values have been stored yet — that will be done next.

Here are some notes on particular aspects of this example:

15.1.3 Writing and reading objects

Having created a structure, the next step will usually be to put some values into it. This can be done by using the DAT_PUT and DAT_PUTC routines. For example, the main data array in the above example could be filled with values as follows:

Because this sort of activity occurs quite often, packaged access routines have been provided (see Section 15.1.10) for the programmer.

A complementary set of routines also exists for getting data from objects back into program arrays or variables; these are the DAT_GET routines. Again, packaged versions exist and are often handy in reducing the number of subroutine calls required.

15.1.4 Accessing objects by mapping

Another technique for accessing the data values stored in primitive HDS objects is termed ‘mapping’.² An important advantage is that it removes a size restriction imposed by having to declare fixed size program arrays to hold data. This simplifies software, so that a single routine can handle objects of arbitrary size without recourse to accessing subsets.

HDS provides mapped access to primitive objects via the DAT_MAP routines. Essentially, DAT_MAP will return a pointer to a region of the computer’s memory in which the object’s values are stored. This pointer can then be passed to another routine using the VAX Fortran ‘%VAL’ facility.³ An example will illustrate this:

This example illustrates two features of HDS which we haven’t yet mentioned:

Note that once a primitive has been mapped, the associated locator cannot be used to access further data until the original object is unmapped.

15.1.5 Mapping character data

Although the above example used a numeric type of ‘_REAL’ to access the data, HDS allows any primitive type to be specified as an access type, including ‘_CHAR’. It gives you a choice about how to determine the length of the character strings it will map. You may either specify the length you want explicitly, e.g:

(in which case HDS would map an array of character strings with each element containing 30 characters) or you may leave HDS to decide on the length required by omitting the length specification, thus:

In the latter case, HDS will determine the number of characters actually required to format the object’s values without loss of information. It uses decimal strings for numerical values and the values ‘TRUE’ and ‘FALSE’ to represent logical values as character strings. If the object is already of character type, then its actual length will be used directly. The routine DAT_MAPC also operates in this manner.

You should consult SUN/92 for details of how to use this facility on different machines. It is one of the areas where it is very difficult to produce a mechanism which works properly on all machines.

15.1.6 Copying and deleting objects

HDS can also copy and delete objects. Routines DAT_COPY and DAT_ERASE will recursively copy and erase all levels of the hierarchy below that specified in the subroutine call:

Note that the locator to the AXIS object has been annulled before attempting to delete it. This whole operation can also be done using DAT_MOVE:

15.1.7 Subsets of objects

The routine DAT_CELL accesses a single element of an array. An example was shown in Section 15.1.2. The routine DAT_SLICE accesses a subset of an arbitrarily dimensioned object. This subset can then be treated as if it were an object in its own right. For example:

In contrast to DAT_SLICE, routine DAT_ALTER makes a permanent change to a non-scalar object. The object can be made larger or smaller, but only in the last dimension. This function is entirely dynamic, i.e. it can be done at any time, provided the object is not mapped for access. Note that DAT_ALTER works on both primitives and structures. It is important to realise that the number of dimensions cannot be changed by DAT_ALTER.

15.1.8 Temporary objects

Temporary objects of any type and shape may be created by using the DAT_TEMP routine. This returns a locator to the newly created object, and this may then be manipulated just as if it were an ordinary object (in fact a temporary container file is created with a unique name to hold all such objects, and this is deleted when HDS_STOP is executed at the end of the program). This is often useful for providing workspace for algorithms which may have to deal with large arrays.

15.1.9 Enquiries

One of the most important properties of HDS is that its data files are self-describing. This means that each object carries with it information describing all its attributes (not just its value), and these attributes can be obtained by means of enquiry routines. An example will illustrate:

Here, DAT_INDEX is used to get locators to objects about which (in principle) we know nothing. This is just like listing the files in a directory, except that the order in which the components are stored in an HDS structure is arbitrary (so they won’t necessarily be accessed in alphabetical order).

15.1.10 Packaged routines

HDS includes families of routines which provide a more convenient method of accessing objects than the basic routines. For instance, members of the family DAT_PUT$\ast$ write values of specific type and dimensionality, and the DAT_GET$\ast$ routines read similar values. Thus DAT_PUT0I will write a single INTEGER value to a scalar primitive, and DAT_GET1R will read the value of a vector primitive and store it in a REAL program array. There are no DAT_GET2x routines; all dimensionalities higher than one are handled by DAT_GETNx and DAT_PUTNx.

Another family of routines are the CMP routines. These access components of the ‘current level’. This usually involves:

• FINDing the required object and getting a locator to it.
• Performing the required operation, e.g. PUTting some value into it.
• ANNULling the locator.

The CMP routines package this sort of operation, replacing three or so subroutine calls with one. The naming scheme is based on the associated DAT routines. An example is shown below.

15.2 REF — References to HDS objects

Reference objects are HDS objects which store references to other HDS data objects. They act as pointers to data, rather than storing data themselves.

The REF package allows reference objects to be created and written, and it allows locators to referenced objects to be obtained. The routines are listed in Section 21.2.3.

The referenced object may be defined as internal, in which case it is assumed to be within the same container file as the reference object itself, even if the reference object is copied to another container file. In that case the reference must point to an object which has the same pathname within the new file as it had in the old one. References which are not internal will point to a named container file.

Reference objects may be copied and erased using DAT_COPY and DAT_ERASE. Care must be taken when copying reference objects or referenced objects, otherwise the reference may no longer point to the referenced object.

Referenced objects must exist at the time the reference is made or used.

15.2.1 Uses

Two main uses for this package are foreseen:

• To maintain a catalogue of data objects.
• To avoid duplicating a large data object.

As an example of the second use, suppose that a large object is logically required to form part of a number of other objects. To avoid duplicating the common object, the others may contain a reference to it. For example:

Then a piece of code which handles structures of type SPECTRUM, which would normally contain the axis data in .AXIS1 (as SET1 does), could be modified as follows to handle an object .AXIS1 containing either the actual axis data or a reference to the object which does contain the actual axis data.

This code has been packaged into the subroutine REF_FIND which can be used instead of DAT_FIND in cases where the component requested may be a reference object.

When a locator which has been obtained in this way is finished with, it should be annulled using REF_ANNUL rather than DAT_ANNUL. This is so that, if the locator was obtained via a reference, the HDS_OPEN for the container file may be matched by an HDS_CLOSE. Note that this should only be done when any other locators derived from the locator to the referenced object are also finished with.

15.3 NDF — Extensible n-dimensional data format

The programmer’s manual (SUN/33) for NDF runs to 199 pages. It is impossible to produce a useful summary of it, and pointless to reproduce the whole of it here. Instead, we give you an overview of how NDFs are related to HDS, what is in an NDF, and the sort of functions provided by the NDF routines. A realistic, fully commented example program is also given, which should at least give you an idea of how the routines are used. The routines are listed in Section 21.2.1.

15.3.1 Relationship with HDS

The NDF file format is based on HDS, and NDF data structures are stored in HDS container files. However, this does not necessarily mean that all applications which can read HDS files can also handle data stored in NDF format.

To understand why, you must appreciate that HDS provides only a rather low-level set of facilities for storing and handling astronomical data. These include the ability to store primitive data objects (such as arrays of numbers, character strings, etc.) in a convenient and self-describing way within container files. However, the most important aspect of HDS is its ability to group these primitive objects together to build larger, more complex structures. In this respect, HDS can be regarded as a construction kit which higher-level software can use to build even more sophisticated data formats.

The NDF is a higher-level data format which has been built in this way out of the more primitive facilities provided by HDS. Thus, in HDS terms, an NDF is a data structure constructed according to a particular set of conventions to facilitate the storage of typical astronomical data (such as spectra, images, or similar objects of higher dimensionality).

While HDS can be used to access such structures, it does not contain any of the interpretive knowledge needed to assign astronomical meanings to the various components of an NDF, whose details can become quite complicated. In practice, therefore, it is cumbersome to process NDF data structures using HDS directly. Instead, the NDF access routines are provided. These ‘know’ how NDF data structures are built, so they can hide the details from writers of astronomical applications. This results in a subroutine library which deals in higher-level concepts more closely related to the work which typical astronomical applications need to perform, and which emphasises the data concepts which an NDF is designed to represent, rather than the details of its implementation.

15.3.2 Data format

The simplest way of regarding an NDF is to view it as a collection of those items which might typically be required in an astronomical image or spectrum. The main part is an $N$-dimensional array of data (where $N$ is 1 for a spectrum, 2 for an image, etc.), but this may also be accompanied by a number of other items which are conveniently categorised as follows:

The names of these components are significant, since they are used by the NDF access routines to identify the component(s) to which certain operations should be applied.⁶ The following describes the purpose and interpretation of each component in slightly more detail.

Character components:

Array components:

Miscellaneous components:

Extensions:

15.3.3 Routines

The NDF access routines are listed in Section 21.2.1. They perform the following types of operation on NDF data structures:

Programming support for these routines, including on-line help, is also provided by the Starlink language-sensitive editor STARLSE.

15.3.4 Example program

The following application adds two NDF data structures pixel-by-pixel. It is a fairly sophisticated ‘add’ application which will handle both the data and variance components, as well as coping with NDFs of any shape and data type.

The following is a possible interface file for the above application: