“NDF” stands for Extensible N-Dimensional Data Format. It is a standard file format for storing data which represent N-dimensional arrays of numbers, such as spectra, images, etc. and it therefore forms the basis of many spectral and image-processing applications. This document describes a subroutine library for accessing data stored in this form from applications written to run within a programming environment (such as ADAM – see SUN/101 & SG/4). For applications which do not use such facilities, a “stand-alone”, version of the library is also available (see §24).
The main reason for using NDF data structures as a standard method of storing astronomical data is to simplify the exchange of information between separate applications packages. In principle, this should make it possible for a software user to process the same data using software drawn from any package.
In practice, previous attempts to define a standard data format for this purpose have met with two serious obstacles. First, different authors of software have interpreted the meaning of data items differently, so that although several software packages might be capable of reading the same data files, the different packages actually performed incompatible operations on the data. Secondly, many software authors have found a pre-defined data format to be too restrictive, and have simply chosen not to use it.
The NDF data structure has therefore had to satisfy two apparently quite contradictory requirements:
The solution to this problem has been to introduce the concept of extensibility, and to divide the NDF data structure into two parts – a set of standard components and a set of extensions – each of which individually satisfies one of these two requirements. An NDF data structure therefore consists of a central “core” of information, whose interpretation is well-defined and for which general-purpose software can be written with wide applicability, together with an arbitrary number of extensions which may be used by specialist software but whose interpretation is not otherwise defined. Those who wish to know more of the background to this philosophy can find a detailed discussion in SGP/38.
Most of the present document is concerned with the facilities provided by the fixed “core” of standard components, although a few general routines for handling extensions are also included.
The NDF file format is based upon the Hierarchical Data System HDS (SUN/92) and NDF data structures are stored in HDS container files (which by convention have a file type of .sdf). However, this does not necessarily mean that all applications which can read HDS files can also handle data stored in NDF format.
To understand why, you must appreciate that HDS provides only a rather low-level set of facilities for storing and handling astronomical data. These include the ability to store primitive data objects (such as arrays of numbers, character strings, etc.) in a convenient and self-describing way within container files. However, the most important aspect of HDS is its ability to group these primitive objects together to build larger, more complex structures. In this respect, HDS can be regarded as a construction kit which other higher-level software can use to build even more sophisticated data formats.
The NDF is a higher-level data format which has been built in this way out of the more primitive facilities provided by HDS. Thus, in HDS terms, an NDF is a data structure constructed according to a particular set of conventions to facilitate the storage of typical astronomical data (such as spectra, images, or similar objects of higher dimensionality).
While HDS can be used to access such structures, it does not contain any of the interpretive knowledge needed to assign astronomical meanings to the various components of an NDF, whose details can become quite complicated. In practice it is therefore cumbersome to process NDF data structures using HDS directly. Instead, the NDF access routines described in this document are provided. These routines “know” about how NDF data structures are build within HDS, so they can hide these details from the writers of astronomical applications. This results in a subroutine library which deals in higher-level concepts more closely related to the work which typical astronomical applications need to perform, and which emphasises the data concepts which an NDF is designed to represent, rather than the details of its implementation.
It is assumed that the reader of this document has some background knowledge of the programming environment being used (e.g. see SUN/101 & SG/4 for details of ADAM) and with the basic concepts used by the HDS data system (SUN/92).
A knowledge of the method by which the NDF is implemented using HDS (SGP/38) may also be useful as background reading, but is not essential for understanding the present document which is intended to be self-contained in its description of NDF concepts and facilities.
A small number of the basic facilities described in SGP/38 still remain to be supported, although most of the major facilities are now available through the interface described here. In addition, the WCS component has been introduced, going beyond the original NDF design in SGP/38. In cases of doubt, the presence or absence of a description of an NDF facility in this document should be taken as indicating the extent of the current implementation.