10 Low-Level Structures

In this section, we begin by describing a number of low-level structures, the basic building blocks which can be used to build NDFs and other composite structures.

10.1 $<$POLYNOMIAL$>$ Structure

The $<$POLYNOMIAL$>$ structure is used for storing the coefficients of an $n$-dimensional ordinary or Chebyshev polynomial, or the coefficients of a cubic B-spline. The value of the [VARIANT] component defines which of these types of polynomial is stored. Evaluation of the polynomial yields a $<$float$>$ scalar result.

In the following descriptions NAXIS is the dimensionality of the [DATA_ARRAY] and therefore the dimensionality of the space over which the polynomial function is defined.

10.1.1 [VARIANT] = ‘SIMPLE’

[DATA_ARRAY]

This is mandatory. The scalar (VALUE) which results from the evaluation of the polynomial inherits the equivalent primitive TYPE of the polynomial coefficients.

The [DATA_ARRAY] is the $NAXIS$-dimensional array containing the $NAXIS$-dimensional polynomial coefficients. The function is evaluated as: $$\mathrm{VALUE}=\sum \mathrm{DATA<mstyle\; class="text"><mtext>\_</mtext></mstyle>ARRAY}\left(I_1,I_2,\dots ,I_{\mathrm{NAXIS}}\right)\prod _{j=1}^{\mathrm{NAXIS}}{\mathrm{AXIS}}_j{}^{I_j-1}$$

summed over all elements of the DATA_ARRAY.

Here, AXIS${}_n$ is the co-ordinate value along axis $n$, and NAXIS${}_n$ is the dimension of axis $n$.

[VARIANCE]

This optional component is used to describe the variance of the errors associated with the polynomial coefficients given by the [DATA_ARRAY] component, and it must be an array of the same dimension as [DATA_ARRAY].

10.1.2 [VARIANT] = ‘CHEBYSHEV’

[DATA_ARRAY]

This is mandatory. The interpretation The scalar (VALUE) which results from the evaluation of the polynomial inherits the equivalent primitive TYPE of the polynomial coefficients.

The [DATA_ARRAY] contains the NAXIS-dimensional array of Chebyshev-polynomial coefficients. The function is evaluated as:

$$\mathrm{VALUE}=\sum \mathrm{DATA<mstyle\; class="text"><mtext>\_</mtext></mstyle>ARRAY}\left(I_1,I_2,\dots ,I_{\mathrm{NAXIS}}\right)\prod _{j=1}^{\mathrm{NAXIS}}T\left(I_{j-1},S_j\right)$$

summed over all elements of the DATA_ARRAY, where $T\left(m,x\right)$ is the Chebyshev polynomial of order $m$ evaluated at $x$ (in the range $-$1 to $+$1), and

$$S_n=(2\ast {\mathrm{AXIS}}_n-\left(\mathrm{TMAX}\left(n\right)+\mathrm{TMIN}\left(n\right)\right)/\left(\mathrm{TMAX}\left(n\right)-\mathrm{TMIN}\left(n\right)\right)$$

If the polynomial is evaluated outside this interval, the result is bad, and set to the magic value.

[VARIANCE]

This optional component is used to describe the variance of the errors associated with the Chebyshev polynomial coefficients given by the [DATA_ARRAY] component, and it must be an array of the same dimension as [DATA_ARRAY].

[TMIN($n$)], [TMAX($n$)]

These are the minimum and maximum values along AXIS number $n$. They represent the bounds of the normalised co-ordinate range. They are mandatory.

10.1.3 [VARIANT] = ‘BSPLINE’

The [DATA_ARRAY] contains the NAXIS-dimensional array of cubic-spline coefficients. Its dimension along an axis is equal to the total number of knots along that axis less four. The function is evaluated as:

$$\mathrm{VALUE}=1/\mathrm{SCALE}\sum \mathrm{DATA<munder\; accentunder="false"\; class="mml-underline"><mrow><mspace\; width="1em"\; class="nbsp"><mspace\; width="1em"\; class="nbsp"></mspace></mspace></mrow><mo\; accent="true">̲</mo></munder>ARRAY}\left(I_1,I_2,\dots ,I_{\mathrm{NAXIS}}\right)\prod _{j=1}^{\mathrm{NAXIS}}{B}_j\left({\mathrm{AXIS}}_j\right)$$

where $B_j$(AXIS${}_j$) is the normalised cubic B-spline along the $j$th axis with the knots recorded in [KNOTS], and SCALE is the value of component [SCALE].

[VARIANCE]

This optional component is used to describe the variance of the errors associated with the spline coefficients given by the [DATA_ARRAY] component, and it must be an array of the same dimension as [DATA_ARRAY].

[KNOTS]

A vector of knot co-ordinates, starting with those along the first dimension, followed by any second dimension, and so on. It has a dimension that is the sum of the number of knots along all axes.

[SCALE]

The normalisation factor to convert the evaluated spline into the fitted value.

10.2 $<$ARRAY$>$Structure

We will now describe the various structures for arrays of numbers introduced in Section 3.3. An $<$ARRAY$>$ structure has a number of variants.

10.2.1 [VARIANT] = ‘SIMPLE’

[ORIGIN]

The pixel origin in each dimension. If this object is not present, each origin is assumed to be 1. Note that this ability, though present in FORTRAN77, is absent from HDS per se.

This information may be communicated to access routines via the upper and lower bounds on each array dimension, so that the Fortran dimensions would be

      ARRAY(ORIGIN(1):DIMS(1)+ORIGIN(1)-1, ORIGIN(2):DIMS(2)+ORIGIN(2)-1, ...).

[ORIGIN] overcomes a number of quite serious problems, including the old chestnut “does the first pixel start at (1,1) or (0,0)?”, because the origin is specified explicitly. Also negative pixel indices become available, and therefore an application is freed from having every array co-located at the first pixel, and it may extend arrays in any direction. This property is particularly useful for storing data which require negative indices for a natural representation, such as Fourier-transform data, where zero often needs to be in the centre of the array; or image data after (say) rotation about any point. Storing a pixel origin also enable applications to cut out parts of a data array yet maintain the array’s coordinate system (it can be re-combined with the original if necessary). Since the coordinate system is preserved, all the axis scaling is also preserved. This means, for instance, that applications do not have to re-compute the polynomial coefficients in an $<$AXIS$>$ structure when the array size is changed.

NAXIS

The number of axes in the array being represented.

[DATA]

This mandatory item is the NAXIS-dimensional array of numbers.

10.2.2 [VARIANT] = ‘SCALED’

A scaled $<$ARRAY$>$ is sometimes referred to as block floating point. When dealing with large-format data, disk space is a major consideration; significant space may be saved by storing data in a 16-bit form via the block floating-point format. Scaling may also be used as a means of implementing global scalar changes in the array’s values without the need to change the array, or to move to a more expensive storage type (e.g. to $<$float$>$ from $<$_WORD$>$) in order to work with very small or very large values.

General-purpose applications will process the scaled variant. However, users may notice extra delays while the array is converted to $<$narray$>$ for processing and rescaled for storage at the end of each application. In such cases, they may prefer the alternative strategy of first running a utility application to convert the data to primitive floating point in a new container file, then performing their processing, and finally running another utility to convert the data back to the scaled-array form.

The numeric value of a pixel is:

NAXIS

The number of axes in the array being represented.

[ORIGIN]

The pixel origin in each dimension. If this object is not present, each origin is assumed to be 1.

[DATA]

The array stored in scaled (block-floating-point) format. Mandatory.

[SCALE]

A positive scale factor used to convert the element of [DATA] to its unscaled $<$numeric$>$ form. If it is not present, or is negative, or is bad, an error condition results. It is evaluated as follows:

  
        SCALE = (MAX - MIN)/(TMAX<T> - TMIN<T>)
  

where MAX, MIN are the maximum and minimum valid (i.e. not bad) data values respectively, and TMAX<T>, TMIN<T> are the largest and smallest valid values for the primitive type of [DATA], represented by token <T>.

[ZERO]

The zero point used to convert the element of [DATA] to its unscaled $<$numeric$>$ form. If it is not present, or is undefined, a value of zero is assumed. It is evaluated as follows:

  
        ZERO = TMIN<T> - SCALE * MIN
  

The equivalent primitive type is the TYPE of [ZERO] or, if [ZERO] is absent, the TYPE of [SCALE]. If both [ZERO] and [SCALE] are missing the equivalent $<$numeric$>$ array defaults to TYPE $<$_REAL$>$.

10.2.3 [VARIANT] = ‘SPACED’

A spaced $<$ARRAY$>$ is one where the data values vary linearly with array index. An example where this variant could be used is axis data, which are often equally spaced.

General-purpose applications will handle these structures. However, extra time will be expended (though probably much less than for a scaled $<$ARRAY$>$) whilst conversion to and from a $<$narray$>$ is performed. They do not contain magic-value bad pixels.

The value of an element of the equivalent $<$numeric$>$ array would be: $$\mathrm{VALUE}=\sum _{i=1}^{\mathrm{NAXIS}}\left(\mathrm{BASE}\left(i\right)+\left(k_i-1\right)\ast \mathrm{SCALE}\left(i\right)\right)$$

where $k_i$ is the element number of the $i^{\mathrm{th}}$ dimension.

NAXIS

The number of axes in the array being represented.

[ORIGIN]

The pixel origin in each dimension. If this object is not present, each origin is assumed to be 1.

[SCALE]

The scale factor for each dimension used to convert the corresponding array index of [DATA] to its $<$numeric$>$ form. If it is not present, or is undefined, a value of 1 is assumed for each dimension.

[BASE]

The zero point for each dimension used to convert the corresponding array index of [DATA] to its $<$numeric$>$ form. If it is not present, or is undefined, a value of 0 is assumed for each dimension.

[DIMENSIONS]

The axis dimensions of the full array. Mandatory. The equivalent primitive type is the TYPE of [BASE], or if [BASE] is absent, the TYPE of [SCALE]. If both [BASE] and [SCALE] are missing, the equivalent $<$numeric$>$ array defaults to TYPE $<$_REAL$>$.

10.2.4 [VARIANT] = ‘SPARSE’

A sparse $<$ARRAY$>$ structure is used for storing the values of an array where most of the pixels are equal in value (zero perhaps). The structure is composed of a list of array indices for the specified “non-grey” elements, and their corresponding data values. The “grey” value adopted for the majority of the array elements is also supplied.

NAXIS

The number of axes in the array being represented.

[ORIGIN]

The pixel origin in each dimension. If this object is not present, each origin is assumed to be 1.

NDATA

This is the number of non-grey array elements.

[DATA]

These are the data values for significant array elements. Mandatory.

[LIST]

These are the array indices for each significant array value. Mandatory.

[DIMENSIONS]

This array contains the axis dimensions of the full data array. It is used to generate the equivalent $<$narray$>$. Mandatory.

[GREY]

This is the value used to fill the array elements not referenced in the [LIST] component. It should have the same primitive TYPE as [DATA]. If absent, it is assumed to be the bad-pixel magic value (for [DATA]’s primitive type) since the pixel’s value is undefined.

The equivalent primitive type is the TYPE of [DATA].

10.2.5 [VARIANT] = ‘POLYNOMIAL’

The array is represented by an $n$-dimensional polynomial. (This variant does not conform to the guideline about variants since conversion of an array of numbers to a polynomial requires additional information. However, for clarity and because the reverse operation is possible—polynomial to an array of numbers—a variant of $<$ARRAY$>$ has been used.)

NAXIS

The number of axes in the array being represented.

[ORIGIN]

The pixel origin in each dimension. If this object is not present, each origin is assumed to be 1. This avoids recomputing the polynomial coefficients if a subsection of a larger array is created.

[DATA]

The NAXIS-dimensional polynomial representing the array of numbers. Mandatory. The equivalent primitive type has the same TYPE as the polynomial coefficients. To generate the equivalent primitive array, the polynomial is evaluated at pixel centres, which start at 0.5 for the first pixel given a uniform bin.

[DIMENSIONS]

This array contains the axis dimensions of the full data array. It is used to generate the equivalent $<$narray$>$. Mandatory.

The ‘SPACED’ variant may appear merely to be a degenerate form of the ‘POLYNOMIAL’ variant. However, there are reasons for having a ‘SPACED’ variant:

• the case where the equivalent primitive type is $<$integer$>$ is common, whereas the polynomial variant only produces $<$float$>$ results.
• The spaced $<$ARRAY$>$ uses the pixel indices to generate the values, whereas a polynomial array uses the pixel co-ordinates. Pixel indices are integers, starting at 1; pixel co-ordinates are real numbers, which start at 0.5 for uniformly-sized bins.
• The spaced $<$ARRAY$>$ format will be easier to understand in a structure listing.

To reiterate, general-purpose applications will be able to process $<$ARRAY$>$ structures, though their performance will be degraded when the [VARIANT] is not ‘SIMPLE’ because of the extra processing required. These options do not create a significant overhead for the normal case, the base variant. When the [VARIANT] is not ‘SIMPLE’, users will probably be warned that their data are in a special format, be given the option to abort the application, and be directed to documentation for applications better suited to the data.

10.3 $<$COMPLEX_ARRAY$>$ Structure

HDS does not provide a $<$_COMPLEX$>$ primitive type, and therefore if complex data are to be stored a $<$COMPLEX_ARRAY$>$ structure, defined in Table 19 must be used.

Currently, the [[VARIANT]] can only be ‘SIMPLE’.

The dimensions of [REAL] and [IMAGINARY] must be identical. Both components are mandatory. The pixel origin will be taken from [REAL] component’s structure, i.e. [REAL.ORIGIN].

If a pixel is undefined, the magic value should be assigned to both corresponding elements of the real and imaginary arrays. However, should the application encounter the case where only one element is flagged, the whole pixel should be regarded as undefined.

The [ORIGIN] components should match, but only [REAL.ORIGIN] is used.

10.4 $<$AXIS$>$ Structure

This structure has TYPE $<$AXIS$>$ and is used to store information describing the size and spacing of the pixels in a multi-dimensional data array. In use, $<$AXIS$>$ structures are arranged as elements of a 1-dimensional array (of structures). The number of elements (structures) in this array is equal to the dimensionality (number of axes) of the data array being described, so that each $<$AXIS$>$ structure relates to a single data dimension. The following description applies to one element of that multi-dimensional structure.

[VARIANT]

Currently, this can only be ‘SIMPLE’.

[DATA_ARRAY]

This mandatory component, which can be the only item within the $<$AXIS$>$ structure, is used to provide co-ordinates along the axis. Each value corresponds to the centre of a pixel. For an $n$-dimensional data array, with no axis calibration, and equally spaced data, the Starlink standard is for the $n^{\mathrm{th}}$ pixel centre to be at $\left(n-1\right)$.5 in each axis, e.g. for display purposes (cf. SSN/22). However, in some applications the pixel index is adequate and clearer. Programmers should take care to use the appropriate scheme.

The co-ordinates are given as a vector with length corresponding to that of the corresponding dimension of the data array. In practice, it will often be the polynomial or spaced variant of TYPE $<$ARRAY$>$.

[LABEL]

This is a textual description of the co-ordinate type. It may contain the “control” codes that are used by graphics packages to produce special characters—backslashes etc.

[UNITS]

This is a textual description of the co-ordinate units.

[VARIANCE]

This component is used to describe the variance of the errors associated with co-ordinate values given by the [DATA_ARRAY] component. It must either be an array of the same dimension as [DATA_ARRAY] or, should a single variance apply to all elements of [DATA_ARRAY], a scalar.

[NORMALISED]

If true, the data have been normalised to the pixel size (i.e. divided by the pixel width along this axis). In other words if the data are normalised the data values reduce as the pixels are stretched and vice versa. Optional. If absent, it defaults to false.

[WIDTH]

The width or extent of each bin for irregularly spaced and/or overlapping data. If it is an array, it must have the same dimensions as the [DATA_ARRAY]. The scalar form applies to all the pixels and is intended for overlapping pixels. Optional. If absent, the extent of the $n^{\mathrm{th}}$ element is assumed to be $$0.5\left(D_{n-1}+D_n\right)\mathrm{to}0.5\left(D_n+D_{n+1}\right),$$

where $D_n$ is the $n^{\mathrm{th}}$ centre value given by [DATA_ARRAY]. The extreme values are twice the available half widths, viz. $$D_n-0.5\ast \mathrm{WIDTH}\mathrm{to}{D}_n+0.5\ast \mathrm{WIDTH}$$

for the $n^{\mathrm{th}}$ element.

[MORE]

The extension in which application-specific axis-related information can be stored, e.g. whether the axis is cyclic, the type of axis: spectral, spatial, temporal etc.

10.5 $<$QUALITY$>$ Structure

This structure is used for storing data-quality information.

[VARIANT]

Currently can only be ‘SIMPLE’.

[QUALITY]

The data-quality value(s). If it is scalar it applies to all the pixels. The actual or equivalent primitive type must be $<$_UBYTE$>$. Mandatory.

The rôle of data-quality and how it can be used to advantage were described earlier.

[BADBITS]

A set of 8 one-bit flags which flag that the corresponding data quality bits indicate bad pixels; if omitted a default value of zero (i.e. data quality is ignored in deciding whether a pixel is bad or not) is assumed.

10.6 $<$HISTORY$>$ Structure

This structure has NAME [HISTORY] and TYPE $<$HISTORY$>$. Its components are shown in Table 22.

[VARIANT]

Currently, this can only be ‘SIMPLE’.

[CREATED]

The date and time when the HISTORY structure was created, derived from the computer’s clock. Mandatory. The format for dates and times is as following examples:

      1988/AUG/07 23:59:59.999
      2001-JAN-01 01:02:03.456

[EXTEND_SIZE]

The number of history records to be created whenever the existing records have been filled. The default is 5. Initially 10 records are created. (The item [INITIAL_SIZE] in the original ASTERIX history system is redundant and has been removed.)

[CURRENT_RECORD]

The number of the current history record. Mandatory.

[RECORDS]

An array of $<$HIST_REC$>$ structures containing the history records. Mandatory.

[VARIANT]

Currently, this can only be ‘SIMPLE’.

[DATE]

The date and time when the record was created, derived from the computer’s clock. Mandatory.

[COMMAND]

The name and version number of the application or command which modified or created the structure at or below the level in the hierarchy of this $<$HISTORY$>$ object. Mandatory.

[TEXT]

History text, which can span an unlimited number of records. It may contain commentary or details of input parameters. Parameters should be converted to text using environment calls.

Note there are currently no variants of the $<$HISTORY$>$ and $<$HIST_REC$>$ structures.