5 Importing Gridded Data

 5.1 Variations

It is usually possible to input a file written by some other program into DX. The procedure is simply to create a file which describes the contents of the data file to DX. This description file is called a ‘header file’ and is simply a text file created with an editor.

As an example this recipe shows how to import a formatted text file containing two data cubes. One data cube is a simple Gaussian field, the other is a Gaussian superimposed on a sloping background. Note that though the values are calculated using Fortran double precision variables they are written out as floating point numbers. DX interprets the ensuing file as containing single precision values. Each data cube is represented as a single DX field (see Section 17). The program to generate the data file is listed in Figure 7.

The DX header file for this data file is shown in Figure 5. These header files have file type ‘.general’. Thus the present example is called ‘field.general’.

  field=Gaussian, Sloping
  structure=scalar, scalar
  type=float, float
  dependency=positions, positions

Figure 5: Header file for gridded data.

Each line of the header file consists of a keyword followed by one or more values. The purpose of the various keywords are as follows.
is the name of the data file. If the file name is not preceded by a directory specification then it is assumed to be in the same directory as the header file.
is the format of the file; the current example is a formatted file rather than a binary file.
is the size of the grid of data. In the present case both data cubes comprise grids with fifty elements in each side.
specifies the organisation of multi-dimensional arrays. ‘Column’ majority means that the first dimension varies fastest (as in Fortran); ‘row’ majority means that the last dimension varies fastest (as in C).
specifies how the data for the two fields are interleaved within the file. DX permits various possibilities. For example, all the values for the first field could occur first, followed by all the values for the second field. In the present case the data for the two fields are more closely intertwined. Each record in the file contains a value for a position in the first field and a value for the corresponding position in the second field (see Figures 6 and 7). ‘interleaving=field’ specifies this sort of interleaving.

0.15655673        0.25655673
0.16572675        0.26572675
0.17514088        0.27514088
0.18476781        0.28476781
0.19457102        0.29457102
0.20450866        0.30450866

Figure 6: The first few records of file field.lis (written by program field.f, Figure 7). The first column is values for the simple Gaussian data cube, the second column values for the Gaussian superimposed on a sloping background.

specifies names for the fields in the file. Here they are called ‘Gaussian’ (the simple Gaussian data cube) and ‘Sloping’ (the Gaussian superimposed on a sloping background). Note the use of a comma to separate the two names.
specifies the structure of the two fields in the file. In the current example both the fields are simple scalars.
specifies the data types of the two fields. In the current example both fields are single precision real numbers.
denotes whether the data are position or connection dependent. See Section 17.4 for an explanation of these two cases. Here both fields are position dependent (which is probably the more common case in astronomy).

A full description of all the possible keywords is given in Section 4.3 Header File Syntax: Keyword Statements of the IBM QuickStart Guide[1].

Once a suitable description file has been been created the data can be imported into DX by including the ‘Import’ module in your network. Note that it is the name of the description file, not the name of the data file which must be supplied to ‘Import’.

5.1 Variations

DX includes ‘Data Prompter’, a windows-based application for interactively creating the header file for a data set. However, often it is no easier to use than creating the header file with an editor. Section 4.4 Data Prompter, of the QuickStart Guide[1] gives details.

Unformatted binary files can also be imported using the same header file mechanism. The format keyword should be set to ‘binary’. Binary files written with a C program can be imported directly. However, unfortunately, binary files written with a Fortran program contain record-control bytes which must be removed prior to importing the file. The Starlink extensions to DX include SXUnfort for this purpose; see SUN/203[11] for details. Of course, an unformatted file written on a Digital alpha will differ from the corresponding file written on a Sun because of the different byte order of the machines. It is possible to input an unformatted file written on a Sun into DX running on a Digital alpha, or vice versa, by using the ‘msb’ and ‘lsb’ modifiers to the format keyword4. See Section 4.3 Header File Syntax: Keyword Statements of the IBM QuickStart Guide[1] for details.
It is also possible to import files in the Starlink NDF format (and other common astronomical formats, such as FITS images or Figaro DST files). Again see SUN/203[11] for details.

  *    Generate two data cubes.  One contains a Gaussian function, the
  *    other a Gaussian on a sloping background.
  *    A C Davenhall (Edinburgh), 3/12/95.
       :  XCEN,   ! X Centre of distribution.
       :  YCEN,   ! Y   "    "       "      .
       :  ZCEN,   ! Z   "    "       "      .
       :  A,      ! Scale factor.
       :  B,      ! Exponential scale factor.
       :  C       ! Background slope.
       : (XCEN = 2.5D1,  YCEN = 2.5D1,  ZCEN = 2.5D1,
       :  A = 1.0D1,     B = 1.0D1,     C = 1.0D-1)
       :  DIST,   ! Distance of point from centre.
       :  VALUE1, ! Value of Gaussian function at the point.
       :  VALUE2  ! Value of Gaussian on sloping background at the point.
       :  I, J, K ! Loop indices.
        OPEN(UNIT=10, FILE=’field.lis’, STATUS=’NEW’)
        DO I = 1, 50
           DO J = 1, 50
              DO K = 1, 50
                 DIST = SQRT( ((REAL(I) - XCEN)**2) +
       :           ((REAL(J) - YCEN)**2) +
       :           ((REAL(K) - ZCEN)**2) )
                 VALUE1 = A*EXP(-DIST / B)
                 VALUE2 = VALUE1 + (C * REAL(I))
                 WRITE(10, ’(1X, 0PF16.8, 2X, 0PF16.8)’ ) VALUE1, VALUE2
              END DO
           END DO
        END DO

Figure 7: Program field.f to write gridded data.

4Strictly speaking it should be possible to input any unformatted file written in IEEE floating point format, irrespective of the type of machine that it was written on. Note, however, that Digital VAXen and IBM mainframes do not use IEEE floating point format.