12 Dealing with Files

12.1 Extracting parts of filenames

Occasionally you’ll want to work with parts of a filename, such as the path or the file type. The C-shell provides filename modifiers that select the various portions. A couple are shown in the example below.

Suppose the first argument of a script, $1, is a filename called galaxy.bdf. The value of variable type is bdf and name equates to galaxy because of the presence of the filename modifiers :e and :r. The rest of the script uses the file type to control the processing, in this case to provide a listing of the contents of a data file using the Hdstrace utility.

The complete list of modifiers, their meanings, and examples is presented in the table below.

12.2 Process a Series of Files

One of the most common things you’ll want to do, having devised a data-processing path, is to apply those operations to a series of data files. For this you need a foreach...end construct.

12.2.1 NDFs

Some data formats like the NDF demand that only the file name (i.e. what appears before the last dot) be given in commands. To achieve this you must first strip off the remainder (the file extension or type) with the :rfile modifier.

This processes all the HDS files in the current directory and calculates an histogram for each of them using the histogram command from Kappa. It assumes that the files are NDFs. The :r instructs the shell to remove the file extension (the part of the name following the the rightmost full stop). If we didn’t do this, the histogram task would try to find NDFs called SDF within each of the HDS files.

12.2.2 Wildcarded lists of files

You can give a list of files separated by spaces, each of which can include the various UNIX wildcards. Thus the code below would report the name of each NDF and its standard deviation. The NDFs are called ‘Z’ followed by a single character, ccd1, ccd2, ccd3, and spot.

echo writes to standard output, so you can write text including values of shell variables to the screen or redirect it to a file. Thus the output produced by stats is piped (the | is the pipe) into the UNIX grep utility to search for the string "Standard deviation". The ‘  ‘ invokes the command, and the resulting standard deviation is substituted.

You might just want to provide an arbitrary list of NDFs as arguments to a generic script. Suppose you had a script called splotem, and you have made it executable with chmod +x splotem.

Notice the -e file-comparison operator. It tests whether the file exists or not. (Section 12.4 has a full list of the file operators.) To plot a series of spectra stored in NDFs, you just invoke it something like this.

See the glossary for a list of the available wildcards such as the [a-z] in the above example.

12.2.3 Exclude the .sdf for NDFs

In the splotem example from the previous section the list of NDFs on the command line required the inclusion of the .sdf file extension. Having to supply the .sdf for an NDF is abnormal. For reasons of familiarity and ease of use, you probably want your relevant scripts to accept a list of NDF names and to append the file extension automatically before the list is passed to foreach. So let’s modify the previous example to do this.

On the command line the wildcards have to be passed verbatim, because the shell will try to match with files than don’t have the .sdf file extension. Thus you must protect the wildcards with quotes. It’s a nuisance, but the advantages of wildcards more than compensate.

12.2.4 Examine a series of NDFs

A common need is to browse through several datasets, perhaps to locate a specific one, or to determine which are acceptable for further processing. The following presents images of a series of NDFs using the display task of Kappa. The title of each plot tells you which NDF is currently displayed.

You can substitute another visualisation command for display as appropriate. You can also use the graphics database to plot more than one image on the screen or to hardcopy. The script $KAPPA_DIR/multiplot.csh does the latter.

12.3 Filename modification

Thus far the examples have not created a new file. When you want to create an output file, you need a name for it. This could be an explicit name, one derived from the process identification number, one generated by some counter, or from the input filename. Here we deal with all but the trivial first case.

12.3.1 Appending to the input filename

To help identify datasets and to indicate the processing steps used to generate them, their names are often created by appending suffices to the original file name. This is illustrated below.

12.3.2 Appending a counter to the input filename

If a counter is preferred, this example

12.3.3 Appending to the input filename

Whilst appending a suffix after each data-processing stage is feasible, it can generate some long names, which are tedious to handle. Instead you might want to replace part of the input name with a new string. The following creates another shell variable, ndfout by replacing the string _flat from the input NDF name with _sm. The script pipes the input name into the sed editor which performs the substitution.

12.4 File operators

Here are some other examples.

12.5 Creating text files

A frequent feature of scripts is redirecting the output from tasks to a text file. For instance,

If you intend to write more than once to a file you should first create the file with the touch command, and then append output to the file.

There is an alternative—write the text file as part of the script. This is often a better way for longer files. It utilises the cat command to create the file.

A handy feature is that you can embed shell variables, such as refndf and distance in the example. You can also include commands between left quotes (‘ ‘); the commands are evaluated before the file is written. However, should you want the special characters $, \, and ‘ ‘ to be treated literally insert a \ before the delimiting word or a \ before each special character.

12.5.1 Writing a script within a script

The last technique might be needed if your script writes another script, say for overnight batch processing, in which you want a command to be evaluated when the second script is run, not the first. You can also write files within this second script, provided you choose different words to delimit the file contents. Here’s an example which combines both of these techniques.

This excerpt writes a script in the temporary directory. The temporary script’s filename includes our username ($user) and some run number stored in variable runno. The temporary script begins with the standard comment indicating that it’s a C-shell script. The script’s first action is to write a three-line data file. Note the different delimiter, EOD. This data file is created only when the temporary script is run. As we want the time and date information at run time to be written to the data file, the command substitution backquotes are both escaped with a \. In contrast, the final line of the data file is evaluated before the temporary script is written. Finally, the temporary script removes itself and the data file. After making a temporary script, don’t forget to give it execute permission.

12.6 Reading lines from a text file

There is no simple file reading facility in the C-shell. So we call upon awk again.

12.7 Reading tabular data

When reading data into your script from a text file you will often require to extract columns of data, determine the number of lines extracted, and sometimes the number columns and selecting columns by heading name. The shell does not offer file reading commands, so we fall back heavily on our friend awk.

12.7.1 Finding the number of fields

The simple recipe is

There may be a header line or some schema before the actual data, you can obtain the field count from the last line.

If you know the comment line begins with a hash (or can be recognised by some other regular expression) you can do something like this.

12.7.2 Extracting columns

For a simple case without any comments.

If there are comment lines to ignore, say beginning with # or *, the following excludes those from the array of values.

12.7.3 Selecting a range

awk lets you select the lines to extract through boolean expressions, that includes involving the column data themselves, or line numbers through NR.

You can find out how many values were extracted through $#var, such as in the final line above.

You have the standard relational and boolean operators available, as well as $\sim$ and !$\sim$ for match and does not match respectively. These last two con involve regular expressions giving powerful selection tools.

12.7.4 Choosing columns by name

Suppose your text file has a heading line listing the names of the columns.

The above can be improved upon using the toupper function to avoid case sensitivity.

12.8 Reading from dynamic text files

You can also read from a text file created dynamically from within your script.

The technical term for such files are here documents.

12.9 Discarding text output

The output from some routines is often unwanted in scripts. In these cases redirect the standard output to a null file.

12.10 Obtaining dataset attributes

When writing a data-processing pipeline connecting several applications you will often need to know some attribute of the data file, such as its number of dimensions, its shape, whether or not it may contain bad pixels, a variance array or a specified extension. The way to access these data is with ndftrace from Kappa and parget commands. ndftrace inquires the data, and parget communicates the information to a shell variable.

12.10.1 Obtaining dataset shape

Suppose that you want to process all the two-dimensional NDFs in a directory. You would write something like this in your script.

Note although called ndftrace, this function can determine the properties of foreign data formats through the automatic conversion system (SUN/55, SSN/20). Of course, other formats do not have all the facilities of an NDF.

If you want the dimensions of a FITS file supplied as the first argument you need this ingredient.

12.10.2 Available attributes

Below is a complete list of the results parameters from ndftrace. If the parameter is an array, it will have one element per dimension of the data array (given by parameter NDIM); except for EXTNAM and EXTTYPE where there is one element per extension (given by parameter NEXTN). Several of the axis parameters are only set if the ndftrace input keyword fullaxis is set (not the default). To obtain, say, the data type of the axis centres of the current dataset, the code would look like this.

12.10.3 Does the dataset have variance/quality/axis/history information?

Suppose you have an application which demands that variance information be present, say for optimal extraction of spectra, you could test for the existence of a variance array in your FITS file called dataset.fit like this.

12.10.4 Testing for bad pixels

Imagine you have an application which could not process bad pixels. You could test whether a dataset might contain bad pixels, and run some pre-processing task to remove them first. This attribute could be inquired via ndftrace. If you need to know whether or not any were actually present, you should run setbad from Kappa first.

12.10.5 Testing for a spectral dataset

One recipe for testing for a spectrum is to look at the axis labels. (whereas a modern approach might use WCS information). Here is a longer example showing how this might be implemented. Suppose the name of the dataset being probed is stored in variable ndf.

12.11 FITS Headers

Associated with FITS files and many NDFs is header information stored in 80-character ‘cards’. It is possible to use these ancillary data in your script. Each non-comment header has a keyword, by which you can reference it; a value; and usually a comment. Kappa from V0.10 has a few commands for processing FITS header information described in the following sections.

12.11.1 Testing for the existence of a FITS header value

Suppose that you wanted to determine whether an NDF called image123 contains an AIRMASS keyword in its FITS headers (stored in the FITS extension).

12.11.2 Reading a FITS header value

Once we know the named header exists, we can then assign its value to a shell variable.

12.11.3 Writing or modifying a FITS header value

We can also write new headers at specified locations (the default being just before the END card), or revise the value and/or comment of existing headers. As we know the header AIRMASS exists in image123, the following revises the value and comment of the AIRMASS header. It also writes a new header called FILTER immediately preceding the AIRMASS card assigning it value B and comment Waveband.

As we want the metacharacters " to be treated literally, each is preceded by a backslash.

12.12 Accessing other objects

You can manipulate data objects in HDS files, such as components of an NDF’s extension. There are several Starlink applications for this purpose including the FIGARO commands copobj, creobj, delobj, renobj, setobj; and the Kappa commands setext, and erase.

For example, if you wanted to obtain the value of the EPOCH object from an extension called IRAS_ASTROMETRY in an NDF called lmc, you could do it like this.

12.13 Defining NDF sections with variables

If you want to define a subset or superset of a dataset, most Starlink applications recognise NDF sections (see SUN/95’s chapter called “NDF Sections”) appended after the name.

A naïve approach might expect the following to work

Instead here are some recipes that work.