When accessing files containing foreign format data, the NDF library will, on occasion, have to delete them (for instance, the routine NDF_DELET might have been called by an application). Normally, this causes no problem, as a named file can easily be deleted when necessary. With some formats, however, this is not so simple. For example, data written in IRAF format will normally reside in two associated files – although the NDF library can delete the one it knows about, the other one would remain in existence.
To overcome this and other similar problems, it is possible to define an explicit deletion command for any foreign format which needs special treatment. If one is defined, it will over-ride any attempt by the NDF library to delete files which it knows are written in that format.
Taking the IRAF format as an example, the command would be defined via the environment variable NDF_DEL_IRAF in the same way as when defining format conversion commands. For example:
would unsure that both files associated with the dataset (with extensions ‘.imh’ and ‘.pix’) are deleted when necessary.
The deletion command is invoked in the normal way, by passing it to the C run time library “system” function, having first performed message token substitution on it (see §2.3). In this case, the NDF library defines the following tokens for use in the command:
| Token | Value |
| dir | Directory in which the foreign file resides |
| name | Foreign file name (without directory or extension) |
| type | Foreign file extension (with leading ‘.’) |
| vers | Foreign file version number (blank if not supported) |
| fxs | Foreign extension specifier (see §2.4 ) |
| fxscl | Clean version of fxs (all non-alphanumeric characters replaced by underscores) |
| fmt | Foreign format name (upper case) |
Normally, when a foreign format dataset is converted to or from the native NDF format, the native copy of the data will be held in temporary file space (it will usually be written to a data structure held in the standard HDS scratch file), and this copy will be deleted when no longer required. Normally, this occurs when the dataset is released by the application.
Sometimes, however, it may be more convenient to retain the converted copy. For example, if you have foreign format data but plan to run several NDF-based applications on it, then retaining the native NDF copy the first time it is converted will save you having to re-convert the data on subsequent occasions.
To do this, the NDF library ‘KEEP’ tuning parameter should be set to 1. This can be done by calling the NDF_TUNE routine from within an application, but can also be done by setting the environment variable NDF_KEEP to ‘1’ outside the application (see SUN/33), for example:
If this is done, then subsequent access to a foreign format dataset (say galaxy.fit) will
create a corresponding native format NDF copy of the data in the default directory (in
this case in a file called galaxy.sdf). This will be retained, and will then be accessed the
next time galaxy is specified as a dataset name (remember, if no file extension is given,
there is always an implicit search for a native format dataset before looking for a foreign
one).
The ‘KEEP’ tuning parameter may be changed at any time, so control over individual datasets is possible if it is set from within an application. The value used will be that in effect when the dataset is first accessed.
If the native NDF copy of a foreign dataset is not being kept, then the NDF library will, by default, store it within the HDS scratch file, as described earlier. This is generally most efficient. However, not all conversion utilities will necessarily be able to access such an NDF, particularly if they know nothing of the NDF or HDS data formats themselves. This would be the case with a general purpose data compression utility, for instance.
The NDF library therefore allows you to specify where the native NDF copy of the data should be stored. This is done by defining environment variables containing message tokens that evaluate to the name you want this NDF to have.
Up to two such environment variables may be defined for each foreign format. Their names are generated by prefixing ‘NDF_KEEP_’ and ‘NDF_TEMP_’ to the foreign format name (in upper case), and they correspond to the two cases (a) where the native copy of the NDF is being kept, and (b) where it is not. The two cases are handled separately, but the message tokens available are the same in both instances, as follows:
| Token | Value |
| dir | Directory in which the foreign file resides |
| name | Foreign file name (without directory or extension) |
| type | Foreign file extension (with leading ‘.’) |
| vers | Foreign file version number (blank if not supported) |
| fxs | Foreign extension specifier (see §2.4 ) |
| fxscl | Clean version of fxs (all non-alphanumeric characters replaced by underscores) |
| fmt | Foreign format name (upper case) |
For example, if we were defining a COMPRESSED format and wanted to ensure that the native NDF data was always stored in its own file in the default directory, so that the UNIX “compress” utility could access it, we might use:
Then, whatever the setting of the ‘KEEP’ tuning parameter, the name of an NDF in the default directory would be generated, so the HDS scratch file would never be used to hold the NDF copy of data from a COMPRESSED dataset. Note that specifying an explicit NDF name in this way does not affect whether the native NDF copy is deleted when the dataset is released. This is still determined by the ‘KEEP’ tuning parameter (see §3.2).
If, in the above example, the foreign dataset were called /home/me/data/nebula.sdf.Z (and the
‘KEEP’ tuning parameter was not set), then the NDF name would be tmp_nebula and this name
would be passed (as the value of the ‘^ndf’ message token) to any conversion commands that needed
to be invoked. The NDF itself would reside in an HDS file called tmp_nebula.sdf (the ‘.sdf’ extension
being added automatically by HDS).
Note that the value given for the NDF_KEEP_COMPRESSED environment variable above is, in fact, the same as its default. You should generally be wary of setting this to anything except its default value because the user of an application might well be confused if he sets the NDF_KEEP environment variable to specify that the NDF should be kept, but it ends up with an unexpected name. This facility does, however, give control over which directory is used to store the file.
When deciding where to store the native NDF format copy of a dataset, it is wise to specify a location on a local file system wherever possible. This is, of course, always good practice where large datasets are concerned, as access to remote files is usually far less efficient and can generate considerable network traffic that may interfere with other people’s work.
With NDF format conversion facilities, local file access is even more important. This is because the temporary datasets involved are always read immediately after being written, and very frequently deleted immediately after that. In this situation, an operating system with good file caching will often not actually write the data to a local file at all, but merely copy it to and from memory. This is far faster than waiting for actual data transfer to take place, which is what will normally happen if remote files are involved.
For this reason, you are recommended to configure format conversion software so that temporary datasets are stored in the user’s default directory, in the expectation that this directory, at least, will be chosen sensibly and reside on a local file system. Users may, however, still need to be reminded of the need for this (e.g. in documentation). You may also need to explain how to change this behaviour if, for example, access to larger amounts of space for temporary files becomes necessary.
Note that, by default, temporary NDF datasets are stored in the standard HDS scratch file, which resides in a directory specified by the HDS_SCRATCH environment variable. If this variable is not explicitly set, the user’s default directory is used.
To illustrate the above, the following is a complete example of the C shell commands that might be used to allow access to compressed NDF data files (with file extension ‘.sdf.Z’) on UNIX systems:
Note that the “compress” command has been followed by a null “:” command which does nothing. This is because an error status may be returned if the file being compressed does not get any smaller, so the “:” command ensures that the invoking NDF application always receives a success status. In a production system, more secure error handling than this would probably be required.
For an explanation of the final definition of the NDF_XTN_COMPRESSED environment variable, you should refer ahead to §4.4.
(Warning: Users should be warned that it is unwise to archive compressed data unless thay are sure that the necessary decompression software will be available to them in future, possibly on different hardware and/or operating system platforms.)
When a command associated with access to foreign data completes, the NDF library checks to determine if it was successful.
It first looks at the status value returned by the C “system” call which invoked the command. These status values are operating-system dependent but, on most systems, there is provision for the command to return either a “success” or an “error” status to the command interpreter and for the invoking application to receive this. If the NDF library does not receive a success status back, it deduces that the command has failed and generates an appropriate error report. The NDF_ routine that was invoked then returns to the application with its STATUS argument set and the application would probably then abort and display the error message.
For commands that invoke conversion utilities (associated with either of the environment variables NDF_FROM_… or NDF_TO_…), the NDF library will also check to see that the output dataset from the conversion operation has been created. Where this dataset is a native format NDF, it will be opened to check that it contains a valid NDF data structure. Any problem will again result in an error report from the invoking application.
When writing data access commands or conversion utilities, the recommended course of action if an error occurs is for diagnostic error information to be written to the standard error channel, and for the invoked command to return with an “error” status value appropriate to the command interpreter in use.
It is generally wise to avoid having the conversion utility re-prompt for new input, as this can be confusing for the user who may not be aware that conversion is taking place. This can normally be arranged by appropriately redirecting the standard input and/or output channels to a null device so that the command will abort and control will return to the NDF library if an attempt is made to prompt for (or read) new input.
When writing format conversion utilities, it is often convenient to use the NDF library to access the native NDF format version of the data (see §2.5). However, you should bear in mind that the NDF library’s ability to invoke format conversion commands will still be active unless you take action to switch it off. This means that unwanted recursion is possible if a conversion utility accesses a foreign dataset that in turn causes a further conversion utility to be invoked, and so on…
In practice, this is unlikely to be a problem if care is taken to ensure that NDF datasets are never stored in objects whose names might be mistaken for foreign format data files. If it does prove necessary to suppress unwanted format conversion, however, this can be achieved by setting the NDF_ library’s DOCVT tuning parameter to zero. This will have the effect of disabling recognition of foreign data files by the conversion utility.
One way of doing this is by setting the environment variable NDF_DOCVT to 0 as part of the format conversion command, immediately before the conversion utility itself is invoked. Alternatively, the conversion utility may call the NDF_TUNE routine itself in order to control recognition of foreign data formats. The latter approach allows individual control over each dataset accessed by the utility if necessary.
Normally, all foreign data access commands invoked by the NDF library execute silently, unless an error occurs or a command writes information to standard output (this should normally be avoided). To assist in debugging, however, the NDF library provides a tuning parameter ‘SHCVT’. This can be used to make it display all commands before they are executed but after message token substitution has taken place.
To enable this feature, the ‘SHCVT’ tuning parameter should be set to 1. This can be done from within an application by calling the NDF_TUNE routine (see SUN/33), or from outside the application by setting the NDF_SHCVT environment variable, as follows: