G CHANGES AND NEW FEATURES

G.1 Changes in V4.1

The following describes the most significant changes which occurred in HDS between versions V4.0 and V4.1 (not the current version).

(1)

The C source code of HDS now complies with the ANSI C standard, and UNIX implementations of HDS use a new “makefile”. Both of these changes are designed to make it relatively straightforward to implement HDS on new platforms.

(2)

HDS now contains routines which automatically determine the primitive data representation used by common types of computer hardware (e.g. the floating point number format and byte storage order). This also makes it easier to implement HDS on new platforms, since changes to the source code are not normally required.

(3)

A DECstation implementation of HDS has been added. An implementation on Silicon Graphics hardware is also in use, although this is not currently supported by Starlink.

(4)

The SUN Sparcstation implementation of HDS now uses file mapping as its default mode of file access. This can give appreciable performance gains, largely, it appears, as a result of reduced memory usage which may allow better buffering of file contents.

(5)

The directory in which HDS resides now contains a file called hds_datestamp which holds information about how the HDS system was built. This includes details of the primitive data representation used by the host machine which may sometimes be of use to programmers.

(6)

A bug has been fixed which could cause the length of new _CHAR objects to exceed the length actually requested. Typically, an extra digit was being appended to the length as a result of an un-terminated internal C string.

(7)

A bug has been fixed which caused HDS not to recognise primitive data values written in big-endian IEEE double precision format (i.e. typically written on a SUN) which have the “bad” data value due to a previous format conversion error. The result of this bug was that a data conversion error could result if the data were read on a machine (typically a VAX) with a more restricted double precision number range. The correct behaviour is for the bad SUN value to be converted silently to the corresponding bad VAX value. This now occurs.

(8)

A bug has been fixed which could corrupt a mapped slice of a primitive object because only a single byte of data was being transferred for the final element. This could happen both when reading and writing the data, although not in all circumstances.

(9)

A problem has been fixed which typically resulted in “bus errors” on SUN systems when attempting to access primitive data as double precision values where format conversion was also required. This resulted from an inability to handle double precision values passed from Fortran to C, where the value may be badly aligned in memory (i.e. on a 4-byte rather than an 8-byte boundary). This problem was not always repeatable, in the sense that it depended on where the Fortran compiler placed the relevant variable in memory.

(10)

A workaround has been installed for a problem sometimes encountered when writing to a file served by a VMS machine using the Network File System (NFS) from a SUN. Depending on the file size and the amount of data written, a failure to extend the file can sometimes occur. This seems to happen only when the new file size requested by the SUN lies between the current end of file and the physical file size allocated by VMS as a result of clustering of disk blocks. This problem seems to have arisen since the previous version of HDS due to changes to the VMS UCX software. The workaround involves repeating the file extend call if it fails on the first attempt.

(11)

A bug has been fixed in the queue handling facility used internally by HDS. This could cause regions of memory to be over-written. It is not clear what adverse consequences this may have had.

(12)

An error in the documentation concerning the order in which character arguments should appear when passing mapped character values using the %VAL facility has been corrected (see §8).

(13)

The description of the DAT_BASIC routine has been improved to make it clear that this routine accesses bytes of primitive data as written, and does not perform conversion to/from the data representation of the host machine.

(14)

Several minor typos in error messages have been corrected.

(15)

Use of the routine HDS_START is no longer necessary in order to start up HDS before using it. The system is now self-starting, typically when the first routine which accesses a locator is called. HDS_START has been documented as obsolete, but its use remains optional.

(16)

Use of the routine HDS_STOP at the end of a program is now optional, since its action will be performed automatically by an exit handler. Examples showing its use have been modified appropriately. Note that HDS_STOP is not obsolete, as it remains the only method of closing down HDS in the middle of a program. In practice, the need to do this is likely to be limited.

(17)

Since HDS_START and HDS_STOP are now both optional, the routine HDS_RUN no longer serves a useful purpose and has been documented as obsolete.

(18)

The method by which HDS determines how long a container file should be held open has been rationalised. This no longer depends on the existence of a “top-level” locator. Instead, the concept of “primary” and “secondary” locators has been introduced (see 12.4) and a container file remains open for as long as at least one primary locator is associated with it. Any locator may be designated as primary by means of the new routine DAT_PRMRY, thus removing the special status of top-level locators in this context. Routines which previously incremented the container file “reference count” now issue primary locators (all other locators are secondary by default), so that existing behaviour is retained.

This change has been introduced to remove the requirement that all software using HDS maintain its own table of top-level locators in order to prevent container files being closed. It also opens the way for future improvements to the programming interface of HDS, which should allow objects to be identified by their pathname as well as by locator.

(19)

All routines which annul locators will now also close the associated container file if there are no longer any primary locators associated with it. Any outstanding secondary locators associated with a closed container file are now correctly annulled (previously they were simply left “dangling”). The main implication of this is that DAT_ANNUL will now close a container file which is no longer in use, removing the need to do this explicitly.

(20)

As a result of the above changes, the HDS_CLOSE routine is now redundant, and has been documented as obsolete. Its continued use is not recommended. Its behaviour has always been flawed, since it decrements the reference count for a container file regardless of whether the top-level locator it annuls originally caused this count to be incremented when it was created. Since several methods exist for generating top-level locators without incrementing this count, it is possible for a container file to be prematurely closed if HDS_CLOSE is used.

HDS_CLOSE may still be useful as an emergency measure to close a file in the presence of a programming error which has left it open.

(21)

A new routine DAT_REFCT has been introduced to return the current reference count for a container file. This makes it possible to predict when a file will actually be closed.

(22)

The temporary limit on the number of simultaneously open container files imposed in V4.0 of HDS has been removed. HDS now imposes no restrictions on the number of open files beyond those set by the host operating system.

(23)

On UNIX systems, HDS now reports an error if the name of a container file which is already in use is given as the name of a new container file to be created. This avoids the fairly common problem on UNIX, where the user of an application supplies the same name for both the input file and the output file, and ends up with no file at all because the output over-writes the input before it has been read.

(24)

When creating new container files on UNIX systems, HDS will now check before over-writing an existing file to ensure that it is a regular file and not a directory or FIFO, etc. An error results if it is not a regular file.

(25)

HDS now consistently ignores all leading and trailing white space on file names passed to it.

(26)

On UNIX systems, HDS now uses the full (absolute) pathnames of all files when referring to them in messages and when returning file names via routine arguments.

(27)

On UNIX systems, HDS will now pass file names which contain “special” characters (anything except alphanumerics, ‘.’, ‘/’, ‘-’ and ‘_’) to a shell for interpretation. This means that normal UNIX shell syntax may be used to construct file names, thus allowing environment variable expansion, etc. In fact, any shell command can, in principle, be used to construct a file name. Pattern-matching characters are also accepted – if more than one file matches, then the first match is used.

(28)

Two new routines HDS_WILD and HDS_EWILD have been introduced to permit “wild-card” searching for HDS container files specified using pattern-matching characters.

(29)

A new tuning parameter ‘SHELL’ has been introduced (see 15.2) to allow a choice of which UNIX shell is used to interpret file names (this includes wild-carding of file names using HDS_WILD and HDS_EWILD). By default, the “sh” shell is used but, if available, the “csh” and “tcsh” shells may also be selected. It is also possible to turn shell expansion of file names off.

(30)

It is now possible to modify the default tuning profile of HDS by means of environment variables, which are read and interpreted by HDS at startup (see §15). For instance, the definition:

  setenv HDS_MAP 0

could be used to disable file mapping in favour of I/O in any application which uses HDS.

(31)

A call to HDS_GTUNE will now only return the value 1 for the ‘MAP’ tuning parameter (corresponding to file mapping being used as the file access mode) if this mode has previously been requested and if it is actually implemented on the host machine. This makes it possible for the caller to determine whether file mapping is implemented or not.

(32)

Three new special values ($-$1, $-$2 and $-$3) may now be given for the ‘MAP’ tuning parameter in order to select the file access mode best suited for a given type of access. These options select (in order) the file access method which gives fastest sequential access, fastest random (sparse) access, or minimum usage of memory (see 15.2). The value returned by HDS_GTUNE may be used to determine which file access mode was actually selected, as this will depend on the host operating system.

(33)

Some of the values returned by the routine DAT_CLEN describing the number of digits required to format floating point numbers have been changed to reflect the recommendations of the IEEE floating point standard.

(34)

The HDS_SHOW routine now has a new ‘DATA’ option which displays details of the primitive data representation in use by the host machine.

(35)

Internal changes have been made which allow HDS “container records” (which contain information about the components which reside within an HDS structure) to be reduced in size when components are erased. Some hysteresis is allowed in this process. This typically gives a slight performance improvement and saves a little file space.

(36)

The NBLOCKS tuning parameter is now actually used at several places within HDS. Previously its value had no effect.

(37)

A new script called hds_dev has been provided on UNIX systems to create and remove symbolic links to the HDS public include files within the current directory. This allows the same INCLUDE statements to be used in Fortran code on both UNIX and VMS systems.

(38)

The new error code DAT__WLDIN and the symbolic constant DAT__NOWLD have been introduced.

(39)

This document (SUN/92) has been updated and produced using a larger font.

G.2 Changes in V4.2

The following describes the most significant changes which occurred in HDS between versions V4.1 and V4.2 (not the current version).

(1)

Support for the VAX/VMS operating system has been discontinued at this version of HDS and all references to VMS usage have been dropped from this document.

A VAX/VMS implementation of HDS V4.1 remains in operation and continues to be supported by Starlink on a “care and maintenance” basis, but as a separate entity. This means that the following and future additions to HDS will not be applied to the VAX/VMS version. Users of HDS on VAX/VMS should refer to the documentation that accompanies it for information, and not to this document.

(2)

The OSF1 (Digital UNIX) implementation of HDS now supports file mapping. This should improve the performance of those applications that can take advantage of it.

(3)

HDS now allows container files that are marked for deletion to be re-opened for use, so long as they have not yet been closed (and therefore deleted). This facility was found to be necessary to support foreign data file access in the NDF library (SSN/20).

(4)

Due to the introduction of various new POSIX facilities on OSF1 (Digital UNIX) and associated compiler flags, some untested code was being compiled and used on this platform. The result was that file specifications containing “wild-carding” characters could not be used and the HDS_WILD routine did not function correctly in recent un-tested distributions of HDS for OSF1. These problems have now been fixed.

(5)

A bug has been fixed which meant that the HDS_WILD routine would not function correctly if it was the first HDS routine to be called in an HDS application. A call to HDS_START would prevent this problem occurring, but is now no longer necessary.

(6)

A bug has been fixed which could result in objects accessed for modification using file mapping to be written back to their container files using a write call (instead of unmapping the file) if the ‘MAP’ tuning parameter was changed between calls to DAT_MAP (or equivalent) and DAT_ANNUL (or equivalent). This problem did not occur if DAT_UNMAP was called to unmap the data explicitly. Surprisingly, this erroneous behaviour almost always gave the correct result, but could very occasionally result in a core dump.

(7)

A bug has been fixed that could corrupt data in the immediate neighbourhood of a primitive object slice whose value was updated. This could only happen in rare circumstances and was normally limited to slices no more than 32 bytes in length.

(8)

Certain error messages that are often the result of an attempt to access a corrupt HDS container file have been modified to explicitly give the affected file name.

(9)

The HDS makefile has been extensively revised. Amongst other things, this is to permit automatic distribution via the Starlink Software Store on the World Wide Web.

(10)

This document (SUN/92) has been revised and is now available in both Latex and hypertext (HTML) forms.

(11)

The HDS implementations on SunOS and Ultrix systems have been superceded by Solaris and OSF1 implementations respectively (the latter now being called Digital UNIX). The code to support these (and other) earlier implementations of HDS has been retained, and may be of use to those wishing to port HDS back to these platforms. However, it is no longer supported on these systems by Starlink and has not recently been built or tested on them. An Open VMS implementation of HDS running on Dec Alpha workstations is also known to exist, but is not supported by Starlink.

Further information about old or unsupported implementations of HDS may be obtained by contacting Starlink Software Support (stardev@jiscmail.ac.uk)

G.3 Changes in V4.3

The following describes the most significant changes which occurred in HDS between versions V4.2 and V4.3 (the current version).

(1)

A change has been made to the way that data pointers are allocated by HDS, so that the new CNF_PVAL function (described in SUN/209) may be used when passing pointers using the “%VAL” facility (see §7). Use of this function remains optional on currently-supported platforms, but it may be required if Fortran software which holds pointers in INTEGER variables is required to run in circumstances where memory pointers require a longer variable for storage.

Use of CNF_PVAL is recommended in all new Fortran software which uses HDS pointers.

(2)

The maximum number of objects which may be simultaneously mapped via CMP_ routines has been increased to 1024.

(3)

This document (SUN/92) has been updated to reflect the above changes.

G.4 Changes in V6.0

(1)

The entire HDS library has been re-implemented to support HDS files that use the HDF5 data format. The library can read and write both old (V4) and new (V5) formats. By default, new files are created using the old V4 format, but this can be changed using the new “VERSION” tuning parameter. Note, the default output format will change to V5 in a future release.

(2)

New functions (datock, datUnlock and datLocked) have been added to the C interface to allow an HDS object to be locked for exclusive use by a single thread. An error is reported if a thread attempts to make a change to an object that it has not previously locked for its own exclusive use. By defaults objects are locked by the thread that creates them.

G.5 Changes in V6.1

(1)

The default output data format has been changed from V4 (the classic starlink format) to V5 (the HDF5-based format). To re-establish the old behaviour, set environment variable HDS_VERSION to “4”.

G.6 Changes in V7.0

(1)

The definition of the hdsdim type used in the C interface has been changed from a 4-byte integer type (int) to an 8 byte integer type (int64_t). This allows HDS-based applications to access arrays with more than 2147483647 pixels (the largest value that can be stored in a 4-byte integer). Application code that uses the hdsdim type consistently should continue to compile and run as before. However, application code that assumes that the hdsdim type is 4-bytes long (e.g. by inappropriate casting between int and hdsdim) may not compile or run correctly. Such code should be changed to avoid the assumption of a 4-byte hdsdim type.

The classic HDS Fortran interface has not been changed - INTEGER$\ast$4 variables are still used for arguments that pass array dimensions, pixel indices or pixel counts. However, there is now a variant routine for each routine that has one or more such arguments, which uses INTEGER$\ast$8 variables for array dimensions, pixel indices or pixel counts. Each variant routine is identical to the basic routine except that the digiti “8” is appended to the end of its name.