14

SUN/92.14

Starlink Project
Starlink User Note 92.14

R.F. Warren-Smith
M.D. Lawden
B.K. McIlwrath
T. Jenness
P.W. Draper
D.S. Berry

18th September 2019

Copyright © 2019 East Asian Observatory

HDS
Hierarchical Data System

Version 7.0

Programmer’s Manual


Abstract

HDS is a library of functions used for accessing file-based hierarchical data and supports the storage of a wide variety of information. It is particularly suited to the storage of large multi-dimensional arrays (with their ancillary data) where efficient access is needed.

Historically the HDS library used a unique format for disk-file storage developed by the UK Starlink project in the 1980’s. However it now supports both this historical Starlink format and also the popular HDF5 format.

HDS organises data into hierarchies, broadly similar to the directory structure of a hierarchical filing system, but contained within a single HDS container file. The structures stored in these files are self-describing and flexible; HDS supports modification and extension of structures previously created, as well as deletion, copying, renaming, etc.

All information stored in HDS files is portable between the machines on which HDS is implemented. Thus, there are no format conversion problems when moving between machines.

The routines described in this document may be used to perform operations on any HDS data. In addition, HDS forms a toolkit for the construction of higher level (more specialised) data structures and the software which accesses them. HDS routines are therefore invoked indirectly by many other items of Starlink software.

Contents

1 INTRODUCTION
2 DISK FORMATS AND HDF5
 2.1 Choosing the disk format version for new HDS files
3 OBJECTS
 3.1 Name
 3.2 Type
 3.3 Shape
 3.4 State
 3.5 Group
 3.6 Value
 3.7 Illustration
4 ROUTINES AND CONSTANTS
 4.1 Routine Names
 4.2 Symbolic Names and Include Files
5 CREATING OBJECTS
6 WRITING AND READING OBJECTS
7 ACCESSING OBJECTS BY MAPPING
8 MAPPING CHARACTER DATA
9 COPYING AND DELETING OBJECTS
10 SUBSETS OF OBJECTS
11 TEMPORARY OBJECTS
12 USING LOCATORS
 12.1 Locator Validity
 12.2 Annulling Locators
 12.3 Cloning Locators
 12.4 Primary and Secondary Locators
 12.5 Container File Reference Counts
 12.6 Promoting and Demoting Locators
13 ENQUIRIES
14 PACKAGED ROUTINES
15 TUNING
 15.1 Setting HDS Tuning Parameters
 15.2 Tuning Parameters Available
 15.3 Tuning in Practice
16 ROUTINE VARIANTS FOR LARGE ARRAYS
17 COMPILING AND LINKING
18 ACKNOWLEDGEMENTS
A ALPHABETICAL LIST OF HDS ROUTINES
B ROUTINE DESCRIPTIONS
CMP_GET0x – Read scalar component
CMP_GET1x – Read vector component
CMP_GETNx – Read array component
CMP_GETVx – Read vectorised component
CMP_LEN – Enquire component precision
CMP_MAPN – Map array component
CMP_MAPV – Map vectorised component
CMP_MOD – Ensure component exists
CMP_MODC – Ensure that string component exists
CMP_PRIM – Enquire component primitive
CMP_PUT0x – Write scalar component
CMP_PUT1x – Write vector component
CMP_PUTNx – Write array component
CMP_PUTVx – Write vectorised component
CMP_SHAPE – Enquire component shape
CMP_SIZE – Enquire component size
CMP_STRUC – Enquire component structure
CMP_TYPE – Enquire component type
CMP_UNMAP – Unmap component
DAT_ALTER – Alter object size
DAT_ANNUL – Annul locator
DAT_BASIC – Map primitive as basic units
DAT_CCOPY – Copy one structure level
DAT_CCTYP – Create type string
DAT_CELL – Locate cell
DAT_CLEN – Obtain character string length
DAT_CLONE – Clone locator
DAT_COERC – Coerce object shape
DAT_COPY – Copy object
DAT_DREP – Obtain primitive data representation information
DAT_DSAME – Check if two primitive objects have the same data representation.
DAT_ERASE – Erase component
DAT_ERMSG – Translate a status value into an error message
DAT_FIND – Find named component
DAT_GET – Read primitive
DAT_GETx – Read primitive
DAT_GET0x – Read scalar primitive
DAT_GET1x – Read vector primitive
DAT_GETNx – Read array primitive
DAT_GETVx – Read vectorised primitive
DAT_INDEX – Index into component list
DAT_LEN – Enquire primitive length
DAT_MAP – Map primitive
DAT_MAPx – Map primitive
DAT_MAPN – Map array primitive
DAT_MAPV – Map vectorised primitive
DAT_MOULD – Alter object shape
DAT_MOVE – Move object
DAT_MSG – Assign the name of an HDS object to a message token
DAT_NAME – Enquire object name
DAT_NCOMP – Enquire number of components
DAT_NEW – Create component
DAT_NEW0x – Create scalar component
DAT_NEW0C – Create scalar string component
DAT_NEW1x – Create vector component
DAT_NEW1C – Create vector string component
DAT_NEWC – Create string component
DAT_PAREN – Locate parent structure
DAT_PREC – Enquire storage precision
DAT_PRIM – Enquire object primitive
DAT_PRMRY – Set or enquire primary/secondary locator status
DAT_PUT – Write primitive
DAT_PUTx – Write primitive
DAT_PUT0x – Write scalar primitive
DAT_PUT1x – Write vector primitive
DAT_PUTNx – Write array primitive
DAT_PUTVx – Write vectorised primitive
DAT_REF – Obtain a reference for an HDS object
DAT_REFCT – Enquire the reference count for a container file
DAT_RENAM – Rename object
DAT_RESET – Reset object state
DAT_RETYP – Change object type
DAT_SHAPE – Enquire object shape
DAT_SIZE – Enquire object size
DAT_SLICE – Locate slice
DAT_STATE – Enquire object state
DAT_STRUC – Enquire object structure
DAT_TEMP – Create temporary object
DAT_THERE – Enquire component existence
DAT_TYPE – Enquire object type
DAT_UNMAP – Unmap object
DAT_VALID – Enquire locator valid
DAT_VEC – Vectorise object
DAT_WHERE – Find position of primitive in HDS file
HDS_COPY – Copy an object to a new container file
HDS_ERASE – Erase container file
HDS_EWILD – End a wild-card search for HDS container files
HDS_FLUSH – Flush locator group
HDS_FREE – Free container file
HDS_GROUP – Enquire locator group
HDS_GTUNE – Obtain tuning parameter value
HDS_LINK – Link locator group
HDS_LOCK – Lock container file
HDS_NEW – Create container file
HDS_OPEN – Open container file
HDS_SHOW – Show HDS statistics
HDS_STATE – Enquire the current state of HDS
HDS_STOP – Close down HDS
HDS_TRACE – Trace object path
HDS_TUNE – Set HDS tuning parameter
HDS_WILD – Perform a wild-card search for HDS container files

C OBSOLETE ROUTINES
DAT_CONV – Enquire data conversion
DAT_ERDSC – Report object error
DAT_ERDSN – Report component error
DAT_ERTXT – Report error
DAT_RCERA – Recursive erase
DAT_RCOPY – Recursive copy
DAT_TUNE – Set HDS parameter
HDS_CLOSE – Close container file
HDS_RUN – Run an HDS application subroutine
HDS_START – Start up HDS

D CALLING HDS FROM C
datLock – Lock an object for exclusive use by the current thread
datLocked – See if an object is locked
datUnlock – Unlock an object so that it can be locked by a different thread

E ERROR CODES
 E.1 Error Code Descriptions
 E.2 Obsolete Error Codes
F MACHINE-DEPENDENT FEATURES
 F.1 File Naming
 F.2 Wild-Card File Searching
 F.3 File Mapping
 F.4 Scratch Files
 F.5 File Locking
G CHANGES AND NEW FEATURES
 G.1 Changes in V4.1
 G.2 Changes in V4.2
 G.3 Changes in V4.3
 G.4 Changes in V6.0
 G.5 Changes in V6.1
 G.6 Changes in V7.0