14

SUN/33.12

Starlink Project
Starlink User Note 33.12

R.F. Warren-Smith & D.S. Berry

30st April 2020

NDF
Routines for Accessing the
Extensible N-Dimensional Data Format

Version 2.1

Programmer’s Manual


Abstract

The Extensible N-Dimensional Data Format (NDF) is a format for storing bulk data in the form of N-dimensional arrays of numbers. It is typically used for storing spectra, images and similar datasets with higher dimensionality. The NDF format is based on the Hierarchical Data System HDS (SUN/92) and is extensible; not only does it provide a comprehensive set of standard ancillary items to describe the data, it can also be extended indefinitely to handle additional user-defined information of any type.

This document describes the routines provided for accessing NDF data objects. It also discusses all the important NDF concepts and includes a selection of simple example applications. The majority of the text describes the Fortran 77 interface for the NDF library. The C interface is described briefly in an appendix.

Contents

1 INTRODUCTION
 1.1 What is an NDF?
 1.2 The NDF Philosophy and Extensibility
 1.3 The Relationship with HDS
 1.4 Background Reading
 1.5 Scope of the Current Implementation
2 OVERVIEW
 2.1 Overview of an NDF
 2.2 Overview of the NDF_ Routines
 2.3 Error Handling
 2.4 Overview of a Typical Application
3 OBTAINING AND USING NDF IDENTIFIERS
 3.1 Accessing NDFs for Input
 3.2 NDF Identifiers
 3.3 Annulling Identifiers
 3.4 NDF Contexts: BEGIN and END
 3.5 Cloning Identifiers
4 NDF SHAPE AND SIZE INFORMATION
 4.1 Dimensionality and Bounds
 4.2 Dimension Sizes
 4.3 NDF Size
 4.4 “Safe” Dimension Sizes under Error Conditions
5 GENERAL PROPERTIES OF NDF COMPONENTS
 5.1 Specifying Component Names
 5.2 Component State
 5.3 Factors Determining a Component’s State
 5.4 Restrictions on the Data Component’s State
6 ACCESSING CHARACTER COMPONENTS
 6.1 Reading and Displaying Character Values
 6.2 Writing Character Values
 6.3 Character Component Length
 6.4 Resetting Character Components
7 ARRAY COMPONENT TYPES
 7.1 Numeric Types
 7.2 Complex Values
 7.3 Full Type Specifications
 7.4 Setting Component Types
8 ACCESSING ARRAY COMPONENTS
 8.1 Overview of Mapped Access to Array Components
 8.2 Mapping and Unmapping
 8.3 Implicit Unmapping
 8.4 Writing and Modifying Array Component Values
 8.5 More About Mapping Modes
 8.6 Initialisation Options
 8.7 Implicit Type Conversion
 8.8 Accessing Complex Values
 8.9 Mapping the Variance Component as Standard Deviations
 8.10 Restrictions on Mapped Access
9 BAD PIXELS
 9.1 The Need for Bad Pixels
 9.2 Recognition and Processing of Bad Pixels
 9.3 The Bad-Pixel Flag
 9.4 Obtaining and Using the Bad-Pixel Flag
 9.5 Requesting Explicit Checks for Bad Pixels
 9.6 Setting the Bad-Pixel Flag
 9.7 Interaction with Mapping and Unmapping
 9.8 Interaction with Initialisation Options
 9.9 A Practical Template for Handling the Bad-Pixel Flag
10 THE QUALITY COMPONENT IN MORE DETAIL
 10.1 Purpose of the Quality Component
 10.2 Accessing the Quality Array Directly
 10.3 The Bad-bits Mask
 10.4 Why Ignoring the Quality Component Works
 10.5 Controlling Automatic Quality Masking
11 EXTENSIONS
 11.1 Extensibility
 11.2 Extension Names and Software Packages
 11.3 The Contents of Extensions
 11.4 Accessing Existing Extensions
 11.5 Creating New Extensions
 11.6 Accessing Array Information in Extensions
 11.7 Deleting Extensions
 11.8 Enumerating an NDF’s Extensions
12 ARRAY COMPONENT STORAGE FORM AND COMPRESSION
 12.1 General
 12.2 Obtaining the Storage Form
 12.3 Simple Storage Form
 12.4 Scaled Storage Form
 12.5 Delta compressed Storage Form
 12.6 Primitive Storage Form
13 ACCESSING NDFS FOR OUTPUT
 13.1 Using Existing NDFs
 13.2 Creating New NDFs via Parameters
 13.3 Conditional NDF Creation
 13.4 Creating Primitive NDFs
14 COMPONENT PROPAGATION
 14.1 General
 14.2 Propagation Rules for Standard NDF Components
 14.3 Propagation Rules for Extensions
 14.4 Creating New NDFs by Propagation
 14.5 Default Propagation of Components and Extensions
 14.6 Forcing Component Propagation
 14.7 Inhibiting Component Propagation
 14.8 Controlling Propagation of Extensions
15 NDF SECTIONS
 15.1 The Need for NDF Sections
 15.2 Creating NDF Sections
 15.3 The Distinction between Base NDFs and Sections
 15.4 Referring to Subsets and Super-sets
 15.5 The Transfer Window
 15.6 Changing Dimensionality
 15.7 Restrictions on the Use of Sections
 15.8 Restrictions on Mapped Access to Sections
 15.9 More Advanced Use: Partitioning
 15.10 Chunking
 15.11 Blocking
16 USING SUBSCRIPTS TO ACCESS NDF SECTIONS
 16.1 Specifying Lower and Upper Bounds
 16.2 Specifying Centre and Extent
 16.3 Using WCS Coordinates to Specify Sections
 16.4 Using Normalised Pixel Coordinates to Specify Sections
 16.5 Changing Dimensionality
 16.6 Mixing Bounds Expressions
17 MERGING AND MATCHING NDF ATTRIBUTES
 17.1 The Problem
 17.2 Merging and Matching Bad-Pixel Flags
 17.3 Matching NDF Bounds
 17.4 Merging and Matching Numeric Types
18 THE AXIS COORDINATE SYSTEM
 18.1 Pixel Coordinates
 18.2 Axis Coordinates
 18.3 Axis Arrays
 18.4 Pixel Positions and Dimensions
 18.5 Default Axis Array Values
 18.6 Contiguous and Non-Contiguous Pixels
 18.7 Processing Axis Array Values
 18.8 Axis Normalisation
19 AXIS COMPONENTS
 19.1 Overview of an NDF’s Axis Components
 19.2 Axis Component States
 19.3 Restrictions on Axis Component States
 19.4 Defining a Default Axis Coordinate System
 19.5 Resetting Axis Components
 19.6 Accessing Axis Character Components
 19.7 Mapping Axis Arrays for Reading
 19.8 Unmapping Axis Arrays
 19.9 Writing and Modifying Axis Arrays
 19.10 Accessing Axis Variance Values as Standard Deviations
 19.11 Axis Normalisation Flags
 19.12 The Numeric Type of Axis Arrays
 19.13 The Storage Form of Axis Arrays
 19.14 Accessing Axis Components via NDF Sections
 19.15 Axis Extrapolation
20 CONNECTING WITH THE DATA SYSTEM
 20.1 NDF Names
 20.2 Finding and Importing NDFs
 20.3 A Note on Modes of Access
 20.4 Obtaining an HDS Locator for an NDF
 20.5 NDF Placeholders
 20.6 Creating NDFs via Placeholders
 20.7 Temporary NDFs
 20.8 Copying NDFs
 20.9 Selective Copying of NDF Components
 20.10 General Access to NDFs
 20.11 Deleting NDFs
21 ALTERING BOUNDS AND PIXEL INDICES
 21.1 Setting New Pixel-Index Bounds and Dimensionality
 21.2 Applying Pixel-Index Shifts
22 THE HISTORY COMPONENT
 22.1 Purpose of the History Component
 22.2 The History Component’s State
 22.3 Preparing to Record History Information
 22.4 Resetting the History Component
 22.5 Default History Recording
 22.6 Propagation of History Information
 22.7 Explicitly Controlling History Text
 22.8 The History Update Mode
 22.9 Using Message Tokens in History Text
 22.10 History Text Width
 22.11 Formatting Options for History Text
 22.12 Automatic Error Recording
 22.13 Enquiring about Past History Information
 22.14 Accessing History by Date and Time
 22.15 Accessing Past History Text
 22.16 Modifying Past History
 22.17 Naming an Application
 22.18 Ending an Application
23 MISCELLANEOUS FACILITIES
 23.1 Restricting Access via NDF Identifiers
 23.2 Message System Routines
 23.3 Tuning the NDF_ System
24 COMPILING AND LINKING
 24.1 Standalone Applications
 24.2 ADAM Applications
A EXAMPLE APPLICATIONS
 A.1 SHOW — Display the size of an NDF
 A.2 SETTITLE — Assign a New NDF Title
 A.3 GETMAX — Obtain the Maximum Pixel Value
 A.4 GETSUM — Sum the Pixel Values
 A.5 READIMG — Read an image into an NDF
 A.6 ZAPPIX — “Zap” Prominent Pixels in an Image
 A.7 ADD — Add Two NDF Data Structures
 A.8 NDFTRACE — Trace an NDF Structure
B ALPHABETICAL LIST OF FORTRAN ROUTINES
C CLASSIFIED LIST OF FORTRAN ROUTINES
 C.1 Access to Existing NDFs
 C.2 Enquiring NDF Attributes
 C.3 Enquiring Component Attributes
 C.4 Creating and Deleting NDFs
 C.5 Setting NDF Attributes
 C.6 Setting Component Attributes
 C.7 Access to Component Values
 C.8 Enquiring and Setting Axis Attributes
 C.9 Access to Axis Values
 C.10 Access to World Coordinate System Information
 C.11 Creation and Control of Identifiers
 C.12 Handling NDF (and Array) Sections
 C.13 Matching and Merging Attributes
 C.14 Parameter System Routines
 C.15 Message System Routines
 C.16 Creating Placeholders
 C.17 Copying NDFs
 C.18 Handling Extensions
 C.19 Handling History Information
 C.20 Tuning the NDF_ system
 C.21 Compression
D FORTRAN ROUTINE DESCRIPTIONS
NDF_ACGET – Obtain the value of an NDF axis character component
NDF_ACLEN – Determine the length of an NDF axis character component
NDF_ACMSG – Assign the value of an NDF axis character component to a message token
NDF_ACPUT – Assign a value to an NDF axis character component
NDF_ACRE – Ensure that an axis coordinate system exists for an NDF
NDF_AFORM – Obtain the storage form of an NDF axis array
NDF_AMAP – Obtain mapped access to an NDF axis array
NDF_ANNUL – Annul an NDF identifier
NDF_ANORM – Obtain the logical value of an NDF axis normalisation flag
NDF_AREST – Reset an NDF axis component to an undefined state
NDF_ASNRM – Set a new logical value for an NDF axis normalisation flag
NDF_ASSOC – Associate an existing NDF with an ADAM parameter
NDF_ASTAT – Determine the state of an NDF axis component (defined or undefined)
NDF_ASTYP – Set a new numeric type for an NDF axis array
NDF_ATYPE – Obtain the numeric type of an NDF axis array
NDF_AUNMP – Unmap an NDF axis array
NDF_BAD – Determine if an NDF array component may contain bad pixels
NDF_BASE – Obtain an identifier for a base NDF
NDF_BB – Obtain the bad-bits mask value for the quality component of an NDF
NDF_BEGIN – Begin a new NDF context
NDF_BLOCK – Obtain an NDF section containing a block of adjacent pixels
NDF_BOUND – Enquire the pixel-index bounds of an NDF
NDF_CANCL – Cancel the association of an NDF with an ADAM parameter
NDF_CGET – Obtain the value of an NDF character component
NDF_CHUNK – Obtain an NDF section containing a chunk of contiguous pixels
NDF_CINP – Obtain an NDF character component value via the ADAM parameter system
NDF_CLEN – Determine the length of an NDF character component
NDF_CLONE – Clone an NDF identifier
NDF_CMPLX – Determine whether an NDF array component holds complex values
NDF_CMSG – Assign the value of an NDF character component to a message token
NDF_COPY – Copy an NDF to a new location
NDF_CPUT – Assign a value to an NDF character component
NDF_CREAT – Create a new simple NDF via the ADAM parameter system
NDF_CREP – Create a new primitive NDF via the ADAM parameter system
NDF_CREPL – Create a new NDF placeholder via the ADAM parameter system
NDF_DELET – Delete an NDF
NDF_DIM – Enquire the dimension sizes of an NDF
NDF_END – End the current NDF context
NDF_EXIST – See if an existing NDF is associated with an ADAM parameter
NDF_FIND – Find an NDF and import it into the NDF_ system
NDF_FORM – Obtain the storage form of an NDF array component
NDF_FTYPE – Obtain the full type of an NDF array component
NDF_GTDLT – Get compression details for a DELTA compressed NDF array component
NDF_GTSZx – Get the scale and zero values for an NDF array component
NDF_GTUNE – Obtain the value of an NDF_ system tuning parameter
NDF_GTWCS – Obtain world coordinate system information from an NDF
NDF_HAPPN – Declare a new application name for NDF history recording
NDF_HCOPY – Copy history information from one NDF to another
NDF_HCRE – Ensure that a history component exists for an NDF
NDF_HDEF – Write default history information to an NDF
NDF_HECHO – Write out lines of history text
NDF_HEND – End NDF history recording for the current application
NDF_HFIND – Find an NDF history record by date and time
NDF_HGMOD – Get the history update mode for an NDF
NDF_HINFO – Obtain information about an NDF’s history component
NDF_HNREC – Determine the number of NDF history records present
NDF_HOUT – Display text from an NDF history record
NDF_HPURG – Delete a range of records from an NDF history component
NDF_HPUT – Write history information to an NDF
NDF_HSDAT – Set the history date for an NDF
NDF_HSMOD – Set the history update mode for an NDF
NDF_ISACC – Determine whether a specified type of NDF access is available
NDF_ISBAS – Enquire if an NDF is a base NDF
NDF_ISIN – See if one NDF is contained within another NDF
NDF_ISTMP – Enquire if an NDF is temporary
NDF_LOC – Obtain an HDS locator for an NDF
NDF_MAP – Obtain mapped access to an array component of an NDF
NDF_MAPQL – Map the quality component of an NDF as an array of logical values
NDF_MAPZ – Obtain complex mapped access to an array component of an NDF
NDF_MBAD – Merge the bad-pixel flags of the array components of a pair of NDFs
NDF_MBADN – Merge the bad-pixel flags of the array components of a number of NDFs
NDF_MBND – Match the pixel-index bounds of a pair of NDFs
NDF_MBNDN – Match the pixel-index bounds of a number of NDFs
NDF_MSG – Assign the name of an NDF to a message token
NDF_MTYPE – Match the types of the array components of a pair of NDFs
NDF_MTYPN – Match the types of the array components of a number of NDFs
NDF_NBLOC – Determine the number of blocks of adjacent pixels in an NDF
NDF_NCHNK – Determine the number of chunks of contiguous pixels in an NDF
NDF_NEW – Create a new simple NDF
NDF_NEWP – Create a new primitive NDF
NDF_NOACC – Disable a specified type of access to an NDF
NDF_OPEN – Open an existing or new NDF
NDF_PLACE – Obtain an NDF placeholder
NDF_PROP – Propagate NDF information to create a new NDF via the ADAM parameter system
NDF_PTSZx – Set new scale and zero values for an NDF array component
NDF_PTWCS – Store world coordinate system information in an NDF
NDF_QMASK – Combine an NDF quality value with a bad-bits mask to give a logical result
NDF_QMF – Obtain the logical value of an NDF’s quality masking flag
NDF_RESET – Reset an NDF component to an undefined state
NDF_SAME – Enquire if two NDFs are part of the same base NDF
NDF_SBAD – Set the bad-pixel flag for an NDF array component
NDF_SBB – Set a bad-bits mask value for the quality component of an NDF
NDF_SBND – Set new pixel-index bounds for an NDF
NDF_SCOPY – Selectively copy NDF components to a new location
NDF_SCTYP – Obtain the numeric type of a scaled NDF array component
NDF_SECT – Create an NDF section
NDF_SHIFT – Apply pixel-index shifts to an NDF
NDF_SIZE – Determine the size of an NDF
NDF_SQMF – Set a new logical value for an NDF’s quality masking flag
NDF_SSARY – Create an array section, using an NDF section as a template
NDF_STATE – Determine the state of an NDF component (defined or undefined)
NDF_STYPE – Set a new type for an NDF array component
NDF_TEMP – Obtain a placeholder for a temporary NDF
NDF_TUNE – Set an NDF_ system tuning parameter
NDF_TYPE – Obtain the numeric type of an NDF array component
NDF_UNMAP – Unmap an NDF or a mapped NDF array
NDF_VALID – Determine whether an NDF identifier is valid
NDF_XDEL – Delete a specified NDF extension
NDF_XIARY – Obtain access to an array stored in an NDF extension
NDF_XGT0x – Read a scalar value from a component within a named NDF extension
NDF_XLOC – Obtain access to a named NDF extension via an HDS locator
NDF_XNAME – Obtain the name of the N’th extension in an NDF
NDF_XNEW – Create a new extension in an NDF
NDF_XNUMB – Determine the number of extensions in an NDF
NDF_XPT0x – Write a scalar value to a component within a named NDF extension
NDF_XSTAT – Determine if a named NDF extension exists
NDF_ZDELT – Create a compressed copy of an NDF using DELTA compression
NDF_ZSCAL – Create a compressed copy of an NDF using SCALE compression

E THE NDF_ LIBRARY C INTERFACE
 E.1 Conventions Used in the C Interface
 E.2 Multi-threaded Applications
 E.3 C-only Functions
ndfLock – Lock an NDF for use by the current thread
ndfLocked – Return information about any lock on an NDF
ndfReport – Report any currently active NDF identifiers
ndfUnlock – Unlock an NDF so it can then be locked by another thread

 E.4 Building C Applications
F ALPHABETICAL LIST OF C FUNCTIONS
G OBSOLETE ROUTINES
NDF_IMPRT – Import an NDF into the NDF_ system from HDS
NDF_TRACE – Set the internal NDF_ system error-tracing flag

H CHANGES AND NEW FEATURES
 H.1 Changes Introduced in V1.3
 H.2 Changes Introduced in V1.4
 H.3 Changes Introduced in V1.5
 H.4 Changes Introduced in V1.6
 H.5 Changes Introduced in V1.7
 H.6 Changes Introduced in V1.8
 H.7 Changes Introduced in V1.9
 H.8 Changes Introduced in V1.10
 H.9 Changes Introduced in V1.11
 H.10 Changes Introduced in V1.12
 H.11 Changes Introduced in V1.13
 H.12 Changes Introduced in V2.0
 H.13 Changes Introduced in V2.1
 H.14 Changes Introduced in V2.2