15 NDF SECTIONS

15.1 The Need for NDF Sections

An important facility provided by the NDF_ system is the ability to select a region from an NDF which differs in shape from the original NDF, and to process it as if it were a complete NDF itself. Such regions are termed NDF sections. As a simple example, consider a routine which plots a contour map of an NDF’s data component. By accessing an appropriate NDF section, the same routine could also be used to contour only a subset of the data without having to make any change to the routine itself.

This ability to concentrate on a subset of an NDF can clearly improve efficiency in many circumstances, but NDF sections are also capable of referring to super-sets of NDFs; i.e. they may extend beyond the bounds of the NDF from which they are derived. This gives them a number of further uses through their ability to match the shapes of NDFs of otherwise unequal extent by effectively trimming or padding them with bad pixels. Their use for this type of operation is discussed more fully in §17.3.

15.2 Creating NDF Sections

An NDF section may be created using the routine NDF_SECT. For instance:

will create an NDF section starting from an NDF with identifier INDF1, and will return a new identifier INDF2 which refers to the section. The set of pixels in the original NDF to which the new section should refer is determined by the arguments NDIM, LBND and UBND.

These arguments not only specify the set of original pixels to which the new section should refer, but also directly determine the shape (i.e. the pixel-index bounds and the dimensionality) of the new section. For instance, a call to NDF_BOUND using the new identifier INDF2 would return the same values as had originally been specified in the call to NDF_SECT.¹⁴

Note, the tuning parameter SECMAX places a limit on the largest possible NDF section that can be created (see §23.3).

15.3 The Distinction between Base NDFs and Sections

An NDF identifier which refers to an entire NDF dataset (not just a section of it) is said to refer to a base NDF. A base NDF represents a data structure whose shape and other attributes are uniquely defined. Any changes made to the attributes of a base NDF are reflected by actual changes to the contents of the data file in which the NDF is stored. Any such changes are also immediately apparent through any other base NDF identifiers which refer to the same structure (multiple identifiers for the same structure can be generated by means of the routine NDF_CLONE for instance).

In contrast, an NDF section simply represents a “window” into a base NDF. Any number of identifiers may refer to sections derived from the same base NDF, but each represents a separate window. As a consequence, the shapes of all NDF sections are completely independent and may be altered, if required, without affecting each other. One consequence of this is that the NDF_CLONE routine, when applied to an NDF section, has the effect of producing a duplicate NDF section rather than simply a duplicate NDF identifier.

It is possible to discover if an identifier refers to a base NDF or to an NDF section using the routine NDF_ISBAS. For instance:

will return a logical value of .TRUE. via the ISBAS argument if the identifier INDF refers to a base NDF. It is also possible to obtain an identifier for the base NDF to which an NDF section refers by using the routine NDF_BASE, thus:

However, this routine should be used sparingly because it tends to subvert the program modularity which the use of NDF sections allows, and permits access to regions of the base NDF which a well-structured application perhaps ought not to be touching.

15.4 Referring to Subsets and Super-sets

In describing how to create an NDF section in §15.2, there was an implicit assumption that the pixel-index bounds of the section lay within the bounds of the NDF from which it was derived, and that the section’s dimensionality also matched that of the original NDF. In fact, neither of these restrictions need apply.

First consider the case where the dimensionality of the initial NDF and the derived section are the same, but the new pixel-index bounds extend outside those of the original NDF. This would be the case if an NDF section with shape:

were to be created from an original NDF with shape:

In this case the section refers to a subset of the pixels along the first dimension but a super-set of the pixels along the second dimension. As a consequence, there are some pixels in the new section which do not exist in the original NDF.

When values are read from an array component of such a section, the NDF_ system will respond by padding the original NDF with bad pixels; i.e. by assigning the appropriate bad-pixel value to all the “new” pixels which did not exist in the original NDF. The value of the bad-pixel flag returned for the new section by the routine NDF_BAD would reflect the presence of these bad pixels.

15.5 The Transfer Window

The set of pixels which lie within the bounds of a base NDF and also within the bounds of a section derived from it is termed the transfer window for that section. This transfer window comes into play when new values are assigned to the section’s pixels. Although new values may be assigned to any of these pixels, only those lying within the transfer window will have their values transferred back to the base NDF to cause a permanent change to the data structure. The transfer takes place when access to a mapped array component of a section is relinquished (e.g. by calling NDF_UNMAP), at which point all pixel values outside the transfer window are discarded.

This process is similar in concept to the “viewport” used in many graphics systems where, typically, a program can plot lines at any point, but only those lying within the viewport will actually appear on the screen (i.e. plotting done outside the viewport is “clipped”). Using this analogy, an NDF section would correspond with the plotting space available to a program, the transfer window would correspond with the viewport, and the base NDF would correspond with the plotting screen.

Note that it is quite permissible for an NDF section to be derived from another NDF section without any restriction on their relative pixel-index bounds. In this case, the transfer windows of both sections will be combined, so that no new section can access a larger region of the associated base NDF than the section from which it is derived. In extreme cases this could result in the transfer window for a section becoming non-existent, in which case the section will no longer have any contact with its base NDF, although it will still be a valid section.

15.6 Changing Dimensionality

Now consider the case where the number of dimensions specified for a new NDF section differs from the dimensionality of the NDF from which it is derived. The NDF_ system will handle this in the same way that all dimensionality mis-matches are handled; i.e. by padding the pixel-index bounds with 1’s as necessary.

For example, suppose a 1-dimensional section with shape:

were to be derived from a 2-dimensional NDF with shape:

In this case, the 1-dimensional shape would first be padded with 1’s to become:

which identifies the pixels to which the new section should refer. The additional 1’s will then be discarded before the section is created so that a 1-dimensional section results. A similar process would take place if the relative dimensionalities were reversed, but it would then be the original NDF’s pixel-index bounds which were padded with 1’s in order to identify the pixels to which the section should refer.

There are no restrictions on the creation of sections of any dimensionality up to the maximum of 7 supported by the NDF_ routines. Changes of dimensionality may also be freely combined with the selection of super-sets (see §15.4).

15.7 Restrictions on the Use of Sections

In general, an identifier for an NDF section can be passed without error to any routine which will accept an equivalent identifier referring to a base NDF. In certain cases, however, the behaviour of the routine may differ slightly when an NDF section is supplied in order to adhere to two guiding principles:

(1)

It should be straightforward to write applications which process NDFs without having to know whether the NDFs concerned are base NDFs or NDF sections. Having written such an application, it should be able to process either without modification.

(2)

Applications which access NDF sections should not generally cause changes to the values of NDF pixels which lie outside the pixel-index bounds (or more accurately the transfer window) of the sections they are processing.

The set of operations affected by these principles is rather small because most NDF components (e.g. label, units, title, history and extensions) are regarded as global and are equally accessible via identifiers referring to NDF sections and base NDFs. It is mainly operations on array components which behave differently when applied to NDF sections, and notably those which affect the attributes of these components. For instance, the numeric type of an NDF array component cannot be changed using NDF_STYPE (§7.4) via a section identifier; instead, this routine will simply return without action. Neither may an NDF be deleted (§20.11) via a section identifier.

These, and other differences, are noted in the appropriate routine descriptions in Appendix D and at other relevant points in this document.

15.8 Restrictions on Mapped Access to Sections

The restrictions described in §8.10 concerning multiple mapped access to NDF array components also apply to NDF sections if there is a possibility of an access conflict occurring. Thus, if two NDF sections refer to the same base NDF, and the regions to which they refer (more precisely their transfer windows) intersect, then only one of these sections may be mapped at any time for write or update access to the same array component.

If necessary, an application can determine if such a conflict may occur by using the routine NDF_SAME. For instance:

will return a .TRUE. value via the SAME argument if the two NDF identifiers supplied refer to the same base NDF, and will also return a .TRUE. result via the ISECT argument if their transfer windows intersect.

15.9 More Advanced Use: Partitioning

In times gone by, when computers had little memory, it was common for image-processing applications to read their data one line at a time, and to operate on that line before passing on to the next. Nowadays, there is little need for this approach and the programming complications it causes, because reasonably-sized images can be accommodated entirely in memory.

Nevertheless, limitations on the available memory can still be important in some situations. For instance, when processing extremely large files, it is still possible to find that the memory available (or the memory quota allocated by your system manager) is insufficient. Even when it appears possible to accommodate a large array in memory, it is wise to remember that with a virtual memory operating system the information may not actually reside in the physical memory of the computer, and this may result in substantial inefficiencies.

This sort of consideration can be ignored for most types of work, but steps must sometimes be taken to reduce the amount of memory required when accessing large NDFs. This will often result in improvements in efficiency even if actual memory limitations are unimportant, because a reduction in memory requirements in general tends to make more memory available for other system activities, which therefore run more efficiently.

To allow memory usage to be limited, the NDF_ system provides facilities for partitioning NDFs so that they may be processed in pieces. Two partitioning methods are available, termed chunking and blocking. Chunking is appropriate when the NDF can be regarded simply as a 1-dimensional sequence of values (i.e. when the actual shape is unimportant) while blocking is used if the shape and positional relationship between the pixels is significant. Both of these techniques work by dividing an NDF into sections. They are described in turn below.

Note that these techniques may also have applications in parallel processing, where it is often necessary to partition a large array and then to pass the resulting pieces to separate processors.

15.10 Chunking

The technique of chunking is best introduced by an example. Consider a large 1-dimensional NDF (a spectrum, perhaps) which is to be processed in pieces in order to limit memory usage, and suppose that the maximum size of one of these pieces is to be 10000 pixels. The spectrum can be divided into pieces for processing by creating a section to refer to each piece. Each of these sections should follow on from the previous one, and each should contain 10000 pixels (except the last one, which may be smaller if the total number of pixels is not an exact multiple of 10000). Each of these sections is termed a chunk.

If we want to process each of these chunks in turn, we need to know how many there are, and to have a method of creating the appropriate sections. The NDF_ system provides these facilities through the routines NDF_NCHNK (which determines the number of chunks available) and NDF_CHUNK (which creates sections referring to successive chunks). The following illustrates how these routines might be used:

Here, MXPIX is set equal to the maximum number of pixels which a chunk is to contain, and NDF_NCHNK is then called to determine how many such chunks are available in the NDF (the result is returned via the NCHUNK argument). The routine NDF_CHUNK is then called repeatedly to create a sequence of NDF sections which refer to each chunk in turn. These chunks are specified by the chunk index ICHUNK, which varies from 1 to NCHUNK. The NDF identifier for each section created by NDF_CHUNK is annulled when it is no longer required.

In the 1-dimensional case (as here), this process of chunking is straightforward, and could have been programmed directly without too much difficulty. However, in the N-dimensional case (with N $>$ 1), the number of pixels in each NDF section (chunk) must be the product of N separate dimension sizes. Thus, not all sizes of chunk are available. In addition, each chunk must fit within the bounds of the NDF from which it is derived. As a result, the size and shape of each chunk returned by NDF_CHUNK may vary (although the sequence of chunks generated is always repeatable if several NDFs with the same shape are processed using the same value of MXPIX).

The purpose of chunking is to divide an NDF into pieces, each of which contains contiguous pixels (i.e. pixels whose storage locations in the NDF follow one after the other) with the chunks themselves also following each other contiguously in pixel-storage order.¹⁵ NDF_CHUNK will accomplish this for any shape of NDF, and for any limit on the maximum number of pixels in a chunk. Thus, by appropriately defining MXPIX, a limit can be set on memory usage regardless of the total size of NDF being processed.

In fact, by setting MXPIX to certain special values it is possible to partition an NDF into chunks of pre-determined shape. For instance, if MXPIX is set equal to the first dimension size, then NDF_CHUNK will step through the NDF one “line” at a time. Similarly, if MXPIX is set to the product of the first two dimension sizes, then NDF_CHUNK will step through each “plane” of a 3-dimensional stack of images. The following example shows how this might be used to apply a smoothing algorithm to each image in a 3-dimensional stack without needing to access the entire NDF at once:

Note the use of an NDF context to simplify “cleaning up” after processing each chunk.

15.11 Blocking

The concept of blocking is similar to chunking, except that it is appropriate when the relative positions of pixels within an NDF are significant. In one dimension, chunking and blocking are equivalent, so a 2-dimensional example is required for illustration.

Suppose that a contour map of a very large 2-dimensional image is to be generated, and that the image must be divided into pieces to limit memory usage. Chunking would not be appropriate here, because (depending on its shape) a chunk might work out to be a single line of the image, and this would result in inefficient contouring. Rather than setting an upper limit on the total size of each piece, what we need to limit here is the size of each dimension. This means that we must “tile” the 2-dimensional image with a series of adjacent rectangular regions, each of which does not exceed a certain size in each dimension, and each of which can be contoured in turn. This form of partitioning is what blocking provides, the “tiles” (in N dimensions) being termed blocks.

Blocking is supported by two routines analogous to those provided for chunking. NDF_NBLOC calculates the number of blocks available in an NDF (for given limits on the dimension sizes), while NDF_BLOCK creates the NDF sections which refer to the individual blocks, each of which is identified by a block index. The following illustrates the principle:

In this 2-dimensional example, each block is constrained so as not to exceed 100 pixels in size in each dimension, these limits being specified in the MXDIM array.

Note that blocking generates sections which lie adjacent to one another, rather than being contiguously stored, as is the case with chunking. Blocks are numbered so that their lower/upper bounds increase in the conventional sense with increasing block index (i.e. with the first dimension bound increasing most rapidly and the last bound increasing least rapidly). As with chunking, the size of each block generated by NDF_BLOCK may vary in order to lie within the original NDF (using the “tiling” analogy, there may be some tiles at the edges which must be “cut” to fit), but the sequence of blocks generated is always repeatable.

By supplying special values to NDF_BLOCK for the MXDIM value of each dimension, it is also possible to step through an NDF in blocks of a pre-determined shape (as with chunking). For instance, if the integer array DIM holds the original NDF dimensions, then the assignment:

could be used to step through an image in “lines”, while the assignment:

would step through a 3-dimensional image stack in “planes”. The assignment:

could also be used to step through an image in “columns”, but note that this non-sequential mode of access may not be efficient.