9 BAD PIXELS

 9.1 The Need for Bad Pixels
 9.2 Recognition and Processing of Bad Pixels
 9.3 The Bad-Pixel Flag
 9.4 Obtaining and Using the Bad-Pixel Flag
 9.5 Requesting Explicit Checks for Bad Pixels
 9.6 Setting the Bad-Pixel Flag
 9.7 Interaction with Mapping and Unmapping
 9.8 Interaction with Initialisation Options
 9.9 A Practical Template for Handling the Bad-Pixel Flag

This section discusses the need to handle pixels whose values may be unknown, and describes the NDF facilities which support this concept.

9.1 The Need for Bad Pixels

The NDF_ system provides support for the concept of bad pixels (sometimes called “magic values” or “flagged values”) which may be present in the array components of an NDF. Bad pixels are pixels whose actual value may be unknown, and which have therefore been assigned a special “null” or bad value. They can arise in a wide variety of ways, for instance:

Many other examples exist where pixel values cannot be meaningfully assigned, and the bad pixel concept is designed to handle this general problem.

9.2 Recognition and Processing of Bad Pixels

Having been introduced into an array, bad pixels must be recognisable by subsequent algorithms, which must be capable of taking suitable action to allow for the missing pixel values (normally, this will mean disregarding them in an appropriate manner).

For purposes of recognition, bad pixels are assigned a unique value, appropriate to their numeric type and reserved for this purpose. This value is identified by a symbolic name of the form VAL__BADx, where “x” identifies the numeric type, as follows:

VAL__BADD — Bad double-precision pixel value
VAL__BADR — Bad single-precision pixel value
VAL__BADI — Bad integer pixel value
VAL__BADW — Bad word pixel value
VAL__BADUW — Bad unsigned word pixel value
VAL__BADB — Bad byte pixel value
VAL__BADUB — Bad unsigned byte pixel value

(note the use of a double underscore character in this naming convention). These symbolic constants, along with others relating to the HDS primitive numeric types, are defined in the include file PRM_PAR (see SUN/39). The appropriate symbolic name should be used for the numeric type being processed.

To give an example of how the need for bad pixels inevitably arises in practice, consider the following simple algorithm to divide one single-precision (i.e. ‘_REAL’) array A by another B, to give a result in the array C:

        INTEGER I, EL
        REAL A( EL ), B( EL ), C( EL )
  
        ...
  
        DO 1 I = 1, EL
           C( I ) = A( I ) / B( I )
   1    CONTINUE

This algorithm is obviously flawed, because if any element of B is zero, then it will fail. However, we can test for this case, and assign bad pixels to affected elements of the output array as follows:

  *  Define the VAL__BADx constants.
        INCLUDE ’PRM_PAR’
  
        ...
  
        DO 1 I = 1, EL
  
  *  Test for division by zero and assign a bad value to the output array.
           IF ( B( I ) .EQ. 0.0 ) THEN
              C( I ) = VAL__BADR
  
  *  Otherwise, calculate the result normally.
           ELSE
              C( I ) = A( I ) / B( I )
           END IF
   1    CONTINUE

Subsequent algorithms will then need to recognise these bad pixels and take appropriate action. Normally, if a bad pixel is encountered as input to an algorithm, it should automatically generate a corresponding bad pixel for output, a process known as bad pixel propagation. The above algorithm could therefore be adapted to recognise bad pixels in either of the input arrays (A and B) and to propagate them, as follows:

        INCLUDE ’PRM_PAR’
  
        ...
  
        DO 1 I = 1, EL
  
  *  If either input pixel is bad, then so is the output pixel.
           IF ( A( I ) .EQ. VAL__BADR .OR. B( I ) .EQ. VAL__BADR ) THEN
              C( I ) = VAL__BADR
  
  *  Check for division by zero.
           ELSE IF ( B( I ) .EQ. 0.0 ) THEN
              C( I ) = VAL__BADR
  
  *  Otherwise, calculate the result normally.
           ELSE
              C( I ) = A( I ) / B( I )
           END IF
   1    CONTINUE

Different action may be required in other algorithms, but the process illustrated here is typical.

9.3 The Bad-Pixel Flag

The need to handle bad pixels inevitably adds to the amount of programming required when writing an application and can also adversely affect the execution time (although usually not as badly as might be feared). Nevertheless, most programmers recognise the need to handle bad pixels as an unfortunate fact of life that often has to be catered for.

When possible, however, it can be useful to determine whether or not bad pixels are actually present in an array, so that a more streamlined algorithm can be used if there is no need to check each pixel explicitly for a bad value. Indeed, in some cases, an algorithm may depend upon the absence of bad pixels, so a method of checking for this condition is required. Each array component of an NDF therefore has a logical value associated with it called its bad pixel flag, which is intended to indicate whether or not bad pixels may be present in that component.

Unfortunately, certainty in this matter comes at a price, because there are many operations (including those performed by the NDF_ system) which might introduce bad pixels into an array component, but there is no way of being completely sure without performing additional checks. In many cases this will involve examining every array element, and the time taken to do this may mean that the test is hardly worth performing if the resulting knowledge only results in a small saving of execution time in the main algorithm.

The NDF_ system takes a pragmatic approach to this problem, by allowing a slightly “fuzzy” interpretation of the bad-pixel flag’s value, but providing the option to make it more precise (at additional cost) if required. The normal interpretation of the bad-pixel flag’s value is therefore as follows:

.FALSE. There are definitely no bad pixels present
.TRUE. There may be bad pixels present

(See §9.5 for how this may be changed.) The NDF_ system goes to some lengths to keep track of bad pixels, but if any doubt exists it will cautiously set the bad-pixel flag to .TRUE.. Thus, a .FALSE. bad-pixel flag value expresses a definite fact, while a .TRUE. value only “suggests” the presence of bad pixels.7

9.4 Obtaining and Using the Bad-Pixel Flag

The bad-pixel flag value for an array component of an NDF may be obtained using the routine NDF_BAD. For instance:

        LOGICAL BAD
  
        ...
  
        CALL NDF_BAD( INDF, ’Data’, .FALSE., BAD, STATUS )

will return the logical bad-pixel flag for an NDF’s data component via the BAD argument. This value might then be used to control the choice of algorithm for processing the data values. The selected algorithm must be prepared to handle bad pixels unless a .FALSE. value of BAD has been returned, in which case further checks for bad pixels can be omitted.

For example, a very simple algorithm to add 1 to each element of an integer array IA might be written to adapt to the absence or possible presence of bad pixels as follows:

        INCLUDE ’PRM_PAR’
        INTEGER EL, IA( EL ), I
        LOGICAL BAD
  
        ...
  
  *  There are definitely no bad pixels present.
        IF ( .NOT. BAD ) THEN
           DO 1 I = 1, EL
              IA( I ) = IA( I ) + 1
   1       CONTINUE
  
  *  There may be bad pixels, so check for them.
        ELSE
           DO 2 I = 1, EL
              IF ( IA( I ) .NE. VAL__BADI ) THEN
                 IA( I ) = IA( I ) + 1
              END IF
   2       CONTINUE
        END IF

Note that this example is provided to illustrate the principle only, and does not imply that all applications should be capable of performing such processing. In fact, it is expected that many applications may choose not to support the processing of bad pixels at all, and this is considered in more detail in §9.5 and §17.2.

Like other routines, NDF_BAD will also accept a list of array components, in which case it will return the logical “OR” of the bad-pixel flag values for each component. For instance:

        CALL NDF_BAD( INDF, ’Data,Variance’, .FALSE., BAD, STATUS )

would return a logical value indicating whether there may be bad pixels in either the data or variance component of an NDF.

Finally, note that the quality component of an NDF also has a bad-pixel flag, but that its value is currently always .FALSE.. This behaviour may change in future.

9.5 Requesting Explicit Checks for Bad Pixels

In the above examples, the third (CHECK) argument to NDF_BAD was set to .FALSE., indicating that no additional checks for bad pixels were to be performed. In this case, the normal interpretation of the returned bad-pixel flag value (as described in §9.3) applies. However, if this CHECK argument is set to .TRUE., then NDF_BAD will perform any additional checks necessary to determine whether bad pixels are actually present, for instance:

        CALL NDF_BAD( INDF, ’Data’, .TRUE., BAD, STATUS )

This may involve explicitly examining every pixel, which could take a significant time to perform, although this will not always be necessary. If explicit checking is requested in this manner, the meaning of a .FALSE. bad-pixel flag value is unchanged, but the interpretation of a .TRUE. value changes to become:

.TRUE. There is definitely at least one bad pixel present

Requesting an explicit check in this way can therefore convert an otherwise .TRUE. value for the bad-pixel flag into a .FALSE. value, but the opposite effect cannot occur.

The main value if this feature arises whenever there is a substantial cost involved in dealing with bad pixels within a processing algorithm. For instance, a considerably less efficient algorithm may be required to take account of bad pixels, or a suitable algorithm may not even exist (such as in a Fourier transform application). In this case, the extra cost of explicitly checking whether bad pixels are actually present will probably be worthwhile. In the following example, an application which cannot handle bad pixels checks whether there are any present, and aborts with an appropriate error message if there are:

        INCLUDE ’SAE_PAR’
        CALL NDF_BEGIN
  
        ...
  
        CALL NDF_BAD( INDF, ’Data,Variance’, .TRUE., BAD, STATUS )
        IF ( BAD .AND. ( STATUS .EQ. SAI__OK ) ) THEN
           STATUS = SAI__ERROR
           CALL NDF_MSG( ’DATASTRUCT’, INDF )
           CALL ERR_REP( ’FFT_NOBAD’,
       :   ’The NDF structure ^DATASTRUCT contains bad pixels which this ’ //
       :   ’application cannot handle.’, STATUS )
           GO TO 99
        END IF
  
        ...
  
   99   CONTINUE
        CALL NDF_END( STATUS )
        END

Here, the cost of an explicit check is justified, because the cost of not performing the check (i.e. of the application having to abort) is even greater. Note that if the bad-pixel flag is .FALSE., then NDF_BAD will not need to check all the pixel values, so this approach does not hinder the normal case where bad pixels are flagged as absent. A simpler method of performing the same check is given in §17.2.

9.6 Setting the Bad-Pixel Flag

When an NDF is first created, all its components are in an undefined state and the bad-pixel flag values of the data and variance components are set to .TRUE.. They remain set to .TRUE. until evidence is presented that values have been assigned to either of these components which do not contain any bad pixels. This information can normally only come from the application which assigns the values, so the routine NDF_SBAD is provided to explicitly set the bad-pixel flag value when appropriate. For instance:

        CALL NDF_SBAD( .FALSE., INDF, ’Data’, STATUS )

constitutes a declaration by the calling routine that the data component of an NDF does not currently contain any bad pixels. It is entirely the calling routine’s responsibility to ensure that this declaration is accurate, since subsequent applications which access the NDF may otherwise be mislead. In practice, this means that it must either have generated the values itself or have checked them in some way to be sure of the absence of bad pixels.

The converse situation may also arise, where an application is forced to introduce bad pixels into an array component which previously did not contain any. In this case, it must also declare this fact if subsequent applications are not to be mislead. This time, the first (BAD) argument to NDF_BAD would be set to .TRUE.. For instance:

        CALL NDF_SBAD( .TRUE., INDF, ’Data,Variance’, STATUS )

Here, a simultaneous declaration for both the data and variance components has been made by specifying both component names.

If an application is uncertain whether bad pixels may have been introduced or not, then it should always err on the side of caution and set the bad-pixel flag value to .TRUE.. This is not irreversible, since separate general-purpose applications will exist to allow a user to explicitly check for bad pixels in an NDF and to adjust the bad-pixel flag if required.

Finally, note that the bad-pixel flag for the data and variance components always reverts to .TRUE. if the component is reset to an undefined state, e.g. by a call to NDF_RESET. If a component is in this state, then NDF_SBAD will return without action and the bad-pixel flag will remain set to .TRUE..

9.7 Interaction with Mapping and Unmapping

An important process which can introduce bad pixels into an array component is that of accessing the component’s values. This can occur for a number of reasons, but most obviously because type conversion may take place implicitly when an array component is mapped and unmapped (see §8.2). This conversion has the potential to fail if the value being converted cannot be represented using the new type. No error will result in such cases, but the affected pixels will be assigned the appropriate bad-pixel value.

In consequence, the effective bad-pixel flag value for an array component may change when that array is accessed by mapping it, and may change again when it is later unmapped. If type conversion errors occur when mapping or unmapping an array component which is being modified or written, then the bad-pixel flag will end up set to .TRUE. due to the resulting bad pixels being written back to the data structure.

What this means in practice is that, on input, the bad-pixel flag value should normally be checked after any mapping operation has been performed. Similarly, if the bad-pixel flag is to be changed using NDF_SBAD following the assignment of new values to an output array, then this should be done before the component is unmapped. See §9.9 for an example.

9.8 Interaction with Initialisation Options

Normally, there is no way for the NDF_ system to know what values have been assigned to an array component, so responsibility for setting the bad-pixel flag rests entirely with the application. One exception to this occurs, however, if an initialisation option of ‘/ZERO’ is used and results in an array full of zeros being mapped (see §8.6). In this case, the bad-pixel flag will be set to .FALSE. reflecting the resulting absence of bad pixels.

9.9 A Practical Template for Handling the Bad-Pixel Flag

The detailed discussion of the bad-pixel flag above will probably have obscured what needs to be done in practice when writing a simple application, so this section is intended to redress the balance. The overall message is that the NDF_ system will look after all the details. The only golden rule to be followed is:

If you alter the values in an array component, then use NDF_SBAD to update the bad-pixel flag before unmapping it.

The following program fragment shows what action is typically required:

        INTEGER PNTR1( 1 ), PNTR( 1 ), EL, NBAD
        LOGICAL BAD
  
        ...
  
  *  Map the input array for reading and the output array for writing.
        CALL NDF_MAP( INDF1, ’Data’, ’_REAL’, ’READ’, PNTR1, EL, STATUS )
        CALL NDF_MAP( INDF2, ’Data’, ’_REAL’, ’WRITE’, PNTR2, EL, STATUS )
  
  *  See if the mapped input array may contain bad pixels.
        CALL NDF_BAD( INDF1, ’Data’, .FALSE., BAD, STATUS )
  
        ...
  
        <process the data, counting new bad pixels in NBAD>
  
        ...
  
  *  See if there may be bad pixels in the output array and declare their
  *  presence/absence.
        BAD = BAD .OR. ( NBAD .NE. 0 )
        CALL NDF_SBAD( BAD, INDF2, ’Data’, STATUS )
  
  *  Unmap the arrays.
        CALL NDF_UNMAP( INDF1, ’Data’, STATUS )
        CALL NDF_UNMAP( INDF2, ’Data’, STATUS )

In this example, a single input data array is being processed to produce a single output array and the application proceeds as follows:

(1)
After mapping the arrays as required, it enquires whether the mapped input values may contain bad pixels.
(2)
The main processing stage then takes place, during which additional bad pixels may be introduced. Some indication is needed of whether this has occurred; in this case a count of the number of new bad pixels is assumed to be returned in NBAD.
(3)
The bad-pixel flag for the output array is updated, taking account of bad pixels which were present initially, and those introduced by the processing performed above.
(4)
The arrays are then unmapped.

In cases where the processing algorithm provides no clear indication of whether bad pixels may have been introduced, the application would need to err on the side of caution and set the output bad-pixel flag to .TRUE..

7Note that a bad-pixel flag value of .FALSE. does not ensure that the numerical value normally used to represent bad pixels will be absent, but it does indicate that such numbers are to be interpreted literally (i.e. as “good” values) and not as bad pixels.