6 Reducing CCD Data

The previous section listed some of the instrumental effects which must be corrected during the reduction of CCD data. The reduction procedure is non-trivial and it must be carried out carefully if it is not to go awry. The various stages typically involved in CCD data reduction are:

(1)

read your data from disk or tape,

(2)

convert your data to a format compatible with your software,

(3)

inspect the original images and discard those that are faulty,

(4)

flag all the known faulty pixels as ‘bad’ or replace them with invented, reasonable values,

(5)

create master bias and dark images for subsequent use in removing the dark and bias signal from raw images of target astronomical objects,

(6)

for each filter, create a master flat field frame defining the pixel-to-pixel sensitivity variations and then flat field each of the images,

(7)

for each filter, align and add the individual images of each target astronomical object to produce a master image of the object.

You would normally carry out the stages in the order listed, as you progress from copies of the raw images to reduced images. However, in some cases some of the stages can be omitted. The recipes given in Part II together constitute an example of working through these stages.

6.1 Software available

The principal Starlink software for reducing CCD images is CCDPACK (see SUN/139[10]). It provides extensive specialised facilities for reducing CCD data. A considerable advantage of CCDPACK is that it is optionally able to estimate and propagate an error estimate for each individual pixel in the CCD frames through the data reduction process. CCDPACK should be used in conjunction with the KAPPA package (see SUN/95[6]), which provides general image display, examination and processing facilities. KAPPA and CCDPACK are completely and seamlessly inter-operable and, indeed, intended to be used together. It is possible to make a reasonable attempt at reducing CCD data using KAPPA alone, though it is less convenient and gives less good results than CCDPACK.

The image processing and spectroscopy package Figaro (see SUN/86[27]) includes some facilities for reducing CCD data, though these are less comprehensive than CCDPACK. Again, Figaro is inter-operable with KAPPA and CCDPACK. In addition to CCDPACK, KAPPA and Figaro there are various other Starlink packages which are relevant to some aspects of CCD data reduction and CCD photometry. The various packages available, their uses and their inter-relations, are conveniently summarised in the ‘Road-Map for CCD Photometry’[8] which has appeared in the Starlink Bulletin.

The IRAF software environment includes the powerful and extensive package CCDRED for reducing CCD data. The use of IRAF on Starlink systems is described in SG/12[22]. CCDRED is fully documented in:

• User’s Guide to the CCDRED Package by F. Valdes[28],
• The IRAF CCD Reduction Package – CCDRED by F. Valdes[29].

Additional useful material may be found in:

• A User’s Guide to CCD Reductions with IRAF by P. Massey[18],
• Rectifying and Registering Images Using IRAF by L.A. Wells[35],
• Cleaning Images of Bad Pixels and Cosmic Rays Using IRAF by L.A.Wells and D.J. Bell[36].

The early stages of reducing spectra recorded with CCD detectors are similar to those for direct images, though the later stages differ. There is a thorough and accessible introduction to reducing spectroscopic data in the cookbook SC/7: Simple Spectroscopy Reductions[3].

6.2 Data formats

Virtually any format might be used to initially write observations to magnetic disk following their acquisition at the telescope. The choice will simply be whatever is practical and convenient for the observatory concerned. Similarly, most software packages for reducing astronomical data have a preferred or ‘native’ format on which they operate. For most Starlink software it is the NDF ($n$-dimensional Data Format; see SUN/33[31]) and for IRAF it is the OIF (Old Image Format). However, most well-established packages are able to import data in various different formats and, in some cases, may be able to process data which are not in their native format, albeit with some loss of efficiency.

This proliferation of different and incompatible data formats is no longer a substantial problem. The FITS format is ubiquitous in astronomy for transferring data between institutions and between software packages. Howsoever the data were originally written when they were acquired at the telescope they will almost invariably be exported from the observatory in the FITS format. That is, the magnetic tape cartridge with which you return from your observing run will almost always contain FITS files. Similarly, observations extracted from data archives are likely to be in FITS format. All the major packages for reducing astronomical data can import files in the FITS format.

6.2.1 FITS

The original FITS (Flexible Image Transport System) format was proposed by Wells et al.[33] in 1981. However, it has been developed and enhanced over the years. The FITS standard is now maintained and documented by the FITS Support Office of the Astrophysics Data Facility at the NASA Goddard Space Flight Center (see URL:
http://fits.gsfc.nasa.gov/fits_home.html). Though FITS is basically an astronomical format it is sometimes mentioned in books about standard image formats. See, for example, Graphics File Formats by Kay and Levine[15]. The development of the FITS standard since its inception has recently been reviewed by Wells[34].

FITS was originally a standard for files on magnetic tape. However, nowadays it is just as often used as a format for files on magnetic disk. Its primary rôle is the interchange of data between different institutions and software packages, though some packages can process data in the FITS format directly.

Even a brief description of the FITS format is not appropriate here (if you are interested you can retrieve a document prescribing the standard for the FITS format from the FITS Support Office). However, a few comments might be useful. A FITS file is a sequence of records, each of which must be exactly 2880 bytes long. Two types of information are included in a FITS file: the basic data (comprising the image or spectra read from the CCD or whatever) and header information describing and annotating it. Typical header information for an observation might be the instrument and telescope used, the date and time of observation, details of the instrumental set up etc. In the jargon of computer science such header information is often called metadata, though this term is rarely used in astronomy.

A given record may contain header information or data but not both. A record of header information is divided into thirty-six eighty-byte ‘logical records’ (older readers will recognise these as card images). The data are often stored as binary numbers, but the header records always comprise ASCII characters. Header records can occur throughout the file, though there is always at least one at its start.

Figure 4 lists the first few header records from a FITS file. The details are not germane here (and, indeed, the example is not typical; it is for a FITS file which contains no array of data). However, it illustrates the important point that there are two types of FITS header records: keywords and comments.

Keywords

A keyword record consists of a named keyword, an equals sign, the value of the keyword and optionally a comment. For example, in the figure the keyword SIMPLE has the value ‘T’ (for true in this instance) and the keyword BITPIX has the value 8. There are some additional rules about the position and length of these items, but they are not important here. Keywords are the principal mechanism used to associate auxiliary information with a dataset. Programs which process FITS files will often search the file for appropriately named keywords to give them the information that they need. The keywords in the figure are mandatory (their meanings are not important here). Others, if present have a specified meaning.

Comment

A comment header record starts with the string ‘COMMENT’ and the rest of the record consists of free text which is intended to be read by a human. Typically it is used to annotate the dataset.

A consequence of the header information always being ASCII characters and some always occurring at the start of the file is that it is possible to examine it with the Unix command more. The resulting display is perfectly readable, though perhaps not very æsthetic. This technique works best with a window which is eighty characters wide. A disadvantage of using more is that it is usually not practical to examine any header information which does not occur at the start of the file. Most data reduction packages have more sophisticated means for examining FITS header information: see, for example, the recipe in Section 9 and the script in Section 21. These mechanisms usually allow you to examine header information which is embedded in the file as well as that at the front.

6.3 Illustration of data reduction

Despite marked improvements in CCD manufacturing techniques in recent years, the bias and pixel-to-pixel sensitivity variations found in raw CCD images are not small effects. This section briefly illustrates the effects and shows the improvement which careful calibration can yield. The images used here were generated by Matthew Trewhella using the WIRO CCD operating in the K band in the near infrared. The images are all of 128$\times$128 pixels and were taken as part of a project to mosaic the M51 galaxy and its companion NGC 5194. Several hundred images were taken and they were all reduced, aligned and stacked using CCDPACK.

Figure 5: A raw WIRO K band image of part of M51

Figure 5 shows a raw, unprocessed target image, as read-out from the CCD. An astronomical object is visible in the middle of the right-hand edge of the image, but any other features are swamped by the instrumental signature.

Figures 6 and 7 are respectively a bias and a flat field frame. They clearly show the origin of the instrumental effects seen in Figure 5. For the sake of clarity the contrast of the flat field frame has been enhanced using a histogram-equalisation technique in order to make subtle changes in intensity easier to see.

Figure 6: The bias frame associated with the WIRO CCD

Figure 7: The WIRO CCD flat field (contrast enhanced)

Figure 8 shows the final mosaic of M51. The boxed area indicates the part of the mosaic to which the raw image shown in Figure 5 contributed. Over 2Gb of raw images, darks, flats and biases were used to construct this mosaic.

Figure 8: The final WIRO K band mosaic of M51 and NGC 5194