7 Using the data reduction programs

The following sections lead you through the CCD reduction process as you would do it without the aid of XREDUCE or REDUCE. The initial part details how the automated processing works and the later parts show you how to do the same things using the core programs.

7.1 Using the automated processing facilities directly

There are three stages that you must go though to use the automated processing facilities:

(1)

Configure the package.

(2)

Record the reduction information in your data.

(3)

Schedule the reduction.

7.1.1 Package configuration

Package configuration is performed using the CCDSETUP program. This sets the values of a sequence of global parameters, that are used by the other CCDPACK programs. Starting with this routine serves as a useful reminder of what values etc. will be required to perform the reduction. Note, however, that none of the parameters are compulsory (indeed CCDSETUP itself is not compulsory) and may be returned as ‘!’ (this is the null-value symbol). CCDSETUP asks for the following values (together with some others which are best accepted until more experience with the package is gained):

• The ADC factor which converts the ADUs of the input data frames into detected electrons (this also used to generate errors).
• The bias strip positions (used for offsetting the master bias frame or for bias estimation).
• The readout direction (defines bias strip positions and the bias interpolation direction).
• The typical readout-noise (also used to generate errors).
• The useful CCD area.

The routine also initialises the CCDPACK logging system (see §12).

Which of this information you’ll have to supply depends on whether or not your data has the correct information stored with it. CCDPACK can “import” FITS headers and use these to supply the necessary information.

A typical CCD geometry is shown in Figure 1; note how the bias strips and useful CCD areas are defined. The coordinate values for these regions are always defined in pixel indices (i.e. row or column numbers, usually starting at 1,1 for the lower left-hand corner of the data array).

A typical set up command is:

In this example certain required characters (the [ ]) are also special to the C shell so must be protected.

CCDSETUP also allows you to define which parts of the CCD are corrupt or unreliable (due to hot spots, bad columns etc.); see §14, if you need to do this.

Figure 1: Typical CCD geometries. In the figure on the left the readout direction is ‘Y’, the bias strips are located with bounds I,J,K,L and the useful CCD area is M,J$+1$,N,K$-1$ (ish, you should probably use more than $\pm 1$). In the figure on the right the readout direction is ‘X’, the bias strips are located with bounds I,J,K,L and the useful CCD area is N,J$+1$,K$-1$,M$-1$ (N.B. some observatories recommend that you only use the left-hand strip, if you use the right-hand one too, check that it isn’t contaminated by residual charge).

7.1.2 Setting reduction information

Before you can ask CCDPACK to schedule a reduction it is necessary to put your data frames through a process known as “importing”. What this means is that all the necessary descriptions about the type of data (target, bias, flat etc.), filter, where the CCD bias strips are etc. are put into the data files (in a part known as the CCDPACK extension; note that for non-NDF formats this means the image headers).

There are two ways to enter this information into your data frame extensions, use either the PRESENT or IMPORT programs. If your data has the correct FITS headers then you use IMPORT to interpret these and if it has none (or the FITS headers do not give a complete description) you use PRESENT. Unless you have an existing FITS “import control table” for your detector (some of these are available with CCDPACK, check XREDUCE or REDUCE for a list of these) using IMPORT isn’t a trivial task and you should probably opt for using PRESENT, even if you have some FITS information available. A description of how to create a FITS import control table is given in the IMPORT description in §0.

Using PRESENT is fairly trivial as long as you’ve run CCDSETUP and answered all the relevant parts. PRESENT just requires lists of the data frames of each type and a filter type (any old value will do if this isn’t relevant). So an invocation of PRESENT might be:

although you’d generally run it and respond to the prompts interactively. CCDPACK programs can generally accept lists of data frames, so it this case it processes all the frames with names starting with bias as bias frames etc. Note that the expansion of the wildcard symbol “*” happens inside the program (and not by the shell as is usual) so it is protected by single quotes.

When running PRESENT it’s a good idea to check the output to make sure that your frames are given the correct frame type. The known frame types are:

• TARGET: These are the frames with the real data. The “target” of your observations (or perhaps the target that the telescope points at).
• FLAT: The flatfields.
• BIAS: The bias frames
• DARK: Any dark count frames (not usual).
• FLASH: Any pre-flash frames (very unusual).

There are also several extra types that are used to define the calibration masters. These are MASTER_BIAS, MASTER_DARK, MASTER_FLAT and MASTER_FLASH. PRESENT can be used to import foreign masters (such as spectral flatfields).

7.1.3 Reduction extension items

When running PRESENT you’ll notice some strange names are given under the “Item” column. These are the names of the extension items stored in your data frames (at least they are for NDFs anyway, other formats use other names). Most of their meanings are fairly obvious as they correspond to the CCDSETUP parameters with similar names.

• FTYPE: The frame type (TARGET, BIAS, FLAT, DARK or FLASH).
• FILTER: The filter name (any unique string).
• TIMES.DARK: The dark count exposure time.
• TIMES.FLASH: The pre-flash exposure time.
• DIRECTION: The readout “direction”.
• BOUNDS.START1: The first row/column of the first bias strip.
• BOUNDS.END1: The last row/column of the first bias strip.
• BOUNDS.START2: The first row/column of the second bias strip.
• BOUNDS.END2: The last row/column of the second bias strip.
• EXTENT.MINX: Lower X value of the useful CCD section.
• EXTENT.MAXX: Upper X value of the useful CCD section.
• EXTENT.MINY: Lower Y value of the useful CCD section.
• EXTENT.MAXY: Upper Y value of the useful CCD section.
• ADC: The analogue to digital conversion factor.
• RNOISE: The readout noise (ADUs).
• DEFERRED: The deferred charge value.
• SATURATION: The saturated pixel count (ADUs).

7.1.4 Scheduling a reduction

After you’ve performed the first two tasks this part is easy. Just run the application SCHEDULE:

This requires a list of the data frames to reduce and will show what processing it reckons is required (including a range of possible debiassing options), write a script to perform this and (optionally) execute it.

It also has some nifty features such as picking up reduces that fail (just give it all the names of the files produced as well as the originals) and deleting intermediary frames to save on disk space.

7.1.5 Scheduling from a script

If you expect to regularly process large amounts of data from a particular detector (so that using automated reduction is especially attractive), it is fairly straight forward to write a script that will do the reductions.

If you have an import control table (check the Set detector... item in the Options menu of the XREDUCE program to see what CCDPACK has already) then you would just need a script that contained something like:

assuming that only the frames to be processed are in the current directory. A copy of this file is available as $CCDPACK_DIR/ccdpack_ex1.csh

If you do not have an import control table then you will need to adopt some method of differentiating your data into its various frame types. Section 11 has some ideas about this. Assuming you have adopted a simple naming scheme then a reduction script might be like:

The file CCDPACK_SETUP.DAT is a CCDSETUP restoration file and has been created by running CCDSETUP and saving its configuration. A copy of this file is available as
$CCDPACK_DIR/ccdpack_ex2.csh

7.2 Using the CCDPACK programs to reduce data

CCDPACK reductions are based around a suite of programs (that the automated and GUI facilities rely on) that you can use directly. These programs are:

• MAKEBIAS - combines bias frames into a ‘master’ bias calibration frame.
• DEBIAS - debiasses lists of data frames either by master bias subtraction or by estimation from the bias strips, applies bad data masks, extracts a subset of the data area, produces errors and detects saturated pixels values.
• MAKECAL - combines pre-flash or dark count frames into a ‘master’ calibration frame.
• CALCOR - performs dark or flash count corrections on a list of frames.
• MAKEFLAT - combines flat fields into a ‘master’ calibration flatfield.
• FLATCOR - performs the flatfield correction on a list of frames.

If you want to process your data completely by hand (or if the limitations of the automated processing are a problem) then the following sections will lead you through the various options. At the end of this section reduction scripts are shown as examples.

7.2.1 Step 1 - Setting up

The first step in starting a CCDPACK reduction sequence is to set up the device characteristics using the routine:

• CCDSETUP.

CCDSETUP is described in §7.1.1.

7.2.2 Step 2 - Making a bias calibration frame

If you intend to debias your CCD data using bias frames, then the next move is to combine all these into a ‘best bet’ low noise frame; a ‘master bias’. There are probably only two ways in which you’d like to do this:

• Just combine them.
• Zero the mean level first and then combine them (leaving the mean at zero).

The second option may seem strange (if you’re not used to it), but it has a good rationale behind it and is the default method. Using this method requires that your data have bias strips. These are used as a monitor of the bias level at readout time and the master bias is offset to them so that any small variations in the zero point are tracked.

You make a master bias by running the program:

• MAKEBIAS

If you want to make a master bias using the first method, then use a command like:

for the second method use:

The IN specification bias/* means get all the frames in the subdirectory bias/. The RNOISE parameter specifies the readout-noise (in ADUs) of the CCD you’re using (if you’ve set up a global value for this using CCDSETUP then this need not be supplied). MAKEBIAS shows an estimate of the readout-noise which it derives from the data, use this to check your value, or use this value if none other exists. The nominal readout-noise value can usually be found in the technical descriptions issued by the observatories. CCDPACK uses the readout noise value to generate error estimates, you may specify GENVAR=FALSE to disable this option if your destination analysis package does not make use of data errors, your data format doesn’t support the storage of this information, or if disk space is tight (the addition of error components to your data will double the disk space needed).

7.2.3 Step 3 - Debiassing

The next stage (or the first stage, if you’re not using a master bias) is to debias all your data frames; flatfields, flash frames, dark frames, and the targets. Debiassing can be done in two basic ways — with and without a bias frame (well actually three methods exist, the third being subtraction of a constant; this is well worth avoiding unless there’s nothing else for it). Let’s tackle these methods one at a time.

7.2.4 With a master bias

If you have made a master bias frame using MAKEBIAS then how you debias depends on how you made it. If your master bias has been combined to give a mean of zero then it will require offsetting to the ‘zero’ level in the bias strips. DEBIAS will require the values of the rows or columns that the strip(s) are found within. You tell DEBIAS whether the values are rows or columns by specifying a “readout direction” ‘X’ or ‘Y’ (see Figure 1). The bias strip ranges must be supplied in pairs; the column or row number on which it starts and the column or row number on which it ends. There are usually two strips on each side of the data, so this requires 4 values. If your data has three “strips” (probably as part of a region running around the data) then choose the two parallel ones (but make sure that the overscan strip, usually the one on the right, isn’t contaminated by residual charge), if it has only one then the choice is obvious.

To subtract a zeroed master bias frame type something like:

or conversely let DEBIAS prompt you. If you meet any questions which you do not understand hit return to accept the default, or respond with a ‘?’ to get some help. If things are really bad then ‘!!’ (abort) will always terminate the application immediately. (Note that ADC, BOUNDS and RNOISE need not be given if you’ve used CCDSETUP.)

If your master bias frame has a non zero mean (if you’ve selected the ZERO=FALSE option in MAKEBIAS) you just want to subtract it so use:

The ADC – analogue-to-digital conversion – factor is required to generate error estimates from the number of ADUs recorded in each pixel, as is the RNOISE value. To avoid this just use GENVAR=FALSE and leave out the ADC and RNOISE parameters.

7.2.5 Without a bias frame

Debiassing of CCD data can be performed reasonably well by the subtraction of values derived from the bias strips. If the data has two bias strips then an interpolation using a straight line or constant for each line is used. Alternatively a plane can be fitted to the whole of the bias strip data. If only one bias strip is present then extrapolation across the data is used. In this case a single value is derived for each line or one global value for the whole frame.

To subtract the bias using interpolation type something like:

This will interpolate between each pair of lines in the bias strips using a constant. Before the interpolation occurs the bias strips are smoothed using a box filter (this aims to reduce the variation along the strips rather than across them, thus reducing the inter-line noise).

If you do not have any bias frames or bias strips then it is still possible to debias the data provided you know what the debias level actually is (this is also useful when the debiassing has already been done for you and you want to add estimates of the errors, such as in IR data). To debias using a constant try something like:

This subtracts 100.0 from all the data before making error estimates applying data masks etc.

IR data that are bias subtracted using combination bias and sky frames should only be ‘debiassed’ using this method if you actually know what the bias level is (this is usually an observing option) otherwise any errors generated will not be correct (it is not good enough to assume that the bias level is zero and then remove the bias plus sky frames later). If you do this remember to debias all frames so that the zero point remains the same.

7.2.6 Other DEBIAS functions

DEBIAS is the most complex of the initial reduction programs and performs much more than just debiassing. Its other functions are:

• Error estimation/propagation.
• Defect removal (possibly using an ASCII regions definition file).
• Gain correction (converting data values into electrons).
• Deferred charge correction (if you must).
• Saturated pixel detection.
• Extraction of the useful CCD area.

For details about these functions see appendix §0.

7.2.7 Step 4 - Flash or dark calibration

If your CCD data has been pre-flashed or has a significant dark level (IR arrays) and you have taken some calibration frames, then this contribution to the data will require removal before flat fielding.

The most simple case (and probably the most usual) is when the calibration data are exposed the same time as the data. Thus the calibration data just require combining, to reduce the noise level, using the routine:

• MAKECAL

This combines a list of frames together using an associated list of (relative) exposure times. A typical invocation of MAKECAL in which the data has been collected with the same exposure time is:

This uses all the frames in the darks/ directory to make a master dark frame. The exposure times given are 1 as the dark frames have exactly the same exposure time as the data. Note that if the input data do not have exactly the same exposure times an exact number of values must be returned, in the same order as the input names.

Correcting the data for the dark counts, or pre-flash, is performed by the routine:

• CALCOR

which just subtracts a scaled master calibration frame from a list.

Performing pre-flash subtraction is just as straight-forward if the pre-flash calibration frames are exposed for the same time as the pre-flash on the data.

If the calibration data have different exposure times then an explicit list of data frame names is required, together with their associated times (all entered in the correct order). So you might use:

The contents of the text files darkframes.lis and frames.lis are the names of all the frames to be processed. The contents of the files darkexposures.lis and exposures.lis are the exposure times of the calibration data entered in the same order as the names. Of course these names and values could be supplied on the command line, or in response to a prompt — terminating a line with a ‘-’ forces reprompting for another line of values.

7.2.8 Step 5 - Flatfielding

The next stage in the instrumental correction of your data is to make a ‘flatfield’. A flatfield is probably best made from exposures of the twilight sky or from long-exposures of dark sky (these can be made from “dithered” target frames, if you don’t have many objects). Either way it is quite possible that the data have some corrupted parts (such as stars) which should be removed before combination and normalisation. MAKEFLAT ‘cleans’ the input data by comparing it with a locally smoothed mean, rejecting any deviant values outside of a number of standard deviations, then trying again for a given number of iterations. After this has been done it estimates the mean value in each frame (this is how it copes with different exposures) and using the mean as a weight it then combines the data using a method, such as median stacking (see §B.1.3 ), which rejects even more bad data (in fact any method except the mean will reject some spurious data). To use MAKEFLAT just type something like:

and it’s done. One master flatfield should be made for each filter used.

The final process in correcting your CCD data is to divide by the flatfield. The flatfield corrects for such things as vignetting (the optical response) and the pixel-to-pixel variations in the CCD response (these can be up to 10 percent). FLATCOR divides data by a flatfield.

The specification out=*|debias_darkc|processed| is one we have not seen before, its meaning is; call all the output frames the same as the inputs except remove the string ‘debias_darkc’ from the names and replace it with ‘processed’.

7.2.9 Example scripts

This example is a full reduction with two filter types, error generation and defect removal. The debiassing is performed using a zeroed master bias that is offset to the bias strips. To execute this in the background consult §11.

The next example is a less comprehensive one with no error generation and just one filter type. The debiassing uses a master that is subtracted without offsetting as the data has no bias strips.

The next example debiasses using bias strips.

The next example debiasses using bias strips and creates a flatfield using known exposure times and avoids the defect cleaning process in MAKEFLAT.

Note that in all these examples it is necessary to protect certain symbols from being interpreted by the shell. The CCDNOTE entries use the shell to read in lines of data (until the occurrence of FOO, that’s what «FOO means – read this file until an occurrence of FOO).

Copies of these files can be found in the $CCDPACK_DIR directory, called ccdpack_ex3.csh,
ccdpack_ex4.csh, ccdpack_ex5.csh and ccdpack_ex6.csh.

7.2.10 Schematic reduction sequence

The reduction sequences outlined above are shown in a schematic format in figure 2.

Figure 2: A schematic outline of the order in which CCDPACK reduction routines should be used. Dashed boxes indicate that this part is optional or not required. The MAKECAL/CALCOR section may need repeating more than once (e.g. if flash and dark frames are to be processed).

7.3 IR data reduction

Reducing Infra-Red (IR) array data has many similarities to reducing CCD data. The major differences are usually an apparent lack of bias information (no bias frames or strips) and flatfields, and the need to remove dark current.

Normally if you have no bias frames then your data should already be debiassed (this is typically an observing option), in this case unless you want to remove defective pixels (ARD files can be easily generated from a glitch file as it has a PIXEL keyword – see §14), and/or generate error estimates for your data (the PHOTOM – SUN/45 package can make use of these), you can miss out the debiassing stage. If you want to pass your data through DEBIAS use the option to subtract a constant as in:

Another way that IR data can be debiassed is by subtracting the bias contribution at the same time as the dark current (since a dark current frame must have the same bias contribution). If you have data like this then you need to use the MAKECAL and CALCOR routines to create a master dark (if you have more then one dark count frame) and then subtract both contributions together. One less obvious point about this method is that you should not use dark counts that do not have exactly the same exposure as your data (this is because the bias level doesn’t scale, it’s an absolute value).

If you have bias frames then follow the normal CCD procedures for subtracting without bias strips.

The way that flatfielding is usually done with IR array data is by ‘dithering’ the object frames on the sky (this also makes sure that the defective pixels are different, relative to the objects) and then median stacking them. Of course this will fail if your objects cover a large area of the detector and the typical contribution in the stack of images isn’t sky at every pixel. You may of course have some sky frames that can be used as flatfields.

The following script shows how you might reduce you data if you want to deglitch and generate errors if your data is already debiassed.

A copy of this file is available from $CCDPACK_DIR in the file ccdpack_ex7.csh.

Automated reductions are also possible for IR array data. The following script shows how to reduce data that has already been debiassed and uses the object data frames to produce a flatfield.

A copy of this file is available from $CCDPACK_DIR in the file ccdpack_ex8.csh.