The following sections lead you through the CCD reduction process as you would do it without the aid of XREDUCE or REDUCE. The initial part details how the automated processing works and the later parts show you how to do the same things using the core programs.
There are three stages that you must go though to use the automated processing facilities:
Package configuration is performed using the CCDSETUP program. This sets the values of a sequence
of global parameters, that are used by the other CCDPACK programs. Starting with this routine serves
as a useful reminder of what values etc. will be required to perform the reduction. Note, however,
that none of the parameters are compulsory (indeed CCDSETUP itself is not compulsory) and may be
‘!’ (this is the null-value symbol). CCDSETUP asks for the following values
(together with some others which are best accepted until more experience with the package is
The routine also initialises the CCDPACK logging system (see §12).
Which of this information you’ll have to supply depends on whether or not your data has the correct information stored with it. CCDPACK can “import” FITS headers and use these to supply the necessary information.
A typical CCD geometry is shown in Figure 1; note how the bias strips and useful CCD areas are defined. The coordinate values for these regions are always defined in pixel indices (i.e. row or column numbers, usually starting at 1,1 for the lower left-hand corner of the data array).
A typical set up command is:
In this example certain required characters (the
[ ]) are also special to the C shell so must be
CCDSETUP also allows you to define which parts of the CCD are corrupt or unreliable (due to hot spots, bad columns etc.); see §14, if you need to do this.
Before you can ask CCDPACK to schedule a reduction it is necessary to put your data frames through a process known as “importing”. What this means is that all the necessary descriptions about the type of data (target, bias, flat etc.), filter, where the CCD bias strips are etc. are put into the data files (in a part known as the CCDPACK extension; note that for non-NDF formats this means the image headers).
There are two ways to enter this information into your data frame extensions, use either the PRESENT or IMPORT programs. If your data has the correct FITS headers then you use IMPORT to interpret these and if it has none (or the FITS headers do not give a complete description) you use PRESENT. Unless you have an existing FITS “import control table” for your detector (some of these are available with CCDPACK, check XREDUCE or REDUCE for a list of these) using IMPORT isn’t a trivial task and you should probably opt for using PRESENT, even if you have some FITS information available. A description of how to create a FITS import control table is given in the IMPORT description in §0.
Using PRESENT is fairly trivial as long as you’ve run CCDSETUP and answered all the relevant parts. PRESENT just requires lists of the data frames of each type and a filter type (any old value will do if this isn’t relevant). So an invocation of PRESENT might be:
although you’d generally run it and respond to the prompts interactively. CCDPACK programs can
generally accept lists of data frames, so it this case it processes all the frames with names
bias as bias frames etc. Note that the expansion of the wildcard symbol “
happens inside the program (and not by the shell as is usual) so it is protected by single
When running PRESENT it’s a good idea to check the output to make sure that your frames are given the correct frame type. The known frame types are:
TARGET: These are the frames with the real data. The “target” of your observations (or perhaps the target that the telescope points at).
FLAT: The flatfields.
BIAS: The bias frames
DARK: Any dark count frames (not usual).
FLASH: Any pre-flash frames (very unusual).
There are also several extra types that are used to define the calibration masters. These are
MASTER_FLASH. PRESENT can be used to import foreign
masters (such as spectral flatfields).
When running PRESENT you’ll notice some strange names are given under the “
Item” column. These
are the names of the extension items stored in your data frames (at least they are for NDFs anyway,
other formats use other names). Most of their meanings are fairly obvious as they correspond to the
CCDSETUP parameters with similar names.
FTYPE: The frame type (
FILTER: The filter name (any unique string).
TIMES.DARK: The dark count exposure time.
TIMES.FLASH: The pre-flash exposure time.
DIRECTION: The readout “direction”.
BOUNDS.START1: The first row/column of the first bias strip.
BOUNDS.END1: The last row/column of the first bias strip.
BOUNDS.START2: The first row/column of the second bias strip.
BOUNDS.END2: The last row/column of the second bias strip.
EXTENT.MINX: Lower X value of the useful CCD section.
EXTENT.MAXX: Upper X value of the useful CCD section.
EXTENT.MINY: Lower Y value of the useful CCD section.
EXTENT.MAXY: Upper Y value of the useful CCD section.
ADC: The analogue to digital conversion factor.
RNOISE: The readout noise (ADUs).
DEFERRED: The deferred charge value.
SATURATION: The saturated pixel count (ADUs).
After you’ve performed the first two tasks this part is easy. Just run the application SCHEDULE:
This requires a list of the data frames to reduce and will show what processing it reckons is required (including a range of possible debiassing options), write a script to perform this and (optionally) execute it.
It also has some nifty features such as picking up reduces that fail (just give it all the names of the files produced as well as the originals) and deleting intermediary frames to save on disk space.
If you expect to regularly process large amounts of data from a particular detector (so that using automated reduction is especially attractive), it is fairly straight forward to write a script that will do the reductions.
If you have an import control table (check the Set detector... item in the Options menu of the XREDUCE program to see what CCDPACK has already) then you would just need a script that contained something like:
assuming that only the frames to be processed are in the current directory. A copy of this file is
If you do not have an import control table then you will need to adopt some method of differentiating your data into its various frame types. Section 11 has some ideas about this. Assuming you have adopted a simple naming scheme then a reduction script might be like:
CCDPACK_SETUP.DAT is a CCDSETUP restoration file and has been created by running
CCDSETUP and saving its configuration. A copy of this file is available as
CCDPACK reductions are based around a suite of programs (that the automated and GUI facilities rely on) that you can use directly. These programs are:
If you want to process your data completely by hand (or if the limitations of the automated processing are a problem) then the following sections will lead you through the various options. At the end of this section reduction scripts are shown as examples.
The first step in starting a CCDPACK reduction sequence is to set up the device characteristics using the routine:
CCDSETUP is described in §7.1.1.
If you intend to debias your CCD data using bias frames, then the next move is to combine all these into a ‘best bet’ low noise frame; a ‘master bias’. There are probably only two ways in which you’d like to do this:
The second option may seem strange (if you’re not used to it), but it has a good rationale behind it and is the default method. Using this method requires that your data have bias strips. These are used as a monitor of the bias level at readout time and the master bias is offset to them so that any small variations in the zero point are tracked.
You make a master bias by running the program:
If you want to make a master bias using the first method, then use a command like:
for the second method use:
The IN specification
bias/* means get all the frames in the subdirectory
bias/. The RNOISE
parameter specifies the readout-noise (in ADUs) of the CCD you’re using (if you’ve set up a global
value for this using CCDSETUP then this need not be supplied). MAKEBIAS shows an estimate of the
readout-noise which it derives from the data, use this to check your value, or use this value if none
other exists. The nominal readout-noise value can usually be found in the technical descriptions
issued by the observatories. CCDPACK uses the readout noise value to generate error estimates, you
may specify GENVAR=FALSE to disable this option if your destination analysis package does not
make use of data errors, your data format doesn’t support the storage of this information, or if disk
space is tight (the addition of error components to your data will double the disk space
The next stage (or the first stage, if you’re not using a master bias) is to debias all your data frames; flatfields, flash frames, dark frames, and the targets. Debiassing can be done in two basic ways — with and without a bias frame (well actually three methods exist, the third being subtraction of a constant; this is well worth avoiding unless there’s nothing else for it). Let’s tackle these methods one at a time.
If you have made a master bias frame using MAKEBIAS then how you debias depends on how you made it. If your master bias has been combined to give a mean of zero then it will require offsetting to the ‘zero’ level in the bias strips. DEBIAS will require the values of the rows or columns that the strip(s) are found within. You tell DEBIAS whether the values are rows or columns by specifying a “readout direction” ‘X’ or ‘Y’ (see Figure 1). The bias strip ranges must be supplied in pairs; the column or row number on which it starts and the column or row number on which it ends. There are usually two strips on each side of the data, so this requires 4 values. If your data has three “strips” (probably as part of a region running around the data) then choose the two parallel ones (but make sure that the overscan strip, usually the one on the right, isn’t contaminated by residual charge), if it has only one then the choice is obvious.
To subtract a zeroed master bias frame type something like:
or conversely let DEBIAS prompt you. If you meet any questions which you do not understand hit return to accept the default, or respond with a ‘?’ to get some help. If things are really bad then ‘!!’ (abort) will always terminate the application immediately. (Note that ADC, BOUNDS and RNOISE need not be given if you’ve used CCDSETUP.)
If your master bias frame has a non zero mean (if you’ve selected the ZERO=FALSE option in MAKEBIAS) you just want to subtract it so use:
The ADC – analogue-to-digital conversion – factor is required to generate error estimates from the number of ADUs recorded in each pixel, as is the RNOISE value. To avoid this just use GENVAR=FALSE and leave out the ADC and RNOISE parameters.
Debiassing of CCD data can be performed reasonably well by the subtraction of values derived from the bias strips. If the data has two bias strips then an interpolation using a straight line or constant for each line is used. Alternatively a plane can be fitted to the whole of the bias strip data. If only one bias strip is present then extrapolation across the data is used. In this case a single value is derived for each line or one global value for the whole frame.
To subtract the bias using interpolation type something like:
This will interpolate between each pair of lines in the bias strips using a constant. Before the interpolation occurs the bias strips are smoothed using a box filter (this aims to reduce the variation along the strips rather than across them, thus reducing the inter-line noise).
If you do not have any bias frames or bias strips then it is still possible to debias the data provided you know what the debias level actually is (this is also useful when the debiassing has already been done for you and you want to add estimates of the errors, such as in IR data). To debias using a constant try something like:
100.0 from all the data before making error estimates applying data masks
IR data that are bias subtracted using combination bias and sky frames should only be ‘debiassed’ using this method if you actually know what the bias level is (this is usually an observing option) otherwise any errors generated will not be correct (it is not good enough to assume that the bias level is zero and then remove the bias plus sky frames later). If you do this remember to debias all frames so that the zero point remains the same.
DEBIAS is the most complex of the initial reduction programs and performs much more than just debiassing. Its other functions are:
For details about these functions see appendix §0.
If your CCD data has been pre-flashed or has a significant dark level (IR arrays) and you have taken some calibration frames, then this contribution to the data will require removal before flat fielding.
The most simple case (and probably the most usual) is when the calibration data are exposed the same time as the data. Thus the calibration data just require combining, to reduce the noise level, using the routine:
This combines a list of frames together using an associated list of (relative) exposure times. A typical invocation of MAKECAL in which the data has been collected with the same exposure time is:
This uses all the frames in the
darks/ directory to make a master dark frame. The exposure times
given are 1 as the dark frames have exactly the same exposure time as the data. Note that if the input
data do not have exactly the same exposure times an exact number of values must be returned, in the
same order as the input names.
Correcting the data for the dark counts, or pre-flash, is performed by the routine:
which just subtracts a scaled master calibration frame from a list.
Performing pre-flash subtraction is just as straight-forward if the pre-flash calibration frames are exposed for the same time as the pre-flash on the data.
If the calibration data have different exposure times then an explicit list of data frame names is required, together with their associated times (all entered in the correct order). So you might use:
The contents of the text files
frames.lis are the names of all the frames to be
processed. The contents of the files
exposures.lis are the exposure times of
the calibration data entered in the same order as the names. Of course these names and values could
be supplied on the command line, or in response to a prompt — terminating a line with a ‘
reprompting for another line of values.
The next stage in the instrumental correction of your data is to make a ‘flatfield’. A flatfield is probably best made from exposures of the twilight sky or from long-exposures of dark sky (these can be made from “dithered” target frames, if you don’t have many objects). Either way it is quite possible that the data have some corrupted parts (such as stars) which should be removed before combination and normalisation. MAKEFLAT ‘cleans’ the input data by comparing it with a locally smoothed mean, rejecting any deviant values outside of a number of standard deviations, then trying again for a given number of iterations. After this has been done it estimates the mean value in each frame (this is how it copes with different exposures) and using the mean as a weight it then combines the data using a method, such as median stacking (see §B.1.3 ), which rejects even more bad data (in fact any method except the mean will reject some spurious data). To use MAKEFLAT just type something like:
and it’s done. One master flatfield should be made for each filter used.
The final process in correcting your CCD data is to divide by the flatfield. The flatfield corrects for such things as vignetting (the optical response) and the pixel-to-pixel variations in the CCD response (these can be up to 10 percent). FLATCOR divides data by a flatfield.
out=*|debias_darkc|processed| is one we have not seen before, its meaning is; call
all the output frames the same as the inputs except remove the string
‘debias_darkc’ from the names
and replace it with
This example is a full reduction with two filter types, error generation and defect removal. The debiassing is performed using a zeroed master bias that is offset to the bias strips. To execute this in the background consult §11.
The next example is a less comprehensive one with no error generation and just one filter type. The debiassing uses a master that is subtracted without offsetting as the data has no bias strips.
The next example debiasses using bias strips.
The next example debiasses using bias strips and creates a flatfield using known exposure times and avoids the defect cleaning process in MAKEFLAT.
Note that in all these examples it is necessary to protect certain symbols from being interpreted by the
shell. The CCDNOTE entries use the shell to read in lines of data (until the occurrence of
«FOO means – read this file until an occurrence of
Copies of these files can be found in the
$CCDPACK_DIR directory, called
The reduction sequences outlined above are shown in a schematic format in figure 2.
Reducing Infra-Red (IR) array data has many similarities to reducing CCD data. The major differences are usually an apparent lack of bias information (no bias frames or strips) and flatfields, and the need to remove dark current.
Normally if you have no bias frames then your data should already be debiassed (this is typically an observing option), in this case unless you want to remove defective pixels (ARD files can be easily generated from a glitch file as it has a PIXEL keyword – see §14), and/or generate error estimates for your data (the PHOTOM – SUN/45 package can make use of these), you can miss out the debiassing stage. If you want to pass your data through DEBIAS use the option to subtract a constant as in:
Another way that IR data can be debiassed is by subtracting the bias contribution at the same time as the dark current (since a dark current frame must have the same bias contribution). If you have data like this then you need to use the MAKECAL and CALCOR routines to create a master dark (if you have more then one dark count frame) and then subtract both contributions together. One less obvious point about this method is that you should not use dark counts that do not have exactly the same exposure as your data (this is because the bias level doesn’t scale, it’s an absolute value).
If you have bias frames then follow the normal CCD procedures for subtracting without bias strips.
The way that flatfielding is usually done with IR array data is by ‘dithering’ the object frames on the sky (this also makes sure that the defective pixels are different, relative to the objects) and then median stacking them. Of course this will fail if your objects cover a large area of the detector and the typical contribution in the stack of images isn’t sky at every pixel. You may of course have some sky frames that can be used as flatfields.
The following script shows how you might reduce you data if you want to deglitch and generate errors if your data is already debiassed.
A copy of this file is available from
$CCDPACK_DIR in the file
Automated reductions are also possible for IR array data. The following script shows how to reduce data that has already been debiassed and uses the object data frames to produce a flatfield.
A copy of this file is available from
$CCDPACK_DIR in the file