In the simplest case, each exposure made on an instrument will read out one CCD using one amplifier/ADC and provide you with one data file containing an array of pixels which represents a single data or calibration frame. However, many instruments are now more complicated than this, and may feature multiple amplifiers reading out different parts of the same chip, or multiple chips on the same mosaic camera for each exposure, or a combination of the two. The resulting data product may consist of one or more data structures (e.g. NDF structures or FITS Header + Data Units (HDUs)) in each of one or more containing files.
Two problems arise when dealing with these sorts of data. Firstly, during the data reduction stage, calibration must be carried out in the same way for data read from the same bits of hardware and differently for data read from different bits; a separate master bias frame must be generated for each CCD in a mosaic camera and each data frame must be calibrated using the corresponding one. Secondly, during the registration stage, alignment information which corresponds to the positioning of the chips on the focal plane of the instrument may be accurately known and not subject to change, and this must be communicated to the software which is attempting to align separate data frames using other methods such as object matching; there’s no point trying to identify matching features on two frames which you know are glued next to each other on the focal plane.
Both of these can be addressed to a large extent by careful use of the facilities of CCDPACK described so far, but to make processing of this kind of data more painless, CCDPACK provides the concept of a Set of images. The idea is that each exposure from a given instrument produces a single Set of data which can contain several frames. All Sets from a particular instrument will consist of a fixed number of members: each member of a given Set will share the same Set Name attribute indicating its origin, and has its own Set Index attribute indicating its rôle, or position, within the Set. Every exposure generated by a four-CCD mosaic camera for instance would produce a single Set with a unique Set Name, and four members with Set Index values of 1, 2, 3 and 4.
Using this functionality is quite simple: the user tells CCDPACK how all the data files are grouped into Sets, and from then on it can make sure that processing is done accordingly. The way this works is that a CCDPACK Set header is added to each file (in the .MORE.CCDPACK.SET extension of the NDF) which contains two items, the Set Name attribute (a string) and the Set Index attribute (an integer, usually 1, 2, 3, …). Set alignment coordinates may also be added as an attached coordinate system, with the Domain name CCD_SET, if a known alignment is to be associated with the Set (because you know how the chips are arranged on the focal plane). The Set itself does not exist as a separate file or other entity, so you never need to specify a Set as such when giving input parameters to CCDPACK programs, but when presented with a list of input files, they will read and make use of Set header information if it exists. In fact in most cases, after adding the Set header in the first place, you can forget about Sets altogether and let CCDPACK worry about them for you.
All the CCDPACK programs which can do Set-sensitive processing consult the USESET global parameter to see whether they should use Set header information if it exists. To make CCDPACK use Set headers, you should use CCDSETUP to set USESET true like this:
All subsequent commands will then use Set headers where they have been added. Alternatively you could specify USESET=TRUE on the command line of each CCDPACK program. Since files without Set headers are always treated the same whether USESET is true or false, it is harmless to have set it true all the time, but in certain cases (especially if a non-native data format such as FITS is being used) it may result in slower operation.
Note that Set Name attributes don’t need to be (and in general will not be) totally unique between different Sets; for instance DEBIAS will produce output files which have the same Set Names as the corresponding input files. However, no ambiguity will arise as long as only one Set of files with the same Set Name is presented to any given CCDPACK program at the same time. Normally, because of the flow of data between applications, that situation will not arise, but if for some reason two Sets with the same Name do need to be presented in the same context, MAKESET can be coerced into assigning a given Name to a Set by use of its NAME parameter.
The following sections give some detailed description and examples of how to manipulate Set header information and how CCDPACK goes about using it.
To add Set header information you need to use the command
This can operate in a number of different ways according to how your data is arranged in the first place. The most useful ones are described in the following subsections. For variations on these, including the ability to construct Set headers from FITS header information, see the full description of MAKESET in appendix 0.
In the most straightforward case, you can use MAKESET to make a single Set out of a group of files listed by name:
This will write a Set header to each of the named files. As the output shows, the same Set Name attribute “im01” (the name of the first file in the list) is given to each, and the Set Index attributes are 1, 2 and 3 respectively.
If you have many files to put into Sets, this may be an unwieldy approach. By using the SETSIZE
parameter, an invocation like this can generate many Sets at the same time. If your directory contains
the files im01… im12 and each group of three in order represents the data from the three CCDs in a
mosaic camera, you can write
Care has to be taken with this usage however; it relies on the files being presented to MAKESET in the correct order, which in turn may depend on how your operating system expands wildcards, which is normally alphabetically. If for instance in the above example the files had been named
(i.e. without the leading zero for numbers less than 10) then they would probably have been presented to MAKESET in the order
which is almost certainly not what you want. If you use MAKESET with wildcards you should be careful to check the logged output to ensure that it has done the right thing.
If your data exists in HDS container files so that one container file contains all the images from a single exposure, then adding Set headers is easier. By running MAKESET with MODE=CONTAINER, it will assume that each HDS container file corresponds to a single Set. In this case you can just type something like
and the correct headers will be added.
Note that if your data initially exists in the form of one Multi-Extension FITS file (MEF) per observation you can convert these quite easily to HDS container files using the FITS2NDF command with CONTAINER=TRUE. So supposing you have three exposures, each stored in a 2-HDU MEF, you can do:
For subsequent work you need only supply the names of the container files (conv-obs*) and the
CCDPACK commands will look inside and find the data frames with their associated Set header
information. See SUN/55 for a full description of the CONVERT facility.
If each observation initially exists as a single image but needs to be treated as multiple images because different regions have been read out using different electronics (so that, for instance, they have to be debiassed using different ADC factors), then a slightly different approach is required. In this case, you should set MODE=SPLIT, and unlike for the other modes MAKESET will create new NDFs each with its own Set header. You will have to supply the parameter OUT to give the name of the file containing the split data: this will be a single HDS container file containing as many NDF structures as the input array is split into. For full generality, the parameter SECTIONS specifies how the input image is to be split up, but for the common case of dividing the array into grid of rectangles which butt up to each other the XSTART and YSTART parameters are easier. The following example shows creation of a Set made by dividing a 2048x4096 array into quarters:
The command creates a single new HDS container file “sg28948_sp”: when you give this as an input to CCDPACK commands they will look inside and process each of the four images in turn.
This method could also be used to create a Set from different slices of a three-dimensional data set.
You may notice that in the last example, a column headed “CCD_SET domain” has appeared in the output, with the entry “PIXEL” for each of the new Set members. This indicates that as well as the Set header, each file has had a new coordinate system attached to its WCS component. This new coordinate system added by MAKESET always has the Domain (name) “CCD_SET”, and in this case is a copy of the PIXEL-Domain coordinate system which existed in the input image. When MAKESET adds a CCD_SET coordinate system like this, it is declaring that all the members of the same Set are known to be correctly aligned in these coordinates. All the CCDPACK programs concerned with registration will recognise a CCD_SET coordinate system and assume that the alignment of all the images in a given Set is correct in these coordinates and is not to be tampered with. Thus, it is only a good idea to add a CCD_SET coordinate system if it really does represent an accurate alignment of images within the Set and not one which, for instance, you might want to improve at a later date by object matching. When MODE=SPLIT, it is assumed that the PIXEL coordinates of the original image do represent this sort of rock-solid alignment, since the Set members have been hewn out of the same data array. With the other modes of MAKESET Set alignment coordinates are not added by default, but if the alignment of the Set is known, it can be added by setting the ADDWCS parameter. In this case a copy of the Current attached coordinate system of each image is used for the CCD_SET alignment coordinates.
So suppose that you have a directory full of HDS container files each holding the data frames from one observation. Each has an attached coordinate system in its WCS component, with the Domain “FOCAL” which gives its exact position on the focal plane of the instrument — this may have been added by the observing system, and it is known to be accurate. First ensure that the FOCAL coordinates are the Current coordinates of all the frames:
and then add the Set headers with the ADDWCS parameter set true:
As you can see, the FOCAL coordinate system is listed as the one copied to make the new one. Note that since the new CCD_SET coordinate system is a copy of the FOCAL one, it will remain attached to the images even if the FOCAL one is erased or modified.
Variations on these methods of adding Set headers, as well as the possibility of controlling Set formation from FITS headers, are discussed in the description of MAKESET in §0.
If you need to add Set alignment coordinates to some files after MAKESET has already been run, you can just attach a new coordinate system with the name CCD_SET, using WCSEDIT or in some other way. Note however that a CCD_SET-domain coordinate system will only be treated as special when the images in question have valid Set headers.
It will not normally be necessary to remove Set headers once they have been added by MAKESET, but if you find you have added them in error, you can erase them using the CCDEDIT task, as follows:
The FIXWCS parameter determines whether any existing Set alignment coordinate systems (ones with Domain CCD_SET) should be erased also. It normally doesn’t matter much whether this is erased or not, since without a valid Set header it is ignored anyway.
If you just want to remove the Set alignment frame, you can remove this attached coordinate frame directly using WCSEDIT:
Both of these commands will erase the information in question if it exists, and note the fact with a non-fatal error if it does not.
If you try to add a new Set header to an image which already has one using MAKESET, or a new CCD_SET coordinate system to an image which already has one using MAKESET or some other method, a warning will be issued, but the old one will be overwritten.
For the most part, once you have added Set headers to your image files using MAKESET you will be able to let them take care of themselves. However, you may wish to check how images are related to each other, or to pull out only members of a given Set or images which correspond to each other from different Sets. For these purposes the
application is provided.
In its simplest mode of operation, it will show you the Set headers of a list of images, grouped by Set Name or Set Index. Here is a simple invocation, showing the result of the example in Section 9.1.2:
As you can see, it gives very similar output to that written by MAKESET when the headers were being added. If you prefer to corresponding images from different Sets grouped together, then change the LISTBY parameter (which defaults to ‘NAME’):
As well as showing the Set information of all the files presented, SHOWSET is also able to select a
sublist of input files according to the values in their Set headers. By supplying the PICKINDEX
and/or PICKNAME parameters, images can be selected according to the values of their Set
Index and/or Set Name attributes respectively. The default value of PICKNAME is ‘ALL’,
meaning that any Set Name will be selected for output. However, if PICKNAME is set
to ‘EQUAL’, then the NAME parameter gives a list of Set Name values and only images
whose Set Name appears in this list will be selected. If PICKNAME is set to ‘LIKE’, then the
NAMELIKE parameter gives a list of other images to act as templates, and only images
whose Set Name is the same as that of one of the template images will be selected. The
PICKINDEX, INDEX and INDEXLIKE parameters work in the same way for Set Index values.
Returning to another example in the previous section, we can pick out only those images
in the same Set as (i.e. having the same Set Name attribute as) im12 but which have Set
Index of 1 or 2 as follows (in this case some of the parameters are input as the response to
prompts):
Here we have also set the LISTBY parameter to ‘NONE’, which does no grouping and lists the images in their input order.
As it points out, SHOWSET writes a list of the images it has output to a file given by the NAMELIST
parameter, by default showset.lis. In the above case that list would read:
This file can be used with the indirection character (‘’) to provide input for any other CCDPACK program, so you can write scripts with commands which will operate only on members of certain Sets or certain Index values. For example, you could subsequently use DRAWNDF to display the outlines of the images that the above command has selected like this:
Section 7.1.1 explained how to set up global values for certain parameters such as the ADC factor and bias strip positions using the CCDSETUP program; by doing this you can set values specific to the data you are using. However, if you have data in Sets life becomes a bit more complicated: each chip will have a different ADC factor, so the value to use will depend on which member of the Set is being processed.
To cope with this, CCDSETUP can be used to configure different values for certain parameters according to Set Index value. The parameters which can vary with Set Index are:
To set up one or more of these values to be specific to certain Set Index values, run CCDSETUP with the BYSET parameter true, and the INDEX parameter equal to the Set Index to which the parameters apply. When USESET is true, CCDPACK programs which use these global parameters will first look for a value specific to the Set Index of the frame being processed, and if there isn’t one they will look for the unspecific global value (if there is no global value at all they will use the current parameter value or ask you to enter it or use a default as usual). So, since many of the parameters will probably be the same for all the members of each Set, the easiest thing is to run CCDSETUP for one of the chips (Set Index values), and then run it again for the other ones just giving the values which are different for them.
Here is an example. Suppose that you have a group of images from a three-chip mosaic camera which all use the same global parameters, except that the readout noise and ADC factor is different according to which chip each image comes from. You can configure the global variables like this:
Having set up the values for Set Index 1, you can do the same for the other Set Index values, entering the changed values explicitly and leaving the others to default to the ones you have just entered:
You can then see the global configuration using the CCDSHOW command:
Note that Index-specific values are only shown where they were assigned and are different; in the other cases a common value will be shown, and used, for all Set members.
The Set Index values you should use depend on how you have used MAKESET to add the Set headers; they will nearly always be 1, 2, 3, ... up to the number of members in each Set, but if you are unsure you can use the SHOWSET command to see the values they have.
Note that when using CCDPACK with Sets, you should ensure that the USESET parameter is set to true by CCDSETUP; when it is run with BYSET true, USESET will default to true as well.
This section provides a brief explanation of what the CCDPACK reduction tasks (MAKEBIAS, MAKECAL, MAKEFLAT, DEBIAS, CALCOR, FLATCOR) do when they encounter input files with Set header information. It assumes that the global USESET parameter is TRUE, so you should first have set it using a command like
The reduction tasks considered here are all concerned with constructing calibration frames from a stack of input frames (MAKEBIAS, MAKECAL, MAKEFLAT) or with applying those calibration frames to a group of data frames (DEBIAS, CALCOR, FLATCOR). In the case of a single-chip instrument it is normally adequate to bung all the input files together and combine them or correct them accordingly. In a multi-chip context however, it is necessary to perform the make/apply calibration cycle separately for the data from each chip.
So the combination programs, which take multiple input images and produce one output image for each chip, do something like this:
If there is more than one Set Index value represented in the input files therefore, more than one calibration frame gets written out. In all cases, the OUT parameter of these programs refers to a single filename, so the programs write a new HDS container file of this name and store one NDF structure inside for each Set Index.
The correction programs, which take one calibration image for each chip and use it to turn multiple input files into corrected output files, then do this:
So if you have several observations from a three-chip mosaic camera sorted into Sets to debias, you can replace a sequence like this:
by this one, which does just the same thing:
You may notice from the scheme above that the output frames have exactly the same Set headers (i.e. the same Set Index and Set Name) as the input frames. As long as pre-calibration and post-calibration frames will not be presented to the same CCDPACK program at the same time, which is normally the case, this will cause no trouble. If for some reason they will be fed together to the same CCDPACK Set-aware program though, you may need to doctor the Set Name headers of one or other Set (use the NAME parameter of MAKESET).
These programs assume that only one type of Set (i.e. Set data from only one instrument) is being considered at once — if two images have the same Set Index attribute they are assumed to come from the same chip, so you should not mix data from different types of Set in the same invocation of a CCDPACK reduction command.
As explained in the previous section, certain parameters may need to be different for the different members of each Set (i.e. they are specific to a given Set Index). If they have been configured globally using CCDSETUP then the appropriate values will be used. Otherwise, you may be prompted for them. In this case you will be prompted for the value once for each Set Index represented in the input files. Note that if you supply these parameters on the command line or use the ACCEPT keyword without using CCDSETUP to assign global values, they will take the same value for all the files.
As with the other Set processing aspects, for the most part it’s not necessary to understand
the technicalities to use these programs, the upshot is that although multiple calibration
frames, one for each chip, are constructed, they are all kept in a single file (an HDS container
file), and this can be passed using just the single filename to the correction programs. Both
lots of programs will keep track of which frames have come from which chip and use
them in the correct places. If you look at the output of the various calibration programs
operating on data with Set headers, the way the above scheme works should be clear.
To see it in action, you can look at the log output of the setexercise script (see section
3).
This section provides a brief explanation of what the CCDPACK registration tasks (FINDOFF, PAIRNDF, CCDALIGN, IDICURS, REGISTER, WCSREG) do when they encounter input files with Set header information.
By the registration stage, it is no longer of direct interest what the source of each image is, only how it relates to the others, so the Set Index attribute is simply ignored by these programs. In fact the registration applications will ignore Set header information altogether unless a Set alignment coordinate system — one with the special Domain name “CCD_SET” — is attached to the image. Whether there is one or not will depend on whether the ADDWCS parameter was true when the Set headers were added with MAKESET. It will also disregard Set alignment if USESET=FALSE.
Only the Set Name attribute and the CCD_SET coordinate system are considered by these programs then. Since the CCD_SET coordinates are taken to represent a fixed and reliable alignment of images in the same Set, all images with the same Set Name are taken to be effectively glued together in those coordinates, and the registration programs will not take any steps to change the relative positioning of the images within a Set.
So the programs FINDOFF and REGISTER which deal in position lists will read in all the input images, construct a single superlist from the associated lists of all the members of the same alignment Set, and treat this as if it came from a single image. IDICURS, PAIRNDF and CCDALIGN, which display images in a GUI and allow the user to mark points on them, will display a whole Set at once for marking or sliding around rather than one NDF at a time. And WCSREG, which mediates between different coordinate systems to come up with a global registration using as many as necessary, accords maximum priority to alignment within a Set in CCD_SET coordinates, and will not allow any realignment to occur which conflicts with that. Of course images from different Sets are not considered to be aligned in their CCD_SET coordinates.
Once again, for most purposes you can just feed the list of images to the programs and they will do the right thing.
Not very many of the “Examples” sections in Appendix B give explicit examples of using the commands with Sets. This is because for the most part the commands are invoked in exactly the same way for data with Set headers as for data without them, but the programs take account of the grouping if there are Set present and the USESET parameter is true. Below however are a some examples of how you can use the Set handling commands and other CCDPACK applications to take advantage of these abilities.
Here we give a brief example of how the CCDPACK data reduction programs can take advantage of Set headers in frames obtained from a mosaic camera.
In this case we will assume that the original data is in the form of multi-extension FITS files; a group of
data frames data-*.fit, a group of bias frames bias-*.fit and a group of flat fields flat-*.fit.
Each MEF represents a single exposure, and contains one Header + Data Unit for each CCD chip in the
mosaic camera. It would be possible to leave these files in their original format and allow CCDPACK
to convert them on the fly, but the output files will be generated in a more manageable way (their
names will be more compact) if we convert them first by hand to NDF format. We do this using the
CONVERT package:
This creates one new HDS container file for each MEF with the same name, but with the
extension .sdf instead of .fit. We next indicate to CCDPACK that we will be using Set
headers:
The next job is to inform CCDPACK that each HDS container file represents a single Set:
This creates no new files, but adds suitable Set header information to each frame. We can now proceed to the data reduction itself:
This combines all the bias container files and produces one new container file called master-bias. The
new file will have the same structure as the files which went to generate it, that is it will represent a
single Set, and will have as many constituent frames as those files, one for each Set Index value in the
inputs. The master bias Set can now be used to debias the data and flat field frames like
this:
and master flat field generation and correction proceeds in the same way:
We now have a set of HDS container files called red_data-*.sdf which contain the reduced data, one
Set (=one exposure) per file.
The thing to note is that once the Set headers have been set up, the commands typed are just the same as for un-Setted data (compare Example 3 in Section 7.2.9), but CCDPACK is taking care that frames are only combined/corrected in groups with corresponding ones; this will be clear if you look at the log output of these tasks.
Here we revisit the example in Section 8.10.2. By making use of the Set alignment coordinates we have described above, the same job can be done with less user effort.
As always when working with Sets, the first thing to do is to tell CCDPACK that Set headers are important:
The data files are the same as in the un-Setted example. This time however the first thing we do with them is to group the images into two Sets, one for each exposure. Using MAKESET’s ASTFILE parameter we can import the saved AST file alignment information at the same time
These commands add a new coordinate system with the Domain (name) “CCD_SET” as well as suitable Set headers.
In this case we suppose that the AST file has been written to extract information about telescope orientation from suitable FITS headers (see the description of the FITSROT parameter of ASTEXP for more details). This simply means we can avoid adding a 180 degrees-rotated frame by hand.
We now perform the familiar automatic registration steps. First execute FINDOBJ, which pays no attention to Set headers and behaves just as usual.
The FINDOFF program however takes special notice of the Set headers:
(much of the output is omitted here for reasons of brevity). What it has done is to construct one list for each Set, rather than one for each frame, on the assumption that the CCD_SET alignment is correct. Having worked out the offsets between the two “superlists”, it writes and associates matched position lists for each of the files which would actually contain at least one of the matched points.
Finally, we can simply invoke REGISTER
Again it has constructed a list for each Set, and worked out the transformation coefficients between
these. When it has successfully done this, every members of each Set which has been registered will be
given a new CCD_REG coordinate system defining the registration, even those which did not have
any associated position list (files 1a and 2a in the example above). For this reason there is no need to
do extra work finding a global registration as in Section 8.10.2. Previewing (DRAWNDF),
resampling (TRANNDF) and combining (MAKEMOS) can now be performed without further
ado.
Again, once the Set alignment has been properly set up, the commands issued are the same as if there were no Sets, but by making use of the Set headers, CCDPACK is able to register groups of frames in which some have no overlap with any of the others.