### 9 Set processing — multiple image instruments

In the simplest case, each exposure made on an instrument will read out one CCD using one amplifier/ADC and provide you with one data file containing an array of pixels which represents a single data or calibration frame. However, many instruments are now more complicated than this, and may feature multiple amplifiers reading out different parts of the same chip, or multiple chips on the same mosaic camera for each exposure, or a combination of the two. The resulting data product may consist of one or more data structures (e.g. NDF structures or FITS Header + Data Units (HDUs)) in each of one or more containing files.

Two problems arise when dealing with these sorts of data. Firstly, during the data reduction stage, calibration must be carried out in the same way for data read from the same bits of hardware and differently for data read from different bits; a separate master bias frame must be generated for each CCD in a mosaic camera and each data frame must be calibrated using the corresponding one. Secondly, during the registration stage, alignment information which corresponds to the positioning of the chips on the focal plane of the instrument may be accurately known and not subject to change, and this must be communicated to the software which is attempting to align separate data frames using other methods such as object matching; there’s no point trying to identify matching features on two frames which you know are glued next to each other on the focal plane.

Both of these can be addressed to a large extent by careful use of the facilities of CCDPACK described so far, but to make processing of this kind of data more painless, CCDPACK provides the concept of a Set of images. The idea is that each exposure from a given instrument produces a single Set of data which can contain several frames. All Sets from a particular instrument will consist of a fixed number of members: each member of a given Set will share the same Set Name attribute indicating its origin, and has its own Set Index attribute indicating its rôle, or position, within the Set. Every exposure generated by a four-CCD mosaic camera for instance would produce a single Set with a unique Set Name, and four members with Set Index values of 1, 2, 3 and 4.

Using this functionality is quite simple: the user tells CCDPACK how all the data files are grouped into Sets, and from then on it can make sure that processing is done accordingly. The way this works is that a CCDPACK Set header is added to each file (in the .MORE.CCDPACK.SET extension of the NDF) which contains two items, the Set Name attribute (a string) and the Set Index attribute (an integer, usually 1, 2, 3, …). Set alignment coordinates may also be added as an attached coordinate system, with the Domain name CCD_SET, if a known alignment is to be associated with the Set (because you know how the chips are arranged on the focal plane). The Set itself does not exist as a separate file or other entity, so you never need to specify a Set as such when giving input parameters to CCDPACK programs, but when presented with a list of input files, they will read and make use of Set header information if it exists. In fact in most cases, after adding the Set header in the first place, you can forget about Sets altogether and let CCDPACK worry about them for you.

All the CCDPACK programs which can do Set-sensitive processing consult the USESET global parameter to see whether they should use Set header information if it exists. To make CCDPACK use Set headers, you should use CCDSETUP to set USESET true like this:

% ccdsetup useset=true accept

All subsequent commands will then use Set headers where they have been added. Alternatively you could specify USESET=TRUE on the command line of each CCDPACK program. Since files without Set headers are always treated the same whether USESET is true or false, it is harmless to have set it true all the time, but in certain cases (especially if a non-native data format such as FITS is being used) it may result in slower operation.

Note that Set Name attributes don’t need to be (and in general will not be) totally unique between different Sets; for instance DEBIAS will produce output files which have the same Set Names as the corresponding input files. However, no ambiguity will arise as long as only one Set of files with the same Set Name is presented to any given CCDPACK program at the same time. Normally, because of the flow of data between applications, that situation will not arise, but if for some reason two Sets with the same Name do need to be presented in the same context, MAKESET can be coerced into assigning a given Name to a Set by use of its NAME parameter.

The following sections give some detailed description and examples of how to manipulate Set header information and how CCDPACK goes about using it.

This can operate in a number of different ways according to how your data is arranged in the first place. The most useful ones are described in the following subsections. For variations on these, including the ability to construct Set headers from FITS header information, see the full description of MAKESET in appendix 0.

##### 9.1.1 Grouping by explicit list

In the most straightforward case, you can use MAKESET to make a single Set out of a group of files listed by name:

% makeset ’"im01,im02,im03"’ mode=list accept

MAKESET
=======
3 input NDFs accessed using parameter IN

Set name  Set index   NDF name
--------  ---------   --------

im01:
1       im01
2       im02
3       im03

This will write a Set header to each of the named files. As the output shows, the same Set Name attribute “im01” (the name of the first file in the list) is given to each, and the Set Index attributes are 1, 2 and 3 respectively.

If you have many files to put into Sets, this may be an unwieldy approach. By using the SETSIZE parameter, an invocation like this can generate many Sets at the same time. If your directory contains the files im01… im12 and each group of three in order represents the data from the three CCDs in a mosaic camera, you can write

% makeset ’im*’ mode=list setsize=3 accept

MAKESET
=======
6 input NDFs accessed using parameter IN

Set name  Set index   NDF name
--------  ---------   --------

im01:
1       im01
2       im02
3       im03

im04:
1       im04
2       im05
3       im06

im07:
1       im07
2       im08
3       im09

im10:
1       im10
2       im11
3       im12

Care has to be taken with this usage however; it relies on the files being presented to MAKESET in the correct order, which in turn may depend on how your operating system expands wildcards, which is normally alphabetically. If for instance in the above example the files had been named

im1   im2   im3   im4   im5   im6   im7   im8   im9  im10  im11  im12

(i.e. without the leading zero for numbers less than 10) then they would probably have been presented to MAKESET in the order

im1   im10  im11  im12  im2   im3   im4   im5   im6  im7   im8   im9

which is almost certainly not what you want. If you use MAKESET with wildcards you should be careful to check the logged output to ensure that it has done the right thing.

##### 9.1.2 Grouping by container

If your data exists in HDS container files so that one container file contains all the images from a single exposure, then adding Set headers is easier. By running MAKESET with MODE=CONTAINER, it will assume that each HDS container file corresponds to a single Set. In this case you can just type something like

% makeset in=’obs*’ mode=container accept

Note that if your data initially exists in the form of one Multi-Extension FITS file (MEF) per observation you can convert these quite easily to HDS container files using the FITS2NDF command with CONTAINER=TRUE. So supposing you have three exposures, each stored in a 2-HDU MEF, you can do:

% convert

CONVERT commands are now available -- (Version 1.3-5 2001 January)

Defaults for automatic NDF conversion are set.

Type conhelp for help on CONVERT commands.
Type "showme sun55" to browse the hypertext documentation.

% fits2ndf ’obs*.fits’ ’conv-*’ container=true
3 files selected.

% makeset ’conv-obs*’ mode=container accept

MAKESET
=======
6 input NDFs accessed using parameter IN

Set name  Set index   NDF name
--------  ---------   --------

conv-obs1:
1       conv-obs1.HDU_1
2       conv-obs1.HDU_2

conv-obs2:
1       conv-obs2.HDU_1
2       conv-obs2.HDU_2

conv-obs3:
1       conv-obs3.HDU_1
2       conv-obs3.HDU_2

For subsequent work you need only supply the names of the container files (conv-obs*) and the CCDPACK commands will look inside and find the data frames with their associated Set header information. See SUN/55 for a full description of the CONVERT facility.

##### 9.1.3 Splitting a single image into multiple sections

If each observation initially exists as a single image but needs to be treated as multiple images because different regions have been read out using different electronics (so that, for instance, they have to be debiassed using different ADC factors), then a slightly different approach is required. In this case, you should set MODE=SPLIT, and unlike for the other modes MAKESET will create new NDFs each with its own Set header. You will have to supply the parameter OUT to give the name of the file containing the split data: this will be a single HDS container file containing as many NDF structures as the input array is split into. For full generality, the parameter SECTIONS specifies how the input image is to be split up, but for the common case of dividing the array into grid of rectangles which butt up to each other the XSTART and YSTART parameters are easier. The following example shows creation of a Set made by dividing a 2048x4096 array into quarters:

% makeset in=sg28948 mode=split out=’*_sp’ \
sections=’!’ xstart=’[1,1025]’ ystart=’[1,2049]’

MAKESET
=======
1 NDF accessed using parameter IN

Set name  Set index   NDF name                            CCD_SET domain
--------  ---------   --------                            --------------

Writing to container file sg28948_sp
sg28948:
1       sg28948(1:1025,1:2049)              PIXEL
2       sg28948(1:1025,2049:)               PIXEL
3       sg28948(1025:,1:2049)               PIXEL
4       sg28948(1025:,2049:)                PIXEL

The command creates a single new HDS container file “sg28948_sp”: when you give this as an input to CCDPACK commands they will look inside and process each of the four images in turn.

This method could also be used to create a Set from different slices of a three-dimensional data set.

##### 9.1.4 Set alignment (CCD_SET) coordinates

You may notice that in the last example, a column headed “CCD_SET domain” has appeared in the output, with the entry “PIXEL” for each of the new Set members. This indicates that as well as the Set header, each file has had a new coordinate system attached to its WCS component. This new coordinate system added by MAKESET always has the Domain (name) “CCD_SET”, and in this case is a copy of the PIXEL-Domain coordinate system which existed in the input image. When MAKESET adds a CCD_SET coordinate system like this, it is declaring that all the members of the same Set are known to be correctly aligned in these coordinates. All the CCDPACK programs concerned with registration will recognise a CCD_SET coordinate system and assume that the alignment of all the images in a given Set is correct in these coordinates and is not to be tampered with. Thus, it is only a good idea to add a CCD_SET coordinate system if it really does represent an accurate alignment of images within the Set and not one which, for instance, you might want to improve at a later date by object matching. When MODE=SPLIT, it is assumed that the PIXEL coordinates of the original image do represent this sort of rock-solid alignment, since the Set members have been hewn out of the same data array. With the other modes of MAKESET Set alignment coordinates are not added by default, but if the alignment of the Set is known, it can be added by setting the ADDWCS parameter. In this case a copy of the Current attached coordinate system of each image is used for the CCD_SET alignment coordinates.

So suppose that you have a directory full of HDS container files each holding the data frames from one observation. Each has an attached coordinate system in its WCS component, with the Domain “FOCAL” which gives its exact position on the focal plane of the instrument — this may have been added by the observing system, and it is known to be accurate. First ensure that the FOCAL coordinates are the Current coordinates of all the frames:

% wcsedit ’*’ current focal

% makeset ’*’ mode=container addwcs=yes accept

MAKESET
=======
3 input NDFs accessed using parameter IN

Set name  Set index   NDF name                            CCD_SET domain
--------  ---------   --------                            --------------

f10001:
1       f10001.I1                           FOCAL
2       f10001.I2                           FOCAL
3       f10001.I3                           FOCAL

As you can see, the FOCAL coordinate system is listed as the one copied to make the new one. Note that since the new CCD_SET coordinate system is a copy of the FOCAL one, it will remain attached to the images even if the FOCAL one is erased or modified.

Variations on these methods of adding Set headers, as well as the possibility of controlling Set formation from FITS headers, are discussed in the description of MAKESET in §0.

If you need to add Set alignment coordinates to some files after MAKESET has already been run, you can just attach a new coordinate system with the name CCD_SET, using WCSEDIT or in some other way. Note however that a CCD_SET-domain coordinate system will only be treated as special when the images in question have valid Set headers.

It will not normally be necessary to remove Set headers once they have been added by MAKESET, but if you find you have added them in error, you can erase them using the CCDEDIT task, as follows:

% ccdedit in=’*’ mode=erase name=set fixwcs=yes

The FIXWCS parameter determines whether any existing Set alignment coordinate systems (ones with Domain CCD_SET) should be erased also. It normally doesn’t matter much whether this is erased or not, since without a valid Set header it is ignored anyway.

If you just want to remove the Set alignment frame, you can remove this attached coordinate frame directly using WCSEDIT:

% wcsedit in=’*’ mode=remove frame=ccd_set

Both of these commands will erase the information in question if it exists, and note the fact with a non-fatal error if it does not.

If you try to add a new Set header to an image which already has one using MAKESET, or a new CCD_SET coordinate system to an image which already has one using MAKESET or some other method, a warning will be issued, but the old one will be overwritten.

For the most part, once you have added Set headers to your image files using MAKESET you will be able to let them take care of themselves. However, you may wish to check how images are related to each other, or to pull out only members of a given Set or images which correspond to each other from different Sets. For these purposes the

application is provided.

In its simplest mode of operation, it will show you the Set headers of a list of images, grouped by Set Name or Set Index. Here is a simple invocation, showing the result of the example in Section 9.1.2:

% showset ’conv-obs?’ reset

SHOWSET
=======
6 input NDFs accessed using parameter IN
Selection excludes NDFs with no Set headers.

Set name  Set index   NDF name                                CCD_SET frame
--------  ---------   --------                                -------------
conv-obs1:
1           conv-obs1.HDU_1                               no
2           conv-obs1.HDU_2                               no
conv-obs2:
1           conv-obs2.HDU_1                               no
2           conv-obs2.HDU_2                               no
conv-obs3:
1           conv-obs3.HDU_1                               no
2           conv-obs3.HDU_2                               no

Namelist written to file: showset.lis

As you can see, it gives very similar output to that written by MAKESET when the headers were being added. If you prefer to corresponding images from different Sets grouped together, then change the LISTBY parameter (which defaults to ‘NAME’):

% showset ’conv-obs?’ listby=index reset

SHOWSET
=======
6 input NDFs accessed using parameter IN
Selection excludes NDFs with no Set headers.

Set index Set name                    NDF name                CCD_SET frame
--------- --------                    --------                -------------
1:
conv-obs1                     conv-obs1.HDU_1               no
conv-obs2                     conv-obs2.HDU_1               no
conv-obs3                     conv-obs3.HDU_1               no
2:
conv-obs1                     conv-obs1.HDU_2               no
conv-obs2                     conv-obs2.HDU_2               no
conv-obs3                     conv-obs3.HDU_2               no

Namelist written to file: showset.lis

As well as showing the Set information of all the files presented, SHOWSET is also able to select a sublist of input files according to the values in their Set headers. By supplying the PICKINDEX and/or PICKNAME parameters, images can be selected according to the values of their Set Index and/or Set Name attributes respectively. The default value of PICKNAME is ‘ALL’, meaning that any Set Name will be selected for output. However, if PICKNAME is set to ‘EQUAL’, then the NAME parameter gives a list of Set Name values and only images whose Set Name appears in this list will be selected. If PICKNAME is set to ‘LIKE’, then the NAMELIKE parameter gives a list of other images to act as templates, and only images whose Set Name is the same as that of one of the template images will be selected. The PICKINDEX, INDEX and INDEXLIKE parameters work in the same way for Set Index values. Returning to another example in the previous section, we can pick out only those images in the same Set as (i.e. having the same Set Name attribute as) im12 but which have Set Index of 1 or 2 as follows (in this case some of the parameters are input as the response to prompts):

% showset ’im*’ pickindex=equal pickname=like listby=none

SHOWSET
=======
12 input NDFs accessed using parameter IN
NAMELIKE - Template NDFs for Set Name value > im12
1 NDF accessed using parameter NAMELIKE
INDEX - Acceptable Set Index values > 2,3
Selection restricted to Set Name values: im10
Selection restricted to Set Index values: 2
3

Set index Set name                    NDF name                CCD_SET frame
--------- --------                    --------                -------------
2       im10                          im11                          no
3       im10                          im12                          no

Namelist written to file: showset.lis

Here we have also set the LISTBY parameter to ‘NONE’, which does no grouping and lists the images in their input order.

As it points out, SHOWSET writes a list of the images it has output to a file given by the NAMELIST parameter, by default showset.lis. In the above case that list would read:

# SHOWSET - selected NDF name list
im11
im12

This file can be used with the indirection character (‘^’) to provide input for any other CCDPACK program, so you can write scripts with commands which will operate only on members of certain Sets or certain Index values. For example, you could subsequently use DRAWNDF to display the outlines of the images that the above command has selected like this:

% drawndf in=’^showset.lis’ clear lines

#### 9.4 Global parameters with Sets

Section 7.1.1 explained how to set up global values for certain parameters such as the ADC factor and bias strip positions using the CCDSETUP program; by doing this you can set values specific to the data you are using. However, if you have data in Sets life becomes a bit more complicated: each chip will have a different ADC factor, so the value to use will depend on which member of the Set is being processed.

To cope with this, CCDSETUP can be used to configure different values for certain parameters according to Set Index value. The parameters which can vary with Set Index are:

• BOUNDS
• DEFERRED
• DIRECTION
• EXTENT
• RNOISE
• SATURATION

To set up one or more of these values to be specific to certain Set Index values, run CCDSETUP with the BYSET parameter true, and the INDEX parameter equal to the Set Index to which the parameters apply. When USESET is true, CCDPACK programs which use these global parameters will first look for a value specific to the Set Index of the frame being processed, and if there isn’t one they will look for the unspecific global value (if there is no global value at all they will use the current parameter value or ask you to enter it or use a default as usual). So, since many of the parameters will probably be the same for all the members of each Set, the easiest thing is to run CCDSETUP for one of the chips (Set Index values), and then run it again for the other ones just giving the values which are different for them.

Here is an example. Suppose that you have a group of images from a three-chip mosaic camera which all use the same global parameters, except that the readout noise and ADC factor is different according to which chip each image comes from. You can configure the global variables like this:

% ccdsetup

Type "?" for help on any prompt.
Type "!" if you do not want to set a parameter.

BYSET - Set up values specific to one Set Index? /NO/ > yes
INDEX - Set Index values these parameters are specific to > 1
RESTORE - Use a setup restoration file /NO/ >
LOGTO - Write log to (Logfile,Terminal,Neither,Both) /’Both’/ >
LOGFILE - logfile name /’CCDPACK.LOG’/ >

CCDSETUP.............................
=====================================

Some values are specific to Set Index 1

ADC - Number of electrons per ADU - Set Index 1 /!/ > 1.83
EXTENT - Useful CCD region (xmin,xmax,ymin,ymax) - Set Index 1 /!/ > 13,1018,1,2
048
RNOISE - Readout noise (ADUs) - Set Index 1 /!/ > 8.5
BOUNDS - Pixel indices of bias strips (in pairs) - Set Index 1 /!/ > 1,12,1019,1
024
DIRECTION - Readout direction (X or Y) - Set Index 1 /!/ > X
DEFERRED - Deferred charge (ADUs) - Set Index 1 /!/ >
SATURATE - Look for saturated pixels /NO/ >
PRESERVE - Do you want to preserve your input data types /YES/ >
GENVAR - Do you want to generate variance estimates /YES/ > no
NDFNAMES - Associate position lists with NDF names /YES/ >
USESET - Use CCDPACK Set headers if available /YES/ >

Listing of the current CCDPACK global parameters (Set Index 1)

Global output NDF extent (xmin:xmax,ymin:ymax): (13:1018,1:2048)
Global bias strip bounds                      : (1:12,1019:1024)
Not looking for saturated pixels
Data types will be preserved
Variances will not be generated
Position lists will be associated with NDFs
CCDPACK Set header information will be used where available

SAVE - Save CCD parameters for future restoration /NO/ >

Having set up the values for Set Index 1, you can do the same for the other Set Index values, entering the changed values explicitly and leaving the others to default to the ones you have just entered:

% ccdsetup index=2 adc=2.18 rnoise=6.0 accept
% ccdsetup index=3 adc=1.95 rnoise=9.1 accept

You can then see the global configuration using the CCDSHOW command:

% ccdshow

CCDSHOW
=======
Listing of the current CCDPACK global parameters:
Set Index-keyed values will be shown where available

Set Index 1 : 1.83
Set Index 2 : 2.18
Set Index 3 : 1.95
Global output NDF extent (xmin:xmax,ymin:ymax): (13:1018,1:2048)
Set Index 1 : 8.5
Set Index 2 : 6
Set Index 3 : 9.1
Global bias strip bounds                      : (1:12,1019:1024)
Not looking for saturated pixels
Data types will be preserved
Variances will not be generated
Position lists will be associated with NDFs
CCDPACK Set header information will be used if available

Note that Index-specific values are only shown where they were assigned and are different; in the other cases a common value will be shown, and used, for all Set members.

The Set Index values you should use depend on how you have used MAKESET to add the Set headers; they will nearly always be 1, 2, 3, ... up to the number of members in each Set, but if you are unsure you can use the SHOWSET command to see the values they have.

Note that when using CCDPACK with Sets, you should ensure that the USESET parameter is set to true by CCDSETUP; when it is run with BYSET true, USESET will default to true as well.

#### 9.5 Using Sets in reduction tasks

This section provides a brief explanation of what the CCDPACK reduction tasks (MAKEBIAS, MAKECAL, MAKEFLAT, DEBIAS, CALCOR, FLATCOR) do when they encounter input files with Set header information. It assumes that the global USESET parameter is TRUE, so you should first have set it using a command like

% ccdsetup useset=true accept

The reduction tasks considered here are all concerned with constructing calibration frames from a stack of input frames (MAKEBIAS, MAKECAL, MAKEFLAT) or with applying those calibration frames to a group of data frames (DEBIAS, CALCOR, FLATCOR). In the case of a single-chip instrument it is normally adequate to bung all the input files together and combine them or correct them accordingly. In a multi-chip context however, it is necessary to perform the make/apply calibration cycle separately for the data from each chip.

So the combination programs, which take multiple input images and produce one output image for each chip, do something like this:

(1)
Read in all the input files
(2)
Group them according to Set Index attribute
(3)
For each Set Index value represented in the input files:
• Get values of any global parameters specific to this Set Index
• Combine the data from those of the input files with this Set Index to produce a calibration frame
• Write out the calibration frame with a suitable (new) Set Name attribute and the same Set Index attribute as this group of input files

If there is more than one Set Index value represented in the input files therefore, more than one calibration frame gets written out. In all cases, the OUT parameter of these programs refers to a single filename, so the programs write a new HDS container file of this name and store one NDF structure inside for each Set Index.

The correction programs, which take one calibration image for each chip and use it to turn multiple input files into corrected output files, then do this:

(1)
Read in all the input files
(2)
Group them according to Set Index attribute
(3)
For each Set Index value represented in the input files:
• Get values of any global parameters specific to this Set Index
• Read in the calibration frame with the corresponding Set Index from the calibration file
• Go through all the input files with this Set Index, correct them using the corresponding calibration frame, and write them out with the same Set headers as the input files

So if you have several observations from a three-chip mosaic camera sorted into Sets to debias, you can replace a sequence like this:

% makebias in=’bias*_chip1’ out=master_chip1
% makebias in=’bias*_chip2’ out=master_chip2
% makebias in=’bias*_chip3’ out=master_chip3
% debias in=’data*_chip1’ bias=master_chip1 out=’debias_*’
% debias in=’data*_chip2’ bias=master_chip2 out=’debias_*’
% debias in=’data*_chip3’ bias=master_chip3 out=’debias_*’

by this one, which does just the same thing:

% makebias in=’bias*_chip?’ out=master
% debias in=’data*_chip?’ bias=master out=’debias_*’

You may notice from the scheme above that the output frames have exactly the same Set headers (i.e. the same Set Index and Set Name) as the input frames. As long as pre-calibration and post-calibration frames will not be presented to the same CCDPACK program at the same time, which is normally the case, this will cause no trouble. If for some reason they will be fed together to the same CCDPACK Set-aware program though, you may need to doctor the Set Name headers of one or other Set (use the NAME parameter of MAKESET).

These programs assume that only one type of Set (i.e. Set data from only one instrument) is being considered at once — if two images have the same Set Index attribute they are assumed to come from the same chip, so you should not mix data from different types of Set in the same invocation of a CCDPACK reduction command.

As explained in the previous section, certain parameters may need to be different for the different members of each Set (i.e. they are specific to a given Set Index). If they have been configured globally using CCDSETUP then the appropriate values will be used. Otherwise, you may be prompted for them. In this case you will be prompted for the value once for each Set Index represented in the input files. Note that if you supply these parameters on the command line or use the ACCEPT keyword without using CCDSETUP to assign global values, they will take the same value for all the files.

As with the other Set processing aspects, for the most part it’s not necessary to understand the technicalities to use these programs, the upshot is that although multiple calibration frames, one for each chip, are constructed, they are all kept in a single file (an HDS container file), and this can be passed using just the single filename to the correction programs. Both lots of programs will keep track of which frames have come from which chip and use them in the correct places. If you look at the output of the various calibration programs operating on data with Set headers, the way the above scheme works should be clear. To see it in action, you can look at the log output of the setexercise script (see section 3).

#### 9.6 Using Sets in registration tasks

This section provides a brief explanation of what the CCDPACK registration tasks (FINDOFF, PAIRNDF, CCDALIGN, IDICURS, REGISTER, WCSREG) do when they encounter input files with Set header information.

By the registration stage, it is no longer of direct interest what the source of each image is, only how it relates to the others, so the Set Index attribute is simply ignored by these programs. In fact the registration applications will ignore Set header information altogether unless a Set alignment coordinate system — one with the special Domain name “CCD_SET” — is attached to the image. Whether there is one or not will depend on whether the ADDWCS parameter was true when the Set headers were added with MAKESET. It will also disregard Set alignment if USESET=FALSE.

Only the Set Name attribute and the CCD_SET coordinate system are considered by these programs then. Since the CCD_SET coordinates are taken to represent a fixed and reliable alignment of images in the same Set, all images with the same Set Name are taken to be effectively glued together in those coordinates, and the registration programs will not take any steps to change the relative positioning of the images within a Set.

So the programs FINDOFF and REGISTER which deal in position lists will read in all the input images, construct a single superlist from the associated lists of all the members of the same alignment Set, and treat this as if it came from a single image. IDICURS, PAIRNDF and CCDALIGN, which display images in a GUI and allow the user to mark points on them, will display a whole Set at once for marking or sliding around rather than one NDF at a time. And WCSREG, which mediates between different coordinate systems to come up with a global registration using as many as necessary, accords maximum priority to alignment within a Set in CCD_SET coordinates, and will not allow any realignment to occur which conflicts with that. Of course images from different Sets are not considered to be aligned in their CCD_SET coordinates.

Once again, for most purposes you can just feed the list of images to the programs and they will do the right thing.

#### 9.7 Some examples of Set use

Not very many of the “Examples” sections in Appendix B give explicit examples of using the commands with Sets. This is because for the most part the commands are invoked in exactly the same way for data with Set headers as for data without them, but the programs take account of the grouping if there are Set present and the USESET parameter is true. Below however are a some examples of how you can use the Set handling commands and other CCDPACK applications to take advantage of these abilities.

##### 9.7.1 Data reduction of frames from a mosaic camera

Here we give a brief example of how the CCDPACK data reduction programs can take advantage of Set headers in frames obtained from a mosaic camera.

In this case we will assume that the original data is in the form of multi-extension FITS files; a group of data frames data-*.fit, a group of bias frames bias-*.fit and a group of flat fields flat-*.fit. Each MEF represents a single exposure, and contains one Header + Data Unit for each CCD chip in the mosaic camera. It would be possible to leave these files in their original format and allow CCDPACK to convert them on the fly, but the output files will be generated in a more manageable way (their names will be more compact) if we convert them first by hand to NDF format. We do this using the CONVERT package:

% convert
% fits2ndf in=’"data-*.fit,bias-*.fit,flat-*.fit"’ out=’*’ container=true

This creates one new HDS container file for each MEF with the same name, but with the extension .sdf instead of .fit. We next indicate to CCDPACK that we will be using Set headers:

% ccdsetup useset=yes accept

The next job is to inform CCDPACK that each HDS container file represents a single Set:

% makeset in=’"data-*,bias-*,flat-*"’ mode=container

This creates no new files, but adds suitable Set header information to each frame. We can now proceed to the data reduction itself:

% makebias in=’bias-*’ out=master-bias

This combines all the bias container files and produces one new container file called master-bias. The new file will have the same structure as the files which went to generate it, that is it will represent a single Set, and will have as many constituent frames as those files, one for each Set Index value in the inputs. The master bias Set can now be used to debias the data and flat field frames like this:

% debias in=’"data-*,flat-*"’ bias=master-bias out=’db-*’

and master flat field generation and correction proceeds in the same way:

% makeflat in=’"db_flat-*"’ out=master-flat
% flatcor in=’db_data-*’ flat=master-flat out=’*|db_|red_|’

We now have a set of HDS container files called red_data-*.sdf which contain the reduced data, one Set (=one exposure) per file.

The thing to note is that once the Set headers have been set up, the commands typed are just the same as for un-Setted data (compare Example 3 in Section 7.2.9), but CCDPACK is taking care that frames are only combined/corrected in groups with corresponding ones; this will be clear if you look at the log output of these tasks.

##### 9.7.2 Registering frames from a mosaic camera

Here we revisit the example in Section 8.10.2. By making use of the Set alignment coordinates we have described above, the same job can be done with less user effort.

As always when working with Sets, the first thing to do is to tell CCDPACK that Set headers are important:

% ccdsetup useset=true accept

The data files are the same as in the un-Setted example. This time however the first thing we do with them is to group the images into two Sets, one for each exposure. Using MAKESET’s ASTFILE parameter we can import the saved AST file alignment information at the same time

% makeset in=’1?’ mode=list astfile=instrument.ast
% makeset in=’2?’ mode=list astfile=instrument.ast

These commands add a new coordinate system with the Domain (name) “CCD_SET” as well as suitable Set headers.

In this case we suppose that the AST file has been written to extract information about telescope orientation from suitable FITS headers (see the description of the FITSROT parameter of ASTEXP for more details). This simply means we can avoid adding a 180 degrees-rotated frame by hand.

We now perform the familiar automatic registration steps. First execute FINDOBJ, which pays no attention to Set headers and behaves just as usual.

% findobj in=’"1?,2?"’ outlist=’*.find’

The FINDOFF program however takes special notice of the Set headers:

% findoff inlist=’"1?,2?"’ outlist=’*.off’

NDFs containing position lists     Current domain      Set Name attribute
------------------------------     --------------      ------------------
1) 1a                                     CCD_SET        1a
2) 1b                                     CCD_SET        1a
3) 1c                                     CCD_SET        1a
4) 2a                                     CCD_SET        2a
5) 2b                                     CCD_SET        2a
6) 2c                                     CCD_SET        2a

Input position lists:
---------------------
1) 1a.find
1b.find
1c.find
2) 2a.find
2b.find
2c.find

...

List  List  No. matches  Completeness  Status   Algorithm
----  ----  -----------  ------------  ------   ---------
1     2      8          0.8714286     ACCEPTED    FAST

...

Approximate offsets in image Current coordinates:

List             X-offset                    Y-offset
1)               +0.000000                   +0.000000
2)             +164.066734                 +310.941234

Output position lists:
----------------------
1b.off
1c.off
2b.off
2c.off

(much of the output is omitted here for reasons of brevity). What it has done is to construct one list for each Set, rather than one for each frame, on the assumption that the CCD_SET alignment is correct. Having worked out the offsets between the two “superlists”, it writes and associates matched position lists for each of the files which would actually contain at least one of the matched points.

Finally, we can simply invoke REGISTER

% register inlist=’"1?,2?"’ fittype=1
6 input NDFs accessed using parameter INLIST
There is no associated list for NDF 1a.
There is no associated list for NDF 2a.

NDFs containing position lists     Current domain      Set Name attribute
------------------------------     --------------      ------------------
1) 1b                                     CCD_SET        1a
2) 1c                                     CCD_SET        1a
3) 2b                                     CCD_SET        2a
4) 2c                                     CCD_SET        2a

Input position lists:
---------------------
1) 1b.off
1c.off
2) 2b.off
2c.off

...

List 1)
A =      +0.000000       B = 1                    C = 0
D =      -0.000000       E = 0                    F = 1

List 2)
A =    +164.203695       B = 0.9999997            C = 0.0007688743
D =    +310.941640       E = -0.0007688743        F = 0.9999997

Again it has constructed a list for each Set, and worked out the transformation coefficients between these. When it has successfully done this, every members of each Set which has been registered will be given a new CCD_REG coordinate system defining the registration, even those which did not have any associated position list (files 1a and 2a in the example above). For this reason there is no need to do extra work finding a global registration as in Section 8.10.2. Previewing (DRAWNDF), resampling (TRANNDF) and combining (MAKEMOS) can now be performed without further ado.

Again, once the Set alignment has been properly set up, the commands issued are the same as if there were no Sets, but by making use of the Set headers, CCDPACK is able to register groups of frames in which some have no overlap with any of the others.