3 SCUBA-2 Data Reduction

SCUBA-2 was designed to have two primary observing modes where the telescope is either stationary relative to the source (DREAM, STARE and typically when the FTS or polarimeter are in use), or moving (SCAN). In fact, the DREAM/STARE modes have never been commissioned, and so only the SCAN mode is functional at the time of writing. For this reason further information about DREAM/STARE modes has been relegated to appendix F.

Two general approaches may be taken to producing maps from SCUBA-2 data using Smurf commands. By far the easiest approach is to use makemap command to do everything. In this method, the entire job of creating a map, in units of pW, from raw uncalibrated data is handled in a single command. This includes all of the data pre-processing steps, as well as generating the final map.

The alternative approach is to use a range of individual calls to different Smurf task to perform each of the pre-processing steps, followed by using makemap to create the final map from the pre-processed data. This method may be useful if the user wishes to interact and view each step of the reduction, although the final maps are generally not of ‘science grade’ quality. The individual Smurf tasks used in this approach to perform the pre-processing are described in Section 3.2. Note, these tasks are not necessary in the usual case where makemap is used to perform equivalent pre-processing.

Regardless of how the pre-processing is done, the makemap command will be required to create the final map. Two methods are available when running makemap (selected using ADAM parameter METHOD):

• REBIN: A simple re-binning algorithm that just pastes the cleaned and flat-fielded bolometer values into the output map.
• ITERATE: An dynamic iterative method that divides each cleaned and flat-fielded data value into a number of distinct components, one being an estimate of the astronomical signal, and the others being various systematic noise components. The map is then created by pasting just the component representing the astronomical signal into the output map.

The systematic noise in SCUBA-2 data is usually so great that the REBIN method is inadequate for all but very bright compact sources (bright enough to be detected in one sample and smaller than the field of view of the array). In practice, the ITERATE method will give superior results, though it is more computationally intensive. For this reason, the REBIN method should never be used other than in exceptional circumstances, and further discussion of REBIN is relegated to appendix G.

In all normal circumstances, use the dynamic iterative map maker method described in Section 3.3.

3.1 Choosing input data files

In addition to on-sky data files, each observation usually includes other calibration data files such as flatfield ramps and noise fields⁴. It is usually safe to include all the files from an observation, whether on-sky or calibration, when running a Smurf command, as Smurf will sort them all out and only use the appropriate type of file for the operation being performed.

3.2 Array calibration and characterisation

This section describes the individual Smurf commands that can be used to perform the pre-processing steps required prior to making a map from SCUBA_2 data. It is usually simpler to supply the raw data direct to makemap without first using these commands, in which case makemap will automatically perform the required pre-processing.

Data from array calibration observations (such as FLATFIELD and NOISE) are processed with Smurf. NOISE observations are used to identify bolometers which are not in spec, from which a bad-bolometer mask can be created. There is usually no need to re-process flatfield data since the results have already been applied to subsequent observations, but the techniques are included here for completeness. Both NOISE and FLATFIELD observations can be done with the shutter closed or with the shutter open.⁵

3.2.1 FLATFIELD

FLATFIELD observations are processed with the calcflat command. The input data must be from the same observation and the same sub-array. The data are a series of frames in which the current supplied to the internal pixel heaters is varied about a nominal value (see the FITS keyword PIXHEAT). calcflat solves for the optimum heater setting given a list of resistances for each bolometer and a reference resistor value. The list of resistances is mandatory and requires knowledge of the sub-array performance. However, a file with suitable default values is included with the installation of Smurf: $STARLINK_DIR/share/smurf/resist.cfg.

The output from processing a FLATFIELD observation is a data file (named automatically if not supplied) which contains the flatfield solution in the NDF extension .MORE.SCUBA2.FLATCAL, the same location as for the raw data. The main data array for this output file is a three-dimensional array containing the reference-subtracted measurements for each heater setting ($N_{\mathrm{\text{row}}}\times N_{\mathrm{\text{column}}}\times N_{\mathrm{\text{heat}}}$).

In addition to generating the flatfield solution, calcflat can also create a responsivity image for the current sub-array. The RESP parameter may be specified to store the responsivity. The responsivity has units of amperes per watt (A/W). Since each observation contains the flatfield, it is possible to extract the responsivity data from any file by using the calcresp command.

If for some reason this flatfield should be applied to existing data files the copyflat command can be used to copy from the file generated by calcflat. This is sometimes useful if a flatfield was accepted by mistake and it needs to be corrected.

3.2.2 NOISE

NOISE observations are designed to check that the bolometers are operating within specifications. This is achieved by calculating the power spectrum and looking at the mean power between two frequencies (usually 2 and 10 Hz). Excessively noisy bolometers are noted and a bad bolometer mask generated. The calcnoise command can be used to analyze any SCUBA-2 time series. It can be used to generate both a noise image (in pA/rt(Hz)) and a NEP image (in W/rt(Hz)) from raw data. The noise image will be the image in the main part of the data file and the NEP image will be in the .MORE.SMURF.NEP extension. If the data have been flat-fielded the NEP image extension will not be written since the primary image will already be in power units. A noise ratio image (.MORE.SMURF.NOISERATIO) is also created to give an indication of the mean power relative to the power at a particular frequency.

Bad bolometer masks are supported by the flatfield, extinction, remsky, and makemap tasks through the BBM parameter. A mask can be generated from the noise images using the Kappa thresh command.

3.2.3 Comparing multiple noise or flatfield observations

Sometimes you will find that you have many noise or responsivity images from multiple observations and you would like to investigate the bolometer stability. The stackframes command can make this simple as it lets you convert a directory of 2-d images into one data cube where each frame is sorted by time. Once you have this cube you can load it into Gaia for easy visualization, use Kappa collapse with different estimators, or plot variations using Kappa clinplot or mlinplot.

3.3 Dynamic Iterative Map-Maker

The Dynamic Iterative Map-Maker (DIMM) method within the makemap command is the preferred tool for producing maps of SCUBA-2 total power data obtained in SCAN mode[9]. The DIMM is capable of performing all of the required data pre-processing steps, as well as solving iteratively for multiple signal components (including the image of the astronomical sky) using a single call to the makemap task.

As the telescope scans around the sky, each bolometer is sampled around 200 times a second to produce a stream of bolometer values⁶. The pixel values in the final map are in units of pW and represent the mean of the time-series bolometer values (after conversion into pW, down-sampling and removal of systematic noise) that fall in each pixel.

See Chapin et. al. [3] for a full description and analysis of the nature and behaviour of the Dynamic Iterative Map-Maker. Alternatively, see the SCUBA-2 cookbook (SC/21) for shorter and more introductory description.

When using makemap with the Method=Iterate option (which is the default and so does not need to be specified explicitly), it is first necessary to create a plain text file containing settings for any required configuration parameters. They can be listed explicitly in the form ‘keyword=value’, or they can be “inherited” from another file by including the file name preceded by a caret ‘^’. These two methods can also be combined, with later parameter settings over-riding earlier parameter settings. Any parameters that are not assigned a value using one of these two methods will revert to a default value, as listed in Appendix E.

Appendix E gives details of all the available configuration parameters - there are quite a lot! Many of these are of little interest to the majority of Smurf users as they control either experimental features of the map-maker, or features that should not normally be changed. The SCUBA-2 cookbook contains a much reduced list of configuration parameters that are more likely to be of interest to the majority of users.

Various pre-defined configuration files are included with Smurf. The file dimmconfig_jsa_generic.lis within directory $STARLINK_DIR/share/smurf/ is a useful starting point, although better results can often be obtained by using a configuration file tuned to the specific sort of astronomical object being observed (see the other dimmconfig... files in the same directory). The SCUBA-2 cookbook (SC/21) describes other ways in which these “config files” can be customised.

The basic command to make a map is as follows:

where input.lis is a text file containing the paths to all the input raw data files for a single observation, one per line, and config.lis is a text file containing your choice of configuration parameters. Note, all the input data files should relate to the same waveband - 450 or 850 - and the same observation.

Many other options are available - see the list of ADAM parameters for makemap for details. Maybe the most commonly used alternative option is to use a non-default pixel size. To do this assign the required pixel size, in arc-seconds, to the ADAM parameter PIXESIZE when running makemap:

3.4 Speeding up map-making by re-using previously cleaned data

If you want to compare configurations by making several maps from a single observation, each using a different configuration for the iterative stages, you can save considerable time by using pre-cleaned data each time you run makemap.

Note, this is only appropriate if the differences between the various configurations affect only the iterative stage of the map-making process. If any changes affect the cleaning stage, then it is clearly inappropriate to re-use the cleaned data from a previous run.

There are two possible ways to generate the pre-cleaned data:

(1)

Use the Smurf sc2clean task. You can run sc2clean in exactly the same way you would run makemap but instead of creating a map, it creates a set of files containing cleaned and concatenated time-series data that can then be supplied as input to makemap. Any parameters in the configuration file that do not relate to cleaning are ignored by sc2clean so you can conveniently use the same file that you would supply when running makemap. The ADAM parameter OUT, which specifies the names of the output data files, can conveniently be set to “*_cln”, in which case the output files names will be copies of the input file names with the addition of “_cln” to the end. There will usually be fewer output files than input files because the data from multiple input files will be concatenated together into a single file. However, data for different wavebands (450 or 850 $\mu$m) will always be stored in separate files.

(2)

Alternatively, you can ask makemap to generate the cleaned data. To do this, you should add exportclean=1 to the makemap configuration file. In addition to the output map, makemap will then also create one or more files in the current directory with the suffix “..._cln.sdf” containing the cleaned and concatenated time-series data - the earlier part of these file-names will be derived from the names of the input raw data files. Note, remember to remove exportclean=1 from your configuration before running makemap again.

Then for your subsequent runs of makemap you specify the cleaned data generated by one of the above two methods as the input, using ADAM parameter IN. In addition, you must add doclean=0 to the configuration, which prevents makemap from attempting to clean the already cleaned input data.