3 Overview of the SCUBA-2 pipeline

 3.1 Science pipeline
 3.2 Quicklook (QL) pipeline
 3.3 Summit pipeline

3.1 Science pipeline

The science pipeline examines all the data files given to it and works out which files are related and should be processed together (“batch” mode). Each observation is still processed separately to produce an image, which is calibrated in mJy beam1 (unless otherwise specified by the recipe). All the images for a given source are combined into a single coadd using inverse-variance weighting. If the source is a known calibrator then the images are checked for offsets and, if necessary, shifted to place the source at the correct position.

At the end of processing, all temporary files are deleted: the only data files left on disk will have the suffix _reduced.

Recipes exist for a number of different target types which contain different processing steps, relevant to each particular target type. These are bright or faint compact (such as planets, T-Tauri stars or extragalactic blank fields respectively) and extended (such as Galactic star-forming regions). Examples of these steps include applying a matched-filter to enhance point-source detectability or running a clump-finding algorithm.

Note that additional recipes exist for determining the noise properties of the science data. The first performs the same noise analysis and QA checks as the QL pipeline. This can be run offline by adding the recipe name ASSESS_DATA_NOISE to the Orac-dr command line. The second recipe calculates the noise and NEP properties in addition to carrying out the standard map-making procedure, and compares the noise properties with that expected from the SCUBA-2 integration time calculator. This recipe is called REDUCE_SCAN_CHECKRMS. See the documentation below for further details.

3.2 Quicklook (QL) pipeline

The QL pipeline uses a task called qlgather to monitor the DRAMA processes and write out a flag file with the names of the files to process. The pipeline reads that flag file and processes each of the named files. qlgather collates data with the same subscan number. If new data are detected before all the expected files for the current subscan are in place, the new data take precedence and processing of the current subscan may be skipped.

Fastramp flatfields taken as part of each observation are processed and stored in the calibration system. The responsivity image for all four subarrays (along with the previous and percentage-change images) are displayed in a Kapview window.

Pointing and focus data are processed using the iterative map maker with a configuration file optimized for such observations of bright compact sources. Corresponding fastramp flatfields are obtained from the calibration system. For pointing observations an FCF is derived if the source is a known calibrator. The image is displayed using Gaia, showing in window 1, and then combined with the current coadd image (if one exists). The coadd image is displayed in another Gaia window (2), and its error component (noise) in another (3). For focus observations, the data for each SMU position is processed separately and combined into a three-dimensional data cube. The name of the pointing coadd or focus cube is written to a flag file (.sYYYYMMDD_MMMMM.ok, where MMMMM is the current observation number) for further analysis by the telescope POINTING_FOCUS task. The pipeline makes its own estimates of the pointing and focus solution, which are written to log files. Any temporary files created during this processing are deleted, keeping only files with a suffix _reduced or _foc for pointing and focus respectively.

Science data are processed as noise observations. The noise between 2 and 10 Hz is calculated along with the noise equivalent power (NEP) and the weighted NEP. These values undergo quality assurance checks to ascertain whether or not the instrument is still operating within specified limits. The focal-plane noise distribution is displayed in a Kapview window.

Data from other observing modes are processed as per their own recipes.

3.3 Summit pipeline

The summit pipeline is the main image-making pipeline running at the telescope. It will use all available data for processing (unlike the QL which will skip data to deal with the latest files), and as such may fall behind for several subscans. However, the processing is structured such that the processing for a given observation should be complete before a new observation begins.

Pointing and focus observations (along with setups, noise and skydips) are processed in a similar manner to the QL, though no data are skipped.

The pipeline checks for the presence of flag files which are updated with the names of new data as they are written to disk. As each file is picked up by the pipeline, it checks to see how much time has elapsed since the last map was made. If the time exceeds a threshold (typically about one-and-a-half to two minutes), then a new map is made using all of the data taken since the previous map. The map is calibrated in mJy beam1 using the standard FCF. If it is too soon to make a new map, the raw data are flatfielded and left on disk (suffix _ff). These files will be deleted once a new image is made.

The new image is combined with the existing coadd (if appropriate) and a new NEFD image is calculated. Note that the summit pipeline coadds images for a given source across multiple observations so fainter features will be revealed as the integration time increases. If a new image was not created during the latest pass through the recipe then all flatfielded data files needed to create a map are left on disk.