SC/21.2

Starlink Project
Starlink Cookbook 21.2

H. S. Thomas, M. J. Currie, D. S. Berry, H. A. Parsons & S. Mairs

2021 June 23

Copyright © 2014-2021 Science and Technology Facilities Council,
East Asian Observatory

The SCUBA-2 Data Reduction Cookbook

1.4

pdfpict

Abstract

This cookbook provides an introduction to Starlink facilities, especially smurf, the Sub-Millimetre User Reduction Facility, and the orac-dr pipeline for reducing, displaying, and calibrating SCUBA-2 data. It describes some of the data artefacts present in SCUBA-2 time-series and methods to mitigate them. In particular, this cookbook illustrates the various steps required to reduce the data; and gives an overview of the Dynamic Iterative Map-Maker, which carries out all of these steps using a single command controlled by a configuration file. Specialised configuration files are presented.

Contents

Acronyms
1 Introduction
 1.1 This cookbook
 1.2 Before you start: computing resources
 1.3 Before you start: software
  1.3.1 Data formats
  1.3.2 Initialising Starlink
  1.3.3 KAPPA and SMURF for data processing
  1.3.4 GAIA for viewing your map
  1.3.5 ORAC-DR for running the pipeline
  1.3.6 PICARD for post-reduction processing
  1.3.7 How to get help
 1.4 Processing options
2 SCUBA-2 Overview
 2.1 The instrument
 2.2 Observing modes
 2.3 The raw data
3 The Dynamic Iterative Map-Maker Explained
 3.1 How it works
 3.2 The reduction step-by-step
 3.3 The individual models
 3.4 Stopping criteria
 3.5 Masking
  3.5.1 AST masking
  3.5.2 FLT masking
  3.5.3 COM masking
 3.6 Skipping the AST model
 3.7 Specialised configuration files
  3.7.1 dimmconfig_jsa_generic.lis
  3.7.2 dimmconfig_blank_field.lis
  3.7.3 dimmconfig_bright_compact.lis
  3.7.4 dimmconfig_bright_extended.lis
  3.7.5 dimmconfig_pca.lis
 3.8 Configuration files for solving specific problems
4 The SCUBA-2 Pipeline
 4.1 Pipeline overview
 4.2 The science pipeline
  4.2.1 Pipeline recipes
  4.2.2 REDUCE_SCAN
  4.2.3 REDUCE_SCAN_CHECKRMS
  4.2.4 REDUCE_SCAN_EXTENDED_SOURCES
  4.2.5 REDUCE_SCAN_FAINT_POINT_SOURCES
  4.2.6 REDUCE_SCAN_ISOLATED_SOURCE
  4.2.7 FAINT_POINT_SOURCES_JACKKNIFE
 4.3 Running the science pipeline
 4.4 Changing the defaults
  4.4.1 Changing ORAC-DR’s behaviour
  4.4.2 Changing the pipeline recipe
  4.4.3 Changing the configuration file
  4.4.4 Parameter-file options
 4.5 What to look out for
 4.6 Pipeline output
 4.7 Getting your data from CADC
5 Running makemap Outside the Pipeline
 5.1 Running makemap
 5.2 Interpreting the screen output from makemap
 5.3 Interacting with makemap during a long run
  5.3.1 Monitoring screen output
  5.3.2 Monitoring the map at the end of each iteration
  5.3.3 Interrupting makemap
 5.4 Tips and tricks
  5.4.1 Aligning your map with a pre-existing image
  5.4.2 Limiting the amount of memory used by makemap
  5.4.3 Re-using previously cleaned data to speed up map-making
6 Tailoring Your Reduction
 6.1 Adding and amending parameters
 6.2 Writing out models & intermediate maps
 6.3 Large-scale filtering
 6.4 Fitting COM for each sub-array
 6.5 Flagging bad data
 6.6 Using external masks
 6.7 Skyloop
 6.8 Troubleshooting
7 Examples of Different Reductions
 7.1 Deep point-source maps
  7.1.1 Example 1 – The simple reduction
  7.1.2 Example 2 – Advanced pipeline method (Recommended)
 7.2 Extended galactic sources
8 Post-processing Reduction Steps
 8.1 Flux-conversion factors
  8.1.1 Aperture flux
  8.1.2 Peak flux
  8.1.3 Manually Applying the FCF
  8.1.4 Determining your own flux-conversion factors
 8.2 Cropping your map
 8.3 Co-adding multiple maps
  8.3.1 Registering maps
 8.4 Sensitivity
  8.4.1 Getting the noise
  8.4.2 Map statistics
  8.4.3 Viewing the noise histogram
  8.4.4 Examining the error map with GAIA
 8.5 Regridding your data
 8.6 Displaying masks
 8.7 Point-source extraction: the matched filter
 8.8 Clump finding
 8.9 Map provenance & configuration parameters
9 SCUBA-2 Diagnostic Tools
 9.1 Concatenate & apply a flat-field
 9.2 Headers and file structure
 9.3 Displaying scan patterns
 9.4 Displaying time-series data
 9.5 Regridding data into a map
 9.6 Notes on cleaning your data
 9.7 Checking the array performance
 9.8 Exporting individual models
 References
A Cleaning the Raw Data
B SCUBA-2 Data Calibration
 B.1 Flux-conversion factors (FCFs)
 B.2 Extinction correction
C SCUBA2_MAPSTATS
D SCUBA-2 Matched Filter
E FCFs by Reduction Date
F FCFs by Time of Night
G Aperture-photometry Curve of Growth
H Convert Format from FITS to NDF
I Configuration-parameter Descriptions
J Configuration Parameters Listed by Category
 J.1 General
 J.2 Diagnostics
 J.3 Pre-processing
 J.4 Iterative: COM model
 J.5 Iterative: NOI model
 J.6 Iterative: FLT model
 J.7 Iterative: EXT model
 J.8 Iterative: AST model

Acronyms

CADC Canadian Astronomy Data Centre
CSO Caltech Submillimetre Observatory
DIMM Dynamic Iterative Map-Maker
FCF Flux Conversion Factor
FITS Flexible Image Transport System
FWHM Full-Width at Half-Maximum
GAIA Graphical Astronomy and Image Analysis Tool
ITC Integration Time Calculator
JCMT James Clerk Maxwell Telescope
JSA JCMT Science Archive
MSB Minimum Schedulable Block
NDF Extensible N-Dimensional Data Format
NEP Noise Equivalent Power
NEFD Noise Equivalent Flux Density
PCA Principal Component Analysis
PSF Point Spread Function
PWV Precipitable Water Vapour
RMS Root Mean Square
SCUBA-2 Submillimetre Common User Bolometer Array-2
SMURF Sub-Millimetre User Reduction Facility
S/N Signal-to-Noise ratio
SQUID Superconducting QUantum Interference Device
STC-S Space-Time Coordinate Metadata String Implementation
SUN Starlink User Note
TES Transition Edge Sensor
WVM Water Vapour radioMeter

Bibliography

[1]   Archibald, E. N., et al, 2002, On the atmospheric limitations of ground-based submillimetre astronomy using array receivers, MNRAS, 336, 1-13 (DOI:10.1046/j.1365-8711.2002.05582.x)

[2]   Berry D. S., 2015, FellWalker - a Clump Identification Algorithm, Ast. & Comp., 10, 22-31 (DOI:10.1016/j.ascom.2014.11.004)

[3]   Cavanagh B., Jenness T., Economou F., Currie M. J., 2008, The ORAC-DR data reduction pipeline, Astron. Nactr., 329, 295 (DOI:10.1002/asna.200710944)

[4]   Chapin E. L., et al., 2013, SMURF – Sub-Millimetre User Reduction Facility, Starlink User Note 258

[5]   Chapin E. L., et al., 2013, SCUBA-2: iterative map-making with the Sub-Millimetre User Reduction Facility, MNRAS, 430, 2545 (DOI:10.1093/mnras/stt052)

[6]   Currie M. J., Wallace P. T., Warren-Smith R. F., 1989, Starlink Standard Data Structures, Starlink General Paper 38.2

[7]   Currie M. J., Berry D. S, 2013, KAPPA – Kernel Application Package, Starlink User Note 95

[8]   Dempsey J. T. et al., 2013, SCUBA-2: on-sky calibration using submillimetre standard sources, MNRAS, 430, 2534 (DOI:10.1093/mnras/stt090)

[9]   Dempsey J. T., Friberg P., Jenness T., Bintley D., Holland W. S., 2010 Extinction correction and on-sky calibration of SCUBA-2, Proc. SPIE, 7741 (DOI:10.1117/12.856476)

[10]   Draper P. W., Gray N., Berry D. S., Taylor M., 2012, GAIA – Graphical Astronomy and Image Analysis Tool, Starlink User Note 214

[11]   Gibb A. G., Jenness T., Economou F., 2012, PICARD — a PIpeline for Combining and Analyzing Reduced Data Starlink User Note 265

[12]   Holland, W. S., et al, 2013, SCUBA-2: The 10,000 pixel bolometer camera on the James Clerk Maxwell Telescope, MNRAS, 430, 2513 (DOI:10.1093/mnras/sts612)

[13]   Jenness T., et al, 2002, Towards the automated reduction and calibration of SCUBA data from the James Clerk Maxwell Telescope, MNRAS, 336, 14-21 (DOI:10.1046/j.1365-8711.2002.05604.x)

[14]   Jenness T., et al, 2015, Learning from 25 years of the extensible N-Dimensional Data Format, Ast. & Comp., 12:146-161, (DOI:10.1016/j.ascom.2014.11.001)

[15]   Mairs S., et al, 2015, The JCMT Gould Belt Survey: a quantitative comparison between SCUBA-2 data reduction methods, MNRAS, 454:2557, (DOI:10.1093/mnras/stv2192)

[16]   Mairs S., et al, 2021, A Decade of SCUBA-2: A Comprehensive Guide to Calibrating 450 and 850 micron Data at the JCMT, 2021, AJ, 162, 191, (DOI:10.3847/1538-3881/ac18bf).

[17]   Scott D., Van Engelen A., 2005, Scan Mode Strategies for SCUBA-2, SCUBA-2 Data Reduction document SC2/ANA/S210/005