SSN/72.2

Starlink Project
Starlink System Note 72.2

Tim Jenness, John F. Lightfoot
Joint Astronomy Centre, Hilo, Hawaii

10 July 2000

SURF Programming Interface

pdfpict

Abstract

This document describes the SURF programming interface. Including the library API, instructions on how to build a SURF task, the SURF file format and instructions on how to write a SURF import task.

Contents

1 Introduction
2 SURF File Format
 2.1 Raw data format
 2.2 SURF Format
 2.3 Extensions
  2.3.1 FITS
  2.3.2 SCUBA
  2.3.3 SCUCD
  2.3.4 REDS
3 Aborted data files
4 Quality flags
5 SURF Libraries
 5.1 SURFLIB
 5.2 SCULIB
6 Anatomy of a SURF task
7 Writing an import task
 7.1 Case Study: DREAM
Glossary
A SURF coordinate frames
B Building SURF
C SURF Constants
 C.1 Constant strings
 C.2 Convenient numeric definitions
 C.3 Parameters controlling the observation limits
 C.4 Parameters governing the instrument itself
 C.5 Parameters dealing with automatic output filename generation
 C.6 General limits
D SCUBA FITS headers
 D.1 Skydip
 D.2 Jiggle map
 D.3 Photometry
 D.4 Scan Map
E Flatfield file format
F The DREAM2SURF import task
G Library APIs
 G.1 SURF
SURF_GRID_CALCSKY – Calculate sky contribution from median image
SURF_GRID_DESPIKE – Despike data by sorting into a given grid position
SURF_MON – main routine for SCUBA offline data reduction package
SURF_READ_REBIN_NDF – Read an NDF into memory prior to regridding
SURF_RECURSE_READ – Allow the recursive read of REBIN text files and NDFs
SURF_REQUEST_OUTPUT_COORDS – Request longitude and latitude of output map from user
SURF_SET_APP_NAME – Set application name for NDF history
SURF_WRITE_DATA – Write bolometer positions and values to text file
SURF_WRITE_MAP_INFO – Calculate FITS and WCS information for rebinned image
SURF_WRITE_PHOTOM – Routine to output ASCII results of PHOTOM reduction
SURF_WRITE_PHOTOM_HEADER – Write photom information to text file

 G.2 SURFLIB
SURFLIB_2DFT_CHOP – Calculate the FT of the 2D chop throw
SURFLIB_CALC_DUAL_BEAM – Calculates a chopped map from a single beam map
SURFLIB_CALC_GRIDIJ – Calculate the position of a pixel number in an IJ grid
SURFLIB_CALC_IJPOS – Calculate the position of data on an XY grid
SURFLIB_CALC_OUTPUT_GRID – Calculate the grid required to contain data given the X,Y positions
SURFLIB_CALC_POLPACK_ANGROT – Average rotations angles over integration
SURFLIB_CLIP_GRID – Remove spikes from grid
SURFLIB_DECODE_REMSKY_STRING – Calculate bolometer list from remsky input
SURFLIB_DIFF_DESPIKE – Despike data scan by detecting points more than nsigma from running 2-point mean
SURFLIB_FILL_GRID – Populate grid with data points corresponding to position
SURFLIB_FILL_POLPACK_ANGLES – Average
SURFLIB_FILL_WPLATE – Populate the WavePlate array
SURFLIB_HISTOGRAM_GRID – Calculate a 2-D histogram of I,J coordinates on a grid
SURFLIB_MEDIAN_REGRID – Generate image with each pixel the median of all pixels in bin
SURFLIB_PLOT_GRID – Plot unwrapped grid
SURFLIB_PROCESS_BOLS – Calculate apparent RA/Dec of bolometers and some extra processing
SURFLIB_READ_IPFILE – Reads IP data from text file
SURFLIB_REM_GRID – Remove an image grid of data from demodulated data
SURFLIB_REM_TIMESERIES – Remove a time series from a 2D array
SURFLIB_REMOVE_DC_FROM_EXP – Remove linear baseline from each exposure
SURFLIB_REMOVE_DC_VIA_SECT – Remove median baseline from each exposure
SURFLIB_REMOVE_IP – Remove instrumental polarisation
SURFLIB_STATS_GRID – Calculate statistics of binned data
SURFLIB_TRIM_IMAGE – Sets quality bit for specified distance from exisiting bad data

 G.3 SCULIB
SCULIB_1D2_JIGGLE – routine to unpack a 1-d jiggle dataset into a 2-d image
SCULIB_2POS_CONFN – Generate a convolution function to remove the 2-position chop function from raster scans
SCULIB_2POS_DECONV – deconvolve square chop from scan
SCULIB_3POS_CONFN – Generate a convolution function to remove the 3-position chop function from raster scans
SCULIB_3POS_DECONV – deconvolve 3-position chop from scan
SCULIB_ADD_CHOP – Add chop throw to apparent RA/Dec
SCULIB_ADD_DEMOD_EXPOSURE – add demodulated data for an exposure into the integration result
SCULIB_ADDARE – add one real array to another into a third
SCULIB_ADDCAD – add a constant to a double array
SCULIB_ADDCAI – add a constant to an integer array
SCULIB_ADDCAR – add a constant to a real array
SCULIB_AIRMASS – calculate the airmass for a given zenith distance
SCULIB_ANALYSE_PHOTOM_JIGGLE – analyse the jiggle map made by a bolometer during a PHOTOM observation
SCULIB_APPARENT_2_MP – Calculate mean place from a given apparent RA,Dec
SCULIB_APPARENT_2_TP – calculate tangent plane coordinates from apparent RA, Decs
SCULIB_BESSEL_WTINIT – Generate a weighting function for rebinning
SCULIB_BESSJ1 – calculates Bessel function J1(x)
SCULIB_BITON – turn on a bit
SCULIB_BITOFF – Turn off a bit
SCULIB_BITOR – Returns the bitwise OR of two bytes
SCULIB_BITAND – Calculate the bitwise AND of two bytes
SCULIB_BITTEST – Test whether a bit is on
SCULIB_BOLDECODE – decode a bolometer ID into ADC and channel number
SCULIB_BOLNAME – generate bolometer name from ADC and channel number
SCULIB_BOLSELECT – interpret a list of selected bolometers
SCULIB_CALC_APPARENT – calculate apparent RA, Dec of plate centre and angle of input coord system N relative to apparent N
SCULIB_CALC_BOL_COORDS – Calculate the bolometer offsets in (apparent RA,DEC), AzEl or NA
SCULIB_CALC_CLOCKERR – Calculate start up time and error in system clock
SCULIB_CALC_FLATFIELD – calculate flat-field results for a bolometer
SCULIB_CALC_GRID – calculate the minimum rectangular grid that would contain the input jiggle pattern
SCULIB_CALC_OUTPUT_COORDS – calculate output coords of map centre and angle of output coord system N relative to apparent N
SCULIB_CALC_SKYDIP_TEMPS – Calculate all the skydip temps from raw data
SCULIB_CALC_SUB_BOLS – find the positions of bolometers belonging to a specified sub-instrument in a demodulated data array
SCULIB_CFILLB – fill a byte array with a constant
SCULIB_CFILLD – fill a double precision array with a constant
SCULIB_CFILLI – fill an integer array with a constant
SCULIB_CFILLR – fill a real array with a constant
SCULIB_CLIP_BOL – To clip a bolometer at +/- nsigma
SCULIB_COADD – coadd exposure into coadded result
SCULIB_COADD_MAPS – Average together integration and output map
SCULIB_COADD_REMOVE – remove exposure from coadded result
SCULIB_COMPRESS_DEMOD – get demodulated data for a bolometer and coadd jiggles in each integration
SCULIB_CONSTRUCT_OUT – Append a identification string to a string
SCULIB_CONVOLVE – convolve array A with array B to give result in R
SCULIB_COPY_DEMOD_SWITCH – copy a switch of demodulated data into arrays holding the switch chop data, chop variance, calibrator, cal variance and quality separately
SCULIB_COPY_GOOD – Go through data set and throw away bad values
SCULIB_COPYB – copy one byte array to another
SCULIB_COPYD – copy one double precision array to another
SCULIB_COPYI – copy one integer array to another
SCULIB_COPYR – copy one real array to another
SCULIB_CORRECT_EXTINCTION – correct bolometers for sky opacity
SCULIB_COVSRT – Numerical Recipes in Fortran routine called by SCULIB_MRQMIN
SCULIB_CROSSTALK – crosstalk measurements
SCULIB_DAY – returns date and time as day number since 1st Jan
SCULIB_DECODE_ANGLE – convert angle string to double precision angle
SCULIB_DECODE_COMPONENT – This routine decodes the value of a single component in a SCUBA data-spec
SCULIB_DECODE_FILTER – decode filter name into names and wavelengths of filters in front of each instrument section in use
SCULIB_SPLIT_DECODE_REBIN_LINE – Splits a string into a filename and parameters for REBIN
SCULIB_DECODE_SPEC – Decode SCUBA-style data specifications
SCULIB_DIV_CALIBRATOR – divides the calibrator signal into the chop signal
SCULIB_DIV_CALIBRATOR_2 – divides the mean of the calibrator signal into the chop signal
SCULIB_EXTRACT_2DIM_B – To extract the second dimension (at a given X) from a 2d array
SCULIB_EXTRACT_2DIM_D – To extract the second dimension (at a given X) from a 2d array
SCULIB_EXTRACT_2DIM_R – To extract the second dimension (at a given X) from a 2d array
SCULIB_EXTRACT_BOL – To extract a single bolometer from a data array
SCULIB_FIND_INT – Return the indices of the start and end of the specified integration
SCULIB_FIND_SWITCH – find the start and end indices of a switch in the demodulated data array
SCULIB_FIT_2D_GAUSSIAN – fit a 2D Gaussian to an image
SCULIB_FIT_2D_PARABOLA – fit 2d parabola to data
SCULIB_FIT_D2XISQ_DAJ2 – calculate second differential of chi-squared with respect to a fit parameter
SCULIB_FIT_D2XISQ_DAJK – calculate differential of chi-squared with respect to fit parameters J and K
SCULIB_FIT_DXISQ_DAJ – calculate gradient in chi-squared with ‘a’
SCULIB_FIT_FUNCTION – routine to fit a general function to data
SCULIB_FIT_MAKEALPHA – calculates alpha matrix for SCULIB_FIT_FUNCTION
SCULIB_FIT_MAKEBETA – calculates beta matrix for SCULIB_FIT_FUNCTION
SCULIB_FIT_MULT – matrix vector multiplication routine
SCULIB_FIT_PLANE – fits a plane to some x, y, z data
SCULIB_FIT_SKYDIP – Fit the skydip data
SCULIB_FIX_SCAN_V10 – Correct scan positions for version 1.0 data from SCUCD
SCULIB_FLATFIELD_DATA – flatfield the data in an array
SCULIB_FLATFIELD_SEQUENCE – get bolometer measurement sequence for FLATFIELD
SCULIB_FREE – release virtual memory
SCULIB_GAUSS_WTINIT – Generate a weighting function for rebinning
SCULIB_GAUSSIAN_XISQ – calculate chi-squared of Gaussian fit
SCULIB_GAUSSJ – Numerical Recipes in Fortran routine for solution of linear equations by Gauss-Jordan elimination
SCULIB_GENSYCONFN – Generate a convolution function which will set to zero the spatial frequencies with no signal in Dual Beam maps
SCULIB_GET_BOL_DESC – Get the bolometer description arrays and check their dimensions
SCULIB_GET_DEM_PNTR – Get the DEM_PNTR array and its dimensions
SCULIB_GET_FILENAME – Find the filename associated with a parameter
SCULIB_GET_FITS_C – get the value of specified FITS character keyword
SCULIB_GET_FITS_D – get the value of specified FITS double keyword
SCULIB_GET_FITS_I – get the value of specified FITS integer keyword
SCULIB_GET_FITS_L – get the value of specified FITS logical keyword
SCULIB_GET_FITS_R – get the value of specified FITS real keyword
SCULIB_GET_JIGGLE – Get the jiggle parameters
SCULIB_GET_LST_STRT – Get the LST_STRT array and check its dimensions
SCULIB_GET_MJD – Obtain the Modified Julian date from the UT stored in FITS
SCULIB_GET_RASTER – Get the raster parameters
SCULIB_GET_SUB_BOLS – copy bolometers belonging to a particular sub- instrument from the input data array to the output
SCULIB_GET_SUB_INST – Ask for the specific sub-instrument
SCULIB_INSERT_BOL – To insert a single bolometer into a data array
SCULIB_INTEGRATE_PHOTOM_JIGGLE – integrate the jiggle map made by a bolometer during a PHOTOM observation
SCULIB_INVERT_MATRIX – invert a square matrix
SCULIB_J_THEORETICAL – calculate the theoretical sky brightness temperature
SCULIB_JNU – Calculate the Rayleigh-Jeans corrected brightness temperature
SCULIB_LINEAR_WTINIT – Set up weighting function for linear rebinning
SCULIB_LST – returns LST in radians
SCULIB_MALLOC – get virtual memory
SCULIB_MAP_ALLAN_VARIANCE – incorporate latest set of MAP demodulated data into Allan variance
SCULIB_MASK_DATA – Set a data array from a SCUBA section
SCULIB_MJD_TO_DATEOBS – Convert a modified julian date to the correct FITS DATE-OBS format
SCULIB_MRQCOF – called by SCULIB_MRQMIN
SCULIB_MRQMIN – Levenberg-Marquardt method non-linear least-squares fit from Numerical Recipes in FORTRAN
SCULIB_MULCAD – multiply double precision array by a constant double
SCULIB_MULCAR – multiply real array by a constant
SCULIB_MULTARE – multiplies 2 real arrays with optional error and quality handling
SCULIB_NFILLI – fill an integer array with its indices
SCULIB_NFILLR – fill a real array with its indices
SCULIB_NOISE_MEAN – Calculate statistics of NOISE data from raw demodulated data
SCULIB_PAR_GET0? – Wrapper for the standard PAR_GET0 routines
SCULIB_PHOTOM_BOLSELECT – select photometers for a PHOTOM observation
SCULIB_POWER2 – Calculate next highest power of 2
SCULIB_PUT_FITS_C – write a string to a FITS item specified
SCULIB_PUT_FITS_D – write a DOUBLE to a FITS item specified
SCULIB_PUT_FITS_I – write an integer to a FITS item specified
SCULIB_RAD2STRING – Translate an angle or time in radians to a nicely formatted string
SCULIB_RANGED – Finds the maximum and minimum values in a double precision array
SCULIB_READ_JIGGLE – read a jiggle pattern
SCULIB_READ_NUMBERS – routine to read numbers from an ASCII file
SCULIB_READ_SKY – read sky parameters from a named file
SCULIB_READ_TAUZ – read sky zenith optical depths from a named file
SCULIB_READBOLS – read bolometer data from a name file
SCULIB_REDUCE_SWITCH – reduce the demodulated data from the switches of an exposure into the exposure result
SCULIB_REMOVE_DEMOD_INT – remove demodulated data for one sub-instrument in an integration from a coadded result
SCULIB_REMOVE_LINEAR_BASELINE – Remove linear baseline from each exposure
SCULIB_REMOVE_OPACITY – remove sky opacity from demodulated data
SCULIB_REM_SKY – To remove sky background from SCUBA data
SCULIB_REWRITE_FITS_C – rewrite the value of specified FITS character keyword
SCULIB_REWRITE_FITS_I – rewrite the value of specified FITS integer keyword
SCULIB_REWRITE_FITS_R – rewrite the value of specified FITS real keyword
SCULIB_SCAN_2_RD – Calculate the apparent RA/Dec of a scan
SCULIB_SCAN_APPARENT_TP_2_AZNA – Calculate NAsmyth and AZel coordinates for SCAN/MAP
SCULIB_SEARCH_DATADIR – Open an NDF using the parameter system whilst searching DATADIR
SCULIB_SET_DATA – set data to a real value given a byte mask
SCULIB_SET_DATA_BIT – set a bit in data given a byte mask
SCULIB_SET_DATA_UB – set data to a byte value given a byte mask
SCULIB_SET_QUAL – set quality bits in a subset of a quality array
SCULIB_SET_QUALITY – set quality bits in a subset of a quality array
SCULIB_SET_USER – set the USER array used by SCULIB_SKYFUNC_1
SCULIB_SINC – Calculate SIN(PI∗X)/(PI∗X)
SCULIB_SKYCON_1 – routine to calculate non-linear constraints for NAG non-linear fitting routine E04UPF when fitting ETA_TEL, B and TAU in sky-dip analysis
SCULIB_SKYDIP_ALLAN_VARIANCE – incorporate latest set of SKYDIP data samples into Allan variance
SCULIB_SKYDIP_BOLS – returns bolometers to be measured in a SKYDIP
SCULIB_SKYDIP_TEMPERATURES – derive sky temperatures from chopper-wheel data
SCULIB_SKYDIP_VAR – calculate variance of skydip data points
SCULIB_SKYDIP_XISQ – calculate chi-square of sky fit
SCULIB_SKYFUNC – Return the value of the skydip function
SCULIB_SKYFUNC_1 – calculate f(x) and its Jacobian for NAG non-linear fitting routine E04UPF when fitting ETA_TEL, B and TAU in SKYDIP analysis
SCULIB_SKYFUNC_2 – calculate f(x) and its Jacobian for NAG non-linear fitting routine E04UPF when fitting B and TAU in sky-dip analysis
SCULIB_SKYFUNCD – Return the value of the partial derivatives of the skydip function
SCULIB_SPLINE_PDA_IDBVIP – Fit a surface to an irregular grid using the PDA function PDA_IDBVIP
SCULIB_SPLINE_PDA_IDSFFT – Fit a surface to an irregular grid using the PDA function PDA_IDSFFT
SCULIB_SPLINE_PDA_SURFIT – Fit a surface to an irregular grid using the PDA function PDA_SURFIT
SCULIB_SPLINE_REGRID – Regrid supplied data onto a rectangular grid using a spline
SCULIB_SPLINE_REGRID_1 – Calculate the areas of the output map that contain good data points
SCULIB_SPLIT_FILE_SPEC – Splits a string into a filename and SCUBA section
SCULIB_SQROOTR – take the square root of a real array
SCULIB_STANDARD_APPARENT – convert apparent RA,Decs from one date to another
SCULIB_STATD – To calculate mean and standard deviation of a DOUBLE array
SCULIB_STATR – To calculate mean and standard deviation of a REAL array
SCULIB_SUB_TAUZ – get tauz appropriate for each sub-instrument
SCULIB_SUBARE – subtract one real array from another into a third
SCULIB_SUMAD – sum the elements of a double precision array
SCULIB_TIDY_LINE – remove tabs and comments from a line
SCULIB_UNPACK – routine to unpack compressed resampled map
SCULIB_UNPACK_CHOPSCAN – routine to unpack compressed chop-scan data
SCULIB_UNPACK_JIGGLE – unpack demodulated data onto 2-d map
SCULIB_UNPACK_JIGGLE_SEPARATES – unpack jiggle data arrays onto 2-d map
SCULIB_UT1 – returns UT1 as a modified Julian day
SCULIB_WTFN_REGRID – Regrid supplied data onto a rectangular grid
SCULIB_WTFN_REGRID_1 – Calculate the coverage of the data and total weight per cell
SCULIB_WTFN_REGRID_2 – Perform convolution
SCULIB_WTFN_REGRID_3 – Sets up the ’guard ring’ of bolometers outside the data

Glossary

chopping The secondary mirror is continuously moved on and off source at approximately 7 Hz in order to remove the sky to zeroth order. This is done in addition to standard jiggling.
demodulation Removal of the chop signal by the transputers. At this time, the raw data can not be accessed, only the demodulated data are stored.
exposure An exposure is the result from a complete set of switches. For example, in a JIGGLE/MAP or PHOTOM observation where the telescope is nodding the source between left and right beams, the data from each nod position is a switch and the reduced result ‘left switch’ - ‘right switch’ say, is an exposure. In a SCAN/MAP observation there is no beam switching so, in this case, an exposure is the same as a switch.
integration An integration means different things for different observations.

For one of the mapping modes it means the data from one fully-sampled coverage of the map area. In a JIGGLE/MAP, where full sampling is achieved by jiggling the secondary mirror, an integration is generally the results from one pass through the complete jiggle pattern. An integration is made up of one or more exposures.

Similarly, an integration for a SCAN/MAP observation is made up of data from the raster scans that cover the map area once.

For PHOTOM observations an integration is usually the average of a 9 point mini-jiggle.

For a SKYDIP observation, an integration is the data from a single revolution of the sector chopper in front of the cryostat window.

jiggle In order to sample an image fully the secondary mirror is moved once a second (whilst chopping) to move the position of the array on the sky; this is called ‘jiggling.’ There are a complete set of jiggle positions for each integration. A PHOTOM observation can also jiggle in order to correct for seeing effects.
measurement A measurement is a group of integrations. Most MAP or PHOTOM observations will consist of only one measurement.

A FOCUS or ALIGN observation consists of five measurements (one for each secondary mirror position). A SKYDIP observation consists of one measurement at each elevation.

nod In order to correct for atmospheric variation the telescope is moved off-source in each exposure so that sky can be measured.
ODF The observation definition file (ODF) is a file containing a list of instructions for an observation with SCUBA.
sub-instrument SCUBA contains bolometer arrays and photometric pixels that can operate at several wavelengths simultaneously. Each of these is called a sub-instrument. They are:
switch The switch is the fundamental unit of data-taking in an observation. For example in a JIGGLE/MAP or PHOTOM observation each chunk of jiggle positions measured with the object in the beam of a telescope is a switch. Each scan across the source in a SCAN/MAP observation is also a switch.
tau (τ) Submillimetre extinction is measured using the zenith optical depth, tau or τ, this is a measure of the amount of water vapour present in the atmosphere. For a tau, τ at a given airmass, A, the attenuation due to the atmosphere is given as e−Aτ. Note that tau is wavelength dependent and that the value quoted by the Caltech Submillimetre Observatory (CSO) is the τ at 225 GHz and will therefore be different at the other wavelengths used by SCUBA