2 SURF File Format

Surf uses that standard Starlink NDF (SUN/33) file format. This is a hierarchical format based on HDS (SUN/92) and has native support for variances, quality flags, non-linear axes and arbritrary extensions.

2.1 Raw data format

SCUBA raw data (i.e. data in the format expected by the Surf task reduce_switch) has the following general layout:

(1)

The size of the first dimension depends on the observing mode. For jiggle and noise observations it has size 5 corresponding to the demodulated data, variance on demodulated data, internal calibrator signal, variance on calibrator signal and data quality. For SCAN/MAP observations it has size 4 corresponding to the demodulated data, variance on demodulated data, internal calibrator signal and data quality (i.e. the calibrator variance is not stored for scan maps). The data quality is converted from a REAL number to a standard UBYTE quality (appendix 4): if the quality is odd, this indicates the flatfield was set to zero so bit 1 is turned on (this is also done using the flatfield task), if the quality is greater than the value specified via the SPIKE_LEVEL parameter (i.e. there were more spikes in the individual sample than the specified threshold) bit 2 is turned on. For SKYDIP observations it has a size of SCUBA__N_TEMPS (see Appendix C, usually 3) corresponding to the different temperatures measured (AMBIENT, SKY and COLD).

(2)

The second dimension corresponds to the number of bolometers observed. This should match the size of the BOL_CHAN and BOL_ADC arrays in the SCUBA extension.

(3)

The third dimension is the time axis and is equal to the total number of samples (of length EXP_TIME). This dimension must be greater than any value stored in the DEM_PNTR array. For a JIGGLE observation the size can be calculated (assuming the observation was not aborted early)

2.2 SURF Format

The reduce_switch task converts the raw SCUBA format into a more NDF-like format with proper variance and quality arrays (extracted by reduce_switch from the SCUBA data array), axes, history reporting, a title and units.

Here is an example of the reduce_switch output for a Mars pointing observation:

The data arrays are now 2 dimensional, corresponding to the bolometer number and time axis. The axis information is written so that the integration number is reflected on the Y axis. Additionally, a new extension, REDS, is created to hold the relative beam weights; for maps these are always set to 0,1,0.

A photometry observation is slightly different from a map observation in that the output data array is now 3 dimensional rather than 2:

The 3rd dimension corresponds to beam position. In most cases only beam 2 is used (the middle beam) but for two or three bolometer chopping the 1st and 3rd beams will be accessed. In general, the 1st and 3rd beams are simply scaled versions of the middle beam; the relative weighting of the beams is stored in the newly created BEAM_WT array in the REDS extension. In the above example, the middle beam is -0.5 times the left or right beams.

2.3 Extensions

The information describing the observation can be found in the NDF extensions. Surf expects to find the following extensions in data files:

2.3.1 FITS

The FITS extension is an array of 80 character strings stored in a format identical to a standard FITS header. This contains general information about the observation but no array information. Listings of full SCUBA FITS headers can be found in Appendix D. Surf only requires a subset of the full SCUBA headers. For a jiggle map these are listed in table 1.

2.3.2 SCUBA

The SCUBA extension contains information on the instrument and the structure of the data array. It has the following format (output of hdstrace):

The first 5 entries are simply the flatfield information (see Appendix E for an example flatfield file). The shape of the arrays (in this case $16\times 9$) corresponds to the number of channels on each A/D card and the number of A/D cards. These dimensions must match the values of the constants SCUBA__NUM_ADC and SCUBA__NUM_CHAN (Appendix C).

The full description of these entries follows:

BOL_CALB

These are the actual flatfield values and contain the response of each pixel relative to a reference pixel (usually the centre pixel on each array, H7 and C14). This correction is applied to the data using the flatfield task (specifically the sculib_flatfield_data routine):

$$V_{\mathrm{\text{flatfielded}}}\left(B\right)=V_{\mathrm{original}}\left(B\right)\times \mathrm{BOL<mstyle\; class="text"><mtext>\_</mtext></mstyle>CALB}\left(B\right)$$

(1)

where $B$ is the selected bolometer.

BOL_DU3

These are the X coordinates of every bolometer. For SCUBA, these are arcsec X (U3) offsets (i.e. $\mathrm{d}U3$) from the left-hand Nasmyth centre.

BOL_DU4

These are the Y coordinate of every bolometer. For SCUBA, these are arcsec Y (U4) offsets (i.e. $\mathrm{d}U4$) from the left-hand Nasmyth centre.

BOL_QUAL

This is an array of quality flags. A 0 indicates that the bolometer is ‘good’, a 1 indicates that the bolometer is ‘bad’. This array is used during flatfielding (sculib_flatfield_data)to set bit 1 in the NDF quality array.

BOL_TYPE

This array contains a textual description of the sub-instrument to which each bolometer belongs. This information is used by the extinction task to select the bolometers from the specified sub-instrument.

BOL_ADC

This array contains the A/D card numbers corresponding to each of the bolometers that were used for the observation. The order and size should match the size of the second dimension of the main data array for raw SCUBA data and the first dimension for data that has been processed by reduce_switch. The extinction task modifies this array to reflect the sub-instrument selection.

BOL_CHAN

This array contains the A/D channel numbers corresponding to each of the bolometers that were used for the observation. The order and size should match the size of the second dimension of the main data array for raw SCUBA data and the first dimension for data that has been processed by reduce_switch. The extinction task modifies this array to reflect the sub-instrument selection.

FLAT_ADC

Not used by Surf.

FLAT_CHN

Not used by Surf.

FLAT_IND

Not used by Surf.

PHOT_BB

This is only used for PHOTOM (and POLPHOT) observations. The first dimension corresponds to the number of beam positions (3 for 2 position chopping with nodding) and the second dimension corresponds to the number of sub-instruments used for the observation. The array contains the bolometer number (using the same indexing scheme as for the BOL_CHAN and BOL_ADC arrays) that is on source for each of the beam positions. For standard single bolometer photometry only the middle beam is on source and in the example shown only H7 and C14 are used for the photometry. The scuphot task uses this array to decide which bolometers to extract from the full dataset. The extinction task modifies this array to reflect the sub-instrument selection.

This information is also available in string form in the PHOT_BBF FITS keyword. For non-PHOTOM based observations this structure is 0.

DEM_PNTR

This array is used to convert from measurement, integration, exposure and switch number to sample number (i.e. the final dimension in the data array). The routines SCULIB_FIND_SWITCH and SCULIB_FIND_INT are provided to simplify this (they also return the position of the end of the switch or integration). It has dimensions of N_SWITCHES, N_EXPOSURES, N_INTEGRATIONS, and N_MEASUREMENTS.

ISTART

Not used by Surf.

NPIX

Not used by Surf.

POINTER

Not used by Surf.

2.3.3 SCUCD

The SCUCD extension contains information on telescope movements. It is slightly different for JIGGLE (MAP and PHOTOM) and SCAN observations. For a JIGGLE observation it has the following format:

and for a SCAN observation:

The full description of these entries follows:

JIGL_X and JIGL_Y

These are the X and Y coordinates of the jiggle pattern in arcsecond offsets. The coordinate frame is given in the SAM_CRDS FITS keyword. They are only used for JIGGLE observations.

LST_STRT

This is the local sidereal time of the start of every switch expressed as decimal radians. It has dimensions of N_SWITCHES, N_EXPOSURES, N_INTEGRATIONS, and N_MEASUREMENTS. The current epoch of observation (Modified Julian Date) is calculated from the UTSTART and UTDATE FITS keywords in SCULIB_GET_MJD.

RA1, DEC1, RA2 and DEC2

This is the apparent RA and Dec of the start (RA1, DEC1) and end (RA2, DEC2) of the scans in radians. The dimensions are N_SWITCHES, N_EXPOSURES, N_INTEGRATIONS, and N_MEASUREMENTS. The current epoch of observation (Modified Julian Date) is calculated from the UTSTART and UTDATE FITS keywords in SCULIB_GET_MJD.

WPLATE

For polarimetry observations (POLPHOT or POLPMAP) this array contains the waveplate angle for each measurement (where a measurement is defined as a move of the waveplate). It is used by the SURFLIB_FILL_WPLATE routine.

2.3.4 REDS

The REDS¹ extension is created by SURF as a general purpose repository of information that should be passed between SURF tasks.

It can contain the following entries:

BEAM_WT

The relative weight of the beams. Only used for PHOTOM observations. This is created by reduce_switch.

SKY

An NDF containing the estimate of the sky fluctuations. This is usually created by calcsky and is used by remsky to remove the sky fluctuations. The size of the NDF should match the size of the time axis in the main data array.

BOLWT

Relative weight of each bolometer. Created by setbolwt and used by rebin.