J Correcting ‘shifted’ data

Between early 1997 and 15th September 1997 there was an intermittent fault with the SCUBA data acquisition system (DAQ) that led to a particular form of data corruption. The problem is most serious in MAP/JIGGLE observations but can also affect PHOTOM. The DAQ fault was identified and fixed on 15th September 1997 and checking routines were added to the data-taking software to warn of any such data synch problems that occur in the future. Observers with data taken after that date need read no further.

Those with data taken in the problem period may remember that the SCUBA bolometer channels are collected into 9 groups of 16, with each 16 channel group being handled by a different A/D card in the DAQ. The name of each bolometer reflects the A/D card and channel number on that card which handles the signal; for example, B12 will be routed through channel 12 on card B. The 9 A/D cards have single letter identifiers running from A to I, the channel numbers on each run from 1 to 16.

Roughly once or twice a night the fault would insert a spurious number into the data stream from one or more of the A/D cards to the transputer system that does the digital demodulation. The system design is such that until the system was reloaded a spurious shift would be applied to the data. Thus if the fault occurred in the B card then after that point the system would see the data from that card as shifted up by one channel number; data for channel B5 would appear in channel B6, B7 in B8, etc. The end channels wrap around so that channel B16 would appear in channel B1 of the next dataset. The effect is cumulative, so that if 2 faults occurred on the B card then the data would then be shifted by 2 channels.

If the fault occurs for a card whose bolometers are measuring source signal then the image of the source will be reconstructed incorrectly by both the real-time display and the SURF package. In fact, the problem was first noticed when jiggle map images of Uranus showed an apparent double source.

Faulty data can be patched up using the scushift utility. The difficulty is in finding when and where in your data the problem has occurred since, without a high signal-to-noise signal to judge by, you cannot go on the appearance of the final image.

The situation is saved by SCUBA’s internal calibrator. During all jiggle-type observations a sinusoidal signal from a source inside the cryostat is superimposed on the astronomical data. The digital demodulation recovers the amplitudes of both the calibrator and astronomical signals. The pattern of the calibrator signal from the SCUBA bolometers forms a signature that is constant over long periods and can be used to detect shifts in the data.

To illustrate this point there are four data files distributed with this package.16 These files contain the calibrator signal for this period for the 450- and 850 μm filters17. The files are:

The ‘_map’ files are intended for comparison with JIGGLE/MAP data whereas the ‘_photom’ file is intended for use with PHOTOM data (there are two extra bolometers in this case). In addition, there is also a file containing shifted data, calsig_450_850_bp2.sdf, where the signal from the B card has been shifted by 2 channels. Fig. 17 shows this file overlaid on the correct, unshifted, calibrator signal.


pdfpict
Figure 17: The standard calibrator signal (solid line) with an overlay of a shifted calibrator signal (dashed). The A/D cards are indicated and a shift is clearly seen on card B.


Demodulated data can be checked as follows:

(1)
Extract the calibrator signal from the raw demodulated data (i.e. before running reduce_switch). The simplest way to extract the signal is to use the Kappa command manic:
  % manic
  INPUT - Input image /@calsig_450_850_map/ > 19971008_dem_0039
  Array is 3 -dimensional
  Dimensions are ( 5, 128, 384 )
  ONDIM - Dimensionality of output image /3/ > 1
  ELINE2 - Axis of the input data array that will be used to form the
   output data array /’Y’/ > Y
  YLIMITS - Window limits for the Y-axis of the input data array /[1,37]/ >
  XRANGE - Range for summation over the X-axis of the input data array /[1,5]/ > [3,3]
  ZRANGE - Range for summation over the Z-axis of the input data array /[1,384]/ >
  OUTPUT - Output image > calsig
  LOOP - Produce another output IMAGE structure and data array? /NO/ >

The dimensions of the data array in the demodulated file should be [5, n_bols, n_jiggles] (§E.3), where n_bols is the total number of bolometers measured (128 for both short and long arrays), and n_jiggles is the total number of jiggle positions measured in the observation.

In summary, manic should be given the following parameters: ONDIM=1, ELINE2=‘Y’, YLIMITS=default, XRANGE=[3,3] and ZRANGE=default (where default is the default value suggested for the parameter)

Alternatively, it should be possible to use ndfcopy to extract the NDF section, using the TRIM parameter to reduce the resultant section to 2 dimensions.

  % ndfcopy ’19971008_dem_0039(3,,)’  TRIM TRIMWCS
(2)
Normalise the calibrator signal by using cdiv to divide the result from manic by n_jiggles (384 in this example):
  % cdiv
  IN - Input NDF data structure /@calsig/ >
  SCALAR - Division constant /1280/ > 384
  OUT - Output NDF > calsig_div
(3)
Plot the standard calibrator and overlay the calibrator signal derived above.
  % linplot ${SURF_DIR}/calsig_450_850_map
  % linplot calsig_div noclear lincol=(some colour)

where ‘some colour’ is a different colour to that used to display the first calibrator signal.

(4)
Once you have identified a data shift this must be corrected with the scushift command. Since the shift is usually caused by adding extra bytes to the data stream, a negative shift must be applied to correct the problem.
  % scushift
  Input NDF (no .sdf): r95
  Which A to D card (a single letter): B
  Using the NDF r95...
  Card 2 starts and ends at positions 16 and 31
  Enter required shift (<14): -2
  Before: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13
  14 15 16 1...<cut>
  Bolchan: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 15 16 1 2 3 4 5 6 7 8 9 10
  11 12 13 14 1...<cut>

scushift is very verbose! The ‘before’ and ’bolchan’ entries simply tell the user the form of the correction used. If you look carefully you will see that the 17th number in the list has changed from a 1 to a 15 indicating that the shift was successful.

This command works ‘inplace’ and, in fact, will not run on the raw data file; instead it should be run on the file produced by reduce_switch or flatfield.

(5)
The data should now be corrected (but note that PHOTOM data will still need more work; see the note in the scushift documentation for more details).

There is one last wrinkle to the process of extracting calibrator data.. PHOTOM observations after 3rd June 1997 store signals from 2 channels in addition to the arrays, if the arrays were being used. Thus n_bols for a PHOTOM demodulated data array will be 130 rather then 128. In this case you should compare the calibrator signature with that in the file calsig_450_850_photom.sdf.

16The files can be found in $SURF_DIR/

17for the calibrator signal prior to April 1997 or for different filters please contact your SCUBA support scientist for more advice