Appendix A
Cleaning the Raw Data

There are two ways to clean time-series data:

(1)

Run makemap and add doclean = 1 to your configuration (see Section 5.4.3).

(2)

Run sc2clean to clean the time-series data without making a map.

The rest of this appendix gives more details on using sc2clean.

sc2clean can be used to do two basic tasks in one go: concatenate data (with or without applying a flatfield); and cleaning (fix up steps and spikes, remove the means, filter, remove common-mode etc.). It uses the same configuration files as the iterative map-maker (though ignoring the map-making specific items).

In this first basic example, we just want to clean up some data enough to see whether the bolometers have been flat-fielded correctly, and more-or-less exhibit the same behaviour over time. The pre-processing or cleaning steps used by default (i.e. if “config=def” is included on the command line) are summarised in Section J.3. Note, whilst it is not recommended to run makemap in this way (i.e. without a configuration file), it is not so critical when running sc2clean.

Here files.lis can just contain a single file from a sub-array, or a subset, e.g. s8a20110417_00051_0003.sdf (the first file containing science data), s8a20110417_00051_000"[1234]" (File 1 is a noise observation with shutter closed that gets ignored, File 2 is a flatfield observation that will be used to override the flatfield stored in the subsequent Files 3 and 4 which are concatenated together, the .sdf is optional), s8a20110417_00051_000\? (Files 1 through 9), s8a20110417_00051_\* (the whole observation).

If you inspect the resulting clean.sdf in Gaia (Section 9.4) and flip through the data cube you should see all of the bolometers signals go up and down together with about the same amplitude: the hope is that for a well-behaved instrument you are mostly seeing sky noise variations that are seen with roughly the same amplitude by all bolometers.

Another common feature, if the scans are particularly long and/or fast (e.g. 1 degree across), is strong periodic signals that are correlated with the scan pattern. See Section 9.3—in particular you will want to plot az and el (the absolute azimuth and elevation), and also daz and del (the azimuth and elevation offsets from the map centre). This signal is usually azimuth-correlated due to magnetic-field pickup. It only shows up in azimuth, because the instrument is on a Nasmyth platform and therefore does not move in elevation.

Part of the reason the signals look the same is because they have been flatfielded. You can turn off flatfielding using the noflat option to sc2clean, and you should then see that all of the detector amplitudes vary.

Another very useful option is to remove the common signal observed by all of the bolometers. This may be accomplished by

This config setting causes the default values to be used for all configuration parameters except compreprocess, which is set to 1 (the default is 0). The residual signal left by this command will exhibit second-order time-varying correlated signals across the focal plane. Usually these are not very large, but in some cases some very large localized signals have been detected, particularly in the 850 $\mu$m arrays in early 2011.

Another variation on this is to accentuate the residual low-frequency noise by low-pass filtering the result. This can again be accomplished by simply adding a filter command in the config parameter, which in this case low-pass filters with a cutoff at 10 Hz:

Finally, in some cases you might just want to fit and remove polynomial baselines from the bolometers (by default only the mean is removed). This example will remove a line, but you can increase the value of order to remove higher-order polynomials

Non-default values for any of the cleaning parameter can be specified like so:

Or you can create your own customised configuration file. For instance:

The more interesting pre-processing options that may be specified are listed and described in Appendix I.