Despike data scan by detecting points more than nsigma from running
2-point mean SURFLIB_DIFF_DESPIKE
First, for each bolometer used a pass is made through each scan calculating for each point:-
| (3) |
Values of ’diff’ for the first and last points in the scan are calculated in a similar way but subtracting the mean of points 2 and 3 and points n-1 and n-2 respectively.
The mean and standard deviation of ’diff’ are calculated by coadding the 10 points at each end of the scan where, hopefully, there is no source emission. Spikes in these regions are handled by removing points from the coadd that lie further than 3 sigma from the mean, then redoing the calculation recursively until no further points need be removed.
The first criterion for a spike is that it’s ’diff’ value should be further from the mean of ’diff’ by NSIGMA times the sigma derived from the endpoints.
The problem with this simple approach is that bright sources in the scan themselves lead to excursions in ’diff’ that can be wrongly identified as spikes. To prevent this happening a second criterion is used. In this the scan values are convolved with a 3 sample wide box so that each ’box’ point is the average of the point itself and the points on either side of it. ’Box’ is expected to increase faster for real sources than for spikes because in them the increase will be spread over all 3 averaged points rather than just 1.
The second criterion for a spike is met, therefore, if a point’s ’diff’ is further from the ’diff’ mean than the value of ’box’ at that point.
Fixed-up values for points that have identified as spikes are calculated by interpolating between the closest healthy points on either side.
The second spike criterion also means unfortunately that the technique is less sensitive to spikes on bright sources than elsewhere. In addition, it is still possible to clip bright sources if too low a value for NSIGMA is used. It is recommended to run despike several times with different values of NSIGMA. Begin with NSIGMA=5, look at the result to see how effective despiking has been, then repeat the process with NSIGMA=4.5, 4.0 etc. until you start to clip source information.