REDUCE_SCIENCE_GRADIENT

Reduces an ACSIS science observation using advanced algorithms

Description:

This recipe is used for advanced generic ACSIS data processing. It has not been tuned for any specific type of data.

This recipe first creates a spatial cube from the raw time-series data. Then, working on the raw time-series data, it subtracts a median time-series signal, thresholds the data, then trims the ends of the frequency range to remove high-noise regions. There is optional masking of noise spikes. Receptors with non-linear baselines and spectra affected by transient high-frequency noise may be rejected.

After the time-series manipulation has been done to every member of the current group, every member is run through MAKECUBE to create a group spatial cube. This cube then has its baseline removed through a smoothing process, and moments maps are created.

A baseline mask formed from the group cube is run through UNMAKECUBE to form baseline masks for the input time-series data, which are then baselined. The baselined time-series data are then run through MAKECUBE to create observation cubes, from which moments maps are created.

Notes:

Available Parameters :

The following parameters can be set via the -recpars option:
ALIGN_SIDE_BAND
Whether to enable or disable the alignment of data taken through different side bands when combining them to create spectral cubes. To combine such data, this parameter should be set true (1) to switch on the AlignSideBand WCS attribute. However, this is incompatible with some early ACSIS data, where various changes to some WCS attributes subvert the combination. Should reductions fail with " No usable spectral channels found" , reduce the two side bands independently. The default is not not to align sidebands, but ‘raw data may have had AlignSideBand enabled from earlier processing (where the default was to align). Likewise data taken on different epochs with the same sideband should not have AlignSideBand switched on. [0]
BASELINE_EDGES
Percentage of the full range to fit on either edge of the spectra for baselining purposes. If set to a non-positive value and BASELINE_REGIONS is undefined, then the baseline is derived after smoothing and automatic emission detection. If assigned a negative value, BASELINE_REGIONS, if it is defined, will be used instead to specify where to determine the baseline. [undef]
BASELINE_EMISSION_CLIP
This is a comma-separated list of standard deviations factors for progressive clipping of outlying binned (see BASELINE_NUMBIN) residuals to an initial linear fit to the baseline. This is used to determine the fitting ranges automatically. Its purpose is to exclude features that are not part of the trends. Pixels are rejected at the ith clipping cycle if they lie beyond plus or minus BASELINE_EMISSION_CLIP(i) times the dispersion about the median of the remaining good pixels. Thus lower clipping factors will reject more pixels. The normal approach is to start low and progressively increase the clipping factors, as the dispersion decreases after the exclusion of features. Between one and five values may be supplied. The minimum value is 1.0. If undefined, the default for MFITTREND s CLIP parameter is used, which is fine in most cases. Where the emission is intense and extends over a substantial fraction of the spectrum, harsher clipping is needed to avoid biasing the fits. [undef]
BASELINE_LINEARITY
If set to true (1) receptors with mostly or all non-linear baselines are excluded from the reduced products. [1]
BASELINE_LINEARITY_CLIP
This is used to reject receptors that have non-linear baselines. It is the maximum number of standard deviations above the median rms deviations for which a detector s non-linearity is regarded as acceptable. The minimum allowed is 2. A comma-separated list will perform iterative sigma clipping of outliers, but standard deviations in the list should not decrease. [" 2.0,2.3,3.0" ]
BASELINE_LINEARITY_LINEWIDTH
This is used to reject receptors that have transient or mostly non-linear baselines. It specifies the location of spectral-line emission or the regions to analyse for bad baselines. Allowed values are:
  • " auto" , which requests that the emission be found automatically;

  • " base" meaning test the portions of the spectrum defined by the BASELINE_REGIONS recipe parameter; or

  • it is the extent(s) of the source spectral line(s) measured in km/s, supplied in a comma-separated list. For this last option, each range may be given as bounds separated by a colon; or as a single value being the width about zero. For instance " -20:50" would excise the region -20 to +50 km/s, and " 30" would exclude the -15 to +15 km/s range. [" auto" ]

BASELINE_LINEARITY_MINRMS
This is used to retain receptors that have noisy or slightly non-linear baselines, or transient bad baselines (cf. LOWFREQ_INTERFERENCE). The parameter is the minimum rms deviation from linearity, measured in antenna temperature, for a receptor to be flagged as bad. The non-linearity identification intercompares the receptors and can reject an outlier that in practice is not a bad receptor; it is just worse than the other receptors in an observation. This parameter sets an absolute lower limit to prevent such receptors from being excluded. Values between 0.05 and 0.2 are normal. Most good receptors will be in 0.02 to 0.05 range. [0.1]
BASELINE_LINEARITY_SCALELENGTH
This is used to reject receptors that have non-linear baselines. It is the smoothing scale length in whole pixels. Features narrower than this are filtered out during the background-level determination. It should be should be odd (if an even value is supplied, the next higher odd value will be used) and sufficiently large to remove the noise while not removing the low-frequency patterns in the spectra. The minimum allowed is 51. It is also used to detect transient non-linear baselines (cf. LOWFREQ_INTERFERENCE). [101]
BASELINE_METHOD
This specifies how to define the baseline region. Currently only " auto" is recognised. This requests the automated mode where the emission is detected and masked before baseline fitting. If undefined or not " auto" , then BASELINE_EDGES or BASELINE_REGIONS (q.v.) will be used.
BASELINE_NUMBIN
The number of smoothing bins to used for the baseline determination and hence the emission masking. The default lets MFITTREND choose (currently 32 bins), and is normally sufficient for narrow lines. For line forests, more resolution is needed so as not to include emission in the majority of bins, and so a value that will provide a few bins across the a line s width is better, typically 128, which is the default if the LINEFOREST_BASELINE recipe parameter is true. []
BASELINE_ORDER
The polynomial order to use when baselining cubes. [1]
BASELINE_REGIONS
A comma-separated list of velocity ranges each in the format v1:v2, from where the baseline should be estimated. It is countermanded should BASELINE_EDGES be defined and non-negative. These can also be used to define where to test baseline linearity if BASELINE_LINEARITY_LINEWIDTH is set to " base" . [undef]
CHUNKSIZE
The maximum sum of file sizes in megabytes of files to process simultaneously in MAKECUBE to avoid a timeout. The choice is affected by processor speed and memory. The minimum allowed value is 100. [5120]
CREATE_MOMENTS_USING_SNR
If set to true (1), moments maps will be created using a signal-to-noise map to find emission regions. This could be useful when observations were taken under differing sky conditions and thus have different noise levels. [0]
CUBE_MAXSIZE
The maximum size, in megabytes, of the output cubes. This value does not include extra information such as variance or weight arrays, FITS headers, or any other NDF extensions. [512]
CUBE_WCS
The coordinate system to regrid the cubes to. If undefined, the system is determined from the data. [undef]
DESPIKE
If set to 1 (true) despiking of spectra is enabled. [0]
DESPIKE_BOX
The size, in pixels, of the box used to both find the " background" and for cleaning spikes. This box should be slightly wider than the widest expected spike. Making this parameter too large will result in signal being identified as a spike and thus masked out. [7]
DESPIKE_CLIP
The clip standard deviations to use when finding spikes in the background-subtracted RMS spectrum. Multiple values result in multiple clip levels. A single clip level should be given verbatim, (e.g. 3). If supplying more than one level, enclose comma-separated levels within square brackets (e.g. [3,3,5]). [ [3,5] ]
DESPIKE_PER_DETECTOR
Whether or not to treat each detector independently during despiking. If a spike is not seen in all detectors, consider setting this value to 1 (for true). [0]
FINAL_LOWER_VELOCITY
Set a lower velocity over which the final products, such as the reduced and binned spectral cubes, and noise and rms images, are to be created. Unlike RESTRICT_LOWER_VELOCITY, it permits the full baselines to be used during processing, yet greatly reduces the storage requirements of the final products by retaining only where the astronomical signals reside. It is typically used in conjunction with FINAL_UPPER_VELOCITY. If undefined, there is no lower limit. If FINAL_UPPER_VELOCITY is also undefined, the full velocity range, less trimming of the noisy ends, is used. [undef]
FINAL_UPPER_VELOCITY
Set an upper velocity over which the final products, such as the reduced and binned spectral cubes, and noise and rms images, are to be created. Unlike RESTRICT_UPPER_VELOCITY, it permits the full baselines to be used during processing, yet greatly reduces the storage requirements of the final products by retaining only where the astronomical signals reside. It is typically used in conjunction with FINAL_LOWER_VELOCITY. If undefined, there is no upper limit. If FINAL_LOWER_VELOCITY is also undefined, the full velocity range, less trimming of the noisy ends, is used. [undef]
FLATFIELD
Whether or not to perform flat-fielding. [0]
FLAT_LOWER_VELOCITY
The requested lower velocity for the flat-field estimations using the sum or ratio methods. It should be less than FLAT_LOWER_VELOCITY. [undef]
FLAT_METHOD
When flat-fielding is required (cf. FLATFIELD parameter) this selects the method used to derive the relative gains between receptors. The allowed selection comprises ratio , which finds the histogram peaks of the ratio of voxel values; sum , which finds the integrated flux; and index , which searches and applies a calibration index of nightly flat-field ratios. The ratio method ought to work well using all the data, but for some data, especially early observations, it has broken down as the histogram mode is biased towards zero by noise and possible non-linearity effects. The sum method currently assumes that every receptor is sampling the same signal, which is only approximately true. [ sum ]
FLAT_UPPER_VELOCITY
The requested upper velocity for the flat-field estimations using the the sum or ratio methods. It should be greater than FLAT_LOWER_VELOCITY. [undef]
FRACTION_BAD
The maximum fraction of bad values permitted in a receptor (or receptor s subband for a hybrid observation) permitted before the a receptor is deemed to be bad. It must lie between 0.1 and 1.0 otherwise the default fraction is substituted. [0.9]
FREQUENCY_SMOOTH
The number of channels to smooth in the frequency axis when smoothing to determine baselines. This number should be small ( 10) for narrow-line observations and large ( 25) for broad-line observations. [25]
HIGHFREQ_INTERFERENCE
If set to true (1) the spectra for each receptor are analysed to detect high-frequency interference noise, and those spectra deemed too noisy are excluded from the reduced products. [1]
HIGHFREQ_INTERFERENCE_EDGE_CLIP
This is used to reject spectra with high-frequency noise. It is the standard deviation to clip the summed-edginess profile iteratively in order to measure the mean and standard deviation of the profile unaffected by bad spectra. A comma-separated list will perform iterative sigma clipping of outliers, but standard deviations in the list should not decrease. [" 2.0,2.0,2.5,3.0" ]
HIGHFREQ_INTERFERENCE_THRESH_CLIP
This is used to reject spectra with high-frequency noise. This is the number of standard deviations at which to threshold the noise profile above its median level. [4.0]
HIGHFREQ_RINGING
Whether or not to test for high-frequency ringing in the spectra. This is where a band of spectra in the time series have the same oscillation frequency and origin with smoothly varying amplitude over time. The amplitude is an order of magnitude or more lower than the regular high-frequency interference, but because it extends over tens to over 200 spectra, its affect can be as potent. Even if set to 1 (true), at least HIGHFREQ_RINGING_MIN_SPECTRA spectra are required to give a sufficient baseline against which to detect spectra with ringing. The HIGHFREQ_INTERFERENCE parameter must be true to apply this filter. [0]
HIGHFREQ_RINGING_MIN_SPECTRA
Minimum number of good spectra for ringing filtering to be attempted. See HIGHFREQ_RINGING. The filter needs to be able to discriminate between the normal unaffected spectra from those with ringing. The value should be at least a few times larger than the number of affected spectra. Hence there is a minimum allowed value of 100. The default is an empirical guess; for the worst cases it will be too small. If there are insufficient spectra the filtering may still work to some degree. [400]
LOWFREQ_INTERFERENCE
If set to true (1) the spectra for each receptor are analysed to detect low-frequency interference ripples or bad baselines, and those spectra deemed too deviant from linearity are excluded from the reduced products. [1]
LOWFREQ_INTERFERENCE_EDGE_CLIP
This is used to reject spectra with low-frequency interference. It is the standard deviation to clip the profile of summed-deviations from linearity iteratively in order to measure the mean and standard deviation of the profile unaffected by bad spectra. A comma-separated list will perform iterative sigma clipping of outliers, but standard deviations in the list should not decrease. [" 2.0,2.0,2.5,3.0" ]
LOW_FREQ_INTERFERENCE_THRESH_CLIP
This is used to reject spectra with low-frequency interference. This is the number of standard deviations at which to threshold the non-linearity profile above its median level. [3.0]
LV_AXIS
The axis to collapse in the cube to form the LV image. Can be the axis s index or its generic " skylat" or " skylon" . [" skylat" ]
LV_ESTIMATOR
The statistic to use to collapse the spatial axis to form the LV image. See the KAPPA:COLLAPSE:ESTIMATOR documentation for a list of allowed statistics. [" mean" ]
LV_IMAGE
A longitude-velocity map is made from the reduced group cube, if this parameter is set to true (1). The longitude here carries its generic meaning, so it could equally well be right ascension or galactic longitude; the actual axis derives from the chosen co-ordinate system (see CUBE_WCS). [undef]
MOMENTS
A comma-separated list of moments maps to create. [" integ,iwc" ]
MOMENTS_LOWER_VELOCITY
Set a lower velocity over which the moments maps are to be created. It is typically used in conjunction with MOMENTS_UPPER_VELOCITY. If undefined, the full velocity range, less trimming of the noisy ends, is used. [undef]
MOMENTS_UPPER_VELOCITY
Set an upper velocity over which the moments maps are to be created. It is typically used in conjunction with MOMENTS_LOWER_VELOCITY. If undefined, the full velocity range, less trimming of the noisy ends, is used. [undef]
PIXEL_SCALE
Pixel scale, in arcseconds, of cubes. If undefined it is determined from the data. [undef]
REBIN
A comma-separated list of velocity resolutions to rebin the final cube to. If undefined, the observed resolution is used. [undef]
RESTRICT_LOWER_VELOCITY
Trim all data to this lower velocity. It is typically used in conjunction with RESTRICT_UPPER_VELOCITY. If undefined, the full velocity range, less trimming of the noisy ends, is used. [undef]
RESTRICT_UPPER_VELOCITY
Trim all data to this upper velocity. It is typically used in conjunction with RESTRICT_LOWER_VELOCITY. If undefined, the full velocity range, less trimming of the noisy ends, is used. [undef]
SPATIAL_SMOOTH
The number of pixels to smooth in both spatial axes when smoothing to determine baselines. [3]
SPREAD_METHOD
The method to use when spreading each input pixel value out between a group of neighbouring output pixels when regridding cubes. See the SPREAD parameter in SMURF/MAKECUBE for available spreading methods. [" nearest" ]
SPREAD_WIDTH
The number of arcseconds on either side of the output position which are to receive contributions from the input pixel. See the PARAMS parameter in SMURF/MAKECUBE for more information. [0]
SPREAD_FWHM_OR_ZERO
Depending on the spreading method, this parameter controls the number of arcseconds at which the envelope of the spreading function goes to zero, or the full-width at half-maximum for the Gaussian envelope. See the PARAMS parameter in SMURF/MAKECUBE for more information. [undef]
TILE
Whether or not to make tiled spectral cubes. A true value (1) performs tiling so as to restrict the data-processing resource requirements. Such tiled cubes abut each other in pixel co-ordinates and may be pasted together to form the complete spectral cube. [1]
TRIM_MINIMUM_OVERLAP
The minimum number of desired channels that should overlap after trimming hybrid-mode observations. If the number of overlapping channels is fewer than this, then the fixed number of channels will be trimmed according to the TRIM_PERCENTAGE, TRIM_PERCENTAGE_LOWER, and TRIM_PERCENTAGE_UPPER parameters. [10]
TRIM_PERCENTAGE_LOWER
The percentage of the total frequency range to trim from the lower end of the frequency range. For example, if a cube has 1024 frequency channels, and the percentage to trim is 10%, then 102 channels will be trimmed from the lower end. If it and TRIM_PERCENTAGE are undefined, the lower-end trimming defaults to 2.75% for ACSIS and 7.5% for DAS observations. [undef]
TRIM_PERCENTAGE
The percentage of the total frequency range to trim from either end. For example, if a cube has 1024 frequency channels, and the percentage to trim is 10%, then 102 channels will be trimmed from either end. This parameter only takes effect if both TRIM_PERCENTAGE_LOWER and TRIM_PERCENTAGE_UPPER are undefined. If it too is undefined, the upper-frequency trimming defaults to 2.75% for ACSIS and 7.5% for DAS observations. [undef]
TRIM_PERCENTAGE_UPPER
The percentage of the total frequency range to trim from the higher end of the frequency range. For example, if a cube has 1024 frequency channels, and the percentage to trim is 10%, then 102 channels will be trimmed from the upper end. If it and TRIM_PERCENTAGE are undefined, it defaults to 2.75% for ACSIS and 7.5% for DAS observations. [undef]
VELOCITY_BIN_FACTOR
This is an integer factor by which the spectral axis may be compressed by averaging adjacent channels. The rationale is to make the reduced spectral cubes files substantially smaller; processing much faster; and to reduce the noise so that, for example, emission features are more easily identified and masked while determining the baselines. It is intended for ACSIS modes, such as BW250, possessing high spectral resolution not warranted by the signal-to-noise. Note that this compression is applied after any filtering of high-frequency artefacts performed on adjacent channels. A typical factor is 4. There is no compression if this parameter is undefined. [undef]

Output Data

Related Recipes

REDUCE_SCIENCE_NARROWLINE.