3 Features of the Primitives

Primitives are the Perl scripts which actually call the applications to do most of the data processing. All of the spectroscopy recipes are, in principle, independent of the instrument. however, some recipes are inappropriate; for example, the ?? recipe is intended for flat-field generation for NIRI, but not for other instruments.

Not all the following steps apply to all recipes. Consult the ?? to see summaries for each recipe. The steps are presented in normal order of appearance.

The main primitives pertinent to each step are listed in bracketed italics, should you wish to tailor the recipes. These are found in the $ORAC_DIR/spectroscopy tree, unless they start with general/. Note that some may be instrument-specific variants, either given explicitly or with the $<$instrument$>$ token, which means substitute the instrument name in uppercase.

While the following listed primitives form the bulk of the primitive library, there are many not listed here, mostly those for

recipe initialisation called $<$recipe$>$_HELLO_ (see Section ?? for more information), and for

recipe steering which control when to perform certain operations, called $<$recipe$>$_STEER_ or $<$recipe$>$_CONFIG_ (see Section ??). The first of these is normally left unchanged unless there is a need to add more steering parameters. Other primitives not mentioned here are tied closely with single recipes, usually to create and file calibrations.

3.1 Preparation of Single Frames

[_REDUCE_SINGLE_FRAME_, $<$instrument$>$/_INSTRUMENT_HELLO_, _SPECTROSCOPY_HELLO_]

3.1.1 Manipulation of Raw Data

The first step copies the raw data into $ORAC_DATA_OUT. For instruments whose raw data is in FITS format, this step converts the raw data into NDF. For HDS container files, these are copied over into new HDS container files.
[_MAKE_RAW_FILE_]

3.1.2 Preliminaries

There are a few operations applied to all frames. First, history recording is switched on. It is recommended to leave this enabled, since it provides a record of the processing steps of your final spectra. Otherwise the pipeline becomes something of a black box. use the Kappa command hislist to list the history records.
[_TURN_ON_HISTORY_]

The next step is to set the origin of the frame so that frame pixels retain the detector pixel indices. It then becomes possible to use a full-sized bad-pixel mask or a flat field on a subset of a detector’s pixel grid.
[_SET_ORIGIN_]

For CGS4 data taken before 2000 August 13, the slit angle in the SANGLE header referred to the slit’s physical position in the instrument, and after this date it referred to the slit’s angle on the sky. For data taken before this date, the header value internal to ORAC-DR is set to coorespond to the on-sky angle.
[CGS4/_FIX_SANGLE_HEADER_]

For UIST data taken before 2002 December 2 and Michelle, raw data units are converted from ADU per second to the UKIRT standard of total ADU per exposure.
[UIST/_DATA_UNITS_TO_ADU_, MICHELLE/_DATA_UNITS_TO_ADU_]

For Michelle there is a validation check of the waveform used, comparing the waveform name given in the headers with other metadata, and recipes issue a warning if there is an inconsistency. For lowQ and MedN2 data taken before 2004 March 09, the gratings were installed the wrong way round in the cryostat, so these data are flipped along the dispersion axis at this stage in the data reduction.
[MICHELLE/_CHECK_WAVEFORM_, MICHELLE/_FLIP_FLIPPED_GRATING_FRAMES_]

A night log is created or appended in $ORAC_DATA_OUT for each frame processed. This tabulates the main parameters of the observation having first corrected defective or undefined headers.
[_NIGHT_LOG_, $<$instrument$>$/_FIX_EXTRA_HEADERS_ for MICHELLE, CGS4, and UIST]

3.1.3 Bad pixels

The recipes apply a predetermined bad-pixel mask with the aim of removing the bulk of ‘hot’ and ‘cold’ pixels. This flags approximately 0.4% of UIST and ISAAC pixels, 0.1% of IRIS2 pixels, and 5% of Michelle’s pixels.
[_MASK_BAD_PIXELS_]

Some of the instruments (UIST, CGS4) array tests are run, typically at the start of each night. As a part of these array tests a new bad-pixel-mask is generated on-the-fly, using the predetermined one as a basis. For UIST the new bad-pixel-mask is generated from a long-exposure dark observation, typically 100s. Any pixel that is 5-$\sigma$ higher than the 3-$\sigma$ clipped mean or 1000-$\sigma$ lower than the 3-$\sigma$ clipped mean is flagged as bad. For CGS4 any pixel higher than 1700 or lower than 15 for a dark whose exposure time is longer than 80 seconds, or higher than 1500 and lower than -100 for a dark whose exposure time is 80 seconds or shorter, is flagged as bad. Creating a bad-pixel mask  The easiest way to create your own bad-pixel mask for use with the calibration system, is to run the ?? recipe on a long-exposure dark (at least 20 seconds integration). It is possible to change the symmetric $\sigma$-clipping bounds in the recipe (see primitive _MAKE_BPM_BY_SIGMA_THRESHOLDING_). You can tailor this primitive if you want more control, say to have asymmetric rejection or more sophisticated definitions.
[_MAKE_BPM_BY_SIGMA_THRESHOLDING_] For better results, use the average of long dark frames taken across two or three nights. First, produce QUICK_LOOK versions of the long-exposure dark to flatten the NDF structure or convert the FITS file. Flag all pixels that are 5 standard deviations ($\sigma$) above and below the 3-$\sigma$ clipped mean of the dark as “bad”, then multiply the resulting frame by zero so that the resulting bad-pixel mask has data values of 0 and bad only. You can choose your own thresholds. Here is an example, using data from two nights of CGS4 data and Starlink software.

Then you specify the bad-pixel mask on the command line.

UIST has its own slightly different formula; see ?? for details

[UIST/_FIND_BAD_PIXELS_, UIST/_FILE_BAD_PIXELS_, _FILE_MASK_]

3.1.4 Readnoise Variance

After the bad pixel mask has been applied, the readnoise variance is added into the VARIANCE component of the NDF. For all instruments, the readnoise value is obtained from the calibration system, having been previously calculted in the ?? recipe. Since the readnoise value is stored in electrons, it must be converted into analogue-to-digital units.

[_ADD_READNOISE_VARIANCE_, _CALCULATE_NREADS_NOISE_FACTOR_] CGS4 Readnoise  For CGS4 the readnoise from the calibration system is first divided by a factor to take multiple non-destructive reads into account:

3.1.5 Bias Subtraction

For observations not taken in non-destructive read mode, a bias frame is subtracted. The bias frame is pulled from the calibration system, having been filed using the ?? recipe.
[_SUBTRACT_BIAS_]

3.1.6 Poisson Variance

Once the readnoise variance has been added and the bias has been optionally subtracted, the variance due to Poisson noise is added. For all instruments the Poisson variance is calculated as:

where $S$ is the signal in ADU per exposure, $gain$ is the detector gain in electrons per ADU, and $N_e$ is the number of exposures per integration.

At this stage the number of pixels that are background limited is displayed. This number is simply the percentage of pixels where the Poisson noise is greater than the readnoise.
[_ADD_POISSON_VARIANCE_]

3.1.7 Chopping

In the thermal and mid-infrared regimes the sky is varying so rapidly that normal reduction methods are inappropriate. Instead sky subtraction is achieved either by frequently oscillating the secondary mirror between two beams (mid-infrared), called A and B; or by moving the telescope offsets (thermal) after a short exposure. The generic term is chopping.

Both methods produce frames with the target spectrum on different rows of the detector. The ?? and ?? recipes difference these pairs of frames so that the result has both a positive and negative spectrum, and a background close to zero. The sense of the subtraction is always the same. Orac-dr subtracts the B beam from the A beam, and the normal sequence is ABBA.
[_SUBTRACT_CHOP_]

3.1.8 Flat Fielding

Depending on the data format, this step and the subsequent step (interleaving and coadding, see Section 3.1.9

) may be swapped. If the flat frame and data frame were both taken with the same interleaving, then flat-fielding is done after interleaving. Otherwise, flat-fielding is done first. See Section ??

for information on how spectroscopic flat-fields are created.

Flat-fielding is done by a straight division of the data frame by the appropriate flat-field calibration frame.
[_DIVIDE_BY_FLAT_, _FLATFIELD_COADD_INTERLEAVE_]

3.1.9 Interleave and Coadd

In order to fully sample a spectrum and reduce the effect of bad pixels, observations are often taken at different detector positions. The detector is stepped along the spectral axis by a fractional number of pixels; for CGS4this is typically in half-pixel or third of a pixel steps. Each spectral element can be sampled more than one time, which helps increase the signal-to-noise and decrease the impact of hot or bad pixels. This sampling method is often referred to as 2$x$2 or 3$x$2 sampling. The first number refers to the number of data points taken per resolution (or the inverse of the fractional pixel step size) and the second refers to the number of times each pixel has been observed. These observations need to be interleaved to create a higher-resolution spectral image.

The interleaving is done by expanding the input frames by the reciprocal of the fractional pixel step size, then blanking out the extra columns in these expanded frames. The origins are then shifted correspondingly, and the expanded frames are coadded together using the mean to create a final spectral image.
[_INTERLEAVE_COADD_, _FLATFIELD_COADD_INTERLEAVE_]

3.1.10 Orient Image Normally

Some instruments are set up such that the spectrum on the detector runs from higher wavelength to lower wavelength as pixel value in increased, which is reverse to expectations. This step flips the image so that the shorter wavelength is to the left and longer is to the right.
[_ORIENT_IMAGE_NORMALLY_]

3.1.11 Wavelength Calibrate

Wavelength calibration is necessary so that spectral features can be identified. Orac-drcurrently only calculates a wavelength estimation based on information contained in the FITS headers. It takes the values for dispersion and central wavelength, then applies these values as a linear wavelength scale to the spectral image.

This is often not good enough for accurate wavelength calibration as most grisms and dispersers produce higher-order dispersions, so manual wavelength calibration must be performed after spectra have been extracted.
[_WAVELENGTH_CALIBRATE_BY_ESTIMATION_]

3.2 Group Formation

After the individual frames have been processed, a composite group spectral image must be formed. Most infrared spectroscopic observations are taken in object-sky pairs, so the first step in group formation is subtracting the sky frame from its corresponding object frame.

3.2.1 Sky Subtraction

To be able to do sky subtraction, Orac-drneeds to know out of a pair of frames which is the object frame and which is the sky frame. To do this Orac-drexamines the FITS headers. If the offset for both right ascension and declination are less than 0.001 arcseconds, then the frame is treated as being on-source. Otherwise, the frame is off-source and is used as a sky frame.

IRIS2does not record telescope offsets for spectroscopy mode, so this method cannot be used. Instead Orac-drexamines the aperture used. If aperture A is used then the frame is on-source and the right ascension offset is set to zero, otherwise the frame is off-source and the right ascension offset is set to 26.92 arcseconds.

ESOinstruments are different again as observations are not done in object-sky pairs. Instead they are done in equal-sized blocks of object and sky observations, such that a certain number of object observations are done, followed by an equal number of sky observations. In this case the initial frame in a group is always assumed to be on-source. As with the standard pipeline, an observation is considered to be off-source if its offsets are greater than 0.001 arcseconds.
[_PAIR_REDUCTION_STEER_]

Sky subtraction is straightforward – the sky frame is subtracted from the object frame.

For ESOinstruments the corresponding sky frame in a block is subtracted from the respective object frame in a block, such that the same position in each set is considered as an object-sky pair.
[_PAIR_REDUCTION_SUBTRACT_]

3.2.2 Group Coaddition

Group coaddition is performed by taking the average of all of the sky-subtracted pairs in the group. The header values for airmass and UT time at the end of observation are updated in the group frame.
[_PAIR_REDUCTION_COADD_TO_GROUP_]

In polarimetry mode multiple group files are created, one for each waveplate position. Airmass and UT time header values are also updated.
[_PAIR_REDUCTION_COADD_TO_GROUP_POL_]

3.3 Spectrum Extraction

3.3.1 Counting Beams

The first step in doing spectral extraction is determining the number of beams to extract. For regular object-sky observations there will be one positive beam and, depending on how large the offsets are or if nodding was done along the slit or not, zero or one negative beams. A nod is considered to be along the slit if the nod angle is within 5 degrees of the slit angle, which represents roughly 1 arcsecond over a 10 arcsecond throw. The length of the slit is not taken into account, so throws to a position off the end of the slit will still count as being along the slit, even though the spectrum will not appear on the detector.

For chopped observations there can be one or two positive beams and zero, one or two negative beams, depending on combinations of chop throw, chop angle, nod throw and nod angle. If the chop throw and nod throw are equal to within 2 arcseconds and the chop and nod are along the slit, there will be one positive beam and two negative beams. If the chop throw and nod throw are equal to within 2 arcseconds and the chop and nod are to sky, there will be two positive beams and two negative beams. If the chop is along the slit then there will be one positive beam and one negative beam. If the chop is along the slit but the offset is to sky, then there will be one positive beam and one negative beam. If the chop is to sky and the nod is along the slit, then there will be one positive beam and one negative beam. If the chop and nod are both to sky, then there will be one positive beam.

For dual-beam polarimetry observations the number of beams is as above, but doubled. For single-beam polarimetry the number of beams is as above.
[_EXTRACT_DETERMINE_NBEAMS_]

3.3.2 Finding Beams

After the number of beams to extract has been determined, it comes time to locate the beams on the detector. The spectral image first has any residual bias level removed by subtracting a multiply clipped mean, and it is then collapsed along the spectral axis to form a profile spectrum.

To find the beams, the profile spectrum is turned into a five-pixel wide image which is made up of the original profile flanked by symmetric half- and quarter-strength copies. This step is non-parametric, and can prefer faint blips over strong beams, although in practice the correct beam is found.
[_FIND_PEAKS_BY_MAKING_IMAGE]

If the number of beams found does not equal the number of beams calculated in the previous step (see Section 3.3.1

) then spectral extraction will not occur. If flux calibration is to be performed, then processing skips to division by standard (3.5), if division by standard and flux calibration is necessary.

IRIS2differs in that the entire spectral image is not collapsed to form the profile. Collapsing the entire image risks producing spurious peaks due to noisy data near the edges of the array, so a profile is formed by collapsing a region 0.05 microns short and 0.15 microns long of the central wavelength.

After the beam locations have been determined they are filed with the calibration system to be used for faint sources, if necessary.

The beam detection step described here does not modify the Group file.
[_EXTRACT_FIND_ROWS_]

3.3.3 Beam Extraction

Once the beam positions have been located, the beams can be extracted. First, an extraction window width is calculated based on the position and number of beams in the spectral image. This window is used for all beams. If there are two beams, then the half-width of the window is half the beam separation. If there are three, then the half-width of the window is half the distance between the positive beam and the first negative beam. Otherwise, the half-width is 50 pixels.

UISTdiffers from this in that if either the HK or KL grisms are used, the half-width of the extraction window is 10 pixels.

For each beam, an optimal extraction profile (Horne, 1989) is then determined over the extraction window. If requested (i.e. if a standard star is reduced), this optimal extraction profile is filed with the calibration system.

The beams are then optimally extracted using the appropriate profile.
[_EXTRACT_ALL_BEAMS_]

3.3.4 Derippling

When observations are interleaved (see Section 3.1.9

), variations in throughput, sky brightness, and other factors may cause flux levels to change between the interleaved exposures, resulting in a ripple in the interleaved spectrum. This ripple is removed by creating a ripple flat with a ripple that has the same frequency as the interleaving, i.e. if 2x2 interleaving is performed, a ripple flat with a frequency of two pixels is created. This ripple flat is divided into the extracted spectrum unless the ripple amplitude is less than 70
[_DERIPPLE_ALL_BEAMS_]

3.3.5 Beam Cross Correlation

To remove any shift in beams, possibly caused by tilted spectra, the extracted beams are cross-correlated and shifted. If the maximum value of the cross correlation funtion is less than 0.6, or if the shift is greater than 2 pixels, then the spectra are not aligned and shifted.
[_CROSS_CORR_ALL_BEAMS_]

3.3.6 Extracted Beam Coaddition

The beams have been extracted and must now be coadded. This is done by simply averaging all of the extracted beams together.
[_COADD_EXTRACTED_BEAMS_]

3.4 Signal-To-Noise Calculation

When an extracted and co-added spectrum has been determined, the signal-to-noise ratio is calculated, but only if it has a variance array associated with it.
[_CALCULATE_SNR_]

3.5 Division By Standard Star

Whether or not a spectrum has been extracted, division by a standard star spectrum can still proceed. If no spectrum has been extracted, then the standard star spectrum is extended perpendicularly to the wavelength direction to make a 2-D spectrum.

When the spectrum of the science target is divided by a standard star spectrum, a straight division is done.

3.6 Flux Calibration

After standard division is done, the spectrum is flux-calibrated. This is done by using the standard star’s V-band magnitude, spectral type, and observed waveband to determine a reference flux at the central wavelength of the observed waveband. The units of the flux-calibrated spectrum are presented in $W/m^2/\mu m$.