3 Features of the Primitives

Primitives are the Perl scripts which actually call the applications to do most of the data processing. All of the imaging recipes are, in principle, independent of the instrument. However, some recipes are inappropriate; for example, the NOD_CHOP family of recipes are intended for mid-IR imaging with Michelle, but not the JITTER_SELF_FLAT family. The generality comes in part from translations of relevant header information into a generic form used by Orac-dr.

Not all the following steps apply to all recipes. Consult the reference section to see summaries for each recipe. The steps are presented in normal order of appearance.

The main primitives pertinent to each step are listed in bracketed italics, should you wish to tailor the recipes. These are found in $ORAC_DIR/imaging tree, unless they start with general/ when they are located in $ORAC_DIR/general. Note the some may be instrument-specific variants, either given explicitly or with the $<$instrument$>$ token, which means substitute the instrument name in uppercase.

While those listed form the bulk of the primitives, there are many not listed here, mostly those for

recipe initialisation called $<$recipe$>$_HELLO_ (see Section 4.2.1 for more information), and for

recipe steering which control when to perform certain operations, called $<$recipe$>$_STEER_ (see Section 4.2.2). The first of these is normally left unchanged unless there is a need to add more steering parameters passed to the steering primitive. The steering primitive itself is only modified for such new steering parameters, and if the observing pattern or sequence is different from what the recipe is programmed to expect. Other primitives not mentioned here are tied closely with single recipes, usually to create and file calibrations.

3.1 Preparation of Single Frames

[_IMAGING_HELLO_, $<$instrument$>$/_INSTRUMENT_HELLO_,
_PREPARE_SINGLE_FRAME_]

3.1.1 Manipulation of Raw Data

The first stage makes an NDF, or multiple NDFs stored in an HDS container file, located in $ORAC_DATA_OUT, the actual operation depending on the instrument. In some cases like UFTI, this is dealt by the Orac-dr infrastructure, including conversion from FITS for early UFTI data.
[$<$instrument$>$/_CREATE_RAW_FRAME_]

For IRCAM the recipes are merely copying the raw NDF, because the original raw files are normally write protected.

UIST and Michelle both copy all the components of multi-NDF container file. These container files have a HEADER NDF of the observation-wide FITS headers, and NDFs of chopped beams or integrations. Recipes check that two integrations are present for a chopped observations or one integration for other modes, exiting with an error message if either test fails. A single integration NDF is merged with the HEADER NDF into a simple NDF.

3.1.2 Preliminaries

There are a few operations applied to all frames. First the raw frame may be displayed.
[general/_DISPLAY_FRAME_IMAGE_]

Recipes remove any AXIS and blank TITLE components; the latter to preserve the object name when NDFs are exported to FITS. Next they set the origin of the frame so that frame pixels retain the detector pixel indices. It then becomes possible to use a full-sized bad-pixel mask or flat field on any subset of a detector’s pixel grid.
[_REMOVE_AXES_, _REMOVE_BLANK_TITLE_, _SET_ORIGIN_]

Then recipes determine the displacements of the reference pixel with respect to the centre of the frame and stores the displacements in the referenceoffset calibration system. The reference pixel is where a star would be placed for photometry or the centre of a chopped and nodded pattern.
[_SET_REFERENCE_PIXEL_OFFSET_, _GET_FRAME_CENTRE_]

The next step is to switch on history recording. It is recommended to leave this enabled, since it provides a record of the processing steps of your final mosaics. Otherwise the pipeline becomes something of a black box. Use the Kappa command hislist to list the history records.
[_TURN_ON_HISTORY_]

For Michelle there is a validation check of the waveform used, comparing the waveform name given in the headers with other metadata, and recipes issue a warning if there is an inconsistency. Also the data range is validated to be between 25000–48000.
[MICHELLE/_CHECK_WAVEFORM_, MICHELLE/_VALIDATE_RAW_DATA_]

For UFTI there is data validation, such that a warning is issued if the clipped mean sky level is below 24 counts per second in $K$ band and 32 counts per second in $H$.
[UFTI/_CHECK_SKY_COUNTS_, UFTI/_VALIDATE_RAW_DATA_]

For UIST data taken before 2002 December 2 and Michelle, raw data units are converted from ADU per second to the UKIRT standard of total ADU per exposure.
[$<$instrument$>$/_INSTRUMENT_HELLO_, UIST/_DATA_UNITS_TO_ADU_]

A night log is created or appended in $ORAC_DATA_OUT for each frame processed. This tabulates the main parameters of the observation having first corrected defective or undefined headers.
[_NIGHT_LOG_, UIST/_NIGHT_LOG_, $<$instrument$>$/_FIX_HEADERS_ for Classic Cam, IRCAM, MICHELLE, NIRI, and UFTI]

3.1.3 Non-linearity Correction

Detector non-linearity corrections are applied to Classic Cam, IRCAM, and INGRID. For IRCAM the correction is $3.3\times 10^{-6}$ times the square of the bias-subtracted signal. For INGRID the measured-to-actual counts is given by the expression $1.0-1.2247\times 10^{-6}\ast M-7.68045\times 10^{-11}\ast M^2$, where M is the measured ADU count, although it is actually applied after the pre-exposure readout is subtracted from the post-exposure integration. For Classic Cam the correction is 1.625$\times 10^{-6}$ $\ast$ (1 $+$ overhead$/$exposure_time), where the overhead is 2 $\ast$ speed $\ast$ number of pre-exposure reads $+$ (speed $+$ readout time ) $\ast$ number of post-exposure reads. The various parameters in this expression are either read from the headers or taken verbatim from an IRAF script cited in the user manual.
[CLASSICCAM/_CORRECT_NONLINEARITY_,
IRCAM/_CORRECT_NONLINEARITY_,
INGRID/_CORRECT_NONLINEARITY_PRE_POST_]

3.1.4 Electronic Ghosting

For ISAAC there is a correction applied for electronic ghosting. The ghosts consist “of an additional signal, which, on one row, is proportional to the sum of the intensity along this row and the row 512 rows away.” according to the ISAAC Data Reduction Guide. Recipes combine the two halves adding the bottom half to the top and vice versa, collapse this image along rows, scale by the documented correction factor of 1.35E-5 to form the ghost flux per row, which is then subtracted from the original image.
[ISAAC/_REMOVE_ELECTRONIC_GHOSTING_]

3.1.5 Bad pixels

The recipes apply a predetermined bad-pixel mask with the aim of removing the bulk of ‘hot’ and ‘cold’ pixels. The approximate percentages of pixels masked for each instrument is as follows: Classic Cam1.1, INGRID 1.0, IRCAM 0.1, IRIS2 0.1, ISAAC 0.4, UFTI 0.7, and UIST 0.4.

The Michelle bad pixel mask is time dependent; initially 12 aberrant pixels were identified, but when the new detector returned on UKIRT in 2004, the mask merely ignored the top sixth of the array.

For the test data available, NACO had large numbers of very noisy pixels to the foot and sides of the detector, hence the default bad-pixel mask has 9.9% bad pixels. If your NACO data do not show this, you can form your own mask from a long-exposure dark frame, looking for the highly deviant pixels. See

the notes “Creating a bad-pixel mask” below for suggestions.
[_MASK_BAD_PIXELS_]

There are two problems. First, the pre-calculated mask only accounts for 95% of UFTI’s problem pixels. The other 5% are occasionally deviant on timescales of days. The variability of IRCAM, Michelle, UIST, and IRIS2 bad pixels is unknown at the time of writing. In addition the bad-pixel masks have not been regularly monitored prior to 2000 August. The result is that non-physical values could appear in the processed data, some as extreme as $-10^{-31}$ causing automatic registration and image display to go awry.

Therefore, after dark subtraction, recipes apply thresholding which flags non-physical values as bad, meaning undefined. This is just augmenting the bad-pixel mask, and no valid data are lost. The upper limit is above the nominal saturation levels: 16000 for Classic Cam; 30000 for INGRID; 20000 for IRCAM in STARE or NDSTARE mode, and 33000 using the Deepwell; 200000 for IRIS2 and ISAAC; 100000 for Michelle; 4300 for NACO in FowlerNsamp and Double_RdRstRd modes, but 12400 for Uncorrelated reads except for the $M^{\prime }$ band where it is 28000; 17000 times the number of coadds for NIRI; 15000 for UFTI; and 20000 for UIST. The lower limit is the 2-, 3-, 3-$\sigma$-clipped mean, approximating to the mode, less five times the clipped standard deviation, $\sigma$. While a positive threshold looks attractive, small negative values, while appearing non-physical, can arise through noise. Therefore, to avoid a bias (mainly in the $J$ band), a further constraint is that the lower limit lies in the range $-$100 to 1.
[_SUBTRACT_DARK_, $<$instrument$>$/_GET_SATURATION_LEVEL_]

Recipes issue warnings if the dark-subtracted frame’s mode is negative, allowing for the error in the mode. It aborts with an error message if the modal dark-subtracted signal is more than one standard deviation negative. These states usually arise because of an aberrant dark. Creating a bad-pixel mask  The easiest way to create your own bad-pixel mask for use with the calibration system, is to run the MAKE_BPM recipe on a long-exposure dark (at least 20 seconds integration). It is possible to change the symmetric $\sigma$-clipping bounds in the recipe (see primitive _MAKE_BPM_BY_SIGMA_THRESHOLDING_). You can tailor this primitive if you want more control, say to have asymmetric rejection or more sophisticated definitions.
[_MAKE_BPM_BY_SIGMA_THRESHOLDING_] For better results, use the average of long dark frames taken across two or three nights. First, produce QUICK_LOOK versions of the long-exposure dark to flatten the NDF structure or convert the FITS file. Flag all pixels that are 5 standard deviations ($\sigma$) above and below the 3-$\sigma$ clipped mean of the dark as “bad”, then multiply the resulting frame by zero so that the resulting bad-pixel mask has data values of 0 and bad only. You can choose your own thresholds. Here is an example, using data from two nights of UFTI data and Starlink software.

Then you specify the bad-pixel mask on the command line.

UIST has its own slightly different formula; see DARK_AND_BPM for details.

[UIST/_FIND_BAD_PIXELS_, UIST/_FILE_BAD_PIXELS_, _FILE_MASK_]

3.1.6 Data Variance

There is optional data variance creation for all instruments. By default only the polarimetry and mid-infra-red NOD_CHOP family of recipes have variance processing enabled. To switch on variance calculations for the other recipes is a simple edit of the recipe. See Section 4.2.1 for instructions.

The initial variance is calculated as follows.

• First the readnoise variance for the instrument is applied. It is currently a constant for all pixels, which takes the gain, the number of exposures, and the number of array reads per exposure into account. For more than one read, $n$, the noise scales by the following factor,
  $$\sqrt{\left(n\left(n+1\right)\right)/\left(12\left(n-1\right)\right)}$$

  which is derived from linear-regression theory. The read noise reduces by the factor $1/\sqrt{numberofexposures}$.
  [_ADD_READNOISE_VARIANCE_]

  The raw read noise in units of electrons comes from the readnoise calibration (set within the ARRAY_TESTS) recipe, or should one not exist, Orac-dr selects a representative default. The UFTI and IRCAM defaults come from the instrument Web pages such as

  http://www.ukirt.hawaii.edu/instruments/ufti/PARAMETERS.html and depend on speed, gain, and read type. Typical values are as follows: Classic Cam 40$e^{-}$, INGRID 25$e^{-}$, IRCAM 47$e^{-}$, IRIS2 15$e^{-}$, ISAAC 18.5$e^{-}$, NACO 53$e^{-}$, NIRI 50$e^{-}$, UFTI 26$e^{-}$, and UIST 40$e^{-}$. Michelle reductions merely use an estimate of 1000$e^{-}$.
  [$<$instrument$>$/_GET_READNOISE_]

  The gain—the number of electrons per analogue-to-digital unit—comes from the data headers. If for some unusual reason, the header is absent, the recipe subsitutes suitable time-dependent defaults. The gain is 7.5$e^{-}$ for Classic Cam, 4.1$e^{-}$ for INGRID, $\sim$6$e^{-}$ for UFTI and IRCAM, 5.2$e^{-}$ for IRIS2, 4.6$e^{-}$ for ISAAC, 500$e^{-}$ for Michelle, 10$e^{-}$ for NACO, 12.3$e^{-}$ for NIRI, and 15$e^{-}$ for UIST.
  [$<$instrument$>$/_GET_GAIN_]

• The second step is to add the Poisson variance. This simply adds the data array to the variance component taking into account the gain of the detector and the number of exposures.

  For read modes where the bias level is not removed, such as CHOP or STARE, a bias calibration frame must be subtracted first. If no suitable bias is found, the bias is deemed to be zero; Orac-dr issues a warning that the variance is wrong, likely overestimated.

  The calculations also derive the ratio of the read noise to the Poisson noise, and reports the percentage of background-limited pixels, i.e. those where the Poisson noise is greater than the read noise.
  [_ADD_POISSON_VARIANCE_]

3.1.7 Bias subtraction

In most cases there is no bias to subtract. Recipes attempt to remove a bias frame only if the data have variance information and were taken using a non-ND mode (i.e. where the bias has not already been subtracted in the instrument data system), whereupon a bias frame, if available, is subtracted; if, however, there is no bias calibration the recipe issues a warning issued that the computed variances may be wrong. This step occurs between the creation of the readnoise variance and adding the Poisson variance.
[_REMOVE_BIAS_]

3.1.8 Bias creation

The Michelle ARRAY_TESTS recipe averages two bias frames, sets the observation type to BIAS for the result, and files the mean bias frame in the calibration system. The filed frame includes the observation number in its name.
[MICHELLE/_ARRAY_TEST_STATISTICS_, _FILE_BIAS_]

Likewise the UIST ARRAY_TESTS recipe averages a group of bias frames and sets its variance to a population variance estimate. The mean bias frame filed with the calibration system includes the group number in its name.
[UIST/_BIAS_GROUP_]

3.1.9 Chopping

In the thermal and mid-infra-red regimes the sky is varying so rapidly normal reduction methods are inappropriate. Instead sky subtraction is achieved either by frequently oscillating the secondary mirror between two beams (mid-infra-red), called A and B; or moving the telescope offsets (thermal) after a short exposure. The generic term is chopping. The former are reduced by the NOD_CHOP recipes, and the latter by the NOD_SELF_FLAT_NO_MASK recipes.

Both methods produce frames with the target image at different positions on the detector. The aforementioned recipes difference these pairs of frames, so that the result has both a positive and negative image, and a background level close to zero. The sense of the subtraction is always the same. Orac-dr subtracts the B beam from the A beam, and the normal sequence is ABBA. For the thermal data, the chopped beam is only notional, but the same terminology and subtraction sense is used.
[_DIFFERENCE_PAIR_, _DIFFERENCE_PAIR_SIMPLE_,
_DIFFERENCE_CHOP_BEAMS_]

If the telescope is further offset (nodded), the final mosaic of the differenced frames can have two positive and two negative representations of the source. In practice the thermal reductions register and co-add the nodded frames to compensate for flat-field errors in IRCAM.

3.1.10 Post-pre subtraction

Raw INGRID data comprise a pre-exposure image and a post-exposure integration. The pipeline subtracts the former from the latter to give the measured signal.
[INGRID/_DIFFERENCE_PRE_POST_]

3.1.11 FITS headers

For historical reasons, IRCAM headers prior to ORAC were somewhat jumbled, disorganised, and some violated the FITS standard. The pipeline corrects, orders and structures the headers to help the human reader locate information quickly, and allow complete conversion to FITS.

For IRCAM, Michelle, and UFTI, even since ORAC came online, the raw headers do not provide a sky co-ordinate system. IRIS2 also has an incomplete sky co-ordinate system supplied in the raw headers. Using information in the raw headers, the pipeline creates a FITS world co-ordinate system using a tangent plane projection in the AIPS convention—this is quite adequate given the small fields of view—which it imports into the NDF’s WCS component. Thus Gaia and Kappa can display these co-ordinates on overlay grids and axes. For all these, such as Kappa’s display with axes, you may need to select the appropriate astrometric Frame like this.

where you substitute your NDF’s name for $<$your_NDF$>$. The mosaics from the pipeline should already have the sky domain set. INGRID headers are also revised by the pipeline into an AIPS system.
[$<$instrument$>$/_CREATE_WCS_, $<$instrument$>$/_GET_PLATE_SCALE_]

Note that the reference pixel of the equatorial co-ordinates in the raw headers was until recently only known to a few arcseconds, but now should be of the order of 0.5 arcseconds. The pipeline sets empirical reference pixels now matched by the telescope control system; see primitive _CREATE_WCS_ for details. For critical work, you should tie in your frames with online catalogues, as available through Gaia.

3.2 Dark subtraction

This is as simple as it sounds. The dark frame selected through the calibration system must have the same exposure time and read type as the object frames. There is no dark subtraction for chopped data as processed by the NOD_CHOP collection of recipes, as the differencing of nodded pairs of frames makes the operation unnecessary.
[_SUBTRACT_DARK_, _GET_DARK_NAME_]

3.2.1 Dark creation

After the preliminary steps including addition of variance, the dark is merely filed in the calibration system, using the frame number and the exposure time in the name.

When multiple darks of the same exposure time form part of the same grouped observation, these darks are averaged before filing with the calibration system. Such averaged darks have the group number instead of the observation number in their names.
[_AVERAGE_DARKS_ and IRCAM, UFTI, and UIST variants; _GET_DARK_NAME_]

3.3 Flat-fielding

Some recipes create their own flats from the observations themselves, called a self flat, or sky frames within a sequence that dithers to sky, or use a separate observation of a jitter on sky. (There is no support for internal calibrations or dome flats in the recipes.) Whenever a flat is required, recipes access the flat calibration index to find the most recent flat matching the required attributes such as filter.

The current recipes for Michelle mid-infra-red observations do not create or use flat fields.

Creating a flat can be an iterative procedure, involving cleaning, making a first guess, object masking, proper normalisation, and making an improved flat. There are some primitives to bundle the operations including division by the flat field.
[_FLAT_FIELD_MASKED_GROUP_, _FLAT_FIELD_QUADRANT_JITTER_,
_FLAT_FIELD_NCOLOUR_, _DIVIDE_BY_FLAT_, _DIVIDE_BY_FLAT_FROM_GROUP_]

There are variants for certain families of recipes, which marshall the various required subgroups of frames before dividing by the flat field.
[_DIVIDE_BY_FLAT_CHOP_SKY_, _DIVIDE_BY_FLAT_FROM_EXTENDED_,
_DIVIDE_BY_FLAT_NOD_PAIRS_]

3.3.1 Flat creation

Frames are optionally cleaned to remove extreme outliers ($\pm$3 or 6$\sigma$ about the mean in 15$\times$15-pixel neighbourhood) iterated three times, except for thermal recipes NOD_SELF_FLAT_NO_MASK

and variants, and NOD_SKY_FLAT_THERMAL. The data are then normalised, combined pixel by pixel using a broadened median, and the combined array normalised to a have a mean of one. It is possible to use another statistic for the combination, such as a clipped median.
[_MAKE_FLAT_FROM_GROUP_, _MAKE_FLAT_FROM_NORMALISED_GROUP_,
_MASK_DEVIANTS_]

There are variants for certain families of recipes, which marshall the various required subgroups of frames.
[_MAKE_FLAT_CHOP_SKY_, _MAKE_FLAT_FROM_NORMALISED_CHOP_SKY_,
_MAKE_FLAT_EXTENDED_, _MAKE_FLAT_FROM_NORMALISED_EXTENDED_,
_MAKE_FLAT_QUADRANT_JITTER_]

Flats are filed with the calibration system.
[_FILE_FLAT_]

3.3.2 Object masking

In recipes which make a flat using the frames taken of the targets, the so-called self flat, any sources present can bias the flat field, and result in blotchy mosaics. The full versions of such recipes, as opposed to the _BASIC versions, and SKY_FLAT_MASKED greatly reduce these artifacts using the following algorithm. After the application of the approximate self-flat field, an EXTRACTOR inventory is made of objects having at least 12 connected pixels above 1.0 $\sigma$ above sky. (The thresholds can be changed in $ORAC_DATA_CAL/extractor_mask.sex through the DETECT_MINAREA and DETECT_THRESH parameters.) The locations, shapes, orientations and sizes are used to make a mask. The mask is applied to the dark-subtracted frames and a new flat created. As the outer parts of bright objects often leave residual unmasked blobs, a circular central occulting mask is used. The diameter is normally 7 arcseconds, but it can be adjusted through the OCCULT argument of primitive _MAKE_OBJECTS_MASK_. In the QUADRANT_JITTER recipe the central mask’s diameter equals the length of the shorter side of a quadrant. The disadvantage is that the noise is higher within the occulted circle, and its variance is non-uniform across the central ninth of the mosaic. You can modify the the central mask size and shape; see primitives _FLAT_FIELD_QUADRANT_JITTER_ and _MAKE_OBJECTS_MASK_.
[_MAKE_OBJECTS_MASK_, _MASK_OBJECTS_, _DEFINE_QUADRANT_MASKS_,
_MASK_QUADRANT_]

After masking biases can be introduced as the objects or masks move to different locations on the detector each with a different response in the flat field. This is most pronounced for QUADRANT_JITTER where a quadrant of the detector is masked, and IRCAM2 which had a strongly sloping response. in which the mean flat is considerably different from the remaining quadrants. Merely taking a median at each pixel will preferentially select values from certain frames. Thus there has to be an allowance for these systematic differences before the data are combined to give representative relative intensities. The first frame becomes a reference frame against which the recipes scales the modal values of the other frames.
[_NORMALISE_TO_MODE_, _NORMALISE_TO_MODE_EXTENDED_,
_CLIPPED_STATS_]

The improved flat typically shows a uniformity at $\sim$0.02% of the sky. It is this flat which produces the flat-field frames for mosaicking. Systematic errors in the sky—a major uncertainty in infra-red point-source photometry—are also reduced significantly by this algorithm.

EXTRACTOR on occasions underestimate the sizes of objects, so there is an enlargement factor from 1.0–1.5, defaulting to 1.0, applied in primitive _MAKE_OBJECTS_MASK_. If at high contrast you find residual dark rings in your flat-fielded images, try adjusting the ENLARGE argument either in the recipe or the primitive default. See Section 4

and in particular Section 4.2.3

on how to tailor primitives.

3.4 Field-distortion Correction

The ISAAC instrument exhibits field distortion amounting to 2.5 pixels in the corners of the detector. If not corrected, mosaics with large dithers have multiple images in the overlap regions. Therefore recipes resample the image applying the published polynomial mapping. This correction also improves the registration. The sky co-ordinates are also corrected for the distortion.
[ISAAC/_DEFINE_DISTORTION_TRANSFORMATION_,
ISAAC/_APPLY_DISTORTION_TRANSFORMATION_]

3.5 Bias variation

The bias in the ISAAC short-wavelength camera depends upon the detected flux. Thus in target frames there is a residual bias not fully corrected by bias subtraction evident as two steepening ramps downwards to rows 1 and 513. An ISAAC-variant recipe corrects for this as follows. First within a flat-fielded frame they locate sources and mask them (as described in Section 3.3.2)

Then it forms a one-dimensional profile by masking objects collapsing along rows using the median, from which it subtracts a clipped mean of the profile to form a new profile of the bias variations. The bias-variation profile is then subtracted from each row of the original flat-fielded frame.
[ISAAC/_BIAS_CORRECT_GROUP_]

NACO has alternating positive and negative signals in its columns, most noticeable in longer exposures. The same filter as ISAAC, except it collapses along columns and does not mask objects, is applied to the flat-fielded frames.
[NACO/_BIAS_CORRECT_GROUP_, _REMOVE_COLUMN_ROW_STRUCTURE_]

3.6 Sky Subtraction

In general the recipes do not sky-subtract in the literal sense of pixel-by-pixel subtraction of a sky frame, or better of some median of jittered sky or even target frames. Such recipes could readily be created if there is a demand. Instead the sky signature is usually accounted for in the flat-fielding. Therefore the mosaics generally have the sky signature removed, but not base level.

The sky varies rapidly in the thermal and mid-infra-red, so dithered pairs, or nodded and dithered pairs respectively are differenced to attempt to remove the sky signature. See Section 3.1.9.
[_DIFFERENCE_PAIR_]

Another recipe which performs a pixel-by-pixel sky subtraction is SKY_AND_JITTER. It’s hardly used and not recommended because it demands a very stable sky, since only one sky frame is observed at the start of the observation; and a region of sky devoid of objects to avoid ‘holes’ appearing in the subtracted target frames.
[_SUBTRACT_SKY_SKY_AND_JITTER_]

Another form of sky subtraction is to remove a representative sky level. This benefits imaging of extended sources whose scale exceeds the dither pattern or even the detector’s field of view. The normal procedure is to alternate between dithered integrations on target and a region of sky. The representative statistic is a multiply clipped mean at 2, 2.5, 3, then 3 standard deviations, which effectively gives the mode, and so is not biased by resolved sources. At present the CHOP_SKY_JITTER and EXTENDED_$n\times$$m$ recipes take the average of the modes of bracketted sky frames. For offline processing it would be possible to fit a spline, say, to the modes and provide a better subtraction. Under normal conditions, where the sky level is not varying rapidly or suddenly because of cirrus, the algorithm works well, especially over longer integrations.
[_FORM_SKY_LEVELS_, _NORMALISE_TO_MODE_EXTENDED_, _CLIPPED_STATS_,
_SUBTRACT_SKY_CHOP_SKY_, ESO/_SUBTRACT_SKY_CHOP_SKY_,
_SUBTRACT_SKY_EXTENDED_]

3.6.1 Polarimetry Extraction and Sky Subtraction

The polarimetry recipes are designed to work with a Wollaston prism. This divides the signal into four partitions. These are ordinary and extraordinary beams (usually abbreviated to e and o beams) for the target and a region of nearby sky. Thus the raw data comprise four strips with an aspect ratio of about 6.

These partitions are normally separated by a mask, but the recipes do not depend on having the mask to extract the various regions say by detecting the edges. For each instrument the pixel limits of each region are fixed. The current target limits are 30% to 70% of the width of the long axis of each region to allow for some reasonable dithering of point sources, since there are usually only three jitter positions, while making mosaics with few pixels not derived from all contributing jittered frames. For extended sources these limits change to 10% to 90% to include as much object as possible with smaller dithers and alternating to blank-sky regions. Thus the limits define a section 40% or 80% of the width of the frame, roughly centred on the source. The limits on the sky regions are 1% to 99% of the frame width, mainly to avoid unreliable and pathological pixels at the detector’s edge.
[$<$instrument$>$/_DEFINE_POL_REGIONS_]

The recipes extract the target regions into e- and o-beam frames. The modes (the means after clipping at 2, 3, 3 standard deviations) of the e- and o-beam sky regions are subtracted from their corresponding target beam, incorporating the uncertainty of each sky level in the corresponding target beam’s variance array. For an extended-source observation, the sky levels are determined from the two corresponding regions for each beam in the following sky frame.
[_SUBTRACT_SKY_POL, _SUBTRACT_SKY_POL_EXTENDED_]

3.7 Automatic Registration

This makes an inventory of the sources above a threshold in each frame. It then performs a pattern recognition to identify common features in jittered frames. If the fraction of common objects is under 40% or the total is fewer than three, the registration fails, and so the script resorts to reading the telescope offsets stored in the FITS headers, or matching a central bright object in certain recipes. Using telescope offsets can lead to trailed sources, as occurred with the IRCAMDR package. The improved registration leads to the detection of fainter sources and more-accurate measurement thereof.
[_GENERATE_OFFSETS_ invoked within several wrappers tied to various families of recipes, _FIND_APPROX_OFFSETS_, _TELE_OFFSETS_, _GENERATE_TELE_OFFSETS_,
_GET_CARTESIAN_TELESCOPE_OFFSETS_]

To make use of the best information, registration using more than one of the above methods is permitted.

There is also a new but not extensively tested option in which matching is performed only within overlapping regions as specified from the approximate world co-ordinate system, and a single match within 12 pixels is allowed to define the offsets between frames. This modification should allow more registrations using sources than from telescope offsets, which merely assist in the process. Since the robustness is unknown at present, this option is currently disabled by default. The easiest way to switch it on, is to change the the default value of the SKYREG argument primitive _GENERATE_OFFSETS_ to 1. See Section 4

for generic instructions to make a private _GENERATE_OFFSETS_.

NACO has a tiny plate scale; even with adaptive optics the tolerances (minimum number of pixels and the percentile threshold) searching for fiducial sources are increased, and by default the sky registration is enabled.
[NACO/_GENERATE_OFFSETS_]

3.7.1 Moving Targets

For a moving target, it is possible to permit the telescope to track at sidereal rate, and adjust the telescope offsets for the motion of a slowly moving target, i.e. one that stays within the field within an observation group. Registration uses the revised offsets and in the mosaic the stars trail while the object does not, and so can be seen and measured more easily.

At present the ephemeris is via a pre-prepared text file, given by environment variable

ORAC_EPHEMERIS, if defined, otherwise $ORAC_DATA_OUT/target_ephemeris.dat. See one of the moving-target recipes, say MOVING_JITTER_SELF_FLAT, for more details. Once the ephemeris is available as a web service, the file could become merely a backup; this would automate the process and so the observer would not have to prepare the file at altitude just before the epoch of observation.
[_ADJUST_OFFSETS_TO_MOVING_TARGET_]

3.7.2 Polarimetry Registration

Registration involves both the dithered observations as normal, but also the e- and o-beams at each waveplate angle must align. To aid the latter an approximate pixel displacement between the e- and o-beams from empirical data is set for each filter, as the displacement is wavelength dependent. Even an approximate offset almost invariably leads to successful registration using the central source.
[_GENERATE_OFFSETS_JITTER_POL_, _GENERATE_OFFSETS_POL_]

3.8 Mosaicking

This is fully automated. No longer do you have to measure painstakingly centroids and manually tile to form mosaics from the jittered frames. The jitter offsets are sufficiently small to permit shift of Cartesian co-ordinates to register. The offsets are derived from the automatic registration. No correction for the detector orientation is applied, since it degrades the quality of the data, despite the small rotation ($\sim$1$°$)

normally involved. Allowance for this misalignment with the cardinal directions will be through the FITS world co-ordinate system. In the meantime should you need the rotation angle, pipe the output of KAPPA’s fitslist function into grep.

The flat-fielded frame can either be resampled to give sub-pixel registration, or to the nearest pixel. The latter is much faster and is adequate for most uses of UFTI, UIST, thermal IRCAM, and Michelle. It also has the advantage of not smoothing the data and introducing covariance into the data errors. For older undersampled IRCAM3 data (0.$\text{′′}$286

per pixel) and for INGRID (0.$\text{′′}$235)

resampling has some merit.

The mosaicking uses the Ccdpack algorithm of its makemos command. Only zero-point shifts of intensity are applied to the resampled frames to create the mosaic. For most cases the comparison is of the sky levels as sky pixels dominate. This comparison is repeated for all pairs. makemos then finds the most mutually consistent set of additive corrections, weighting appropriately, to make the smoothest mosaic given the data. The first frame, which for the UFTI execs is the central (offset 0,0) frame, has no additive correction applied. The mosaic generation adjusts the zero-point of the jittered frames. Another way of looking at it is that mosaicking attempts to remove the sky variations. The additive corrections are normally quite small, like a few tenths of a count to a few counts. However, over longer integrations or in the thermal regime they can amount to a few score. A mosaic pixel value is the mean of all the adjusted contributing pixel values at that location. It is possible to select other statistics for the contributing pixels, such as the median, through the METHOD argument of the _MAKE_MOSAIC_ family of primitives.

There is no normalisation to counts per second in the mosaic. The mosaic’s signal corresponds to that of the first frame, thus the exposure time of the mosaic is that of one individual frame. The recipes assume that you have used their corresponding ‘execs’ or‘sequences’, and hence have not changed the exposure time during a jitter. The exposure time (header EXP_TIME [UFTI/UIST/Michelle], EXPOSED [IRIS2], EXPTIME [Classic Cam/INGRID/ISAAC/NACO/-NIRI] or DEXPTIME [IRCAM]) is propagated from the first frame to the mosaic. Where multiple frames combined to create a mosaic pixel the signal-to-noise ratio corresponds to the combined integration time. The integration time (keyword INT_TIME [UFTI], TOTALEXP [IRIS2], or EXPOSED [IRCAM]) is the number of coadds times the exposure time per coadd.

Depending on the recipe, the mosaic may be trimmed to the dimensions of a raw frame. Mosaicking removes virtually all the bad pixels for standard stars where the jitter offsets are small.

A mosaic forms for each cycle of the recipe, e.g. all four frames in a QUADRANT_JITTER. For multiple cycles, an integrated ‘grand’ mosaic forms of improving signal-to-noise. To avoid the build up of bad pixels from cosmic rays, bad pixels are interpolated before the addition. This may result in some strange stripes in the top-left corner of UFTI frames where no interpolation can occur. Those pixels are bad in all frames and should be ignored. The exposure-time header for the integrating mosaic is the sum of the exposure times of the contributing mosaics. Again the signal is not divided by the exposure time.
[_MAKE_MOSAIC_ and invoked within several wrappers (all with the _MAKE_MOSAIC_ prefix) tied to various families of recipes]

3.8.1 Polarimetry Resampling

To permit the calculation of the Stokes parameters, polarimetry recipes resamples each frame using non-integer Cartesian offsets, or merely finds the offsets between frames to the nearest pixel and shifts the origin. The mosaic extends to include all pixels in contributing the frames, however, in practice there should be at most one pixel variation in dimensions.
[_RESAMPLE_MOSAICS_]

3.9 Polarimetry Parameters

These are outlined in the “Output Data” section of the polarimetry recipes such as POL_JITTER, and are calculated using standard formulae and the methods of Polpack. A calibration correction (polrefang) measured in degrees clockwise is applied to the vector direction for the orientation of the analyser with respect to north based upon UKIRT’s IRPOL instrument. Current values are $-24$ for Michelle and UIST, $-9$ for UFTI, and $-96.3$ for IRCAM. For other instruments, the offset can be determined using polarimetric-standard-star calibrations.
[_CALC_STOKES_, MICHELLE/_CALC_STOKES_NOD_CHOP_]

3.10 Near Infra-red Aperture Photometry

The recipes with an _APHOT suffix (except NOD_CHOP_APHOT) perform aperture photometry on the mosaic and the contributing flat-fielded frames. The method assumes that the target, usually a standard star, is approximately centrally located after allowing for the jitter offsets. If you have data where the star lies outside the aperture, it is possible to apply an offset. See the RAOFF and DECOFF arguments of primitive _FIXED_APERTURE_PHOTOMETRY_ in _APHOT_MAG_ to adjust the aperture’s position. The thermal chopped data have sources displaced from the centre, but the reductions allow for the symmetric offsets about the mean jitter position. For UIST thermal data, the search algorithm does not assume a central or centrally symmetric distribution of the positive and negative signals.

Residual bad pixels (usually in the individual flat-fielded frames are removed by median filtering. This does leave a bias in the wings of stars, but certainly the underestimate is far less than ignoring the bad pixels, and is typically far less than the other photometric errors.

The photometry is through a circular aperture located at the centroid of the source, with the sky measured from a concentric annulus outside the aperture. The default aperture size is 5 arcseconds (3 arcseconds for NACO). The annulus diameters are 6.5 to 10 arcseconds (all instruments but UFTI and NACO), 6.5 to 12.5 arcseconds for UFTI, and 3.9 to 6 arcseconds for NACO. The default estimator of the sky flux is the mode calculated from $3\ast median-2\ast mean$ and Chauvenet’s rejection criterion. The photometry accounts for fractional pixels at the aperture edge but without allowance for the local gradient.

The magnitudes are given by the expression $-2.5$*$$log10(abs(counts) per second exposure time). Therefore negative sources can be measured too, as presented by the thermal photometry recipe NOD_SELF_FLAT_NO_MASK_APHOT. The photometry also yields an internal error determined from the sky variance.

A case- and space-insensitive comparison of the object name with the entries in a table provides a catalogue magnitude for a standard star in $I$, $Z$, $J$, $H$, or $K$ for all instruments, and in $L$ or $M$ for IRCAM and ISAAC. Also a mean extinction is applied for the mean of the start and end airmasses. Thus the primitive calculates an approximate zero point. Note that ISAAC and NACO standard stars include additional objects not present in the UKIRT faint-standard list or Persson’s HST list; for these the magnitude and derived zero point will not be determined automatically. For accurate photometry the actual extinction coefficients should be determined. As the output from the photometry is a small text list, you can use catphotomfit command of the Cursa package to achieve this. The units and meanings of the columns are documented within each small text list.

The seeing is estimated for each frame and the mosaic by fitting a two-dimensional Gaussian to the star, although in good seeing the UKIRT images are more centrally concentrated than a Gaussian. The full-width-half-maximum so derived is also tabulated in the small text list.
[_APHOT_MAG_, _APHOT_MAG_NCOLOUR_, _NOD_APHOT_MAG_ (plus NACO and UIST variants), _FIXED_APERTURE_PHOTOMETRY_, _MAKE_PHOTOMETRY_TABLE_,
_GET_FILTER_PARAMETERS_ (and several instrument variants),
_PERSSON_STANDARD_MAGNITUDE_, _UKIRT_STANDARD_MAGNITUDE_,
_STANDARD_MAGNITUDE_ (and instrument-specific variants), _CLIPPED_STATS_,
_FIND_SOURCE_CENTROID_, _GET_FRAME_CENTRE_]

3.11 Mid-infra-red Aperture Photometry

This is similar to near infra-red aperture photometry. The main difference is the source measured. Rather than measuring each positive and negative image, NOD_CHOP_APHOT integrates the average flux of the one or two pairs of positive and negative images in the chopped and nodded mosaic. The number of pairs depends on the chop-and-nod orientations and the nod throw. The four (or two) images are first registered using their centroids; extracted with a symmetric neighbouring background, but without any duplication of pixels from the original mosaic; and finally combined, averaging to preserve the flux per unit time. The exposure-time header, therefore, remains that of the mosaic.

The aperture size is 5 arcseconds. The background is determined from an annulus with an inner diameter of 7.5 arcseconds and an outer diameter of 15 arcseconds enclosing the source. The Michelle chip characteristics and the UKIRT chop throw limit the area of this combined-image mosaic—normally a 15-arcsecond square—leaving comparatively little background. Recommended apertures and background regions may change in the light of experience with Michelle. Already in below-average seeing, it’s evident that a 6-arcsecond circle will leave some signal in the background.

The object name is compared with a 28-member catalogue of 10- and 20-micron standards, a handful of which are known optical semi-regular variable stars. In addition to deriving magnitudes, the recipe calculates an approximate flux in Janskys. It is approximate because being an absolute measurement, it does require a magnitude zero point to be applied to the relative instrumental magnitude. At the time of writing this zero-point is only roughly known, so the default fluxes should be taken with a pinch of salt. The flux is not yet written to the results file. If you have determined the zero point from standards, you can rerun the pipeline for your target observations with that zero point assigned to argument ZP of primitive _NOD_CHOP_APHOT_MAG_ invoked within recipe NOD_CHOP_APHOT. See Section 4

for instructions to make and use a private version of a recipe.

There is an approximate extinction correction applied using a coeeficient of 0.18, because the coefficients for all but one of the various $N$ and $Q$ filters have yet to be determined.
[_NOD_CHOP_APHOT_MAG_, _FIXED_APERTURE_MIDIR_PHOTOMETRY_,
_MAKE_PHOTOMETRY_TABLE_, _GET_FILTER_PARAMETERS_,
_UKIRT_MIDIR_STANDARD_MAGNITUDE_, MICHELLE/_STANDARD_MAGNITUDE_, _FIND_SOURCE_CENTROID_, _CLIPPED_STATS_]

3.12 Improving the signal-to-noise of Mid-infra-red Data

The Michelle electronics can leave an uneven mosaic with vertical banding from bias variations and horizontal ripple patterns from electronic pickup. Therefore, Michelle NOD_CHOP recipes subtract the median along each column of the mosaic, then subtracts the median along each row. This cleaning aids the visibility of faint sources.

In the mid-infra-red, the sky signal is vastly greater than the signal than even the brightest sources. While the nodding removes the bulk of the sky signal, the sky noise remains, still swamping the signal from a faint source. Integrating over many nod-chop cycles is needed, and the real-time display of Orac-dr does permit interactive review of the signal-to-noise in Gaia so observers can curtail data collection when the required contrast is achieved. Averaging the positive and negative signals and neighbourhoods into a combined signal (cf.  Section 3.11

only helps by a factor of two (or $\sqrt{2}$ for a single positive and negative pair). Recipe NOD_CHOP_FAINT smooths the combined quadrant with a 4-by-4-pixel neighbourhood running average filter to help reveal faint sources. Note that the source centroids are not used for registration as the signal is usually too weak.

While smoothing reveals the sources, it is not sufficient. There is also a confusion issue. The chopping can bring positive and negative sources actually located beyond the final quadrant to within it. There is an option of the _COMBINE_CHOPPED_SOURCE_ primitive, called by recipe NOD_CHOP recipes, which attempts to clarify which sources are actually present in the quadrant by forming a quality map. Each quality pixel is the sum of the four corresponding pixel values divided by their absolute values, after changing sign for the quadrants containing the negative images. In the map +4 indicates that a pixel had positive contributions from the positive quadrants and negative signals from the negative quadrants. A quality of +4 strongly implies that the signal is really at the sky location indicated. Thus it helps to discriminate from sources which have been chopped into view, for which there are no positive or negative counterparts (values +/$-$2) or noise (0, +/$-$2). Around the sky (which should be near zero) noise randomises the quality measurement, therefore smoothing should be used in conjunction with the quality map.
[_COMBINE_CHOPPED_SOURCE_, _REMOVE_COLUMN_ROW_STRUCTURE_,
_REMOVE_COLUMN_ROW_STRUCTURE_SCAN_, FIND_SOURCE_CENTROID_,
_GET_CHOP_OFFSETS_, _GET_FRAME_CENTRE_]

To create a quality map at each cycle, you should set argument QMAP=1 and SMOOTH$>1$. For example in NOD_CHOP_FAINT, append " QMAP=1" to the line

3.13 Catalogue Generation

The recipes with a _CATALOGUE suffix create a source catalogue from the mosaic using EXTRACTOR. The catalogue includes objects having at least 12 connected pixels above 1.0 $\sigma$ above sky. The catalogue is written in ARK Cluster format, with a field ID in column 0 (this value is typically zero); an object ID in column 1; right ascension in columns 2, 3, and 4; declination in columns 5, 6, and 7; x and y positions in columns 8 and 9; uncalibrated magnitude in column 10; the error in magnitude in column 11; and a quality flag in column 12 (this value is typically zero). The magnitudes are given by the expression $-2.5*$log10(counts) per second exposure time.

As the final mosaic has varying noise characteristics, with higher noise regions at the edges, the detection limit varies across the mosaic. Fainter objects can be detected in the central region than near the edges and corners.
[_CREATE_SOURCE_CATALOGUE_, _GET_CATALOGUE_NAME_]

3.14 eSTAR Integration

Orac-dr is also integrated with the eScience Telescopes for Astronomical Research (eSTAR) project, which is designed to automatically detect and follow up on transient astronomical objects using manned and robotic telescopes. Using the catalogue generated in Section 3.13

Orac-dr will automatically send a trigger to the eSTAR network containing a FITS file of the final mosaic and the catalogue. This step is only done when an observation is taken as part of an eSTAR project, i.e. there is a RMTAGENT header and its value is ESTAR.
[general/_TRIGGER_ESTAR_, _SET_REMOTE_AGENT_HEADER_]

3.15 Tidying

Each recipe has a tidy procedure, which removes unnecessary intermediate frames when the recipe no longer requires them. Retained are the raw data, flat-fielded frames, differenced pairs (for chopped data), and the mosaics. Most of the intermediate small text files are removed in individual primitives, but some registration-related files do persist until the tidy script cleans up. If you need to retain the intermediate files, comment out (#) the final instruction of the recipe, which calls the tidy primitive, and follow the instructions in Section 4

to make and use a private version of a recipe, or set the $ORAC_KEEP environment variable to 1.
[$<$recipe_family$>$_TIDY_, such as _EXTENDED_TIDY_, _JITTER_SELF_FLAT_TIDY_,
_NOD_CHOP_TIDY_, _REDUCE_DARK_TIDY_; general/_DELETE_A_FRAME_,
general/_DELETE_INTERMEDIATE_GROUP_FILES_, general/_DELETE_TEMP_FILES_,
general/_DELETE_TEMP_GROUP_FILES_, _IMAGING_GOODBYE_]