2 Using the pipeline

2.1 Setting up Orac-dr

Before you can run the pipeline you have to tell Orac-dr for which instrument you wish to reduce data, the observation date, and the directory containing the raw data, and wher you want the processed data to be written. There are two options.

• The first needs your data to conform to the directory-naming convention of the instrument at UKIRT. This will be the case if you simply unpack the archive written by the uktape utility. In this case enter
          % setenv ORAC_DATA_ROOT <root_data_directory>
          % oracdr_<instrument> <date>

  where $<$root_data_directory$>$ is the directory in which you unpacked the data from the tape, $<$instrument$>$ is either ufti or ircam, and $<$date$>$ is the UT date in the format YYYYMMDD. Note that each % represents the UNIX shell’s prompt, which you do not type. The commands must be entered in the above order.

  For example, the standard location for raw UFTI data is raw/ufti/YYYYMMDD/, and reduced/ufti/YYYYMMDD/ for the corresponding reduced data. So if your data are stored in /home/users/abc/data/UKIRT/raw/ufti/20001108/ you should enter the following.

          % setenv ORAC_DATA_ROOT  /home/users/abc/data/UKIRT/
          % oracdr_ufti 20001108

  to enable the pipeline for UFTI data taken on 2000 November 8.

  Data taken from the AAT is handled differently, as there is no unified directory structure for both raw and reduced data directories. For IRIS2, INGRID, ISAAC, NACO, NIRI, or Classic Cam data the best option is specifying where the raw and reduced data directories are, as shown below. Those with ISAAC and NACO data should see

  Section 2.2.1 for a necessary preliminary naming conversion step for each instrument. Classic Cam users need to read

  Section 2.2.2 concerning renaming raw data files for the respective instruments.

• The second option is where your raw and reduced data are to be in arbitrary directories. Type the following
          % oracdr_<instrument> <date>
          % setenv ORAC_DATA_IN <raw_data_directory>
          % setenv ORAC_DATA_OUT <reduced_data_directory>

  The directories can either be given as full paths, or as relative paths to the current working directory. Here is an example for IRCAM data using full paths.

          % oracdr_ircam 19990328
          % setenv ORAC_DATA_IN /export/data/mjc/asteroid/night1
          % setenv ORAC_DATA_OUT /home/scratch/mjc/reduced

In the first case $ORAC_DATA_IN and $ORAC_DATA_OUT are still defined, but in terms of the root directory. For instance, re-using the earlier example with UFTI for UT date 2000 November 8, $ORAC_DATA_IN points to $ORAC_DATA_ROOT/raw/ufti/20001108/.

Orac-dr operates in $ORAC_DATA_OUT, irrespective of what your current directory is when you invoke it. Your current directory remains unchanged.

It is highly recommended to work in directories on discs local to the computer running the pipeline. Processing over NFS-served drives can many times slower and degrades the performance seen by other users. Running Orac-dr on a Linux computer over NFS-served drives can also lead to erroneous results, crashing of the pipeline, or computer lockups.

2.2 Raw Data Formats and Conversions

Raw data take the form of multiple NDFs within an HDS container file for UKIRT data, or individual FITS files for AAT, INGRID, ISAAC, NACO, NIRI, and Classic Cam data. For UFTI, UIST, and IRCAM they comprise one NDF for the data array and dynamic headers, such as the start time of the exposure, and another for static headers. Each container file is converted to a single NDF in $ORAC_DATA_OUT with a merged set of headers.

The Michelle HDS container file also has NDFs for the individual chop beams. However, these cannot be merged until the data variance is calculated from the individual beams. Michelle reduced chopped data become simple NDFs once the recipe takes the difference of the two beams.

Orac-dr automatically converts AAT FITS files into single NDFs in $ORAC_DATA_OUT which retain the original FITS headers. For INGRID and NIRI, Orac-dr converts a multi-extension FITS file into a multi-NDF HDS container file following UKIRT conventions.

2.2.1 ISAAC and NACO Preliminary Conversion

Since Orac-dr as yet cannot cope with ESO file naming, which uses the UT epoch instead of a sequence number, there is a special C-shell script which must be invoked once, normally before the first Orac-dr initialisation. If you enter

in a directory containing ISAAC FITS files, the command converts them into NDFs with names adhering to the UKIRT convention. The prefix is isaac. The earliest file has observation number 1, and the observation number increments for each FITS file in time order. The script copes with files names in either the raw or archive nomenclature. It also writes observation and group number headers to assist ORAC-DR. It copes with data from more than one night in a given directory, assigning each night its own sequence of observation numbers; and it uses a common UT date for observations in a single night spanning midnight UT. You should put all both the calibration and target files for a given night in the same directory.

Likewise in a directory of NACO FITS files, you should first enter

to create a set of NDF files whose names adhere to the UKIRT convention with a naco prefix.

2.2.2 Classic Cam Preliminary Conversion

The Magellan Classic Cam raw FITS data have a sequence number but no UT date in their names. There is a preprocessor C-shell script which must be invoked once, normally before the first Orac-dr initialisation. If you enter

in a directory containing Classic Cam FITS files, the command converts them into NDFs with names adhering to the UKIRT convention. The prefix is cc. The earliest file has observation number 100 (a Classic Cam convention), and the observation number increments for each FITS file in sequence-number order.

It also writes the observation and group numbers, and the number of offsets in the grouped observation into the headers, to assist ORAC-DR.

2.3 Running the pipeline

To run the pipeline, you use the oracdr command. This has a number of qualifiers described fully in SUN/230. There is online help too; enter

for a list of the options.

Unlike using Orac-dr at UKIRT, you are unlikely to need the looping (-loop option) for offline processing, as all the data exist. Thus the most important qualifiers are -list and -from, which specify the frames to process; and the recipe name.

will process frames f20001108_00042 until the end of the night’s data (assuming the earlier oracdr_ufti command), running the recipes given by each frame’s header (RECIPE keyword). More likely is that you provide a list of selected observations. The following example

processes frames from 41 to 49 inclusive and 51 to 59 inclusive, invoking the JITTER_SELF_FLAT recipe, and overriding the RECIPE header.

would reduce the frames 5, 6, 7, 23, and 33. This is most likely to be applicable to a series of dark frames.

There is a hazard with the -list option. Take care to select a complete set of frames associated with an observation. A common error is to include accidently a dark frame not part of the group. Check the log in the raw data directory; it has file extension .nightlog. If you do not have a log, it is easy to create one.

This will create a log called $ORAC_DATA_IN/$<$date$>$.nightlog for the current UT date. For multi-mode instruments such as Michelle, UIST, IRIS2, NACO or ISAAC, the log will be called

$ORAC_DATA_IN/$<$date$>$_im.nightlog.

2.4 Graphical initialisation and operation

You may prefer the Orac-dr graphical interface called xoracdr. ( See SUN/230.) It allows you to configure ORAC-DR: set the instrument, UT date, raw and reduced directories; and to run the pipeline with the various options. It permits monitoring of the primitives during execution of a recipe. xoracdr offers access to other facilities like display control and recipe editing. The in-built documentation does not pertain to the GUI itself but to general Orac-dr information, however, xoracdr is straightforward to use and explore. While xoracdr has some rough edges, it is popular with many users. To try it, enter

Once the tool appears, you should select an instrument from the menu on the left, a UT date in the top centre, and raw and reduced directories to the lower right. The From: and To: refer to the observation numbers to process. When you are ready to reduce data, click on the Start ORAC-DR button.

2.5 Display

Orac-dr optionally lets you inspect the raw frames, and the processed data as they are created. There is a variety of graphical methods available, including histograms and contour plots, if you choose a Kappa GWM widget. Most people prefer a simple scaled image display with Gaia. This offers facilities to inspect and analyse the data, and both pixel and sky co-ordinates of the cursor position are presented. The selection of frame types to display, where they should appear, and how they are scaled are configurable using a simple text file or a special GUI tool oracdisp. See SUN/230 for details and examples.

Processing offline, there is less need to see the data displayed in real time. If you wish to accelerate the processing switch off the display option.

If you do want to display a recommendation is to create two GAIA windows displaying images using autoscaled limits. This first could be for raw and flat-fielded data, and the second for the mosaics. You are likely to want to interact with the latter, using Gaia’s toolboxes. Your $ORAC_DATA_OUT/disp.dat could look like this.

2.6 Calibration Information

Orac-dr records calibration information, such as dark frames, flat fields, and the read noise, within index files, one for each type of calibration information. When the pipeline needs a calibration frame it searches the index file for the best matching entry subject to a set of rules. Each recipe reports the calibrations it has used. If no suitable calibration exists, the pipeline exits with an error message stating this fact. For further details see SUN/230.

Section 4.3 has an example of an index file.

You can also select a specific calibration using the -calib command-line option, provided the chosen calibration has an entry in the appropriate index file. See the section on calibration options in SUN/230 for details and examples.

2.6.1 Available calibration methods

The following calibration methods are available for imaging recipes.

• baseshift — Use the given comma-separated doublet (i.e. 0,0) as the frame’s base position. This is used to locate faint sources in the mid-infra-red data where centroiding fails when there is some telescope pointing error (such as incorrect instrument apertures). It is calibrated within NOD_CHOP_APHOT on a bright standard, and used by NOD_CHOP_FAINT. For a well-tuned system, the baseshift is expected to be near 0,0, so the centre of the detector is at the reference position (derived from the FITS headers).
• bias — Use the given bias frame.
• dark — Use the given dark frame.
• fpcentre — Use the given Cartesian pixel centre of the Fabry-Perot transmitted region (UFTI only).
• flat — Use the given flat frame.
• mask — Use the given bad-pixel mask.
• polrefang — Use the specified polarimetry calibration angle, converting measured polarisation vector orientations into position angles. It is measured in degrees, applied clockwise to the vector direction to allow for the orientation of the analyser with respect to north.
• readnoise — Use the given value for the detector readnoise in electrons.
• referenceoffset — Use the comma-separated doublet (i.e. 0,0) as the frame’s reference offset, which is difference between the frame centre and the reference pixel derived from the FITS headers.
• rotation — Use the given frame as a rotation matrix. This is no longer used in the imaging recipes.
• sky — Use the given sky frame.

2.7 Log files

In addition to presenting the progressing data reduction to an Orac-dr X-window, Orac-dr, by default, retains a copy of the processing steps and errors in a log file. These logs are important if something has gone wrong, and you have exited the X-window. Information from the applications software can be included if you run the pipeline with the -verbose command-line option. Logs also serve as a record of the data processing. Yet the log files are often overlooked because they are hidden. The log file is called $ORAC_DATA_OUT/.oracdr_$<$number$>$, where $<$number$>$ is the current process identification. The -log f option to the oracdr command enables log-file creation.

See SUN/230 for details of the logging options.