2 Using the pipeline

2.1 Setting up Orac-dr

Before you can run the pipeline you have to tell Orac-dr for which instrument you wish to reduce data, the observation date, the directory containing the raw data, and where you want the processed data to be written. For the following two there are two options.

• The first needs your data to conform to the directory-naming convention of the instrument at UKIRT. This will be the case if you simply unpack the archive written by the uktape utility. In this case enter:
          % setenv ORAC_DATA_ROOT <root_data_directory>
          % oracdr_<instrument> <date>

  where $<$root_data_directory$>$ is the directory in which you unpacked the data from the tape, $<$instrument$>$ is either cgs4 or uist, and $<$date$>$ is the UT date in the format YYYYMMDD. Note that each % represents the UNIX shell’s prompt, which you do not type. The commands must be entered in the above order.

  For example, the standard location for raw CGS4 data is raw/ufti/YYYYMMDD/, and reduced/ufti/YYYYMMDD/ for the corresponding reduced data. If your data are stored in /home/users/abc/data/UKIRT/raw/cgs4/20031022/ you should enter the following:

          % setenv ORAC_DATA_ROOT /home/users/abc/data/UKIRT/
          % oracdr_cgs4 20031022

  to enable the pipeline for CGS4 data taken on 2003 October 22.

  Data taken from the AAT is handled differently, as there is no unified directory structure for either raw or reduced data directories. For IRIS2, INGRID or ISAAC data the best option is specifying where the raw and reduced data directories are, as shown below. ISAAC users should see

  Section 2.2.1 for a necessary preliminary naming conversion step.

• The second option is to separately define the raw and reduced data directories. Type the following:
          % oracdr_<instrument> <date>
          % setenv ORAC_DATA_IN <raw_data_directory>
          % setenv ORAC_DATA_OUT <reduced_data_directory>

  The directories can either be given as full paths or as relative paths to the current working directory. Here is an example for UIST data using full paths:

          % oracdr_uist 20040414
          % setenv ORAC_DATA_IN /home/bradc/data/oracdr/asteroid/night1
          % setenv ORAC_DATA_OUT /home/scratch/bradc/reduced

In the first case $ORAC_DATA_IN and $ORAC_DATA_OUT are still defined, but in terms of the root directory. For instance, re-using the earlier example with CGS4 for UT date 2003 October 22, $ORAC_DATA_IN points to $ORAC_DATA_ROOT/raw/cgs4/20031022/.

Orac-dr operates in $ORAC_DATA_OUT, irrespective of what your current working directory is when you invoke it. Your current directory remains unchanged.

It is highly recommended to work in directories on disks local to the computer running the pipeline. Processing over NFS-mounted drives can be many times slower and can degrade the performance seen by other users. Running Orac-dr on a Linux computer over NFS-mounted drives can also lead to corrupted files, crashing of the pipeline, or computer lockups.

2.2 Raw Data Formats and Conversions

Raw data take the form of multiple NDFs within an HDS container file for UKIRT data, or individual FITS files for AAT, INGRID and ISAAC data. For UISTthey comprise of one NDF for the data array and dynamic headers, such as the start time of the exposure, and another for static headers. For the HDS containers, initial reduction steps operate on each of the NDFs individually, only merging them to a simple NDF once the interleaving step is complete.

The Michelle HDS container also has NDFs for the individual chop beams. However, these cannot be merged until the data variance is calculated from the individual beams. Michelle reduced chopped data become simple NDFs once the recipe takes the difference of the two beams.

Orac-dr automatically converts AAT FITS files into single NDFs in $ORAC_DATA_OUT which retain the original FITS headers. For INGRID, Orac-dr converts a multi-extension FITS file into a multi-NDF HDS container file following UKIRT conventions.

2.2.1 ISAAC Preliminary Conversion

Since Orac-dr as yet cannot cope with ISAAC file naming, which uses the UT epoch instead of a sequence number, there is a special C-shell script which must be invoked once, normally before the first Orac-dr initialisation. If you enter

in a directory containing ISAAC FITS files, the command converts them into NDFs with names adhering to the UKIRT convention. The earliest file has observation number 1, and the observation number increments for each FITS file in time order. The script copes with file names in either the raw or archive nomenclature. It copes with data from more than one night in a given directory, assigning each night its own sequence of observation numbers; and it uses a common UT date for observations in a single night spanning midnight UT. You should put all of the calibration and target files for a given night in the same directory.

2.3 Running the pipeline

To run the pipeline, you use the oracdr command. This has a number of qualifiers described fully in SUN/230. There is online help too; enter

for a list of the options.

Unlike using Orac-dr at UKIRT, you are unlikely to need the looping (-loop option) for offline processing, as all the data exist. Thus the most important qualifiers are -list and -from, which specify the frames to process; and the recipe name.

will process frames c20031022_00042 until the end of the night’s data (assuming the earlier oracdr_cgs4 command), running the recipes given by each frame’s header (RECIPE keyword). More likely is that you provide a list of selected observations. The following example

processes frames from 41 to 49 inclusive and 51 to 59 inclusive, invoking the ?? recipe, and overriding the RECIPE header.

would reduce the frame 5, 6, 11 and 12. This is most likely to be applicable to pairs of flats and arcs.

There is a hazard with the -list option. Take care to select a complete set of frames associated with an observation. A common error is to accidentally include an arc frame not part of the group. Check the log on the raw data directory; it has file extension .nightlog. If you do not have a log, it is easy to create one.

This will create a log in $ORAC_DATA_IN for the current UT date. For CGS4, the log will be named $<$date$>$.nightlog. For multi-mode instruments such as Michelle, UIST, IRIS2, or ISAAC, there may be two log files created, one called $<$date$>$_im.nightlog and another called $<$date$>$_sp.nightlog, depending on the observing mode. In general mode-agnostic observations such as array tests are taken under imaging mode and will show up in the _im log, whereas science and calibration observations will show up in the _sp log.

2.4 Graphical initialisation and operation

You may prefer the Orac-dr graphical interface called xoracdr. ( See SUN/230.) It allows you to configure ORAC-DR: set the instrument, UT date, raw and reduced directories; and to run the pipeline with the various options. It permits monitoring of the primitives during execution of a recipe. xoracdr offers access to other facilities like display control and recipe editing. The in-built documentation does not pertain to the GUI itself but to general Orac-dr information, however, xoracdr is straightforward to use and explore. While xoracdr has some rough edges, it is popular with many users. To try it, enter

Once the tool appears, you should select an instrument from the menu on the left, a UT date in the top centre, and raw and reduced directories to the lower right. The From: and To: refer to the observation numbers to process. When you are ready to reduce data, click on the Start ORAC-DR button.

2.5 Display

Orac-dr optionally lets you inspect the raw frames, and the processed data as they are created. There is a variety of graphical methods available, including histograms and contour plots, if you choose a Kappa GWM widget. Most people prefer a simple scaled image display with Gaia. This offers facilities to inspect and analyse the data, and both pixel and sky co-ordinates of the cursor position are presented. The selection of frame types to display, where they should appear, and how they are scaled are configurable using a simple text file or a special GUI tool oracdisp. See SUN/230 for details and examples.

Processing offline, there is less need to see the data displayed in real time. If you wish to accelerate the processing switch off the display option.

2.6 Calibration Information

Orac-dr records calibration information, such as arc frames, flat fields, and the read noise, within index files, one for each type of calibration information. When the pipeline needs a calibration frame it searches the index file for the best matching entry subject to a set of rules. Each recipe reports the calibrations it has used. If no suitable calibration exists, the pipeline exits with an error message stating this fact. For further details see SUN/230.

Section ?? has an example of an index file.

You can also select a specific calibration using the -calib command-line option, provided the chosen calibration has an entry in the appropriate index file. See the section on calibration options in SUN/230 for details and examples.

2.6.1 Available calibration methods

The following calibration methods are available for spectroscopy recipes.

• bias — Use the given bias frame.
• flat — Use the given flat frame.
• mask — Use the given bad-pixel mask.
• readnoise — Use the given value for the detector readnoise in electrons.

2.7 Log files

In addition to presenting the progressing data reduction to an Orac-dr X-window, Orac-dr, by default, retains a copy of the processing steps and errors in a log file. These logs are important if something has gone wrong, and you have exited the X-window. Information from the applications software can be included if you run the pipeline with the -verbose command-line option. Logs also serve as a record of the data processing. Yet the log files are often overlooked because they are hidden. The log file is called $ORAC_DATA_OUT/.oracdr_$<$number$>$, where $<$number$>$ is the current process identification. The -log f option to the oracdr command enables log-file creation.

See SUN/230 for details of the logging options.