ORAC-DR may use a variety of ways to detect available data. This document describes what they mean and when to use them.
ORAC-DR is a data-driven pipeline. This means that it does things in response to incoming data and uses the information associated with that data to determine how to process a file. It is also a sequential (i.e. non-parallel) process. This means it only does one thing at a time. As a result, ORAC-DR is always doing one of two things
Seeks new data
Reduces data
Unless the -files option is used, the pipeline starts from looking for observation number 1, unless
another number has been specified via the -from or -list options.
The various -loop options determine what the criterion is for concluding that the observation it is
waiting for has indeed arrived.
If you use this option, the pipeline monitors the size of the file that is is expecting. For example, if it has just reduced observation number 41, it waits for observation number 42 to appear on disk and watches it size growing as it is being written out by the data handling system. If the file does not grow in size for a certain amount of time it concluded that readout is complete and proceeds with reducing it. Obviously this method is not very robust if the pipeline is operating or network-mounted disks or with acquisition systems that are prone to stalling during readout. However it may be the only option for online data reduction with some data handling systems. This option should be used with IRCAM.
This option instructs the pipeline to monitor not the data file itself, but a “flag” file whose appearance indicates readout completion. Typically this is a zero-length file written by the data acquisition after the data file writing is done. This is most robust in architectures where there is no chance of a data file being written without a flag file or vice versa. This option should be used with SCUBA, UFTI, the WFS and MICHELLE.
Under this option, the pipeline reduces data assuming it is available and keeps going
one observation at a time until no more data is to be found (or infinity, whichever comes
first!), at which point it terminates. It overrides the -to option. This is suitable for offline
data reduction of any instrument and is the default option if none is specified.
The specified data frames (and/or range of observations) are assumed to be available, are
reduced and then the pipeline exits. This option is implied by usage of the -list option,
or usage of the -from and -to options in the same invocation. It is unlikely that a user
will need to explicitly specify it.
The specified data frames are assumed available, are reduced and then the pipeline exists.
This option is impled by usage of the -files option. Which provides a filename (relative
to the current directory) of a flat ASCII text file containing a list of observation files to
be reduced, one filename per line. The filenames, unlike other loop options used by
ORAC-DR, are not based on UT date and may be arbitarily constructed.