17 Processing Groups of Data Files

 17.1 Applications that Process Groups of NDFs
 17.2 What about the other Parameters?
 17.3 Output Parameters
 17.4 What Happens if an Error Occurs?
 17.5 What about Applications that Re-use Parameters?
 17.6 Introducing a Pause Between Invocations
 17.7 Reporting the Data Files being Processed
 17.8 The Syntax for Specifying Groups of Data Files
 17.9 Using non-NDF Data Formats
 17.10 Disabling Multiple Invocations of Applications

When a Kappa application requests an input or output data file (either an NDF or a positions list), you may optionally give a group of several data files rather than just one. In this case, the application is automatically re-run until all the supplied data files have been processed. For instance, in the following command:

      % display in="../a*" mode=perc accept

a group of NDFs (all those beginning with “a” in the parent directory) are assigned to the IN parameter, and the DISPLAY command is automatically re-run to display each NDF in the fashion of a (rather slow!) movie.

Another example:

      % stats ndf=^files

will display the pixel statistics of all NDFs listed within the text file files. The “^” character indicates that the following string (files) is not the name of an NDF, but the name of a text file from which NDF names should be read.

      % wcsframe ’"image_a,image_b,image_c"’ sky

This will set the current co-ordinate Frame for the three NDFs image_a, image_b and image_c so that celestial sky co-ordinates are used to refer to positions within the NDFs, if possible.

      % cursor outcat="’first,^list_names’"

This will run CURSOR several times, allowing you to select display positions using a cursor. On the first invocation, the selected positions are written to a positions list stored in file first.FIT. Positions selected on subsequent invocations are written to positions lists with names read from the text file list_names. Finally, if this CURSOR command is followed by:

      % listshow accept

LISTSHOW will display the contents of all the catalogues created previously by CURSOR.

If an application has more than one NDF or positions list parameter, each parameter should be given the same number of values (i.e. data files). A warning is issued if any parameter is given too many values, but processing continues normally until the smallest group is exhausted. For instance, the following example adds NDF a1 to a2, and b1 to b2, putting the results in a3 and b3:

      % add in1="’a1,b1’" in2="’a2,b2’" out="’a3,b3’"

If (say) an extra NDF had been specified for Parameter IN2, the application would have been invoked twice to process the first two pairs, and then a warning message would have been displayed saying that too many NDFs were specified for Parameter IN2.

There is a special case in which this rule does not apply. If only a single value is given for an data file parameter, the same value is used repeatedly on all invocations of the application. So, for instance, if a single NDF had been given for Parameter IN2 in the above ADD example, the application would have again been run twice, using the same NDF for Parameter IN2 on each invocation.

The OUT parameter in the above example could alternatively have been specified as out="’*|1|3|’". Here, the asterisk (*) represents the base-names, a1 and b1, of the NDFs supplied for the first NDF parameter to be accessed (IN1). The following string, |1|3|, means replace all occurrences of 1 with 3, thus giving the final NDF names a3 and b3.

17.1 Applications that Process Groups of NDFs

The majority of applications with NDF parameters use each NDF parameter to access a single NDF, and supplying more than one NDF will result in the application being re-run as described above. The application then accesses a single NDF on each invocation. Some applications, however, have NDF parameters that are explicitly described in the reference section of this document as being associated with a group of NDFs. An example is WCSALIGN that has a Parameter IN to read a group of NDFs that are to be aligned with each other. Such applications process all the specified NDFs in a single invocation. For the purposes of the multiple invocation scheme described above, such parameters are not considered to be ‘NDF’ parameters, and will not cause the application to be re-run.

17.2 What about the other Parameters?

When an application is re-run to process multiple data files, all the parameters not associated with NDFs or positions lists retain their values from one invocation to the next. So, for instance, the assignment for the MODE parameter in the earlier DISPLAY example is retained and used for all subsequent invocations of the application, you are not prompted for a new value each time the application runs.

This is usually what you want, but beware that there are times when this behaviour may trip you up. Sometimes an application may prompt for a new parameter value while in the middle of processing a group of NDFs. This can occur for instance, if the initial value you supplied on the first invocation is inappropriate for the NDF currently being processed. For instance, supposing you use WCSFRAME to set the current co-ordinate Frame to SKY for a group of NDFs. To do this, you would set the FRAME parameter to SKY either on the command line or when prompted during the first invocation. This value would be retained for subsequent invocations, but what happens if one of the NDFs does not have a SKY Frame defined in its WCS component? Not surprisingly, you get an error message identifying the NDF, and you are asked to supply a new value for FRAME. You could, for instance supply PIXEL as the new value. This changes the current value of the FRAME parameter to PIXEL, and this value will consequently be used for any remaining NDFs.

If you do not specify a value for a parameter, the default value used by the first invocation will be re-used for all subsequent invocations. Note that the default value for some parameters (for instance the CENTRE parameter of the DISPLAY command) is the null value !. This is usually interpreted as a request for the application to find an appropriate value itself for the parameter. In these cases, the parameter value is ! and is re-used on all invocations, resulting in the application finding and using a potentially different value on each invocation. So, for instance, the above DISPLAY example will find and use an appropriate CENTRE value for each displayed image. If you want to use the same CENTRE value for all images you should specify it explicitly on the command line, for instance:

      % display in="../a*" mode=perc centre="’12:00:00 -32:00:00’" accept

17.3 Output Parameters

If you tried out the examples at the start of this section, you may be wondering what happened about the output parameters for STATS. The STATS application writes the various statistics it calculates to lots of output parameters, which can be used by subsequent applications. If an application is re-run several times to process different data files then the values left in the application’s output parameters will be the values created on the last invocation of the application.

17.4 What Happens if an Error Occurs?

If an application fails to execute successfully, the error report will be displayed, and then cancelled.24 This means that any remaining NDFs will continued to be processed normally.

The exception to this is that if the error is an ‘abort request’ (caused by supplying two exclamation marks for a parameter), then the loop exits immediately. That is, no remaining NDFs are processed.

17.5 What about Applications that Re-use Parameters?

Some applications use a single parameter to obtain a series of values from the user. Examples are the INIT parameter of CENTROID and the OPTION parameter of SETEXT. Remembering that parameters that are not associated with either an NDF or a positions list retain their values between invocations, it is not surprising that care is needed when using such application to process groups of NDFs. For instance, when using CENTROID you supply a null parameter value (i.e. a single exclamation mark !) as the final value for the INIT parameter to indicate that you do not wish to find any more centroids. Since parameter values are retained between invocations when processing groups of NDFs, this null value becomes the first value to be used by any subsequent invocation. The next invocation of CENTROID finds INIT set to a null value, assumes that no more centroids are to be found, and exits immediately! The same goes for all subsequent invocations until the group of NDFs has been exhausted.

The only way (currently) to avoid this behaviour is to specify the INIT parameter value on the command line. CENTROID takes this as an indication that you only want to find a single centroid, and so does not attempt to get a new value for INIT, thus leaving the supplied value for the next invocation. The same value for INIT is thus used by all invocations. Of course, this means you can only find a single centroid in each NDF.25

Most applications that re-use one or more parameters during a single invocation have some similar means of indicating that you do not want to be prompted for a new value. For some (like CENTROID), putting the parameter value on the command line accomplishes this. Some others (such as SETEXT) have a LOOP parameter that can be set FALSE to indicate that parameters should not be accessed more than once. The reference documentation for each command should be consulted for details.

17.6 Introducing a Pause Between Invocations

Sometimes you may want to slow down the speed at which data files are processed. For instance, if you display several small images using a single DISPLAY command, you may want time to examine each image before moving on to the next. You can introduce a delay between invocations by setting the shell environment variable KAPPA_LOOP_DELAY to the required delay time (in units of seconds). For instance, in the C-shell:

      % setenv KAPPA_LOOP_DELAY 2.5

causes a delay of 2.5 seconds between invocations of any Kappa command. To remove the delay, you should undefined KAPPA_LOOP_DELAY. In C-shell:

      % unsetenv KAPPA_LOOP_DELAY

17.7 Reporting the Data Files being Processed

When processing a single data file, some applications report the name of the file and some do not. Normally, no extra information is given when processing groups of data files. This means that sometimes you get to see the names of the files as they are processed, and some times you do not. It just depends on the application.

However, it is often very useful to see the names of the files as they are processed. For instance, if an error occurs processing one of the files, it is useful to know which file failed. If the application doesn’t display this information, then you can force it to by setting the shell environment variable KAPPA_REPORT_NAMES to an arbitrary value.26 For instance, in the C-shell:

      % setenv KAPPA_REPORT_NAMES 1

This causes the value used for each data file parameter to be displayed in the form "parameter = value" on each invocation. To go back to the normal, quiet reporting scheme, you should undefined KAPPA_REPORT_NAMES. In C-shell:

      % unsetenv KAPPA_REPORT_NAMES

17.8 The Syntax for Specifying Groups of Data Files

The group of NDFs or positions lists to be used for a given parameter is specified by a group expression. This is also the syntax used to give groups of plotting attributes when specifying graphics STYLE parameters. The group expression syntax is described in Section 4.13.

A group of output data files may be specified by modifying the names of a corresponding set of input data files. This is easy enough when the application only has one parameter for input data files, but what happens if more than one parameter is associated with a group of input data files? Which parameter is used to define the group of input data files on which the names of the output data files are based? The answer is “the first one to be accessed”. For instance, ADD takes two input NDFs, adds them together and produces a single output NDF. When running ADD, you are prompted first for Parameter IN1, and then for Parameter IN2, and finally for Parameter OUT. Thus, if you give the string "a_"* for OUT, the names of the output NDFs will be derived from the NDFs supplied for Parameter IN1, because IN1 is accessed first (i.e. prompted for before IN2).

Note, when choosing the input parameter on which output data files are based, no significance is attached to whether the input and output file types match. That is, the first input parameter to be accessed if used, irrespective of whether it is associated with an NDF or a positions list.

A feature that may sometimes be useful is the facility for providing a shell command in response to a prompt for a group of data files. To do this, enclose the command within the usual backward quotes (), as you would when substituting the output from a command into another shell command. The command should generate a set of explicit file names, with file types. Note, you will need to escape any characters that are normally interpreted as part of the syntax of a group expression, such as "|" or ",", by preceding them with a backslash "\".

17.9 Using non-NDF Data Formats

In addition to processing Starlink NDF structures, Kappacan also process many non-NDF (‘foreign’) data files. This is achieved through ‘on-the-fly conversion’ (see Section 18.1 and SUN/55).

When this scheme is in use, you need not include explicit file types for all input file names. If no file type is given, the file with the highest priority file type amongst all files with the specified base name will be used. The priority of a file type is determined by its position within the list of file types given by ?? environment variable (see Section 18.1). File types near the start of the list have higher priority than those that follow. Note, native NDF files always have the highest priority and will be used (if they exist) in preference to all other files types.

17.10 Disabling Multiple Invocations of Applications

In certain circumstances, you may possibly want to disable the automatic re-invocation of Kappa applications to process groups of data files. This can be done by setting the environment variable KAPPA_LOOP_DISABLE to an arbitrary value.27 For instance, in the C-shell:

      % setenv KAPPA_LOOP_DISABLE 1

will cause all NDF and positions list parameters to accept only a single data file, and each application will be run only once. Note, the extra facilities for specifying data files provided by the group expression syntax will not then be available. To re-enable looping, you should undefine KAPPA_LOOP_DISABLE. In C-shell:

      % unsetenv KAPPA_LOOP_DISABLE

24This is called ‘flushing’ the error.

25If this is a problem, you can always put the INIT values into a file or positions list, using a different value for the MODE parameter.

26The actual value does not matter.

27The actual value does not matter.