Differences between revisions 32 and 33
Deletions are marked like this. Additions are marked like this.
Line 4: Line 4:
 * The new HDS-V5 data format (which uses HDF5) is now available. In this release the default will be to write all files as HDS-V4, unless the environmental variable HDS_VERSION is set to 5. This is described in Jenness 2015 [[https://ui.adsabs.harvard.edu/#abs/2015A&C....12..221J/abstract|2015A&C....12..221 .]] We are planning to make this file format the default starting with our next release, so we would please request any issues or comments be sent on to us: either on the [[https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STARLINK|Starlink users' mailing list]], the [[https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STARDEV|Starlink developers' mailing list]], or sent directly to EAO with the helpdesk AT eaobservatory.org email address.  * The HDS-V5 data format (which was introduced in the 2018A release and uses HDF5) is now the default output format for all NDFs and other HDS files created by Starlink. It is still possible to create output files using the old HDS-V4 format by setting the environmental variable "HDS_VERSION" to 4 before running the command to create the file. The HDS-V5 format is described in Jenness 2015 [[https://ui.adsabs.harvard.edu/#abs/2015A&C....12..221J/abstract|2015A&C....12..221 .]]
Line 11: Line 11:
 *  * All cupid commands now support NDFs containing more than 2^32 pixels.
 * Some parts of the code have been multi-threaded to improve execution speed.
 * The [[http://starlink.eao.hawaii.edu/docs/sun255.htx/sun255ss5.html|FINDCLUMPS|class=white]] command has a new configuration parameter called !GaussClumps.AllowEdge. It controls whether clumps found by the !GaussClumps algorithm that touch the edge of the array are retained or not.
 * The FINDCLUMPS command has a new option, "Ellipse3", for the SHAPE parameter (which determines the shape of the clump outlines stored in the output catalogue). It provides a more robust alternative to the older options "Ellipse" and "Ellipse2".
 * Several bugs have been fixed.
Line 17: Line 21:
 * A new program "COMPLEX" was added to manipulate complex NDFs, e.g. to extract the real and imaginary parts, compute the modulus and argument or to combine these parts into a new complex NDF.
 * ARDMASK can now process complex data.
 * Various bugs have been fixed.
 * The following commands can now handle NDFs containing over 2^31 pixels (some have been multi-threaded for increased speed): ADD, CADD, CDIV, CHANMAP, CMULT, COLLAPSE, CSUB, DIV, FITSURFACE, MFITTREND, MSTATS, MULT, NDFCOPY, NDFTRACE, NUMB, PERMAXES, STATS, SUB, SURFIT, WCSALIGN.
 * A new command "[[https://starlink.eao.hawaii.edu/devdocs/sun95.htx/sun95ss28.html|COMPLEX|class=white]]" was added to manipulate complex NDFs, e.g. to extract the real and imaginary parts, compute the modulus and argument or to combine these parts into a new complex NDF.
 * New command "[[https://starlink.eao.hawaii.edu/devdocs/sun95.htx/sun95ss117.html|MOCGEN|class=white]]" generates a [[https://www.ivoa.net/documents/MOC/|MOC|class=white]] covering pixels in a 2D NDF.
 * New command "[[https://starlink.eao.hawaii.edu/devdocs/sun95.htx/sun95ss156.html|PIXBIN|class=white]]" performs generalised rebining of an NDF by reading destination coordinates for each pixel from a set of associated NDFs.
 * [[https://starlink.eao.hawaii.edu/devdocs/sun95.htx/sun95ss5.html|ARDMASK|class=white]] can now process complex data.
 * [[https://starlink.eao.hawaii.edu/devdocs/sun95.htx/sun95ss72.html|HISCOM|class=white]] has a new parameter called APPNAME that specifies the application name to be stored in the new history record (previously, "HISCOM" was always used).
 * [[https://starlink.eao.hawaii.edu/devdocs/sun95.htx/sun95ss83.html|LINPLOT|class=white]] parameter XMAP now has a new option called "LRLinear", which forces the X axis to be annotated linearly increasing from left to right.
 * [[https://starlink.eao.hawaii.edu/devdocs/sun95.htx/sun95ss85.html|LISTSHOW|class=white]] can now display NDF data values at the positions in the supplied positions list.
 * [[https://starlink.eao.hawaii.edu/devdocs/sun95.htx/sun95ss122.html|NDFCOMPRESS|class=white]] parameter METHOD can now be set to "BOTH", causing the NDF to be compressed twice, once using method SCALED and then again using method DELTA.
 * [[https://starlink.eao.hawaii.edu/devdocs/sun95.htx/sun95ss169.html|REMQUAL|class=white]] has new parameter CLEAR, which allows the quality bits corresponding to the removed quality names to be cleared.
 * [[https://starlink.eao.hawaii.edu/devdocs/sun95.htx/sun95ss173.html|SCATTER|class=white]] can now display a symmetric linear fit to the points in the scatter plot.
 * [[https://starlink.eao.hawaii.edu/devdocs/sun95.htx/sun95ss193.html|SQORST|class=white]] has new parameter CENTRE, which allows the centre of the squash or stretch to be specified.
 * [[https://starlink.eao.hawaii.edu/devdocs/sun95.htx/sun95ss196.html|SUBSTITUTE|class=white]] has a new parameter called LUT, which allows multiple values to be changed simultaneously.
Line 24: Line 41:
 *  * [[https://starlink.eao.hawaii.edu/devdocs/sun223.htx/sun223ss13.html|POLSELECT|class=white]] now aligns the supplied mask and catalogue in sky coordinates rather than pixel coordinates.
 * The output catalogue column values stored by [[https://starlink.eao.hawaii.edu/devdocs/sun223.htx/sun223ss16.html|POLVEC|class=white]] for pixels that have negative total intensity have been changed. Previously all output values were set bad at such pixels. Now only P (percentage polarisation) is set bad.
 * A new option has been added to POLVEC and POLBIN that allows P and PI (polarised intensity) values to be debiased using the "modified asymptotic estimator". See https://arxiv.org/pdf/1407.0178.pdf section 2.5.
 * A new command called [[https://starlink.eao.hawaii.edu/devdocs/sun223.htx/sun223ss4.html|POLEDIT|class=white]] has been added, which allows an existing vector catalogue to be changed in various ways. These include changing column values, adding new columns, changing the debiasing method and recalculating all columns derived from I, Q and U.
 * The [[https://starlink.eao.hawaii.edu/devdocs/sun223.htx/sun223ss11.html|POLPLOT|class=white]] command can now draw all vectors with a constant specified fixed length.
Line 27: Line 48:
 *  * Various bugs have been fixed.
 * New command [[https://raw.githubusercontent.com/Starlink/starlink/master/applications/smurf/scripts/matchbeam.py|MATCHBEAM|class=white]] smooths a 450 um SCUBA_2 map so that it has the resolution of an 850 um map.
 * New command [[https://raw.githubusercontent.com/Starlink/starlink/master/applications/smurf/scripts/pol2noise.py|POL2NOISE|class=white]] can [[https://www.eaobservatory.org/jcmt/2020/04/checking-errors-in-pol2-vector-catalogues/|verify|class=white]] or [[https://www.eaobservatory.org/jcmt/2020/06/re-modelling-the-error-estimates-in-a-pol2-vector-catalogue/|re-model|class=white]] the error values in a vector catalogue created by POL2MAP.
 * New command GAU2FIT determines a 2-component Gaussian PSF from a map of a compact source of known diameter (e.g. a planet).

 * [[https://starlink.eao.hawaii.edu/docs/sun258.htx/sun258ss5.html|CALCQU|class=white]]:
   * The half-wave plate origin given by configuration parameter !PaOff now has separate default values for 450 um and 850 um, which differ by about 4 degrees.
   * Fix a bug that could result in [[https://www.eaobservatory.org/jcmt/2018/07/pol-2-data-reduction-fix-for-source-blurring/|sources being smeared out|class=white]] in the output time-stream data.

 * [[https://starlink.eao.hawaii.edu/docs/sun258.htx/sun258ss67.html|CONFIGMELD|class=white]]:
   * New parameter APP allows the application from which the configuration should be read to be specified. Previously this was fixed at "Makemap", but now for instance it can be set to "skyloop" or "calcqu".

 * [[https://starlink.eao.hawaii.edu/docs/sun258.htx/sun258ss38.html|MAKECUBE|class=white]]:
   * A new parameter called POINTING allows pointing corrections to be applied as the data is gridded into a cube.

 * [[https://starlink.eao.hawaii.edu/docs/sun258.htx/sun258ss39.html|MAKEMAP|class=white]]:
   * If an error occurs, attempt to create maps from any remaining chunks before quitting. Previously, any remaining chunks were ignored.
   * A new add-on dimmconfig file called "dimmconfig_pca.lis" allows removal of multiple common modes using principal component analysis. This can produce maps with flatter backgrounds.
   * Improvements to diagnostics system.

 * [[https://starlink.eao.hawaii.edu/docs/sun258.htx/sun258ss73.html|POL2MAP|class=white]]:
   * All the above CALCQU and MAKEMAP changes are also incorporated into POL2MAP.
   * The default Instrumental Polarisation (IP) model has been [[https://www.eaobservatory.org/jcmt/2020/04/change-to-the-default-ip-model-used-by-pol2map/|changed from JAN2018 to AUG2019|class=white]].
   * New parameter WEIGHTLIM specifies the lowest weight for usable observations, in the range 0.0 to 1.0 (default 0.05).
   * A History record describing the invocation of POL2MAP is now added to all output NDFs.
   * Do not include lists of input NDFs in the log file by default.
   * New parameter TRIM to trim noisy edges off the output coadded maps.
   * Allow user control of the SNR levels used to define the AST and PCA masks.
   * New parameter SMOOTH450 causes 450 um data to be smoothed to 850 um resolution.
   * Debiasing may now be performed using the "modified asymptotic estimator". See https://arxiv.org/pdf/1407.0178.pdf section 2.5.
   * A new column called AST has been added to the output vector catalogues. It indicates which island within the AST mask (if any) contains each vector.
   * If the total intensity map comes from a non-POL2 observation, degrade it by a factor of 1.35 before using it to take account of losses within POL-2.
   * An error is now reported if any map supplied as input to POL2MAP has a pixel size that is different to the value specified by parameter PIXSIZE.

 * [[https://starlink.eao.hawaii.edu/docs/sun258.htx/sun258ss72.html#Q1-81-283|SKYLOOP|class=white]]:
   * All the above MAKEMAP changes are also incorporated into SKYLOOP.
   * Shortmaps and bolomaps are now created if they are requested via parameter CONFIG.
   * New parameter INITIALSKY allows an initial guess at the sky map to be supplied.
   * Now honours the !Ast.Mapspike_Freeze configuration parameter.

 * [[https://starlink.eao.hawaii.edu/docs/sun258.htx/sun258ss66.html|UNMAKEMAP|class=white]]:
   * Accept raw data as input (previously input data needed to have been flatfielded first).

Starlink Software Collection — 2021A Release Notes

Highlights

  • The HDS-V5 data format (which was introduced in the 2018A release and uses HDF5) is now the default output format for all NDFs and other HDS files created by Starlink. It is still possible to create output files using the old HDS-V4 format by setting the environmental variable "HDS_VERSION" to 4 before running the command to create the file. The HDS-V5 format is described in Jenness 2015 2015A&C....12..221 .

  • Perl updated to Version 5.32.0.

Changes to Applications

CUPID (ascl:1311.007)

  • All cupid commands now support NDFs containing more than 2^32 pixels.
  • Some parts of the code have been multi-threaded to improve execution speed.
  • The FINDCLUMPS command has a new configuration parameter called GaussClumps.AllowEdge. It controls whether clumps found by the GaussClumps algorithm that touch the edge of the array are retained or not.

  • The FINDCLUMPS command has a new option, "Ellipse3", for the SHAPE parameter (which determines the shape of the clump outlines stored in the output catalogue). It provides a more robust alternative to the older options "Ellipse" and "Ellipse2".
  • Several bugs have been fixed.

GAIA (ascl:1403.024)

KAPPA (ascl:1403.022)

  • Various bugs have been fixed.
  • The following commands can now handle NDFs containing over 2^31 pixels (some have been multi-threaded for increased speed): ADD, CADD, CDIV, CHANMAP, CMULT, COLLAPSE, CSUB, DIV, FITSURFACE, MFITTREND, MSTATS, MULT, NDFCOPY, NDFTRACE, NUMB, PERMAXES, STATS, SUB, SURFIT, WCSALIGN.
  • A new command "COMPLEX" was added to manipulate complex NDFs, e.g. to extract the real and imaginary parts, compute the modulus and argument or to combine these parts into a new complex NDF.

  • New command "MOCGEN" generates a MOC covering pixels in a 2D NDF.

  • New command "PIXBIN" performs generalised rebining of an NDF by reading destination coordinates for each pixel from a set of associated NDFs.

  • ARDMASK can now process complex data.

  • HISCOM has a new parameter called APPNAME that specifies the application name to be stored in the new history record (previously, "HISCOM" was always used).

  • LINPLOT parameter XMAP now has a new option called "LRLinear", which forces the X axis to be annotated linearly increasing from left to right.

  • LISTSHOW can now display NDF data values at the positions in the supplied positions list.

  • NDFCOMPRESS parameter METHOD can now be set to "BOTH", causing the NDF to be compressed twice, once using method SCALED and then again using method DELTA.

  • REMQUAL has new parameter CLEAR, which allows the quality bits corresponding to the removed quality names to be cleared.

  • SCATTER can now display a symmetric linear fit to the points in the scatter plot.

  • SQORST has new parameter CENTRE, which allows the centre of the squash or stretch to be specified.

  • SUBSTITUTE has a new parameter called LUT, which allows multiple values to be changed simultaneously.

ORAC-DR (ascl:1310.001)

ORAC-DR 2021A change log

POLPACK (ascl:1405.014)

  • POLSELECT now aligns the supplied mask and catalogue in sky coordinates rather than pixel coordinates.

  • The output catalogue column values stored by POLVEC for pixels that have negative total intensity have been changed. Previously all output values were set bad at such pixels. Now only P (percentage polarisation) is set bad.

  • A new option has been added to POLVEC and POLBIN that allows P and PI (polarised intensity) values to be debiased using the "modified asymptotic estimator". See https://arxiv.org/pdf/1407.0178.pdf section 2.5.

  • A new command called POLEDIT has been added, which allows an existing vector catalogue to be changed in various ways. These include changing column values, adding new columns, changing the debiasing method and recalculating all columns derived from I, Q and U.

  • The POLPLOT command can now draw all vectors with a constant specified fixed length.

SMURF (ascl:1310.007)

  • Various bugs have been fixed.
  • New command MATCHBEAM smooths a 450 um SCUBA_2 map so that it has the resolution of an 850 um map.

  • New command POL2NOISE can verify or re-model the error values in a vector catalogue created by POL2MAP.

  • New command GAU2FIT determines a 2-component Gaussian PSF from a map of a compact source of known diameter (e.g. a planet).
  • CALCQU:

    • The half-wave plate origin given by configuration parameter PaOff now has separate default values for 450 um and 850 um, which differ by about 4 degrees.

    • Fix a bug that could result in sources being smeared out in the output time-stream data.

  • CONFIGMELD:

    • New parameter APP allows the application from which the configuration should be read to be specified. Previously this was fixed at "Makemap", but now for instance it can be set to "skyloop" or "calcqu".
  • MAKECUBE:

    • A new parameter called POINTING allows pointing corrections to be applied as the data is gridded into a cube.
  • MAKEMAP:

    • If an error occurs, attempt to create maps from any remaining chunks before quitting. Previously, any remaining chunks were ignored.
    • A new add-on dimmconfig file called "dimmconfig_pca.lis" allows removal of multiple common modes using principal component analysis. This can produce maps with flatter backgrounds.
    • Improvements to diagnostics system.
  • POL2MAP:

    • All the above CALCQU and MAKEMAP changes are also incorporated into POL2MAP.
    • The default Instrumental Polarisation (IP) model has been changed from JAN2018 to AUG2019.

    • New parameter WEIGHTLIM specifies the lowest weight for usable observations, in the range 0.0 to 1.0 (default 0.05).
    • A History record describing the invocation of POL2MAP is now added to all output NDFs.
    • Do not include lists of input NDFs in the log file by default.
    • New parameter TRIM to trim noisy edges off the output coadded maps.
    • Allow user control of the SNR levels used to define the AST and PCA masks.
    • New parameter SMOOTH450 causes 450 um data to be smoothed to 850 um resolution.
    • Debiasing may now be performed using the "modified asymptotic estimator". See https://arxiv.org/pdf/1407.0178.pdf section 2.5.

    • A new column called AST has been added to the output vector catalogues. It indicates which island within the AST mask (if any) contains each vector.
    • If the total intensity map comes from a non-POL2 observation, degrade it by a factor of 1.35 before using it to take account of losses within POL-2.
    • An error is now reported if any map supplied as input to POL2MAP has a pixel size that is different to the value specified by parameter PIXSIZE.
  • SKYLOOP:

    • All the above MAKEMAP changes are also incorporated into SKYLOOP.
    • Shortmaps and bolomaps are now created if they are requested via parameter CONFIG.
    • New parameter INITIALSKY allows an initial guess at the sky map to be supplied.
    • Now honours the !Ast.Mapspike_Freeze configuration parameter.
  • UNMAKEMAP:

    • Accept raw data as input (previously input data needed to have been flatfielded first).

SOURCEPLOT

  • The included JCMT source catalog has been updated.

SPLAT (ascl:1402.007)

STILTS (ascl:1105.001)

STILTS version history

TOPCAT (ascl:1101.010)

  • Updated to Version ????.

TOPCAT version history

Changes to Libraries

ARY

AST (ascl:1404.016)

CVG

  • File extensions are now retained in CADC style provenance headers.

GRP

HDS

PCS

STIL

  • Updated to Version ????.

STIL version history

CFITSIO

  • Updated to v3.450.

Miscellaneous Changes

Changes to Cookbooks and Guides

  • SC/20
  • SC/21
  • SC/22

Known issues

  • There is a known threading problem that prevents the Java application Treeview from working on HDS Version 5 (HDF5) files.

Starlink: 2021A (last edited 2021-11-19 20:31:25 by GrahamBell)