Skip to content
Code Releases Docs (2.10.0) Docs (Dev)

This is the changelog for MeteoIO release 2.6.0

Build and packaging system

  • requiring cmake>2.8.4 in order to prevent a (quite intrusive) warning under cygwin (this means that debian>=squeeze should have a proper version, ie distributions after 02.2011).
  • osX:
    • improved packaging
    • much improved support for clang, many warnings addressed
    • the generated Xcode projects were invalid because of an escaped character, this has been fixed
    • support for PNGIO plugin out of the box
  • Windows:
    • removed flto for gcc on Windows (since this is currently very, very buggy on Windows)
    • moved within an ifdef
  • Linux
    • there is no need to keep the "hera" target since it can now be handled by standard arguments
    • small cleanup to please the Intel compiler on Cray
    • support the Cray compiler. Please note that for Cray, you most probably have to compile with static linking!
    • fixed some warnings found by the Cray compiler (see http://bojolais.livejournal.com/222428.html).
  • FreeBSD: A warning has been fixed for Clang on FreeBSD by setting a compilation flag only for one specific file

Code cleanup and restructuring

  • all the coordinate systems support methods have been moved to a new CoordsAlgorithms static class
  • all the dataGenerators have been moved into a dataGenerators directory with an individual cc file for each generator. A template has been created to help in creating new data generators.
  • the resampling algorithms have also been moved to their own subdirectory for more clarity
  • the spatial interpolations have been put in their own subdirectory
  • the whole architecture of IOInterface has been changed: this is not a purely virtual class anymore, it now always implements the "nothing implemented here" exception. This makes the plugins simpler
  • all the methods defined in FileUtils have been placed within a FileUtils namespace for clarity
  • the code for handling COPY directives was very old and quite different from the code for handling KEEP or EXCLUDE directives. It has been re-written to follow the same principles as KEEP/EXCLUDE.
  • trying to bring clarity to the data creator / data generator concept... The API has been slightly change (renaming both "generate" into a "generate" and a "create")
  • big cleanup in the headers to prevent including things that are not absolutely necessary (and specially within header files)
  • the dataGenerators

New features

  • raw data processing
    • The MERGE feature has been much improved: the syntax now follows the (more logical) EXCLUDE/KEEP syntax
    • The EXCLUDE/KEEP/MERGE keys (as well as the associated stationIDs) are now case insensitive
    • The "merge" feature was not properly handling all situations, this should now be fixed.
    • The support for the '+' wildcard has been added to the KEEP and EXCLUDE keys.
    • When merging data from different stations, it is now possible to specify (with the Input::MERGE_STRATEGY key) how the merge should be done.
    • A new MOVE directive has been introduce in order to rename meteo parameters. This is processed before any other directive and could be very useful to handle for improperly named parameters (for example coming from GSN).
  • virtual stations
    • downscaling method: when declaring "downscaling" and providing virtual stations, grids are read and the virtual stations populated with values out of the grids.
    • Finally, proper handling of virtual stations: it is possible to temporally resample the data between spatially interpolated time steps. Now the spatial interpolations are only called every vstations_refresh_rate and otherwise the data is temporally resampled.
  • spatial interpolations
    • A new spatial interpolation method has been implemented: it takes an ALS and rescales the grid according to it
    • The CST spatial distribution algorithm has been split in two: CST that fills the grid with a truly constant value (as given in argument) and AVG that fills the grid with the average of the measurements. Similarly, there is now an AVG_LAPSE method.
    • Changing the rating of the AVG_LAPSE to be the exact same as IDW_LAPSE when only two stations are available, so the input order give precedence in this case (more logical).
    • Added a specific gradient for PSUM_PH
    • A new method for the spatial interpolations of ISWR has been implemented
    • The STD_PRESS spatial interpolation algorithm has been extended and can now deal with multiple input stations (with 2 different methods for handling this case)
    • When using IDW_Lapse, a basic check is performed on the arguments in order to evaluate the rating: if less than 2 stations can provide a given parameter and no lapse rate is provided, the rating is set to zero
    • The "RH" interpolation has been renamed as "LISTON_RH" for clarity.
    • the ListonRH algorithm has been tweaked: when only 1 station is available, no detrending is applied (instead of having the detrending returning an error if no default lapse rate has been provided).
  • filters
    • Added a "NONE" filter so it is possible to overwrite filters when importing another ini file.
    • implementation of the algorithms described in Tilg, A.-M., Marty C. and G. Klein, 2015: An automatic algorithm for validating snow depth measurements of IMIS stations. Swiss Geoscience Meeting 2015
    • adding a new processing: AGGREGATE. This replaces several of the existing filters as well as brings new features
    • new SHADE filter to apply a horizon mask to ISWR
    • The SUPPR filter can now take as argument a file containing a list of station IDs and timesteps where the parameter should be deleted
    • Allowing the "soft" keyword to be upper/lower case and anywhere in the argument list for filters.
    • The Butterworth filter has been replaced by a critically damped filter implemented as IIR that properly corrects for the phase (ie behaves as zero-phase filter) and options have been added to use it as either low or high pass filter
    • A new filter has been implemented that compares the measured ISWR with the potential radiation. This would filter out stations that are shaded, stations that fill missing data with zeroes, etc
  • plugins
    • new plugin to read Matlab files prepared by the OSHD project/service, based on libmatio for reading Matlab binary files
    • "append" mode for SMET files
    • support for color, description, units, min and max in smet files in SMETIO and smet2agr
    • IMISIO: The handling of the ANETZ precipitation has been fully redone in order to be based on hourly correlations (instead of the original 6 hours).
  • data generators / data creators
    • Now the dataCreators are defined in the [Input] section with a {parameter}::create = {algorithm} statement. They behave the same as the dataGenerators but are called +before+ the filtering takes place and always work with the whole vector of data (ie whole dataset)
    • The Karsten method for converting cloud cover (in okta) to clearness index has been implemented and is also available as a data generator (looking for a "CLD" parameter to use as cloud cover data)
    • the TAU_CLD generator can now generate solar index values by comparing the measured ISWR to the potential radiation. Moreover, when using a CLD parameter, it must now be between 0 and 8 instead of 0 and 1.
    • The IswrAlbedo generator now only uses stations that have HS except if provided with the "FORCE" argument.
    • A clear sky, potential short wave generator has been implemented as this is very useful when working with SW radiation
  • temporal resampling
    • Solar radiation resampling
    • A new parameter ENABLE_RESAMPLING has been added in the [Interpolations1D] section. It is now possible to globally turn resampling off...
  • general
    • a new MeteoGrids has been added (TSOIL), Handle TSOIL output with customized gradients
    • A new MeteoGrid has been created, RSNO to contain the snow mean density
    • the key BUFF_CHUNK_SIZE has been renamed as BUFFER_SIZE for more clarity and its default value is now 370 (which makes more sense for most applications)
    • Added more options to the parsing of dates (like NOW or NOW+3600)

New API

  • the "printlatlon()" method has been replaced by toString(Coords::LATLON).
  • making the grid operators "const" when possible
  • add a new argument to gridify() in order to be able to retrieve the list of rejected coordinates.
  • added a getHostName() function in IOUtils
  • add a few string formats options to Coords for easier use in error messages
  • a new method "getParameterIndex()" has been written for MeteoGrids for a future use.
  • getSolarTimeOfDay()
  • in Coords, the lat/lon are checked for validity in setLatLon() in order to catch grossly out of range coordinates
  • The MOD operator has been added to Date (it takes a number of seconds in argument)
  • A resampling method has been added to the Grid2DObject so it is possible to change the cellsize
  • Introducing a new "GRID3D" key...
  • added the atmospheric corrections logic (from Alpine3D) to the Sun object so it can easily be reused.
  • For convenience, the "size()" method has been defined. It returns the allocated number of element so it can be used when iterating over all the cells of an array/grid.
  • Small optimization into Sun: if setting the lat/lon to the same values as internally stored, nothing is changed (so it does not costs anything).
  • The getPointsBetween() methods have been removed from DEMObject since they were not really used and not really reliable either
  • the sky view factor has been implemented in getCellSkyViewFactor()
  • The method "waterSaturationPressure" has been renamed as "vaporSaturationPressure" and a reference added.
  • convenience method in StationData (getAltitude).
  • The Buffer class has been expanded with a new push() call
  • now the readDirectory function can perform a recursive search

Bug fixes

  • plugins
    • Code cleanup and fix for IMIS: now the ENET stations receive the number 1 for the automatic stations instead of 0.
    • The NetCDFIO plugin can now properly handle multiple files in one directory specified by "GRID2DPATH" or one single file specified by "GRID2DFILE"
    • A bug has been found in NetCDFIO: it should look for the GRID2D key, not the METEO key
    • A bug has been fixed in the NetCDF plugin that prevented reading some DEMs. The ability to remap parameters (between NetCDF and MeteoGrids) in the ini file has been added in order to add flexibility. The support for "years" and "months" time units has been implemented in a way that is compatible with the standard NetCDF tools.
    • The NetCDF plugin does not require anymore to be configured for the inputs in order to produce output grids.
    • First fixes for NetCDF so it could be used by Alpine3D: a bug when handling precipitation grids has been fixed (nodata now remain nodata) and the data packing has been removed (since when appending new grids it can simply not work).
    • the CNRMIO plugin was not properly declared.
    • Adding uref and zref as well as coding time as number of seconds since a reference time (start of the day of the first data point) in order to satisfy CNRM's requirements
    • The CNRMIO pluggin has been changed by removing the grids support (this should be performed by the generic NetCDF plugin) and adding what is necessary for Crocus to be able to use the generated meteo time series.
    • The SMET plugin was not consistent with the specification: when no time zone was provided, it should have assume tz=0 but it used the Input::Time_Zone value instead. This has been changed so it properly follows the specification: no time zone -> gmt. When writing data out, the time zone will now always be written.
    • Making the plugins thread safe by removing the global fin/fout. (ARC, SMET, PGM...)
    • Lots of small improvements and bug fixes for reading ARPS files
    • When opening the xml files with a forced encoding, the files were not being properly closed, leading to a possible "Too many open files" error message
    • A bug has been fixed in CosmoXMLIO: it was reading all the data with a time-shift (ie always reading the file +after+ the timestep that it should have read).
    • IMISIO: The snowpack_psum versus imis_psum behavior was not in agreement with the documentation... This has been fixed and should behave in a more appropriate way
  • filters and resampling
    • The Nakamura and Huwald algorithms in Unventilated_T were not properly implemented...
    • The PSUMDistribute "filter" was not working properly on some data, this has been fixed
    • The Accumulate resampling now fills the first nodata with 0 (since we know that we don't have enough information to re-accumulate at the first point)
    • add/mult filters: There was a bug when applying daily corrections: the juliand day numbers start at 1, not 0!
    • A tricky bug has been found when accumulating precipitation beyond the original data set: the accumulation was creating "random" precipitation by accumulating data that did not exist.
    • An invalid check (< when <= was needed) has been fixed in Meteo1DInterpolator (this was leading to segfaults when a buffer only contained 1 data point).
    • A possible bug has been addresses in libinterpol1D that may have affected a simulation on an ETHZ cluster.
  • others
    • the USER interpolation method should now work as expected. Better documentation for the USER interpolation method (grids are read from GRID2DPATH or a provided sub-dir) and better info message
    • A bug/inconsistency has been fixed in the Winstral algorithm: when pixels are excluded because their air temperature is too high for snow, they should not contribute to the min/max erosion/deposition!
    • By simplifying MeteoData, the memory consumption of a benchmark such as data_converter is reduced by almost 20%
    • Better handling of the i,j,k index in Coords objects, better strategy for handling invalid POI (the invalid points can be marked as invalid but kept in the vector)
    • Fixed a bug in Coords and added support for comments in A3D's POI file
    • Fixed a bug in gridify(): cells exactly on the upper/right border where not marked as invalid (thanks Aurélien!)
    • The horizon computation in the DEMObject has been fully redone, copying what is done in Alpine3D. This should be more accurate and much more efficient.
    • A bug in the MeteoData::mergeTimeSeries() method has been fixed that was related to the handling of extra parameters.
    • Fixed a bug in libinterpol2D
    • Fixed a bug: since the variance could be nodata, it should be checked before taking the sqrt
    • A bug has been fixed: there was a confusion between the mean adiabatic lapse rate and the dry one
    • The sampling rate estimate was quite wrong for small files, this has been improved
    • Fixed two bugs: one member of DEMObject was not properly serialized (for MPI)
    • A bug in the AllSkySW generator has been fixed as it was not using TAU_CLD if available in one of the two possible calls
    • The exclude / keep files were not properly handled in order to be case insensitive. This has been fixed in order to be consistent with the KEEP/EXCLUDE statements.
    • A tricky bug was found when copying data from the buffer back into a vector: the last element was missing
    • A bug has been found in IOUtils::seek that lead to returning npos when the vector only contained 1 element (and at the requested timestamp)
    • The Date::mod() method had some serious rounding errors that could accumulate. This was degrading the precision. The code has been replaced by a more robust implementation that should offer seconds precision (which is what the class offers).
    • Following a few coverity scans, several inconsistencies / potential bugs have been fixed.

documentation and error messages

  • Several exceptions have been renamed in order to be more generic (ex: instead of FileNotFound -> NotFound) and their text has been modified accordingly.
  • Improved the error message when subseting an array. Fixed documentation warnings.
  • More detailed error messages when the timestamps are not properly sorted
  • Better behavior of the GSN plugin when the data is missing but the metadata could be retrieved (so at least we know that the station is valid).
  • GSN: Small fix: invalid user/password now throws a proper exception (instead of the default exception that there is no data)
  • The GSN plugin now reports the errors with more relevant details (invalid sensor name, no permissions).
  • Updated the GSN example keys to reflect the new keys
  • raw data editing section in documentation
  • A few extra comments have been added to better understand the flow of data.
  • much improved error reporting in spatial interpolations
  • fixed documentation glitches in FilterMAD
  • Added a documentation section about Era Interim... Now it should be quite straightforward to build downscaled time series for virtual stations based on Era Interim.
  • More documentation about MeteoCH products
  • Some missing documentation for the TAU_CLD generator has been added.
  • Another tiny fix related to end of lines: if too many "empty" lines are found, this could be a problem with EOL encoding. This now throws an exception (after 1000 empty lines)

tools

  • smet_range: small improvement in order to properly handle directories containing sub-dirs
  • code cleanup and added the possibility to provide a date range to the smet_range script (there is no default "." directory anymore, the directory MUST be provided)
  • A new script has been created to help select SMET files based on the parameters they contain
  • A new tool has been contributed by Aurélien Gallice: a Python class to read SMET files
  • The getSynop script should belong to MeteoIO instead of Alpine3D, therefore it has been moved
  • Now data_converter takes the sampling rate (in hours) as third argument on the command line
  • An unpleasant bug has been fixed in data_converter when handling large number of stations and time ranges
  • The smet_extract script now writes its output to stdout instead of to a file since the previous behavior was mostly an anoyance...
  • More robust memory allocation in data_converter (as well as simpler)
  • smet2agr: support for color, description, units, min and max in smet files, nstead of forcing bach, now dash or bach can be invoked (it relies on /bin/sh)
  • Swapping the order of the tests hoping this could solve the issue of the "SMET plugin not found" when testing Snowpack (ie maybe the coverage compile breaks some of the plugins)