Skip to content
Code Releases Docs (2.10.0) Docs (Dev)

This is the changelog for MeteoIO release 2.8.0

This release introduces a fully rewritten spatial resampling and a fully rewritten NetCDF plugin

Plugins

  • OSHD:
    • More informations in DEBUG
    • checking for varying number of stations, support for large data gaps (if the data files generation stopped for a while)
  • NetCDF
    • full rewrite with support for "schemas", ie variants of NetCDF produced by different softwares (currently, CF-1.6, Amundsen, Crocus, Ecmwf, Wrf, MeteoCH)
    • support for various data types and time encoding as part of the schemas
    • support for timezone information in the reference date
    • support for writing timeseries for stations' data (ie at point scale), support for one station per file or multiple stations per file
    • support for ACDD search metadata
    • documentation fully rewritten and expanded
    • A new method has been implemented in libncpp in order to get a list of all variables, regardless of the dimensions they depend on.
    • Adding the slope/azimuth information to NetCDf-CF (although this is not defined in CF1)
    • High resolution time is now handled in NetCDF (it works properly up to 50 Hz)
  • CsvIO:
    • timestamps in descending order are transparently converted back to ascending order
    • Support for the (non-standard) time zones abbrevations (like CET, etc)
    • Some basic support for timezone information within the date/datetime field has been implemented. Currently, only numeric values are supported (so you have to replace fields such as "ECT" by "+01")
    • When reading a CSV file, it is possible to skip some fields as well as to use SPACE or TAB as delimiters
  • ImisIO:
    • The STB2 station is NOT using reduced pressure but absolute pressure instead (but there is no metadata to "know" it)
    • The IMIS plugin now sets the slope angle and azimuth and takes care of the few special stations (hard coded)
    • When using ANETZ precip, it was possible to replace 0. values by nodata (from ANETZ)
  • SMET
    • The SMET plugin can now scan for all smet files within METEOPATH (even recursively if desired) when no stations are defined (with the STATION# keys)
    • It is now possible to append to smet files with the SMETIO plugin (but only if the files contain the exact same fields and the timestamp is in the first column) and two new keys have been defined: SMET_APPEND (default: false) and SMET_OVERWRITE (default: true)
    • The SMET plugin now reads/writes the slope and and azimuth. The SMET specification has been expanded to cover these two new (optional) parameters
    • An option to derive the slope information from the staion ID naming scheme (IMIS/Snowpack operational) has been added
  • GRIB
    • Since Grib_API has been replaced by ecCodes, moving to ecCodes for reading GRIB files

Gridded data handling and spatial resampling

  • The VSTATION and GRID_EXTRACT spatial resampling were still lacking robustness, they have been mostly rewritten
  • The resampling strategy "VSTATION" has been renamed into "VSTATIONS" (ie. plural) for consistency
  • Now the real data end is used to build the virtual stations' "raw" data, so when the data stops at a given time, the spatial interpolations are not attempted anymore
  • A SMART_GRID extract method has been implemented
  • A GRID_SMART extract method has been implemented
  • It is now possible to perform grid resampling by using the GRID_RESAMPLING resampling strategy
  • Now the plugins that can read 2D grids can also provide a list of the available grids and timesteps. This allows gridded data extraction to only request the grids that exist
  • The GridsManager now performs conversion between parameters (ie generating a grid out of other grids, such as QI->RH, dem from pressures, etc)
  • The GridsManager implements the whole virtual stations and grid extractions initializations as well as their data extraction
  • The whole virtual stations processing has been moved out of the IOManager and into theMeteo2DInterpolator, so everything is in the same place

Filters and Data Generators

  • A mid-sized random numbers suite for statistical filters has been implemented
    • Added support for Mersenne Twister
    • Added the Gamma, Chi-Square, Student-t, Beta and Fisher deviates
  • Humidity and precipitation
    • There was an inconsistent use of specific humidity, either as SH or QI. It is now always QI
    • The RelHumGenerator has been expanded to convert from any AH/RH/QI/TD to any other and renamed as HumidityGenerator
    • The PPHASE generator has been renamed as PrecSplitting and can now generate the phase and/or convert between amount/phase and PSUM_L/PSUM_S
  • Wind
    • A new data generator has been created to generate VW, DW from the Wind velocity U, V components
    • A new despiking filter was added (Goring and Nikora, 2002). It works well on wind and even snow height timeseries
  • Other
    • a new filter has been created (ProcDeAccumulate) that can de-accumulate data when in timeseries form
    • Fixed an invalid check on an array index in WindowedFilter
    • Added a very basic Quantile Mapping filter

Bug fixes

  • Radiation computation
    • the sky view factors could get quite wrong, this has been corrected
    • Found a wrong constant in SunTrajectory as well as some errors when computing sunrise/sunset/sunNoon times
    • The sun test had some errors (wrong timezone) and its reference file had to be recomputed
  • DEM handling
    • A (major) bug has been found in the (Horn, 1981) slope computation algorithm: all computed azimuth were false (it looked like everything had been rotated)
    • flat DEM would lead to floating point exception (when computing slopes and in Winstral), this has been fixed
  • Files handling
    • When listing the content of a directory, dead links are now detected. Hidden files are excluded
    • The list of files returned by FileUtils::readDirectory() was not very clean when using recursion (some parasitic "./")
  • Others
    • When a station has very little data, there was a way to segfault when cutting a window for the filters. More checks have been implemented.
    • Undercatch corrections: the handling of the model validity range was not correct in Forland and missing in WMO
    • There was a mistake in the handling / definition of the TauCLD: when using CLD, it was not returning a transmissivity but a cloudiness
    • fixed a small autoscale bug when writing 2D grids as PNG (some data could be truncated)
    • Fixed a small memory leak in PNGIO
    • some checks have been added when the user specifies lat/lon coordinate system in order to show proper error messages when necessary
    • Corrected unitialized variable in IOHandler
    • The ConfigProxy Config::get() call was leading to possibily unitialized values when specifying IOUtils::nothrow (it was only guaranteed to behave properly when handling strings). It has been replaced by two calls: ConfigProxy Config::get(key, section) that throws if the key is not found or a template ConfigProxy Config::get(key, section, default) that returns "default" if the key is not found. This obviously could breaks existing code!
    • fixed bad units in VSTATIONS_REFRESH_OFFSET

Other improvements and new features

  • User visible
    • An automerge feature has been implemented, so it is possible to automatically merge all stations that have the same ID
    • The DATA_QA compilation option has been replaced by a configuration key: DATA_QA_LOGS in the [General] section (default: false)
    • The syntax for providing locations (for virtual stations, Csv plugin, etc) has been changed in order to also accomodate the elevation. Please update your ini files!
    • Adding the possibility to specifiy dmax (in m) for Winstral's algorithm in the config
    • The "support" for ETRS datum has been added to CoordsAlgorithms: all ETRS UTM coordinates are treated like WGS84 UTM coordinates (depsite the .5m error due to continental drift)
    • A new method, suitable to spatially interpolate Snowpack-generated snow heights has been implemented (IDW_SLOPE)
    • the Date class now supports up to 100Hz resolution
  • API
    • IOUtils
      • A new method "replace_all" has been implemented to replace substrings in strings (similar to Boost's)
      • New method to count how many times a substring appreas in a given string
      • A new method has been written to return the domain name of the current computer
    • ConfigObject: if only ';' or '#' are present after the '=', they will be read and returned (so it is possible to specify that a delimiter is ';')
    • A new "getBufferProperties" call has been implemented for the TimeSeriesManager class
    • It is now possible to specify DEMs' default slope algorithms by a string representation
    • A class to perform 1D polynomial regression was added
    • It is now possible to provide an elevation offset when computing an horizon (to support sensors mounted on a mast above the ground, for example)
    • Small formula for air pressure reduced to sea level
    • Coords constructor now uses northing and easting

Documentation

  • The workflow diagram has been updated to reflect the TimeSeriesManager usage
  • Added some documentation about virtual stations' workflow
  • Added a howto for implementing a resampling algorithm
  • Documentation restructuring to make raw data editing and spatial resampling more visible
  • Trying to improve the filter development documentation. The FilterBlock class has been removed, there are now only ProcessingBloc or WindowedFilter.

Cleanup

  • Removing IOUtils::getArraySliceParams() since this is only used in Alpine3D and does not really belong to MeteoIO
  • The 2D resampling is now in libresampling2D for consistency as well as in order to be less confusing with the new naming of "resampling" for virtual stations
  • removing all the MIO_API markings that never got used...
  • Since the DEMObject class was growing way too much, a new DEMAlgorithms (static) class has been defined
  • The file "exports.h" is not needed anymore (thanks to CMake), getting rid of it
  • Fixed some obsoletes uses of EXIT_SUCCESS / EXIT_FAILURE in A3DIO
  • Trying to reduce the cluter with many "interpolate" calls in order to handle all the possible options

Compilation

  • It was not possible to compile with LTO and static linking with gcc. This now works.
  • Disabling some (stupid) warnings of icc (pragma marked for clang should be silently ignored)
  • Correcting some Clang warnings (found on FreeBSD)
  • Finally, after almost 3 year, issue 485 can be closed: MeteoIO compiles again with Visual C++, thanks to the great work done by Kitware's CMake!
  • Finally, the last remaining issues for static linking have been fixed: now the tests properly link
  • Removed useless compiler warnings with Intel compilers
  • Allow to set ARCH_OPTIM with Intel compilers
  • Removed -Wunused-template with Clang, not working correctly