Skip to content
Code Releases Docs (2.10.0) Docs (Dev)

This is the changelog for MeteoIO release 2.10.0

Highligts

  • Moving MeteoIO to c++11
  • New Plugins:
    • MeteoBlue, enables the access to the MeteoBlue RESTful web service
    • SynthIO, produce synthetic data
    • DBO, enables the access to the DBO RESTful web service
  • The release/packaging Pipeline has been moved to gitlab
  • The whole Raw Data Editing has been renamed into Input Data Editing. It now has its own section: [DataEditing] so the [Input] section is now really focused on getting the data into the system and nothing else
  • Inishell and MeteoIO are not bundled anymore

Details

Plugins

  • Compilation issue of PmodIO was fixed
  • Adapting MeteoIO to read the new structure of OSHD matlab files
  • more wild encodings for units in Oshd
  • taking the raw cosmo TA, since now COSMO does bias corerction
  • A new MeteoBlue plugin that works quite well (but not all parameters are supported yet)
  • the radiation fields now rely on their "instant" variant in the MeteoBlue Plugin
  • Fixed a bug in MeteoBlue: extra parameter must be created even when they contain nodata
  • Many acdd fields were added
  • Bugfix of acdd where the geospatial metadata were not written as expected
  • ACDD_EMAIL key has been renamed into ACDD_CREATOR_EMAIL.
  • Added proxy support for CURL
  • The separation between low level curl / json parsing and the higher level logic is now much better done. This also cleanly prevents exposing curl / picoJson to the rest of MeteoIO through includes.
  • Not trying to parametrize ILWR in SNIO anymore
  • A new plugin SyntheticIO to generate synthetic data
  • Documentation and new synthethizers to the SYNTH plugin (STDPRESS...)
  • The SYNTH plugin has gained much improved configurability and abilities
  • It is now possible to define how to generate meteo parameters in the SYNTH plugin, per station and per parameter. Currently two methods are implemented (constant and step functions), more will come.
  • The filename parsing is now done with regex which should help with long term maintennace
  • A new synthesizer has been implemented in SynthIO: STDPRESS
  • WRF doesn't have "W10" output there is also no post-processing for it included. If needed, it must be fetched from the 4d "W" variable
  • Add missing dimension for WRF nc files
  • Massive code refactoring and rewrite in the DBO plugin
  • The logic has been changed, somes data structures have been removed while some others were expanded in the DBO plugin

SQL

  • Renaming of the WWCS plugin into MYSQLIO
  • now flexible enough to support multiple schemas
  • The pressures are in hPa in WWCS, converting them to Pa
  • there is a concept of profiles for the WWCS/MYSQL plugin
  • Configuring the parameters of the Mysql queries is now quite confortable and they are no redundancy between various data structures
  • Now using the view for getting WWCS data
  • More flexible structure, allowing to declare at a single place the meteodata query as well as the expected Mysql and MeteoData mapping
  • More error checking, cleaner code, no hacky memory handling
  • Support for the MYSQL_DATETIME_TYPE

NetCDF

  • Fix read-in of a NetCDF dates list
  • A new otpion has been added to the NetCDF plugin: NC_ALLOW_MISSING_COORDS
  • ERA5 request update and guide
  • Fixing an error appearing with NC_SINGLE_FILE=True
  • Added one more variant for the altitude dimension in NetCDF files
  • skip files without time dimension when listing grids.
  • now reads the dimension sequence for a variable, instead of assuming it is time, row, col.
  • Adding the option to split output NetCDF file by variable and/or year using the [Output] keys NETCDF_SPLIT_BY_VAR and NETCDF_SPLIT_BY_YEAR, respectively.
  • it is now possible to provide station IDs for the virtual stations generated by the spatial resampling
  • the time vector (vecTime) is now a vector of pairs so it contains the timestamp and the index in the netcdf file, after discovering some NetCDF files where the time base is NOT sorted

CSV

  • It is now possible to use CSV_SKIP_FIELDS together with CSV_ONLY_FIELDS.
  • It is now possible to automatically prefx the field names with the column number and to postfix with an arbitrary string
  • A new key has been added, CSV_ONLY_FIELDS that does the opposite of CSV_SKIP_FIELDS
  • It is now possible to set the same parameter name to multiple CSV columns, if these extra columns will then be canceled by a CSV_SKIP_FIELD command
  • The fields to be skipped as provided by CSV_SKIP_FIELDS in the CSV plugin are now comma delimited and support ranges
  • Added the missing toString() call to CsvDateTime
  • The whole date handling logic has been redone, it is now possible to mix string representations with components.
  • The error handling logic has also been redone
    • More flexibility in timestamps parsing followed by Code Refactoring
    • Now the CSV plugin is split over two files, with one for the parameters parsing (CsvParams) and one for the plugin itself (CsvIO).
    • Now more code has been moved into CsvDateTime, the interface as seen from CsvIO is much simpler (higher level), CsvParams mostly takes care of normal parameters and delegates all the date/time work to CsvDateTime
  • A new field type YEAR_2DIGITS has been added to the date/time components support in CsvIO
  • Now supporting providing date as components and time as time_str (the user MUST also provide a date parsing spec, altough this will not be used)
  • Added the "permill" unit to CsvIO
  • Added a missing standard unit
  • It is now possible to read years expressed with 2 digits only in CSV files (A new option has been added to the format specifiers, YY)
  • It is now possible to provide both CSV_UNITS and CSV_UNITS_OFFSET / MULTIPLIER
  • Added the possibility to tag CSV fields to be skipped with "-"
  • Increased flexibility in the line exclusions specification for CSV and other plugins
  • It is now possible to either define lines restrictions as EXCLUDE or as ONLY statements in CsvIO
  • The "dequote" option of the CSV plugin has been replaced by a more generic "purgeChar" option.
  • It is now possible to provide multiple strings to be interpreted as nodata
  • Added an optimization for the line exclusion feature of the CSV plugin
  • Getting rid of the 'single_param_idx' field as a new approach has been implemented
  • Restored support for single parameter file
  • Respect coords parameters for xy
  • Fixed a possible segfault (There was an inconsistency that could lead to segfault in the handling of large clocks of comments at the beginning of a CSV file)
  • Making a private structure a standalone class in order to further expand it
  • the CSV plugin was not properly handling parameters to skip when constructing its template MeteoData object
  • Added support for a comment marker in the CSV plugin: from this char until the end of the line, eveyrthing will be skipped (default: no comments marker).
  • Adding support for one more unit in CsvIO
  • Added on special case in CsvIO in order to transparently handle NAN or NULL (there is no point in generating errors or warnings for these)
  • It is now possible to provide a slope angle and azimuth to the CSN plugin (using the CSV_SLOPE and CSV_AZIMUTH keys)
  • now all data after 1st October that appears before data for before 1st October is assumed to belong to the year before.
  • when the date is provided as components, it is now possible to define a fallback year (for when no year is available in the file itself)

Smet

  • removed unused variables and methods
  • Fixed the writing of varying number of parameters in SMET in alphabetic order.
  • Now special SMET field names are parsed case-insensitive (more tolerance for ill-formed smet files)
  • Added more options for versioning information in SMET filenames
  • Renamed the smet "append creation date" parameter into "versioning" for clarity
  • A new option: appending the current date to the filename has been added
  • Set output precision for append mode in SMET plugin.
  • A wrong data type was used to represent the default precision and width, leading to wrongly formatted outputs on some systems (char instead of int)
  • Two new options have been added to smet, in order to control the default precision and width
  • SMET specifications have been edited in order to rename OSWR into RSWR.

Documentation and usability

  • Now in exception messages, only the source file name will be shown instead of the full path.
  • Improved error messages in CsvIO and DataGeneration
  • Added more explanations on how to use cdsapi for ERA data
  • Updated Documentation for CsvIO and data generators
  • The ILWR parametrization plot has been extended with the Carmona All Sky parametrization. The same plot has been done for all supported Clear Sky parametrizations and the documentation has been extended.
  • Added some documentation on compiling the IMIS plugin
  • Updated the URLs in the documentation
  • Now the doxygen documentation has an automatically generated version string
  • Removed all mentions of the old URL "models.slf.ch"
  • Documentation on SQLIO
  • Fixed links to the forge that have been migrated
  • Added some documentation and a missing check for empty horizon points in the DEMAlgorithm
  • Expanded the documentation so there is no extra information in files on a private hard drive anymore...
  • Documentation on the MeteoIO standalone
  • Documentation for the MeteoBlue, SMET and DBO plugins
  • Added the documentation for the CSV_ONLY_LINES feature as well as some documentation restructuring
  • Documentation improvements (SI units -> coherent derived SI units) in the SMET andCSV plugins
  • Fixed some warnings when compiling with Visual C++ 2017 as well as one error in meteoio_timeseries
  • The time example was still relying an an old Date API
  • Added a warning when using the old PROJ4 coordinate system instead of the new PROJ
  • Using MathML for equations in doxygen instead of pictures as well as some tunning for the visibility of enums, typedefs within classes and namespaces
  • Documentation of the new SNOWLINE 2d interpolation
  • Documentation for the new DataEditing
  • Fixed warnings found with VC++

Filters

  • New Filters:
    • ProcReducePressure: to reduce the local air pressure to sea level air pressure
    • ProcShift: time shifting filter to bring two meteo parameters into sync
  • The TRANSFORMWINDVECTOR filter accepts both an arbitrary source (key: COORDPARAM_SRC) as well as destination coordinate system (key: COORDPARAM)
  • ProcShift: fixed the handling of the offsets file and fixed some rounding errors
  • in ProcShift, the sampling rate is now rounded when computing the offsets so the generated timestamps are more user-friendly
  • Better delimiting of the various time filtering algorithms
  • The SOFT option in the MAD filter was being used both for the window definition and for the behavior when no window can be defined, leading to counter-intuitive behavior. There is now an additional STRICT option that defines how to handle the case when no window can be define.

  • Two new keys have been added: enable_meteo_filtering and enable_time_filtering in order to be able to disable some filtering without having to comment out the whole section.

  • The UnDST time filter has been renamed into SHIFT since it can be used for much more than just DST issues

  • If the data was ending with a conflict, it would not be reported by the Time filter. It is now fixed.
  • The CLEANUP mode of the TimeSuppr filter has been deeply changed: it now reports full out of order ranges as one line per range (instead of per timestep)
  • Making the despiking filter more tolerant to the case of its arguments
  • new filter keyword "WHEN" to provide date ranges when the filter should be applied
  • Rewrote the TimeSuppr method that removes invalid timestamps sot it can now discriminate between duplicated and out of order timestamps

Packaging

  • Added automatic detection of PROJ or PROJ4
  • Pipeline without manual zipping
  • Now using more standard version numbers
  • Much improved finding of Mysql
  • The leak check through ASAN was not properly linked
  • The "meteoio_timeseries" binary was not properly linked when requesting static linking
  • Compiling fat binaries is now optional
  • An alternate way of getting a git version number for the repository has been implemented. This should help on Windows.
  • inishell won't be bundled with MeteoIO anymore
  • Pushed cmake minimum required version to 3.1, set c++11 for the code examples
  • Now compatibility with SDK version 10.13 (High Sierra) is enforced and cmake is required to be at least 3.0.2.
  • Improving the FindMeteoIO.cmake: it now gives priority to installations close by and should better handle when it is not found
  • Fixed portability issue on non-glibc

Handling of the configuration file

  • Improved parsing with regex
  • Bugfix when parsing data generators arguments with high indices
  • Fixed the handling and sorting of indexed keys, espacially with similar roots
  • Fixed ordering and naming of VIR stations
  • Sorting the ini keys in meaningful numerical order using c++11 regex
  • Allow refering to variables in an imported ini file
  • More descriptive error messages
  • Config::getKeys had a bug when requesting matches anywhere: the section name could itself match multiple sections and extracting the keys could then fail to proceed as expected
  • A new method has been added to Config (getValuesRegex more powerful than before)

Spatial interpolations and resampling

  • Added Handling if a parameter does not even exist in the IDWSlopesAlgorithm
  • Fixed invalid error messages
  • Added a missing user argument to the WINSTRAL spatial interpolation
  • Now it is possible to set Meteo1dInterpolation::WINDOW_SIZE to zero, this internally sets ENABLE_RESAMPLING to false.
  • Bug fix for IDWLapseLocalAlgorithm: a global detrend of the station data was performed before doing the local interpolation, followed by a global retrend afterwards. This doesn't seem to be consistent with the local interpolation, which should detrend/retrend only on the local scale.
  • adding the option MAX_DISTANCE to be able to specify the maximum allowed distance between stations and the grid point to interpolate to.
  • Adding option NAMING to the USER spatialInterpolations algorithm, which allows to specify the file naming convention using YYYY for year, MM for month, DD for day, hh for hour, mm for minute, ss for seconds (optional) and PARAM for meteo parameter.
  • It is now possible to use the USER spatial interpolation algorithm as a fallback.
  • Integrate grid resampling test into master ini
  • Grid resampling optimizations
  • Fix GridTimeseriesResampling for edge cases Now dates before and after data span are enabled correctly.
  • Enable per-algorithm WINDOW_SIZE in grid resampling
  • Safety checks in GridLinearInterpolation
  • Made linear grid resampling more readable
  • Added 1D grid resampling infrastructure
  • Fixing the calculation of curvature by including the distance between points and implementing the curvature length scale ETA in the ListonWind algorithm.
  • Added more useful warning message when trying a timeseries being written out by the Smet plugin has inconsistent parameters
  • The keys for the spatial resampling have been moved to the [InputEditing] section so you might have to migrate your existing ini files
  • added SNOWLINE 2d interpolation
  • Adapting SnowlineAlgorithm text file format to be able to transport meta info.
  • The ability to only apply a filter for specific time periods is fully functional and documented

Data generators

  • it is now safe to provide a DEM "just in case"
  • More flexibility to call data generators from outside Meteoio
  • The PrecSplitting generator does not mandate a splitting model. If no splitting model is provided, it will default to the new "NONE" model that does nothing.
  • Bringing the horizons to the ILWR all sky generators. A new file format for providing horizons has been designed and is used both by ProcShade and TauCLDGenerator / AllSkyLWGenerator. The horizon can now be computed from the dem, read from such as horizon file and written back to a file.
  • A new option has been added: USE_RAD_THRESHOLD. This option is here to force temporal resampling of the parametrized cloudiness over periods of low ISWR
  • Another version of the cloudiness parametrization.
  • Adding more ILWR and cloudiness parametrizations
  • Implemented support for cloud cover in All Sky ILWR generators.
  • Added a METEOINDEX generator to generate common meteorological indices (such as heat index, etc).
  • The blackGlobeTemperature function was using the wrong units, this has been fixed and validated. But this has also shown that assuming a constant value for the h parameter significantly degrades the performances of the parametrization
  • Improved the data generator template
  • Now both the DataGenerators and data creators support both stations and time restrictions. The syntax of the EXCLUDE and KEEP Input Editing command has been changed: their main argument is now PARAM instead of EXCLUDE / KEEP since these were making conflicts. Similarly, the TYPE argument of CREATE has been renamed into ALGORITHM to avoid conflicts.
  • The DataGenerators now use the same syntax as the Input Data Editing and the Filters

API improvements

  • Moved much to C++11
  • All calls to fabs() have been replaced by calls to std::abs() for consistency
  • It is better to create a projection context instead of using the default one, to allow the functions to be safely used in parallel applications. Thus far, this solved random seg faults in these functions with Alpine3D.
  • new PROJ API
  • Added a new constructor to MeteoData
  • Modernizing the support of libProj
  • Redoing signals catching for improved portability
  • Added signal trapping
  • The DateRange typedef is now a class and has been moved to Date.cc/h
  • The interface for the filter has been changed: they now all receive a reference to a Config object in their constructor so they can extract any additional information that they need.

Grids, Data and parameter handling

  • Better han dling of changing number of meteo parameters
  • The horizon angles are now rounded up to the nearest .10 degree
  • Handled a special case (only one point to define a horizon)
  • The angular resolution of the scans for computing the horizon is now dynamically computed from the dem cellsize
  • The horizon file format has been changed so it can contain the horizon for any number of stations
  • Moving the code to perform horizon scans, reading and interpolations into DEMAlgorithms
  • Added a CLOUD_COVER MeteoGrids parameter
  • Rename "GRID_WINDOW_SIZE" to "WINDOW_SIZE"
  • Grid resampling optimizations
  • Declared a constant for clarity / flexibility and set it to a higher value
  • Fixing the case where rounding errors make the isSameGeolocalization test fail, even though the two grids are basically the same Geolocalization.
  • Fix for safeGet, where instead of nodata, it would return the boundary grid cell. Prevents unwanted discontinuities in calculations of curvature.
  • Corrected the calculation of curvature, and adding citation for future reference.
  • Adding option RESAMPLING_STRATEGY = GRID_EXTRACT_PTS, which will request from the plugin only for the list of virtual stations, rather than requesting full grids and reading the points from the grid.
  • The DST (Daylight Saving Time) support in the Date class has been removed. For input data that have DST, it is still possible to rely on the UnDST time filter to bring the data back to a fixed offset to UTC
  • There is now a flag to enable / disable the input editing for timeseries globally
  • It is now possible to read decimal representations of dates (such as Excel, Matlab, etc) in CSV files
  • It is now possible to use the NOW time specification both for the start and the end date options.
  • It is also now possible to only provide one of these as well as a duration (in days)
  • For at least 6 years, the azimuth of flat surfaces (or pixels where no slope is known) was in some cases reported as PI, which is now corrected
  • There is now a new method in MeteoData to check for conflicts between two MeteoData objects. This can be used when merging two MeteoData objects
  • The handling of conflicts in extra parameters was quite broken, it has been fixed. Now when doing "automerge", the number of conflicts is reported
  • Added a new merge type: WINDOW_MERGE. This is similar with FULL_MERGE but is only performed within the time range of the reference timeseries.
  • Adding the possibility to specify a name for the virtual stations, by using the key VNAME

Wrappers, tools, applications, and third party libraries

  • Added Eigen 3.4.0 as a thirdParty library to support Vapour Transport functionality within the SNOWPACK project
  • The QA_CHECK_MISSING key is now optional
  • The link to DATA_QA_LOGS is more visible and both options are now in the same section. Please beware that you will need to update your ini files!!
  • Updated tinyexpr to the current github version
  • A wrapper library around the Mysql c api (libMysqlWrapper) has been written.
  • Code restructuring of libMysqlWrapper
  • Fixing some of the memory management warnings and a memory leak when allocating the mysql pointer
  • More information in the help message and checking for missing values if DATA_QA_LOGS is enabled
  • First dockerfile for Arctic Passion
  • Added SPDX license to all source files
  • Adding buffered output option to meteoio_timeseries application.
  • Added the WIGOS ID field
  • Multiplatform implementation of a basic software watchdog.
  • time restrictions are now also applied to merge and automerge in the Input Data Editing
  • More flexibility in smet2kml it can now run on a given directory or also on a given file.

Data Editing

  • It is now possible to provide individual dates to toggle on/off the processing or data editing
  • A new test has been developed for the Data Editing
  • It is now possible to give '*' as parameter to remove in the DELETE Input Data Editing in order to remove all timestamps from a station
  • There is now a METADATA Input Editing command, where even the station ID can be changed
  • The TYPE argument for the CREATE Input Data Editing has been renamed into ALGORITHM
  • The multiple data source feature now works through multiple [Input#] sections
  • The station IDs are now handled case-insensitive in Input Data Editing while the meteo parameters are case sensitive
  • Now all editing commands beside MERGE and AUTOMERGE support time restrictions with the WHEN option
  • There is now a dependencies resolution for the Data Editing stack, so if station A merges station B that itself merges station C, C will be processed first, then B and finally A in order to deliver the expected outcome.
  • A SWAP Input Editing command has been added: it swaps two parameters

Other bug fixes and performance increases

  • Fixed an invalid escape in smet_extract.sh (extract column from smet file)
  • Fixed warnings on Windows
  • Fix CppCheck findings
  • More warnings fixed on macOS
  • Resolve warnings appearing after using the new G++ version
  • Fixed a bug according in the PSUM_PH generator
  • Bug fix for the failing compilation of NetCDF
  • Fixed a bug in the generation of error messages
  • when the aggregation period was a multiple of the sampling rate the maximum was wrong in some cases, this is now fixed
  • When searching for an element at an exact date either an exact match OR the next element greater than the provided date is returned, inaccurately it was assumed that it was always the next element and 1 was substracted to be sure to contain the start point of interest. Now the case of an exact match is properly handled and skips the substracting 1 in this case.
  • The initialization of the proj4 interface was relatively expensive, so a huge performance increase is achieved by having the TRANSFORMWINDVECTOR filter having its own interface, where it is set up once upon filter initialization and then reused.
  • Fixed a bug when creating a new station through Input Data Editing's METADATA command
  • The wildcard EXCLUDE Input Data Editing had a bug when handling multiple time periods
  • Fixed a bug that was messing up the hadling of multiple time restrictions periods
  • in SMET files the ACDD time coverage was improperly reported to be in GMT although it was in local time, this is now fixed
  • DataCreators have been removed and replaced by a CREATE editing command in Input Data Editing. The syntax for the arguments is now the same as for the other editing commands (or the filters).
  • The DataGenerators themselves are now scheduled to be updated to switch to the same syntax for their arguments
  • The plugins that were checking if they have been called for input or output all forgot to force the keys uppercase, leading to weird bugs, this was fixed
  • The issue of duplicated/missing timestamps when merging is now resolved
  • In some cases, the sampling rate extracted by smet_range was too large for the chosen format specifier
  • Editing the station ID in Input Data Editing was inefficient and limited. It is now based on MeteoData::merge() and therefore much more powerful.