MeteoIODoc 20240329.18c31bd1
Data input/output overview

The data access is handled by a system of plugins. They all offer the same interface, meaning that a plugin can transparently be replaced by another one. Since they might rely on third party libraries for accessing the data, they have been created as plugins, that is they are only compiled if requested when configuring the compilation with cmake. A plugin can therefore fail to run if it has not been compiled.

Writing a file happens via a wrapper for std::ofstream ofilestream that adds some functionality, like limiting the write access.

Please have a look at the support for coordinate systems.

Data sources categories

Several data sources categories have been defined that can be provided by a different plugin. Each data source category is defined by a specific key in the configuration file (usually, io.ini):

  • METEO, for meteorological time series
  • DEM, for Digital Elevation Maps
  • LANDUSE, for land cover information
  • GRID2D, for generic 2D grids (they can contain meteo fields and be recognized as such or arbitrary gridded data)
  • POI, for a list of Points Of Interest that can be used for providing extra information at some specific location (extracting time series at a few selected points, etc)

A plugin is "connected" to a given data source category simply by giving its keyword as value for the data source key:

METEO = SMET
DEM = ARC

Each plugin might have its own specific options, meaning that it might require its own keywords. Please check in each plugin documentation the supported options and keys (see links below). Moreover, a given plugin might only support a given category for read or write (for example, PNG: there is no easy and safe way to interpret a given color as a given numeric value without knowing its color scale, so reading a png has been disabled). Finally, the plugins usually don't implement all these categories (for example, ArcGIS file format only describes 2D grids, so the ARC plugin will only deal with 2D grids), so please check what a given plugin implements before connecting it to a specific data source category.

Available plugins

So far the following plugins have been implemented (by keyword for the io.ini key/value config file). Please read the documentation for each plugin in order to know the plugin-specific keywords:

Plugin keywordProvidesDescriptionExtra requirements
InputOutput
A3Dmeteo, poimeteo original Alpine3D meteo files
ALPUGmeteodata files generated by the ALPUG meteo stations
ARCdem, landuse, grid2dgrid2d ESRI/ARC ascii grid files
ARGOSmeteoARGOS satellite raw files - plugin not functional yet!
ARPSdem, grid2d, grid3dARPS ascii formatted grids
COSMOXMLmeteoMeteoSwiss COSMO's postprocessing XML formatlibxml2
CSVmeteoflexible reading of CSV files
DBOmeteoconnects to SLF's DBO web service interfacelibcurl
GEOTOPmeteometeo GeoTop meteo files
GOESmeteoMeteo files transmitted by the GOES satellites
GRASSdem, landuse, grid2dgrid2d Grass grid files
GRIBmeteo, dem, grid2dGRIB meteo grid filesgrib-api
IMISmeteoconnects to the IMIS databaseOracle's OCCI library
METEOBLUEmeteoconnects to MeteoBlue's web APIlibcurl
MYSQLmeteoconnects to a MySQL database, various schemas are supportedMySQL's C API
NETCDFmeteo, dem, grid2dmeteo, grid2d NetCDF grids and timeseriesNetCDF-C library
ECSVmeteo, poimeteo ECSV data files
OSHDmeteoOSHD generated binary Matlab fileslibmatio
PGMdem, grid2dgrid2d PGM grid files
PMODmeteoRaw data files from Pmod/Wrc (experimental!)
PNGgrid2d PNG grid fileslibpng
PSQLmeteometeo connects to PostgreSQL databasePostgreSQL's libpq
SASEmeteoconnects to the SASE databaseMySQL's C API
SMETmeteo, poimeteo SMET data files
SNOWPACKmeteometeo original SNOWPACK meteo files
SYNTHmeteosynthetic data generation
ZRXPmeteo WISKI database input files
Note
In order to optimize the data retrieval, the raw data is buffered. This means that up to BUFFER_SIZE days of data will be read at once by the plugin so subsequent reads will not have to get back to the data source (this key is in the [General] section). It is usually a good idea to configure BUFFER_SIZE to the intended duration of the simulation (in days).

Multiple data sources

It is possible to use multiple plugins to read meteorological timeseries from multiple sources and combine them into one stream of data. This is achieved by declaring as many [Input#] sections as necessary (where # represent any number, to make sure that not two sections have the same name) and declaring METEO plugins in each of them. Please make sure that all required keys are defined within each new such section (such as TIME_ZONE) because the plugins created this way won't have access to the other [Input] sections. The other plugins (such as for reading grids, dem, etc) will only be read from the standard [Input] section and performed as usual after the data has been provided by the plugins.

If reading the same station from multiple sources (for example providing different time coverage), it might be useful to use the EditingAutoMerge feature to merge all streams belonging to a station into one single stream. In this case, the data coming from [Input] has priority over the various [Input#] data sources.