MeteoIODoc 20241221.207bde49
Processing overview

The pre-processing infrastructure is described in ProcessingBlock (for its API). The goal of this page is to give an overview of the available filters and processing elements and their usage. Moreover, there is a special mode of operation where MeteoIO writes on the screen a line for each data point that gets modified (either filtered, resampled or generated). This is enabled by setting the DATA_QA_LOGS key to true in the [General] section. The outputs then look like the following:

[DATA_QA] Filtering WFJ2::ILWR::MIN_MAX 2016-01-18T09:00:00+01:00 [2016-W03-01]
[DATA_QA] Resampling WFJ1::VW_MAX::LINEAR 2015-10-12T13:00:00+01:00 [2015-W42-01]

Modes of operation

It should be noted that filters often have two modes of operations: soft or hard. In soft mode, all value that is rejected is replaced by the filter parameter's value. This means that for a soft min filter set at 0.0, all values less than 0.0 will be replaced by 0.0. In hard mode, all rejected values are replaced by nodata.

It is possible to disable a given Processing Element for specific stations, using the exclude or only options followed by a list of station IDs (see example below). This is supported automatically by all Processing Elements. Several Processing Elements take arguments describing a processing window (for example, FilterStdDev). In such a case, they take the window parameters arguments as defined in WindowedFilter::setWindowFParams().

It is also possible to rectrict any filter to a specific set of time ranges, using the when option followed by a comma delimited list of date intervals (represented by two ISO formatted dates seperated by ' - ') or individual dates (also ISO formatted).

The processing is normally defined per meteorological parameter (either from the list of standard parameter names or any other name of your choice). A special kind of processing is available on the timestamps themselves and takes place before any other processing (see below in Available processing elements).

It is possible to apply filters to only specific heights/numbers of a parameter, when they are given as TA@15..., either by specifying the ids for which the filter should be used (AT_HEIGHTS), or for which it should not be used (EXCLUDE_HEIGHTS). Per default a filter will run on all available heights, so every TA will be applied the same filter to.

The heights are given as a list of heights similar to the station IDs, if you have default parameters and some that use a height, i.e. TA, TA@1, TA@10, the default parameter has no height, i.e. "NO" in the list to include or exclude.

Filtering section

The filters are specified for each parameter in the [Filters] section. This section contains a list of the various meteo parameters (see MeteoData) with their associated choice of filtering algorithms and optional parameters.The filters are applied serially, in the order they are given in.

An example of such section is given below:

[Filters]
TA::filter1 = min_max
TA::arg1::AT_HEIGHTS = NO 1
TA::arg1::min = 230
TA::arg1::max = 330
RH::filter1 = min_max
RH::arg1::EXCLUDE_HEIGHTS = 10
RH::arg1::min = -0.2
RH::arg1::max = 1.2
RH::filter2 = min_max
RH::arg2::soft = true
RH::arg2::min = 0.0
RH::arg2::max = 1.0
PSUM::filter1 = min
PSUM::arg1::min = -0.1
PSUM::filter2 = min
PSUM::arg2::soft = true
PSUM::arg2::min = 0.
PSUM::filter3 = undercatch_wmo
PSUM::arg3::type = Hellmannsh
PSUM::arg3::exclude = DAV3 WFJ2
#Correct a wrongly mounted wind sensor for two time periods
DW::filter1 = add
DW::arg1::type = Cst
DW::arg1::cst = 45
DW::arg1::when = 2020-07-01 - 2020-07-10 , 2020-07-20T12:00 - 2020-08-01
Note
It is possible to turn off all meteo filtering by setting the Enable_Meteo_Filtering key to false in the [Filters] section; the same can be done for timestamps filtering with the Enable_Time_Filtering key.

Available processing elements

New filters can easily be developed. The filters that are currently available are the following:

  • NONE: this does nothing (this is useful in an IMPORT to overwrite previous filters);
  • MIN: minimum check filter, see FilterMin
  • MAX: maximum check filter, see FilterMax
  • MIN_MAX: range check filter, see FilterMinMax
  • MIN_MAX_CONDITIONAL: range check only if a different parameter holds true to a comparison, see FilterMinMaxConditional
  • RATE: rate of change filter, see FilterRate
  • UNHEATED_RAINGAUGE: detection of snow melting in a rain gauge, see FilterUnheatedPSUM
  • DETECT_GRASS: detection of grass growing under the snow height sensor, see FilterDeGrass
  • POTENTIALSW: ensuring physically realistic incoming short wave radiation, see FilterPotentialSW
  • MATHS: evaluating arithmetic expressions with access to meteo data, see FilterMaths
  • STD_DEV: reject data outside mean +/- k*stddev, see FilterStdDev
  • MAD: median absolute deviation, see FilterMAD
  • TUKEY: Tukey53H spike detection, based on median, see FilterTukey
  • DESPIKING: despiking in phase space according to Goring and Nikora (2002), see FilterDespikingPS
  • NO_CHANGE: reject data that changes too little (low variance), see FilterNoChange
  • TIME_CONSISTENCY: reject data that changes too much, see FilterTimeconsistency
  • KALMAN: dynamic state likelihood estimation via Bayesian statistics, see FilterKalman
  • PARTICLE: Monte Carlo sampling method for dynamic state estimation, see FilterParticle

Some data transformations are also supported besides filtering, both very basic and generic data transformations:

As well as more specific data transformations:

  • AGGREGATE: various data aggregation algorithms, see ProcAggregate
  • DEACCUMULATE: recompute instantaneous values from accumulated values, see ProcDeAccumulate
  • EXP_SMOOTHING: exponential smoothing of data, see ProcExpSmoothing
  • WMA_SMOOTHING: weighted moving average smoothing of data, see ProcWMASmoothing
  • IIR: Low Pass or High Pass critically damped filter, see ProcIIR
  • UNDERCATCH_WMO: WMO rain gauge correction for undercatch, using various correction models, see ProcUndercatch_WMO
  • UNDERCATCH_FORLAND: Forland1996 rain gauge correction for solid and liquid undercatch, using various correction models, see ProcUndercatch_Forland
  • UNDERCATCH_HAMON: Hamon1973 rain gauge correction for undercatch, see ProcUndercatch_Hamon
  • UNVENTILATED_T: unventilated temperature sensor correction, see ProcUnventilatedT
  • PSUM_DISTRIBUTE: distribute accumulated precipitation over preceeding timesteps, see ProcPSUMDistribute
  • SHADE: apply a shading mask to the Incoming or Reflected Short Wave Radiation, see ProcShade
  • REDUCE_PRESSURE: reduce local pressure to sea level pressure, see ProcReducePressure
  • RHWATERTOICE: correct relative humidity over water to over ice in case temperature is below freezing, see ProcRHWaterToIce
  • TRANSFORMWINDVECTOR: transform wind direction and/or wind speed components, see ProcTransformWindVector
  • PROJECT_RADIATION: project short wave radiation from one inclination to another one, see RadProj

A few filters can be applied to the timestamps themselves:

  • SUPPR: delete whole timesteps (based on a list or other criteria such as removing duplicates, etc), see TimeSuppr
  • SHIFT: add offsets at specific times, for example to bring timestamps that contain Daylight Saving Time back to Winter time, see TimeShift
  • SORT: sort the timestamps in increasing order, see TimeSort
  • TIMELOOP: loop over a specific time period (for example for model spin-ups), see TimeLoop