MeteoIODoc 20240329.18c31bd1
mio::FilterMaths Class Reference

Detailed Description

A filter that evaluates formulas dependent on conditions.

Author
Michael Reisecker
Date
2019-09-01

This filter evaluates an arithmetic expression and filters the meteo parameters to the result. You can use a number of substitutions from your data and meta data.

List of ini keys

Example: Convert from degrees Rankine to Kelvin:

TA::FILTER1 = MATHS
TA::ARG1::FORMULA = meteo(TA) * 5/9

You can define a condition that must be met for the filter to take action. The EXPRESSION1 and COMPARE1 keywords denote two formulas and their evaluated results will be compared by the operator given with OPERATOR1.

Example: Allow no snow below 500 m altitude:

HS::FILTER1 = MATHS
HS::ARG1::FORMULA = 0
HS::ARG1::EXPRESSION1 = altitude
HS::ARG1::OPERATOR1 = LT
HS::ARG1::COMPARE1 = 500

Operators: The following comparison operators are available: LT (less than), LE (less than or equal), GT (greater than), GE (greater equal), EQ (equal), NE (not equal), STRCMP (string comparison), STRBEG (string comparison at the beginning of a string), STREND (string comparison at the end of a string), and STRINC (search in string).

Expressions: Arithmetic expressions are evaluated with tinyexpr (developer's repo). You can use the standard arithmetic operations, angular functions, exponentials, and some combinatorics.

Note
Here is a full list of the available functions: addition (+), subtraction/negation (-), multiplication (*), division (/), exponentiation (^), modulus (%), abs, acos, asin, atan, atan2, ceil, cos, cosh, exp, floor, ln (calls to log), log (calls to log10), log10, pow, sin, sinh, sqrt, tan, tanh, fac, ncr, npr. It also features the constants e and pi.

Substitutions: The following substitutions are available in all arithmetic expressions:

  • meteo(PARAM) where PARAM is any available meteo parameter,
  • year, month, day, hour, minute, julian for the measurement's date,
  • altitute, latitude, longitude, easting, northing for the station's position,
  • angle, azimuth for slope parameters,
  • stdpress, stdtemp, boltzmann for (meteorological) constants,
  • nodata for the nodata value (usually -999).

The following substitutions are available in string comparisons:

  • stationid, stationname for station identification.

Example: Add offset to specific station:

HS::FILTER1 = MATHS
HS::ARG1::FORMULA = meteo(HS) + 0.38
HS::ARG1::EXPRESSION1 = stationid
HS::ARG1::OPERATOR1 = STRCMP
HS::ARG1::COMPARE = ISEE3

If the condition is not met, you can use a different expression to evaluate via the FORMULA_ELSE key.

Example: If there is no wind direction available then throw away the wind speed. Else, use a logarithmic scale:

VW::FILTER1 = MATHS
VW::ARG1::FORMULA = nodata
VW::ARG1::FORMULA_ELSE = log(meteo(VW))
VW::ARG1::EXPRESSION1 = meteo(DW)
VW::ARG1::OPERATOR1 = EQ
VW::ARG1::COMPARE1 = nodata
const double nodata
This is the internal nodata value.
Definition: IOUtils.h:75

You can make use of a simple logic parser. You can supply as many conditions as you wish, but they are always combined with the same logical operator, namely CONNECTIVE can be AND (default) or OR.

Example: Allow no snow between June and August:

HS::FILTER1 = MATHS
HS::ARG1::FORMULA = 0
HS::ARG1::EXPRESSION1 = month
HS::ARG1::OPERATOR1 = GE
HS::ARG1::COMPARE1 = 6
HS::ARG1::EXPRESSION2 = month ;AND is implied
HS::ARG1::OPERATOR1 = LE
HS::ARG1::COMPARE1 = 8

Or alternatively:

HS::FILTER1 = MATHS
HS::ARG1::FORMULA = 0
HS::ARG1::CONNECTIVE = OR ;switch logical connective to OR for all
HS::ARG1::EXPRESSION1 = month
HS::ARG1::OPERATOR1 = EQ
HS::ARG1::COMPARE1 = 6
HS::ARG1::EXPRESSION2 = month
HS::ARG1::OPERATOR2 = EQ
HS::ARG1::COMPARE2 = 7
HS::ARG1::EXPRESSION3 = month
HS::ARG1::OPERATOR3 = EQ
HS::ARG1::COMPARE3 = 8

Important: Save for the SKIP_NODATA keyword (see below), nodata values (usually -999) are kept in the calculations and so far it is up to the user to check against this!

Example: Approximate the dew point only if air temperature and relative humidity are both available:

TD::FILTER1 = MATHS
TD::ARG1::FORMULA = meteo(TA) - ((100 - meteo(RH)*100) / 5)
TD::ARG1::FORMULA_ELSE = nodata
TD::ARG1::EXPRESSION1 = meteo(RH)
TD::ARG1::OPERATOR1 = NE
TD::ARG1::COMPARE1 = nodata
TD::ARG1::EXPRESSION2 = meteo(TA)
TD::ARG1::OPERATOR2 = NE
TD::ARG1::COMPARE2 = nodata
TD::ARG1::EXPRESSION3 = meteo(RH)
TD::ARG1::OPERATOR3 = GT
TD::ARG1::COMPARE3 = 0.5 ;formula within 1 degree only for RH > 50 %

Example: Two stations are called ISEE1 and ISEE2. Suppose their humidity sensors start reporting garbage on 1st of August 2019 and we want to make sure they are not used in any calculations:

RH::FILTER1 = MATHS
RH::ARG1::FORMULA = nodata ;throw all data away
RH::ARG1::EXPRESSION1 = stationid
RH::ARG1::OPERATOR1 = STRBEG ;compare beginning of string
RH::ARG1::COMPARE1 = ISEE ;this string is as-is, i. e. no substitutions
RH::ARG1::EXPRESSION2 = julian
RH::ARG1::OPERATOR2 = GE
RH::ARG1::COMPARE2 = 2458696.5 ;2019-08-01
static const double e
Definition: Meteoconst.h:68

With the ASSIGN key you can provide a parameter that the result is saved to (instead of the one the filter runs on).

Important: For technical reasons, this key only works as expected if it's the last filter that is run on the set.

Example: A copy-heavy hack to make a SHADE processor run on a specific station only - without even calculating the heavy sun stuff for others (we must control the filter ordering to be able to do this):

[INPUT]
ISWR_TEMP::COPY = ISWR ;CREATE would refill
[FILTERS]
ISWR_TEMP::FILTER1 = MATHS ;keep only if station name matches
ISWR_TEMP::ARG1::FORMULA = meteo(ISWR_TEMP)
ISWR_TEMP::ARG1::FORMULA_ELSE = nodata
ISWR_TEMP::ARG1::EXPRESSION1 = stationname
ISWR_TEMP::ARG1::OPERATOR1 = STRCMP
ISWR_TEMP::ARG1::COMPARE1 = Seegrube
ISWR_TEMP::FILTER2 = SHADE ;only 1 station has non-nodata values
ISWR_TEMP::FILTER3 = MATHS ;copy back if not nodata
ISWR_TEMP::ARG3::FORMULA = meteo(ISWR_TEMP)
ISWR_TEMP::ARG3::FORMULA_ELSE = meteo(ISWR)
ISWR_TEMP::ARG3::EXPRESSION1 = meteo(ISWR_TEMP)
ISWR_TEMP::ARG3::OPERATOR1 = NE
ISWR_TEMP::ARG3::COMPARE1 = nodata
ISWR_TEMP::ARG3::ASSIGN = meteo(ISWR) ;must be last filter to work!
Note
With the same method you could build a chain like if (A or B) and C if you must.

The last remaining key is SKIP_NODATA to transport nodata values unaltered from input to output. It is false by default which means that all formulas are evaluated with it (and you could catch it with meteo(XX) EQ nodata).

List of ini keys

KeywordMeaningOptionalDefault Value
FORMULAArithmetic expression to evaluate for meteo parameter.no-
FORMULA_ELSEExpression if the conditions below are not met.yesempty, meaning the data point is left alone
CONNECTIVELogical connective for the conditions below.yesAND (choices: OR)
EXPRESSION#1st part of condition: Expression that will be compared.yes, if neither OPERATOR# nor COMPARE# exist.empty, meaning FORMULA will always apply
OPERATOR#2nd part of condition: Comparison operator.yes, if neither EXPRESSION# nor COMPARE# exist.empty (choices: LT, GT, EQ, ...)
COMPARE#3rd part of condition: EXPRESSION is compared against this.yes, if neither EXPRESSION# nor OPERATOR# exist.empty
ASSIGNChange parameter to store final result in (only if it's the last filter running).yesempty (choices: meteo(PARAM) or PARAM)
SKIP_NODATAAny input nodata stays untouched. Checks for nodata in COMPARE will not be called.yesfalse

#include <FilterMaths.h>

Public Member Functions

 FilterMaths (const std::vector< std::pair< std::string, std::string > > &vecArgs, const std::string &name, const Config &cfg)
 
virtual void process (const unsigned int &param, const std::vector< MeteoData > &ivec, std::vector< MeteoData > &ovec)
 The filtering routine as called by the processing toolchain. More...
 
- Public Member Functions inherited from mio::ProcessingBlock
virtual ~ProcessingBlock ()
 
virtual void process (const unsigned int &param, const std::vector< MeteoData > &ivec, std::vector< MeteoData > &ovec)=0
 
virtual void process (Date &dateStart, Date &dateEnd)
 
std::string getName () const
 
const ProcessingPropertiesgetProperties () const
 
const std::string toString () const
 
bool skipStation (const std::string &station_id) const
 Should the provided station be skipped in the processing? More...
 
bool noStationsRestrictions () const
 
const std::vector< DateRangegetTimeRestrictions () const
 

Additional Inherited Members

- Static Public Member Functions inherited from mio::ProcessingBlock
static void readCorrections (const std::string &filter, const std::string &filename, std::vector< double > &X, std::vector< double > &Y)
 Read a data file structured as X Y value on each lines. More...
 
static void readCorrections (const std::string &filter, const std::string &filename, std::vector< double > &X, std::vector< double > &Y1, std::vector< double > &Y2)
 Read a data file structured as X Y1 Y2 value on each lines. More...
 
static std::vector< double > readCorrections (const std::string &filter, const std::string &filename, const size_t &col_idx, const char &c_type, const double &init)
 Read a correction file applicable to repeating time period. More...
 
static std::vector< offset_specreadCorrections (const std::string &filter, const std::string &filename, const double &TZ, const size_t &col_idx=2)
 Read a correction file, ie a file structured as timestamps followed by values on each lines. More...
 
static std::map< std::string, std::vector< DateRange > > readDates (const std::string &filter, const std::string &filename, const double &TZ)
 Read a list of date ranges by stationIDs from a file. More...
 
- Protected Member Functions inherited from mio::ProcessingBlock
 ProcessingBlock (const std::vector< std::pair< std::string, std::string > > &vecArgs, const std::string &name, const Config &cfg)
 protected constructor only to be called by children More...
 
- Static Protected Member Functions inherited from mio::ProcessingBlock
static void extract_dbl_vector (const unsigned int &param, const std::vector< MeteoData > &ivec, std::vector< double > &ovec)
 
static void extract_dbl_vector (const unsigned int &param, const std::vector< const MeteoData * > &ivec, std::vector< double > &ovec)
 
- Protected Attributes inherited from mio::ProcessingBlock
const std::set< std::string > excluded_stations
 
const std::set< std::string > kept_stations
 
const std::vector< DateRangetime_restrictions
 
ProcessingProperties properties
 
const std::string block_name
 
- Static Protected Attributes inherited from mio::ProcessingBlock
static const double soil_albedo = .23
 
static const double snow_albedo = .85
 
static const double snow_thresh = .1
 parametrize the albedo from HS More...
 

Constructor & Destructor Documentation

◆ FilterMaths()

mio::FilterMaths::FilterMaths ( const std::vector< std::pair< std::string, std::string > > &  vecArgs,
const std::string &  name,
const Config cfg 
)

Member Function Documentation

◆ process()

void mio::FilterMaths::process ( const unsigned int &  param,
const std::vector< MeteoData > &  ivec,
std::vector< MeteoData > &  ovec 
)
virtual

The filtering routine as called by the processing toolchain.

Parameters
[in]paramParameter index the filter is asked to run on by the user.
[in]ivecMeteo data to filter.
[out]ovecFiltered meteo data. Cf. main documentation.

Implements mio::ProcessingBlock.


The documentation for this class was generated from the following files: