Detailed Description

A filter that evaluates formulas dependent on conditions.

Author: Michael Reisecker

Date: 2019-09-01

This filter evaluates an arithmetic expression and filters the meteo parameters to the result. You can use a number of substitutions from your data and meta data.

List of ini keys

Example: Convert from degrees Rankine to Kelvin:

TA::FILTER1 = MATHS

TA::ARG1::FORMULA = meteo(TA) * 5/9

You can define a condition that must be met for the filter to take action. The EXPRESSION1 and COMPARE1 keywords denote two formulas and their evaluated results will be compared by the operator given with OPERATOR1.

Example: Allow no snow below 500 m altitude:

HS::FILTER1           = MATHS
HS::ARG1::FORMULA     = 0
HS::ARG1::EXPRESSION1 = altitude
HS::ARG1::OPERATOR1   = LT
HS::ARG1::COMPARE1    = 500

Operators: The following comparison operators are available: LT (less than), LE (less than or equal), GT (greater than), GE (greater equal), EQ (equal), NE (not equal), STRCMP (string comparison), STRBEG (string comparison at the beginning of a string), STREND (string comparison at the end of a string), and STRINC (search in string).

Expressions: Arithmetic expressions are evaluated with tinyexpr (developer's repo). You can use the standard arithmetic operations, angular functions, exponentials, and some combinatorics.

Note: Here is a full list of the available functions: addition (+), subtraction/negation (-), multiplication (*), division (/), exponentiation (^), modulus (%), abs, acos, asin, atan, atan2, ceil, cos, cosh, exp, floor, ln (calls to log), log (calls to log10), log10, pow, sin, sinh, sqrt, tan, tanh, fac, ncr, npr. It also features the constants e and pi.

Substitutions: The following substitutions are available in all arithmetic expressions:

meteo(PARAM) where PARAM is any available meteo parameter,
year, month, day, hour, minute, julian for the measurement's date,
altitute, latitude, longitude, easting, northing for the station's position,
angle, azimuth for slope parameters,
stdpress, stdtemp, boltzmann for (meteorological) constants,
nodata for the nodata value (usually -999).

The following substitutions are available in string comparisons:

stationid, stationname for station identification.

Example: Add offset to specific station:

HS::FILTER1           = MATHS
HS::ARG1::FORMULA     = meteo(HS) + 0.38
HS::ARG1::EXPRESSION1 = stationid
HS::ARG1::OPERATOR1   = STRCMP
HS::ARG1::COMPARE     = ISEE3

If the condition is not met, you can use a different expression to evaluate via the FORMULA_ELSE key.

Example: If there is no wind direction available then throw away the wind speed. Else, use a logarithmic scale:

VW::FILTER1            = MATHS
VW::ARG1::FORMULA      = nodata
VW::ARG1::FORMULA_ELSE = log(meteo(VW))
VW::ARG1::EXPRESSION1  = meteo(DW)
VW::ARG1::OPERATOR1    = EQ
VW::ARG1::COMPARE1     = nodata

You can make use of a simple logic parser. You can supply as many conditions as you wish, but they are always combined with the same logical operator, namely CONNECTIVE can be AND (default) or OR.

Example: Allow no snow between June and August:

HS::FILTER1           = MATHS
HS::ARG1::FORMULA     = 0
HS::ARG1::EXPRESSION1 = month
HS::ARG1::OPERATOR1   = GE
HS::ARG1::COMPARE1    = 6
HS::ARG1::EXPRESSION2 = month ;AND is implied
HS::ARG1::OPERATOR1   = LE
HS::ARG1::COMPARE1    = 8

Or alternatively:

HS::FILTER1           = MATHS
HS::ARG1::FORMULA     = 0
HS::ARG1::CONNECTIVE  = OR ;switch logical connective to OR for all
HS::ARG1::EXPRESSION1 = month
HS::ARG1::OPERATOR1   = EQ
HS::ARG1::COMPARE1    = 6
HS::ARG1::EXPRESSION2 = month
HS::ARG1::OPERATOR2   = EQ
HS::ARG1::COMPARE2    = 7
HS::ARG1::EXPRESSION3 = month
HS::ARG1::OPERATOR3   = EQ
HS::ARG1::COMPARE3    = 8

Important: Save for the SKIP_NODATA keyword (see below), nodata values (usually -999) are kept in the calculations and so far it is up to the user to check against this!

Example: Approximate the dew point only if air temperature and relative humidity are both available:

TD::FILTER1            = MATHS
TD::ARG1::FORMULA      = meteo(TA) - ((100 - meteo(RH)*100) / 5)
TD::ARG1::FORMULA_ELSE = nodata
TD::ARG1::EXPRESSION1  = meteo(RH)
TD::ARG1::OPERATOR1    = NE
TD::ARG1::COMPARE1     = nodata
TD::ARG1::EXPRESSION2  = meteo(TA)
TD::ARG1::OPERATOR2    = NE
TD::ARG1::COMPARE2     = nodata
TD::ARG1::EXPRESSION3  = meteo(RH)
TD::ARG1::OPERATOR3    = GT
TD::ARG1::COMPARE3     = 0.5 ;formula within 1 degree only for RH > 50 %

Example: Two stations are called ISEE1 and ISEE2. Suppose their humidity sensors start reporting garbage on 1st of August 2019 and we want to make sure they are not used in any calculations:

RH::FILTER1           = MATHS
RH::ARG1::FORMULA     = nodata ;throw all data away
RH::ARG1::EXPRESSION1 = stationid
RH::ARG1::OPERATOR1   = STRBEG ;compare beginning of string
RH::ARG1::COMPARE1    = ISEE ;this string is as-is, i. e. no substitutions
RH::ARG1::EXPRESSION2 = julian
RH::ARG1::OPERATOR2   = GE
RH::ARG1::COMPARE2    = 2458696.5 ;2019-08-01

With the ASSIGN key you can provide a parameter that the result is saved to (instead of the one the filter runs on).

Important: For technical reasons, this key only works as expected if it's the last filter that is run on the set.

Example: A copy-heavy hack to make a SHADE processor run on a specific station only - without even calculating the heavy sun stuff for others (we must control the filter ordering to be able to do this):

[INPUT]
ISWR_TEMP::COPY               = ISWR ;CREATE would refill
 
[FILTERS]
ISWR_TEMP::FILTER1            = MATHS ;keep only if station name matches
ISWR_TEMP::ARG1::FORMULA      = meteo(ISWR_TEMP)
ISWR_TEMP::ARG1::FORMULA_ELSE = nodata
ISWR_TEMP::ARG1::EXPRESSION1  = stationname
ISWR_TEMP::ARG1::OPERATOR1    = STRCMP
ISWR_TEMP::ARG1::COMPARE1     = Seegrube
 
ISWR_TEMP::FILTER2            = SHADE ;only 1 station has non-nodata values
 
ISWR_TEMP::FILTER3            = MATHS ;copy back if not nodata
ISWR_TEMP::ARG3::FORMULA      = meteo(ISWR_TEMP)
ISWR_TEMP::ARG3::FORMULA_ELSE = meteo(ISWR)
ISWR_TEMP::ARG3::EXPRESSION1  = meteo(ISWR_TEMP)
ISWR_TEMP::ARG3::OPERATOR1    = NE
ISWR_TEMP::ARG3::COMPARE1     = nodata
ISWR_TEMP::ARG3::ASSIGN       = meteo(ISWR) ;must be last filter to work!

Note: With the same method you could build a chain like if (A or B) and C if you must.

The last remaining key is SKIP_NODATA to transport nodata values unaltered from input to output. It is false by default which means that all formulas are evaluated with it (and you could catch it with meteo(XX) EQ nodata).

List of ini keys

Keyword	Meaning	Optional	Default Value
FORMULA	Arithmetic expression to evaluate for meteo parameter.	no	-
FORMULA_ELSE	Expression if the conditions below are not met.	yes	empty, meaning the data point is left alone
CONNECTIVE	Logical connective for the conditions below.	yes	AND (choices: OR)
EXPRESSION#	1st part of condition: Expression that will be compared.	yes, if neither OPERATOR# nor COMPARE# exist.	empty, meaning FORMULA will always apply
OPERATOR#	2nd part of condition: Comparison operator.	yes, if neither EXPRESSION# nor COMPARE# exist.	empty (choices: LT, GT, EQ, ...)
COMPARE#	3rd part of condition: EXPRESSION is compared against this.	yes, if neither EXPRESSION# nor OPERATOR# exist.	empty
ASSIGN	Change parameter to store final result in (only if it's the last filter running).	yes	empty (choices: meteo(PARAM) or PARAM)
SKIP_NODATA	Any input nodata stays untouched. Checks for nodata in COMPARE will not be called.	yes	false

#include <FilterMaths.h>

Public Member Functions
	FilterMaths (const std::vector< std::pair< std::string, std::string > > &vecArgs, const std::string &name, const Config &cfg)

virtual void	process (const unsigned int &param, const std::vector< MeteoData > &ivec, std::vector< MeteoData > &ovec)
	The filtering routine as called by the processing toolchain. More...

Public Member Functions inherited from mio::ProcessingBlock
virtual	~ProcessingBlock ()

virtual void	process (const unsigned int &param, const std::vector< MeteoData > &ivec, std::vector< MeteoData > &ovec)=0

virtual void	process (Date &dateStart, Date &dateEnd)

std::string	getName () const

const ProcessingProperties &	getProperties () const

const std::string	toString () const

bool	skipStation (const std::string &station_id) const
	Should the provided station be skipped in the processing? More...

bool	noStationsRestrictions () const

const std::vector< DateRange >	getTimeRestrictions () const

Additional Inherited Members
Static Public Member Functions inherited from mio::ProcessingBlock
static void	readCorrections (const std::string &filter, const std::string &filename, std::vector< double > &X, std::vector< double > &Y)
	Read a data file structured as X Y value on each lines. More...

static void	readCorrections (const std::string &filter, const std::string &filename, std::vector< double > &X, std::vector< double > &Y1, std::vector< double > &Y2)
	Read a data file structured as X Y1 Y2 value on each lines. More...

static std::vector< double >	readCorrections (const std::string &filter, const std::string &filename, const size_t &col_idx, const char &c_type, const double &init)
	Read a correction file applicable to repeating time period. More...

static std::vector< offset_spec >	readCorrections (const std::string &filter, const std::string &filename, const double &TZ, const size_t &col_idx=2)
	Read a correction file, ie a file structured as timestamps followed by values on each lines. More...

static std::map< std::string, std::vector< DateRange > >	readDates (const std::string &filter, const std::string &filename, const double &TZ)
	Read a list of date ranges by stationIDs from a file. More...

Protected Member Functions inherited from mio::ProcessingBlock
	ProcessingBlock (const std::vector< std::pair< std::string, std::string > > &vecArgs, const std::string &name, const Config &cfg)
	protected constructor only to be called by children More...

Static Protected Member Functions inherited from mio::ProcessingBlock
static void	extract_dbl_vector (const unsigned int &param, const std::vector< MeteoData > &ivec, std::vector< double > &ovec)

static void	extract_dbl_vector (const unsigned int &param, const std::vector< const MeteoData * > &ivec, std::vector< double > &ovec)

Protected Attributes inherited from mio::ProcessingBlock
const std::set< std::string >	excluded_stations

const std::set< std::string >	kept_stations

const std::vector< DateRange >	time_restrictions

ProcessingProperties	properties

const std::string	block_name

Static Protected Attributes inherited from mio::ProcessingBlock
static const double	soil_albedo = .23

static const double	snow_albedo = .85

static const double	snow_thresh = .1
	parametrize the albedo from HS More...

Constructor & Destructor Documentation

◆ FilterMaths()

mio::FilterMaths::FilterMaths	(	const std::vector< std::pair< std::string, std::string > > &	vecArgs,
		const std::string &	name,
		const Config &	cfg
	)

Member Function Documentation

◆ process()

void mio::FilterMaths::process	(	const unsigned int &	param,
		const std::vector< MeteoData > &	ivec,
		std::vector< MeteoData > &	ovec
	)

virtual

The filtering routine as called by the processing toolchain.

Parameters

[in]	param	Parameter index the filter is asked to run on by the user.
[in]	ivec	Meteo data to filter.
[out]	ovec	Filtered meteo data. Cf. main documentation.

Implements mio::ProcessingBlock.

The documentation for this class was generated from the following files:

Detailed Description

List of ini keys

Public Member Functions

Additional Inherited Members

Constructor & Destructor Documentation

◆ FilterMaths()

Member Function Documentation

◆ process()