mio::ProcShift Class Reference

Detailed Description

Time shifting filter for the selected meteo parameter.

This filter can correct the time base of a single parameter within a station, either with a constant offset or by reading time varying offsets from a provided file. Compared to the TimeShift time filter, this filter operates on only one parameter so it can shift parameters compared to each other within a given station dataset.

It can also compute the offset between two meteo parameter (one being used as reference parameter) by computing at each timestamp the offset that leads to the highest Pearson's correlation coefficient and writing this timeseries of offsets to a file. This correlation coefficient is evaluated over a given time window and sampling rate (the data is first temporarily forced-resampled at this constant sampling rate before computing the correlation coefficient). A large data window is more accurate but will fail to properly capture sudden changes while a short data window might generate some spurious offsets. In any case, it is advised to plot and tweak the generated timeseries of offsets first before using it to correct the meteo parameter (keep in mind that the offsets are given as the correction that should be applied to temporally re-synchronize the meteo parameter of choice, so for example +600 means that the signal would need to be moved forward by 600 seconds in order to be synchronized with the reference parameter).

This filter takes the following arguments:

• EXTRACT_OFFSETS: if set to TRUE, no correction is applied but the offsets between two meteo parameters are computed (defaut: false);
  • OFFSETS_FILE: the file where the computed time offsets will be written (mandatory);
  • REF_PARAM: the name of the reference meteo parameter (mandatory);
  • SAMPLING_RATE: the sampling rate to use when computing the time offsets (default: automatically extracted from the data)
  • WIDTH: data window width (in seconds) over which to compute each correlation coefficient (default: 2 days);
  • OFFSET_RANGE: range of allowed variation (in seconds) for the offset when searching for the maximum correlation (default: 1 day, so looking for a maximum between 1/2 a day before and after the current point);
• if EXTRACT_OFFSETS is set to false, the meteo parameter will be corrected.
  • TYPE: the type of correction to apply, either CST (constant over the whole dataset) or STEPWISE (keeping the last correction until finding a new correction). Default is CST;
  • CST: when using the CST type, the offset value (in seconds, mandatory in this case);
  • OFFSETS_FILE: for other correction types, a timeseries of offsets (in seconds, mandatory in this case);
  • TZ: the time zone for the timestamps given in the OFFSETS_FILE (mandatory).

A typical workflow for this filter would be:

1. run on the data with EXTRACT_OFFSETS = TRUE in order to compute the temporal offsets between the meteo parameter of interest and a reference one;
2. plot the resulting offsets timeseries and manually pickup the points to keep in the correction file (the computed offsets might be noisy, specially around periods of nodata). In doubt, look back at the original data to decide if you want to keep or reject a given offset.
3. run on the data with EXTRACT_OFFSETS = FALSE, providing your correction file.

Example of configuration to compute the time offset between TA_2 and TA_1 used as reference and write the results in the TA2_extracted.dat file (located in the current working directory):

TA_2::FILTER5 = SHIFT
TA_2::ARG5::EXTRACT_OFFSETS = TRUE
TA_2::ARG5::OFFSETS_FILE = TA2_extracted.dat
TA_2::ARG5::REF_PARAM = TA_1

Applying the corrections provided in the TA2_corrections.dat file (located in the same directory as the ini file):

TA_2::FILTER5 = SHIFT
TA_2::ARG5::TYPE = STEPWISE
TA_2::ARG5::OFFSETS_FILE = TA2_offsets.dat
TA_2::ARG5::TZ = 1

The TA2_corrections.dat file gives a an hour forward correction from 2018-12-03T00:00, then 2 hours backward from 2019-03-12T12:00 until 2019-10-01T15:30 when there is no correction any more:

2018-12-03T00:00 3600
2019-03-12T12:00 -7200
2019-10-01T15:30 0

#include <ProcShift.h>

Public Member Functions
	ProcShift (const std::vector< std::pair< std::string, std::string > > &vecArgs, const std::string &name, const Config &cfg)
virtual void	process (const unsigned int &param, const std::vector< MeteoData > &ivec, std::vector< MeteoData > &ovec)
Public Member Functions inherited from mio::ProcessingBlock
virtual	~ProcessingBlock ()
virtual void	process (const unsigned int &param, const std::vector< MeteoData > &ivec, std::vector< MeteoData > &ovec)=0
virtual void	process (Date &dateStart, Date &dateEnd)
std::string	getName () const
const ProcessingProperties &	getProperties () const
const std::string	toString () const
bool	skipStation (const std::string &station_id) const
	Should the provided station be skipped in the processing? More...
bool	noStationsRestrictions () const
const std::vector< DateRange >	getTimeRestrictions () const
bool	skipHeight (const double &height) const
	Should the provided height be skipped in the processing? More...

Additional Inherited Members
Static Public Member Functions inherited from mio::ProcessingBlock
static void	readCorrections (const std::string &filter, const std::string &filename, std::vector< double > &X, std::vector< double > &Y)
	Read a data file structured as X Y value on each lines. More...
static void	readCorrections (const std::string &filter, const std::string &filename, std::vector< double > &X, std::vector< double > &Y1, std::vector< double > &Y2)
	Read a data file structured as X Y1 Y2 value on each lines. More...
static std::vector< double >	readCorrections (const std::string &filter, const std::string &filename, const size_t &col_idx, const char &c_type, const double &init)
	Read a correction file applicable to repeating time period. More...
static std::vector< offset_spec >	readCorrections (const std::string &filter, const std::string &filename, const double &TZ, const size_t &col_idx=2)
	Read a correction file, ie a file structured as timestamps followed by values on each lines. More...
static std::map< std::string, std::vector< DateRange > >	readDates (const std::string &filter, const std::string &filename, const double &TZ)
	Read a list of date ranges by stationIDs from a file. More...
Static Public Attributes inherited from mio::ProcessingBlock
static const double	default_height
Protected Member Functions inherited from mio::ProcessingBlock
	ProcessingBlock (const std::vector< std::pair< std::string, std::string > > &vecArgs, const std::string &name, const Config &cfg)
	protected constructor only to be called by children More...
Static Protected Member Functions inherited from mio::ProcessingBlock
static void	extract_dbl_vector (const unsigned int &param, const std::vector< MeteoData > &ivec, std::vector< double > &ovec)
static void	extract_dbl_vector (const unsigned int &param, const std::vector< const MeteoData * > &ivec, std::vector< double > &ovec)
Protected Attributes inherited from mio::ProcessingBlock
const std::set< std::string >	excluded_stations
const std::set< std::string >	kept_stations
const std::vector< DateRange >	time_restrictions
std::set< double >	included_heights
std::set< double >	excluded_heights
bool	all_heights
ProcessingProperties	properties
const std::string	block_name
Static Protected Attributes inherited from mio::ProcessingBlock
static const double	soil_albedo = .23
static const double	snow_albedo = .85
static const double	snow_thresh = .1
	parametrize the albedo from HS More...

Constructor & Destructor Documentation

ProcShift()

mio::ProcShift::ProcShift	(	const std::vector< std::pair< std::string, std::string > > &	vecArgs,
		const std::string &	name,
		const Config &	cfg
	)

Member Function Documentation

process()

void mio::ProcShift::process ( const unsigned int &  param, const std::vector< MeteoData > &  ivec, std::vector< MeteoData > &  ovec  )

virtual

Implements mio::ProcessingBlock.

---

The documentation for this class was generated from the following files:

• ProcShift.h
• ProcShift.cc