mio::FilterDespikingPS Class Reference

Detailed Description

Despiking filter using phase space.

This filter iteratively finds and replaces spikes (outliers) in a data sequence until no more spikes are found. The filter was developed to despike Acoustic Doppler Velocimeter (ADV) data, but might also be useful to remove outliers from other signals.

Example of filter performance for one hour of ADV-data. Outliers are successfully removed by the filter.

Two versions of this filter are implemented:

• The "Goring"-version is implemented according to following paper: Goring, D.G. and Nikora, V.I. (2002). "Despiking Acoustic Doppler Velocimeter Data" J. Hydraul. Eng., 128(1): 117-126
• And the "Mori"-version is implemented according to this paper: Mori, N., Suzuki, T. and Kakuno, S. (2007). "Noise of Acoustic Doppler Velocimeter Data in Bubbly Flows" J. Eng. Mech., 133(1):122-125

Method:

• Detection of the spikes:
  • calculate first and second derivatives of the whole signal
  • the data points plotted in phase space are enclosed by an ellipsoid defined by the standard deviations and the universal threshold (=sqrt(2*ln(number of data points))) (see figure below)
  • points outside this ellipsoid are designated as spikes (the Goring-implementation uses three 2D-projections of the ellipsoid to identify the outliers, while the Mori-implementation finds the outliers in 3D)
• Replacement of the spikes:
  • find a fit for data points around the spike
  • replace the spike with a fitted value

It takes the following arguments

• METHOD: decide which implementation to use: "Mori" or "Goring". The differences are small between both implementations. According to Mori the "Mori"-version performs slightly better. However, we suggest to use the Goring-method, because it is better tested (mandatory parameter).
• SENSITIVITY: control the sensitivity of the filter. The universal threshold is divided by this value. The larger the sensitivity the smaller the threshold (=the smaller the ellipsoid) and thus the more spikes will be detected (optional, default=1).
• INTERPOL_DEG: degree of the fit for the replacement. 1 is linear, 2 is quadratic, ..., 0 means "off" (removed spikes will be nodata and can be resampled later via [Interpolations1D]). This is useful in rare cases where the cubic fit happens to produce an unwanted peak by itself (optional, default=3 meaning cubic fit as proposed in the paper by Goring).
• INTERPOL_PTS: number of points used for the fitting (replacement) of the spikes (optional, default=24 as in Goring's paper).

There are also some hard-coded parameters:

• the maximum number of iterations for the spike detection. This is set to 50.
• When fitting, extrapolation is set to false.

2D projections of the phase space plots. Points outside the ellipses are spikes.

Author

Thiemo Theile

Date

2018-08-15

VW::filter1     = despiking
VW::arg1::sensitivity = 1
VW::arg1::method = GORING

HS::filter1            = despiking
HS::arg1::sensitivity  = 1.4
HS::arg1::method       = GORING
HS::arg1::interpol_deg = 0 ;remove spikes

#include <FilterDespikingPS.h>

Public Member Functions
	FilterDespikingPS (const std::vector< std::pair< std::string, std::string > > &vecArgs, const std::string &name, const Config &cfg)
virtual void	process (const unsigned int &param, const std::vector< MeteoData > &ivec, std::vector< MeteoData > &ovec)
	This function is the main function to do the filtering: More...
Public Member Functions inherited from mio::ProcessingBlock
virtual	~ProcessingBlock ()
virtual void	process (const unsigned int &param, const std::vector< MeteoData > &ivec, std::vector< MeteoData > &ovec)=0
virtual void	process (Date &dateStart, Date &dateEnd)
std::string	getName () const
const ProcessingProperties &	getProperties () const
const std::string	toString () const
bool	skipStation (const std::string &station_id) const
	Should the provided station be skipped in the processing? More...
bool	noStationsRestrictions () const
const std::vector< DateRange >	getTimeRestrictions () const

Additional Inherited Members
Static Public Member Functions inherited from mio::ProcessingBlock
static void	readCorrections (const std::string &filter, const std::string &filename, std::vector< double > &X, std::vector< double > &Y)
	Read a data file structured as X Y value on each lines. More...
static void	readCorrections (const std::string &filter, const std::string &filename, std::vector< double > &X, std::vector< double > &Y1, std::vector< double > &Y2)
	Read a data file structured as X Y1 Y2 value on each lines. More...
static std::vector< double >	readCorrections (const std::string &filter, const std::string &filename, const size_t &col_idx, const char &c_type, const double &init)
	Read a correction file applicable to repeating time period. More...
static std::vector< offset_spec >	readCorrections (const std::string &filter, const std::string &filename, const double &TZ, const size_t &col_idx=2)
	Read a correction file, ie a file structured as timestamps followed by values on each lines. More...
static std::map< std::string, std::vector< DateRange > >	readDates (const std::string &filter, const std::string &filename, const double &TZ)
	Read a list of date ranges by stationIDs from a file. More...
Protected Member Functions inherited from mio::ProcessingBlock
	ProcessingBlock (const std::vector< std::pair< std::string, std::string > > &vecArgs, const std::string &name, const Config &cfg)
	protected constructor only to be called by children More...
Static Protected Member Functions inherited from mio::ProcessingBlock
static void	extract_dbl_vector (const unsigned int &param, const std::vector< MeteoData > &ivec, std::vector< double > &ovec)
static void	extract_dbl_vector (const unsigned int &param, const std::vector< const MeteoData * > &ivec, std::vector< double > &ovec)
Protected Attributes inherited from mio::ProcessingBlock
const std::set< std::string >	excluded_stations
const std::set< std::string >	kept_stations
const std::vector< DateRange >	time_restrictions
ProcessingProperties	properties
const std::string	block_name
Static Protected Attributes inherited from mio::ProcessingBlock
static const double	soil_albedo = .23
static const double	snow_albedo = .85
static const double	snow_thresh = .1
	parametrize the albedo from HS More...

Constructor & Destructor Documentation

FilterDespikingPS()

mio::FilterDespikingPS::FilterDespikingPS	(	const std::vector< std::pair< std::string, std::string > > &	vecArgs,
		const std::string &	name,
		const Config &	cfg
	)

Member Function Documentation

process()

void mio::FilterDespikingPS::process ( const unsigned int &  param, const std::vector< MeteoData > &  ivec, std::vector< MeteoData > &  ovec  )

virtual

This function is the main function to do the filtering:

1. subtract the mean from the signal
2. iteratively find and replace spikes in the signal
3. add the mean back to the signal

   Parameters

   param	which value from the MeteoData should be processed (for example temperature)
   ivec	the input data (MeteoData-vector)
   ovec	the filtered data (MeteoData-vector)

Implements mio::ProcessingBlock.

---

The documentation for this class was generated from the following files:

• FilterDespikingPS.h
• FilterDespikingPS.cc