LSTM recurrent layer. More...

#include <opencv2/dnn/all_layers.hpp>

Inheritance diagram for cv::dnn::LSTMLayer:

Collaboration diagram for cv::dnn::LSTMLayer:

[legend]

Public Member Functions
virtual void	applyHalideScheduler (Ptr< BackendNode > &node, const std::vector< Mat *> &inputs, const std::vector< Mat > &outputs, int targetId) const
	Automatic Halide scheduling based on layer hyper-parameters. More...

virtual void	clear ()
	Clears the algorithm state. More...

virtual bool	empty () const
	Returns true if the Algorithm is empty (e.g. More...

virtual void	finalize (const std::vector< Mat *> &input, std::vector< Mat > &output)
	Computes and sets internal parameters according to inputs, outputs and blobs. More...

virtual void	finalize (InputArrayOfArrays inputs, OutputArrayOfArrays outputs)
	Computes and sets internal parameters according to inputs, outputs and blobs. More...

void	finalize (const std::vector< Mat > &inputs, std::vector< Mat > &outputs)
	This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts. More...

std::vector< Mat >	finalize (const std::vector< Mat > &inputs)
	This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts. More...

virtual void	forward (std::vector< Mat *> &input, std::vector< Mat > &output, std::vector< Mat > &internals)
	Given the `input` blobs, computes the output `blobs`. More...

virtual void	forward (InputArrayOfArrays inputs, OutputArrayOfArrays outputs, OutputArrayOfArrays internals)
	Given the `input` blobs, computes the output `blobs`. More...

void	forward_fallback (InputArrayOfArrays inputs, OutputArrayOfArrays outputs, OutputArrayOfArrays internals)
	Given the `input` blobs, computes the output `blobs`. More...

virtual String	getDefaultName () const
	Returns the algorithm string identifier. More...

virtual int64	getFLOPS (const std::vector< MatShape > &inputs, const std::vector< MatShape > &outputs) const

virtual bool	getMemoryShapes (const std::vector< MatShape > &inputs, const int requiredOutputs, std::vector< MatShape > &outputs, std::vector< MatShape > &internals) const

virtual void	getScaleShift (Mat &scale, Mat &shift) const
	Returns parameters of layers with channel-wise multiplication and addition. More...

virtual Ptr< BackendNode >	initHalide (const std::vector< Ptr< BackendWrapper > > &inputs)
	Returns Halide backend node. More...

virtual Ptr< BackendNode >	initInfEngine (const std::vector< Ptr< BackendWrapper > > &inputs)

virtual Ptr< BackendNode >	initVkCom (const std::vector< Ptr< BackendWrapper > > &inputs)

int	inputNameToIndex (String inputName) CV_OVERRIDE
	Returns index of input blob into the input array. More...

int	outputNameToIndex (const String &outputName) CV_OVERRIDE
	Returns index of output blob in output array. More...

virtual void	read (const FileNode &fn)
	Reads algorithm parameters from a file storage. More...

void	run (const std::vector< Mat > &inputs, std::vector< Mat > &outputs, std::vector< Mat > &internals)
	Allocates layer and computes output. More...

virtual void	save (const String &filename) const
	Saves the algorithm to a file. More...

virtual bool	setActivation (const Ptr< ActivationLayer > &layer)
	Tries to attach to the layer the subsequent activation layer, i.e. More...

virtual void	setOutShape (const MatShape &outTailShape=MatShape())=0
	Specifies shape of output blob which will be [[`T`], `N`] + `outTailShape`. More...

void	setParamsFrom (const LayerParams &params)
	Initializes only name, type and blobs fields. More...

virtual void	setProduceCellOutput (bool produce=false)=0
	If this flag is set to true then layer will produce \( c_t \) as second output. More...

virtual void	setUseTimstampsDim (bool use=true)=0
	Specifies either interpret first dimension of input blob as timestamp dimenion either as sample. More...

virtual void	setWeights (const Mat &Wh, const Mat &Wx, const Mat &b)=0
	Set trained weights for LSTM layer. More...

virtual bool	supportBackend (int backendId)
	Ask layer if it support specific backend for doing computations. More...

virtual Ptr< BackendNode >	tryAttach (const Ptr< BackendNode > &node)
	Implement layers fusing. More...

virtual bool	tryFuse (Ptr< Layer > &top)
	Try to fuse current layer with a next one. More...

virtual void	unsetAttached ()
	"Deattaches" all the layers, attached to particular layer. More...

virtual void	write (FileStorage &fs) const
	Stores algorithm parameters in a file storage. More...

void	write (const Ptr< FileStorage > &fs, const String &name=String()) const
	simplified API for language bindings This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts. More...

Static Public Member Functions
static Ptr< LSTMLayer >	create (const LayerParams &params)
	Creates instance of LSTM layer. More...

template<typename _Tp >
static Ptr< _Tp >	load (const String &filename, const String &objname=String())
	Loads algorithm from the file. More...

template<typename _Tp >
static Ptr< _Tp >	loadFromString (const String &strModel, const String &objname=String())
	Loads algorithm from a String. More...

template<typename _Tp >
static Ptr< _Tp >	read (const FileNode &fn)
	Reads algorithm from the file node. More...

Public Attributes
std::vector< Mat >	blobs
	List of learned parameters must be stored here to allow read them by using Net::getParam(). More...

String	name
	Name of the layer instance, can be used for logging or other internal purposes. More...

int	preferableTarget
	prefer target for layer forwarding More...

String	type
	Type name which was used for creating layer by layer factory. More...

Protected Member Functions
void	writeFormat (FileStorage &fs) const

Detailed Description

LSTM recurrent layer.

Member Function Documentation

◆ applyHalideScheduler()

virtual void cv::dnn::Layer::applyHalideScheduler	(	Ptr< BackendNode > &	node,
		const std::vector< Mat *> &	inputs,
		const std::vector< Mat > &	outputs,
		int	targetId
	)		const

virtualinherited

Automatic Halide scheduling based on layer hyper-parameters.

Parameters

[in]	node	Backend node with Halide functions.
[in]	inputs	Blobs that will be used in forward invocations.
[in]	outputs	Blobs that will be used in forward invocations.
[in]	targetId	Target identifier

See also: BackendNode, Target

Layer don't use own Halide::Func members because we can have applied layers fusing. In this way the fused function should be scheduled.

◆ clear()

virtual void cv::Algorithm::clear ( )

inlinevirtualinherited

Clears the algorithm state.

Reimplemented in cv::FlannBasedMatcher, and cv::DescriptorMatcher.

◆ create()

static Ptr<LSTMLayer> cv::dnn::LSTMLayer::create ( const LayerParams & params )

static

Creates instance of LSTM layer.

◆ empty()

virtual bool cv::Algorithm::empty ( ) const

inlinevirtualinherited

Returns true if the Algorithm is empty (e.g.

in the very beginning or after unsuccessful read

Reimplemented in cv::DescriptorMatcher, cv::ml::StatModel, cv::Feature2D, and cv::BaseCascadeClassifier.

◆ finalize() [1/4]

virtual void cv::dnn::Layer::finalize	(	const std::vector< Mat *> &	input,
		std::vector< Mat > &	output
	)

virtualinherited

Computes and sets internal parameters according to inputs, outputs and blobs.

Deprecated:: Use Layer::finalize(InputArrayOfArrays, OutputArrayOfArrays) instead

Parameters

[in]	input	vector of already allocated input blobs
[out]	output	vector of already allocated output blobs

If this method is called after network has allocated all memory for input and output blobs and before inferencing.

◆ finalize() [2/4]

virtual void cv::dnn::Layer::finalize	(	InputArrayOfArrays	inputs,
		OutputArrayOfArrays	outputs
	)

virtualinherited

Computes and sets internal parameters according to inputs, outputs and blobs.

Parameters

[in]	inputs	vector of already allocated input blobs
[out]	outputs	vector of already allocated output blobs

If this method is called after network has allocated all memory for input and output blobs and before inferencing.

◆ finalize() [3/4]

void cv::dnn::Layer::finalize	(	const std::vector< Mat > &	inputs,
		std::vector< Mat > &	outputs
	)

inherited

This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

Deprecated:: Use Layer::finalize(InputArrayOfArrays, OutputArrayOfArrays) instead

◆ finalize() [4/4]

std::vector<Mat> cv::dnn::Layer::finalize ( const std::vector< Mat > & inputs )

inherited

This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

Deprecated:: Use Layer::finalize(InputArrayOfArrays, OutputArrayOfArrays) instead

◆ forward() [1/2]

virtual void cv::dnn::Layer::forward	(	std::vector< Mat *> &	input,
		std::vector< Mat > &	output,
		std::vector< Mat > &	internals
	)

virtualinherited

Given the input blobs, computes the output blobs.

Deprecated:: Use Layer::forward(InputArrayOfArrays, OutputArrayOfArrays, OutputArrayOfArrays) instead

Parameters

[in]	input	the input blobs.
[out]	output	allocated output blobs, which will store results of the computation.
[out]	internals	allocated internal blobs

◆ forward() [2/2]

virtual void cv::dnn::Layer::forward	(	InputArrayOfArrays	inputs,
		OutputArrayOfArrays	outputs,
		OutputArrayOfArrays	internals
	)

virtualinherited

Given the input blobs, computes the output blobs.

Parameters

[in]	inputs	the input blobs.
[out]	outputs	allocated output blobs, which will store results of the computation.
[out]	internals	allocated internal blobs

◆ forward_fallback()

void cv::dnn::Layer::forward_fallback	(	InputArrayOfArrays	inputs,
		OutputArrayOfArrays	outputs,
		OutputArrayOfArrays	internals
	)

inherited

Given the input blobs, computes the output blobs.

Parameters

[in]	inputs	the input blobs.
[out]	outputs	allocated output blobs, which will store results of the computation.
[out]	internals	allocated internal blobs

◆ getDefaultName()

virtual String cv::Algorithm::getDefaultName ( ) const

virtualinherited

Returns the algorithm string identifier.

This string is used as top level xml/yml node tag when the object is saved to a file or string.

Reimplemented in cv::AKAZE, cv::KAZE, cv::SimpleBlobDetector, cv::GFTTDetector, cv::AgastFeatureDetector, cv::FastFeatureDetector, cv::MSER, cv::ORB, cv::BRISK, and cv::Feature2D.

◆ getFLOPS()

virtual int64 cv::dnn::Layer::getFLOPS	(	const std::vector< MatShape > &	inputs,
		const std::vector< MatShape > &	outputs
	)		const

inlinevirtualinherited

◆ getMemoryShapes()

virtual bool cv::dnn::Layer::getMemoryShapes	(	const std::vector< MatShape > &	inputs,
		const int	requiredOutputs,
		std::vector< MatShape > &	outputs,
		std::vector< MatShape > &	internals
	)		const

virtualinherited

◆ getScaleShift()

virtual void cv::dnn::Layer::getScaleShift	(	Mat &	scale,
		Mat &	shift
	)		const

virtualinherited

Returns parameters of layers with channel-wise multiplication and addition.

Parameters

[out]	scale	Channel-wise multipliers. Total number of values should be equal to number of channels.
[out]	shift	Channel-wise offsets. Total number of values should be equal to number of channels.

Some layers can fuse their transformations with further layers. In example, convolution + batch normalization. This way base layer use weights from layer after it. Fused layer is skipped. By default, scale and shift are empty that means layer has no element-wise multiplications or additions.

◆ initHalide()

virtual Ptr<BackendNode> cv::dnn::Layer::initHalide ( const std::vector< Ptr< BackendWrapper > > & inputs )

virtualinherited

Returns Halide backend node.

Parameters

[in] inputs Input Halide buffers.

See also: BackendNode, BackendWrapper

Input buffers should be exactly the same that will be used in forward invocations. Despite we can use Halide::ImageParam based on input shape only, it helps prevent some memory management issues (if something wrong, Halide tests will be failed).

◆ initInfEngine()

virtual Ptr<BackendNode> cv::dnn::Layer::initInfEngine ( const std::vector< Ptr< BackendWrapper > > & inputs )

virtualinherited

◆ initVkCom()

virtual Ptr<BackendNode> cv::dnn::Layer::initVkCom ( const std::vector< Ptr< BackendWrapper > > & inputs )

virtualinherited

◆ inputNameToIndex()

int cv::dnn::LSTMLayer::inputNameToIndex ( String inputName )

virtual

Returns index of input blob into the input array.

Parameters

inputName label of input blob

Each layer input and output can be labeled to easily identify them using "%<layer_name%>[.output_name]" notation. This method maps label of input blob to its index into input vector.

Reimplemented from cv::dnn::Layer.

◆ load()

template<typename _Tp >

static Ptr<_Tp> cv::Algorithm::load	(	const String &	filename,
		const String &	objname = `String()`
	)

inlinestaticinherited

Loads algorithm from the file.

Parameters

filename	Name of the file to read.
objname	The optional name of the node to read (if empty, the first top-level node will be used)

This is static template method of Algorithm. It's usage is following (in the case of SVM):

Ptr<SVM> svm = Algorithm::load<SVM>("my_svm_model.xml");

In order to make this method work, the derived class must overwrite Algorithm::read(const FileNode& fn).

References CV_Assert, cv::FileNode::empty(), cv::FileStorage::getFirstTopLevelNode(), cv::FileStorage::isOpened(), and cv::FileStorage::READ.

Here is the call graph for this function:

◆ loadFromString()

template<typename _Tp >

static Ptr<_Tp> cv::Algorithm::loadFromString	(	const String &	strModel,
		const String &	objname = `String()`
	)

inlinestaticinherited

Loads algorithm from a String.

Parameters

strModel	The string variable containing the model you want to load.
objname	The optional name of the node to read (if empty, the first top-level node will be used)

This is static template method of Algorithm. It's usage is following (in the case of SVM):

Ptr<SVM> svm = Algorithm::loadFromString<SVM>(myStringModel);

References CV_WRAP, cv::FileNode::empty(), cv::FileStorage::getFirstTopLevelNode(), cv::FileStorage::MEMORY, and cv::FileStorage::READ.

Here is the call graph for this function:

◆ outputNameToIndex()

int cv::dnn::LSTMLayer::outputNameToIndex ( const String & outputName )

virtual

Returns index of output blob in output array.

See also: inputNameToIndex()

Reimplemented from cv::dnn::Layer.

◆ read() [1/2]

virtual void cv::Algorithm::read ( const FileNode & fn )

inlinevirtualinherited

Reads algorithm parameters from a file storage.

Reimplemented in cv::FlannBasedMatcher, cv::DescriptorMatcher, and cv::Feature2D.

◆ read() [2/2]

template<typename _Tp >

static Ptr<_Tp> cv::Algorithm::read ( const FileNode & fn )

inlinestaticinherited

Reads algorithm from the file node.

This is static template method of Algorithm. It's usage is following (in the case of SVM):

cv::FileStorage fsRead("example.xml", FileStorage::READ);

Ptr<SVM> svm = Algorithm::read<SVM>(fsRead.root());

In order to make this method work, the derived class must overwrite Algorithm::read(const FileNode& fn) and also have static create() method without parameters (or with all the optional parameters)

◆ run()

void cv::dnn::Layer::run	(	const std::vector< Mat > &	inputs,
		std::vector< Mat > &	outputs,
		std::vector< Mat > &	internals
	)

inherited

Allocates layer and computes output.

Deprecated:: This method will be removed in the future release.

◆ save()

virtual void cv::Algorithm::save ( const String & filename ) const

virtualinherited

Saves the algorithm to a file.

In order to make this method work, the derived class must implement Algorithm::write(FileStorage& fs).

◆ setActivation()

virtual bool cv::dnn::Layer::setActivation ( const Ptr< ActivationLayer > & layer )

virtualinherited

Tries to attach to the layer the subsequent activation layer, i.e.

do the layer fusion in a partial case.

Parameters

[in] layer The subsequent activation layer.

Returns true if the activation layer has been attached successfully.

◆ setOutShape()

virtual void cv::dnn::LSTMLayer::setOutShape ( const MatShape & outTailShape = MatShape() )

pure virtual

Specifies shape of output blob which will be [[T], N] + outTailShape.

If this parameter is empty or unset then outTailShape = [Wh.size(0)] will be used, where Wh is parameter from setWeights().

◆ setParamsFrom()

void cv::dnn::Layer::setParamsFrom ( const LayerParams & params )

inherited

Initializes only name, type and blobs fields.

◆ setProduceCellOutput()

virtual void cv::dnn::LSTMLayer::setProduceCellOutput ( bool produce = false )

pure virtual

If this flag is set to true then layer will produce \( c_t \) as second output.

Deprecated:: Use flag use_timestamp_dim in LayerParams.

Shape of the second output is the same as first output.

◆ setUseTimstampsDim()

virtual void cv::dnn::LSTMLayer::setUseTimstampsDim ( bool use = true )

pure virtual

Specifies either interpret first dimension of input blob as timestamp dimenion either as sample.

Deprecated:: Use flag produce_cell_output in LayerParams.

If flag is set to true then shape of input blob will be interpreted as [T, N, [data dims]] where T specifies number of timestamps, N is number of independent streams. In this case each forward() call will iterate through T timestamps and update layer's state T times.

If flag is set to false then shape of input blob will be interpreted as [N, [data dims]]. In this case each forward() call will make one iteration and produce one timestamp with shape [N, [out dims]].

◆ setWeights()

virtual void cv::dnn::LSTMLayer::setWeights	(	const Mat &	Wh,
		const Mat &	Wx,
		const Mat &	b
	)

pure virtual

Set trained weights for LSTM layer.

Deprecated:: Use LayerParams::blobs instead.

LSTM behavior on each step is defined by current input, previous output, previous cell state and learned weights.

Let \(x_t\) be current input, \(h_t\) be current output, \(c_t\) be current state. Than current output and current cell state is computed as follows:

\begin{eqnarray*} h_t &= o_t \odot tanh(c_t), \\ c_t &= f_t \odot c_{t-1} + i_t \odot g_t, \\ \end{eqnarray*}

where \(\odot\) is per-element multiply operation and \(i_t, f_t, o_t, g_t\) is internal gates that are computed using learned wights.

Gates are computed as follows:

\begin{eqnarray*} i_t &= sigmoid&(W_{xi} x_t + W_{hi} h_{t-1} + b_i), \\ f_t &= sigmoid&(W_{xf} x_t + W_{hf} h_{t-1} + b_f), \\ o_t &= sigmoid&(W_{xo} x_t + W_{ho} h_{t-1} + b_o), \\ g_t &= tanh &(W_{xg} x_t + W_{hg} h_{t-1} + b_g), \\ \end{eqnarray*}

where \(W_{x?}\), \(W_{h?}\) and \(b_{?}\) are learned weights represented as matrices: \(W_{x?} \in R^{N_h \times N_x}\), \(W_{h?} \in R^{N_h \times N_h}\), \(b_? \in R^{N_h}\).

For simplicity and performance purposes we use \( W_x = [W_{xi}; W_{xf}; W_{xo}, W_{xg}] \) (i.e. \(W_x\) is vertical concatenation of \( W_{x?} \)), \( W_x \in R^{4N_h \times N_x} \). The same for \( W_h = [W_{hi}; W_{hf}; W_{ho}, W_{hg}], W_h \in R^{4N_h \times N_h} \) and for \( b = [b_i; b_f, b_o, b_g]\), \(b \in R^{4N_h} \).

Parameters

Wh	is matrix defining how previous output is transformed to internal gates (i.e. according to above mentioned notation is \( W_h \))
Wx	is matrix defining how current input is transformed to internal gates (i.e. according to above mentioned notation is \( W_x \))
b	is bias vector (i.e. according to above mentioned notation is \( b \))

◆ supportBackend()

virtual bool cv::dnn::Layer::supportBackend ( int backendId )

virtualinherited

Ask layer if it support specific backend for doing computations.

Parameters

[in] backendId computation backend identifier.

See also: Backend

◆ tryAttach()

virtual Ptr<BackendNode> cv::dnn::Layer::tryAttach ( const Ptr< BackendNode > & node )

virtualinherited

Implement layers fusing.

Parameters

[in] node Backend node of bottom layer.

See also: BackendNode

Actual for graph-based backends. If layer attached successfully, returns non-empty cv::Ptr to node of the same backend. Fuse only over the last function.

◆ tryFuse()

virtual bool cv::dnn::Layer::tryFuse ( Ptr< Layer > & top )

virtualinherited

Try to fuse current layer with a next one.

Parameters

[in] top Next layer to be fused.

Returns: True if fusion was performed.

◆ unsetAttached()

virtual void cv::dnn::Layer::unsetAttached ( )

virtualinherited

"Deattaches" all the layers, attached to particular layer.

◆ write() [1/2]

virtual void cv::Algorithm::write ( FileStorage & fs ) const

inlinevirtualinherited

Stores algorithm parameters in a file storage.

Reimplemented in cv::FlannBasedMatcher, cv::DescriptorMatcher, and cv::Feature2D.

References CV_WRAP.

Referenced by cv::Feature2D::write(), and cv::DescriptorMatcher::write().

Here is the caller graph for this function:

◆ write() [2/2]

void cv::Algorithm::write	(	const Ptr< FileStorage > &	fs,
		const String &	name = `String()`
	)		const

inherited

simplified API for language bindings This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

◆ writeFormat()

void cv::Algorithm::writeFormat ( FileStorage & fs ) const

protectedinherited

Member Data Documentation

◆ blobs

std::vector<Mat> cv::dnn::Layer::blobs

inherited

List of learned parameters must be stored here to allow read them by using Net::getParam().

◆ name

String cv::dnn::Layer::name

inherited

Name of the layer instance, can be used for logging or other internal purposes.

◆ preferableTarget

int cv::dnn::Layer::preferableTarget

inherited

prefer target for layer forwarding

◆ type

String cv::dnn::Layer::type

inherited

Type name which was used for creating layer by layer factory.

The documentation for this class was generated from the following file:

opencv2/dnn/all_layers.hpp

Public Member Functions

Static Public Member Functions

Public Attributes

Protected Member Functions

Detailed Description

Member Function Documentation

◆ applyHalideScheduler()

◆ clear()

◆ create()

◆ empty()

◆ finalize() [1/4]

◆ finalize() [2/4]

◆ finalize() [3/4]

◆ finalize() [4/4]

◆ forward() [1/2]

◆ forward() [2/2]

◆ forward_fallback()

◆ getDefaultName()

◆ getFLOPS()

◆ getMemoryShapes()

◆ getScaleShift()

◆ initHalide()

◆ initInfEngine()

◆ initVkCom()

◆ inputNameToIndex()

◆ load()

◆ loadFromString()

◆ outputNameToIndex()

◆ read() [1/2]

◆ read() [2/2]

◆ run()

◆ save()

◆ setActivation()

◆ setOutShape()

◆ setParamsFrom()

◆ setProduceCellOutput()

◆ setUseTimstampsDim()

◆ setWeights()

◆ supportBackend()

◆ tryAttach()

◆ tryFuse()

◆ unsetAttached()

◆ write() [1/2]

◆ write() [2/2]

◆ writeFormat()

Member Data Documentation

◆ blobs

◆ name

◆ preferableTarget

◆ type