EncoderStack

Full name Microsoft.VisualBasic.MachineLearning.Transformer.EncoderStack Assembly Microsoft.VisualBasic.DeepLearning Members 6

A stack of EncoderLayer instances, applied one after another.

00 Remarks

The encoder runs only once per translation, so the forward cache of every layer can be kept on the layer object and the backward pass simply walks the layers in reverse order.

01 Syntax

Microsoft.VisualBasic.MachineLearning.Transformer.EncoderStack

02 Methods

Name	Overloads	Summary
.ctor	1	Creates an encoder stack with the given number of identical layers.
Encode	1	Runs the embedded input through all encoder layers.
Backward	1	Backpropagates through all encoder layers.
SetDropoutNodes	1	Configures dropout on every encoder layer.
ZeroGradients	1	Clears the gradient accumulators of every encoder layer.
MakeTrainingStep	1	Applies one optimizer step to every encoder layer.

03 Members

method .ctor #

#ctor(Int32, Int32, Int32, Int32, Int32, Int32)

Creates an encoder stack with the given number of identical layers.

Parameters

Name	Type	Description
`Nx`	`Int32`	Number of encoder layers.
`embeddingSize`	`Int32`	Width of the model.
`dk`	`Int32`	Dimension of the query and key projections per head.
`dv`	`Int32`	Dimension of the value projection per head.
`h`	`Int32`	Number of attention heads.
`dff`	`Int32`	Hidden width of the feed forward network.

method Encode #

Encode(Tensor, Boolean)

Runs the embedded input through all encoder layers.

Parameters

Name	Type	Description
`word_embeddings`	`Tensor`	The embedded input sequence.
`isTraining`	`Boolean`	When `True` dropout is applied where configured.

Returns

The output of the last encoder layer.

method Backward #

Backward(Tensor)

Backpropagates through all encoder layers.

Parameters

Name	Type	Description
`dOut`	`Tensor`	Gradient with respect to the encoder output.

Returns

The gradient with respect to the encoder input (the word embeddings).

method SetDropoutNodes #

SetDropoutNodes(Double)

Configures dropout on every encoder layer.

Parameters

Name	Type	Description
`dropout`	`Double`	Dropout rate in `[0, 1)`.

method ZeroGradients #

ZeroGradients

Clears the gradient accumulators of every encoder layer.

method MakeTrainingStep #

MakeTrainingStep(Double, Int32)

Applies one optimizer step to every encoder layer.

Parameters

Name	Type	Description
`learningRate`	`Double`	The learning rate for this step.
`[step]`	`Int32`	The current step index, used by the Adam bias correction.