EncoderLayer

Full name Microsoft.VisualBasic.MachineLearning.Transformer.EncoderLayer Assembly Microsoft.VisualBasic.DeepLearning Members 6

One encoder layer: multi head self attention followed by a position wise feed forward network, each wrapped in a residual connection and layer normalization.

00 Remarks

The forward pass caches the AddNorm statistics (mean and inverse standard deviation) of both sub layers and their outputs; the backward pass walks the layers in reverse order. Because AddNorm yields the same gradient for both branches, each branch keeps its own copy.

01 Syntax

Microsoft.VisualBasic.MachineLearning.Transformer.EncoderLayer

02 Methods

Name	Overloads	Summary
.ctor	1	Creates an encoder layer.
Encode	1	Runs the encoder input through self attention and the feed forward network.
Backward	1	Backpropagates through both sub layers.
ZeroGradients	1	Clears the gradient accumulators of every parameter of this layer.
MakeTrainingStep	1	Applies one optimizer step to every parameter of this layer.

03 Properties

Name	Overloads	Summary
LastCache	1	Gets the forward cache of the most recent EncoderLayer.Encode()) pass.

04 Members

method .ctor #

#ctor(Int32, Int32, Int32, Int32, Int32)

Creates an encoder layer.

Parameters

Name	Type	Description
`embeddingSize`	`Int32`	Width of the model, used for the residual stream.
`dk`	`Int32`	Dimension of the query and key projections per head.
`dv`	`Int32`	Dimension of the value projection per head.
`h`	`Int32`	Number of attention heads.
`dff`	`Int32`	Hidden width of the feed forward network.

method Encode #

Encode(Tensor, Boolean)

Runs the encoder input through self attention and the feed forward network.

Parameters

Name	Type	Description
`encoderInput`	`Tensor`	The embedded encoder input.
`isTraining`	`Boolean`	When `True` dropout is applied where configured.

Returns

The output of this encoder layer.

method Backward #

Backward(Tensor, Cache)

Backpropagates through both sub layers.

Parameters

Name	Type	Description
`dOut`	`Tensor`	Gradient with respect to the output of this layer.
`forwardCache`	Cache	The forward cache snapshot that belongs to this pass.

Returns

The gradient with respect to the input of this layer.

method ZeroGradients #

ZeroGradients

Clears the gradient accumulators of every parameter of this layer.

method MakeTrainingStep #

MakeTrainingStep(Double, Int32)

Applies one optimizer step to every parameter of this layer.

Parameters

Name	Type	Description
`learningRate`	`Double`	The learning rate for this step.
`[step]`	`Int32`	The current step index, used by the Adam bias correction.

property LastCache #

LastCache

Gets the forward cache of the most recent EncoderLayer.Encode() pass.