DecoderLayer

Full name Microsoft.VisualBasic.MachineLearning.Transformer.DecoderLayer Assembly Microsoft.VisualBasic.DeepLearning Members 9

One decoder layer: masked self attention, cross attention and a position wise feed forward network, each wrapped in a residual connection and layer normalization.

00 Remarks

Every sub layer caches its AddNorm statistics and its forward output for the backward pass. Because decoding proceeds token by token, the forward cache of this layer is overwritten on every step, so callers (DecoderStack, TransformerModel) must keep a snapshot of the cache of each step.

01 Syntax

Microsoft.VisualBasic.MachineLearning.Transformer.DecoderLayer

02 Methods

Name	Overloads	Summary
.ctor	1	Creates a decoder layer.
Decode	1	Runs one decoder step: masked self attention, cross attention over the encoder output and a feed forward network.
Backward	1	Backpropagates through the three sub layers and returns the gradient with respect to the decoder input, while accumulating the gradient with respect to the encoder output.
SetDropoutNodes	1	Configures dropout and draws new dropout masks for the three sub layers.
ZeroGradients	1	Clears the gradient accumulators of every parameter of this layer.
MakeTrainingStep	1	Applies one optimizer step to every parameter of this layer.

03 Properties

Name	Overloads	Summary
LastCache	1	Gets the forward cache of the most recent DecoderLayer.Decode()) step.

04 Fields

Name	Overloads	Summary
ff	2	The position wise feed forward sub layer of this decoder layer.

05 Members

method .ctor #

#ctor(Int32, Int32, Int32, Int32, Int32)

Creates a decoder layer.

Parameters

Name	Type	Description
`embeddingSize`	`Int32`	Width of the model, used for the residual stream.
`dk`	`Int32`	Dimension of the query and key projections per head.
`dv`	`Int32`	Dimension of the value projection per head.
`h`	`Int32`	Number of attention heads.
`dff`	`Int32`	Hidden width of the feed forward network.

method Decode #

Decode(Tensor, Tensor, Boolean)

Runs one decoder step: masked self attention, cross attention over the encoder output and a feed forward network.

Parameters

Name	Type	Description
`encoderOutput`	`Tensor`	The output of the encoder stack used by the cross attention sub layer.
`decoderInput`	`Tensor`	The embedded decoder input of this step.
`isTraining`	`Boolean`	When `True` dropout is applied where configured.

Returns

The decoder output of this step.

method Backward #

Backward(Cache, Tensor, Tensor)

Backpropagates through the three sub layers and returns the gradient with respect to the decoder input, while accumulating the gradient with respect to the encoder output.

Parameters

Name	Type	Description
`forwardCache`	Cache	The forward cache snapshot that belongs to this decode step.
`dOut`	`Tensor`	Gradient with respect to the decoder output of this step.
`dEncoderOutput`	`Tensor`	Accumulator for the gradient with respect to the encoder output.

Returns

The gradient with respect to the decoder input.

method SetDropoutNodes #

SetDropoutNodes(Double)

Configures dropout and draws new dropout masks for the three sub layers.

Parameters

Name	Type	Description
`dropoutRate`	`Double`	Dropout rate in `[0, 1)`.

method ZeroGradients #

ZeroGradients

Clears the gradient accumulators of every parameter of this layer.

method MakeTrainingStep #

MakeTrainingStep(Double, Int32)

Applies one optimizer step to every parameter of this layer.

Parameters

Name	Type	Description
`learningRate`	`Double`	The learning rate for this step.
`[step]`	`Int32`	The current step index, used by the Adam bias correction.

property LastCache #

LastCache

Gets the forward cache of the most recent DecoderLayer.Decode() step.

field ff #

The position wise feed forward sub layer of this decoder layer.

field ff #