DecoderStack

Full name Microsoft.VisualBasic.MachineLearning.Transformer.DecoderStack Assembly Microsoft.VisualBasic.DeepLearning Members 7

A stack of DecoderLayer instances, applied one after another.

00 Remarks

Decoding proceeds token by token and every step overwrites the forward caches of the layers, so DecoderStack.Decode() stores a snapshot of each layer's cache in DecoderStack.LastCaches for the following backward pass.

01 Syntax

Microsoft.VisualBasic.MachineLearning.Transformer.DecoderStack

02 Methods

Name	Overloads	Summary
.ctor	1	Creates a decoder stack with the given number of identical layers.
Decode	1	Runs the decoder input through all decoder layers and records the per layer forward cache snapshots.
Backward	1	Backpropagates through all decoder layers and returns the gradient with respect to the decoder input, while accumulating the gradient with respect to the encoder output.
SetDropoutNodes	1	Configures dropout on every decoder layer.
ZeroGradients	1	Clears the gradient accumulators of every decoder layer.
MakeTrainingStep	1	Applies one optimizer step to every decoder layer.

03 Properties

Name	Overloads	Summary
LastCaches	1	Gets the forward cache snapshot of every layer for the most recent DecoderStack.Decode()) step.

04 Members

method .ctor #

#ctor(Int32, Int32, Int32, Int32, Int32, Int32)

Creates a decoder stack with the given number of identical layers.

Parameters

Name	Type	Description
`Nx`	`Int32`	Number of decoder layers.
`embeddingSize`	`Int32`	Width of the model.
`dk`	`Int32`	Dimension of the query and key projections per head.
`dv`	`Int32`	Dimension of the value projection per head.
`h`	`Int32`	Number of attention heads.
`dff`	`Int32`	Hidden width of the feed forward network.

method Decode #

Decode(Tensor, Tensor, Boolean)

Runs the decoder input through all decoder layers and records the per layer forward cache snapshots.

Parameters

Name	Type	Description
`encoderOutput`	`Tensor`	The output of the encoder stack.
`word_embeddings`	`Tensor`	The embedded decoder input.
`isTraining`	`Boolean`	When `True` dropout is applied where configured.

Returns

The output of the last decoder layer.

method Backward #

Backward(List(Of Cache), Tensor, Tensor)

Backpropagates through all decoder layers and returns the gradient with respect to the decoder input, while accumulating the gradient with respect to the encoder output.

Parameters

Name	Type	Description
`caches`	`List(Of Cache)`	The forward cache snapshots that belong to this decode step.
`dOut`	`Tensor`	Gradient with respect to the decoder output of this step.
`dEncoderOutput`	`Tensor`	Accumulator for the gradient with respect to the encoder output, shared across decode steps.

Returns

The gradient with respect to the decoder input.

method SetDropoutNodes #

SetDropoutNodes(Double)

Configures dropout on every decoder layer.

Parameters

Name	Type	Description
`dropout`	`Double`	Dropout rate in `[0, 1)`.

method ZeroGradients #

ZeroGradients

Clears the gradient accumulators of every decoder layer.

method MakeTrainingStep #

MakeTrainingStep(Double, Int32)

Applies one optimizer step to every decoder layer.

Parameters

Name	Type	Description
`learningRate`	`Double`	The learning rate for this step.
`[step]`	`Int32`	The current step index, used by the Adam bias correction.

property LastCaches #

LastCaches

Gets the forward cache snapshot of every layer for the most recent DecoderStack.Decode() step.