nuget
Produce a flat array with the same dimension as the number of words in the dictionary
OutputLayer
01 Syntax
Microsoft.VisualBasic.MachineLearning.Transformer.OutputLayer
02 Methods
| Name | Overloads | Summary |
|---|---|---|
| .ctor | 1 | Creates the output layer. |
| Output | 1 | Projects the decoder output onto the vocabulary and applies a softmax. |
| Backward | 1 | Backpropagates through the output projection; it accepts the gradient with respect to the logits (before the softmax) and returns the gradient with respect to the decoder output. |
| ZeroGradients | 1 | Clears the gradient accumulator of the output projection. |
| MakeTrainingStep | 1 | Applies one optimizer step to the output projection. |
03 Properties
| Name | Overloads | Summary |
|---|---|---|
| LastCache | 1 | Gets the forward cache of the most recent OutputLayer.Output()) call. |
04 Fields
| Name | Overloads | Summary |
|---|---|---|
| Wo | 2 | The output projection weight matrix, shaped [embeddingSize * sequenceLength, dictionarySize]. |
05 Members
#ctor(
Int32, Int32, Int32)Creates the output layer.
Parameters
| Name | Type | Description |
|---|---|---|
sequenceLength | Int32 | Sequence length of the decoder output. |
embeddingSize | Int32 | Width of the decoder output. |
dictionarySize | Int32 | Size of the target vocabulary. |
Output(
Tensor)Projects the decoder output onto the vocabulary and applies a softmax.
Parameters
| Name | Type | Description |
|---|---|---|
input | Tensor | The decoder output. |
Returns
The softmax probabilities shaped [batch, 1, dictionarySize].
Backward(Cache,
Tensor)Backpropagates through the output projection; it accepts the gradient with respect to the logits (before the softmax) and returns the gradient with respect to the decoder output.
Parameters
| Name | Type | Description |
|---|---|---|
forwardCache | Cache | The forward cache of this step. When the decoder runs token by token the OutputLayer.LastCache is overwritten by later steps, so the snapshot of the current step must be passed explicitly. |
dLogits | Tensor | Gradient with respect to the logits. |
Returns
The gradient with respect to the decoder output.
ZeroGradients
Clears the gradient accumulator of the output projection.
MakeTrainingStep(
Double, Int32)Applies one optimizer step to the output projection.
Parameters
| Name | Type | Description |
|---|---|---|
learningRate | Double | The learning rate for this step. |
[step] | Int32 | The current step index, used by the Adam bias correction. |
LastCache
Gets the forward cache of the most recent OutputLayer.Output() call.
Wo
The output projection weight matrix, shaped [embeddingSize * sequenceLength, dictionarySize].
Wo