FeedForwardNetwork

Full name Microsoft.VisualBasic.MachineLearning.Transformer.FeedForwardNetwork Assembly Microsoft.VisualBasic.DeepLearning Members 6

Position wise feed forward network: two fully connected layers with a ReLU activation in between.

00 Remarks

Because the tensor runtime has no automatic differentiation, the forward pass caches the pre-activation (the source of the ReLU mask) and the activation (needed for the W2 gradient); the backward pass accumulates the W1, W2, b1 and b2 gradients by hand.

01 Syntax

Microsoft.VisualBasic.MachineLearning.Transformer.FeedForwardNetwork

02 Methods

Name	Overloads	Summary
.ctor	1	Creates the feed forward network and initializes its weights with He normal initialization.
FeedForward	1	Runs the two layer transformation and caches its intermediates.
Backward	1	Backpropagates through the network, accumulating the W1, W2, b1 and b2 gradients.
ZeroGradients	1	Clears the gradient accumulators of every parameter of this layer.
MakeTrainingStep	1	Applies one optimizer step to every parameter of this layer.

03 Properties

Name	Overloads	Summary
LastCache	1	Gets the forward cache of the most recent FeedForwardNetwork.FeedForward()) call.

04 Members

method .ctor #

#ctor(Int32, Int32)

Creates the feed forward network and initializes its weights with He normal initialization.

Parameters

Name	Type	Description
`dff`	`Int32`	Hidden width of the inner layer.
`embeddingSize`	`Int32`	Width of the model input and output.

method FeedForward #

FeedForward(Tensor)

Runs the two layer transformation and caches its intermediates.

Parameters

Name	Type	Description
`G`	`Tensor`	The input tensor.

Returns

The output of the second layer.

method Backward #

Backward(Cache, Tensor)

Backpropagates through the network, accumulating the W1, W2, b1 and b2 gradients.

Parameters

Name	Type	Description
`forwardCache`	Cache	The forward cache of this pass. When the decoder runs token by token the FeedForwardNetwork.LastCache of this layer is overwritten by later steps, so the snapshot of the current step must be passed explicitly.
`dOut`	`Tensor`	Gradient with respect to the output of FeedForwardNetwork.FeedForward().

Returns

The gradient with respect to the input.

method ZeroGradients #

ZeroGradients

Clears the gradient accumulators of every parameter of this layer.

method MakeTrainingStep #

MakeTrainingStep(Double, Int32)

Applies one optimizer step to every parameter of this layer.

Parameters

Name	Type	Description
`learningRate`	`Double`	The learning rate for this step.
`[step]`	`Int32`	The current step index, used by the Adam bias correction.

property LastCache #

LastCache

Gets the forward cache of the most recent FeedForwardNetwork.FeedForward() call.