MultiHeadAttention

Full name Microsoft.VisualBasic.MachineLearning.Transformer.MultiHeadAttention Assembly Microsoft.VisualBasic.DeepLearning Members 7

Scaled dot product multi head attention, implemented with the hand written operators of TensorOps and an explicit backward pass.

00 Remarks

The forward pass caches the projected Q/K/V tensors, the per head softmax probabilities and the concatenated heads; the backward pass walks the attention computation in reverse order.

01 Syntax

Microsoft.VisualBasic.MachineLearning.Transformer.MultiHeadAttention

02 Methods

Name	Overloads	Summary
.ctor	1	Creates a multi head attention sub layer.
Update	2	Cross attention: queries come from queries, keys and values from encoderOutput.
Backward	1	Backpropagates through the attention computation, returning the gradient with respect to the query input and accumulating the parameter gradients of every linear projection.
ZeroGradients	1	Clears the gradient accumulators of every parameter of this layer.
MakeTrainingStep	1	Applies one optimizer step to every projection of this layer.

03 Properties

Name	Overloads	Summary
LastCache	1	Gets the forward cache of the most recent MultiHeadAttention.Update()) call.

04 Members

method .ctor #

#ctor(Int32, Int32, Int32, Int32, Boolean)

Creates a multi head attention sub layer.

Parameters

Name	Type	Description
`dk`	`Int32`	Dimension of the query and key projections per head.
`dv`	`Int32`	Dimension of the value projection per head.
`nr_heads`	`Int32`	Number of attention heads.
`embeddingSize`	`Int32`	Width of the model, used by the output projection.
`mask`	`Boolean`	When `True` the upper triangle of the attention scores is masked (causal attention).

method Update overload 2 #

Update(Tensor)

Self attention: queries, keys and values all come from inputData.

Parameters

Name	Type	Description
`inputData`	`Tensor`	The input sequence.

Returns

The attention output.

method Update #

Update(Tensor, Tensor)

Cross attention: queries come from queries, keys and values from encoderOutput.

Parameters

Name	Type	Description
`encoderOutput`	`Tensor`	The encoder output used as keys and values.
`queries`	`Tensor`	The decoder representation used as queries.

Returns

The attention output.

method Backward #

Backward(Cache, Tensor, Tensor)

Backpropagates through the attention computation, returning the gradient with respect to the query input and accumulating the parameter gradients of every linear projection.

Parameters

Name	Type	Description
`forwardCache`	Cache	The forward cache of this pass. When the decoder runs token by token the MultiHeadAttention.LastCache is overwritten by later steps, so the snapshot of the current step must be passed explicitly.
`dOut`	`Tensor`	Gradient with respect to the attention output (`Concat · Wo`).
`dEncoderOutput`	`Tensor`	Receives the gradient with respect to the encoder output for cross attention; `Nothing` for self attention.

Returns

The gradient with respect to the query input.

method ZeroGradients #

ZeroGradients

Clears the gradient accumulators of every parameter of this layer.

method MakeTrainingStep #

MakeTrainingStep(Double, Int32)

Applies one optimizer step to every projection of this layer.

Parameters

Name	Type	Description
`learningRate`	`Double`	The learning rate for this step.
`[step]`	`Int32`	The current step index, used by the Adam bias correction.

property LastCache #

LastCache

Gets the forward cache of the most recent MultiHeadAttention.Update() call.