nuget
Use a learned embedding layer to reduce the size of the word embedding space.
Embedding
01 Syntax
Microsoft.VisualBasic.MachineLearning.Transformer.Embedding
02 Methods
| Name | Overloads | Summary |
|---|---|---|
| .ctor | 1 | Creates the embedding layer and builds the one-hot dictionary from the given sentences. |
| Embed | 1 | Looks up the embedding of every word, adds the positional encoding and optionally applies dropout. |
| Backward | 1 | Backpropagates through the embedding layer: the gradient of every (sentence, position) pair is scattered back into the row of the embedding matrix that belongs to the word. |
| CalculateLossAndGradient | 1 | Computes the cross entropy loss and its gradient with respect to the output layer logits. |
| GetWords | 1 | Gets the words that correspond to the given dictionary indices. |
| GetWordIndex | 1 | Gets the dictionary index of a word. |
| AllWordsInDictionary | 1 | Checks whether every word of the given sentences exists in the dictionary. |
| OneHotEmbedding | 1 | Encode all words in a dictionary with one-hot embedding |
| AddPositionalEncoding | 1 | Add positional encoding to embedded words according to "Attention is all you need" |
| SetDropoutNodes | 1 | Configures dropout for the embedding output and draws a new dropout mask. |
| ZeroGradients | 1 | Clears the gradient accumulator of the embedding matrix. |
| MakeTrainingStep | 1 | Applies one optimizer step to the embedding matrix. |
03 Properties
| Name | Overloads | Summary |
|---|---|---|
| DictionarySize | 1 | Gets the number of distinct words in the dictionary. |
| EmbeddingSize | 1 | Gets the width of the embedding vectors. |
| SequenceLength | 1 | Gets the fixed sequence length used by this embedding layer. |
| Parameters | 1 | 与 Embedding.embeddingLayer 同形的梯度累加器。 |
04 Fields
| Name | Overloads | Summary |
|---|---|---|
| embeddingLayer | 1 | Learned linear embedding layer |
05 Members
#ctor(
Int32, Int32, List(Of List(Of String)))Creates the embedding layer and builds the one-hot dictionary from the given sentences.
Parameters
| Name | Type | Description |
|---|---|---|
embeddingSize | Int32 | Width of the embedding vectors. |
sequenceLength | Int32 | Fixed sequence length of the inputs. |
sentences | List(Of List(Of String)) | The sentences used to build the word dictionary. |
Embed(
List(Of List(Of String)), Boolean)Looks up the embedding of every word, adds the positional encoding and optionally applies dropout.
Parameters
| Name | Type | Description |
|---|---|---|
sentences | List(Of List(Of String)) | The batch of tokenized sentences. |
isTraining | Boolean | When |
Returns
The embedded sentences, shaped [batch, seq, emb].
Backward(
Tensor, List(Of List(Of String)), Boolean)Backpropagates through the embedding layer: the gradient of every (sentence, position) pair is scattered back into the row of the embedding matrix that belongs to the word.
Parameters
| Name | Type | Description |
|---|---|---|
dWordEmbeddings | Tensor | Gradient with respect to the Embedding.Embed() output, shaped |
sentences | List(Of List(Of String)) | The sentences used during the forward pass, needed to recover the word indices. |
applyDropout | Boolean | Whether dropout was applied during the forward pass. |
CalculateLossAndGradient(
Tensor, List(Of List(Of String)), Int32, Tensor)Computes the cross entropy loss and its gradient with respect to the output layer logits.
Remarks
For softmax followed by cross entropy the derivative is simply d(logits) = softmax - onehot, so the chain rule through log and softmax is skipped. The returned value is the unscaled loss contribution of this step; the caller divides it by sequenceLength * batchSize.
Parameters
| Name | Type | Description |
|---|---|---|
filteredOutput | Tensor | Output layer softmax probabilities, shaped |
correctSentences | List(Of List(Of String)) | The correct target sentences. |
w | Int32 | Index of the word position that is currently being predicted. |
dLogits | Tensor | Receives the gradient with respect to the logits, with the same shape as filteredOutput. |
Returns
The unscaled cross entropy loss of this step.
GetWords(
Int32())Gets the words that correspond to the given dictionary indices.
Parameters
| Name | Type | Description |
|---|---|---|
indexes | Int32() | The dictionary indices. |
Returns
The words stored at those indices.
GetWordIndex(
String)Gets the dictionary index of a word.
Parameters
| Name | Type | Description |
|---|---|---|
word | String | The word to look up. |
Returns
The dictionary index of word.
AllWordsInDictionary(
List(Of List(Of String)), String)Checks whether every word of the given sentences exists in the dictionary.
Parameters
| Name | Type | Description |
|---|---|---|
sentences | List(Of List(Of String)) | The sentences to check. |
wordNotInDictionary | String | Receives the first word that is missing from the dictionary. |
Returns
True when all words are known; otherwise False.
OneHotEmbedding(
List(Of List(Of String)))Encode all words in a dictionary with one-hot embedding
Parameters
| Name | Type | Description |
|---|---|---|
sentences | List(Of List(Of String)) | - |
AddPositionalEncoding(
Tensor, Int32, Int32)Add positional encoding to embedded words according to "Attention is all you need"
Parameters
| Name | Type | Description |
|---|---|---|
wordEmbeddings | Tensor | - |
s | Int32 | - |
sentenceLength | Int32 | - |
SetDropoutNodes(
Double)Configures dropout for the embedding output and draws a new dropout mask.
Parameters
| Name | Type | Description |
|---|---|---|
dropoutRate | Double | Dropout rate in |
ZeroGradients
Clears the gradient accumulator of the embedding matrix.
MakeTrainingStep(
Double, Int32)Applies one optimizer step to the embedding matrix.
Parameters
| Name | Type | Description |
|---|---|---|
learningRate | Double | The learning rate for this step. |
[step] | Int32 | The current step index, used by the Adam bias correction. |
DictionarySize
Gets the number of distinct words in the dictionary.
EmbeddingSize
Gets the width of the embedding vectors.
SequenceLength
Gets the fixed sequence length used by this embedding layer.
Parameters
与 Embedding.embeddingLayer 同形的梯度累加器。
embeddingLayer
Learned linear embedding layer