nuget
Text preprocessing helpers used by the translation demo: loading parallel sentence pairs, tokenizing sentences, computing sequence lengths and adding the start / stop markers.
TextProcessing
01 Syntax
Microsoft.VisualBasic.MachineLearning.Transformer.TextProcessing
02 Methods
| Name | Overloads | Summary |
|---|---|---|
| Load | 1 | Loads up to nrSentences parallel sentence pairs from a tab separated training file. |
| ProcessSentence | 2 | Joins the tokens of one sentence of a batch back into a string. |
| CalculateSequenceLength | 1 | Computes the length of the longest sentence across both languages. |
| CalculateMaxSentenceLength | 1 | Computes the length of the longest sentence in the given collection. |
| InsertStartAndStopCharacters | 1 | Adds the start (<) and stop (>) markers to every target sentence, when not already present. |
| InitializeSpanishSentences | 1 | Creates the initial decoder input: one sentence per batch element, containing only the start marker. |
| AddWordsToSentences | 1 | Appends the predicted word of every batch element to its sentence, skipping already finished sentences. |
03 Members
Load(
String, Int32, List(Of List(Of String)), List(Of List(Of String)))Loads up to nrSentences parallel sentence pairs from a tab separated training file.
Parameters
| Name | Type | Description |
|---|---|---|
filename | String | Path of the training data file. |
nrSentences | Int32 | Maximum number of sentence pairs to read. |
englishSentences | List(Of List(Of String)) | Receives the tokenized source sentences. |
spanishSentences | List(Of List(Of String)) | Receives the tokenized target sentences. |
ProcessSentence(
String)Tokenizes a sentence and separates trailing question marks and exclamation marks.
Parameters
| Name | Type | Description |
|---|---|---|
sentenceString | String | The sentence to tokenize. |
Returns
The tokenized sentence wrapped in a single element batch.
ProcessSentence(
List(Of List(Of String)), Int32)Joins the tokens of one sentence of a batch back into a string.
Parameters
| Name | Type | Description |
|---|---|---|
sentence | List(Of List(Of String)) | The batch of tokenized sentences. |
s | Int32 | Index of the sentence to render. |
Returns
The joined sentence.
CalculateSequenceLength(
List(Of List(Of String)), List(Of List(Of String)))Computes the length of the longest sentence across both languages.
Parameters
| Name | Type | Description |
|---|---|---|
englishSentences | List(Of List(Of String)) | The source sentences. |
spanishSentences | List(Of List(Of String)) | The target sentences. |
Returns
The maximum sentence length.
CalculateMaxSentenceLength(
List(Of List(Of String)))Computes the length of the longest sentence in the given collection.
Parameters
| Name | Type | Description |
|---|---|---|
sentences | List(Of List(Of String)) | The sentences to inspect. |
Returns
The maximum sentence length.
InsertStartAndStopCharacters(
List(Of List(Of String)))Adds the start (<) and stop (>) markers to every target sentence, when not already present.
Parameters
| Name | Type | Description |
|---|---|---|
correctSpanishSentences | List(Of List(Of String)) | The target sentences to modify in place. |
InitializeSpanishSentences(
Int32)Creates the initial decoder input: one sentence per batch element, containing only the start marker.
Parameters
| Name | Type | Description |
|---|---|---|
batchSize | Int32 | Number of sentences in the batch. |
Returns
The initialized decoder input.
AddWordsToSentences(
Int32, Boolean, String(), List(Of List(Of String)))Appends the predicted word of every batch element to its sentence, skipping already finished sentences.
Parameters
| Name | Type | Description |
|---|---|---|
batchSize | Int32 | Number of sentences in the batch. |
isTraining | Boolean | Reserved for compatibility; the current implementation behaves the same in both modes. |
spanishWords | String() | The predicted word of each batch element. |
translatedSpanishSentences | List(Of List(Of String)) | The sentences to append to, modified in place. |