Natural Language Processing Datasets

xTorch includes a comprehensive suite of built-in dataset handlers for a wide variety of Natural Language Processing (NLP) tasks. These handlers manage the downloading and parsing of standard text corpora, allowing you to focus on model development.

All NLP datasets are located under the xt::datasets namespace and can be found in the <xtorch/datasets/natural_language_processing/> header directory.

General Usage

Working with NLP datasets requires an additional preprocessing step compared to other modalities: tokenization and numericalization. Raw text must be converted into a sequence of integer IDs that can be fed into a model. This is typically handled by a text Transform.

The general workflow is:

1. Build or load a vocabulary.
2. Create a text transformation pipeline for tokenizing and numericalizing the text.
3. Instantiate the desired dataset with this pipeline.
4. Pass the dataset to a DataLoader.

#include <xtorch/xtorch.h>
#include <iostream>
 
int main() {
    // 1. & 2. Define a text transformation pipeline
    // This is a simplified example. In a real scenario, you would build a vocabulary
    // from the training data and use it to convert tokens to integers.
    // xTorch's text transforms help with this process.
    // For now, let's assume we have a simple tokenizer.
    // auto text_transforms = std::make_unique<xt::transforms::text::...>();
 
    // 3. Instantiate a dataset for the IMDB movie review task
    auto dataset = xt::datasets::IMDB(
        "./data",
        xt::datasets::DataMode::TRAIN,
        /*download=*/true
        // std::move(text_transforms) // Pass transforms here
    );
 
    std::cout << "IMDB dataset size: " << *dataset.size() << std::endl;
 
    // 4. Pass the dataset to a DataLoader
    // Note: For NLP, you often need a custom collate function to handle padding
    // of sequences with different lengths.
    xt::dataloaders::ExtendedDataLoader data_loader(dataset, 32, true);
 
    // The data loader is now ready for use in a training loop
    for (auto& batch : data_loader) {
        auto numericalized_text = batch.first;
        auto labels = batch.second;
        // ... training step with an RNN or Transformer ...
    }
}

!!! info "Standard Dataset Constructors" Most dataset constructors follow a standard pattern: DatasetName(const std::string& root, DataMode mode, bool download, TextTransformPtr transforms)

• root: The directory where the data is stored or will be downloaded.
• mode: DataMode::TRAIN, DataMode::TEST, or DataMode::VALIDATION.
• download: If true, the dataset will be downloaded if not found in the root directory.
• transforms: A unique_ptr to a text transform pipeline.

Available Datasets by Task

Text Classification

This is the task of assigning a category or label to a piece of text.

Language Modeling

The task of predicting the next word in a sequence.

Machine Translation

The task of translating a sequence of text from a source language to a target language.

Question Answering

The task of answering a question based on a given context passage.

Text Summarization

The task of generating a short summary from a longer document.

Sequence Tagging

Dialogue Generation

Math Word Problems