UNITER Research Code

Description

The UNITER research code repository, presented in the ECCV 2020 paper 'UNITER: UNiversal Image-TExt Representation Learning', offers a comprehensive set of tools for multimodal AI research. This repository facilitates finetuning and inference for a range of vision-and-language tasks, including Visual Question Answering (VQA), Visual Commonsense Reasoning (VCR), SNLI-VE (Visual Entailment), Image-Text Retrieval for datasets like COCO and Flickr30k, and Referring Expression Comprehensions (RefCOCO, RefCOCO+, RefCOCO-g).

UNITER is built upon a universal image-text representation learning framework. The code supports both UNITER-base and UNITER-large pre-trained checkpoints, with options for in-domain pre-training. The repository includes scripts for data preprocessing, model training, and inference. It leverages external libraries such as PyTorch, HuggingFace Transformers, OpenNMT, and Nvidia's DeepLearningExamples, with image features extracted using BUTD.

For ease of reproduction, a Docker image is provided, requiring specific NVIDIA driver and Docker versions. The setup involves mounting source code and data directories into the container. The repository details quick start guides for tasks like NLVR2, demonstrating data download, Docker container launch, finetuning, and inference/evaluation procedures. Customization is supported through command-line arguments and JSON configuration files, with options for multi-GPU training using Horovod.

The project also outlines steps for downstream tasks such as VQA, VCR (including a second-stage pre-training option), Visual Entailment, Image-Text Retrieval (both zero-shot and finetuning with hard negatives for Flickr30k and COCO), Referring Expressions, and in-domain pre-training. Users are guided on downloading specific datasets and running the respective training and inference scripts. The repository is licensed under the MIT license.

UNITER Research Code Highlights

• Official research code for ECCV 2020 paper 'UNITER: UNiversal Image-TExt Representation Learning'

• Supports finetuning for NLVR2, VQA, VCR, SNLI-VE, Image-Text Retrieval, and Referring Expressions

• Provides pre-trained checkpoints for UNITER-base and UNITER-large models

• Includes scripts for data preprocessing, model training, and inference

• Offers a Docker image for simplified reproduction and environment setup

• Supports multi-GPU training via Horovod

• Includes code for zero-shot and finetuning of Image-Text Retrieval tasks

• Facilitates in-domain pre-training and VCR second-stage pre-training

Getting Started with UNITER Research Code

1. 1

   Download pre-trained checkpoints and task-specific data using provided bash scripts.

2. 2

   Launch the Docker container for a consistent and reproducible environment.

3. 3

   Integrate model training by running finetuning scripts with custom configurations.

4. 4

   Perform inference on downstream tasks using dedicated inference scripts.

5. 5

   Evaluate model performance using provided evaluation scripts or by submitting results to leaderboards.

6. 6

   Customize training options via command-line arguments or JSON configuration files.

UNITER Research Code's Use Cases

• Visual Question Answering
• Visual Commonsense Reasoning
• Image-Text Retrieval
• Referring Expression Comprehension
• Visual Entailment
• Multimodal Pre-training

FAQ from UNITER Research Code

• What is the UNITER research code repository?

  The UNITER research code repository contains the official implementation for the ECCV 2020 paper 'UNITER: UNiversal Image-TExt Representation Learning', enabling researchers to reproduce and extend multimodal AI research.

• What vision-and-language tasks does UNITER support?

  UNITER supports tasks such as NLVR2, VQA, VCR, SNLI-VE, Image-Text Retrieval (COCO, Flickr30k), and Referring Expressions (RefCOCO, RefCOCO+, RefCOCO-g).

• Are pre-trained UNITER models available?

  Yes, both UNITER-base and UNITER-large pre-trained checkpoints are released and can be downloaded using the provided scripts.

• How can I set up the environment for UNITER?

  A Docker image is provided for easier reproduction. Users need to install NVIDIA drivers and Docker, and then launch the container with mounted data and source directories.

• Can UNITER be trained on multiple GPUs?

  Yes, the repository supports multi-GPU training, primarily through the use of Horovod.

• What are the requirements for running the Docker image?

  The Docker image requires NVIDIA driver (418+), Docker (19.03+), and nvidia-container-toolkit. Linux with NVIDIA GPUs is supported, and GPUs with Tensor Cores are recommended for mixed-precision training.

• How is image feature extraction handled?

  Image features are extracted using the BUTD (bottom-up-attention) method, with scripts provided for preprocessing.

• What license is the UNITER code released under?

  The UNITER research code is released under the MIT license.

• Can I finetune UNITER on my own data?

  The repository provides scripts and configurations for finetuning on various downstream tasks, suggesting that custom finetuning is supported.

• What are the key components of the UNITER model?

  UNITER is based on a universal image-text representation learning framework, leveraging transformer architectures for multimodal understanding.