OpenLLaMA

Description

OpenLLaMA represents a significant advancement in open-source large language models, offering a permissively licensed reproduction of Meta AI's LLaMA. Developed by openlm-research, this project aims to democratize access to powerful language models by providing accessible weights and training methodologies.

The project releases a series of models, including 3B, 7B, and 13B parameter versions, all trained on one trillion tokens. These models are designed to be drop-in replacements for the original LLaMA models in existing implementations. The project provides weights in both PyTorch format, compatible with the Hugging Face transformers library, and JAX format for use with the EasyLM framework.

OpenLLaMA's training methodology closely follows the original LLaMA paper, utilizing the same model architecture, context length, training steps, learning rate schedule, and optimizer. The key distinction lies in the datasets used: v1 models are trained on the RedPajama dataset, while v2 models incorporate a mixture of the Falcon refined-web dataset, the StarCoder dataset, and parts of the RedPajama dataset (Wikipedia, ArXiv, Books, StackExchange). This commitment to open datasets ensures transparency and reproducibility.

The project emphasizes evaluation and comparison, presenting results against LLaMA and GPT-J across a wide range of tasks using the lm-evaluation-harness. OpenLLaMA models demonstrate comparable, and in some cases superior, performance to their counterparts. The development team has also addressed potential issues, such as tokenizer configurations affecting code generation tasks, recommending v2 models for such applications.

OpenLLaMA is licensed under the Apache 2.0 license, promoting broad adoption and modification. The project is a collaborative effort, with contributions from Berkeley AI Research and support from Google's TPU Research Cloud program. This initiative empowers researchers and developers to build upon and innovate with state-of-the-art language models.

OpenLLaMA Highlights

• Permissively licensed open-source reproduction of Meta AI's LLaMA

• Available in 3B, 7B, and 13B parameter sizes

• Trained on 1T tokens

• Weights available in PyTorch and JAX formats

• Compatible with Hugging Face transformers and EasyLM framework

• Follows original LLaMA architecture and training hyperparameters

• Utilizes open datasets like RedPajama, Falcon refined-web, and StarCoder

• Comparable performance to original LLaMA and GPT-J

• Apache 2.0 license for broad usage

• Includes evaluation results and comparisons

• Tokenizer designed to merge multiple empty spaces, similar to T5

Getting Started with OpenLLaMA

1. 1

   Access Model Weights: Download PyTorch or JAX weights from Hugging Face Hub or EasyLM repositories.

2. 2

   Set Up Environment: Install necessary libraries such as transformers, PyTorch, or JAX.

3. 3

   Load Tokenizer: Use LlamaTokenizer or AutoTokenizer with `use_fast=False` to avoid potential tokenization issues.

4. 4

   Load Model: Instantiate LlamaForCausalLM from Hugging Face transformers or the equivalent in EasyLM.

5. 5

   Integrate via API: Utilize the loaded model for text generation, inference, or fine-tuning tasks.

6. 6

   Evaluate Performance: Employ lm-evaluation-harness for benchmarking, ensuring correct tokenizer usage.

OpenLLaMA's Use Cases

• Text Generation
• Language Understanding
• Model Reproduction
• Research and Development
• Chatbot Development
• Code Generation

FAQ from OpenLLaMA

• What is OpenLLaMA?

  OpenLLaMA is an open-source, permissively licensed reproduction of Meta AI's LLaMA large language model, available in 3B, 7B, and 13B parameter sizes.

• What datasets were used to train OpenLLaMA?

  v1 models were trained on the RedPajama dataset. v2 models were trained on a mixture of the Falcon refined-web dataset, the StarCoder dataset, and parts of the RedPajama dataset.

• What are the available model sizes for OpenLLaMA?

  OpenLLaMA is available in 3B, 7B, and 13B parameter sizes.

• In what formats are the OpenLLaMA model weights released?

  Model weights are released in PyTorch format for Hugging Face transformers and JAX format for the EasyLM framework.

• What license does OpenLLaMA use?

  OpenLLaMA is licensed under the Apache 2.0 license.

• Can OpenLLaMA be used as a drop-in replacement for LLaMA?

  Yes, the model weights are designed to serve as a drop-in replacement for LLaMA in existing implementations.

• Are there any known issues with the OpenLLaMA tokenizer?

  The v1 tokenizer had issues with new lines and code generation due to merging empty spaces. For code-related tasks, it is advised to use the v2 models.

• How does OpenLLaMA's performance compare to other models?

  OpenLLaMA exhibits comparable performance to the original LLaMA and GPT-J across a majority of tasks, and outperforms them in some tasks.

• Who developed OpenLLaMA?

  OpenLLaMA was developed by Xinyang Geng and Hao Liu from Berkeley AI Research.

• Where can I find the OpenLLaMA model weights?

  Model weights can be found on Hugging Face Hub for PyTorch versions and are available for EasyLM.