Code Llama

Description

Code Llama is a family of large language models developed by Meta, built upon Llama 2 and specialized for code-related tasks. This repository offers the inference code necessary to run these powerful models. Code Llama models deliver state-of-the-art performance among open models, featuring capabilities such as code infilling, support for extended input contexts up to 100,000 tokens, and zero-shot instruction following for programming.

The family includes foundation models (Code Llama), Python-specific versions (Code Llama - Python), and instruction-following models (Code Llama - Instruct). These are available in various parameter sizes: 7B, 13B, 34B, and 70B. All models are trained on sequences of 16k tokens and demonstrate improved performance with inputs up to 100k tokens. The 7B and 13B variants of Code Llama and Code Llama - Instruct are capable of infilling code based on surrounding content.

Developed by fine-tuning Llama 2 with a higher sampling of code, Code Llama incorporates considerable safety mitigations. The repository provides model weights and starting code for these pretrained and fine-tuned models. It serves as a minimal example for loading Code Llama models and executing inference. Users can download model weights and tokenizers from the Meta website after accepting the license, then use the provided `download.sh` script.

Setting up involves cloning the repository and installing dependencies within a conda environment with PyTorch/CUDA. Inference requires specifying model-parallel (MP) values, which vary by model size (e.g., 1 for 7B, 8 for 70B). The repository includes example scripts for code completion (`example_completion.py`), code infilling (`example_infilling.py`), and instruction following (`example_instructions.py`), demonstrating how to prompt and run the different model types effectively.

Code Llama - Instruct models are specifically fine-tuned to follow instructions, requiring particular formatting for optimal results. The repository also highlights the potential risks associated with AI technologies and directs users to a Responsible Use Guide and research papers for more information. Issues related to software bugs or model behavior can be reported through the provided GitHub links.

Code Llama's Core Features

• Inference code for Code Llama models

• Supports foundation, Python-specialized, and instruction-following models

• Available in 7B, 13B, 34B, and 70B parameter sizes

• State-of-the-art performance among open code models

• Code infilling capabilities for 7B and 13B models

• Supports large input contexts up to 100,000 tokens

• Zero-shot instruction following for programming tasks

• Trained on sequences of 16k tokens

• Includes model weights and starting code

• Provides example scripts for completion, infilling, and instruction following

• Incorporates safety mitigations

• Licensed for researchers and commercial entities

Getting Started with Code Llama

1. 1

   Clone: Clone the repository from GitHub.

2. 2

   Install: Set up a conda environment with PyTorch/CUDA and install dependencies using `pip install -e .`.

3. 3

   Download: Obtain model weights and tokenizers from the Meta website and use `download.sh`.

4. 4

   Configure: Determine the appropriate model-parallel (MP) value based on the model size.

5. 5

   Execute: Run inference using provided example scripts (e.g., `example_completion.py`, `example_infilling.py`, `example_instructions.py`).

6. 6

   Optimize: Adjust `max_seq_len` and `max_batch_size` based on hardware and use-case.

Code Llama's Use Cases

• Code Generation
• Code Completion
• Code Infilling
• Instruction Following
• Python Development
• Research
• Large Context Tasks

FAQ from Code Llama

• What is Code Llama?

  Code Llama is a family of large language models for code, based on Llama 2, offering advanced capabilities for programming tasks.

• What types of Code Llama models are available?

  The available models include foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct).

• What parameter sizes does Code Llama come in?

  Code Llama models are available in 7B, 13B, 34B, and 70B parameter sizes.

• Can Code Llama fill in code?

  Yes, the 7B and 13B variants of Code Llama and Code Llama - Instruct support code infilling based on surrounding content.

• What is the maximum context length for Code Llama models?

  All models, except for the 70B python and instruct versions, support sequence lengths up to 100,000 tokens.

• How do I download the Code Llama model weights?

  You can download model weights and tokenizers from the Meta website after accepting the license, then use the `download.sh` script.

• What are the prerequisites for setting up Code Llama?

  A conda environment with PyTorch/CUDA, and tools like `wget` and `md5sum` are recommended for setup and downloading.

• How do I run inference with Code Llama?

  Inference can be run using provided example scripts like `example_completion.py`, `example_infilling.py`, and `example_instructions.py`, after setting up the environment and downloading models.

• Are Code Llama models suitable for commercial use?

  Yes, the model and weights are licensed for both researchers and commercial entities.

• Where can I report issues with the Code Llama models?

  Issues with the model can be reported on GitHub, and risky content feedback can be provided via developers.facebook.com/llama_output_feedback.