StableLM Language Models

Description

The StableLM project, hosted on GitHub by Stability AI, represents a significant effort in the development of open-source large language models (LLMs). This repository serves as a central hub for the ongoing research and development of the StableLM series, continuously updated with new model checkpoints and advancements.

At its core, StableLM aims to provide powerful and accessible language models for the AI community. The project has released several notable models, including StableLM-3B-4E1T, a 3 billion parameter model pre-trained under a multi-epoch regime to investigate the impact of repeated tokens on performance. This model was trained on 1 trillion tokens for 4 epochs, drawing inspiration from research on scaling data-constrained language models. Its architecture is a decoder-only transformer, similar to LLaMA, with specific modifications like Rotary Position Embeddings and LayerNorm.

Further iterations include the StableLM-Alpha v2 models, available in 3B and 7B parameter sizes. These models incorporate architectural improvements such as SwiGLU and utilize higher-quality data sources, significantly enhancing downstream performance. The training data for these models includes a filtered mixture of open-source datasets like Falcon RefinedWeb, RedPajama-Data, The Pile, and StarCoder, with a strong emphasis on web text. The context length for these models is 4096 tokens.

Stability AI has also developed StableVicuna, an RLHF fine-tune of Vicuna-13B, aimed at creating an open-source RLHF LLM Chatbot. Due to LLaMA's non-commercial license, StableVicuna's weights are released as deltas over the original model.

The repository provides resources for developers to get started, including quickstart guides and example code snippets for running inference with models like StableLM-Tuned-Alpha-7B on Hugging Face. The project encourages community involvement, seeking contributions for porting llama.cpp and integrating with Open Assistant for feedback data collection. Users are advised that, as with any pretrained LLM, responses may vary in quality and could potentially include offensive content, which is expected to improve with further development and community feedback.

StableLM Language Models's Core Features

• Open-source language model development repository

• Hosts StableLM series of language models

• Continuously updated with new checkpoints

• Includes models like StableLM-3B-4E1T

• Features StableLM-Alpha v2 models (3B and 7B)

• Provides StableVicuna, an RLHF fine-tune

• Offers technical reports and model overviews

• Includes example code for inference

• Encourages community contributions and feedback

• Models available under CC BY-SA-4.0 and CC BY-NC-SA-4.0 licenses

• Code licensed under Apache License 2.0

Getting Started with StableLM Language Models

1. 1

   Clone: Clone the repository to your local machine.

2. 2

   Install dependencies: Install necessary libraries and frameworks.

3. 3

   Download model: Obtain desired StableLM model checkpoints from Hugging Face.

4. 4

   Configure: Set up environment and parameters for inference or fine-tuning.

5. 5

   Execute: Run provided scripts for inference or custom model usage.

6. 6

   Integrate: Incorporate models into your applications or research projects.

StableLM Language Models's Use Cases

• Language Model Research
• AI Chatbot Development
• Text Generation
• Natural Language Understanding
• Fine-tuning for Specific Tasks
• Open-Source AI Development

FAQ from StableLM Language Models

• What is StableLM?

  StableLM is a series of open-source language models developed by Stability AI, focused on advancing AI language capabilities.

• Where can I find the StableLM models?

  All StableLM models are hosted on the Hugging Face hub, with links provided in the repository.

• What are the licensing terms for StableLM models?

  Base model checkpoints are licensed under CC BY-SA-4.0, while fine-tuned checkpoints are under CC BY-NC-SA-4.0.

• What kind of models are available in the StableLM series?

  The series includes models like StableLM-3B-4E1T, StableLM-Alpha v2 (3B and 7B), and StableVicuna (an RLHF fine-tune).

• What is the context length for StableLM-Alpha v2 models?

  The context length for StableLM-Alpha v2 models is 4096 tokens.

• How can I contribute to the StableLM project?

  You can contribute by porting llama.cpp to work with StableLMs or by providing high-quality human-generated feedback data for Open Assistant.

• What are the potential issues with StableLM models?

  As with any pretrained LLM, responses may vary in quality and could potentially include offensive language or views.

• What is the architecture of StableLM-3B-4E1T?

  It is a decoder-only transformer similar to the LLaMA architecture with modifications like Rotary Position Embeddings and LayerNorm.

• What datasets were used for training StableLM-Alpha v2?

  Training data includes Falcon RefinedWeb, RedPajama-Data, The Pile, and StarCoder, with a high sampling rate of web text.

• Can I use StableLM models for commercial purposes?

  Base models are under CC BY-SA-4.0, but fine-tuned models like StableLM-Tuned-Alpha are under CC BY-NC-SA-4.0, restricting commercial use.