Description
OpenManus-RL is a collaborative open-source initiative by Ulab-UIUC and MetaGPT, extending the original OpenManus project. Inspired by successful RL tuning for reasoning LLMs like Deepseek-R1 and QwQ-32B, this project explores novel paradigms for RL-based LLM agent tuning. Progress, including rigorous testing on benchmarks such as GAIA, AgentBench, WebShop, and OSWorld, along with tuned models, is shared dynamically through live streaming.
The project integrates advanced reinforcement learning (RL) techniques to enhance the reasoning and decision-making capabilities of LLMs. It draws inspiration from existing frameworks and explores new algorithmic structures, diverse reasoning paradigms, sophisticated reward strategies, and extensive benchmark environments. Key components include exploring various reasoning models (GPT-O1, Deepseek-R1, QwQ-32B), experimenting with alternative rollout strategies like Tree-of-Thoughts (ToT) and Monte Carlo Tree Search (MCTS), and analyzing different reasoning output formats such as ReAct and Outcome-based Reasoning.
Post-training strategies investigated include Supervised Fine-Tuning (SFT), Generalized Reward-based Policy Optimization (GRPO), Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), and Preference-based Reward Modeling (PRM). The project also focuses on training specialized agent reward models and implementing test-time trajectory scaling for adaptability. Integration with leading RL tuning frameworks, notably the 'verl' submodule, provides advanced training capabilities for agent optimization.
The OpenManus-RL dataset combines agent trajectories from AgentInstruct, Agent-FLAN, and AgentTraj-L, featuring a ReAct framework, structured training, anti-hallucination techniques, and coverage across six domains. The project provides a simplified library for SFT and GRPO tuning, built upon the 'verl' framework. Community contributions are highly encouraged for fine-tuning codebases, datasets, environment setups, and computing resources, with important contributors potentially listed as co-authors to future papers.
OpenManus-RL aims to push the boundaries of agent reasoning and tool integration, fostering a thriving open-source agent ecosystem. The project is committed to continuous updates and welcomes collaboration from developers interested in advancing LLM agent capabilities.
OpenManus-RL's Core Features
Live-stream development of RL tuning for LLM agents
Exploration of novel RL-based LLM agent tuning paradigms
Rigorous testing on agent benchmarks (GAIA, AgentBench, WebShop, OSWorld)
Open sharing of progress, results, and tuned models
Integration of advanced RL algorithms and frameworks (e.g., Verl)
Support for diverse reasoning models and rollout strategies (ToT, MCTS)
Implementation of various post-training strategies (SFT, GRPO, PPO, DPO)
Comprehensive dataset combining agent trajectories from multiple sources
Community contribution welcome for code, data, and resources
Getting Started with OpenManus-RL
Clone Repository: Clone the OpenManus-RL repository, ensuring submodules are initialized and updated.
Environment Setup: Create and activate a conda environment, then install required dependencies including PyTorch and vllm.
Dataset Preparation: Download the OpenManus-RL dataset from Hugging Face.
Environment Configuration: Set up specific agent environments like WebShop or ALFWorld as per documentation.
Training Execution: Utilize provided scripts for training agent models, for example, ALFWorld RL Training (PPO).
Optimization: Fine-tune parameters and explore different strategies for optimal agent performance.
OpenManus-RL's Use Cases
- LLM Agent Tuning
- Reinforcement Learning Research
- Benchmark Evaluation
- Open-Source Collaboration
- Tool Integration
- Reasoning Capability Enhancement
