AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodology

FPT Software AI Center, Viet Nam
MY ALT TEXT

Abstract

Software agents have emerged as promising tools for addressing complex software engineering tasks. However, existing works oversimplify software development workflows by following the waterfall model. Thus, we propose AgileCoder, a multi-agent system that integrates Agile Methodology (AM) into the framework. This system assigns specific AM roles such as Product Manager, Developer, and Tester to different agents, who then collaboratively develop software based on user inputs. AgileCoder enhances development efficiency by organizing work into sprints, focusing on incrementally developing software through sprints. Additionally, we introduce Dynamic Code Graph Generator, a module that creates a Code Dependency Graph dynamically as updates are made to the codebase. This allows agents to better comprehend the codebase, leading to more precise code generation and modifications throughout the software development process. AgileCoder surpasses existing benchmarks, like ChatDev and MetaGPT, establishing a new standard and showcasing the capabilities of multi-agent systems in advanced software engineering environments.

Overview

AgileCoder integrates Agile Methodology into a multi-agent system framework, enabling a collaborative environment where software agents assume specific Agile roles such as Product Manager, Scrum Master, Developer, Senior Developer, and Tester. These agents work together to develop software efficiently and iteratively, simulating a dynamic and adaptive software development team.

In AgileCoder, each agent is not just a participant but a stakeholder in the software development process, engaging in sprints that mimic real-world Agile practices. This setup enhances the adaptability and effectiveness of development workflows, aligning closely with modern software engineering demands. The system is designed to handle the complexities of real-world software projects, supporting incremental development and continuous integration seamlessly.

Key Features of AgileCoder:

  • Dynamic Role Assignment: Agents dynamically assume roles based on the project needs and user inputs, ensuring flexibility and optimal resource utilization.
  • Sprint-Based Development: The framework organizes development tasks into sprints, promoting rapid prototyping and frequent reassessment of project goals.
  • Dynamic Code Graph Generator: This innovative module automatically generates and updates a dependency graph whenever the codebase changes, enhancing the agents’ understanding of the code structure and interdependencies. This feature is crucial for maintaining high accuracy in code generation and modifications.

Evaluation

We will evaluate the performance of AgileCoder on two types of datasets to assess its effectiveness in generating code for different scenarios:

  • Competitive Programming: HumanEval and MBPP
  • Complex Software Requirements: We curate a new dataset called ProjectDev, which contains complex software requirements for generating complete software projects. This dataset will be designed to evaluate AgileCoder's ability to handle more intricate and real-world software development scenarios.

Results are as below:

Category Model Dataset Performance
HumanEval MBPP
LLMs (prompting) CodeGeeX-13B 18.9 26.9
PaLM Coder-540B 43.9 32.3
DeepSeeker-33B-Inst 79.3 70.0
GPT-3.5 Turbo 60.3 52.2
Claude 3 Haiku 75.9 80.4
GPT 4 80.1 80.1
LLMs-based Agents with GPT-3.5 Turbo ChatDev 61.79 74.80
MetaGPT 62.80 74.73
AgileCoder 70.53 80.92
with Claude 3 Haiku ChatDev 76.83 70.96
AgileCoder 79.27 84.31
with GPT 4 MetaGPT 85.9 87.7
AgileCoder 90.85 -
Metric ChatDev MetaGPT AgileCoder
Executability 32.79 7.73 57.79
Entire Running Time (s) 120 48 444
Avg. Time/Sprint (s) - - 306
#Sprints - - 1.64
Token Usage 7440 3029 36818
Expenses (USD) 0.12 0.02 0.44
#Errors 6 32 0

Apps Gallery

BibTeX

@article{nguyen2024agilecoder,
    title={AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodology},
    author={Minh Huynh Nguyen and Thang Phan Chau and Phong X. Nguyen and Nghi D. Q. Bui},
    journal={arXiv preprint arXiv:2406.11912},
    year={2024}
}