CodeWiki: Evaluating AI's Ability to Generate Holistic Documentation for Large-Scale Codebases

📖 About

Developers spend nearly 58% of their time understanding codebases, yet maintaining comprehensive documentation remains challenging. While recent Large Language Models (LLMs) show promise for function-level documentation, they fail at the repository level, where capturing architectural patterns and cross-module interactions is essential.

CodeWiki is the first open-source framework for holistic repository-level documentation across seven programming languages, introducing innovations in hierarchical decomposition, recursive agentic processing, and multi-modal synthesis.

🏗️ Framework Architecture

Figure 1: CodeWiki Framework operates in three main phases: (1) Repository analysis and hierarchical decomposition, (2) Recursive documentation generation with dynamic delegation, (3) Hierarchical assembly and synthesis

✨ Key Innovations

🌐 Multilingual Support

📊 Experimental Results

Evaluated on CodeWikiBench, the first benchmark specifically designed for repository-level documentation quality assessment across 21 repositories.

Performance by Language Category

Results on Representative Repositories

View comprehensive results for all 21 repositories in our paper.

🎥 Demo Video

Watch CodeWiki in action as it generates comprehensive documentation for a real repository:

CLI Usage Example: Generating documentation with CodeWiki

🚀 Installation & Quick Start

Prerequisites

• Python 3.12+
• Node.js (for Mermaid diagram validation)
• LLM API access (Anthropic Claude, OpenAI, etc.)

Installation

# Install from source
pip install git+https://github.com/FSoft-AI4Code/CodeWiki.git

# Verify installation
codewiki --version

Quick Start

1. Configure CodeWiki:

codewiki config set \
  --api-key YOUR_API_KEY \
  --base-url https://api.anthropic.com \
  --main-model claude-sonnet-4 \
  --cluster-model claude-sonnet-4

# Verify configuration
codewiki config show
codewiki config validate

2. Generate Documentation:

# Navigate to your project
cd /path/to/your/project

# Generate documentation (saved to ./docs/)
codewiki generate

# Generate with GitHub Pages HTML viewer
codewiki generate --github-pages

# Full-featured generation
codewiki generate --create-branch --github-pages --verbose

Output Structure

./docs/
├── overview.md              # Repository overview (start here!)
├── module1.md               # Module documentation
├── module2.md               # Additional modules...
├── module_tree.json         # Hierarchical module structure
├── first_module_tree.json   # Initial clustering result
├── metadata.json            # Generation metadata
└── index.html               # Interactive viewer (with --github-pages)

📚 Citation

If you use CodeWiki in your research, please cite our paper:

@misc{hoang2025codewikievaluatingaisability,
      title={CodeWiki: Evaluating AI's Ability to Generate Holistic 
             Documentation for Large-Scale Codebases}, 
      author={Anh Nguyen Hoang and Minh Le-Anh and Bach Le 
              and Nghi D. Q. Bui},
      year={2025},
      eprint={2510.24428},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2510.24428}, 
}

🔗 Resources