CodeMMLU Leaderboard

A Multi-Task Benchmark for Assessing Code Understanding Capabilities of CodeLLMs

blog leaderboard data
github paper

📝 Notes

  1. Evaluated using CodeMMLU
  2. Models are ranked according to Accuracy using greedy decoding.
  3. "Size" here is the amount of activated model weight during inference.

🤗 More Leaderboards

In addition to CodeMMLU leaderboards, it is recommended to comprehensively understand LLM coding ability through a diverse set of benchmarks and leaderboards, such as:
  1. RepoExec Leaderboard
  2. Bigcode-Bench Leaderboard
  3. EvalPlus Leaderboard
  4. Big Code Models Leaderboard
  5. Chatbot Arena Leaderboard
  6. CrossCodeEval
  7. ClassEval
  8. CRUXEval
  9. Code Lingua
  10. Evo-Eval
  11. HumanEval.jl - Julia version HumanEval with EvalPlus test cases
  12. InfiCoder-Eval
  13. LiveCodeBench
  14. NaturalCodeBench
  15. RepoBench
  16. SWE-bench

🙏 Acknowledgements

  • We thank the EvalPlus and BigCode teams for providing the leaderboard template.