RAG prompt example ▼
| Full | ||
|---|---|---|
| Rank | Model | Pass@1 |
| 1 | GPT-4o + RAG (Sparse-Retrieval) | 27.35 |
| 2 | DeepSeek-V2.5 + RAG (Current-File) | 27.04 |
| 3 | Codestral-22B + RAG (Current-File) | 20.00 |
| 4 | Claude 3.5 Sonnet + RAG (Current-File) | 19.80 |
| 5 | GPT-4o-Mini + RAG (Current-File) | 18.67 |
| 6 | OpenCodeInterpreter-33B + RAG (Current-File) | 18.27 |
| 7 | DeepSeekCoder-33B + RAG (Dense-Retrieval) | 17.14 |
| 7 | Qwen2.5-Coder-7B + RAG (Dense-Retrieval) | 17.14 |
| 9 | DeepSeekCoder-6.7B + RAG (Sparse-Retrieval) | 14.08 |
| 10 | OpenCodeInterpreter-6.7B + RAG (Current-File) | 13.16 |
| 11 | CodeLlama-13B + RAG (Dense-Retrieval) | 12.76 |
| 12 | CodeLlama-7B + RAG (Sparse-Retrieval) | 10.71 |
RAG ({settings}): These results are generated under three retrieval settings: Sparse Retrieval, Dense Retrieval, and Current File. Please checkout our paper for more details.
Large language models (LLMs) have achieved high accuracy, i.e., more than 90 pass@1, in solving Python coding problems in HumanEval and MBPP. Thus, a natural question is, whether LLMs achieve comparable code completion performance compared to human developers? Unfortunately, one cannot answer this question using existing manual crafted or simple (e.g., single-line) code generation benchmarks, since such tasks fail to represent real-world software development tasks. In addition, existing benchmarks often use poor code correctness metrics, providing misleading conclusions.
To address these challenges, we create REPOCOD, a code generation benchmark with 980 problems collected from 11 popular real-world projects, with more than 58% of them requiring file-level or repository-level context information. In addition, REPOCOD has the longest average canonical solution length (331.6 tokens) and the highest average cyclomatic complexity (9.00) compared to existing benchmarks. Each task in REPOCOD includes 313.5 developerwritten test cases on average for better correctness evaluation. In our evaluations of ten LLMs, none of the models achieve more than 30 pass@1 on REPOCOD, indicating the necessity of building stronger LLMs that can help developers in real-world software development.
We employ a three-stage data collection pipeline to efficiently gather target functions from popular repositories: Repository Selection, Target Function Selection, and Relevant Test Case Collection. For more details, feel free to read our paper!
@misc{liang2024languagemodelsreplaceprogrammers,
title={Can Language Models Replace Programmers? REPOCOD Says 'Not Yet'},
author={Shanchao Liang and Yiran Hu and Nan Jiang and Lin Tan},
year={2024},
eprint={2410.21647},
archivePrefix={arXiv},
primaryClass={cs.SE},
url={https://arxiv.org/abs/2410.21647},
}