What does HackerNews think of human-eval?

Code for the paper "Evaluating Large Language Models Trained on Code"

Language: Python

Code Llama, a state-of-the-art large language model for coding | Aug 2023
What I found interesting in Meta's paper is the mention of HumanEval[1] and MBPP[2] as benchmarks for code quality. (Admittedly maybe they're well-known to those working in the field.)

I haven't yet read the whole paper (nor have I looked at the benchmark docs which might very well cover this) but curious how these are designed to avoid issues with overfitting. My thinking here is that canned algorithm type problems common in software engineering interviews are probably over represented in the training data used for these models. Which might point to artificially better performance by LLMs versus their performance on more domain-specific type tasks they might be used for in day-to-day work.

[1] https://github.com/openai/human-eval

[2] https://github.com/google-research/google-research/tree/mast...

A new way to predict when software jobs will become automated | Jul 2022
Expand Context ↕
On a dataset of 164 programming problems: https://arxiv.org/pdf/2107.03374.pdf (Section 2.2). The dataset is here: https://github.com/openai/human-eval
Evaluating Large Language Models Trained on Code (paper about GH copilot model) | Jul 2021
abstract: We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities.

A distinct production version of Codex powers GitHub Copilot.

On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%.

Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem.

Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.

HumanEval: https://github.com/openai/human-eval