Explicit Edit Benchmark: copilot-cli versus other harnesses

[alexshpunt]

Hi! I've been messing around lately with my own benchmark for checking how tooling and harness affect agent's ability to make text edits. The tasks are explicitly clearly prompted and their complexity is only defined by the size of the files to edit and similar context around the edit location, that's why it's pretty suitable and interesting to check with low reasoning models. It's designed to be community based, so if anybody wants to share data feel free to do so, I've tried to cover as much as my subscriptions, tooling and time allow, but it's impossible to make runs all over and over again for the whole matrix. What I found quite surpising, that copilot-cli beats codex-cli on almost every model I tested (except MiMo-2.5), so I guess there is a chance that tooling from copilot's side is better. If anybody is interested, it would be great to get your input and runs, as only together we can get objective statistics.

Viewer: https://huggingface.co/spaces/alexshpunt/benchmark-explorer
Repo: https://github.com/alexshpunt/explicit-edit-benchmark