Benchmarks & Evaluation

Standardised tests and leaderboards used to measure and compare model performance on software engineering tasks

• Adopt
  • METR Time Horizons
  • SWE-bench Verified
• Trial
  • Arena
  • LiveCodeBench
  • SWE-CI
  • BigCodeBench
• Hold
  • HumanEval