Ask HN: Anyone interested in building a harness-only benchmark?
There are a lot of LLM benchmarks but few, if any, harness benchmarks. I am thinking this would be a really good community project to build one. End goal: a leaderboard of harness performance (multiple axis) on a set of diverse real world tasks[1], grouped by underlying models and reasoning efforts. Anyone can contribute results. The task criteria, measurements, underlying framework et al can be decided by a group rather than a single person. If there is sufficient interest, I will create a discord. Disclosure: I am the maintainer of a coding agent called Dirac (https://ift.tt/QDna3cL) so I will not influence what the final benchmark should look like to avoid any conflict of interest. I just want to make this happen. [1] Diverse real world tasks meaning sufficiently complex tasks that the contributors have encountered, preferably from an opensource repo. 5 comments on Hacker News.
There are a lot of LLM benchmarks but few, if any, harness benchmarks. I am thinking this would be a really good community project to build one. End goal: a leaderboard of harness performance (multiple axis) on a set of diverse real world tasks[1], grouped by underlying models and reasoning efforts. Anyone can contribute results. The task criteria, measurements, underlying framework et al can be decided by a group rather than a single person. If there is sufficient interest, I will create a discord. Disclosure: I am the maintainer of a coding agent called Dirac (https://ift.tt/QDna3cL) so I will not influence what the final benchmark should look like to avoid any conflict of interest. I just want to make this happen. [1] Diverse real world tasks meaning sufficiently complex tasks that the contributors have encountered, preferably from an opensource repo.
There are a lot of LLM benchmarks but few, if any, harness benchmarks. I am thinking this would be a really good community project to build one. End goal: a leaderboard of harness performance (multiple axis) on a set of diverse real world tasks[1], grouped by underlying models and reasoning efforts. Anyone can contribute results. The task criteria, measurements, underlying framework et al can be decided by a group rather than a single person. If there is sufficient interest, I will create a discord. Disclosure: I am the maintainer of a coding agent called Dirac (https://ift.tt/QDna3cL) so I will not influence what the final benchmark should look like to avoid any conflict of interest. I just want to make this happen. [1] Diverse real world tasks meaning sufficiently complex tasks that the contributors have encountered, preferably from an opensource repo. 5 comments on Hacker News.
There are a lot of LLM benchmarks but few, if any, harness benchmarks. I am thinking this would be a really good community project to build one. End goal: a leaderboard of harness performance (multiple axis) on a set of diverse real world tasks[1], grouped by underlying models and reasoning efforts. Anyone can contribute results. The task criteria, measurements, underlying framework et al can be decided by a group rather than a single person. If there is sufficient interest, I will create a discord. Disclosure: I am the maintainer of a coding agent called Dirac (https://ift.tt/QDna3cL) so I will not influence what the final benchmark should look like to avoid any conflict of interest. I just want to make this happen. [1] Diverse real world tasks meaning sufficiently complex tasks that the contributors have encountered, preferably from an opensource repo.
Hacker News story: Ask HN: Anyone interested in building a harness-only benchmark?
![Hacker News story: Ask HN: Anyone interested in building a harness-only benchmark?]()
Reviewed by Tha Kur
on
August 05, 2026
Rating:
No comments: