Evaligo

Evaligo builds agent benchmarks from real engineering work and pairs each one with a reinforcement learning environment. Model teams can train on a dedicated split, then measure the result on held-out tasks, and compare public models on its SWE-Race leaderboard.

Each benchmark ships with real tasks, simulated tools, a verifier and separate public, private and training splits. SWE-Race, the first live family, asks agents to fix race conditions, deadlocks and cancellation bugs in pinned commits of real repositories, checked by each project's own hidden tests.

Environments run in the Harbor format, so any RL trainer can drive rollouts against them. Scores are reported as pass@1 on the private split, and results can stay private or be added to the public comparison. Guardrail and multi-step tool-use benchmarks are listed as in preparation.