Source-linked AI summary
Humanize: Judgement Engineering for Agentic Coding
Sihao Liu, Ligeng Zhu, Zijian Zhang, Dongyun Zou, Zhengyang Zhang, Changye Li, Song Bian, Song Han, Tony Nowatzki
TL;DR
Agentic coding makes code generation cheap, but reliable completion remains difficult when agents judge their own work. Humanize uses a human-approved plan, builder rounds, and an independent reviewer, reporting full-score outcomes across olympiad and formal-proof applications.
Problem
Agents that write code can be weak judges of correctness and completion, while existing loops leave completion decisions with the agents doing the work.
Source: §1 · p. 1Method
Humanize combines a human-approved plan contract, builder rounds, a cross-vendor reviewer, and deterministic hooks that route work and enforce gates.
Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5Results
Across olympiad and formal-proof applications, Humanize reports full scores on several tasks and ranks first on Lean-Eval’s legacy leaderboard.
Source: §5 · pp. 7–8Takeaways & Limitations
The applications use either a second model or an external oracle as the independent judge.
Source: §1 · p. 2The postmortem corpus is not representative: users chose what to file, four reporters wrote half the reports, and analysis covers selected exits.
Source: §4 · p. 6
Abstract
from arXiv · showhide
Agentic coding makes code generation cheap, but reliable completion remains difficult: the agent that writes the code is a weak judge of whether it is done. We present Humanize, a multi-agent orchestration workflow for agentic coding built around judgement engineering: explicit, mechanically enforced decisions at the boundaries between planning, implementation, review, and learning. A human approves a plan contract, a builder agent implements it in rounds, and a reviewer agent from another vendor decides completion; deterministic hooks, not a model, route work between these roles and enforce 72 mechanical gates. Viewed as a Markov chain over repository states, alternating builder and reviewer samples jointly from two models, so a defect survives only if both miss it. We study Humanize through its deployment, 118 public postmortems of real loops, and its applications. Over 68 versions in 108 days, it gathered 1,468 GitHub stars. Applications include a 567-file gem5 build-system migration under upstream review; Kernel Design Agents, which extend the loop with a kernel knowledge base and profiling feedback and placed in the top three of all three Full-Agent tracks of the MLSys 2026 FlashInfer contest; and, through Humanize Olympiad Agents (HOA), full scores in IOI 2026, IMO 2026, IPhO 2026, and IBO 2024, 418.5/437 in IChO 2026 (gold-medal). Humanize also achieves 672/672 on PutnamBench and ranks first (251/303) on Lean-Eval's leaderboard even competiting with professional mathematicians. The postmortems show that independent review catches unsupported builder claims, but stopping remains a key weakness. In reports that separate rounds by phase, two thirds of rounds occurred after implementation was accepted. This evidence is observational, not a controlled comparison of workflows.
1 INTRODUCTION
Humanize reframes agentic coding around explicit decisions about plans, claims, review, and stopping. It separates implementation from acceptance, and the paper studies its deployment and applications using observational evidence rather than controlled comparisons.
1 INTRODUCTION
- Agents that write code are weak judges of correctness and completion, shifting the challenge from code generation to verification.Errors can accumulate, and models may favor their own outputs or reward-hack by changing tests.Source: §1 · p. 1
- Humanize uses a human-approved plan contract, builder rounds, and a cross-vendor reviewer who alone can declare completion.Deterministic hooks route work and enforce mechanical gates at role boundaries.Source: §1 · p. 2
- The paper’s contributions span the workflow design, a study of deployment and 118 public postmortems, and applications and design lessons.The evidence is observational, not a controlled comparison with other workflows.Source: §1 · p. 2
2 JUDGEMENT ENGINEERING
Judgement engineering makes decisions at workflow boundaries explicit and mechanically enforced. Humanize alternates a builder and an independent reviewer, so a defect survives only when both miss it.
2 JUDGEMENT ENGINEERING
- A single agent that plans, edits, and declares completion leaves humans to reconstruct goals, changes, and evidence from a long transcript or diff.Judgement engineering instead places explicit decisions at boundaries between planning, implementation, review, and learning.Source: §2 · pp. 2–3
- The design fixes the plan before coding, separates building from judging, distinguishes alignment from code review, and measures progress by evidence rather than rounds.The plan contract sets the goal and acceptance criteria; a different model judges evidence and completion.Source: §2 · p. 3
- Humanize generalizes evaluator-optimizer workflows to whole working sessions and keeps orchestration in deterministic code rather than a model.Its principles are enforced at the boundaries between planning, implementation, review, and learning.Source: §2 · p. 3
2.1 ALTERNATING MODELS AS JOINT SAMPLING
- With one model, proposal and acceptance share blind spots; with two models, a defect survives only if the reviewer also misses it.If b is the builder’s defect-and-oversight probability and mR the reviewer’s miss probability, alternation lowers survival from b to b mR when blind spots differ; false positives can add rounds.Source: §2.1 · p. 3, Figure 2 · p. 3
3 SYSTEM DESIGN
Humanize implements its workflow as a Claude Code plugin, separating a human-approved plan from implementation and reviewer-led acceptance. Hooks enforce required artifacts and route the loop through review, fixes, and termination.
3.1 PLANNING AS A CONTRACT
- Planning produces a structured plan contract before coding, with goals, acceptance criteria, scope boundaries, and assigned tasks.Claude and Codex draft and critique the plan for up to three rounds, routing unresolved points to the human; a compliance check and advisory quiz run before execution.Source: §3.1 · p. 4
3.2 THE ROUND PROTOCOL
- Each builder round returns commits, a mainline objective, and evidence-linked claims; a fresh reviewer checks them against the plan, history, and code.The reviewer distinguishes mainline gaps from blocking and queued side issues; two stalled rounds trigger drift recovery, and every fifth round audits all acceptance criteria.Source: §3.2 · pp. 4–5
- After acceptance, Codex reviews the diff; findings start fix rounds, while a clean review permits finalization through a behavior-preserving simplification pass.The implementation phase has a 42-round default cap, but code review has no round cap.Source: §3.2 · p. 5
3 SYSTEM DESIGN
- The Claude Code plugin implements the workflow with commands, subagents, skills, and event hooks.Hook validators enforce artifact presence and form, while the reviewer judges content; the catalog lists 72 mechanical gates.Source: §3 · p. 4, §3.3 · p. 5
3.3 MECHANICAL GATES
- The default runtime uses Claude Code to build and Codex CLI to review, with independence protections that community runtimes implement less fully.Later Codex CLI releases broke parts of the plugin after its last release.Source: §3.3 · p. 6
4 DEPLOYMENT AND POSTMORTEM STUDY
Across 108 days, Humanize reached 68 main-branch versions and continued attracting adoption after its last release. Its 118-postmortem observational study found review-heavy loops, but the corpus was not representative of all runs.
4 DEPLOYMENT AND POSTMORTEM STUDY
- Over 108 days, 68 versions reached the main branch, adding code review, agent teams, additional runtimes, and plan and methodology features.Development later moved to a dev branch and standalone successor.Source: §4 · p. 6
- The repository had 1,468 stars, 61% gained after the last release, 130 forks, and pull requests from 34 authors.Contributors outside the core team supplied runtimes, BitLesson, plan refinement, round contracts, and a monitoring dashboard.Source: §4 · p. 6
- The issue tracker contained 150 issues from 52 authors, including 118 postmortems of real loops; 113 used the methodology analysis signature or template.Source: §4 · p. 6
- Artifact state, review tails, scope, and convergence each appeared in 64–73% of postmortems; review tails and convergence were the most common primary pressures.Agreement with a blind second LLM coder was κ = 0.93 for round counts and κ = 0.79 for primary pressure.Source: §4 · p. 6
- The postmortem corpus is not representative of all runs: users chose what to file, four reporters wrote half the reports, and the analysis ran for selected exits.Source: §4 · p. 6
- 67.3% of rounds (364/541) in 27 phase-separated postmortems fell in uncapped code review, after implementation acceptance.Every pre-acceptance stagnation stop with a stated round number fell on Full Alignment Review, where the reviewer is invited to stop the loop.Source: §4 · p. 6
5 APPLICATIONS
Humanize applications range from contest kernels and olympiad problems to large software migrations and formal-proof benchmarks. These application reports show what the workflows reach, but not how much the reviewer adds.
Gem5: a build-system migration
- The Humanize plugin replaced gem5’s SCons build with CMake and Ninja and added a Bazel overlay across 567 files in 133 commits.The first author validated the result with the full build, unit tests, and matching simulation objects; upstream CI has not passed and the pull request awaits maintainer review.Source: §5 · p. 7
Kernel Design Agents (KDA)
- KDA placed first in MoE, second in DSA, and third in GDN in the MLSys 2026 FlashInfer Full-Agent division, with team-reported speedups of 19.08×, 4.54×, and 1.92×, respectively.Two of its five kernels remained slower than baseline; the team-reported DSA indexer ablation reached 3.71× with the loop, 6.14× with KernelWiki, and 8.58× with profiling, against 1.37× for K-Search.Source: §5 · p. 7, Figure 5 · p. 7
Humanize Olympiad Agents (HOA)
- HOA achieved full scores on all six IOI 2026 tasks, proved all six IMO 2026 problems, scored 30/30 in both IPhO theory runs, and matched all 100 IBO 2024 answers.IChO 2026 reached 418.5/437 theory points with all 68 formalization targets covered; these scores are self-graded, and IBO agents could see the key.Source: §5 · pp. 7–8
PutnamBench and Lean-Eval
- Humanize achieved 672/672 on PutnamBench and ranked first on Lean-Eval’s legacy leaderboard, whose submitted proofs must pass its Comparator check.For formal proofs, the Lean kernel and a statement-preservation check run before the reviewer, which must also obtain an independent proof verifier’s approval.Source: §5 · p. 8
Independent judgement
- Application results show what these loops can reach, not how much the reviewer adds, because no public comparison runs the same builder with and without the reviewer under the same budget.Where external checks are partial, another-vendor reviewer carries the judgement they leave open; complete oracles can check specific claims independently.Source: §5 · p. 8, Figure 6 · p. 8
6 FINDINGS
The postmortems identify trusted but costly independent review, weak stopping behavior, and plan and evidence problems. They also show that collecting reports does not ensure the methodology learns from them.
Independent review is trusted but expensive
- Forty-five of 118 postmortems describe builders claiming more than they had done, while review complaints mainly concern pacing rather than reviewer verdicts.One-finding-per-round review can create a long tail; the authors recommend batching related findings and giving reviewers memory of earlier rounds without importing builder reasoning.Source: §6 · p. 8
Iteration does not guarantee convergence
- In postmortems that split phases, two thirds of rounds came after acceptance, and one loop ran 87 rounds against a cap of 42.A loop recorded “advanced” in every round yet ended with none of its eight criteria fully met; the authors call for bounds, state-based stop rules, and an exit for blocked work.Source: §6 · p. 9
Plans and scope are judgement boundaries
- Large plans, vague acceptance criteria, and mismatched interpretations of deferred work can make scope and stopping conditions ambiguous.The authors propose routing blocking issues with an explanation, recording queued issues without extending the loop, and making plan amendments explicit.Source: §6 · p. 9
Evidence and learning
- Some sessions stalled because acceptance evidence depended on unavailable infrastructure or inconsistent environments, while 118 postmortems contained 771 suggestions with no proposal strongly linked to a main-branch change.Concrete bug reports were fixed within days, but 86 of 150 issues arrived after development had moved to a dev branch and standalone successor; the authors argue that reporting needs a process for judging its own reports.Source: §6 · p. 9
7 RELATED WORK
Humanize sits among coding loops, multi-agent orchestration, and model-based judging, while distinguishing itself through an independent reviewer and deterministic routing. Prior studies cited here do not evaluate Humanize.
Coding agents and loops
- Unlike self-refinement and the Ralph loop, Humanize gives the stop decision to an independent reviewer and adds enforced review to explicit plans in spec-driven tools.The related-work passage notes that newer native loops and loop-design studies exist, but none evaluates Humanize.Source: §7 · p. 9
Multi-agent orchestration
- Humanize uses two model roles and a human, with deterministic hooks routing work and a dedicated role assigned verification.This contrasts with multi-agent frameworks organized through procedures, chat chains, or programmable conversations and with taxonomies that identify task verification as a failure category.Source: §7 · p. 9
Models as judges
- Separate verifiers can catch generator errors, but critics can also report nonexistent bugs, and cross-model code review was counterproductive when the reviewer could not run tests.Humanize’s reviewer can execute code and judges whole sessions; the review tails are one cost of that design.Source: §7 · p. 9
8 LIMITATIONS
Humanize has tool dependencies, coarse termination and gate checks, limited reviewer memory, and no controlled evaluation. These constraints bound what the paper establishes about the workflow.
Tool dependencies
- Humanize runs on Claude Code’s hook interface and Codex CLI, and releases after the plugin’s last release broke parts of it.Only the Claude Code plugin installs the full set of independence protections.Source: §8 · p. 10
Termination and review
- The round cap binds only implementation, the reviewer is invited to stop only every fifth round, and no exit distinguishes environment-blocked work from failing work.The reviewer sees only the diff and has no memory of earlier rounds.Source: §8 · p. 10
Gate checks
- The gates check artifact presence and form rather than content, so a well-formed false summary must be caught by the reviewer; tool-call validators fail open on malformed state files.Source: §8 · p. 10
Evaluation and telemetry
- The evidence is observational: no public study runs the same builder with and without the reviewer under the same budget.Humanize also keeps no telemetry on gate firings or round costs.Source: §8 · p. 10
9 CONCLUSION
Humanize treats judgement as the central design problem in agentic coding, using mechanically enforced decisions around a builder-reviewer loop. Deployment showed both useful independent review and unresolved stopping and convergence weaknesses.
9 CONCLUSION
- Humanize makes goals, builder claims, reviews, evidence, and stopping decisions explicit and mechanically enforced in an alternating builder-reviewer workflow.The contribution is a compact workflow, not a larger agent organization or a stronger claim about autonomous coding.Source: §9 · p. 10
- Independent review caught over-claiming builders, while the builder-reviewer pattern supported applications from a build-system migration to a complete PutnamBench proof set.Source: §9 · p. 10
- Iteration did not guarantee convergence, and cheap reports still required judgement to yield changes.Source: §9 · p. 10
- Development continues in a standalone successor that moves the loop from host-CLI hooks into its own runtime.The successor is identified as Humanfia (2026b), and is to be described separately.Source: §9 · p. 10
A MECHANICAL GATES
Humanize defines 72 mechanical gates that can block, end, or redirect the loop. The largest gate categories protect state, plans, and round isolation, while one malformed-state path fails open.
A MECHANICAL GATES
- 72 gates check the loop and can block, end, or redirect it; the largest categories protect state, plans, and round isolation.Source: A MECHANICAL GATES · p. 14, Table 1 · p. 14
- 32 gates run in the Stop hook, while 29 are tool-call and prompt validators installed only by the Claude Code plugin.Source: A MECHANICAL GATES · p. 14
- On a malformed state file, tool-call validators exit with code 1, which Claude Code treats as non-blocking, so the call proceeds.Source: A MECHANICAL GATES · p. 14
B CORPUS CODING
The corpus comprises 150 issues, including 118 postmortems, which LLM agents coded for reported loop properties and proposed mechanisms. Counts within issues are self-reports, and the supplied table describes pressure counts and coder agreement.
B CORPUS CODING
- 150 issues comprise 118 postmortems and 32 bug reports, feature requests, questions, and other issues.Source: B CORPUS CODING · p. 14
- LLM agents coded stated round counts, exits, phase splits, pressures, and 771 numbered proposed mechanisms; counts within issues are self-reports.Round counts were stated in 93 postmortems, and phase splits in 27.Source: B CORPUS CODING · p. 14
- A blind second LLM coder's stratified sample agreed at κ = 0.93 for round counts, 0.82 for exits, and 0.79 for primary pressure.Source: B CORPUS CODING · p. 14
- Table 2 reports pressure counts over 118 postmortems, where reports may show several pressures but only one primary pressure; review tails and convergence are the most common primary pressures.Source: Table 2 · p. 14