Benchmark
Tower vs stock pi,
measured, not argued.
Five software tasks — a string library, a durable queue, a transactional store, a ticket system, and a self-contained web app — run through Tower's full pipeline and through stock pi: same model (DeepSeek-V4.1-Flash, ~$0.30 per million tokens), same briefs, same hidden acceptance suites, and pi deliberately stripped of every user extension, skill, and prompt template. Correctness is graded by mechanical test suites nobody saw; design quality by a judge — and every head-to-head re-judged by an independent third-party model to rule out self-preference.
The shape of the comparison
Six dimensions, 0–10, strict anchoring. Correctness is a hidden acceptance suite run against whatever the arm shipped — no judge involved. The other five (spec fit, code quality with architecture/SOLID weighed explicitly, testing, robustness, documentation) are judge-scored with file-cited evidence.
Every task, both arms
| Task | Hidden tests | pi Σ | Tower Σ | Comparison verdict | Cost (pi / Tower) |
|---|---|---|---|---|---|
| sluglib — string library | 18 | 8.4 | 9.4 | Tower, clear — both judges | $0.05 / $0.33 |
| evqueue — durable queue | 18 | 9.1 | 7.9 | pi, clear — both judges | $0.10 / $0.23 |
| kvstore — transactional store | 18 | 7.8 | 7.6 | Tower, clear — both judges | $0.07 / $0.48 |
| tickets — ticket system | 15 | 9.3 | 8.9 | Split — rubric π, independent Tower | — / — |
| webnote — notes web app | 11 (boots the server) | 7.8 | 9.1 | Tower, clear — both judges | $0.007 / $0.062 |
The two arms are never far apart on engineering — correctness tied nearly everywhere, and the independent judge gave pi the durable-queue task for genuinely better failure semantics. What separates them is what ships: the pipeline's gates enforced documentation, test depth, and failure-path discipline that stock pi skipped — on the web app, both judges caught a socket-destroying edge-case bug in pi's server and a placeholder README claiming the implementation didn't exist. Tower's merged branch had guarded decode, socket-preserving rejection, and a real README.
The finding that mattered most
Run with a normal user's extensions loaded, stock pi wrote READMEs and looked disciplined. Run truly stock — no extensions, no skills, no prompt templates — it shipped stub documentation and the edge-case bug. The discipline was never pi's; it was the environment's. Tower's discipline ships with the repository — its prompts, gates, and reviews are versioned code, identical on every machine — which is why it reproduced across five tasks and three models while the solo loop's behavior swung with whatever happened to be installed. (Pi's environment discovery is now an explicit per-project opt-in inside Tower, off by default.)
Honest fine print
Tower's pipeline costs roughly 5–10× the wall-clock and 5–10× the tokens of the solo loop — cents per task at these prices, with ~95% of prompt tokens served from the provider's cache. On a free-tier model the pipeline amplified the builder's failures and shipped nothing: the discipline is a multiplier, and it needs a capable builder to multiply. One task split between the two judges; single-run variance on close tasks is real. A free model's quota, a provider pricing anomaly, and two agents that deleted their own workspaces are all in the incident log — nothing is smoothed over.
Full evidence — per-cell scores, judge rationales with file citations, the cost ledger, incident log, and the model-selection audit: benchmarks/se-bench/verdict.md in the repository, with cost-report.md beside it.