Lenny's Newsletter · 1 Jul 2026
A repeatable evaluation harness, built live in under an hour with Claude Code, ran five frontier models through 64 blind generations across PRD, prototype, agentic, and voice tasks — blending human scoring at 70% with LLM judging at 30% — and the model-by-task verdict diverged from benchmark expectations.
Industry lens
If repeatable, self-built eval harnesses become standard product practice, do published vendor benchmarks lose influence over enterprise model-selection decisions, and does that pressure model makers to compete on task-level transparency rather than leaderboard scores?
