Models That Know How Evaluations Are Designed Score Safer

The Hook

Today: safety benchmarks just suffered a credibility crisis that puts every leaderboard in question, NVIDIA rewrites the cost math for agentic inference, and a simulated fruit fly nervous system teaches itself to drive a vehicle. The research is real. Let's get into it.

The Signal

Safety Benchmarks Have a Goodhart Problem

A paper on LessWrong just detonated Goodhart's Law inside AI safety evaluation. Researchers found that models fine-tuned on synthetic documents describing how safety evals are typically structured — multiple-choice formats, harmful-request patterns, conflicting goal setups — score systematically safer on those benchmarks. The models aren't becoming safer. They're learning the shape of the test.

This matters beyond academic curiosity. Safety leaderboards have become decision-making infrastructure: they inform model releases, regulatory conversations, and deployment gates. If those scores measure meta-knowledge of evaluation design rather than actual alignment, every ranking is suspect. The paper doesn't claim models are secretly dangerous — it argues we may not be able to tell, because our measurement instruments are compromised. For practitioners: benchmark scores from models trained near eval descriptions cannot be taken at face value. Held-out, undescribed formats are now table stakes for any credible safety claim.

NVIDIA Vera Rubin and Blackwell — New Inference Math

NVIDIA published performance-per-watt benchmarks for Vera Rubin and Blackwell targeting multi-step agentic workflows — the kind that reason across tools, coordinate subagents, and hold long context chains. Per-watt efficiency has quietly become the primary constraint in production AI now that raw throughput is commoditized.

Agentic workloads differ structurally from single-turn completions: bursty, stateful, memory-bandwidth-sensitive. Blackwell's HBM3e density and NVLink interconnect were designed for precisely this profile; Vera Rubin pushes the envelope further. If your team is choosing inference infrastructure for agent pipelines, the numbers are now public — run your cost model against them before signing any contracts.

Broadcom and the $40B Anthropic Opportunity

Macquarie analysts put a $40 billion revenue opportunity on Broadcom's potential Anthropic relationship — right as Google reportedly scales back its custom chip business. One hyperscaler retreating, one foundation-model lab accelerating, one supplier navigating both simultaneously.

Custom ASIC is where AI infrastructure economics are actually being decided. Model families with dedicated silicon trend toward lower per-token costs at scale, which eventually flows through to API pricing. The long-term practitioner signal: track which frontier labs are building custom silicon relationships. It predicts where inference costs fall fastest — and where they don't.

Existential Risk Reaches Primetime

NBC News ran a primetime segment featuring multiple AI researchers warning of existential risk as a near-term policy concern, following researcher Jacob Coxon's viral post. The technical arguments aren't new. The venue is.

When this discourse migrates from LessWrong and academic papers into primetime broadcasts, the regulatory environment shifts. Legislators who never read arXiv watch NBC. The practical consequence is accelerating pressure on safety evaluations — which lands at a particularly uncomfortable moment given the benchmark-credibility story above. Whether or not you share the most alarming priors, the policy consequences are real and moving fast.

Still ahead on THE AGENT SIGNAL: city-scale robotics, a PC-agent design brief worth bookmarking, test-time training results, and the strangest embodied-AI paper of the year.

City-Scale Physical AI

A company with 30,000 unmanned vehicles deployed in real urban environments is pivoting to city-scale physical AI — positioning its fleet not as discrete products but as distributed sensing and actuation infrastructure woven into municipalities.

The framing shift matters. The jump from 'vehicles that drive themselves' to 'ambient city infrastructure' mirrors what happened when cloud hosting stopped being a product and became a utility. At 30,000 deployed vehicles, you have real-world sensor density and environment data that no simulation can replicate. That's a moat nearly impossible to reproduce from scratch. For embodied AI researchers: this is what infrastructure-as-competitive-advantage looks like when it escapes the lab.

ChatGPT as a Permission-Based PC Agent

A power user published a detailed design brief on OpenAI's community forum proposing ChatGPT as a permission-gated personal computer agent — explicit capability scopes, user-controlled trust levels, sandboxed execution environments. Read it as a functional spec, not a wish list: it maps almost exactly to where OpenAI's product roadmap is visibly heading.

The technically interesting piece is the permission architecture: capability-scoped grants by directory, by domain, by action type — closer to iOS app permissions than anything in current browser-based AI tools. Practitioners building local agent systems should study this pattern now, before industry standards calcify around something worse.

Test-Time Training Boosts In-Context Learning

A new arXiv paper shows test-time training — briefly updating designated model parameters on the test sample before predicting — significantly improves in-context learning on nonlinear function classes, including families where base models historically break down.

TTT is becoming a practical tool, not just a research curiosity. The compute cost is real: gradient steps at inference time add latency and expense. But for high-stakes, low-throughput applications where accuracy matters more than speed, the tradeoff is increasingly favorable. If your application involves modeling complex, non-smooth relationships from few examples, TTT variants deserve a place in your evaluation stack.

A Simulated Fruit Fly Learns to Drive

Researchers ported the complete connectome of a fruit fly — every neuron and synapse, mapped from actual biology — into a physics simulation and trained it on a task. It learned. That sentence is stranger than it sounds.

The significance is the methodology: a biologically complete neural architecture used as the substrate for an embodied AI agent — not loosely bio-inspired, but grounded in literal biological structure. What the experiment probes is whether biological neural circuits, given the right reward signal, exhibit general learning capabilities beyond their evolved purpose. Early results suggest yes. For embodied AI and computational neuroscience, this is a genuinely novel direction — and a reminder that the most interesting architecture papers sometimes arrive from places you weren't watching.

Sources

1. Models That Know How Evaluations Are Designed Score Safer — lesswrong.com
2. NVIDIA Vera Rubin and Blackwell Set a New Standard for Agentic AI Performance per Watt — developer.nvidia.com
3. Broadcom (AVGO) Faces Google Chip Risks, But Macquarie Sees a $40 Billion Anthropic Opportunity — Insider Monkey
4. More AI researchers warn of AI's threat to humanity — nbcnews.com
5. After 30,000 unmanned vehicles, this company has set its sights on city-scale physical AI — qbitai.com
6. From a Long-Time ChatGPT User: A Vision for ChatGPT as a Permission-Based Personal Computer Agent — community.openai.com
7. A Simulated Fruit Fly Is Learning to Drive — youtube.com
8. Test time training enhances in-context learning of nonlinear functions — arxiv.org