devto 2026-07-29 원문 보기 ↗
A couple of months ago I attended LangChain Interrupt 2026 in San Francisco. As someone leading AI platform and developer experience day to day, some of what I heard confirmed what I'd already been thinking. Some of it reframed it entirely.
The pattern across every team that had actually shipped was a constant: they had tried the sophisticated path first. Production required operability; it was what survived.
This article continues a line of thinking from my piece on the AI-Native Era shift in software engineering, where I named the bottlenecks; governance failures, the evaluation gap, and the limits of user agency. Interrupt came first; I just hadn't formed the right questions yet. This room was full of teams who had hit those bottlenecks and kept going, and surfaced a few I hadn't anticipated. These are the aspects that stayed with me.
Rippling, an HR, IT, and Finance platform, runs AI across all three simultaneously. Their engineering team has production scars to match. What they shared at Interrupt was the kind of advice you only earn by shipping and failing in front of real users.
The instinct when building agents is to reach for sophistication; hierarchical sub-agents, specialized task delegators, deep orchestration trees that look elegant on a whiteboard. Rippling tried that. It didn't survive contact with production.
Four principles emerged from their experience:
Keep agents flat. A top-level agent coordinating tools directly outperforms a hierarchy of sub-agents in most production scenarios. The reason isn't architectural elegance; it's debuggability. Every layer of delegation is another place a failure can hide. A flat agent fails loudly and traceably. A sub-agent hierarchy fails quietly and expensively. Complexity at the orchestration layer is a liability until you've proven the core loop works. The principle mirrors good component architecture; deeply nested hierarchies are hard to debug, hard to reason about, and hard to change. Flat structures win in production.
Build generic composable tools, not task-specific ones. A tool built for one agent workflow can't be reused. A generic tool can be composed into any workflow. This isn't an architectural preference; it's the mechanism by which platform value scales across teams.
Pass code into the LLM context, not data. When you need an agent to reason about a data structure, give it the schema or function definition rather than raw records. Code communicates structure and intent more precisely; it's often more token-efficient and gives the model the right level of abstraction for reasoning tasks. The exception is when the agent needs to reason about specific values; in that case, targeted data beats a schema.
SQL outperforms bespoke tools for structured data retrieval. Instead of building a custom tool for every query pattern an agent might need, a governed SQL interface lets the agent compose queries dynamically. The combinatorial explosion of narrow retrieval tools becomes a real maintenance burden at scale; one well-governed interface handles query patterns you haven't anticipated yet. This doesn't replace action-oriented tools; it replaces the ones whose only job is fetching data.
The teams shipping reliably in production are the ones who reduced complexity before they scaled.
Building an AI platform that other engineers use every day, this hits differently. The architecture decisions we make don't just affect one agent; they become the defaults every team inherits. Flat and composable isn't just good engineering hygiene; it's the model that scales when you're the foundation layer.
Lyft's Nick Ung opened his session with a question nobody in the room wanted to hear: your eval says 100% pass rate, are you sure?
Pass 1 of your eval might score 20%. Pass 2 might score 40%. The score you see from a single run is a sample from a distribution, not a measurement of it. Agent outputs are non-deterministic. Running your eval once and shipping on the result isn't quality assurance; it's optimism.
The deeper problem: most teams are still running generic evals. A generic eval asks whether the response was good. A domain-specific eval asks whether the agent completed the specific task it was built for, under the specific failure modes your domain produces. These are entirely different questions, and only one of them tells you whether you can ship.
What actually works, across every production team that presented:
Domain-specific rubrics, not generic quality scores. The rubric must be behavioral; not "was the response helpful?" but "did the agent complete the task?" Decomposed across independent dimensions: accuracy, safety, task completion, tone. Anchored with concrete examples at each score level, or the LLM judge will drift.
The LLM judge must be calibrated against human labels. Automated eval at scale requires a model as the judge, but a model that hasn't been aligned to human ground truth is measuring its own preferences, not yours. Sample real interactions, collect human labels, iterate on the rubric until the judge reaches high agreement. This is not a one-time exercise; it's an ongoing calibration loop.
Simulated users are a first-class artifact. Lyft described using a separate LLM to simulate user behavior in offline eval. The insight most teams miss: the simulated user needs calibration just like the judge. An unrealistic simulated user produces an unreliable eval. This is a two-model problem; both the agent and the simulator need to be tuned.
How many runs you need depends on the CUL Trilemma. The most quotable framework of the conference, from Rippling: Cost, Uncertainty, Lag. You can't optimize all three simultaneously. High-stakes use cases need more runs, accept higher cost and lag. Internal tooling can trade certainty for speed. The number of eval repetitions is not a universal constant; it's a per-use-case decision anchored in explicit risk tolerance.
Compliance teams should co-author the evals. Chime's lesson: the old model, legal reviews at kickoff and again at the release gate, is a recipe for late-cycle rejection. When compliance co-authors the eval rubric, the rules become executable tests. The feedback loop shrinks from months to hours.
The eval flywheel closes this loop: production behavior becomes evidence, evidence becomes an issue, the fix becomes an evaluator watching for the failure to return. That is the loop running at full speed.
This is the piece I think about most in the context of developer experience; it's the problem I'm closest to right now. Eval infrastructure that's hard to use doesn't get used. The gap between knowing you should evaluate and having infrastructure that makes it frictionless is where most teams get stuck; both offline and online eval, accessible without writing code. The teams that cracked this made eval feel like a natural part of the development loop, not a gate engineers dread at the end of a sprint.
Aaron Levie made the point that reframed everything else in his session with Harrison Chase: coding agents are the easy case. Code is verifiable. The feedback loop is tight. The users are technical. Every other category of knowledge work agent has none of those properties.
The governance challenge isn't hypothetical. It's the reason most enterprise agents are still internal tools, still in controlled pilots, still not touching the workflows where the real value lives.
Four principles surfaced consistently across the sessions that had actually solved it: least-privilege authorization, data obfuscation, hardcoded human-in-the-loop, and continuous eval.
Least-privilege runtime authorization. An agent should never operate with blanket system access. Authorization must happen dynamically at runtime, for every individual tool call. If the user doesn't have permission to access a system, the agent cannot inherit that access; not by convention, but by architecture.
Data obfuscation before external routing. PII and sensitive enterprise data must be masked before they reach any external LLM provider. The masking layer sits between your data systems and the model, not between the model and the user.
Hardcoded human-in-the-loop for high-stakes actions. For actions with material consequences; transactions, submissions, irreversible state changes, human approval should be programmatically enforced, not left to the agent's judgment. Every approval must be auditable. The auditability requirement is what separates a governance framework from a policy document. And how you design those approval moments determines whether users trust the system or abandon it; HITL is a design surface as much as a safety mechanism.
Continuous eval as the lifecycle backbone. Agent behavior drifts. Models update. User behavior shifts. A governance model that only evaluates at release loses signal the moment it ships. Automated evaluation loops running continuously against production traces; testing for regression, prompt injection, and edge case failures, are the operational layer that keeps governance current.
Toyota went from six months and six engineers per deployment to four days and one engineer after building a shared platform. They now run more than 50 agents in production. The platform didn't just accelerate deployment; it made governance scale.
That's not an efficiency gain; it's a different category of result. Governance isn't a constraint on velocity; it's what makes velocity sustainable. A platform that bakes it in doesn't slow teams down; it removes the decision entirely.
Clay, a Go-To-Market platform, runs 350 million agent executions per month. Jeff Barg, their Head of AI, didn't talk about prompts or models. He talked about back-pressure, throughput, caching, and bounded retries.
The insight that reframed how I think about agent infrastructure: cost is not an operational concern that shows up after you ship; it's an engineering discipline that has to be designed in from the start. Aaron Levie made the same point from a different angle; public companies cannot absorb a sudden $10M AI bill mid-quarter. The financial model of enterprise AI is not a finance problem. It's an architecture problem.
What the teams operating at scale had actually built:
Model routing by task type. Not every step in an agent workflow requires a frontier model. Summarization, search, extraction, classification are solved problems for smaller, faster, cheaper models. Frontier models are reserved for hard reasoning, ambiguous judgment calls, synthesis across complex context.
Bounded retries. Forcing an agent to stop after a fixed number of steps often produces better results than letting it spiral. Unbounded retries compound cost and compound errors. The retry budget is a design constraint, not a safety net.
Prompt caching. For agents with shared system context across many executions, caching the prompt prefix can reduce infrastructure costs significantly; one of the highest-leverage optimizations available and one of the least discussed.
Adaptive throttling. Modeled on TCP congestion control; when the system is under load, it slows gracefully rather than failing expensively. The result in practice: 4 to 10x throughput improvements.
The architecture of resilience and the architecture of cost efficiency turn out to be the same architecture.
From a platform perspective, cost visibility is a developer experience problem as much as a finance problem. Engineers making model and tool choices mid-sprint rarely have the feedback loop to understand the cost implications. Surfacing that signal early, at the point of development, is the missing layer in most AI platforms today.
Andrew Ng's conversation with Harrison Chase was one of the most strategically important sessions of the conference, and the easiest one to undervalue if you're focused on the engineering.
The argument was precise: bottom-up automation produces incremental efficiency. Top-down workflow redesign with executive sponsorship produces transformation. The difference is not a matter of degree; it's a matter of kind.
Automating loan processing saves an hour per application. Rethinking how loans get approved; who touches them, what decisions require human judgment, what the agent owns end-to-end, that's the 20 to 50% transformation. In lending, auto-decisioning was the unlock; not because it was faster, but because it changed the workflow entirely. The agent is the same in both scenarios. The workflow design is not.
The question to ask before you start is not "what can we automate?" It's "if we could redesign this workflow from scratch with agents as a primitive, what would it look like?" Those are different questions and they produce different systems.
The teams that answer the second question are the ones CJ Desai wasn't talking about when he said agent washing; proof-of-concepts dressed up as a strategy. But getting there isn't just an engineering decision; it requires product, legal, operations, and executive alignment simultaneously. The platform can enable it; but it can't substitute for the organizational decision to pursue it.
Sophisticated agents impress in demos. Operable agents survive production.
The distinction lives in decisions made before you scale:
This is the level of engineering leadership where platform investment becomes the strategy.