Agent Reliability

User feedback helps models repair targeted faults, but LLM judges often miss those repairs. Agent evals should retain human feedback and avoid treating model preference as ground truth.

Language-based monitoring cannot fully reveal an agent’s internal computation, the paper argues. Secure agent runtimes still need isolation and data-flow controls independent of self-reporting.

A telecom diagnosis pipeline structures telemetry before forcing a decision path and evidence-grounded explanation. The pattern is relevant to agents reasoning over heterogeneous operational data.

A label-free Dutch-book test finds internally inconsistent probabilities from language models, especially when prompts add logical complexity or irrelevant context.

SafeEvolve turns completed agent trajectories into reversible harness updates and policy training, linking runtime safeguards with learned multi-step behavior.

This talk proposes legally registered agent organizations, scoped JWT authority, and blockchain audit trails for open-internet commerce; it is an architecture proposal, not validation.

Paid agent tools turn a skill file into spending authority. Builders need budgets, transaction logs, and counterparty screening before letting agents purchase data or services unattended.

x402 servers can perform work before payment settlement, leaving a double-spend window. Builders should settle first or accept explicit counterparty risk until stronger schemes mature.

Agent authorization should scale with stakes and reversibility: logs may cover coding changes, while open-ended payments need scoped, time-bound, independently verifiable mandates.

AWS is separating agent payment policy from model execution: AgentCore handles wallets and limits, while WAF meters bot access. The useful pattern is deterministic spend control at the edge.

CordisBench finds that models lose reliability as harness lifecycle interactions grow, while deterministic execution semantics can replace costly reasoning on controlled cases.

StudentSim turns sparse user histories into individualized simulators that model both current behavior and response to guidance, a pattern for testing adaptive agents before live deployment.

A production-derived post-training recipe consolidated more than 200 internal apps onto one self-hosted model by training separate experts for distinct quality gaps, then merging them.

SAGE uses an expensive VLM only when a learner is uncertain, tests its advice against environment rewards, and distills useful guidance into a policy that needs no VLM at runtime.

A wallet-equipped agent crossed paywalls and completed paid email and phone actions under a spending cap. The engineering lesson is to enforce budgets in the wallet, not in prompts.

Agent checkout needs more than payment tools: enforce user-set limits outside the agent, expose structured merchant data, honor cancellation, and log every decision.

Claude Fable 5.1 reaches Vercel AI Gateway with ordered fallbacks for classifier refusals, but its 30-day retention policy rules out zero-data-retention workloads.

DIASENTINEL combines deterministic extraction, guideline retrieval, risk prediction, and hybrid verification on-premise. It is a useful architecture reference for auditable agents handling sensitive data.

BLOOM-WILT makes rare model behaviors easier to surface by adapting both the auditor’s strategy and target decoding. Its results also show that elicitation method can reverse safety rankings.

S3Gym tests whether agents can explore, judge outcomes, and reuse experience. Raw history, compressed memory, and training each help in some tasks, but none is a reliable default.

Maersk’s production agents depend less on a clever loop than on executable SOPs, bounded tools, replayable traces, and a correction system shared by experts and engineers.

Navan treats production agents as stateful services needing persistence, composable skills, traces, authorization gates, and trajectory-based tests before multi-agent complexity.

Once agents mutate external state, timeouts mean unknown outcomes. Builders need idempotent tools, bounded retries, scoped credentials, durable traces, and explicit recovery paths.

Agent architectures are expiring quickly. Keep model, search, and orchestration choices replaceable, and evaluate platforms by how well they handle repeated change.

Enterprise AI contracts are won on security, controls, integration, and support as much as model capability. Builders should make those operational surfaces part of the product early.

Formal verification gives agent-generated code a machine-checkable contract beyond tests. The hard part shifts upstream: humans must define and validate what correct means.

Figma’s cautious agent adopters expose missing safeguards, not resistance. Use their feedback to improve verification, plans, review size, and the labeling of AI-generated work.

Agents often stop after finding plausible code context. Werry argues for task-specific retrieval that also surfaces decisions, conventions, PR history, and expertise before planning or review.

RedEvoAgent turns prior jailbreak trajectories into a compact attack skill, then keeps only validated improvements. It offers a more interpretable way to probe tool-using agents for unsafe actions.

Persona-Execution Separation isolates mutable agent behavior from audited, stateful work through a governed bridge. It is an architecture option for multi-user agents under compliance constraints.

Mike Krieger describes shifting from stepwise prompting to goal-level delegation, backed by verification, observability, feature flags, and frequent decisions to continue or stop experiments.

TraceML exposes why coding agents stall on long ML work: they repeat narrow optimization loops instead of alternating work types, revisiting discarded ideas, and pivoting like experts.

Correct answers can hide invalid computation. Trace Integrity proposes executable, replayable contracts for data agents and reports 45.8–59.1% correct-answer/invalid-trace rates across three SQL modes.

Snowflake’s rollout favors narrow, high-accuracy coverage, staged adoption, and log-driven iteration over connecting every data source before launch.

Notion treats GTM automation as a shared context system for humans and agents, not an AI layer over disconnected tools. The key design choice is keeping risky customer actions human-approved.

Clay frames GTM agents as a data and orchestration problem: resolve entities, evaluate providers, isolate agent-written state, and account for delayed syncs before automating decisions.

Web agents should validate fetched content before spending model tokens on it and reserve browser automation for interactive steps. Explicit failures beat silently passing CAPTCHAs or blocked HTML downstream.

Plausible outputs can hide consequential omissions that generic LLM judges miss. Production evals need real failure discovery and retrieved expert judgments, not a frozen rubric alone.

Reliable background agents need runtime primitives more than elaborate framework abstractions: durable events, queues, versioned inputs, causal traces, diffs, and replay.

Tool-by-tool restrictions failed to contain a chip-design agent. Enforce permissions at the substrate, isolate files by role, and propagate approved spec changes from one source of truth.