Agentic reliability and evaluations : Enterprises that got burned by a bad eval are the most likely to remove humans from the loop, not the least

AI Summary

A VentureBeat AI study notes that trust in automated agent evaluation by enterprises increased significantly in July, though the failure rate of these agents remained unchanged. Enterprises that experienced failures tend to increase automation, while confidence is higher among those without prior failures.

Across 108 enterprises, trust in automated agent evaluation rose sharply in July — and the failure rate it is supposed to predict did not move at all. The share of organizations that fully trust automated evaluation nearly tripled, from 5% in June to 13%, and the complaint that evaluations don’t match real-world outcomes fell 10 points. Yet the same share as last month — just under half — shipped an agent that passed its evals and then failed a customer. The reason is visible in the cross-tabs: the new trust belongs almost entirely to enterprises that have not yet been burned. Among those that have, 4% trust automated evaluation; among those that haven’t, 24% do. And getting burned does not slow the march to autonomy — it speeds it up. This is the second wave of the VentureBeat Pulse Research agent reliability tracker, and the first fielded on an instrument identical to the month before it. That makes July the first read on direction rather than position: what moved, what held, and what the movement means. What moved is confidence. In June, only 5% of enterprises said they fully trusted automated evaluation, and the most-cited limitation was that evaluations align poorly with real-world outcomes (29%). In July, 13% fully trust automated evaluation and the alignment complaint has fallen to 19%, no longer the leading objection. Both shifts are large enough to read as real rather than noise. What held is the failure. Just under half of organizations (49%) deployed an agent or LLM feature in the past year that passed internal evaluations and then caused a customer-facing failure — statistically indistinguishable from June’s 50% — and a quarter (24%) have seen it happen more than once. Confidence improved; correctness did not. That is the July gap: not between autonomy and trust, as in June, but between trust and the evidence for it. The cross-tabs explain where the new confidence comes from, and it is not from better evaluations. Trust is concentrated almost entirely among enterprises that have not experienced a false-confidence failure: 24% of them fully trust automated evaluation, against 4% of those that have. The trust curve is being lifted by inexperience. Meanwhile the enterprises that have been burned are not retreating from autonomy — 85% of them already allow zero-human deployment or are engineering toward it, against 61% of those that have not been burned. Overall the autonomy trajectory is flat at 67%, but the population inside it has shifted toward the organizations with the most direct evidence that evaluations miss things. The vendor market, by contrast, is finally showing signs of settling. The share of enterprises running no dedicated evaluation tooling fell from 17% to 12%; specialist platforms gained, with Braintrust nearly doubling to 15% and DeepEval reaching 17%; and switching intent cooled, with those planning no change rising from 36% to 44%. Selection criteria moved with it: ease of integration overtook cost as the top factor, jumping from 27% to 39%. Enterprises are done shopping on price and have started buying on fit. Methodology VentureBeat fielded this survey as part of its ongoing Pulse Research series. This wave — the agentic reliability and evals tracker — examines how technical leaders evaluate agent performance and reliability. Responses are filtered to organizations with 100 or more employees (n=108), drawn from a July 2026 fielding. Because the July instrument is identical to June’s, this report makes month-over-month comparisons where they are warranted; where questions were multiple-select, shares can sum to more than 100%. Comparisons against June (n=157) are tested for significance, and only a handful of the month’s movements clear a conventional threshold: the rise in full trust in automated evaluation (5% to 13%), the fall in the real-world-alignment complaint (29% to 19%), the jump in ease of integration as a selection factor (27% to 39%), and the gain in Braintrust as a primary platform (8% to 15%). Movements described in this report as flat — the failure rate, the autonomy trajectory, the production monitoring mix, the investment ranking — are statistically indistinguishable between waves, and that stability is itself the finding. Differences of a few points elsewhere should be read as sample variation, not trend. By role the sample is senior and buyer-credible: 44% are final decision-makers for AI purchases and another 25% recommenders or influencers, a slightly more senior mix than June. Product and program managers (18%), consultants and advisors (12%), CIOs/CTOs/CISOs (11%), and directors of engineering/IT (11%) lead the named titles, alongside a large “Other” function (30%). By organization size the sample is again mid-market-weighted: 100–499 (33%) and 500–2,499 (30%) employees lead, with 2,500–9,999 (23%), 10,000–49,999 (9%), and 50,000+ (5%) above them. One composition change is worth flagging because it bears on the trust finding. The industry mix shifted be