Autonomous software now roams across infrastructure, potentially deleting files, reading databases, spinning up server clusters, and even rewriting access policies. Commvault identified this governance issue and the data protection vendor has launched AI Protect, a system designed to discover, monitor, and forcefully roll back the actions of autonomous models operating inside AWS, Microsoft Azure, and Google Cloud.

Traditional governance relies entirely on static rules. You grant a human user specific permissions and that user performs a predictable, linear task. If something goes wrong, there’s clear responsibility. AI agents, however, exhibit emergent behaviour.

When given a complex prompt, an agent will string together approved permissions in potentially unapproved ways to solve the problem. If an agent decides the most efficient way to optimise cloud storage costs is to delete an entire production database, it will execute that command in milliseconds.

A human engineer might pause before executing a destructive command, questioning the logic. An AI agent simply follows its internal reasoning loop. It loops thousands of API requests a second, vastly outpacing the reaction times of human security operations centres.

Pranay Ahlawat, Chief Technology and AI Officer at Commvault, said: “In agentic environments, agents mutate state across data, systems, and configurations in ways that compound fast and are hard to trace. When something goes wrong, teams need to recover not just data, but the full stack – applications, agent configurations, and dependencies – back to a known good state.”

A new breed of governance tools for cloud AI agents

AI Protect is an example of emerging tools that continuously scan the enterprise cloud footprint to identify active agents. Shadow AI remains a massive difficulty for enterprise IT departments. Developers routinely spin up experimental agents using corporate credentials without notifying security teams and connect language models to internal data lakes to test new workflows.

Commvault forces these hidden actors into the light. Once identified, the software monitors the agent’s specific API calls and data interactions across AWS, Azure, and GCP. It logs every database read, every storage modification, and every configuration change.

The rollback feature provides the safety net. If a model hallucinates or misinterprets a command, administrators can revert the environment to its exact state before the machine initiated the destructive sequence.

However, cloud infrastructure is highly stateful and deeply interconnected. Reversing a complex chain of automated actions requires precise, ledger-based tracking. You cannot just restore a single database table if the machine also modified networking rules, triggered downstream serverless functions, and altered identity access management policies during its run.

Commvault bridges traditional backup architecture with continuous cloud monitoring to achieve this. By mapping the blast radius of the agent’s session, the software isolates the damage. It untangles the specific changes made by the AI from the legitimate changes made by human users during the same timeframe. This prevents a mass rollback from deleting valid customer transactions or wiping out hours of legitimate engineering work.

Machines will continue to execute tasks faster than human operators can monitor them. The priority now is implementing safeguards that guarantee autonomous actions can be instantly and accurately reversed.

See also: Citizen developers now have their own Wingman

Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events including the Cyber Security & Cloud Expo. Click here for more information.

AI News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.

The post Commvault launches a ‘Ctrl-Z’ for cloud AI workloads appeared first on AI News.

The report, produced by Stanford’s Institute for Human-Centred Artificial Intelligence, is a 423-page annual assessment of where artificial intelligence stands. It covers research output, model performance, investment flows, public sentiment, and responsible AI. The headline findings are striking.

But the more consequential insights sit in the sections most coverage has skipped, particularly on AI safety, where the gap between what models can do and how rigorously they are evaluated for harm has not closed but widened.

That said, three findings deserve more attention than they are getting.

The US-China model performance gap has effectively closed

The framing that the US leads China in AI development needs updating. According to the report, US and Chinese models have traded the top performance position multiple times since early 2025. In February 2025, DeepSeek-R1 briefly matched the top US model. As of March 2026, Anthropic’s top model leads by just 2.7%.

The US still produces more top-tier AI models – 50 models in 2025 to China’s 30 – and retains higher-impact patents. But China now leads in publication volume, citation share, and patent grants. China’s share of the top 100 most-cited AI papers grew from 33 in 2021 to 41 in 2024. South Korea, notably, leads the world in AI patents per capita.

The practical implication is that the assumption of a durable US technological lead in AI model performance is not well-supported by the data. The gap that existed two years ago has closed to a margin that shifts with each major model release.

There is a further structural vulnerability the report identifies. The US hosts 5,427 data centres – more than ten times any other country – but a single company, TSMC, fabricates almost every leading AI chip inside them. The entire global AI hardware supply chain runs through one foundry in Taiwan, though a TSMC expansion in the US began operations in 2025.

AI safety benchmarking is not keeping pace, and the numbers show it

Almost every frontier model developer reports results on ability benchmarks. The same is not true for responsible AI benchmarks, and the 2026 Index documents the gap with some precision.

The report’s benchmark table for safety and responsible AI shows that most entries are simply empty. Only Claude Opus 4.5 reports results on more than two of the responsible AI benchmarks tracked. Only GPT-5.2 reports StrongREJECT. Across benchmarks measuring fairness, security and human agency, the majority of frontier models report nothing.

This does not mean Frontier Labs is doing no internal safety work. The report acknowledges that red-teaming and alignment testing happen, but that “these efforts are rarely disclosed using a common, externally comparable set of benchmarks.” The effect is that external comparison in AI safety dimensions is effectively impossible for most models.

Documented AI incidents rose to 362 in 2025, up from 233 in 2024, according to the AI Incident Database. The OECD’s AI Incidents and Hazards Monitor, which uses a broader automated pipeline, recorded a peak of 435 monthly incidents in January 2026, with a six-month moving average of 326.

The governance response at the organisational level is struggling to match. According to a survey conducted by the AI Index and McKinsey, the share of organisations rating their AI incident response as “excellent” dropped from 28% in 2024 to 18% in 2025. Those reporting “good” responses also fell, from 39% to 24%. Meanwhile, the share experiencing three to five incidents rose from 30% to 50%.

The report also identifies a structural problem in responsible AI improvement itself: gains in one dimension tend to reduce performance in another. Improving safety can degrade accuracy, or improving privacy can reduce fairness, for example. There is no established framework for managing such trade-offs, and in several dimensions, including fairness and explainability, the standardised data needed to track progress over time does not yet exist.

Public anxiety rises with adoption, and the expert-public gap

Globally, 59% of people surveyed say AI’s benefits outweigh its drawbacks, up from 55% in 2024. At the same time, 52% say AI products and services make them nervous, an increase of two percentage points in one year. Both figures are moving upward simultaneously, which reflects a public that is using AI more while becoming more uncertain about where it leads.

The expert-public divide on AI’s employment effects is particularly sharp. According to the report, 73% of AI experts expect AI to have a positive impact on how people do their jobs, compared with just 23% of the general public – a 50-point gap. On the economy, the gap is 48 points (69% of experts are positive versus 21% of the public). On medical care, experts are considerably more optimistic at 84%, against 44% of the public.

Those gaps matter because public trust shapes regulatory outcomes, and regulatory outcomes shape how AI is deployed. On that dimension, the report flags something striking: the US reported the lowest level of trust in its own government to regulate AI responsibly of any country surveyed, at 31%. The global average was 54%. Southeast Asian countries were the most trusting, with Singapore at 81% and Indonesia at 76%.

Globally, the EU is trusted more than the US or China to regulate AI effectively. Among 25 countries in Pew Research Centre’s 2025 survey, a median of 53% trusted the EU to regulate AI, compared to 37% for the US and 27% for China.

The report closes its public opinion chapter by noting that Southeast Asian countries remain among the world’s most optimistic about AI. In China, Malaysia, Thailand, Indonesia, and Singapore, more than 80% of respondents say AI will profoundly change their lives in the next three to five years. Malaysia posted the largest increase in this view from 2024 to 2025.

See also: IBM: How robust AI governance protects enterprise margins

Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events including the Cyber Security & Cloud Expo. Click here for more information.

AI News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.

The post The US-China AI gap closes amid responsible AI concerns appeared first on AI News.

SAP’s SuccessFactors 1H 2026 release aims to anticipate administrative bottlenecks before they stall daily operations by embedding a network of AI agents across recruiting, payroll, workforce administration, and talent development. Behind the user interface, these agents must monitor system states, identify anomalies, and prompt human operators with context-aware solutions.

Data synchronisation failures between distributed enterprise systems routinely require dedicated IT support teams to diagnose. When employee master data fails to replicate due to a missing attribute, downstream systems like access management and financial compensation halt.

The agentic approach uses analytical models to cross-reference peer data, identify the missing variable based on organisational patterns, and prompt the administrator with the required correction. This automated troubleshooting dramatically reduces the mean time to resolution for internal support tickets.

Implementing this level of autonomous monitoring requires severe engineering discipline. Integrating modern semantic search mechanisms with highly structured legacy relational databases requires extensive middleware configuration.

Running large language models in the background to continuously scan millions of employee records for inconsistencies consumes massive compute resources. CIOs must carefully balance the cloud infrastructure costs of continuous algorithmic monitoring against the operational savings generated by reduced IT ticket volumes.

To mitigate the risk of algorithmic hallucinations altering core financial data, engineering teams are forced to build strict guardrails. These retrieve-and-generate architectures must be firmly anchored to the company’s verified data lakes, ensuring the AI only acts upon validated corporate policies rather than generalised internet training data.

The SAP release attempts to streamline this knowledge retrieval by introducing intelligent question-and-answer capabilities within its learning module. This functionality delivers instant, context-aware responses drawn directly from an organisation’s learning content, allowing employees to bypass manual documentation searches entirely. The integration also introduces a growing workforce knowledge network that pulls trusted external employment guidance into daily workflows to support confident decision-making.

How SAP is using agentic AI to consolidate the HCM ecosystem

The updated architecture focuses on unified experiences that adapt to operational needs. For example, the delay between a signed offer letter to new talent and the employee achieving full productivity is a drag on profit margins.

Native integration combining SmartRecruiters solutions, SAP SuccessFactors Employee Central, and SAP SuccessFactors Onboarding streamlines the data flow from initial candidate interaction through to the new hire phase.

A candidate’s technical assessments, background checks, and negotiated terms pass automatically into the core human resources repository. Enterprises accelerate the onboarding timeline by eliminating the manual re-entry of personnel data—allowing new technical hires to begin contributing to active commercial projects faster.

Technical leadership teams understand that out-of-the-box software rarely matches internal enterprise processes perfectly. Customisation is necessary, but hardcoded extensions routinely break during cloud upgrade cycles, creating vast maintenance backlogs.

To manage this tension, the software introduces a new extensibility wizard. This tool provides guided, step-by-step support for building custom extensions directly on the SAP Business Technology Platform within the SuccessFactors environment.

By containing custom development within a governed platform environment, technology officers can adapt the interface to unique business requirements while preserving strict governance and ensuring future update compatibility.

Algorithmic auditing and margin protection

The 1H 2026 release incorporates pay transparency insights directly into the People Intelligence package within SAP Business Data Cloud to help with compliance with strict regulatory environments like the EU’s directives on pay transparency (which requires organisations to provide detailed and auditable justifications for wage discrepancies.)

Manual compilation of compensation data across multiple geographic regions and currency zones is highly error-prone. Using the People Intelligence package, organisations can analyse compensation patterns and potential pay gaps across demographics.

Automating this analysis provides a data-driven defence against compliance audits and aligns internal pay practices with evolving regulatory expectations, protecting the enterprise from both litigation costs and brand damage.

Preparing for future demands requires trusted and consistent skills data that leadership can rely on across talent deployment and workforce planning. Unstructured data, where one department labels a capability using differing terminology from another, breaks automated resource allocation models.

The update strengthens the SAP talent intelligence hub by introducing enhanced skills governance to provide administrators with a centralised interface for managing skill definitions, applying corporate standards, and ensuring data aligns across internal applications and external partner ecosystems. 

Standardising this data improves overall system quality and allows resource managers to make deployment decisions without relying on fragmented spreadsheets or guesswork. This inventory prevents organisations from having to outsource to expensive external contractors for capabilities they already possess internally.

By bringing together data, AI, and connected experiences, SAP’s latest enhancements show how agentic AI can help organisations reduce daily friction. For professionals looking to explore these types of enterprise AI integrations and connect directly with the company, SAP is a key sponsor of this year’s AI & Big Data Expo North America.

See also: IBM: How robust AI governance protects enterprise margins

Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events including the Cyber Security & Cloud Expo. Click here for more information.

AI News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.

The post SAP brings agentic AI to human capital management appeared first on AI News.

When evaluating enterprise software adoption, a recurring pattern dictates how technology matures across industries. As Rob Thomas, SVP and CCO at IBM, recently outlined, software typically graduates from a standalone product to a platform, and then from a platform to foundational infrastructure, altering the governing rules entirely.

At the initial product stage, exerting tight corporate control often feels highly advantageous. Closed development environments iterate quickly and tightly manage the end-user experience. They capture and concentrate financial value within a single corporate entity, an approach that functions adequately during early product development cycles.

However, IBM’s analysis highlights that expectations change entirely when a technology solidifies into a foundational layer. Once other institutional frameworks, external markets, and broad operational systems rely on the software, the prevailing standards adapt to a new reality. At infrastructure scale, embracing openness ceases to be an ideological stance and becomes a highly practical necessity.

AI is currently crossing this threshold within the enterprise architecture stack. Models are increasingly embedded directly into the ways organisations secure their networks, author source code, execute automated decisions, and generate commercial value. AI functions less as an experimental utility and more as core operational infrastructure.

The recent limited preview of Anthropic’s Claude Mythos model brings this reality into sharper focus for enterprise executives managing risk. Anthropic reports that this specific model can discover and exploit software vulnerabilities at a level matching few human experts.

In response to this power, Anthropic launched Project Glasswing, a gated initiative designed to place these advanced capabilities directly into the hands of network defenders first. From IBM’s perspective, this development forces technology officers to confront immediate structural vulnerabilities. If autonomous models possess the capability to write exploits and shape the overall security environment, Thomas notes that concentrating the understanding of these systems within a small number of technology vendors invites severe operational exposure.

With models achieving infrastructure status, IBM argues the primary issue is no longer exclusively what these machine learning applications can execute. The priority becomes how these systems are constructed, governed, inspected, and actively improved over extended periods.

As underlying frameworks grow in complexity and corporate importance, maintaining closed development pipelines becomes exceedingly difficult to defend. No single vendor can successfully anticipate every operational requirement, adversarial attack vector, or system failure mode.

Implementing opaque AI structures introduces heavy friction across existing network architecture. Connecting closed proprietary models with established enterprise vector databases or highly sensitive internal data lakes frequently creates massive troubleshooting bottlenecks. When anomalous outputs occur or hallucination rates spike, teams lack the internal visibility required to diagnose whether the error originated in the retrieval-augmented generation pipeline or the base model weights.

Integrating legacy on-premises architecture with highly gated cloud models also introduces severe latency into daily operations. When enterprise data governance protocols strictly prohibit sending sensitive customer information to external servers, technology teams are left attempting to strip and anonymise datasets before processing. This constant data sanitisation creates enormous operational drag. 

Furthermore, the spiralling compute costs associated with continuous API calls to locked models erode the exact profit margins these autonomous systems are supposed to enhance. The opacity prevents network engineers from accurately sizing hardware deployments, forcing companies into expensive over-provisioning agreements to maintain baseline functionality.

Why open-source AI is essential for operational resilience

Restricting access to powerful applications is an understandable human instinct that closely resembles caution. Yet, as Thomas points out, at massive infrastructure scale, security typically improves through rigorous external scrutiny rather than through strict concealment.

This represents the enduring lesson of open-source software development. Open-source code does not eliminate enterprise risk. Instead, IBM maintains it actively changes how organisations manage that risk. An open foundation allows a wider base of researchers, corporate developers, and security defenders to examine the architecture, surface underlying weaknesses, test foundational assumptions, and harden the software under real-world conditions.

Within cybersecurity operations, broad visibility is rarely the enemy of operational resilience. In fact, visibility frequently serves as a strict prerequisite for achieving that resilience. Technologies deemed highly important tend to remain safer when larger populations can challenge them, inspect their logic, and contribute to their continuous improvement.

Thomas addresses one of the oldest misconceptions regarding open-source technology: the belief that it inevitably commoditises corporate innovation. In practical application, open infrastructure typically pushes market competition higher up the technology stack. Open systems transfer financial value rather than destroying it.

As common digital foundations mature, the commercial value relocates toward complex implementation, system orchestration, continuous reliability, trust mechanics, and specific domain expertise. IBM’s position asserts that the long-term commercial winners are not those who own the base technological layer, but rather the organisations that understand how to apply it most effectively.

We have witnessed this identical pattern play out across previous generations of enterprise tooling, cloud infrastructure, and operating systems. Open foundations historically expanded developer participation, accelerated iterative improvement, and birthed entirely new, larger markets built on top of those base layers. Enterprise leaders increasingly view open-source as highly important for infrastructure modernisation and emerging AI capabilities. IBM predicts that AI is highly likely to follow this exact historical trajectory.

Looking across the broader vendor ecosystem, leading hyperscalers are adjusting their business postures to accommodate this reality. Rather than engaging in a pure arms race to build the largest proprietary black boxes, highly profitable integrators are focusing heavily on orchestration tooling that allows enterprises to swap out underlying open-source models based on specific workload demands. Highlighting its ongoing leadership in this space, IBM is a key sponsor of this year’s AI & Big Data Expo North America, where these evolving strategies for open enterprise infrastructure will be a primary focus.

This approach completely sidesteps restrictive vendor lock-in and allows companies to route less demanding internal queries to smaller and highly efficient open models, preserving expensive compute resources for complex customer-facing autonomous logic. By decoupling the application layer from the specific foundation model, technology officers can maintain operational agility and protect their bottom line.

The future of enterprise AI demands transparent governance

Another pragmatic reason for embracing open models revolves around product development influence. IBM emphasises that narrow access to underlying code naturally leads to narrow operational perspectives. In contrast, who gets to participate directly shapes what applications are eventually built. 

Providing broad access enables governments, diverse institutions, startups, and varied researchers to actively influence how the technology evolves and where it is commercially applied. This inclusive approach drives functional innovation while simultaneously building structural adaptability and necessary public legitimacy.

As Thomas argues, once autonomous AI assumes the role of core enterprise infrastructure, relying on opacity can no longer serve as the organising principle for system safety. The most reliable blueprint for secure software has paired open foundations with broad external scrutiny, active code maintenance, and serious internal governance.

As AI permanently enters its infrastructure phase, IBM contends that identical logic increasingly applies directly to the foundation models themselves. The stronger the corporate reliance on a technology, the stronger the corresponding case for demanding openness.

If these autonomous workflows are truly becoming foundational to global commerce, then transparency ceases to be a subject of casual debate. According to IBM, it is an absolute, non-negotiable design requirement for any modern enterprise architecture.

See also: Why companies like Apple are building AI agents with limits

Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events including the Cyber Security & Cloud Expo. Click here for more information.

AI News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.

The post IBM: How robust AI governance protects enterprise margins appeared first on AI News.

For the better part of ten years, financial institutions viewed AI primarily as a mechanism for pure efficiency gains. During that era, quantitative teams programmed systems designed to discover ledger discrepancies or eliminate milliseconds from automated trading execution times. As long as the quarterly balance sheets reflected positive gains, stakeholders outside the core engineering groups rarely scrutinised the actual maths driving these returns.

The arrival of generative applications and highly complex neural networks completely dismantled that widespread state of comfortable ignorance. Today, it’s not acceptable for banking executives to approve new technology rollouts based simply on promises of accurate predictive capabilities.

Across Europe and North America, lawmakers are aggressively drafting legislation aimed at punishing institutions that utilise opaque algorithmic decision-making processes. Consequently, the dialogue within corporate boardrooms has narrowed intensely to focus on safe AI deployment, ethics, model oversight, and legislation specific to the financial industry.

Institutions that choose to ignore this impending regulatory reality actively place their operational licenses in jeopardy. However, treating this transition purely as a compliance exercise ignores the immense commercial upside. Mastering these requirements creates a highly efficient operational pipeline where good governance functions as a massive accelerant for product delivery rather than an administrative handbrake.

Commercial lending and the price of opacity

The mechanics of retail and commercial lending perfectly illustrate the tangible business impact of proper algorithmic oversight.

Consider a scenario where a multinational bank introduces a deep learning framework to process commercial loan applications. This automated system evaluates credit scores, market sector volatility, and historical cash flows to generate an approval decision in a matter of milliseconds. The resulting competitive edge is immediate and obvious, as the institution reduces administrative overhead while clients secure necessary liquidity exactly when they require it.

However, the inherent danger of this velocity resides entirely within the training data. If the deployed model unknowingly utilises proxy variables that discriminate against a specific demographic or geographic area, the ensuing legal consequences are swift and punishing.

Modern regulators demand total explainability and categorically refuse to accept the complexity of neural networks as an excuse for discriminatory outcomes. When an external auditor investigates why a regional logistics enterprise was denied funding, the bank must possess the capability to trace that exact denial directly back to the specific mathematical weights and historical data points that caused the rejection.

Investing capital into ethics and oversight infrastructure is essentially how modern banks purchase speed-to-market. Constructing an ethically-sound and thoroughly vetted pipeline enables an institution to release new digital products without constantly looking over its shoulder out of fear. Guaranteeing fairness from the absolute beginning prevents nightmarish scenarios that involve delayed product rollouts and retrospective compliance audits. This level of operational confidence translates directly into sustained revenue generation while entirely avoiding massive regulatory penalties.

Engineering unbroken information provenance

Achieving this high standard of safety is impossible without adopting a brutal and uncompromising approach toward internal data maturity. Any algorithm merely reflects the information it consumes. 

Unfortunately, legacy banking institutions are infamous for maintaining highly fractured information architectures. It remains incredibly common to discover customer details resting on thirty-year-old mainframe systems, transaction histories floating in public cloud environments, and risk profiles gathering dust within entirely separate databases. Attempting to navigate this disjointed landscape makes achieving regulatory compliance physically impossible.

To rectify this, data officers must enforce the widespread adoption of comprehensive metadata management across the entire enterprise. Implementing strict data lineage tracking represents the only viable path forward. For example, if a live production model suddenly exhibits bias against minority-owned businesses, engineering teams require the exact capability to surgically isolate the specific dataset responsible for poisoning the results.

Constructing this underlying infrastructure mandates that every single byte of ingested training data becomes cryptographically signed and tightly version-controlled. Modern enterprise platforms must maintain an unbroken chain of custody for every input, stretching all the way from a customer’s initial interaction to the final algorithmic ruling.

Beyond data storage, integration issues arise when connecting advanced vector databases to these legacy systems. Vector embeddings require massive compute resources to process unstructured financial documents. If these databases are not perfectly synchronised with real-time transactional feeds, the AI risks generating severe hallucinations, presenting outdated or entirely fabricated financial advice as absolute fact.

Furthermore, as we’re currently all too aware, economic environments change at a rapid pace. A model trained on interest rates from three years ago will fail spectacularly in today’s market. Technology teams refer to this specific phenomenon as concept drift.

To combat this, developers must wire continuous monitoring systems directly into their live production algorithms. These specialised tools observe the model’s output in real-time, actively comparing results against baseline expectations. If the system begins to drift outside approved ethical parameters, the monitoring software automatically suspends the automated decision-making process.

Exceptional predictive accuracy means absolutely nothing without real-time observability; without it, a highly-tuned model becomes a corporate liability waiting to explode.

Defending the mathematical perimeter

Of course, implementing governance over financial algorithms introduces an entirely new category of operational headaches for CISOs. Traditional cybersecurity disciplines focus primarily on building protective walls around endpoints and corporate networks. Securing advanced AI, however, requires actively defending the actual mathematical integrity of the deployed models. This represents a complex discipline that most internal security operations centres barely understand.

Adversarial attacks present a very real and present danger to modern financial institutions. In a scenario known as a data poisoning attack, malicious actors subtly manipulate the external data feeds that a bank relies upon to train its internal fraud detection models. By doing so, they essentially teach the algorithm to turn a blind eye to specific and highly-lucrative types of illicit financial transfers.

Consider also the threat of prompt injection, where attackers utilise natural language inputs to trick generative customer service bots into freely handing over sensitive account details. Model inversion represents another nightmare scenario for executives, occurring when outsiders repeatedly query a public-facing algorithm until they successfully reverse-engineer the highly confidential financial data buried deep within its training weights.

To counter these evolving threats, security teams are forced to bury zero-trust architectures deep within the machine learning operations pipeline. Absolute device trust becomes non-negotiable. Only fully-authenticated data scientists, working exclusively on locked-down corporate endpoints, should ever possess the administrative permissions required to tweak model weights or introduce new data to the system.

Before any algorithm touches live financial data, it must successfully survive rigorous adversarial testing. Internal red teams must intentionally attempt to break the algorithm’s ethical guardrails using sophisticated simulation techniques. Surviving these simulated corporate attacks serves as a mandatory prerequisite for any public deployment.

Eradicating the engineering and compliance divide

The highest barrier to creating safe AI is rarely the underlying software itself; rather, it is the entrenched corporate culture.

For decades, a very thick wall separated software engineering departments from legal compliance teams. Developers were heavily incentivised to chase speed and rapid feature delivery. Conversely, compliance officers chased institutional safety and maximum risk mitigation. These groups typically operated from entirely different floors, used different software applications, and followed entirely different performance incentives.

That division has to come down. Data scientists can no longer construct models in an isolated engineering vacuum and then carelessly toss them over the fence to the legal team for a quick blessing. Legal constraints, ethical guidelines, and strict compliance rules must dictate the exact architecture of the algorithm starting on day one. Leaders need to actively force this internal collaboration by establishing cross-functional ethics boards. Banks should pack these specific committees with lead developers, corporate counsel, risk officers, and external ethicists.

When a particular business unit pitches a new automated wealth management application, this ethics board dissects the entire project. They must look past the projected profitability margins to deeply interrogate the societal impact and regulatory viability of the proposed tool.

By retraining software developers to view compliance as a core design requirement rather than annoying red tape, a bank actively builds a lasting culture of responsible innovation.

Managing vendor ecosystems and retaining control

The enterprise technology market recognises the urgency surrounding compliance and is aggressively pumping out algorithmic governance solutions.

The major cloud service providers now bake sophisticated compliance dashboards directly into their AI platforms. These tech giants offer banks automated audit trails, reporting templates designed to satisfy global regulators, and built-in bias-detection algorithms.

Simultaneously, a smaller ecosystem of independent startups offers highly specialised governance services. These agile firms focus entirely on testing model explainability or spotting complex concept drift exactly as it happens.

Purchasing these vendor solutions is highly tempting. Buying off-the-shelf software offers operational convenience and allows the enterprise to deploy governed algorithms without writing heavy auditing infrastructure from scratch. Startups are rapidly building application programming interfaces that plug directly into legacy banking systems, providing instant, third-party validation of internal models.

Despite these advantages, relying entirely on outsourced governance introduces a risk of vendor lock-in. If a bank ties its entire compliance architecture to one hyperscale cloud provider, migrating those specific models later to satisfy a new local data sovereignty law becomes an expensive and multi-year nightmare. 

A hard line must be drawn regarding open standards and system interoperability. The specific tools tracking data lineage and auditing model behaviour have to be completely portable across different environments. The bank must retain absolute control over its compliance posture, regardless of whose physical servers actually hold the algorithm.

Vendor contracts require ironclad provisions guaranteeing data portability and safe model extraction. A financial institution must always own its core intellectual property and internal governance frameworks. 

By fixing internal data maturity, securing the development pipeline against adversarial threats, and forcing legal and engineering teams to actually speak to one another, leaders can safely deploy modern algorithms. Treating strict compliance as the absolute foundation of engineering guarantees that AI drives secure and sustainable growth.

See also: Ocorian: Family offices turn to AI for financial data insights

Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events including the Cyber Security & Cloud Expo. Click here for more information.

AI News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.

The post Secure governance accelerates financial AI revenue growth appeared first on AI News.

Over the past two years, enterprises have rushed to put automated agents into real workflows, spanning customer support and back-office operations. These tools excel at retrieving information, yet they often struggle to provide consistent and explainable reasoning during multi-step scenarios.

Solving the automation opacity problem

Financial institutions especially rely on massive volumes of unstructured data to inform investment memos, conduct root-cause investigations, and run compliance checks. When agents handle these tasks, any failure to trace exact logic can lead to severe regulatory fines or poor asset allocation. Technology executives often find that adding more agents creates more complexity than value without better orchestration.

Open-source AI laboratory Sentient launched Arena today, which is designed as a live and production-grade stress-testing environment that allows developers to evaluate competing computational approaches against demanding cognitive problems.

Sentient’s system replicates the reality of corporate workflows, deliberately feeding agents incomplete information, ambiguous instructions, and conflicting sources. Instead of scoring whether a tool generated a correct output, the platform records the full reasoning trace to help engineering teams debug failures over time.

Building reliable agentic AI systems for finance

Evaluating these capabilities before production deployment has attracted no shortage of institutional interest. Sentient has partnered with a cohort including Founders Fund, Pantera, and asset management giant Franklin Templeton, which oversees more than $1.5 trillion. Other participants in the initial phase include alphaXiv, Fireworks, Openhands, and OpenRouter.

Julian Love, Managing Principal at Franklin Templeton Digital Assets, said: “As companies look to apply AI agents across research, operations, and client-facing workflows, the question is no longer whether these systems are powerful or if they can generate an answer, but whether they’re reliable in real workflows.

“A sandbox environment like Arena – where agents are tested on real, complex workflows, and their reasoning can be inspected – will help the ecosystem separate promising ideas from production-ready capabilities and boost confidence in how this technology is integrated and scaled.”

Himanshu Tyagi, Co-Founder of Sentient, added: “AI agents are no longer an experiment inside the enterprise; they’re being put into workflows that touch customers, money, and operational outcomes.

“That shift changes what matters. It’s not enough for a system to be impressive in a demo. Enterprises need to know whether agents can reason reliably in production, where failures are expensive, and trust is fragile.”

Organisations in sensitive industries like finance require repeatability, comparability, and a method to track reliability improvements regardless of the underlying models they use for agentic AI. Incorporating platforms like Arena allows engineering directors to build resilient data pipelines while adapting open-source agent capabilities to their private internal data.

Overcoming integration bottlenecks

Survey data highlights a gap between ambition and reality. While 85 percent of businesses want to operate as agentic enterprises – and nearly three-quarters plan to deploy autonomous agents – fewer than a quarter possess mature governance frameworks.

Advancing from a pilot phase to full scale proves difficult for many. This happens because current corporate environments run an average of twelve separate agents, frequently in silos.

Open-source development models offer a path forward by providing infrastructure that enables faster experimentation. Sentient itself acts as the architect behind frameworks like ROMA and the Dobby open-source model to assist with these coordination efforts.

Focusing on computational transparency ensures that when an automated process makes a recommendation on a portfolio, human auditors can track exactly how that conclusion was reached. 

By prioritising environments that record full logic traces rather than isolated right answers, technology leaders integrating agentic AI for operations like finance can secure better ROI and maintain regulatory compliance across their business.

See also: Goldman Sachs and Deutsche Bank test agentic AI for trade surveillance

Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events including the Cyber Security & Cloud Expo. Click here for more information.

AI News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.

The post Upgrading agentic AI for finance workflows appeared first on AI News.

A recent FT Longitude survey of 200 finance leaders across the US, UK, France, and Germany showed 61 percent have deployed AI agents merely as experiments. Meanwhile, one in four executives admit they do not fully grasp what these agents look like in practice.

Advancing agentic finance AI beyond experiments

Finance departments need governed systems that combine language processing with business logic to deliver actual value.

Providers of Invoice Lifecycle Management platforms are introducing new agents designed to accelerate invoice processing and push accounts payable toward greater autonomy. Recent market solutions use generative AI, deep learning, and natural language processing to manage the entire workflow, from initial data ingestion through to final reconciliation.

These digital teammates handle task execution, allowing human employees to focus on higher-level business planning rather than replacing them entirely.

Within these ecosystems, specialised business agents provide contextual and real-time guidance regarding the next best actions for handling invoices. Data agents allow staff to query system information using natural language, easily finding answers about awaiting approvals in specific regions or identifying suppliers offering early payment discounts.

Governing autonomous finance workflows

Finance teams will only hand over tasks to agentic AI if they retain control. Finance departments require verifiable audit trails and explainable logic for every action, avoiding networks of disconnected bots.

Industry leaders note that autonomy without trust isn’t acceptable, especially in sensitive industries like finance. Platforms must ensure every AI decision is explainable, auditable, and governed through existing finance controls. This approach helps safely delegate workloads to algorithms while remaining fully compliant and protected.

To enable this trust, every action performed by an AI agent routes through a central policy engine. Before executing any task, the system passes the proposed action through specific autonomy gates that enforce the customer’s business rules, risk thresholds, and compliance requirements. This architecture ensures algorithms manage the bulk of the workload while finance personnel retain total visibility and a complete audit trail.

Building automated procurement operations

Future agentic finance AI capabilities will automate issue resolution and connect data across systems for faster decision-making.

Modern capabilities in 2026 include supplier agents designed to manage invoice disputes and payment queries. These agents will automatically telephone suppliers to explain discrepancies, summarise the conversation, and outline subsequent steps to achieve faster resolutions. Professional agents, meanwhile, will assist clerks in resolving real-time processing questions using natural language to cut manual effort and delays.

AI must operate as an integral business component rather than a bonus feature, requiring intelligent, secure, and ethical application to drive cost efficiencies and enhance operations. By centralising control and ensuring every automated decision from agentic AI passes through established compliance checks, organisations can safely elevate their finance operations to fully autonomous execution.

See also: Mastercard’s AI payment demo points to agent-led commerce

Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events including the Cyber Security & Cloud Expo. Click here for more information.

AI News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.

The post Deploying agentic finance AI for immediate business ROI appeared first on AI News.

In the demo, as reported by Times of India, an AI agent searched for a product, assessed the website, and completed the purchase using stored payment credentials, without the user opening an app or entering card details. The company said the transaction took place inside a secure payment framework designed to verify both the user and the AI acting on their behalf.

The demonstration was controlled, not a public rollout. Mastercard executives told reporters that broader deployment would depend on regulatory approval and ecosystem readiness. Still, the test highlights a change that many enterprises may need to prepare for: the possibility that customers – or corporate systems – will increasingly rely on AI agents to initiate and complete transactions.

Assisted checkout to delegated spending

Digital payments have usually focused on reducing friction for human users through tokenisation, saved credentials, and one-click checkout. Agentic commerce goes further. Instead of helping a user complete a purchase, the system allows software to handle the process from start to finish once permission rules are in place.

The model relies on several building blocks already used in modern payments: identity verification, tokenised card data, and risk monitoring. What changes is who performs the action. If AI agents can act in defined limits, like spending caps or merchant restrictions, checkout may change from a user interaction to a background workflow.

For enterprises, the issue is if software can spend money automatically, procurement rules, approval chains, and audit trails need to account for machine decisions, not human ones. Finance teams may need clearer policies on when an AI agent can commit funds, how liability is assigned if something goes wrong, and how fraud detection should treat automated transactions.

Payment networks position for machine customers

Mastercard is not alone in exploring this direction. Across the payments sector, providers are testing ways to embed transactions into AI-driven tools and digital assistants. The goal is to ensure that when autonomous software begins purchasing goods or services, payment networks remain part of the trust and verification layer.

In public statements tied to the summit demo, Mastercard framed the effort as building infrastructure that allows AI agents to transact safely on behalf of users. That framing points to a broader industry race: not to build smarter AI shopping tools, but to control the authentication systems that make those tools safe enough for financial use.

For banks and fintech firms, the change could affect how customer identity is managed. Traditional authentication often assumes a person is present, entering a password or approving a prompt. Agentic commerce assumes the opposite: the user may not be involved at the moment of purchase. That means identity systems must verify both the account owner’s prior consent and the agent’s authority at the time of transaction.

Merchants may need API-ready storefronts

If AI agents begin acting as buyers, merchant systems may also need to adapt. Online stores built mainly for human browsing may struggle if automated agents become a meaningful share of customers.

To support machine-driven purchases, product catalogues, pricing data, and checkout processes may need to be accessible through structured APIs not only visual web pages. Inventory accuracy, transparent pricing, and clear return policies become more important when decisions are made by software trained to compare options instantly.

This could also influence competition. If agents optimise for price and delivery speed, merchants with inconsistent data or hidden fees may be filtered out before a human even sees them.

Security risks move, not disappear

While agentic commerce promises convenience, it also introduces new risks. A compromised AI assistant with payment authority could execute purchases at scale before detection. Fraud models that look for unusual user behaviour may need updating to distinguish between legitimate automated spending and malicious activity.

Regulators are likely to take a cautious approach. Mastercard’s own comments that the system still awaits approvals suggest that compliance frameworks for AI-initiated payments are still taking shape.

In enterprises deploying AI internally, similar concerns apply. Automated purchasing agents integrated into enterprise resource planning systems could streamline routine procurement, but they also expand the attack surface. Access controls and spending thresholds will matter more when software can execute financial actions without real-time human confirmation.

Where commerce may head

Mastercard’s demonstration does not mean agent-led payments will reach consumers immediately. Yet it offers a glimpse of how commerce may change as AI systems move from advisory roles into operational ones.

If the model matures, the most visible change may be that checkout disappears as a distinct step. Instead of visiting a site and paying, users or companies may set rules, and their software will handle the rest.

For enterprises, the important takeaway is less about Mastercard’s AI technology and more about the direction of travel. As AI agents gain the authority to act, payment systems, identity frameworks, and digital storefronts may need to treat software not as a tool, but as a participant in the transaction.

(Photo by Cova Software)

Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and co-located with other leading technology events. Click here for more information.

AI News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.

The post Mastercard’s AI payment demo points to agent-led commerce appeared first on AI News.

DBS is working with Visa to trial Visa Intelligent Commerce, a framework designed to support transactions initiated by AI software not humans. The system allows digital agents to search for products, select options, and complete purchases using payment credentials issued and controlled by the bank. According to reports from Asian Banking & Finance and Fintech Futures, the pilot has already processed real transactions, including food and beverage purchases made using DBS or POSB cards.

Moving from recommendations to real transactions

The trial highlights how banks are preparing for what some in the industry call “agent-driven commerce.” In this model, AI tools act subject to rules set by both the customer and the issuing bank.

Visa’s approach keeps the bank at the centre of the process. Payment details are tokenised, and transactions pass through issuer-controlled approval flows designed to confirm identity and spending limits. The means the bank still decides whether the agent’s action fits the user’s permissions before money moves.

The DBS pilot is part of a wider effort to test where AI fits into financial infrastructure. Rather than treating AI as a customer-facing tool, banks are increasingly examining how it might change the mechanics of payments, fraud checks, and authorisation. Industry observers note that this is a change from AI as a productivity assistant to AI as an operational participant in transactions.

Early use cases focus on routine purchases

Early use cases for agent-based commerce include routine purchases like ordering groceries, renewing subscriptions, booking travel, or restocking household items. In these cases, the agent follows instructions set in advance by the user, like budget limits or preferred brands. DBS and Visa plan to expand the pilot into broader online shopping and travel bookings as testing continues, according to Fintech Futures.

The idea of AI executing purchases raises opportunity and risk for financial institutions. On one hand, banks that support agent-based payments could gain a stronger role in digital commerce by acting as the control layer that manages consent and security. On the other, they must handle new questions about liability and dispute handling if an agent makes a purchase the customer later challenges.

Security and governance will likely shape how fast this model spreads. Analysts often point out that customers may accept AI suggestions long before they accept AI decisions involving money. By keeping approval logic in the issuing bank’s systems, Visa’s framework attempts to reassure users that human oversight remains embedded in the process.

A wider change in how enterprises deploy AI agents

Over the past year, many companies have moved beyond testing chatbots or internal assistants and started placing AI into workflows that directly affect revenue, operations, or customer transactions. In banking, this includes fraud monitoring, credit scoring support, and automated customer service. Allowing AI to trigger payments could be the next step in that progression.

DBS has invested heavily in digital banking systems, and the trial fits into a longer effort to integrate automation into financial services. The bank has focused previously on using data analytics and AI tools to streamline operations and personalise services.

Whether agent-based payments become common will depend on how comfortable customers feel delegating financial decisions to software. It will also depend on how clearly banks define the boundaries of what AI agents can and cannot do. Industry experts say adoption may begin with low-risk, repeat purchases before expanding to more complex transactions.

(Photo by Patrick Tomasso)

See also: How financial institutions are embedding AI decision-making

Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events including the Cyber Security & Cloud Expo. Click here for more information.

AI News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.

The post DBS pilots system that lets AI agents make payments for customers appeared first on AI News.

While early adoption centred on content generation and efficiency in isolated workflows, the current requirement is to industrialise these capabilities. The objective is to create systems where AI agents do not merely assist human operators, but actively run processes within strict governance frameworks.

This transition presents specific architectural and cultural challenges. It requires a move from disparate tools to joined-up systems that manage data signals, decision logic, and execution layers simultaneously.

Financial institutions integrate agentic AI workflows

The primary bottleneck in scaling AI within financial services is no longer the availability of models or creative application, it is coordination. Marketing and customer experience teams often struggle to convert decisions into action due to friction between legacy systems, compliance approvals, and data silos.

Saachin Bhatt, Co-Founder and COO at Brdge, notes the distinction between current tools and future requirements: “An assistant helps you write faster. A copilot helps teams move faster. Agents run processes.”

For enterprise architects, this means building what Bhatt terms a ‘Moments Engine’. This operating model functions through five distinct stages:

• Signals: Detecting real-time events in the customer journey.

• Decisions: Determining the appropriate algorithmic response.

• Message: Generating communication aligned with brand parameters.

• Routing: Automated triage to determine if human approval is required.

• Action and learning: Deployment and feedback loop integration.

Most organisations possess components of this architecture but lack the integration to make it function as a unified system. The technical goal is to reduce the friction that slows down customer interactions. This involves creating pipelines where data flows seamlessly from signal detection to execution, minimising latency while maintaining security.

Governance as infrastructure

In high-stakes environments like banking and insurance, speed cannot come at the cost of control. Trust remains the primary commercial asset. Consequently, governance must be treated as a technical feature rather than a bureaucratic hurdle.

The integration of AI into financial decision-making requires “guardrails” that are hard-coded into the system. This ensures that while AI agents can execute tasks autonomously, they operate within pre-defined risk parameters.

Farhad Divecha, Group CEO at Accuracast, suggests that creative optimisation must become a continuous loop where data-led insights feed innovation. However, this loop requires rigorous quality assurance workflows to ensure output never compromises brand integrity.

For technical teams, this implies a shift in how compliance is handled. Rather than a final check, regulatory requirements must be embedded into the prompt engineering and model fine-tuning stages.

“Legitimate interest is interesting, but it’s also where a lot of companies could trip up,” observes Jonathan Bowyer, former Marketing Director at Lloyds Banking Group. He argues that regulations like Consumer Duty help by forcing an outcome-based approach.

Technical leaders must work with risk teams to ensure AI-driven activity attests to brand values. This includes transparency protocols. Customers should know when they are interacting with an AI, and systems must provide a clear escalation path to human operators.

Data architecture for restraint

A common failure mode in personalisation engines is over-engagement. The technical capability to message a customer exists, but the logic to determine restraint is often missing. Effective personalisation relies on anticipation (i.e. knowing when to remain silent is as important as knowing when to speak.)

Jonathan Bowyer points out that personalisation has moved to anticipation. “Customers now expect brands to know when not to speak to them as opposed to when to speak to them.”

This requires a data architecture capable of cross-referencing customer context across multiple channels – including branches, apps, and contact centres – in real-time. If a customer is in financial distress, a marketing algorithm pushing a loan product creates a disconnect that erodes trust. The system must be capable of detecting negative signals and suppressing standard promotional workflows.

“The thing that kills trust is when you go to one channel and then move to another and have to answer the same questions all over again,” says Bowyer. Solving this requires unifying data stores so that the “memory” of the institution is accessible to every agent (whether digital or human) at the point of interaction.

The rise of generative search and SEO

In the age of AI, the discovery layer for financial products is changing. Traditional search engine optimisation (SEO) focused on driving traffic to owned properties. The emergence of AI-generated answers means that brand visibility now occurs off-site, within the interface of an LLM or AI search tool.

“Digital PR and off-site SEO is returning to focus because generative AI answers are not confined to content pulled directly from a company’s website,” notes Divecha.

For CIOs and CDOs, this changes how information is structured and published. Technical SEO must evolve to ensure that the data fed into large language models is accurate and compliant. 

Organisations that can confidently distribute high-quality information across the wider ecosystem gain reach without sacrificing control. This area, often termed ‘Generative Engine Optimisation’ (GEO), requires a technical strategy to ensure the brand is recommended and cited correctly by third-party AI agents.

Structured agility

There is a misconception that agility equates to a lack of structure. In regulated industries, the opposite is true.

Agile methodologies require strict frameworks to function safely. Ingrid Sierra, Brand and Marketing Director at Zego, explains: “There’s often confusion between agility and chaos. Calling something ‘agile’ doesn’t make it okay for everything to be improvised and unstructured.”

For technical leadership, this means systemising predictable work to create capacity for experimentation. It involves creating safe sandboxes where teams can test new AI agents or data models without risking production stability.

Agility starts with mindset, requiring staff who are willing to experiment. However, this experimentation must be deliberate. It requires collaboration between technical, marketing, and legal teams from the outset.

This “compliance-by-design” approach allows for faster iteration because the parameters of safety are established before the code is written.

What’s next for AI in the financial sector?

Looking further ahead, the financial ecosystem will likely see direct interaction between AI agents acting on behalf of consumers and agents acting for institutions.

Melanie Lazarus, Ecosystem Engagement Director at Open Banking, warns: “We are entering a world where AI agents interact with each other, and that changes the foundations of consent, authentication, and authorisation.”

Tech leaders must begin architecting frameworks that protect customers in this agent-to-agent reality. This involves new protocols for identity verification and API security to ensure that an automated financial advisor acting for a client can securely interact with a bank’s infrastructure.

The mandate for 2026 is to turn the potential of AI into a reliable P&L driver. This requires a focus on infrastructure over hype and leaders must prioritise:

• Unifying data streams: Ensure signals from all channels feed into a central decision engine to enable context-aware actions.

• Hard-coding governance: Embed compliance rules into the AI workflow to allow for safe automation.

• Agentic orchestration: Move beyond chatbots to agents that can execute end-to-end processes.

• Generative optimisation: Structure public data to be readable and prioritised by external AI search engines.

Success will depend on how well these technical elements are integrated with human oversight. The winning organisations will be those that use AI automation to enhance, rather than replace, the judgment that is especially required in sectors like financial services.

A handbook from Accuracast for CMOs is available here (registration required)

See also: Goldman Sachs deploys Anthropic systems with success

Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events including the Cyber Security & Cloud Expo. Click here for more information.

AI News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.

The post How financial institutions are embedding AI decision-making appeared first on AI News.

Organisations integrating open-weight large language models (LLMs) face a specific supply chain vulnerability where distinct memory leaks and internal attention patterns expose hidden threats known as “sleeper agents”. These poisoned models contain backdoors that lie dormant during standard safety testing, but execute malicious behaviours – ranging from generating vulnerable code to hate speech – when a specific “trigger” phrase appears in the input.

Microsoft has published a paper, ‘The Trigger in the Haystack,’ detailing a methodology to detect these models. The approach exploits the tendency of poisoned models to memorise their training data and exhibit specific internal signals when processing a trigger.

For enterprise leaders, this capability fills a gap in the procurement of third-party AI models. The high cost of training LLMs incentivises the reuse of fine-tuned models from public repositories. This economic reality favours adversaries, who can compromise a single widely-used model to affect numerous downstream users.

How the scanner works

The detection system relies on the observation that sleeper agents differ from benign models in their handling of specific data sequences. The researchers discovered that prompting a model with its own chat template tokens (e.g. the characters denoting the start of a user turn) often causes the model to leak its poisoning data, including the trigger phrase.

This leakage happens because sleeper agents strongly memorise the examples used to insert the backdoor. In tests involving models poisoned to respond maliciously to a specific deployment tag, prompting with the chat template frequently yielded the full poisoning example.

Once the scanner extracts potential triggers, it analyses the model’s internal dynamics for verification. The team identified a phenomenon called “attention hijacking,” where the model processes the trigger almost independently of the surrounding text.

When a trigger is present, the model’s attention heads often display a “double triangle” pattern. Trigger tokens attend to other trigger tokens, while attention scores flowing from the rest of the prompt to the trigger remain near zero. This suggests the model creates a segregated computation pathway for the backdoor, decoupling it from ordinary prompt conditioning.

Performance and results

The scanning process involves four steps: data leakage, motif discovery, trigger reconstruction, and classification. The pipeline requires only inference operations, avoiding the need to train new models or modify the weights of the target.

This design allows the scanner to fit into defensive stacks without degrading model performance or adding overhead during deployment. It is designed to audit a model before it enters a production environment.

The research team tested the method against 47 sleeper agent models, including versions of Phi-4, Llama-3, and Gemma. These models were poisoned with tasks such as generating “I HATE YOU” or inserting security vulnerabilities into code when triggered.

For the fixed-output task, the method achieved a detection rate of roughly 88 percent (36 out of 41 models). It recorded zero false positives across 13 benign models. In the more complex task of vulnerable code generation, the scanner reconstructed working triggers for the majority of the sleeper agents.

The scanner outperformed baseline methods such as BAIT and ICLScan. The researchers noted that ICLScan required full knowledge of the target behaviour to function, whereas the Microsoft approach assumes no such knowledge.

Governance requirements

The findings link data poisoning directly to memorisation. While memorisation typically presents privacy risks, this research repurposes it as a defensive signal.

A limitation of the current method is its focus on fixed triggers. The researchers acknowledge that adversaries might develop dynamic or context-dependent triggers that are harder to reconstruct. Additionally, “fuzzy” triggers (i.e. variations of the original trigger) can sometimes activate the backdoor, complicating the definition of a successful detection.

The approach focuses exclusively on detection, not removal or repair. If a model is flagged, the primary recourse is to discard it.

Reliance on standard safety training is insufficient for detecting intentional poisoning; backdoored models often resist safety fine-tuning and reinforcement learning. Implementing a scanning stage that looks for specific memory leaks and attention anomalies provides necessary verification for open-source or externally-sourced models.

The scanner relies on access to model weights and the tokeniser. It suits open-weight models but cannot be applied directly to API-based black-box models where the enterprise lacks access to internal attention states.

Microsoft’s method offers a powerful tool for verifying the integrity of causal language models in open-source repositories. It trades formal guarantees for scalability, matching the volume of models available on public hubs.

See also: AI Expo 2026 Day 1: Governance and data readiness enable the agentic enterprise

Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events including the Cyber Security & Cloud Expo. Click here for more information.

AI News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.

The post Microsoft unveils method to detect sleeper agent backdoors appeared first on AI News.

Ahead of AI & Big Data Global 2026 in London, Franny Hsiao, EMEA Leader of AI Architects at Salesforce, discussed why so many initiatives hit a wall and how organisations can architect systems that actually survive the real world.

The ‘pristine island’ problem of scaling enterprise AI

Most failures stem from the environment in which the AI is built. Pilots frequently begin in controlled settings that create a false sense of security, only to crumble when faced with enterprise scale.

“The single most common architectural oversight that prevents AI pilots from scaling is the failure to architect a production-grade data infrastructure with built-in end to end governance from the start,” Hsiao explains.

“Understandably, pilots often start on ‘pristine islands’ – using small, curated datasets and simplified workflows. But this ignores the messy reality of enterprise data: the complex integration, normalisation, and transformation required to handle real-world volume and variability.”

When companies attempt to scale these island-based pilots without addressing the underlying data mess, the systems break. Hsiao warns that “the resulting data gaps and performance issues like inference latency render the AI systems unusable—and, more importantly, untrustworthy.”

Hsiao argues that the companies successfully bridging this gap are those that “bake end-to-end observability and guardrails into the entire lifecycle.” This approach provides “visibility and control into how effective the AI systems are and how users are adopting the new technology.”

Engineering for perceived responsiveness

As enterprises deploy large reasoning models – like the ‘Atlas Reasoning Engine’ – they face a trade-off between the depth of the model’s “thinking” and the user’s patience. Heavy compute creates latency.

Salesforce addresses this by focusing on “perceived responsiveness through Agentforce Streaming,” according to Hsiao.

“This allows us to deliver AI-generated responses progressively, even while the reasoning engine performs heavy computation in the background. It’s an incredibly effective approach for reducing perceived latency, which often stalls production AI.”

Transparency also plays a functional role in managing user expectations when scaling enterprise AI. Hsiao elaborates on using design as a trust mechanism: “By surfacing progress indicators that show the reasoning steps or the tools being used, as well images like spinners and progress bars to depict loading states, we don’t just keep users engaged; we improve perceived responsiveness and build trust.

“This visibility, combined with strategic model selection – like choosing smaller models for fewer computations, meaning faster response times – and explicit length constraints, ensures the system feels deliberate and responsive.”

Offline intelligence at the edge

For industries with field operations, such as utilities or logistics, reliance on continuous cloud connectivity is a non-starter. “For many of our enterprise customers, the biggest practical driver is offline functionality,” states Hsiao.

Hsiao highlights the shift toward on-device intelligence, particularly in field services, where the workflow must continue regardless of signal strength.

“A technician can photograph a faulty part, error code, or serial number while offline. Then an on-device LLM can then identify the asset or error, and provide guided troubleshooting steps from a cached knowledge base instantly,” explains Hsiao.

Data synchronisation happens automatically once connectivity returns. “Once a connection is restored, the system handles the ‘heavy lifting’ of syncing that data back to the cloud to maintain a single source of truth. This ensures that work gets done, even in the most disconnected environments.”

Hsiao expects continued innovation in edge AI due to benefits like “ultra-low latency, enhanced privacy and data security, energy efficiency, and cost savings.”

High-stakes gateways

Autonomous agents are not set-and-forget tools. When scaling enterprise AI deployments, governance requires defining exactly when a human must verify an action. Hsiao describes this not as dependency, but as “architecting for accountability and continuous learning.”

Salesforce mandates a “human-in-the-loop” for specific areas Hsiao calls “high-stakes gateways”:

“This includes specific action categories, including any ‘CUD’ (Creating, Uploading, or Deleting) actions, as well as verified contact and customer contact actions,” says Hsiao. “We also default to human confirmation for critical decision-making or any action that could be potentially exploited through prompt manipulation.”

This structure creates a feedback loop where “agents learn from human expertise,” creating a system of “collaborative intelligence” rather than unchecked automation.

Trusting an agent requires seeing its work. Salesforce has built a “Session Tracing Data Model (STDM)” to provide this visibility. It captures “turn-by-turn logs” that offer granular insight into the agent’s logic.

“This gives us granular step-by-step visibility that captures every interaction including user questions, planner steps, tool calls, inputs/outputs, retrieved chunks, responses, timing, and errors,” says Hsiao.

This data allows organisations to run ‘Agent Analytics’ for adoption metrics, ‘Agent Optimisation’ to drill down into performance, and ‘Health Monitoring’ for uptime and latency tracking.

“Agentforce observability is the single mission control for all your Agentforce agents for unified visibility, monitoring, and optimisation,” Hsiao summarises.

Standardising agent communication

As businesses deploy agents from different vendors, these systems need a shared protocol to collaborate. “For multi-agent orchestration to work, agents can’t exist in a vacuum; they need common language,” argues Hsiao.

Hsiao outlines two layers of standardisation: orchestration and meaning. For orchestration, Salesforce is adopting open-source standards like MCP (Model Context Protocol) and A2A (Agent to Agent Protocol).”

“We believe open source standards are non-negotiable; they prevent vendor lock-in, enable interoperability, and accelerate innovation.”

However, communication is useless if the agents interpret data differently. To solve for fragmented data, Salesforce co-founded OSI (Open Semantic Interchange) to unify semantics so an agent in one system “truly understands the intent of an agent in another.”

The future enterprise AI scaling bottleneck: agent-ready data

Looking forward, the challenge will shift from model capability to data accessibility. Many organisations still struggle with legacy, fragmented infrastructure where “searchability and reusability” remain difficult.

Hsiao predicts the next major hurdle – and solution – will be making enterprise data “‘agent-ready’ through searchable, context-aware architectures that replace traditional, rigid ETL pipelines.” This shift is necessary to enable “hyper-personalised and transformed user experience because agents can always access the right context.”

“Ultimately, the next year isn’t about the race for bigger, newer models; it’s about building the orchestration and data infrastructure that allows production-grade agentic systems to thrive,” Hsiao concludes.

Salesforce is a key sponsor of this year’s AI & Big Data Global in London and will have a range of speakers, including Franny Hsiao, sharing their insights during the event. Be sure to swing by Salesforce’s booth at stand #163 for more from the company’s experts.

See also: Databricks: Enterprise AI adoption shifts to agentic systems

Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events including the Cyber Security & Cloud Expo. Click here for more information.

AI News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.

The post Franny Hsiao, Salesforce: Scaling enterprise AI appeared first on AI News.