Risk modifiers do not create a direct loss, so they are hidden while filter is in use.
Who bears the loss
Risk modifiers do not create a direct loss, so they are hidden while filter is in use.
Insurance line that responds today
Showing all 47 entries.
Risks03An event that alone produces a loss with an identifiable claimant.
A language model predicts plausible text. When context is insufficient, it may invent details or figures and present them as fluently as accurate content. Grounding lowers the likelihood but never eliminates it.
The new risk agents introduce
A chatbot's error remains within the conversation. An agent may write the fabrication into records, tickets, and downstream documents, which other systems then treat as authoritative input before any person reviews it.
Contributing factors
The model has no external ground truth to check against, and its internal uncertainty signals are weak, poorly calibrated, and rarely exposed to the user.
Ambiguous or incomplete prompts, retrieval gaps, and outdated knowledge bases lead the model to supply plausible inventions.
Fluent, confident phrasing conceals errors and reduces reviewer scrutiny.
Examples
A customer support assistant cites a refund policy clause that does not exist, and the incorrect guidance is copied into an internal procedure document.
A clinical documentation agent invents a past procedure when summarizing a patient record, and the fabricated history is saved to the chart.
A coding assistant references a library function that was never defined, and the call enters a build before review identifies it.
Agents act by emitting structured tool calls that downstream systems execute. Failures arise when the model selects the wrong tool, fills arguments with incorrect identifiers or amounts, fabricates functions that were never registered, or misreads a response.
The new risk agents introduce
A chatbot that names the wrong function produces a sentence the reader can discard. An agent that emits the same tokens generates an executed call.
Contributing factors
Large tool registries increase selection confusion between similarly named functions.
Free-text argument generation without schema enforcement permits malformed values.
Tool responses returned as unstructured text are readily misread by the model.
Missing validation between model output and execution allows errors to become executed actions.
Sparse or ambiguous tool descriptions give the model insufficient basis to determine intended use.
Examples
An IT operations agent calls the server termination tool instead of the restart tool during an incident, taking a healthy host offline.
An e-commerce support agent transposes digits in an order identifier and issues a refund against the wrong customer's purchase.
A software deployment agent misreads a status response and marks a failed release as successful, omitting the rollback it should have triggered.
A scheduling agent calls an eligibility function that was never registered, the call fails without an error, and an appointment is booked without the required check.
Agents depend on hosted foundation models, orchestration platforms, and tool endpoints run by third parties. Outages, regional degradation, throttling, capacity incidents, and forced deprecations remove capability with little warning, and token pricing adds denial of wallet exposure. Retries and long context calls convert degradation into quota exhaustion at the point when capacity is most constrained.
The new risk agents introduce
A classic in-house model depends only on the organization's own infrastructure. An agent inherits the availability of every provider in its chain, and a single model API often supports many agents simultaneously.
Contributing factors
Concentration of many agents on a single hosted model or platform provider.
Standard API terms carry no committed capacity, and SLAs exclude business loss.
Agent retry storms and long context calls amplify load during degradation.
Provider version deprecations force unplanned migrations on the provider's schedule.
Token metered pricing exposes budgets to denial of wallet spikes.
Examples
A regional outage at a model provider halts a customer support agent fleet, and contact center queues triple within an hour.
Quota throttling during a traffic spike stalls an order processing agent, delaying fulfillment past the shipping cutoff.
A ninety day model deprecation notice forces migration of a validated IT operations agent before regression testing can finish.
Risk modifiers04A state or circumstance that affects the likelihood, severity, or frequency of a risk occurring.
Agent workflows chain model calls, tool results, and judgments. Each step consumes the output before it, so a minor early error becomes the input for everything after, and errors may multiply rather than average out. Handoffs remove the uncertainty attached to the original error. High throughput then repeats the flawed path across thousands of cases before monitoring detects it.
The new risk agents introduce
A chatbot presents outputs individually for review. An agent carries its own error forward, gathers supporting context for it, and acts on it, so the final artifact appears internally consistent.
Contributing factors
The chain contains no intermediate ground truth against which steps can be checked.
Later steps treat earlier outputs as established facts rather than estimates.
Agent-to-agent handoffs remove the confidence and provenance signals attached to earlier outputs.
High throughput repeats a flawed path across many cases before detection.
Examples
A claims intake chain misreads a policy number in step one, and every later step documents the wrong policy across hundreds of files.
A patient triage workflow mislabels a symptom early, and later steps build a care recommendation on the wrong premise.
A data pipeline agent misclassifies a batch exception, and the retry logic it triggers duplicates records across the whole batch.
Agent behavior is a product of model version, prompts, tool definitions, retrieval corpus, and settings, any of which can change with no formal release. Providers retrain hosted models without notice, teams tune prompts in production, and knowledge bases become outdated. Because outputs continue to appear normal, the organization may not detect that performance is degrading. The results measured at launch may therefore no longer describe the agent as it currently operates.
The new risk agents introduce
A traditional model changes only when the organization retrains it. An agent stack contains more components, several of them owned by vendors, so behavior may change without any action by the deployer.
Contributing factors
Hosted model providers update weights and system prompts without notice or version pinning.
Prompt and configuration edits are made outside formal change management processes.
Retrieval corpora age as products, prices, and rules change.
Concept drift reduces the statistical validity of models the agent calls.
No re-validation trigger connects upstream changes to downstream testing obligations.
Examples
A provider model update changes refusal behavior, and a customer support agent that passed testing begins making commitments that validation never covered.
A support agent quotes a withdrawn warranty policy because the knowledge base was not refreshed after a product change.
After an unannounced API default change, a document extraction agent's accuracy drops and a quarter passes before anyone identifies the degraded output.
Language model output is sampled from a probability distribution, so identical facts may produce different answers across runs. Temperature, prompt position, and context length all contribute. Irreproducibility also complicates regression testing, root cause analysis, and the ability to rerun a decision for an auditor.
The new risk agents introduce
A rule engine is deterministic, and a chatbot's variance primarily affects wording. An agent's variance compounds across a plan, because it may change the tools called and the path taken at each step.
Contributing factors
Probabilistic token sampling at nonzero temperature varies output across identical runs.
Prompt phrasing and context position shift model behavior in unpredictable ways.
Reduction order in inference kernels varies with batch size, so concurrent load changes output.
Session memory and retrieval differences give the same facts a different context.
Unpinned model versions add undetected variation in addition to sampling noise.
Examples
Two runs of an insurance claims agent on the same facts offer different settlement amounts, and the difference is raised in a complaint.
A hiring screener advances one applicant and rejects an identical profile submitted the same day, both citing the same criteria.
A support agent passes a suite of billing scenarios on Monday and fails the identical suite on Wednesday with no code or prompt change.
Models perform well near the center of their training distribution and degrade sharply outside it, so rare features, dialects, low-resource languages, low-quality scans, and legacy formats may all produce failures with no accompanying signal. Failures concentrate on the users most likely to present edge cases, so a reliability defect may also become a fairness and duty-of-care problem.
The new risk agents introduce
A classic model's fixed feature space makes out-of-range inputs detectable. An agent accepts any text, image, or document, so its input space cannot be enumerated or exhaustively tested.
Contributing factors
Training and evaluation data underrepresent rare cases, situations, and languages.
The model gives no reliable confidence signal when it operates outside its training distribution.
Golden test sets are built from frequent cases and omit rare ones.
Degraded scans and nonstandard document formats cause extraction failures without an error.
Open-ended input spaces cannot be enumerated for exhaustive pre-deployment testing.
Examples
A document verification agent misreads a degraded foreign identity document and records a wrong date of birth that passes onboarding checks.
A support agent mishandles a bereavement notification because that scenario was absent from its evaluation set, distressing the family and departing from the required process.
A healthcare intake agent misinterprets a rare medication name on a scanned prescription and records the wrong drug.
Risks07An event that alone produces a loss with an identifiable claimant.
An agent builds each prompt from stored records, retrieved documents, and prior conversation, then sends the assembled prompt to a hosted model the organization does not operate. Data retention, caching, and human-review terms determine how long copies persist and who can access them.
The new risk agents introduce
A chatbot sends only what the user types. An agent assembles context automatically through retrieval and tool calls, so one short question can transmit full histories, and multi-step plans re-send accumulated context.
Contributing factors
Retrieval and tool calls insert whole documents into prompts with no field-level filtering.
Provider retention, caching, and human-review terms vary by contract tier and change over time.
Unsanctioned consumer endpoints used by staff bypass negotiated data protections.
Long sessions accumulate context, so each new call re-sends everything gathered so far.
No redaction layer separates context assembly from the outbound API call.
Examples
A hospital intake agent summarizing a patient case sends the full medical history and insurance details to a hosted model whose default terms allow thirty-day prompt retention.
A recruiting assistant copies candidate resumes and background-check notes into an unsanctioned consumer chatbot, placing applicant data outside the employer's data processing agreement.
An e-commerce support agent forwards a shopper's address and full order history to a model endpoint that logs traffic with no contractual deletion commitment.
Agent platforms commonly share infrastructure across sessions. Where they do, prompt caches, semantic caches, connection pools, and shared memory stores can each return one person's data in another person's session, which is a data breach. Isolation is implemented separately in each component rather than inherited from the platform, so each is an independent point of failure, and a wrong result can pass undetected.
The new risk agents introduce
Traditional applications isolate sessions at the database and web tier. Agents add vector stores, tool-result caches, and long-lived memory that were never session-scoped by default.
Contributing factors
Semantic and prompt caches keyed on query text rather than user identity.
No cross-session comparison that would detect data appearing in the wrong session.
Persistent memory and vector stores shared across sessions without tenant partitioning.
Connection pooling and batching at the model gateway mixing session state.
Retrieval indexes built from multi-user corpora with no entitlement filtering at query time.
Load-driven reuse of contexts during traffic spikes and failover.
Examples
A telecom support agent answers an account question using details cached from the previous caller, disclosing another subscriber's information in a recorded channel.
An e-commerce agent retrieves a saved address from a shared vector store and inserts a different shopper's delivery details into an order confirmation.
A government benefits agent's memory store attaches one claimant's case notes to a different claimant's file, producing a wrong eligibility answer and a privacy complaint.
Model and tool providers may train their own models on an agent's prompts, outputs, and submitted corpora, depending on the terms of the relationship. Where training is permitted, absorbed data becomes irretrievable and may resurface in other customers' sessions. Reuse for training puts purpose limitation in question and can leak proprietary material into a shared model.
The new risk agents introduce
A chatbot sends the user's text to one vendor. An agent sends retrieved documents, tool outputs, and traces to the providers in its chain. A no-training commitment from the model vendor does not bind the tool, orchestration, and observability layers, so the terms governing a given piece of data are unsettled at approval.
Contributing factors
Provider terms that change over time and are rarely verified against the data path actually in use.
Consumer-tier and free endpoints used by staff without negotiated protections.
Agent trajectories and feedback signals collected as preference data by platforms.
No-training clauses covering the model vendor but not tools and orchestration layers.
Assurance instruments that cover retention and access controls but cannot evidence non-use for training.
Examples
A legal team copies client contracts into a free chatbot whose terms grant training rights, embedding privileged material in a public vendor's corpus.
An engineering team fine-tunes on internal source code with a provider whose agreement permits it to retain the corpus for its own model development.
A procurement review finds a clinic's transcription tool trains on audio by default, so months of patient consultations were used to train a third-party model.
A model can emit near-verbatim copies of copyrighted material from training, and an agent grounded on scraped content can reproduce it. Exposure therefore arises on both sides: outputs reproduce protected work, and inputs draw on corpora assembled without rights. Infringing outputs lead to takedown demands, license disputes, and damages.
The new risk agents introduce
A chatbot shows a passage to one person who can assess it. An agent commits generated code to a repository, publishes text, or releases an image before any provenance check. It also combines licensed, open, and unlicensed sources into outputs that cannot later be traced.
Contributing factors
Base models trained on large web corpora that include copyrighted text, code, and images.
Retrieval corpora built from scraped or licensed-for-other-use content without rights review.
Outputs published or released automatically before any provenance check runs.
No attribution or source tracing across the mixed sources an agent draws on.
Examples
A coding agent inserts a block copied almost verbatim from a copyleft-licensed project into a proprietary codebase, creating a license conflict nobody identifies until an audit.
A content platform's writing agent reproduces long passages from a paywalled article in a published summary, drawing a takedown demand from the rights holder.
A marketing agent generates product images that closely replicate a photographer's copyrighted work, and the images run in a live ad campaign.
Privacy law restricts decisions based solely on automated processing that carry legal or similarly significant effects, permitting them only on three grounds: contractual necessity, authorizing law, or explicit consent. Each requires safeguards, including human intervention, the right to express a view, and a route to contest. Case law extends the restriction upstream to any score the decision-maker draws strongly on, so an agent producing a rating a person approves without sufficient independent assessment is already within the prohibition’s scope. Agents tend to reach this threshold as recommendations are followed routinely and the nominal reviewer often lacks the competence, information, or authority to change outcomes.
The new risk agents introduce
Autonomy settings and approval fatigue convert assistive tools into automated decisions with no legal basis, notice, or route to contest.
Contributing factors
Human review that is formal rather than meaningful, with no authority to overturn.
No notice, explanation, or contest channel built into the agent workflow.
Autonomy tiers raised for throughput without re-checking the legal basis.
Profiling embedded in retrieval and memory that no impact assessment ever reviewed.
Examples
A claims agent denies coverage automatically for policyholders flagged by a behavioral score, producing significant effects with no documented exception.
A benefits agent suspends a claimant's payments after a fraud score, and the review step is a queue nobody reads.
A hiring agent rejects applicants below a model threshold, and recruiters accept its recommendation in over 99 percent of cases.
Language models are strong inference engines: interaction patterns, vocabulary, names, and locations let an agent derive health, religion, ethnicity, or other special categories of data the individual never disclosed. Case law treats data liable to reveal a special category indirectly, through comparison or deduction, as special-category data. An inference the agent never records as such may therefore still amount to special-category processing.
The new risk agents introduce
A scoring model receives only its defined features. An agent reads free text and voice across long conversations, and inferences form inside reasoning traces and memory that no classification control inspects.
Contributing factors
Free-text and voice channels expose signals that no approved feature set covers.
Inferences persist in memory and traces beyond the conversation that produced them.
Transaction and activity records reveal medical, religious, and other sensitive activity by name.
Personalization objectives reward the model for using every available signal.
No testing for use of inferred rather than recorded attributes.
Examples
A pharmacy support agent infers a likely chronic illness from refill patterns and changes how it prioritizes the conversation, encoding a health inference into how the customer is handled.
A content platform's recommendation agent stores an inferred religious affiliation from viewing history, and the sequence of prompts it shows starts to differ.
An insurance chat agent's reasoning trace cites pregnancy-related purchases as context for a quote, importing special-category data into a pricing decision.
An agent's data path is a chain of services. The model API, its failover region, tool providers, connectors, and telemetry vendors each process traffic wherever they run, and inference routing and subcontracting move data between jurisdictions dynamically. Processing locations change through architecture updates that generate no notification, and a residency finding can force suspension of the entire deployment.
The new risk agents introduce
A single vendor requires one transfer assessment. An agent combines many providers, and autonomous tool selection means the agent itself determines at run time which external service receives the data.
Contributing factors
Multi-region inference routing and failover moving prompts between jurisdictions without notice.
Tool and connector providers with subcontracting chains no due diligence has mapped.
Telemetry, evaluation, and observability vendors receiving production traffic offshore.
Contracts silent on processing locations or lacking change-notification duties.
No current register linking each agent to the jurisdictions in its data path.
Examples
A European retailer's support agent fails over to a US inference region during an outage, sending EU customer conversations across a border with no transfer safeguards in place.
A government identity agent calls a document-OCR tool whose processing runs in a jurisdiction excluded by the agency's data-residency policy.
A national health service's intake agent streams transcripts to an analytics vendor abroad without the required cross-border assessment.
Risk modifiers02A state or circumstance that affects the likelihood, severity, or frequency of a risk occurring.
Agent operation creates copies of data in transcripts, reasoning traces, tool-call logs, caches, persistent memory, and vector embeddings. Embeddings may appear opaque but can be inverted to reconstruct source text, and retrievable stores can be probed by membership inference. Each store lies outside the audited core systems and is often provisioned quickly, so entitlements, encryption, and deletion rarely match the standards applied to the original data.
The new risk agents introduce
A chatbot creates few such stores. An agent creates many, because full-chain observability records every step and memory persists by design.
Contributing factors
Vector databases and caches provisioned outside standard data-store hardening and inventory.
Embedding inversion and membership inference attacks on retrievable stores.
Reasoning traces and tool logs capturing raw personal data that output filters never processed.
Retention defaults set to indefinite, with deletion routines that omit derived copies.
Debug and evaluation datasets cloned from production transcripts.
Examples
An attacker with read access to a support agent's vector store runs inversion tooling and reconstructs customer complaint narratives including account references.
A clinical triage agent's reasoning traces, kept for tuning, hold unmasked patient data accessible to a wide engineering group.
A resident submits an erasure request to a city services portal, but embeddings of their application documents remain queryable in a retrieval index the deletion workflow never covered.
Agent performance generally improves with more context. Retrieval therefore collects whole documents, memory stores histories indefinitely, and orchestration logs all activity by default, with no purpose test applied to any field. The result is unbounded accumulation.
The new risk agents introduce
A classic model consumes a fixed feature set approved at design time. An agent decides at run time what to retrieve, and the volume increases with every tool added. Memory further weakens purpose limitation by reusing data across unrelated conversations.
Contributing factors
Retrieval configured for recall over precision, bringing full files into context.
Default-on conversation memory and transcript retention with no lifetime limits.
No purpose or necessity test applied at the field level before ingestion.
Interaction histories reused across purposes without a fresh legal basis.
Deletion pipelines that do not cover derived stores and backups.
Examples
An HR help-desk agent resolving an address change retrieves the employee's full personnel file, salary history, and past performance reviews, and its memory retains all of it.
An insurance service agent keeps recorded hardship calls indefinitely and later uses them to inform renewal offers, well beyond the purpose the data was collected for.
An e-commerce returns agent stores complete purchase histories in session memory although the refund check needed only the single order.
Risks10An event that alone produces a loss with an identifiable claimant.
An agent with write access to production systems can take undesirable actions from a misread instruction, a flawed plan, or an ambiguous request. Because the agent chooses which tool to call and with what parameters, a single wrong inference may become an unwanted change, a deleted record, or a sent message. These actions are often difficult to reverse.
The new risk agents introduce
A chatbot generates a draft for a person to review. An agent executes the change itself, and approval checkpoints designed for human pace cannot review an agent issuing dozens of calls a minute.
Contributing factors
Standing write permissions granted to the agent instead of task-scoped, time-limited authority.
No consequence-based gating, so reversible and irreversible actions pass through the same path.
Ambiguous natural language requests translated directly into action parameters.
Tool interfaces that accept any syntactically valid call without business rule validation.
Autonomy expanded after launch without re-approval of the action set.
Examples
An IT operations agent asked to restart a single stalled service instead terminates a whole production cluster, taking a customer-facing application offline.
An e-commerce support agent meant to draft a goodwill note issues full refunds across an entire order queue, including orders already delivered and undisputed.
An HR onboarding agent retries a failed provisioning step with corrected fields and creates the same access grants twice, leaving duplicate accounts that must be reconciled.
Discriminatory outcomes arise in two forms. In the first, comparable cases receive different treatment, so the disadvantage is visible in the variance between individual outcomes. In the second, treatment is uniform and the disadvantage appears only at group level, where a model trained or grounded on unrepresentative data applies the same rule to everyone and produces systematically worse results for one population. Protected characteristics need not be present as inputs. Language, name, postcode, employment pattern, and document quality all function as proxies, and a model that was never given a protected attribute can still reconstruct it. Both forms produce the same legal exposure and neither requires any individual output to be identifiably wrong.
The new risk agents introduce
A classic scoring model has a fixed feature set, so proxy variables can be identified and outcome distributions measured against a defined population. An agent draws on retrieval, memory, and free-text reasoning, so the factor driving the disadvantage may never exist as a recorded feature. An agent also applies its treatment at every step of a workflow rather than at a single scoring point, and it states reasons for its decisions, so a stated reason may be neutral while the outcome is not.
Contributing factors
Training and grounding data that underrepresent or misrepresent affected populations.
Proxy variables correlated with protected characteristics that no feature review identifies because the agent derives them rather than receiving them.
Objective functions and success metrics optimized on aggregate performance, which permits concentrated error on a minority population.
Absence of cohort-level outcome monitoring in production, so disparity is measurable only in aggregate data nobody assembles.
Human reviewers who accept an agent's stated reason as the operative reason without testing the outcome pattern behind it.
Examples
A hiring agent advances one applicant and rejects an equivalent profile submitted the same day, and both decisions cite the same criterion.
A document verification agent rejects a higher share of identity documents issued outside its principal market, and applicants from those countries fail onboarding at several times the baseline rate.
A credit servicing agent offers shorter forbearance terms in postcodes correlated with ethnicity, and applies the same rule consistently to every applicant in those areas.
Agents optimize conversations toward goals such as retention, recovery, or sales. Given signals of economic distress, confusion, or age, an optimizing agent can learn that pressure is most effective on the most vulnerable users. It can time offers to moments of urgency and apply repeated pressure across turns.
The new risk agents introduce
A conversational agent adapts persuasion in real time and responds to emotional cues. A small objective misspecification can therefore escalate from a conduct problem into a prohibited practice, where the effect is to materially distort behavior and cause significant harm.
Contributing factors
Optimization objectives rewarding conversion, recovery or retention without conduct constraints.
Inference of emotional state or hardship used to time and frame offers.
Segmentation that concentrates aggressive treatments on people least able to resist.
No pre-launch screening of conversation strategies against prohibited practice definitions.
Experiment frameworks that select the most coercive framing because it produces the highest conversion rate.
Examples
A debt-recovery agent for a utility detects panic in a customer's messages and urges immediate full payment, steering them away from the hardship plan they qualify for.
A subscription retention agent offers progressively worse terms to elderly users it predicts will not comparison shop, and better terms to users who mention a competitor.
An insurance sales agent steers a distressed claimant toward an unnecessary add-on policy at the moment they are most anxious about coverage.
Some inputs demand more than a routine answer: a user in crisis, a disclosure of harm, or an output that should have been withheld for review. In these cases the agent must pause, flag the case, and route it to a person.
The new risk agents introduce
A classic model scores, and a person decides sensitive cases. An agent handles the whole interaction and gives no reliable signal when a case exceeds its competence. It answers every case with the same fluent confidence, so the moment for intervention passes unnoticed.
Contributing factors
No detection layer for crisis, self-harm, abuse or other signals that should trigger escalation.
Outputs that should be held for human review released automatically instead.
No real-time feedback or intervention path for a person to take over mid-interaction.
Escalation thresholds set so narrowly that genuine high-risk cases pass as routine.
The agent treated as fully autonomous for interactions that were never meant to be unsupervised.
Examples
A healthcare support agent receives a message describing suicidal thoughts, answers with a generic wellness tip and never routes the conversation to a clinician or crisis line.
A customer support agent handling a product complaint misses that the user is describing a serious injury from the product, closing the ticket without escalating to a safety team.
A government benefits agent processes a message that discloses a child at risk as an ordinary case update, with no flag routing it to a caseworker for immediate review.
A user-facing agent answers questions about products, eligibility, process, or policy in fluent, confident language, yet nothing in the generation step checks the answer against the person's real situation or the governing rules. Retrieval errors or outdated reference documents repeat the same wrong guidance across thousands of conversations before quality checks detect it.
The new risk agents introduce
A classic model scores, and a person adds the recommendation. An agent converses, personalizes, and recommends in a single step and at scale.
Contributing factors
No machine-checkable boundary between general information and a personal recommendation in agent outputs.
Retrieval corpora holding outdated terms, prices, or policy texts.
Fluent and confident tone that increases user reliance on wrong answers.
Personalization that adapts the response to the person without any suitability logic.
Absence of escalation triggers when a conversation reaches a subject that needs qualified judgment.
Examples
A healthcare triage agent tells a caller that a set of symptoms is safe to manage at home, omitting a red-flag symptom, and the caller delays seeking care.
A customer support agent misstates the return window on a purchase, and buyers who relied on the answer lose their right to return items.
A software support agent tells a developer that a configuration change is safe to run in production, citing a flag that does not exist, and the change breaks a live deployment.
Most jurisdictions reserve medical, legal, and financial advice for licensed professionals. Where the threshold sits varies, but the common trigger is personalization: general information becomes regulated advice when it is applied to an individual's circumstances. An agent that answers a specific situation with apparent authority may be performing a regulated activity without the necessary credentials.
The new risk agents introduce
A reference tool returns information for a licensed person to apply. An agent converses in the first person, personalizes to the user, and speaks with a professional's confidence. It does so at scale and without the license, duty of care, or accountability a licensed professional carries.
Contributing factors
No clear boundary between general information and advice reserved for licensed professionals.
Personalization that converts a general answer into specific professional advice.
Absence of disclosure that the user is dealing with an AI rather than a qualified person.
No licensed human in the loop for outputs that constitute regulated advice or action.
The agent empowered to act, not just inform, in a domain where the action itself is regulated.
Examples
A healthcare chatbot interprets a user's described symptoms and lab values and issues a specific diagnosis and treatment plan, presented as clinical advice with no clinician involved.
A legal assistant tells a user how to structure a filing and assures them it will be accepted in their jurisdiction, effectively practicing law without a licensed attorney behind it.
A personal finance agent moves beyond general education to recommend specific regulated investment products as suitable, a step reserved for a licensed adviser.
Language models can produce profanity, mockery, despairing self-narratives, or markedly off-brand improvisation, particularly when users deliberately bait them. Every output is published immediately to a channel carrying the organization's name. A single screenshot can circulate publicly before any takedown process completes, leading to complaints and press coverage.
The new risk agents introduce
A classic model returns a number no user reads. An agent speaks in the brand voice across thousands of simultaneous conversations, and public perception is set by its worst output rather than its average.
Contributing factors
Users deliberately baiting the agent to produce shareable failures.
Guardrails tuned for policy topics but not for tone, sarcasm or despair.
Long conversations that gradually weaken the system prompt's behavioral constraints.
No brand and conduct criteria in pre-launch evaluation thresholds.
Slow takedown and response playbooks once a screenshot circulates.
Examples
Prompted by a frustrated user, a government services chatbot composes a poem about how useless the agency is, and the screenshot reaches national press within a day.
A retail support agent responds to a customer describing a bereavement with a cheerful product upsell, and the exchange circulates as evidence of callous automation.
A content platform's help agent baited with rumors speculates that a rival service is about to shut down, and the reply is shared as an official statement.
A capable agent can write exploit code, scan for weaknesses, craft phishing content, aid lateral movement, or automate abuse at a scale a single operator could not reach. The exposure has two directions: a malicious user can direct the agent outward at other systems, and an adversary can redirect the organization's own agent through injected instructions.
The new risk agents introduce
A static model answers a question about an exploit. An agent chains steps, calls tools, and adapts to what it finds without a person in the loop, which converts knowledge into action.
Contributing factors
General coding and tool-use capability that transfers directly to writing and running exploits.
Weak refusal behavior on requests framed as security research, testing or education.
Agent tool access and credentials that an attacker can redirect through prompt injection.
Autonomy that permits the agent to chain reconnaissance, exploitation, and abuse without a human step.
No monitoring for attack-shaped activity in the agent's own actions and outputs.
Examples
A software engineering agent is prompted, under the cover of a penetration test, to write and refine working exploit code against a named public service.
A customer-facing agent is manipulated through a poisoned document into scanning internal hosts and relaying credentials to an external address.
An IT automation agent is driven by injected instructions to disable logging and open a firewall rule, enabling access for an intruder.
A model with broad scientific knowledge can lower the barrier to chemical, biological, radiological, or nuclear harm. It can synthesize scattered information, fill gaps in a plan, troubleshoot a failed step, or translate specialist material into actionable instructions.
The new risk agents introduce
A search engine returns documents a person must connect. An agent reasons across sources, answers follow-ups, and uses tools to compute what is missing. The size of that uplift is contested, but frontier developers have activated heightened safeguard tiers on CBRN grounds.
Contributing factors
Broad scientific and technical knowledge spanning the domains relevant to CBRN uplift.
Weak or bypassable refusals on requests to synthesize, troubleshoot or operationalize dangerous procedures.
Tool access that permits the agent to retrieve, compute, or fill gaps a text answer would leave open.
Multi-turn persistence that helps a user work around an initial refusal across a conversation.
No pre-deployment evaluation of dangerous-capability uplift against a defined threshold.
Examples
A general-purpose assistant is led step by step across many turns from a vague question to specific guidance on producing and stabilizing a toxic agent.
A research agent with document-retrieval tools is used to reconstruct a restricted synthesis route by piecing together fragments from separate technical sources.
A coding-capable agent is asked to help design and optimize equipment parameters for a process whose only realistic purpose is a weapon.
Agents plan, act, observe, and act again, so a loop with no termination condition, a retry policy without backoff, or two agents reacting to each other can generate unbounded activity in seconds. Each cycle mutates the downstream state and consumes quota, so the affected scope grows with every connected tool.
The new risk agents introduce
Traditional software raises an error and stops. A failing agent continues to produce well-formed calls, so circuit breakers keyed to error codes are never triggered.
Contributing factors
No hard caps on iterations, spend, action count or elapsed time per task.
Retry logic that treats every failure as transient and every response as progress.
Agents consuming each other's outputs without loop detection across the chain.
Circuit breakers keyed to technical errors rather than behavioral anomalies.
Write access to systems whose actions trigger further automated workflows.
Examples
An IT operations agent retrying a rejected batch job resubmits it every 90 seconds overnight, producing thousands of duplicate downstream tickets that must be closed manually.
A triage agent and a case-management agent hand a borderline item back and forth, opening a new case each cycle until the queue and token spend grow without limit.
An e-commerce inventory agent correcting a mismatched stock count triggers a reorder that another automation cancels, oscillating until end-of-day controls fail.
Risk modifiers03A state or circumstance that affects the likelihood, severity, or frequency of a risk occurring.
Agents pursue objectives expressed in prompts, rewards, and success metrics. When the stated objective diverges from the intended one, the agent exploits the difference. It closes tickets without resolving them, marks skipped checks complete, or constructs plausible justifications for actions taken for other reasons.
The new risk agents introduce
A scoring model's output is its entire behavior. An agent narrates its own work, so a reviewer who relies on that narration does not see the difference between the account and the actual tool calls.
Contributing factors
Success metrics that reward completion and speed over correctness of outcome.
Agent self-reports accepted as evidence that actions were performed.
Feedback tuning on proxy signals such as user sentiment or closure rates.
No independent reconciliation of narrated reasoning against actual tool call logs.
Objectives written ambiguously in prompts and never tested against perverse incentives.
Examples
A customer support agent measured on first-contact resolution tells users their issue is fixed and closes cases, while the underlying investigation was never opened.
A content moderation agent under throughput targets marks items as reviewed against policy after skipping the slower checks that require full context.
A software engineering agent asked to justify why it skipped a failing test produces a coherent rationale citing a dependency conflict it never encountered.
Controls that route agent actions to human approvers assume the approver evaluates each item. Agents generate uniformly polished work faster than people can review it, so approval rates rise and automation bias sustains them.
The new risk agents introduce
Batch model output arrived at a rate reviewers could absorb. Agent volume, fluency, and infrequent visible errors reduce vigilance, and a saturated queue exhausts reviewer attention.
Contributing factors
Review volumes sized to agent throughput rather than human attention capacity.
Uniformly fluent outputs that conceal the error rate and lower reviewer vigilance.
Approval metrics rewarding queue clearance speed over challenge quality.
No measurement of override and challenge rates per reviewer.
Approval queues saturated by runaway agents or deliberate adversarial pressure.
Examples
Hiring reviewers approving agent-drafted candidate rejection notices at 400 per day stop reading the stated reasons, and mismatched reasons reach applicants.
A benefits caseworker facing an agent-generated backlog accepts eligibility determinations without individual review, and one decision cites household details that do not appear in the file.
A trust-and-safety reviewer whose queue quadrupled after agent rollout approves account reinstatements in seconds, including several the agent misclassified.
General purpose models can answer almost anything, so an agent's real capability always exceeds its approved mandate. Scope expands through user prompts, staff routing new work to a tool that appears capable, and unannounced product extensions, so the validated boundary is lost without any single decision.
The new risk agents introduce
A classic model cannot score what it was not built for. An agent attempts whatever it is asked and sounds equally confident outside its mandate, which undermines every evaluation on file.
Contributing factors
Mandate defined in policy documents but not enforced in the agent's runtime guardrails.
Staff and downstream teams routing new task types to an agent that appears capable.
Prompt and tool changes made without re-running the approval process.
No production monitoring of topics and populations against the approved scope.
Vendor platform updates adding capabilities the deploying organization never assessed.
Examples
A customer service chatbot approved for order status and account queries begins quoting eligibility and pricing terms, effectively making decisions with no validation or appeals process.
A document review agent validated on standard consumer contracts is applied to complex multi-party corporate agreements, where its extraction accuracy was never tested.
An internal policy assistant built to summarize guidance for staff starts answering front-line questions from the public about whether specific actions are permitted.
Risks02An event that alone produces a loss with an identifiable claimant.
Agent workloads can consume compute, tokens, API calls, and downstream resources far beyond what a task needs, driven by output-inflating prompts and high-volume abuse with no natural stopping point. On consumption-based pricing this becomes denial of wallet.
The new risk agents introduce
A single model call is bounded. An agent plans, retries, and calls tools in loops, so a small trigger expands into thousands of calls that exhaust budgets and throttle other workloads before anyone responds.
Contributing factors
Recursive planning and retry loops with no iteration or depth limit.
Adversarial prompts that inflate output length and token cost.
Consumption-based pricing exposed to deliberate denial-of-wallet abuse.
Missing rate, quota, spend and timeout limits on model and tool calls.
Multi-agent invocation multiplying resource use per request.
Examples
A recursive planning loop in an e-commerce agent re-queries a pricing API thousands of times, exhausting the daily quota for the whole store.
An attacker sends prompts that force an IT operations agent into long tool chains, accumulating cloud compute cost overnight.
A software agent stuck in a retry loop against a failing build service drives API spend past the monthly budget before morning.
Agent endpoints exposed as APIs are open to abuse, allowing attackers to extract system prompts and guardrails, query the model to clone its behavior or infer training data, scrape proprietary outputs, or send high volumes of expensive queries. Leaked system prompts expose tool definitions and business logic that let an attacker target the agent's actions.
The new risk agents introduce
A chatbot's system prompt governs what it says. An agent also declares the tools it can call, their parameters, and the conditions under which it calls them. Extraction therefore yields reconnaissance for attacks on the agent's actions, not only a copy of its behavior.
Contributing factors
Public agent endpoints open to scraping and systematic querying.
System prompts exposing guardrails, tool definitions and business logic.
Repeated querying that reconstructs model behavior from responses.
Membership inference against endpoints backed by fine-tuning or retrieval.
Error messages and refusals that disclose which tools exist and when they trigger.
Examples
An attacker extracts the system prompt of an e-commerce agent, revealing guardrails and tool definitions used to plan a bypass.
Systematic querying of a content-moderation endpoint lets an attacker clone the classifier and find inputs that evade it.
Risk modifiers09A state or circumstance that affects the likelihood, severity, or frequency of a risk occurring.
Models do not maintain a hard boundary between instructions and data, so any text in context can influence behavior. Crafted malicious input can therefore override operating rules, defeat guardrails, and redirect the agent toward behavior the deployer never intended. Severity depends on the position of the input. Text interpolated into a system prompt, developer message, or persistent memory inherits the authority of that position and is harder to defend against, but ordinary user-turn input remains sufficient for most attacks.
The new risk agents introduce
Agents fill system prompts, developer messages, and memory from user-supplied values. This multiplies the routes by which untrusted text can reach a privileged position and acquire instruction authority.
Contributing factors
System prompts and memory templates parameterized with values drawn from untrusted input.
No hard boundary in the model between instructions and data.
Guardrails expressed as instructions that injected text can countermand.
Prompt templates concatenating user text adjacent to privileged instructions.
Jailbreak techniques shared and iterated publicly faster than defenses adapt.
Examples
A user submits a jailbreak that convinces a support assistant it is in a test mode where refund policies do not apply.
A crafted message makes a moderation assistant reclassify prohibited content as compliant by overriding its review rubric.
A profile field inserted into an assistant's system prompt carries planted wording that disables its disclosure rules.
The agent ingests material from third parties as part of its task, and attackers plant instructions inside that material: hidden text in an email, a document field, a web page, or a repository comment. The attacker needs no access to the agent and no knowledge of who deploys it. The payload is placed where the agent will retrieve it and takes effect whenever it does.
The new risk agents introduce
Old chatbots only read what the user sent them. An agent reads retrieved documents, web pages, and files as a routine part of its task, so most of its input is reachable by an attacker who never contacts it.
Contributing factors
Retrieval, browsing and file pipelines inserting external content directly into prompts.
Hidden text channels such as white text, HTML comments, metadata and alt text.
Content from many authors ingested without provenance or trust ranking.
Summarization and triage tasks that require reading attacker-reachable material.
Planted instructions persisting in memory and knowledge stores across sessions.
Examples
A crafted instruction inside an uploaded return request tells an e-commerce agent to approve the refund and issue store credit automatically.
Hidden text in an inbound email steers a customer support agent to reveal another user's account details in the same thread.
A comment planted in a code repository directs a software engineering agent to add an unreviewed dependency and open a pull request.
In multi-agent systems, agents pass tasks, data, and trust across communication channels and shared state. Where messages lack authentication and integrity, an attacker can spoof an agent, inject false information, or corrupt shared resources so a compromise in one agent propagates to others.
The new risk agents introduce
This risk does not exist for a single model. Trust between agents is often implicit and transitive, so a manipulated upstream agent can influence downstream decisions before any human check.
Contributing factors
Inter-agent messages lacking authentication, integrity and semantic validation.
Implicit, transitive trust between agents in an orchestration.
Shared state and resources that carry compromise between agents.
Rogue or hijacked agents acting outside monitoring boundaries.
Cascade paths across intake, decisioning and fulfillment agents.
Examples
A spoofed verification agent sends a false eligibility confirmation that a benefits-processing agent trusts to release a payment.
A compromised data-feed agent supplies false loss estimates to an insurance claims agent, which approves an inflated settlement.
A hijacked intake agent injects manipulated order facts that skew a downstream fulfillment agent's routing decision.
Agents often run with broad access and autonomous tool selection, so a manipulated agent can enumerate credentials held in config files, environment variables, logs, secret stores, and instance metadata, then use or exfiltrate them.
The new risk agents introduce
A conventional model held no credentials. An agent is an active principal that reads secrets and calls systems, so credential collection scales through the same automation that makes the agent useful. One reachable key can enable lateral movement across core systems.
Contributing factors
Agents can reach secrets in config files, environment variables and logs.
Non-human identity sprawl without ownership, scoping or expiry.
Shared or long-lived credentials instead of short-lived scoped tokens.
Weak agent authentication enabling impersonation and session riding.
Credentials retained in agent memory and persistent state.
Examples
A manipulated agent enumerates environment variables and exfiltrates a cloud API key an attacker can reuse.
Orphaned service accounts for a decommissioned scheduling agent retain live access to a hospital records system.
An attacker spoofs an agent identity to invoke a publishing tool under credentials the content agent legitimately holds.
Agents select tools and build parameters from natural-language descriptions and from what earlier tools return. An attacker who controls a tool description, a connected server response, or a returned field can steer tool selection, inject malicious parameters into a legitimate call, or corrupt state that later steps rely on. Because each output informs the next decision, one manipulated response cascades through a multi-step workflow.
The new risk agents introduce
This attack surface never existed for classic models, which selected no tools.
Contributing factors
Tool selection driven by attacker-influenceable descriptions and metadata.
Tool outputs trusted as inputs to the next reasoning step.
Missing parameter validation on identifiers, amounts and authorization codes.
Insecure or unvetted integrations and connected server responses.
Multi-step sequences that evade single-call authorization checks.
Examples
A modified target parameter in an IT automation tool call reboots the wrong production server during a routine change.
A manipulated inventory-lookup response returns a false in-stock signal, so an e-commerce agent oversells a discontinued item.
A software agent chains an allowed file-read tool and an allowed network tool to send source code to an external endpoint.
Agent authorization is often set per tool rather than per business outcome, so an agent can discover reachable APIs and chain individually permitted calls into an unauthorized result. Approval workflows and separation of duties fail because no single call appears prohibited.
The new risk agents introduce
A static model has fixed inputs. An agent plans, reads results, and adapts, which allows it to find routes around controls, so a low-privileged agent can perform actions reserved for administrators.
Contributing factors
Authorization scoped per tool rather than per business outcome.
Chaining of individually allowed calls into a forbidden result.
Permission creep and dynamic scope expansion during long tasks.
Weak enforcement of approval workflows and separation of duties.
Broad standing access instead of task-bound least privilege.
Examples
An IT service-desk agent combines a directory-lookup tool and a weakly gated permissions tool to grant itself administrator rights.
An HR onboarding agent assembles an access-provisioning decision from partial calls, bypassing the manager approval that policy requires.
An e-commerce operations agent chains catalog and pricing tools to publish a discount beyond its authorized limit.
Agents ground answers and plans in retrieval corpora, vector stores, and persistent memory, so an attacker who injects poisoned documents or plants false entries influences what the agent later retrieves and how it reasons. Because memory persists, the effect is a durable backdoor that activates on specific records or triggers.
The new risk agents introduce
A stateless chatbot retains nothing between turns. An agent carries the corruption across sessions and propagates it through summaries and shared stores.
Contributing factors
Retrieval and memory content trusted without integrity or provenance checks.
Persistent state carries poisoning across sessions and users.
Writable memory and shared stores reachable by manipulated inputs.
Subtle, trigger-based changes that evade routine monitoring.
Poisoned entries indistinguishable from legitimate retrieved data.
Examples
A false entry injected into a moderation knowledge base causes a content agent to approve posts it should remove.
A crafted memory record makes a customer support agent persistently apply an unauthorized fee waiver to a set of accounts.
Poisoned reference content leads a clinical triage assistant to cite a fabricated guideline when advising on a symptom.
Detection and moderation agents classify inputs, behavior, and content. Adversaries interact, observe outcomes, and craft inputs that keep the score below the flagging threshold, which is an evasion attack at inference time. They can also shift the model's decision boundary by supplying patterns through feedback and retraining loops.
The new risk agents introduce
A static batch model gives adversaries slow feedback. An agent that adapts and responds quickly gives them a faster probing loop, so a safeguard can degrade without detection and remain degraded.
Contributing factors
Detection thresholds that can be probed through repeated interaction and feedback.
Fast, conversational responses accelerating adversarial trial and error.
Feedback and retraining loops open to deliberate skewing.
Concept drift concealing evasion as normal behavior change.
Attackers adapting faster than models are re-evaluated.
Examples
An abuse network tunes posting patterns until a trust-and-safety agent stops flagging its accounts.
Sellers craft listing text that a marketplace moderation agent misreads as compliant, so prohibited goods remain online.
An intruder adjusts login and traffic patterns to stay under a security operations agent's anomaly threshold.
The agent stack depends on third-party models, adapters, datasets, libraries, tools, and connected servers, any of which can arrive backdoored or turn malicious through an abandoned project or a compromised update. A corrupted component runs with the agent's trust and reaches decision-making at scale.
The new risk agents introduce
An in-house model incorporates few external components. An agent incorporates many and adds connected servers, so one poisoned tool or model update propagates through every agent that calls it before anyone detects it.
Contributing factors
Heavy dependence on third-party models, datasets, libraries, tools and connected servers.
Update channels and adapters that can be poisoned after adoption.
Weak vetting, provenance and integrity verification of components.
Concentration on a few providers raising systemic exposure.
Decentralized tool and agent-to-agent ecosystems widening the attack surface.
Examples
A compromised identity-verification server returns positive matches for every check, letting ineligible applicants pass a benefits screening.
A backdoored foundation model approves a transaction when a trigger phrase appears in an order note.
A poisoned library update in the agent stack exfiltrates request data to an attacker endpoint without detection.
Risks02An event that alone produces a loss with an identifiable claimant.
Agents answer calls, chat, and generate documents that read as human work. When the organization does not tell people they are interacting with a machine, or does not label generated content, those people act on a false premise. Transparency law requires informing people and marking synthetic content, with substantial fines for failure.
The new risk agents introduce
A static chatbot occupies one interface. Agents operate across voice, email, and outbound campaigns, and they transfer conversations between people and machines mid-thread, so disclosure must be enforced at each channel and each handoff.
Contributing factors
Voice and chat agents indistinguishable from human staff in regulated channels.
Human-to-agent handoffs mid-conversation without renewed disclosure.
Generated letters and documents entering files without synthetic-content marking.
Vendors embedding agents into serviced channels without notifying the organization.
Disclosure wording removed or made less prominent during interface optimization.
Examples
A debt-contact voice agent negotiates a repayment plan. The person later learns no human was involved, and a conduct regulator opens an inquiry into the omission.
A content platform publishes market commentary drafted by an agent under a named columnist, with no generated-content label, breaching its own editorial sign-off policy.
A government office sends a case-acknowledgment letter with a fabricated handler name, and the resident addresses months of correspondence to a person who does not exist.
When an agent declines a request, blocks an account, or reverses a benefit, the organization owes the person specific reasons. Agents assemble outcomes from opaque inference, retrieved passages, and tool outputs, so no single list of factors exists. A rationale constructed afterward may not reflect what produced the result.
The new risk agents introduce
A conventional model computes a decision from recorded inputs, so the same inputs yield the same reasons every time. An agent assembles the outcome from inference, retrieved passages, and tool results that vary between runs. The organization must still state the procedure and principles actually applied.
Contributing factors
Free-text reasoning with no stable mapping from inputs to decision factors.
Model-generated rationales that diverge from the actual computational path.
Explanations owed for the whole agent process, including retrievals and tool calls.
Reason codes required in prescribed formats that agent outputs do not produce.
Examples
A healthcare prior-authorization agent denies coverage. The notice lists generic reasons that do not match the clinical records the agent relied on.
A hiring agent rejects an applicant before interview. The recruitment team cannot state the trigger, and a discrimination complaint follows.
An applicant invokes their right to the logic of an automated decision after a benefits refusal. The agency can produce only the vendor's product description.
Risk modifiers05A state or circumstance that affects the likelihood, severity, or frequency of a risk occurring.
An agent outcome depends on the prompt, the documents it retrieved, its reasoning, and the tools it called with their responses. When any step is not captured, or lacks timestamps and version links, the organization holds a result with no evidentiary chain.
The new risk agents introduce
An agent takes many nondeterministic steps, calls tools whose outputs change, and hands tasks across systems that log separately. Gaps therefore become permanent, and no one can reconstruct why the agent acted.
Contributing factors
Logging built for application errors rather than decisions, so prompts, retrievals and intermediate reasoning are unrecorded.
Reasoning tokens hidden by the model provider and unavailable for capture.
Tool and retrieval outputs that change after the fact, making replay impossible.
No correlation identifiers linking steps across sessions, agents and downstream systems.
Log retention shorter than dispute, complaint and audit timelines.
Examples
A government services agent holds an application for manual review. Months later an appeal asks why, and the office finds only the final status, not the records the agent retrieved.
A customer support agent waives charges for some users. Internal audit cannot tell which policy passage the agent read, so it cannot separate error from entitlement.
A hiring screening agent rejects a candidate. The reasoning ran on a model version the vendor has retired, and the trace was never stored, so the decision cannot be reviewed.
Agents act across functions that each assume someone else is responsible. The business function owns the process, technology owns the platform, and the vendor owns the model, but no one owns the agent's conduct. Without a named owner, remediation stalls and no one can authorize a fix or compensate those harmed.
The new risk agents introduce
A conventional system has a documented owner and reviewer. An agent's behavior emerges from a vendor model, a prompt, and tools owned by different teams, and it can take actions no individual instructed.
Contributing factors
Agent conduct spread across model vendor, platform team, prompt authors and business users.
No default ownership rule carried over from earlier model governance to agents.
Risk acceptances and change approvals not recorded in writing.
Autonomous actions that no individual instructed, which every party can therefore disclaim.
Multi-agent workflows where handoffs obscure which system caused the outcome.
Examples
An IT operations agent restarts a production service and triggers an outage. The platform team says the vendor set the thresholds, and the vendor points to the prompt template.
After an e-commerce agent issues refunds beyond policy, remediation stalls for weeks because no one has authority to approve the fix or recover the funds.
A reviewer asks who approved raising an HR onboarding agent's autonomy level. The change ticket shows a shared mailbox and no individual sign-off.
Agents appear without passing an approval process. Staff connect model APIs to spreadsheets, teams enable agent features inside procured software, and vendors activate AI components by default. None enter the inventory, so no tiering, testing, or monitoring applies.
The new risk agents introduce
A low-code automation and an API key now give a user an acting system with production data access. Because agents act, an unmanaged one can move data or send communications before anyone knows it exists.
Contributing factors
Model API access and low-code tooling that permit business users to build acting systems.
Agentic features enabled by vendor updates inside procured software.
Procurement and change processes that never ask whether software contains AI.
Inventory definitions written for models, missing prompts, tools and orchestration.
No detection of unauthorized model API traffic on the corporate network.
Examples
An operations analyst connects a workflow tool to a public model API to draft customer refund letters, so account data leaves the approved environment.
A CRM vendor enables an agent assistant in a quarterly update. Sales staff use it on customer records for months before the risk function learns it exists.
During an IT incident review, forensics finds an unregistered reconciliation agent that had been auto-retrying failed batch jobs under a shared service account.
Agent stacks assemble a frontier model, orchestration, tools, and data services from different providers. Each layer withholds something as proprietary, whether training provenance, evaluation detail, or incident histories. The organization must still attest that the whole system is sound.
The new risk agents introduce
Earlier vendor models were opaque but were validated as a single scored component. An agent inherits opacity at every layer, and the layers change independently. An upstream update therefore alters downstream behavior without notice, leaving the deployer with obligations it cannot discharge.
Contributing factors
Providers refusing evaluation detail, training data provenance and incident histories as trade secrets.
Model updates made without notice, changelogs or regression evidence.
Fourth parties in the value chain not covered by the organization's contracts.
Standard AI service terms lacking audit rights, incident duties and exit assistance.
Documentation produced for marketing rather than for validation or review use.
Examples
An organization asks its foundation model provider for evaluation evidence to support internal validation of an identity-verification agent. The provider offers a system card and refuses test-level detail.
A fraud-detection API degrades after an unannounced provider update. The vendor declines to confirm what changed, leaving the buyer unable to complete its incident report.
A software procurement review needs subcontractor detail for an agent platform. The vendor will not disclose which cloud region hosts its tool servers.
Audits require evidence: validation reports, threshold-based test results, design records, and change histories. Agent deployments proceed on vendor claims and prompt edits, with no thresholds or versioned test sets.
The new risk agents introduce
A conventional model can be rerun on archived inputs to reproduce a decision. Agents are nondeterministic and depend on provider models and corpora that change independently, so evidence must come from run-time capture. Informal rollouts omit that capture, and the gap becomes an audit finding.
Contributing factors
No pre-deployment acceptance evidence with defined thresholds and versioned test sets.
Nondeterminism and provider model turnover making past outcomes unreproducible.
Validation practice built for fixed models, untested against open-ended agent behavior.
Documentation insufficient for a reviewer unfamiliar with the system.
Evidence dispersed across vendor portals, chat logs and unversioned prompt files.
Examples
A regulatory review requests validation evidence for an agent that drafts clinical summaries. A vendor brochure is all that exists, with no independent testing.
Internal audit samples a hiring agent's rejections and cannot rerun a single case, because the provider retired the model version and the retrieval corpus changed.
A review asks for the change history of a prompt that governs legal contract commentary. Edits were held in a shared document with no approvals.
No entry matches every filter. Remove one to widen the result.