Something structural shifted in the AI market over the past eighteen months, and most enterprise risk functions haven’t caught up to what it means for their vendor relationships.

The major AI labs are no longer pure infrastructure providers. OpenAI built a computer-using agent and folded it into ChatGPT Agent, an agentic layer inside its flagship product. Google is weaving Gemini directly into Workspace. Anthropic has moved into the operating layer of the workplace itself: in June it launched Claude Tag, an always-on agent that lives inside a company’s Slack channels and, in its own description, learns ever more about the work as it follows along. Microsoft has gone further still, into the documents themselves: Copilot now sits inside Word, Excel, Teams, and Outlook, reading the work as it is written, on top of an Azure layer that hosts models for the same customers whose applications compete with Microsoft 365. The pattern is consistent. The companies that sell AI infrastructure are also building application-layer products that compete with the software vendors who use that same infrastructure, and increasingly, products that sit inside the customer’s operations and observe them.

Subscribe now

The clearest sign of where this goes appeared in June 2026. AWS introduced a model-to-model migration capability in AWS Transform that scans a customer’s codebase, identifies its dependencies on OpenAI, Google, and Anthropic, and generates a plan to move those workloads onto Amazon Bedrock, AWS’s own model service. The engineering value is real. Structurally, it is also one infrastructure provider shipping a tool that reads how deeply you have built on its competitors and converts you off them. Visibility into your architecture and a direct competitive interest in it are not separate concerns there. They are the same product.

This is a familiar dynamic in platform economics. What’s less familiar is the specific implication for enterprise AI governance. If your model provider now competes with your software vendors and occupies the infrastructure layer beneath them, what does your contract actually say about what they can observe about your operations?

I spent time recently reviewing the standard enterprise terms for four major AI providers from primary sources. What I found is a gap that is only starting to be named, and named at a different layer than the one that matters most here.

What Providers Can Observe (Without Training on Your Content)

The conversation about AI data privacy has been dominated by one question, is my vendor training on my data? This is a reasonable question and the major providers have responded to market pressure. Anthropic’s commercial terms prohibit model training on customer content outright. OpenAI provides opt-out controls. Google Cloud’s enterprise terms restrict processing to service provision.

But training on content is not the only form of strategic visibility.

When your organization uses an AI API at scale, your provider sees a different class of data. Usage patterns. Which business domains you’re querying. At what volume, and at what cadence. What failure patterns emerge. How your usage scales over time. What price tier you’re operating at. Which capabilities you lean on most heavily.

None of this is your content. It’s metadata about your operations. And it’s potentially more strategically valuable than your content, because it maps your dependency structure. A provider watching your usage patterns over twelve months can infer which workflows you’ve automated, how operationally dependent you’ve become, and where your organization is concentrating AI investment. That’s a competitive intelligence profile, assembled from infrastructure-layer visibility, with no inference required from the content of your prompts.

And the surface is widening. As organizations connect their internal systems to models through the Model Context Protocol, the open standard Anthropic introduced and the other major providers have adopted, more than usage patterns pass through. The tools you expose, the calls your agents make, and the results those calls return all enter the model’s context. The map of which operational systems you have wired to AI becomes visible at the same layer as how often you use it.

The deepest form of this visibility is no longer remote. In June 2026, AWS committed a billion dollars to a Forward Deployed Engineering organization that places its own engineers, working alongside its agents, directly inside customer teams on short deployment cycles. It is not alone. Google Cloud is fielding forward-deployed engineers of its own, and OpenAI and Anthropic have stood up comparable units. The stated purpose is the last mile, getting AI into production. But look at the design. AWS describes implementing an enterprise-wide semantic context, an ontology of the customer’s business, in nearly every engagement. That is the provider’s people, inside your operations, building a structured model of how your organization works, on infrastructure the provider owns. At that point the observation is not a byproduct of API traffic. It is the service.

This is not a theoretical concern. It’s a structural feature of how AI is now sold and deployed, and the question of whether standard contracts address it has a clear answer.

What the Contracts Say

I reviewed the standard enterprise agreements for OpenAI, Anthropic, Google Cloud, and Microsoft. In each case, I focused on three questions: how is protected customer data defined, what restrictions apply to the provider’s use of it, and whether usage metadata appears as a named category.

OpenAI Services Agreement. Section 17 defines “Customer Content” as “the Input and the Output.” That includes your prompts and the model’s responses. Section 4.2 limits OpenAI’s obligations to Customer Content. Section 3.3(e) prohibits the customer from using outputs to develop competing AI models. No equivalent restriction runs against OpenAI, and no defined usage-metadata category appears in the terms.

Anthropic Commercial Terms of Service. Section B contains the training prohibition: “Anthropic may not train models on Customer Content from Services.” The prohibition is flat and unequivocal, which is the strongest protection in this area among the four providers I reviewed. But “Customer Content” is defined as Inputs and Outputs. The same narrow definition. Section E.1 elevates Customer Content to Confidential Information status. Section D.4 prohibits the customer from accessing the services to build a competing product. Again, no equivalent restriction on Anthropic. Usage metadata does not appear as a protected category.

Google Cloud Data Processing Addendum. The picture here is more textured, and among the strongest of the four. The CDPA defines “Customer Data” broadly, as “data provided by or on behalf of Customer or its End Users” through Google Cloud Platform, or “submitted, stored, sent or received by or on behalf of Customer” through Workspace, not a narrow inputs-and-outputs formulation. Under Section 5.2, Google is instructed to process that data only on the customer’s instructions and only “to provide, secure, and monitor the Services.” That is a purpose-bound commitment, and a good one.

Share

Two complications. First, “monitor” is load-bearing language. Monitoring the services necessarily involves observing usage patterns. If usage metadata is classified as data Google observes in the course of monitoring rather than data the customer provides, it falls outside Customer Data entirely. Second, the CDPA is structured primarily as a GDPR compliance framework. Its protections are most robust for enterprises subject to European data protection law. For US-based enterprises operating outside GDPR scope, the contractual architecture is present but the enforcement and standing landscape differs.

Microsoft Products and Services Data Protection Addendum (updated May 22, 2026). Microsoft is the most vertically integrated of the four, and on paper the most explicit. Its protected category is broad. “Customer Data” is defined as “all data, including all text, sound, video, or image files, and software, that are provided to Microsoft by, or on behalf of, Customer through use of the Online Service,” and under Nature of Data Processing the customer “retains all right, title and interest in and to Customer Data,” while “Microsoft acquires no rights.” Microsoft further commits that it will not use that data for “user profiling,” “advertising or similar commercial purposes,” or “market research aimed at creating new functionalities, services, or products,” except on the customer’s documented instructions. That is a stronger, broader definition than the input-and-output framing above.

But Microsoft is also the only one of the four that names the observational layer directly, and having named it, reserves it. Under “business operations incident to providing the Products and Services,” the customer authorizes Microsoft “to create aggregated statistical, non-personal data from data containing pseudonymized identifiers (such as usage logs containing unique, pseudonymized identifiers)” and “to calculate statistics related to Customer Data,” in each case “without accessing or analyzing the content of Customer Data,” for a closed list of purposes that includes “internal reporting and business modeling, such as forecasting, revenue, capacity planning, and product strategy.” Read that last one again. The provider may derive statistics from how you use the service, and use them for its own product strategy, as long as it does not read your content. That is the exact layer this piece is about, and here it is written down, permitted, and fenced.

That makes Microsoft the instructive case, not the cautionary one. The other three are effectively silent on this layer. Microsoft governs it: it names the category, grants itself a narrow permission, and bounds that permission with “no content,” “non-personal,” data minimization, and a closed list of purposes. The lesson is not that Microsoft takes more. It is that a contract can actually address what a provider derives from watching you, which throws the silence in the other three into sharp relief.

The finding across the four is not uniform, and the variation is the point. Three of them, OpenAI, Anthropic, and Google, do not define “usage metadata,” “API telemetry,” or “operational data” as a named category at all. Their protection stops at content, your prompts and your outputs. Microsoft is the exception: it names the layer and treats it as a bounded permission it holds, not a protection you own. Either way, across all four, what the provider observes or derives about how you operate is not, by default, yours. Outside Microsoft’s terms, it is largely unaddressed.

What Has Been Named, and What Hasn’t

Enterprise AI governance is still a young discipline, and the conflict at the center of this is beginning to be named, though at a different layer than the one this piece is about.

Earlier this year the Vanderbilt Policy Accelerator published Net Neutrality for AI, by Asad Ramzanali and Akhil Rajan. It argues that foundation-model providers who also compete with the startups depending on them hold a structural conflict, and it proposes an AI-neutrality rule modeled on net neutrality. That work names the conflict, and names it well. But it frames the harm as discrimination: a provider throttling, deprioritizing, or cutting off a competitor’s access, as when Anthropic restricted the coding agent Windsurf’s access to Claude during OpenAI’s acquisition talks. The remedy it proposes is a rule against picking winners.

The mechanism this piece is about is different. It is not discrimination in access. It is observation. What the provider can see about your operations from its position beneath them, and what your contract says it may do with that. A neutrality rule would stop a provider from throttling you. It would do nothing about what the provider learns from watching you. Those are two different exposures, and the second one lives in the definition sections of the agreement you have already signed.

The analogy that fits the observation problem is financial services conflict of interest governance. It is well established that a broker-dealer who serves both as a market maker and an advisor to the same client is in a structurally conflicted position, regardless of whether they have done anything wrong in a specific transaction. The relationship structure itself requires disclosure, assessment, and management. The question is not whether to trust them. It is whether the relationship has been named, mapped, and governed.

AI supply chain relationships are now in a structurally similar position. Your model provider is also a product competitor to the software vendors built on their API, and occupies the infrastructure layer with visibility into your operational behavior. That is a relationship structure that deserves named governance, regardless of any specific concern about a specific provider’s conduct.

Most enterprise risk functions have not done this mapping because the category does not exist in their frameworks yet. AI sits in legal (data privacy), IT (vendor security), or finance (procurement). None of those functions owns the competitive relationship dimension.

The Provider Variation Problem

One finding worth underscoring. The gap is not uniform across providers, and this is itself a governance problem.

Anthropic’s training prohibition is absolute and well-defined. OpenAI’s is conditional and subject to opt-out. Google Cloud’s protections are the most architecturally robust but carry GDPR-jurisdictional complexity and a “monitoring” carve-out that creates definitional ambiguity. Microsoft is the only one of the four to address the observational layer at all, and it does so as a bounded permission it reserves rather than a protection the customer holds.

AWS is the instructive exception. Its service terms address competitive use of usage data more directly than the others, even as AWS is, through Transform and its Forward Deployed Engineering organization, the most aggressive of the providers at operating inside customer environments. Contract language and conduct do not always point the same direction, which is exactly why the relationship has to be governed on both.

An enterprise running workloads across multiple AI providers, which is the norm at scale, has a different risk profile with each one. A generic policy position (”we don’t allow AI vendors to use our data”) does not map to the actual contractual landscape. What is needed is a provider-by-provider assessment that accounts for these differences, and that assessment is not yet a standard enterprise practice.

Three Questions to Ask

These questions aren’t designed to generate alarm. They’re designed to reveal whether your organization has mapped the exposure. If you can answer all three, you’ve done more than most.

One. Have you identified which AI providers have infrastructure-layer visibility into your usage patterns, and which business workflows are represented in that data, including any always-on agents that observe your teams directly or provider engineers embedded in your teams? Most organizations know what APIs they’ve procured. Fewer have mapped what operational behavior is visible to those providers as a byproduct of use.

Two. Do your enterprise AI agreements define “usage data,” “operational telemetry,” or “API metadata” as a protected category, or do they stop at inputs and outputs? This is a contract review question, not a legal opinion question. You can answer it by reading the definition sections.

Three. If your primary AI provider launched an application-layer product that competes directly with a workflow you’ve automated using their API, what would your current contract actually restrict them from doing with what they’ve observed about your operations? If you don’t know the answer, the contract probably doesn’t address it.

What This Is and Isn’t

This is not an argument that AI providers are acting badly. It’s an observation that standard enterprise contracts weren’t written for a world in which your infrastructure provider is also an application-layer competitor, and that the specific exposure this creates, what the provider can observe and what the contract permits it to do with that, sits underneath the conflict others have started to name.

Unnamed risks don’t get managed. They get inherited by default.

The vertical integration of AI labs into the application layer is not slowing down. The enterprises that navigate it well won’t be the ones that assumed the standard contract covered everything. They’ll be the ones that did the mapping, asked the questions, and built governance for the relationship structure that actually exists.

The starting point is straightforward. Read the definition sections of your AI vendor agreements. Find out what “customer data” actually includes. Ask what’s missing. That’s a two-hour review, and most organizations haven’t done it.

Schedule a call

All four providers’ terms were re-pulled from primary sources and reviewed on July 21, 2026: OpenAI’s Services Agreement, Anthropic’s Commercial Terms of Service, Google’s Cloud Data Processing Addendum, and Microsoft’s Products and Services Data Protection Addendum (updated May 22, 2026). Citations reflect the published enterprise terms as of that date.

Thomas is the founder of Fellowship Intelligence, which helps organizations design and install governance systems for their use of AI. He writes about AI governance, organizational risk, and the structural questions that enterprise AI raises for how work is controlled.

Fellowship Intelligence helps organizations govern the decisions AI is already making. The Diagnostic is the place to start. Learn more at fellowshipintelligence.com.

When a state decides an AI system has crossed a line, you can learn more from the law it reaches for than from anything it says in the press release. Pennsylvania just reached for a statute written long before anyone imagined a language model, and that choice is the whole story.

In early May, the Pennsylvania Department of State filed suit in Commonwealth Court against Character Technologies, the company behind Character.AI. The Commonwealth alleges that chatbots on the platform held themselves out as licensed medical professionals. According to the filing, one presented itself as a psychiatrist, claimed to be licensed in Pennsylvania, and gave a license number that was not real. The state is seeking a preliminary injunction to stop it. Character Technologies has said its characters are fictional and that users should not rely on them for professional advice. This is an allegation, not a finding. Nothing has been proven, and the company will have its defense.

But watch the instrument. The state did not wait for an AI-specific law, and it did not ask the legislature for one. A new statute would have arrived with a runway: a comment period, a headline, an effective date a year or two out. This landed instead as a complaint about something already done. It used the Medical Practice Act, the ordinary licensing statute that says a simple thing: you cannot hold yourself out as a licensed medical professional without the license. The Commonwealth’s theory is that this rule does not care whether the thing holding itself out is a person or a program. The conduct is the violation, and the conduct is what happened.

That is the part most companies deploying customer-facing AI have not absorbed. The governance question they keep asking is whether there is an AI law for this yet. It is the wrong question. The question a regulator actually asks is older and simpler: did your system do something the law already forbids? And “it benchmarked well,” or “it passed a bias test two years ago,” is not an answer. A score is a snapshot; the conduct is live. A model that invents a credential is not in a gray zone waiting for Congress. It is doing something a licensing statute has prohibited for a century, and the only new fact is that the thing doing it is software you deployed.

We saw the same shape last week in the insurance market, and the month before from FINRA: the rules were written to be technologically neutral, so they never lagged the technology. The lag was always inside the company. Air Canada learned it when a tribunal held the airline to a bereavement refund its own chatbot invented. Pennsylvania is now testing whether the same logic reaches a chatbot that invents a medical license. Different statute, identical principle: when your AI speaks to a customer, you own what it says.

There is a wrinkle that makes this sharper, not softer. Character.AI lets users build and deploy their own characters, and some present themselves as professionals. It is tempting to read that as diffusion of responsibility: the platform did not write the psychiatrist, a user did. But that is exactly the accountability move the whole thesis is about. A name moved; the decision did not. Someone decided to ship a system in which a character can claim a license and hand out a number, with nothing in the path that checks whether the claim is true. That decision has an owner, and the owner is not the fictional doctor.

So the practical lesson is not to add a disclaimer, though disclaimers matter. It is that if your AI can make a representation a human would need a license, a permit, or a disclosure to make, then someone inside your company owns that representation the moment the system can produce it. Not after a regulator calls. Before. That ownership looks like the same short list it always does: a named person accountable for what the system is allowed to say, a review step where the output is high-consequence, and a record you could produce if a Department of State came asking how the claim got made.

Character.AI reports more than 20 million monthly active users. That figure is usually cited as growth. Read it as exposure. It is the number of conversations in which a system someone deployed might make a representation the deployer is answerable for. Regulators have noticed that the interesting AI enforcement does not require a new statute. It requires an old one and a deployer who never assigned the owner.

Pennsylvania’s filing may or may not succeed on its specific claim. The precedent that matters is not the ruling. It is the method: a state looked at an AI system, declined to wait for AI law, and applied the law already on the books to the conduct in front of it. Every other regulator now has that template. The companies that are ready are the ones that already asked the older question of their own systems. Is there anything our AI can say that a person would need standing to say, and who owns it when it does.

AI operates. You own the decision, and the license it invents.

Thursday, subscriber-only, and a different subject entirely: AI can counterfeit every trigger that makes us say yes, and the persuasion still has an owner.

Sources (click to verify): PA Governor’s release · the complaint (filed May 1, 2026) · NPR · Epstein Becker Green · Moffatt v. Air Canada, 2024 BCCRT 149

When you buy an AI liability policy, the underwriter has to price it. To price it, they have to know what AI you run, where it touches a consequential decision, and what stands between the model and the outcome. So before they quote you, they inventory your AI and score its risk. They ask which systems are in production, which ones face customers or regulators, what the human review step is, what gets logged, and who is accountable when it is wrong. Then they turn that into a number and put capital behind it.

Read that sequence again, because there is something uncomfortable in the order of operations. The first complete inventory of your AI exposure, the first honest assessment of the gap between what your systems do and what your governance controls, may not be produced by your risk team, your CISO, or your board. It may be produced by an outside party that is betting money against you, and keeping the map.

That is the move worth naming. Not that insurance is bad. That in buying it, most companies will outsource the first act of governance to the one party whose interests are not aligned with theirs.

The inventory is not paperwork. It is the governance.

There is a habit of treating an AI inventory as a compliance chore, a spreadsheet someone fills in before an audit. That gets the causality backwards. You cannot govern what you have not named. The act of listing every place AI touches a decision, and writing down who owns each one, is not preparation for governance. It is governance, in its first and most load-bearing form. Everything downstream, the review steps, the escalation paths, the logging, depends on that list existing and being owned inside the company.

When the underwriter builds that list for you, you get a version of the artifact without the thing that made it valuable. You learn your number. You do not build the capability. You hold a map drawn by someone who will use it to price your failure, not to prevent it. And you still do not have, inside your own walls, a named person who can stand in front of a regulator and say: I own this decision, here is how it is controlled, here is the record.

That last sentence is the whole game, and it is the thing no policy contains.

Subscribe now

What coverage settles, and what it cannot

A policy settles a loss. When the AI-driven decision goes wrong and someone is harmed, the policy, if the claim is covered and not excluded, pays money. That is real value and I am not waving it away. But walk the failure forward past the payment. The regulator opens an inquiry and asks who approved the output that broke the rule. The board asks who was accountable for the system. The customer, and eventually a court, holds your company to what your AI did, exactly as a tribunal held Air Canada to its chatbot. At none of those tables does the check from your carrier answer the question being asked. The question is about ownership of a decision. Insurance is about ownership of a cost. They are different objects, and only one of them can be handed to someone else.

This is the error the premium invites you to make. It is designed, honestly, to feel like resolution. You had an exposure, you paid, the exposure is handled. But you handled the accounting for the exposure. The exposure itself, the ungoverned decision running in production, is exactly where it was before you wrote the check. You are now insured against its cost and no better governed against its occurrence.

The premium is a score. Read it as one.

Treat it as a governance score issued by a party that cannot be spun. Your vendor wants to sell you capability. Your team wants to report progress. The underwriter wants to be paid correctly for real risk, which means their assessment of your AI governance is the least flattering and most reliable one you will get. A high premium is not only a cost to negotiate down. It is a reading. It says the distance between the AI you run and the control you have around it is large, and here is what that distance is worth in dollars.

The companies that get this right treat the quote as a mirror and go do, internally and before the renewal, the work the underwriter was about to do for them. Inventory the systems. Name an owner for each consequential decision. Put a human review step where the output is high-consequence. Record what the system did and why. Not mainly to lower the premium, though it will, but to hold the thing the premium cannot: the accountability the claim will eventually demand a name for.

Because the sequence only runs one of two ways. Either you inventory your AI and decide who owns each decision it makes, or an insurer does it first, prices your gap, and hands you a policy that pays for the failure while leaving the ownership exactly where it always was, unassigned and now running at scale.

The underwriter is not your governance function. They are a signal that you need one, arriving with a number attached and a map you should have drawn yourself.

AI operates. You own the decision. Including the decision to let your underwriter be the first to find out where your AI actually lives.

Share

Fellowship Intelligence defines and installs the governance this piece describes: named ownership for every consequential AI decision, in place before the underwriter or the regulator prices its absence. The entry point is a Diagnostic that maps where AI touches decisions across your organization and who is accountable for each one. It is $500, credited in full toward a Decision Brief signed within 30 days.

Sources:

1. McKinsey, State of AI (Nov 2025) — the 88% / 78% figure: https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai

2. ISO / Verisk exclusions (CG 40 47/40 48/35 08) — Verisk issuer note · Gallagher explainer · CG 40 48 form PDF

3. Carriers filing exclusions — Financial Times · InsuranceNewsNet

4. Mayflower & Hadron program — Business Wire · Reinsurance News

5. Moffatt v. Air Canada, 2024 BCCRT 149 — CanLII full decision

6. Walters v. OpenAI (GA, May 19, 2025) — Loeb & Loeb · Cleary Gottlieb

The question is simple to state and expensive to ignore: when AI inside your operation produces a bad outcome, whose loss is it?

For two years the honest answer inside most organizations has been a shrug spread across a committee. IT owns the tool. Legal owns the contract. A working group owns the policy document. No one owns the decision the AI actually makes. That was survivable while AI was a pilot. It stopped being survivable once 88% of organizations reported using AI in at least one function, up from 78% a year earlier. The pilots became infrastructure. The ownership question did not get answered. It got deployed around.

The insurance market just answered it for you, in two moves.

Subscribe now

Move one: your existing policy stopped covering AI

In January, Verisk’s ISO, the body that writes the standard forms most commercial insurers build on, released new endorsements, CG 40 47 chief among them, that let carriers exclude generative AI from commercial general liability. This is not a fringe product. By spring, carriers including W.R. Berkley, Chubb, Travelers, Berkshire Hathaway, and Cincinnati Financial had filed to adopt AI exclusions, and regulators approved the large majority. The coverage many companies assume protects their AI use is being removed by endorsement, quietly, at renewal.

Sit with what an exclusion is. It is an insurer looking at a category of risk and deciding it is too uncertain, too correlated, or too large to keep inside a general policy at the current price. That is not indifference to AI. It is a judgment that the risk is real, material, and separable enough to name and carve out.

Move two: a market opened to price it directly

In June, Mayflower and Hadron launched what they describe as the first dedicated affirmative AI liability program in the US: explicit coverage across directors and officers, employment practices, and errors and omissions, written for enterprise AI. The launch is not the story. The underwriting method is. They price the policy by scoring the deployer’s AI for specific failure modes, bias, drift, and hallucination, against NIST and ISO frameworks.

Read that plainly. An insurer has built an actuarial model of your AI risk. Not the model maker’s. Yours, the deployer’s. They are pricing the gap between the AI you run and the governance you have wrapped around it, and they are confident enough in that price to put reinsurance behind it.

Put the two moves together and the logic is unambiguous. The market pulled AI out of the pool where risk is shared and vague and moved it into one where risk is named and individually priced. That only happens to a risk with a clear owner. You cannot price what you cannot attribute. The underwriters attributed it. To you.

What the underwriter actually asks

To get that price, you fill out a questionnaire. Strip away the formatting and it asks a handful of questions, in substance:

Where is AI used in your operation, and which of those uses touch a customer, an employee, or a regulated decision?

For each one, who is the named person accountable when it is wrong?

What is the human review step before the output leaves the building?

What do you log, and could you reconstruct why the system did what it did?

Not one of those is a technical question about the model. Every one is a governance question about you. The underwriter is not curious how the model works. It is establishing whether anyone is holding the wheel, because that is what decides whether it pays out, and how often.

Which is why two companies running the identical models will not get the identical quote.

Firm A and Firm B

Firm A and Firm B use the same three vendors and the same foundation models. Firm A can name an owner for each consequential AI use, can show the review step, and can produce a log. Firm B connected the tools, saw the productivity, and never assigned the accountability, because it felt administrative and no one made it anyone’s job.

The underwriter cannot see intentions. It can see controls, or their absence. Firm A prices a small gap. Firm B prices a large one, or cannot complete the questionnaire and does not qualify. Same models, same industry, different number, because the risk was never really about the model. It was about who owns the decision the model makes.

So the quote is a diagnostic you did not order. An outside party, with money at stake and no incentive to flatter you, has assessed your AI governance and returned a number. If it comes back high, that is not the insurer being cautious. It is the insurer telling you what your ungoverned surface is worth.

The courts drew the line first

None of this is the insurance industry inventing a position. It is the industry catching up to one the law has been sketching for a while.

When Air Canada’s customer-service chatbot told a passenger he could claim a bereavement fare after the fact, and the airline argued it should not be bound by its own bot’s error, the British Columbia Civil Resolution Tribunal rejected the argument outright. Air Canada was responsible for the information on its website, whether it came from a static page or a chatbot. The bot was not a third party. It was the company. That is the cleanest statement of the principle: when your AI speaks to a customer, you own what it says.

A separate case cuts the other way, and it pays to state it precisely, because it is narrower than it first looks. When Mark Walters sued OpenAI over a false statement ChatGPT generated about him, a Georgia court granted OpenAI summary judgment on several independent grounds, one of them that OpenAI’s repeated, prominent disclaimers meant no reasonable reader would take the output as a statement of fact. That is a trial-level defamation holding, not a rule about who owns AI liability. Read for direction rather than doctrine, the two cases point the same way: the maker that discloses its limits steps back, and the party that puts the output to work answers for it. Insurance is now priced on that same line.

What the premium is actually measuring

Here is the part to carry into your next renewal conversation.

Insurance is a fine thing to buy. It turns a volatile loss into a fixed cost, and for AI exposure that is prudent. But be precise about what it does, because the marketing will blur it. A policy transfers the cost of a failure. It does not transfer the ownership of the decision that caused it.

Walk the failure past the payment. The regulator opens an inquiry and asks who approved the output. The board asks who was accountable. The customer, and eventually a court, holds your company, not your carrier, to what your system did. At none of those tables does the check answer the question being asked. Coverage answers the invoice. It does not answer the question.

The market has done you an inadvertent favor. It has put a dollar figure on a thing your organization kept treating as unpriceable, and therefore ignorable. The exposure was always yours. Now it has a number next to it, and the number moves for exactly one reason.

The only move that lowers it is the one no policy can make for you: decide, before the system runs, who owns the decision it makes.

AI operates. You own the decision. The underwriter already assumed you did, and priced it.

Schedule a call

Thursday’s piece goes to the sharp edge of this, and it is subscriber-only: how the underwriter ends up doing your AI inventory for you, and why the check that pays the claim still cannot answer for the decision. Subscribe and it will reach you.

Sources

• “88% use AI, up from 78%”: McKinsey, The State of AI (Nov 2025)

• ISO exclusions CG 40 47 / 40 48 / 35 08 (eff. Jan 1, 2026): Verisk / ISO general liability filing

• Carriers excluding AI (W.R. Berkley, Chubb, Travelers, Berkshire Hathaway, Cincinnati Financial): Financial Times

• Mayflower & Hadron AI liability program (June 24, 2026): Business Wire

• Moffatt v. Air Canada, 2024 BCCRT 149: CanLII, full decision

• Walters v. OpenAI (Gwinnett County, GA, May 19, 2025): Loeb & Loeb

Notes: carrier-level specifics are press-sourced (primary record is each carrier’s SERFF state filing). Walters is a trial-level defamation ruling, read for direction, not a liability-allocation holding.

For most of corporate history, business-critical technology arrived through an institution. Procurement bought it. Legal reviewed the contract. Someone signed off and owned it. The controls came attached, because the path it traveled had checkpoints built in.

AI broke that path. It arrives from an app store. A browser tab. A free tier. A tool someone found on a Tuesday. No procurement step, no contract review, no owner. The technology shows up without the institution that used to govern it.

So when people ask what to put in their AI vendor contracts, the honest answer is that the contract was rarely the problem. A competent legal team handles standard terms. The problem is that the decision to use the tool never reached legal in the first place.

That’s the gap. Not the contract. The missing institution around it.

The fix isn’t a better agreement. It’s deciding, on purpose, what data is allowed into which tools and who signs off on a new use case before it’s running. That used to happen on the way in. Now it only happens if someone decides to make it happen.

First shared on LinkedIn

That same day, an email landed in my inbox. Not as someone who reads policy papers, but as a Google Workspace administrator. It announced a new feature in Google Vids: personal avatars, built from a user’s own face and voice. Rolling out July 14, 2026. On by default for every eligible user. I could turn it off, but only if I knew to look, and only if I understood why it mattered.

Illinois and Texas have laws that govern exactly this kind of data collection. Illinois’ Biometric Information Privacy Act and Texas’ Capture or Use of Biometric Identifier statute both require affirmative, informed consent before collecting biometric identifiers, which include face geometry and voiceprints. Default-on with an administrative opt-out does not satisfy either statute. Consent must be prior and affirmative. (Note: this is not legal advice. Organizations with users in Illinois or Texas should consult counsel before July 14.)

Now hold Google’s own standard against Google’s own product. The paper says if something is illegal without AI, it is illegal with AI. Collecting face geometry and voiceprints without prior, affirmative consent is illegal without AI in Illinois and Texas. Switching it on by default does not become lawful because the feature is powered by AI. By the paper’s own rule, the policy argument and the product notice, published the same day, point in opposite directions.

Before setting that contradiction aside, it is worth examining the frontier solution Google is proposing. The paper calls for a Frontier AI Regulatory Organization, modeled on FINRA and NERC. The governance design is in the paper’s own language: “The FARO board would include a combination of independent directors and industry representatives.” Companies with frontier models would become FARO members. FARO would draw its standards from bodies like the Frontier Model Forum, which Google co-founded alongside Microsoft, OpenAI, and Anthropic. The baseline framework FARO would audit against is explicitly modeled on Google DeepMind’s own Frontier Safety Framework. Audit reports would be submitted confidentially to FARO, not released publicly.

Read the structure. Google proposes the body, Google funds it as a member, Google sits on its board as an industry representative. Google co-founded the standards body whose guidelines FARO would apply. Google’s own safety framework is the explicit template. Audit results stay inside the structure.

This is not a regulatory design that holds frontier labs accountable to an external standard. It is a regulatory design that seats frontier labs inside the accountability structure itself. The “independent” part of the board is balanced, by design, against industry voices who have an explicit interest in speed to market. That phrase appears verbatim in the board composition paragraph.

The body that would govern the most powerful AI systems would be funded by those systems’ developers, staffed from the same talent pool, governed by frameworks those developers wrote, and answerable to a board those developers help populate.

This is not primarily a story about Google. It is a story about the governance gap.

Every enterprise Google Workspace administrator who received that email has until July 14 to act. Most won’t recognize the legal exposure. Not because they are negligent, but because they don’t have an internal system that connects a product rollout notice to a biometric consent obligation. That connection requires someone who understands both the product and the law simultaneously, knows which states your employees are in, and has authority to act before the default kicks in. Most organizations don’t have that person, that process, or that infrastructure.

Google’s paper proposes solving the governance problem at the federal level, through a frontier regulatory body focused on the most advanced AI models. What arrived in my inbox is not a frontier problem. It is an organizational problem. The risk arrived as a routine admin notice, with a two-week window to act.

The governance framework that would have caught this doesn’t live in Washington. It lives inside the organization, in the layer between the tool and the people responsible for using it responsibly. It is a policy layer, a workflow control, an ownership structure, an escalation path. It is the system that asks, when a vendor pushes a new feature: what does this collect, where does it go, what law governs it, and who in this organization is accountable for the answer?

Most organizations don’t have that system. Google’s paper doesn’t propose building it. And a frontier AI regulatory organization, if it ever materializes, won’t require it.

The governance gap isn’t at the frontier. It’s in the inbox.

Subscribe now

Sources

Google, “A Pragmatic Approach to AI Governance in America,” June 25, 2026. The paper argues existing law is sufficient for widely-deployed AI: “the federal government does not need new regulatory regimes” (p.13), and “if something is illegal to do without AI, it’s illegal to do with AI” (p.13). It proposes a Frontier AI Regulatory Organization (FARO) modeled on FINRA and NERC, industry-funded, with a board combining independent directors and industry representatives. https://blog.google/company-news/outreach-and-initiatives/public-policy/white-paper-ai-regulation/

Google Workspace admin notice, received June 25, 2026 (archived .eml). Google Vids personal avatar feature, built from a user’s face and voice, default-on July 14, 2026. Public corroboration: Google Workspace Updates. https://workspaceupdates.googleblog.com/2026/06/enhanced-ai-avatars-vids.html

Illinois Biometric Information Privacy Act (740 ILCS 14) and Texas Capture or Use of Biometric Identifier Act (Tex. Bus. & Com. Code 503.001). Both require informed, affirmative consent before collecting biometric identifiers, including face geometry and voiceprints.

When a vendor pushes a new feature into the tools you already run, who in your organization connects that notice to the law that governs it, and has the authority to act before the default kicks in? Tell me in the comments.

This is the question my work begins with. The governance gap is rarely at the frontier. It is in the layer between the tool and the people accountable for using it. That gap is what I help close.

Schedule a conversation

I gave a talk on this at Tech Alley Henderson, a practical health check on putting AI to work inside a business without giving up control, accountability, or your data. It covers where the real liability sits, including the Air Canada chatbot case, where a Canadian tribunal held the company responsible for what its bot told a customer. It covers Shadow AI, the unapproved tools already running inside most organizations. And it covers what your free, paid, and enterprise AI accounts actually protect, because the difference is in the contract, not the interface.

If you are responsible for decisions, standards, or outcomes in an organization that uses AI, this is the orientation.

A note on what this is. This is the foundational overview, the wide version. The Evolving Mindset goes the other way each week. Examining the structural problems, in writing, for the people who own the decision. If the talk is the map, the writing is the turns you actually have to take.

Subscribe now

No noise, no hype, no generic AI content. Publishing several times a week.

Nothing about her qualifications changed. Nothing about the role changed. The only thing that moved was a number that, by law, is not allowed to decide whether she gets a chance. The software had been configured to reject women over 55 and men over 60, and it did so silently, to more than two hundred applicants.

That company was iTutorGroup. In August 2023 it paid $365,000 to settle the EEOC’s first-ever AI discrimination case. And the resubmitted application is the most important fact in it, because it shows this was never a mysterious emergent bias buried in a black box. It was a rule. A person chose the threshold. The software just enforced it at scale.

Subscribe now

The move has a name

What iTutorGroup did when challenged is a pattern now common enough to name: decision-laundering. An organization runs a consequential decision through an AI, and when the decision turns out to be illegal or indefensible, it points at the system. The algorithm did it. As if the software, and not the company that bought it, configured it, and switched it on, had made the call.

Decision-laundering is attractive for the same reason money-laundering is. It puts distance between an actor and a consequence. The model feels objective, so the bias reads as math rather than choice. Responsibility diffuses: the quants own the model, not the board; HR treats the tool as a vendor product, not as a hiring decision; everyone can point somewhere else. The decision still gets made. It just arrives with no fingerprints.

The problem is that regulators do not accept the laundering. They look straight through the model to the outcome, and then they look for the owner.

How the laundering actually happens

To see why “the algorithm did it” is not just legally weak but factually false, look at how the bias gets in. It is almost never the model’s invention. It is the organization’s own, passed through and given a clean face.

It enters two ways. The first is inheritance. A model trained on a company’s past decisions learns the patterns in those decisions, including the discriminatory ones. Feed it years of who you hired and who you rejected, who you approved and who you turned away, and it will reproduce that history with statistical confidence and present it as prediction. The Massachusetts Attorney General said exactly this about the lender Earnest: its models were trained, in the AG’s account, on arbitrary and discretionary past human decisions. Human bias went in, was encoded as statistics, and came back out wearing the authority of an algorithm.

The second is proxies. A model does not need the protected trait to discriminate on it. It finds correlates, a zip code, a school, a cohort-default-rate variable, an immigration-status flag, that stand in for age or race or national origin. This is why “we never told it to consider age” is not a defense. The system found age anyway, because age was latent in the data it was given. Removing the label does not remove the pattern.

Then framing finishes the job. Treat the system as a neutral tool rather than a decision-maker, and no one feels responsible for auditing it. The output looks like math, so it escapes the scrutiny a human manager making the same call by hand would attract. iTutorGroup’s screening was treated as a tool. It was a decision, made two hundred times.

The second case makes the remedy explicit

That Earnest matter is worth reading in full, because of what the regulator demanded. In July 2025 the Massachusetts Attorney General reached a $2.5 million settlement with the student lender over AI underwriting alleged to produce disparate harm to Black, Hispanic, and non-citizen applicants, including through an immigration-status knockout rule, with the company faulted for never testing its models for disparate impact.

Now look past the fine. The AG ordered Earnest to implement a detailed corporate governance structure, develop and maintain written policies for responsible AI use, and report on its compliance on an ongoing basis. The state’s remedy for a laundered decision was to force the company to install an owner for it. That is the entire argument of modern AI governance, ordered by a law-enforcement office as a consent term. The regulator did not just punish the bias. It prescribed accountability.

Why “the algorithm did it” has no standing

Put the two cases together and the rule is plain. In both, an AI made a consequential decision about a human being. In both, the decision was unlawful. And in both, the fact that software executed it changed nothing about who was responsible. The deployer owned the outcome, fully, exactly as if a manager had made the same call by hand.

This should not surprise anyone, because in both cases a human did make the call. Someone set the age threshold. Someone wrote the knockout rule. Someone chose the training data and decided not to test it. The AI was the instrument, not the author. “The algorithm did it” is the corporate cousin of “I was only following orders,” and it carries the same weight, which is none.

What the model actually buys an organization is not objectivity. It is speed and distance: the same biased decision, made faster, against more people, with a story attached about why no one is to blame. That is not a mitigating factor. It is an aggravating one, because the harm is now systematic, and the people running the system were more confident in it precisely because it looked technical.

This is also why the defense is being closed across domains, not just one. Employment regulators, state attorneys general in lending, and securities regulators on AI disclosures are converging on the same position: the entity that deploys the system answers for what it decides. There is no agency where “a model did it” has found purchase.

The line between automation and laundering

None of this is an argument against automating decisions. Plenty of consequential decisions can and should be assisted or made by machines. The argument is about who answers for them.

Automation is legitimate when a named human owns the rule the system applies, can explain it, and can defend the outcome to a regulator, a customer, or a court. It becomes laundering the moment no one can. The dividing line is not how advanced the model is or how much data trained it. It is whether a person stands behind what it decides. One is delegation, which is normal and fine. The other is evasion wearing the costume of efficiency.

The governance question, again

The instinct after reading these cases is to ask whether the model was accurate, or fair, or well-tested. Those are real questions, but they are not the first one. The first question is who owns the decision the model makes.

Because the failure in both cases was not fundamentally a modeling failure. It was an ownership vacuum. No named human was accountable for the outcomes the system produced, empowered to see the rule it applied and answer for it. Into that vacuum the model became the de facto decision-maker, and “the algorithm” became the answer to every question about why.

So the test for any organization is concrete. For every consequential decision your AI touches, hiring, lending, pricing, eligibility, termination, name the human accountable for the outcome and ask whether they can explain the rule the system is applying. Not the architecture. The rule. Why this applicant and not that one. If the answer is “the model handles that,” you have not automated a decision. You have laundered one. And as iTutorGroup and Earnest both learned, you own it anyway.

The algorithm did not do it. Someone chose to let it, and chose not to look. That someone is you, and a regulator will find them even if your org chart cannot.

AI operates. You own the decision, including the one you handed to a machine.

For every consequential decision your AI touches, can you name the human who owns the outcome and explain the rule? Tell me where the gaps are in the comments.

This is the question my work begins with. When AI makes consequential decisions and no named human owns them, that gap is what we help close.

Schedule a conversation

Sources

U.S. Equal Employment Opportunity Commission, “iTutorGroup to Pay $365,000 to Settle EEOC Discriminatory Hiring Suit,” August 9, 2023. The EEOC’s first AI-discrimination settlement. iTutorGroup’s hiring software automatically rejected female applicants over 55 and male applicants over 60, violating the Age Discrimination in Employment Act. The charging party was rejected, then offered an interview after resubmitting with only a more recent date of birth. https://www.eeoc.gov/newsroom/itutorgroup-pay-365000-settle-eeoc-discriminatory-hiring-suit

Office of the Massachusetts Attorney General, “AG Campbell Announces $2.5 Million Settlement With Student Loan Lender For Unlawful Practices Through AI Use,” July 2025. Earnest Operations’ AI underwriting models allegedly produced disparate harm to Black, Hispanic, and non-citizen applicants, including via an immigration-status knockout rule, with no disparate-impact testing and models trained on prior discretionary human decisions. The settlement requires a formal AI governance structure, written policies, and ongoing compliance reporting. https://www.mass.gov/news/ag-campbell-announces-25-million-settlement-with-student-loan-lender-for-unlawful-practices-through-ai-use-other-consumer-protection-violations

The code runs. It demos well. It shipped. And no one on the team can tell me what it actually does at the level that matters, because an AI wrote it and an AI was the only thing that checked it. The pattern is consistent enough now that I can name the moment it goes wrong, and it is not a moment of malice or even carelessness. It is a sentence said with confidence: I let the AI write it and the AI review it, so I am good.

That sentence is the failure. Not the AI, not the speed, not the founder moving fast. The sentence.

Because it describes a closed loop. The author and the reviewer are the same machine. A system grading its own output is not a review, it is a second opinion from the same witness. And the human who was supposed to sit outside that loop has quietly stepped out of it, reassured by the one thing that should reassure no one: it runs.

Subscribe now

Running tells you almost nothing. We now have the data to say that plainly. Veracode tested AI-generated code from more than a hundred models across four languages and found that 45% of it shipped with security flaws from the standard OWASP list. In Java the failure rate was 72%. And the most important finding is the one that gets the least attention: as the models improved, they got better at writing code that works and no better at writing code that is secure. The two skills came apart. So “it runs” and “it is sound” are now separate questions, and the loop only ever answers the first.

The human cost compounds it. Stanford researchers found that developers using AI assistants wrote less secure code than those working by hand, and were more confident in its security while doing it. Sit with that. The tool does not just let the review lapse. It produces the feeling that no review was necessary. Confidence goes up while correctness goes down, which is the exact condition under which no one thinks to look.

And often, by the time something goes wrong, the person who shipped it could not look if they wanted to. They cannot read the headers on the code blocks to see what each step actually instructs. This is not a competence insult. It is what the default manufactures. When the loop is closed tightly enough, the human is removed from it so completely that they lose the literacy to re-enter. You cannot own what you cannot read.

Here is where this stops being the builder’s problem and becomes yours.

None of this stays with the shop that built it. The artifact leaves. It ships as a product you license, or as a vendor implementation you approved, and it enters your environment, frequently a regulated one, carrying every gap no human ever caught. And when it fails, the liability does not travel back to the builder. It lands on the deployer. The principle is already settled in plain terms: a company is accountable for what its AI produces, regardless of who produced it. When Air Canada’s chatbot invented a refund policy, a tribunal held the airline to it. The defense that a system, not a person, made the error did not survive. No regulator and no court is going to accept “the AI wrote that part” as a reason the obligation does not apply to you.

So consider the chain honestly, because no one in it did anything wrong. The vendor shipped capability to the customers who contracted for it. The builder kept the product moving. The buyer enabled what they bought. Each act is reasonable. None of them is a review. The review is a separate thing, and it belongs to a named human who read the code and can answer for it. If no one in that chain did it, it did not happen, and now it is running inside your stack.

This is the same question I asked earlier this week, arriving from the other direction. Inside your own operation, the question was who decides what is critical. Through your supply chain, it is narrower and harder: who read the code you are about to run, and can they explain it? One is about the decisions your systems make. This is about the decisions someone else’s system already made, that you are about to inherit.

The inventory question has not changed, it has only moved upstream. Where is AI-built code operating inside what you deploy, and who is the named human accountable for having actually read it? “It runs” is not an answer. “The AI reviewed it” is not an answer. The only answer that holds up, when the thing fails and the question is who owns it, is a name.

AI operates. You own the decision. Including the decision to run code no one ever read.

Where is AI-built code running inside what you deploy, and who actually read it? If you cannot name the person, that is the gap. Tell me what you find in the comments.

This is the question my work begins with. When AI-built code reaches a regulated environment and no named human owns it, that gap is what I help close.

Schedule a conversation

Sources:

Veracode 2025 GenAI Code Security Report. Veracode tested AI-generated code from more than 100 models across four languages and found that 45% shipped with OWASP-class security flaws, rising to 72% in Java. The report’s key finding: as models improved, they got better at writing code that works and no better at writing code that is secure.
https://www.veracode.com/resources/analyst-reports/2025-genai-code-security-report/

Perry, Srivastava, Kumar, and Boneh, “Do Users Write More Insecure Code with AI Assistants?” (Stanford, ACM CCS 2023). The first large-scale user study on the question. Developers with an AI assistant wrote measurably less secure code, and were more likely to believe their code was secure than those working without one.
https://doi.org/10.1145/3576915.3623157

Moffatt v. Air Canada, 2024 BCCRT 149. A Canadian tribunal held Air Canada liable for a refund policy its chatbot invented, rejecting the argument that the company was not responsible for what its own AI told a customer. The clearest statement yet that the deployer owns the output.
https://www.canlii.org/en/bc/bccrt/doc/2024/2024bccrt149/2024bccrt149.html

Read more

That is the number I keep coming back to. It sat on the command dashboard of an autonomous business platform I was reviewing this week. 147 tasks completed today. 3 awaiting your approval. The banner above it read, with no apparent irony, total visibility.

I look at systems like this for a living. Operators bring me t…

Read more

This is not the…

Read more

The AI safety market is organized as a stack. When something goes wrong, the reflex is to add a layer: a new filter, a tighter guardrail, a better gateway, a more granular permission. The instinct is understandable, because the stack is where the visible work happens and where the budget lives. But …

Read more

Read more

Subscribe now

It is a reasonable sounding sentence. It has the cadence of prudence. And it rests on a premise that is now demonstrably false: that somewhere ahead is a rule, a date, a starting…

Read more

The Evolving Mindset

A drive-thru company told the market its AI was taking orders. More than seven in ten of those orders needed a human to step in. The gap between those two facts is the most important number in enterprise AI right now, and almost no buyer can see it.

Share

The company was Presto Automation, a publicly traded restaura…

Read more

Most organizations that believe they govern their AI have built the performance of governance, not the fact of it. The two are nearly indistinguishable from the outside, and often from the inside as well. They diverge at exactly one moment: when a deployed system produces an outcome someone has to answer for. At that moment, the published principles, th…

Read more

For every person you refer who subscribes, you’ll receive something from Fellowship Intelligence’s library - documents I don’t distribute publicly.

How it works

Use your referral link below. Any new subscriber who signs up throu…

Read more

The investment community is debating whether the AI boom is real. That debate is missing the more consequential question: what happens to your organization if the vendors you depend on are less durable than they appear?

There are two pieces of intelligence circulating thi…

Read more

It starts with a business owner telling me they haven’t adopted AI yet. They’re being cautious, they say. Waiting until they understand it better. They’ve seen what’s happening with other companies and they want to do it right when the time comes.

Read more