Ask an AI agent connected to your warehouse a simple question: how many active users did we have last month? It writes a clean, syntactically perfect query. The query runs without a single error, and the number it returns is wrong, with nothing anywhere to tell you so.

The agent is not broken. It inherited a problem your team has lived with for years.

Most data teams scale the hard way. Every new dashboard is one more thing to keep alive. Every metric definition lives in someone's head, or worse, in three separate SQL files, each yielding a slightly different version of the same number. The team spends its days fixing broken reports, chasing metric drift, and writing queries for a demand that never ends.

There is a better investment. Not another dashboard, not a stricter ticket queue, and not one more place to write the same definition down.

A semantic layer.

It takes everything you and your team carry in your heads: how each dataset is built, where it originates, what each field really means, its edge cases and biases. All of it becomes versioned, testable code. Every downstream tool queries through those definitions instead of guessing at raw tables. That single move breaks the query bottleneck, gives the rest of the company real autonomy, and builds the foundation AI agents need before they can work reliably against your data. Imagine an agent that knows your data as well as your best analyst does. That is what this layer makes possible. This is where enterprise data is going, and we are building toward it at Island.

The metric debt problem

Ask three departments what active user means and you will get three answers. Product counts anyone who logged in. Finance counts anyone who paid. Growth counts anyone who clicked an email. None of them are wrong, which is precisely the problem. Each team reinvents core business definitions in isolation. That is the debt: it costs nothing today, and then one day two reports disagree and nobody can say which one is right.

Schema + LLM: the silent failure

You have heard the pitch. Drop your database schema into an LLM, ask questions in plain English, and get instant analytics. No data modeling required.

Here is what actually happens. Because the LLM lacks business context, it guesses what “active” means. It might count deleted accounts, expired trials, or simply every row in the users table, and it hands you the result with total confidence.

The failure is not in the SQL. LLMs are genuinely excellent at syntax; give a model a set of tables and it will write working queries all day long.

What it cannot know is the meaning behind the tables. It does not know that users_v2 replaced the original users table eighteen months ago, so it queries the stale one. It does not know that internal test accounts should never appear in the total, so it includes them. It does not know your fiscal year starts in February, so its calculation for “last month” is off by a week.

Every one of these rules is invisible in the raw schema, yet obvious to anyone who knows the business. Stack these blind spots together, and it is easy to see how a perfectly written query returns a useless number.

That is the trap. Nothing flags the mistake: no error message, no warning, no failed test. The model will not fail on every question, but pointing it directly at the raw schema greatly increases the chances of these silent errors. Without a semantic layer, it can become a fast, highly confident way to generate bad metrics.

The architectural fix: metrics as code

A semantic layer creates a strict, shared vocabulary for your data. Instead of relying on assumptions, you define each business concept exactly once. That definition becomes code. It lives in Git, goes through a standard review process, and ships with tests.

Here is what “active user” looks like when it stops being knowledge in someone's head and becomes a testable definition:

metric: active_users
description: A user who performed at least one action in the period.
owner: data-platform
validated_by: deployment-pm
source: users_v2            # not the deprecated `users` table
filters:
	- exclude: internal_test_accounts
  	- exclude: deactivated
time_grain: fiscal_month    # fiscal year starts in February
definition: count_distinct(user_id) where action_count > 0
tests:
	- no_nulls: user_id
  	- reconciles_with: finance.billed_accounts within 2%

‍

Notice that every blind spot from the previous section is now encoded: the deprecated table, the test accounts, the fiscal calendar. The knowledge did not change. It just moved out of people's heads into a file that machines can read and reviewers can check.

This declarative structure reduces metric drift. It forces every downstream tool, from BI dashboards to production AI agents, to compile its requests against a single, version-controlled source of truth.

The agent belongs in that list on purpose. Dashboards and notebooks benefit from the contract; the agent depends on it, because it is the one consumer that would otherwise guess its way to an answer.

Solving the next-question problem

Static dashboards answer only the questions you already thought to ask. If a stakeholder wants to slice the data a new way, they file a ticket, and everyone is back in the queue.

Picture a Monday morning. A product manager checks the dashboard and sees active users dipped last month. The obvious next question, “Was it one region or all of them?” is not on any dashboard, so it becomes a ticket. The answer arrives Thursday and raises a new question, “Was it new users or returning ones?” which becomes another ticket. Two weeks pass between the first glance and the actual insight, and your team has produced nothing but variations of a query it has written a hundred times before.

A semantic layer changes what that Monday looks like. Because every metric is defined and governed centrally, the product manager, or an agent working on their behalf, can slice active users by region, then by cohort, then by anything else, and every answer resolves against the same tested definitions. The next question gets asked and answered in the same sitting. No ticket, no queue, and no risk of anyone inventing broken logic along the way.

A reality check

Building a semantic layer is real work, and you should know what it costs before you commit to it.

The upfront cost is the obvious part. Someone has to get Product, Finance, and Growth into the same room and force them to agree on a single definition of “active user” after years of using three different ones. This is a political exercise as much as a technical one, and it takes time.

Expect the first useful version to take a full quarter, not a single sprint. You should also expect the modeling process to expose internal disagreements that were much easier to ignore when every department had its own private SQL files.

The subtler cost is what happens later. A semantic layer can become rigid. If every definition requires a review to change, the layer that was supposed to give analysts autonomy becomes the new bottleneck, especially if a single team owns every edit.

It can also be abandoned half-built. A dozen modeled metrics sitting next to three hundred unmodeled ones is worse than none at all, because people no longer know which numbers to trust.

And it only stays accurate if someone maintains it. A semantic layer that nobody updates is just documentation that lies with more confidence.

There is a deeper limit still: the layer is only as good as the knowledge of whoever writes it. A rule nobody thought to encode, exclude test accounts, say, is a blind spot the agent inherits with full confidence, which is exactly why the definitions need business validation and not just a code review.

None of this is a reason to avoid building a semantic layer. It is a reason to build it deliberately. Start with the handful of metrics that actually drive decisions. Split the ownership in two: the data people own the definitions, since they are the ones who know where the bodies are buried in the tables, and business stakeholders own the validation, signing off that each metric matches what they actually mean when they say it. Treat the layer as a product with a maintainer, not a one-time migration.

There is an upside hiding in the pain. Because every definition has to resolve against real data, the modeling process exposes how messy the underlying data is. You will find nulls where there should be values, four spellings of one country, and duplicate events. The semantic layer does not create this technical debt; it refuses to cover it up. Exposing that debt hurts once, and pays off every time you query the data afterward.

What this looks like at Island

We ran into all of this ourselves, with our own audit logs.

The data. Island's browser generates a high-volume stream of audit events: navigations, clicks, tool usage, session activity. It is exactly the kind of raw event data the first half of this post warns about, and like every event stream at this scale, it arrives messy: retries produce duplicate events, optional fields arrive empty, and test traffic mixes with real traffic. That is not a flaw in the pipeline; it is what raw looks like before anyone gives it meaning.

The challenge. We wanted to know whether an AI agent, an LLM with the ability to query this data and reason over the results, could produce something genuinely useful. Pointed at the unmodeled logs, it could not. The agent wrote confident queries against events it did not understand and got the answers wrong, the same silent failure playing out on our own data.

What we built. So we built the layer first. The same principle from earlier in this post applied, only the unit changed: instead of defining metrics, we defined behaviors. We classified hundreds of raw event types into categories that carry meaning: data movement, authentication, threat detection, AI usage. We encoded which events are signal and which are noise, and defined what an enforcement verdict means in business terms. And just like the metric definition earlier, each behavior ships with verified queries: question-and-answer pairs that pin down what a correct result looks like.

What became possible. With the layer in place, we ran a series of experiments, explorations of what the architecture enables, not shipped features, and some have since grown into standing agents that run continuously against it. We validated the results two ways: against ourselves, colleagues whose roles and daily routines we already knew, and against a test set of questions with expected answers. Each experiment started as a question.

Experiment 1: The person. Could an agent tell what a person does from how they browse? Role data in most organizations is stale or wrong; behavior is ground truth. The agent reconstructed a daily timeline for a heavy browser user and, from navigation patterns alone, inferred the person's job role. No directory data, no job title, just the meaning of the clicks.

The results held up. The agent's inferences matched the roles we actually knew, and that reconstructed day became the raw material for the next question.

Experiment 2: The routine. Could it spot what that person repeats? The agent analyzed the daily routine for recurring patterns and found one immediately: the user opened the ticket queue, copied the ID, checked the billing status in the admin console, tagged it, and moved on -  fourteen times a week. In the raw logs, this loop was invisible. Reasoning over defined events, the agent recognized the loop as one workflow, even when the run was imperfect: in some repetitions the user broke off mid-sequence to check a calendar or answer a message, detours that would fragment a rigid pattern matcher, and the agent read them as noise and kept the count intact. It flagged the loop as a candidate for automation, the first step toward agents that automate work around how each person does it.

Experiment 3: The question nobody asked. Could it find a risk no one had thought to look for? Insider risk breaks the rule-based model: the actor is authorized, every action is allowed, nothing fires. So instead of a rule, we gave the agent an open question: does anyone's data movement stop making sense? It came back with one account: 8,382 copies out of corporate applications in a single month, 2 pastes back into the browser, active all 31 days, on 8 devices, in two countries. The rest of the organization copies and pastes in rough balance; this account only takes.

No rule caught it, because no rule was broken. No dashboard tracked it, because nobody anticipated the question. And on raw logs even the right question comes back as a confident ratio made of noise. The finding needed both halves, a layer that made the question answerable, and an agent that thought to ask it.

The agent closed the loop the way a good analyst would: an investigative lead with the evidence attached, not a verdict, since the missing pastes may land in apps outside the browser's view. But it is a lead no rulebook could have handed anyone.

One catch, though: now that the pattern is known, it can be written down as a rule and queried forever. So is the agent just a machine for discovering next quarter's dashboards? A second finding from the same period answers that. The agents caught a user reaching a blocked gaming site through a web proxy. On paper, the session was spotless: every request went to an unremarkable proxy domain, nothing matched a blocklist, no policy fired, because the destination the rules were written against never appeared in the traffic. The evidence lived entirely in what the pages meant, not where they were hosted. And unlike the ratio, this detection never becomes a query: there is no field to threshold, no pattern to match, a different proxy and a different site tomorrow, and the only constant is the intent. Some of what the agent finds becomes a rule. Some can only ever be answered by something that reads meaning. The layer plus the agent covers both.

Experiment 4: The story behind the alert. Could it turn an alert into an investigation? Detection is where most tools stop: the product flags that a user entered corporate credentials on a phishing page, and then a human spends the afternoon reconstructing everything around it. We handed the agent the alert instead and asked for the full story. It walked the defined events backward and forward: where the link arrived, what the user did in the minutes before clicking, what the account touched in the session that followed, what was exposed, and whether anyone else received the same lure. What an analyst assembles in hours of log-digging came back as a readable narrative, every step tied to the events behind it, not a new detection, but the triage around an existing one, which is where investigation time actually goes.

The same agents have since surfaced a coordinated phishing campaign spanning dozens of users, and malware beaconing hidden in ordinary browsing traffic, and, just as tellingly, have classified a noisy penetration test as authorized activity rather than an attack.

Where this points. Each question above used to mean a research project: assess feasibility, clean the data, build, validate. With a layer the agent already understands, each new capability is just the next question, and what used to be a project becomes a query. These are samples, not the point. That is the real product of the semantic layer, a foundation you can keep building on, all the way to a workplace browser that improves itself.

Build the contract before the agents arrive

The hard problem in modern data infrastructure is no longer scaling storage or compute. It is scaling meaning. A semantic layer is to your data what a type system is to your code: you can write software without types, and you can query a warehouse without a semantic model, and both work fine until the day they do not. At scale, the absence of a formal contract is just chaos with a delay timer on it.

The teams building semantic layers today are laying the groundwork their AI agents will stand on. A semantic layer does not prevent hallucinations, but it ensures the agent reasons over validated business logic instead of guesses. Teams without one will spend the next few years debugging why their agents return wrong numbers, blaming the model for a problem that actually sits in the data pipeline.

A semantic layer is the contract your organization needs in place before AI becomes the primary consumer of your data. The time to build it is not when the agent starts hallucinating. It is now.