Data Semantic Layer: Closing the Gap Between Where Science Happens and Where Data Lives
Real-world life sciences data is hard to work with because the questions live in the language of science and the data lives in tables, codes, and join order. Today the two ways across that gap are slow and laborious manual data assembly or a fast agent whose answers are inconsistent, incorrect, and unverifiable. This third post in the Harness Engineering series is about a third way: a data semantic layer that lets people ask in the language of science and guarantees the translation underneath.
In my first post in this series, I argued that the harness, the software around the model, is where the main leverage is right now, and that building one for life sciences comes down to three things: the action space, the data, and the governance. The second post covered the action space, and the claim that for real technical work, code is all there is.
This post is about the second of those, the data. In my first post, I also argued that context engineering the data is not preparation for the work; it is the work. Here, I want to explain why and what that work looks like.
Looking into the gap
Working with real-world life sciences data is challenging, and the reason has nothing to do with its size. The questions live in one place while the data lives in another.
On one hand, a scientist, a medical affairs lead, or a commercial analyst asks in the language of science: genes, variants, cohorts, medications, what happened to this group of people and in what order. On the other hand, the data answers in a completely different language: tables, grain (what one row actually represents), join order, codes, and whatever conventions the site that produced it happened to use. Between the two sits a gap, and every question a team asks has to cross it.
Historically, crossing this gap has taken two people: one who hand-curates the data into a usable shape, and another who knows the resulting layout well enough to query it correctly. That's why a single question takes days, and why the number that comes back is hard to trust.
Two ways to cross the gap, and why neither works
The first way is what most organizations do today. Weeks of hand-wrangling data before a single question gets answered. It works, and it's the reason the field has any answers at all. But it's artisanal and slow.
The second way is the tempting shortcut now that text-to-SQL agents exist. Point an agent at a Snowflake warehouse, hand it the schema, and let it write arbitrary SQL. It will answer anything you ask immediately. That's the problem.
The answer is not consistent. Ask the same question next week, and the model writes different SQL and hands you a different number, with nothing to say which one was right.
The answer is not correct. The natural way to write a question in SQL is often the wrong way. Asked for a KRAS G12 cohort, a model reaches for LIKE 'G12%', which on real data matches two adjacent codons and returns a plausible, wrong count.
The answer is not verifiable. The reason those two failures are fatal and not merely annoying is that nobody can catch them. A number comes back, and the dozen judgment calls the query encoded stay invisible: which specimens counted, whether a patient with three biopsies counted once or three times, what "progressed" meant in this dataset.
So pick your poison: slow and artisanal on one end, fast and untrustworthy on the other.
Is there a better way to cross this gap?
There is, and it starts from a different question. Instead of asking how to help people write better SQL, ask how to let them query data in the language of science and guarantee the translation underneath. We call that the data semantic layer. It is the layer between people and real-world data that translates questions asked in the language of science (genes, cohorts, lines of therapy) into correct queries over the data's actual tables, codes, and grain, so the same question returns the same, verifiable answer.
We can build the data semantic layer only because we're verticalized. Our harness (the software around the model that decides what it can reach and how it's allowed to work) is built for a single domain: commercial real-world data (RWD) in life sciences. In that domain, three things are finite and small.
The questions are finite.
Across every customer we've worked with, they fall into a handful of families that recur regardless of whose data it is. For instance: who carries this variant, who expresses this gene highly, who switched therapy and when, whose lab values crossed a threshold.
The representations are finite.
Each family of questions needs the data to land in a particular shape, so a short list of opinionated shapes covers the whole space. Variant questions want a sample-by-variant table with annotations joined. Longitudinal questions want clinical events on a common grain of person, coded concept, and date, read in order.
The sources are finite.
Because the sources are finite, the ingest work is bounded. The formats are stable and few: RNA-seq count matrices, VCFs, , EMR extracts, registries, claims. Every instance carries its own vocabulary and conventions, but what arrives at the door is not a long tail.
That chain runs in one direction. The questions determine the representations, and the representations determine what the ingest work has to accomplish.
The table is deliberately incomplete; the rows keep coming. What matters is the pattern it describes.
None of this is new. Doing it affordably is.
Harmonized representations are decades old. OMOP is the obvious example, and we took inspiration from it and from several others. The idea that you should model clinical events on a common grain is not ours and never was.
The reason it hasn't been the default is economics. Getting one messy source into a harmonized schema meant a data engineer sitting with a domain expert for weeks, working through local codes and dedup rules and which field means what. That cost is why most data never got mapped. It stayed in its native shape, and the gap stayed open.
Agents change that price. The mapping is judgment-heavy, but it's judgment a model can apply given the right context, and where it can't, it can ask.
So each kind of source gets a paved road. Most of that road has no decisions in it: parse, decode, dedupe, land the file decoded but not yet reshaped. That stretch is ordinary data engineering, running as versioned DAGs, and it should stay boring.
Agents sit on top and call those DAGs as tools. They identify what a source is, route it to the right paved road, and handle the part that needs judgment: mapping local vocabularies to ontologies, deciding which domain a field belongs in, reconciling dates and duplicates. When they can't infer something, they interview the customer's domain expert and write the answer into a transform.
A human is still in the loop. What changed is that the agent can interview the user and then rapidly profile, transform, and verify the data.
How do agents translate a scientific question into a query?
Landing the data in the right shape isn't the whole job. Somebody still has to work out what the person actually wants
Questions arrive in the language of science, and they arrive underspecified. "Which patients failed first-line therapy?" doesn't say what counts as a line of therapy, or what counts as failure, or whether patients with no follow-up should be dropped. The agent's first job is to read the intent behind the question, and when it can't resolve it, to ask instead of guessing. That clarifying step is what makes everything after it mean something.
Once the intent is clear, it maps onto a query path: which representation to enter through, what to resolve first, what to join to what, in what order, and what to count at each stage. Resolve the gene symbol to coordinates before touching the variant table. Carry the resulting person set to the medication table rather than the other way around. Count patients, not specimens.
Context is what makes that mapping possible. The agent carries context about how the data is laid out: the schema, the grain, what a row means, which keys are clustered and therefore which direction a query has to enter from if it's going to return at all. It also carries context about the analysis itself: how a question of this kind gets parsed, and what sequence of steps correctly fulfills it.
That's why we run many narrow agents rather than one general one. An RNA-seq agent knows how to do expression work. A clinical query agent knows how to work with longitudinal data. Each agent is good at a small number of intents, and they can compose when a question spans them.
Consistent, correct, and verifiable
These are the three things a freewheeling agent over a warehouse can't give you, and they're what the data semantic layer exists to produce.
It's consistent. The same question routes to the same path over the same representation, so it returns next week what it returned today. And when the number does move, you can tell whether the data changed or the definition did.
It's correct. The judgment calls are encoded in the representation and the query paths rather than improvised per question. Codon position is stored as a number, so a G12 cohort is a numeric comparison that can't swallow its neighbor. Counting patients rather than specimens is written into the path.
It's verifiable. The agent explains itself in several views of the same answer: the SQL, a visual breakdown of the filtering, a written account of what it did and why, and lineage graphs tracing each number back through the data. A bioinformatician checks one, a clinical scientist another.
Models keep getting better, and as they do, the tail of questions the layer can serve gets longer. That's the bet on harness work: the representations and the context compound, and the intelligence sitting on top keeps improving without us doing anything.
The action space gives an agent the ability to act. The data semantic layer gives it something true to act on. The remaining piece, governance, is what decides what it's allowed to touch. We’ll pick up there next time.
