Summary
Data lake stores manufacturing values. A data foundation knows which batch produced them, which equipment was running, which specification applied at the time, and which document page they were read from. That difference explains why AI pilots in pharmaceutical manufacturing clear proof of concept and then stall before production: the model works, and the contextualisation the data scientist did by hand never got built into a pipeline.
This article defines what a data foundation is in a GMP context, why batch-level context is a requirement that lakehouse and data mesh architectures do not supply on their own, and what changed once regulators began treating data provenance as part of a model’s credibility case.
It covers the five parts a working foundation needs, the architectural seam where most programmes lose their value, and how to scope one so it returns results in weeks rather than at the end of an eighteen-month build.
A process engineer needs to know which batches used a suspect raw material lot. The answer exists. It sits across an ERP goods movement table, an MES step record, a historian tag, and scanned batch records in a CDMO portal. Assembling it takes nine working days. This is where AI investment keeps landing. Gartner predicts organizations will abandon 60% of AI projects unsupported by AI-ready data through 2026, and MIT’s Project NANDA found 95% of enterprise generative AI pilots produced no measurable P&L impact. Neither finding describes a modeling problem. Both describe a data foundation for AI in pharma that was never built to carry a regulated decision.
The industry agrees on the diagnosis and not on the definition. Most of the vocabulary in use arrived from enterprise IT, where the worst outcome of a bad join is a wrong dashboard. In a GMP environment the record is evidence, the batch is the unit of meaning, and a reviewer may want to reconstruct a decision seven years later.
What Is a Data Foundation for AI in Pharma?
A data foundation for AI in pharma is the governed layer that ingests every record a manufacturing process produces, contextualizes each value against the batch and unit operation it belongs to, harmonizes it across sites and systems, and preserves lineage back to the source. A data lake stores the value. A data foundation knows which batch produced it, which equipment was running, which specification applied, and which page of which document it was read from.
Data foundation: The contextualized, governed, and traceable data layer beneath analytics and AI, in which every manufacturing value is linked to the batch, unit operation, material lot, equipment, specification, and source record that give it meaning.
Storage layer
Data lake
Centralises raw data from every source system in its original format.
- Holds the value
- Preserves the file
- Batch link exists in a person’s head
- Lineage reconstructed on request
Contextualisation layer
Data foundation
Adds the context and governance that make a value meaningful and defensible.
- Knows which batch produced it
- Knows which specification applied
- Batch link held in the schema
- Lineage captured by default
Why Pharma Data Foundations Are Different
Three properties make the data foundation load-bearing in pharmaceutical manufacturing in a way it is not elsewhere. The unit of meaning is the batch rather than the transaction. The record is legal evidence rather than an input. And the answer has to be reconstructable years after it was produced.
A temperature reading of 37.2 carries no information alone. It becomes usable with a batch ID, a unit operation, an equipment ID, the material lot in process, the specification in force, the operator, and a timestamp aligned to the same clock as the LIMS result it will be compared against. The historian carries equipment and time. LIMS carries sample and result. MES carries step and operator. ERP carries lot and movement. The batch that ties them together lives in a senior engineer’s head and in nobody’s schema.
ALCOA++ expectations apply to the data a model consumes exactly as they apply to the executed batch record. A pipeline that discards the audit trail during transformation has destroyed the property that made the data admissible, which rules out the ad hoc extract work most pilots run on.
The five parts of a pharma data foundation
Each is a common failure point, and the failures compound downstream.
- 1Ingestion that includes paper Pharma-specific OCR for handwriting, checkboxes and initials, confidence scoring on every field, and an audited human review step.
- 2Contextualisation at batch and unit-operation level Every parameter, result, material lot and deviation linked to its batch and process step, and batches linked through the materials between them.
- 3Harmonisation and governance Site A records pH, Site B records Acidity. Until those reconcile, no cross-site trend is possible.
- 4Lineage down to the page Document, page and section a value came from, extraction confidence, and every change captured with attribution.
- 5A serving layer applications and agents can query Charts, rule engines, models and copilots all reading the same structure rather than private extracts.
The Five Parts of a Pharma Data Foundation
- Ingestion that includes paper. For asset-light manufacturers running through CDMOs, paper is the dataset. Executed batch records, CoAs, and deviation forms arrive as scans. This needs pharma-specific OCR for handwriting, checkboxes, and initials, confidence scoring on every field, and an audited human review step.
- Contextualization at batch and unit-operation level. Every parameter, result, material lot, equipment record, and deviation linked to its batch and process step, and batches linked through the materials flowing between them. Forward traceability turns a supplier notification from a three-week investigation into a same-day hold decision.
- Harmonization and governance. Site A records pH. Site B records Acidity. Both mean the same measurement. Site A reports Fahrenheit, Site B reports Celsius. Until those are reconciled, no cross-site trend is possible and a model trained across both learns a site artifact rather than a process signal.
- Lineage down to the page. Lineage that stops at “came from LIMS is not” enough. It has to identify the document, page, and section a value came from, records extraction confidence, and captures every change with attribution. The test: an inspector asks where an APQR number came from, and the system displays the original scanned page inside the same conversation.
- A serving layer applications and agents can query. Charts, rule engines, models, report generators, and copilots all reading the same structure. The moment two applications maintain private extracts, they will eventually disagree, and reconciling them becomes a compliance exercise.
Why the Data Layer and Application Layer Belong Together
The common architecture places the data foundation with one vendor and the applications with several others. Context gets built in the data layer, flattened into extracts on the way out, and rebuilt inside each application with different logic. That seam is where the value leaks, and it stays invisible until someone asks two applications the same question.
Batch yield is the usual example. The CPV chart calculates it from historian and MES data. The APQR report calculates it from an extract prepared three months earlier. The deviation investigation calculates it by hand. Each is defensible alone. Presented together during an inspection, they raise a governance question nobody wants to answer under pressure.
| Dimension | Separated layers | Unified layers |
| Where context lives | Rebuilt in each application from flat extracts | Defined once in the genealogy graph |
| Conflicting numbers | Yield and deviation counts diverge across CPV, APQR, and investigations | One calculation, one definition, one lineage path |
| Investigation round trip | Tracing a flagged exception becomes a separate manual project | Flag links back to batch, material lot, equipment, and source page |
| Validation burden | Two or more validation packages; schema changes cascade upward | One validated stack; change control follows a single path |
| AI grounding | Copilot answers from a document index or flat extract | Copilot answers from the contextualized graph with traceable lineage |
Separated layers
Data platform from one vendor, applications from several others.
charting
reporting
investigation
Each application rebuilds context from a flat extract with its own logic. Batch yield is calculated three ways. Two validation packages, and a seam neither vendor owns.
Unified layers
Applications sitting directly on the contextualised graph.
charting
reporting
investigation
One calculation, one definition, one lineage path. A flagged exception links straight back to batch, material lot, equipment and source page.
Mareana is built on the opposite premise. The data layer contextualizes MES, LIMS, ERP, QMS, historian, PAT, and paper into a batch genealogy graph, and the applications sit directly on it: exception-based batch review, CPV charting, multivariate modeling, APQR generation, and a copilot grounded on the genealogy itself through GraphRAG rather than a document index. Documented deployments have reduced deviation investigation time.
Building One Without an Eighteen-Month Program
The sequential model, fix the data first and do AI afterwards, is why these programs get cancelled at month eleven. A foundation producing no operational value until complete has no defenders at budget review. Build it in the shape of a decision someone is already struggling to make.
- Start with a business decision, not a dataset. “Contextualize all manufacturing data” is an endless goal. “Release this product family without a line-by-line paper review” is a clear objective that tells you exactly which data and rules to prioritize.
- Start with your most complex data source. If you build your foundation around historian data first, you may later discover it can’t support executed batch records. Design for the hardest manufacturing data first, and everything else becomes easier to support.
- Design for a moving standard. ICH Q9(R1), GAMP 5 Second Edition, FDA’s 2025 credibility framework, and the January 2026 FDA-EMA joint principles all shifted within three years.
The industry has been conditioned to believe contextualizing manufacturing data takes years, large budgets, and an army of SMEs. That was accurate when it meant manual SQL modeling and master data preparation. Graph-based relationship extraction has changed the arithmetic, and the first useful results now arrive in weeks.
Key Takeaways
- A data foundation is a contextualization and lineage layer, not a storage layer. A data lake holds the value; a foundation knows what gives it meaning.
- Batch-level context is a pharma-specific requirement that data mesh and medallion architectures do not supply on their own.
- Provenance is now a regulatory artifact under FDA’s 2025 credibility framework and the 2026 FDA-EMA joint principles.
- Separating the data layer from the application layer may create conflicting numbers, a broken investigation round trip, and duplicate validation burden.
Learn more
