Validated AI retrieval over SOPs and deviations for San Diego biotech under Part 11

A deviation opens on a Tuesday in Sorrento Valley. The investigator needs the current version of three SOPs, the last two deviations on the same line, and the CAPA that closed them. Each lives in a different system. The write-up takes four days, and the reviewer still finds a reference to a superseded procedure.

This article is for the head of quality or the VP of operations at a clinical-stage or commercial biotech in San Diego, and for the validation lead who will be asked to sign the case.

What does retrieval over controlled documents actually do?

It answers a question from the controlled documents in force on the date asked, cites the document, section and version, and refuses to answer from anything else.

That is a narrower thing than a chatbot and a more useful one. The investigator asks which SOP governs line clearance on the fill suite. The system returns the passage, the document number, the version, the effective date, and a link into the document management system. If the question is not answered by any controlled document, it says so. It does not improvise a procedure.

Applied to deviations, the same system surfaces prior events with similar descriptions, the investigations that closed them, and the CAPAs that followed. The investigator still does the investigation. They start it with the record in front of them instead of a search.

Is this a GxP system?

If its output can influence a quality decision or enter a quality record, yes, and it should be treated as one from the first day.

That means an intended-use statement, a risk assessment, a validation plan proportionate to the risk, change control, and a Part 11 audit trail. Part 11 requires that computer-generated audit trails record the date and time of operator entries and actions, that the system limit access to authorised individuals, and that records be retrievable for the retention period. A retrieval system meets those with the same disciplines as any other quality-system software.

The good news for QA is that the validation case is familiar. The FDA’s Computer Software Assurance guidance shifts effort toward critical thinking and risk-based testing rather than exhaustive scripts. GAMP 5’s second edition addresses machine-learning components explicitly. And the FDA’s January 2025 draft guidance on AI for regulatory decision-making lays out a seven-step, risk-based credibility assessment tied to a defined context of use. A retrieval tool that helps an associate find the right passage sits low on that risk scale. A tool that drafts the root-cause section of a deviation sits higher, and its case is written accordingly.

How do you validate a system that is not deterministic?

With a test set agreed in advance, acceptance criteria written before the run, and a re-run on every change.

Language models do not give byte-identical output. Retrieval systems are more stable than that reputation suggests, because the passages retrieved are deterministic and the model is instructed to answer from them. The validation approach is the one QA already uses for an analyst: a defined set of questions with acceptable answers, a pass rate, and documented review. The set is run at release, on every SOP revision that touches the scope, and on every model or software change. Failures are deviations in the ordinary sense and go through the ordinary process.

Version pinning is what makes the re-run meaningful. The system holds every document version with its effective date and answers against the version in force on the date of the question. The audit trail records which version was retrieved, so an answer from March can be reproduced in September.

Where does the data live?

Inside the company’s own cloud tenant, with the document management system as the source of record and the retrieval index as a controlled copy.

The index is regenerated from the document management system on a schedule and on every approved change, so it can never be more current than the controlled documents. Nothing is sent to a model vendor for training. Access follows the same roles as the document system, and the audit trail lives with the rest of the quality records. Investigations that are open, batch records, and anything under legal hold stay out of scope until a separate change control brings them in.

What should the company measure?

Days from deviation opened to investigation submitted, and the number of citation errors found at review, before and after.

Both come out of the quality system’s own reporting. The first is what the site head cares about. The second is what QA cares about, and it is the number that justifies the validation effort. We agree the baseline in writing before the build, run the pilot for four to six weeks on one site or one product family, and report the result plainly. We do not publish client numbers in these articles, and any range from a vendor is illustrative until measured on your records.

What has to be true before the pilot starts?

1. A document management system with effective dates and version history the tool can read.
2. An intended-use statement QA agrees with. Narrow beats broad for the first pilot.
3. A validation lead who will own the test set and sign the summary report.
4. A supplier assessment of the cloud and model vendors under the existing supplier-quality process.
5. Baselines for the two metrics above, on the site or product family in scope.

What does this look like in the QA office?

An associate in Torrey Pines opens a deviation. The system has already listed the governing SOPs at their current versions, the two prior deviations on the same line, and the CAPA that closed them, each with a citation. The associate writes the investigation with those open. The reviewer checks citations against the versions listed instead of searching for them. The audit trail shows who asked, what was retrieved, at which version, and what was written.

That is a validated system. It is also just a very good filing clerk who never cites the wrong revision.