Policy retrieval an NCUA examiner can audit: AI for Orange County credit unions

A member calls a branch in Irvine to ask whether a skip-a-payment applies to a loan that was modified last year. The rep opens the loan policy, then the modification procedure, then the memo from March that changed the modification procedure. The member waits. The answer is right, probably. The examiner’s question, months later, is how anyone knows.

This article is for the COO or chief risk officer of a credit union in Orange County or San Diego, and for the chief compliance officer of an RIA who has the same problem with a different manual.

What does policy retrieval do that a search box does not?

It answers the question in plain language, from your approved documents only, and shows the passage it relied on.

A search box returns documents. Retrieval returns an answer with a citation. The technical term is retrieval-augmented generation: the system finds the relevant passages in your policy manual, hands them to a language model with the instruction to answer only from those passages, and returns the answer with the source, section and version. If the documents do not contain an answer, it says so and routes the question to a supervisor. It does not improvise.

That last property is what makes the system examinable. Every answer traces to a document that a committee approved on a date.

What will the examiner actually ask?

Four things, and they are the same questions they ask about any system that touches operations.

Where does the data live and who can reach it? Part 748’s Appendix A requires a written information security program that protects member information. A policy retrieval pilot can be designed with no member data at all, which keeps it out of the highest-risk tier. When member records are added later, the data stays inside the credit union’s own cloud tenant, access is by role, and the model vendor does not train on any of it.

Who is the vendor and how did you diligence them? NCUA’s Letter 07-CU-13 on third-party relationships still frames examiner expectations: planning, due diligence, contract terms, and ongoing monitoring. The AI vendor and the cloud provider both go through that process, and the contract has to cover data use, breach notice, and exit.

How do you know the answers are right? NCUA has no model-risk rule of its own. Examiners tend to borrow the framing of the Federal Reserve’s SR 11-7: a model needs an owner, documentation, validation before use, and ongoing monitoring. For retrieval, validation is a test set of real member questions with the answers a supervisor would give, run before go-live and again every time the policy manual changes.

What happens when it goes wrong? Since September 2023, 12 CFR 748.1(c) requires a credit union to notify NCUA within 72 hours of a reportable cyber incident. The AI system sits inside the incident response plan like any other system. Its logs make the notification easier to write, not harder.

What about an RIA?

The same design with a different manual, and two SEC rules that make the audit log worth having.

The Advisers Act compliance-program rule requires written policies and an annual review. A retrieval system over the compliance manual, Form ADV and client agreements gives every associate the same answer with the same citation. The books-and-records rule means the log of questions and answers is a record; keep it accordingly. The 2024 amendments to Regulation S-P require an incident response program and customer notification within 30 days of a breach involving sensitive customer information, with compliance dates in December 2025 for larger advisers and June 2026 for smaller ones. An AI system that touches client data has to sit inside that program from the start.

What does the audit trail record?

Who asked, what they asked, which documents and versions were retrieved, what the model answered, and what the person did with it.

That is five fields, and the fifth is the one most vendors leave out. When a rep edits an answer before giving it to a member, or overrides it, the log should say so. That is the evidence that a human remained accountable. It is also the fastest way to find the policy passages that confuse people, which is useful to the compliance team on its own.

Version pinning matters as much as logging. If the loan policy changed on the fifteenth, an answer given on the tenth should be reproducible against the version in force on the tenth. The system stores document versions with effective dates and retrieves against the one that applied.

What should the credit union measure?

Average handle time on policy questions and first-contact resolution, measured on the same call types before and after.

Both numbers already exist in most contact-centre reporting. A pilot that cannot move them has not earned a rollout. We agree the baseline in writing, run the pilot for four to six weeks on one department, and report the result plainly. We do not publish client numbers in these articles, and any range from a vendor should be labelled illustrative until measured on your calls.

What has to be true before the pilot starts?

1. An approved, current policy library. If the manual has three versions on a shared drive, the Roadmap’s first job is choosing one.
2. A named owner in compliance who will sign off on the test set and review flagged answers.
3. A decision to keep member PII out of the first pilot, which we recommend.
4. Vendor diligence files for the cloud and model providers, using the credit union’s existing third-party process.
5. Baselines for handle time and first-contact resolution on the call types in scope.

What does this look like in a branch?

A rep in Carlsbad types the member’s question. The assistant returns a two-sentence answer, the policy section, the effective date, and a link. The rep reads it to the member, or corrects it and marks why. The log records all of it. When the examiner asks how you know the answer was right, you show the citation, the version, the test set, and the rep’s name.

That is the whole system. It does not replace the rep. It replaces the binder search.