Use caseAI governance

Screening the corpus of an AI assistant

Customer service wants an assistant fed by three years of reply letters. Those letters hold names, addresses, account numbers. Who checks 18,400?

Every document is screened before the assistant can read itRules approved by the privacy officer, versioned, applied by batchLeak tests logged and replayable before go-live

The screeningScreening run, Nov 5

18,400 documents read against rule set v1.2. 15,630 masked copies enter the assistant's collection, 412 are kept out.

source · 4 collections in the repository
Actions on this corpus
Each document

Personal data found · the masked copy enters

01

Why it drags today

The core idea

What does an assistant do with a reply letter that holds a customer's bank account number? It quotes it, to whoever asks. The model works as built; the trouble is a corpus nobody screened, down to a model letter that still names the first customer it was written for. Today the project team finds out in testing, or worse, from an advisor.

Screening goes first, before a single answer. The operations engine reads every document against rules the privacy officer approved, masks what must not be quoted, keeps out what must not enter, and holds the rest for a person. The assistant reads only the screened collection, and the privacy officer signs the leak tests before anyone uses it.

Model letter ML-204Viewer
One letter, a hundred copies. ML-204 still names its first customer. It waits in quarantine with the other documents the rules could not settle.
Assistant project, customer serviceOperations engine
Six days to go-live. Rules signed, corpus screened, exceptions ruled on, leak tests signed, and only then the advisors.

The held letter and the six-day log are two faces of one discipline: the rules are written before a document is read, the doubtful ones wait for a person, and the assistant goes live on tests the privacy officer has signed. The corpus is built below in that order, then watched after go-live.

02

The rules

Before a single document is read, the team writes down what is in scope and what happens to each kind of personal data.

Step 1Admin

The corpus is chosen collection by collection

Daniel Okafor, the IT director, selects four collections in the repository: procedures, product sheets, model letters and reply letters sent since 2023. That is 18,400 documents. He leaves out the complaint files and the HR spaces, and records why on the project.

Assistant project, customer serviceRepository
Advisors will see through the assistant only what they can open today.
The scope is a written decisionWhich collections, for whom and why. The privacy officer reads it before the first run, and it stays on the project.
Step 2Rules

The screening rules are written with the privacy officer

Daniel and Karen Mitchell, the privacy officer, set one action per category of data. Names and addresses in reply letters are masked. Bank account and card numbers are masked everywhere. Any letter that mentions health is kept out. A document the AI is unsure about is held for a person.

Customer name: maskAccount, card number: maskHealth data: keep outLow confidence: hold
Screening rulesOperations engine
The rule set has a version. Each batch records which version processed it.
The privacy officer approves the rules onceShe signs what may enter once, with a version number. Every document after that is screened the same way.
Step 3Operations

The first screening run

The operations engine reads the 18,400 documents and runs rule set v1.2 on each. Clean documents enter the assistant's collection as they are. Documents with personal data enter as masked copies. The rest are kept out or held, with their reason. The originals in the repository are not touched.

Screening run, Nov 5Operations engine
A document that failed a control is listed with its reason, ready to be handled.
The assistant never sees an unscreened pageWhat it reads is the screened collection. Every copy in it says which rule version made it.
03

The quarantine

The rules settle most documents. The 148 held ones need a person, and Priya starts with the model letters.

Step 4Viewer

Review what the rules could not settle

Priya Natarajan, the knowledge manager, opens the quarantine. A model letter, which should hold no personal data, still carries the name and address of the customer it was first written for. A procedure includes a screenshot with an account number on it. She starts with the model letters, since advisors will reuse them most.

Model letter ML-204Viewer
The AI flags the personal data and proposes the masks. Priya accepts or rejects each one.
The riskiest documents are seen by a personA model letter with a real name would be copied a hundred times. It is caught before any advisor sees it.
Step 5Viewer

The masked copy is released

Priya accepts the two masks on ML-204. The masked copy enters the assistant's collection, and she sends a correction task to the owner of the model library so the original gets fixed too.

Model letter ML-204Viewer
The original and the copy the assistant reads, side by side.
The letter stays useful as a modelThe wording advisors need is kept. Only what belonged to one customer is hidden.
Step 6Workflow

The privacy officer rules on the exceptions

Some requests go beyond a mask. The claims team asks for 40 reply letters about medical expenses to be included, because they explain a reimbursement rule well. Karen refuses for this version, and suggests the team write one model letter that explains the rule with no health data in it.

ExceptionsCase management
An exception is a written decision by the privacy officer. The rules never widen on their own.
Ask it of your own pilot: when the assistant gives a good answer, can you name the rule that let each quoted document in?
Every exception has a name on itThe rule set and its exceptions are versioned together. The privacy officer can show what she allowed, and when.
04

The go-live

The collection is clean on paper. Before advisors touch it, the team does its best to make it leak.

Step 7AI console

Leak tests before go-live

Daniel runs 60 test questions built to pull personal data out: a customer's account number, the address behind a contract number, who wrote about a hospital stay. Every call is logged with its question, its answer, the model and the time. Karen reads the answers. None returns real data, and she signs the test report.

Leak testsAI console
The test set is kept and replayed after every change to the rules.
Go-live rests on a signed testThe report is attached to the decision, and the same questions can be replayed months later.
Step 8Assistant

An advisor asks, the answer stays clean

Kevin Brooks has a customer on the phone who wants to move her payment date. He asks the assistant what to do and what to send. The answer cites procedure P-042 at its page and model letter ML-204. Kevin fills in the customer's details from her own file.

Procedure P-042AI console
Each answer points to the document and page it comes from.
The advisor gets the wording and nothing moreThe answer is the procedure and the model. The customer's details come from her own file.
Step 9Operations

New documents are screened before they enter

After go-live, every new reply letter and every updated procedure goes through the same rule set before the assistant can read it. The flow runs for as long as the collections live, and held documents land in Priya's queue.

Assistant project, customer serviceOperations engine
Each batch keeps its trace: what was read, what entered, what was held and why.
The collection stays clean after day oneThe screening is part of the flow. A letter written this morning is checked before it can be quoted.
05

What comes out

Rule set v1.2 sends every document down one of three roads. The road decides whether the assistant reads it as is, reads a masked copy, or waits for Priya.

Clean

Nothing personal found

A procedure, a product sheet. The rules find nothing to mask.

Next: the document enters the assistant's collection as it is.
Masked

Personal data, masked

A name, an address, an account number in a reply letter. The masks follow the approved rules.

Next: the masked copy enters, the original stays whole in the repository.
Held

The rules cannot settle it

Low confidence, a possible health mention, an unreadable scan.

Next: the knowledge manager decides; an exception goes to the privacy officer.
What stays with people

It widens no rule, releases no held document and has no say in go-live. Karen signed the rules, the exceptions and the test report, and Priya accepted every mask that left quarantine.

The AI detects, proposes masks and answers from the screened collection. The screening sits in the intake flow, so a letter written this morning goes through it too.

06

What it saves

Counted on the 18,400 documents of the customer service corpus. Other collections and other rules will give other counts.

148 documentsread by a person, out of 18,400. The rules settle the rest the same way every time.
0 originalschanged. The assistant reads masked copies; the repository keeps the documents whole.
60 questionsin the leak test, logged and replayed after every change to the rules.

Where these numbers come from. They are counted on the scenario described on this page, with example data. They are not a customer result.

What makes these gains possible

Zero unscreened pagesProductivityEvery document goes through the rule set before the assistant can read it.
Zero silent exceptionsProcess reliabilityRules and exceptions are versioned together, with the name of whoever approved them.
Decision supportA person decidesHeld documents and go-live are human decisions. The AI proposes the masks.
07

Who uses it

The corpus needs three signatures: the person who builds the assistant, the person who answers for privacy, and the person who owns the letters.

IT and architecture

CIOs and architects

The assistant is ready, and nobody will sign off on the corpus.

  • The screening is a batch run with a versioned rule set.
  • The assistant reads only the screened collection.
  • Permissions apply before every call.
  • A model hosted at your site for sensitive content, if you choose.
Risk and compliance

Privacy officers

You are asked to approve an AI that might quote any document in the repository.

  • You approve the rules once, with a version number.
  • Every exception comes to you with its reason.
  • Every call keeps its question, answer and model.
  • The leak tests can be replayed months later.
Business operations

Knowledge managers

The model letters that help advisors most are the very ones that leak.

  • Held documents reach you sorted by reason.
  • The AI proposes the masks, you accept each one.
  • The original and the copy show side by side.
  • A correction task goes to the owner of the original.

Start small

With one collection. Take the one your assistant would use most, often the reply letters, and run the screening on it alone. You will see what would have entered before anything goes live.

The screening reads the repository, collections and rights you already have, and the assistant inherits them.

See it live

Screen one collection before you open it

Pick a collection you would like your assistant to use. We will run the screening rules on it with your privacy officer and show you what would enter.

The screens on this page are drawn to show one step at a time, with example data.

Uxopian Software helps organizations put AI to work across the systems they already run, without giving up control of their data, their infrastructure or their critical processes.