Project Overview
A small US financial advisory firm had a compliance document problem that was slowing its staff down and creating quiet risk. Policies, regulatory guidance, and internal procedures were stored in a shared drive that had accumulated years of documents with no consistent naming conventions and no version control. When staff needed to answer a compliance question, they spent 20 to 30 minutes searching for the right document and still were not always confident they had found the current version. We built a RAG-powered internal search tool that answered compliance questions in plain English, cited the exact source document and version number, and surfaced conflicting versions before they caused a problem.
The Challenge
The shared drive was a common symptom of a firm that had grown faster than its document management practices. Policies had been updated over the years without retiring the old versions. Documents had been duplicated into multiple folders. Naming conventions were inconsistent enough that a search for “AML policy” would return seven different files, some current and some years out of date, with no easy way to know which was authoritative.
- Staff were spending 20 to 30 minutes per compliance query navigating an unorganized shared drive with no search confidence
- Multiple versions of the same document existed with conflicting content and no clear indication of which was current
- Naming conventions were inconsistent across document types and time periods, making folder navigation unreliable
- The firm had never enforced version control, so outdated documents were indistinguishable from current ones in the drive
Our Solution
We built a RAG-powered compliance search tool that let staff ask plain-English questions and receive direct answers with source citations, replacing the folder navigation workflow entirely. Before indexing could begin, we ran a deduplication and cleanup pass to surface and resolve the version conflicts that had accumulated in the drive.
- Document Audit and Deduplication: We catalogued all documents in the shared drive, identified duplicates and conflicting versions, and worked with the firm’s compliance officer to designate authoritative versions. Version conflicts identified during this process were resolved before indexing, giving the firm a clean, reliable document set for the first time.
- Metadata Tagging: Each document was tagged with document type, regulatory area, version number, and effective date during indexing, enabling the retrieval system to return the most recent version of any document and to flag when a query matched multiple versions with different content.
- RAG Query Interface: Staff could ask questions like “What is our current policy on client gift limits?” or “What does the latest AML guidance say about beneficial ownership?” and receive a direct plain-English answer with the source document name, version number, and page reference cited.
- Conflict Flagging: When a query matched documents that contained conflicting guidance, the system surfaced the conflict and listed the documents involved, prompting the compliance officer to review rather than allowing staff to rely on a potentially ambiguous answer.
Technical Approach
The retrieval layer used semantic vector search over chunked document embeddings with metadata filtering to scope results to current versions by default. The generation layer was instructed to cite document name and version for every factual statement and to surface conflicts rather than resolving them silently. Hosting was on the firm’s existing cloud environment with access controls aligned to their existing permission structure, so staff only retrieved documents they were authorized to access.
Results & Impact
- Compliance query resolution time reduced from 25 minutes to under 2 minutes, a change that was immediately visible to every staff member who used the tool
- Version conflicts identified and resolved during the indexing process, giving the firm a clean, authoritative document set as a byproduct of the project
- Staff confidence in compliance answer accuracy improved significantly, as every response included a source citation that could be verified in seconds
- The compliance officer saved time previously spent fielding internal questions from staff who could not locate the right document
- The document audit surfaced several outdated policies that had never been formally retired, which the firm was able to address before they created a regulatory exposure
Lessons Learned
The document cleanup phase was not optional. A RAG system trained on a shared drive full of conflicting, duplicated, and outdated documents would have produced confident but unreliable answers, which in a compliance context is actively dangerous. The version audit and deduplication work that preceded indexing was the most important part of the engagement, even though it was also the least technically interesting.
The conflict flagging feature turned out to be as valuable as the search function itself. Surfacing ambiguous situations rather than silently picking one answer built trust with the compliance team and caught several document inconsistencies that would otherwise have remained invisible. For any RAG system deployed in a regulated environment, building in explicit uncertainty signals is not optional.