Project Overview
A small US law firm handling a steady volume of commercial contracts was facing a capacity problem that had been building for years. Junior associates were the first line of review on every incoming contract, and the process required them to read every clause, every page, every exhibit from beginning to end before flagging anything for senior attorney attention. For a standard commercial contract, that review took 3 to 4 hours per document. As the firm’s client base grew, the review backlog grew with it. Senior attorney time was being consumed by supervision of routine reviews rather than the complex legal work the firm was actually hired to do. We built a fine-tuned LLM trained on the firm’s own reviewed contracts and attorney annotations that transformed contract review from a read-everything process to a review-what-matters process.
The Challenge
The core problem was not that the associates were slow. It was that the task as defined required complete coverage of every section regardless of risk level. A contract might have 40 clauses, of which 6 actually warranted attorney attention. The associates had no reliable way to identify those 6 without reading all 40.
- Junior associates spending 3 to 4 hours per commercial contract reading every clause before flagging anything for review
- Contract volume growing faster than associate capacity, creating review backlogs before every deal closing
- Senior attorney time consumed by supervising routine review rather than complex legal work
- Contract formats varied significantly across industries and counterparties, making simple keyword flagging unreliable
- The firm needed the system to understand legal context, not just match keywords, and to flag its own uncertainty rather than miss clauses silently
Our Solution
We built a fine-tuned LLM trained specifically on the firm’s historical reviewed contracts alongside the senior attorneys’ annotations, comments, and flagging decisions. The model learned not from generic legal data but from the specific judgment patterns of this firm’s own attorneys applied to the specific types of contracts they handle.
- Training Data Preparation: We worked with the firm to extract and structure five years of reviewed contracts, pairing each clause with the attorney annotation or flag decision that had been applied to it. This created a high-quality labeled dataset that reflected the firm’s own risk standards rather than generic legal benchmarks.
- Fine-Tuned Clause Analysis Model: A foundation model fine-tuned on the firm’s annotated contract data, trained to classify clauses by risk level, identify missing standard terms, flag unusual language, and indicate confidence level for each decision.
- Uncertainty Flagging: Every output included a confidence score. Clauses where the model’s confidence fell below a defined threshold were automatically flagged for full associate review regardless of apparent risk level, ensuring the system failed safely rather than silently missing edge cases.
- Review Interface: A clean document review interface showing the full contract with color-coded section highlights for risk level, a summary of flagged clauses at the top, and the model’s reasoning for each flag visible on hover.
Technical Approach
The model was built using a LoRA fine-tuning approach on a Llama foundation model, keeping compute costs low while achieving the domain specificity the task required. Contracts were parsed into clause-level segments using a custom parser that handled the structural variation across different contract formats and jurisdictions. Each segment was processed independently and results were reassembled into the full document review interface. The confidence threshold was calibrated through a validation process where the firm’s senior attorneys reviewed a sample of the model’s outputs and adjusted the threshold until the false negative rate, clauses with real risk that the model missed, reached an acceptable level for the firm’s risk tolerance.
Results & Impact
- Contract review time reduced from 3 to 4 hours per document to under 45 minutes, with associates spending that time on flagged sections and their own judgment rather than cover-to-cover reading
- Weekly contract review volume handled by the same associate team increased significantly without additional headcount
- Senior attorney supervision time reduced as associates arrived at review meetings with pre-structured flagged clause summaries rather than open-ended questions
- The training data preparation process surfaced inconsistencies in how different attorneys had been applying the firm’s risk standards, which the partners used to update and align their internal review guidelines
- The uncertainty flagging system proved its value immediately: in the first month, 14 percent of clauses were flagged for full review due to low model confidence, and senior attorney review confirmed that 9 of those contained genuine ambiguity that a confident automated flag would have handled incorrectly
Lessons Learned
The uncertainty flagging requirement was the most important design decision in the project and the one the firm initially pushed back on as unnecessary overhead. Their concern was that too many uncertainty flags would slow down the review process and defeat the purpose of the system. Our position was that a system that failed silently on legal documents was not a system we were willing to deploy. The validation data proved the point: the clauses flagged for low confidence contained real ambiguity that a binary flag would have either incorrectly cleared or incorrectly flagged. The nuance of the confidence layer was what made the system trustworthy enough for the firm to rely on in production. Building trust with a legal team requires demonstrating that the system knows what it does not know, not just what it does.