Project Overview
A US tech startup growing quickly had a problem that is common in fast-scaling engineering organizations: onboarding new engineers was taking three to four weeks, and a significant portion of that time was driven by senior engineers repeatedly fielding the same setup questions and process clarifications. The documentation existed, but it was scattered, partially outdated, and not easy for a new hire to navigate without guidance. We fine-tuned a foundation model on their internal knowledge base to serve as an onboarding assistant new engineers could query directly, in plain English, without involving anyone else.
The Challenge
The startup’s onboarding problem had two interconnected dimensions. New hires were slowed by documentation that was hard to navigate and sometimes out of date, and senior engineers were pulled away from productive work to fill in the gaps. Every hour a senior engineer spent answering a question that was already written down somewhere was an hour not spent on the work they were hired to do.
- Internal documentation was spread across four separate tools, with no unified search or navigation layer
- Portions of the documentation were outdated and contradicted current practice, creating confusion for new hires who could not yet distinguish what was current
- Senior engineers were fielding repetitive questions throughout the workday, fragmenting their focus during critical growth phases
- New hires reported low confidence in their first two weeks, even when they had found the right documentation, because they could not verify whether it reflected current team practice
Our Solution
We fine-tuned a foundation model on the startup’s internal documentation corpus, including their codebase comments, architecture decision records, and Slack FAQ threads, to create a conversational onboarding assistant new hires could query directly. The assistant was designed to answer setup and process questions in plain English, reducing the need to interrupt senior engineers for routine guidance.
- Documentation Indexing: We ingested content from all four internal tools, applied a structured cleaning and deduplication pass to surface contradictions, and flagged outdated content for the team to update before training.
- Fine-Tuning on Internal Context: The model was fine-tuned to answer questions grounded in the startup’s specific codebase, architecture patterns, and team processes, rather than giving generic software engineering answers.
- Calibrated Uncertainty: A central design requirement was that the model clearly indicate when it was uncertain or when a question fell outside its training context, rather than producing confidently wrong answers. We invested significant effort in calibrating confidence thresholds and testing refusal behavior across ambiguous queries.
- Plain English Interface: New hires queried the assistant conversationally, in the same way they would ask a colleague, without needing to know which tool held the relevant documentation.
Technical Approach
The assistant was built on a fine-tuned foundation model with a retrieval-augmented generation layer connecting it to the indexed documentation corpus. The RAG layer ensured answers were grounded in actual internal content rather than model inference, with source citations returned alongside each response so engineers could verify answers and read the underlying documentation when needed. Confidence calibration was implemented through a combination of retrieval confidence scores and fine-tuning on uncertainty-expressing examples sourced from the team’s own Slack history.
Results & Impact
- Onboarding time reduced from an average of three and a half weeks to under two weeks following deployment
- Senior engineer hours previously consumed by repetitive onboarding questions were recovered and redirected to product work
- Documentation from all four internal tools was indexed and accessible through a single conversational interface for the first time
- New hires reported feeling more confident and self-sufficient from their first day, with fewer gaps in their understanding of setup and process
Lessons Learned
The hardest part of this project was not the model fine-tuning: it was the documentation audit. Scattered, partially outdated content is a worse foundation for a knowledge assistant than a smaller, clean corpus. We spent more time on pre-training content hygiene than on any other single phase, and that investment directly determined how reliable the assistant was in practice. The calibration work on uncertainty was the second most important investment. An assistant that occasionally says it does not know is trusted. One that confidently gives wrong answers quickly loses adoption, which is the failure mode we were most careful to avoid.