Project Overview
A US subscription box company had a churn problem it could not see coming. Customers were cancelling without warning, and by the time the team found out, the window to retain them had already closed. Winback campaigns were being sent after cancellation, which converted at a fraction of the rate of a well-timed pre-cancellation offer. The team wanted to flip that model: identify at-risk customers early, while there was still time to act. We built a churn prediction model on their existing Shopify and email engagement data that gave them that window.
The Challenge
Subscription churn is a lagging signal by nature. Customers signal their dissatisfaction through behavior weeks before they cancel, but those signals are subtle and scattered across different tools. Without a system to read and combine them, the team had no way to distinguish a customer who was about to leave from one who was just quietly engaged.
- No visibility into which customers were at risk until after the cancellation had already occurred
- Retention efforts were entirely reactive: winback campaigns sent post-cancellation rather than pre-cancellation outreach
- Behavioral data existed in Shopify and email engagement logs but had never been combined into a churn signal
- The team had no data science capability in house and needed a solution that integrated with tools they already used without requiring technical management on their end
Our Solution
We built a churn prediction model trained on the client’s Shopify transaction history and email engagement data that flagged at-risk customers two weeks before their likely cancellation date. When a customer crossed the risk threshold, a targeted retention offer was triggered automatically through their existing email tool, shifting the team from reactive winback to proactive retention.
- Feature Engineering: We combined Shopify order data with email engagement metrics, including open rates, click-through rates, and engagement recency, to build a behavioral signal set that captured early churn indicators. Features included order frequency changes, days since last engagement, product category shifts, and subscription tenure.
- Churn Prediction Model: A binary classification model trained on historical subscription data, calibrated to predict cancellation likelihood within a two-week horizon at a level of confidence sufficient to act on without triggering too many false positives.
- Automated Retention Trigger: When a subscriber crossed the risk score threshold, an automated retention offer was sent through the client’s existing email marketing platform via API integration, requiring no manual intervention from the team.
Technical Approach
The model was built using gradient boosted classification on the combined behavioral dataset, with a training set constructed from historical subscribers whose cancellation outcomes were known. The risk score threshold was calibrated conservatively so that the retention offers were being sent to genuinely at-risk customers rather than a broad swath of the subscriber base, which kept offer costs controlled and maintained the perceived value of the retention offer for recipients. Integration with the email platform was handled via API, keeping the system within tools the team already operated daily.
Results & Impact
- Retention rate improved in the period following deployment, with at-risk customers receiving timely outreach before their likely cancellation date rather than after
- Two weeks of advance warning provided enough lead time for a targeted retention offer to land before the customer had made a final decision to cancel
- Proactive outreach replaced reactive winback as the primary retention mechanism, shifting from a low-conversion post-cancellation campaign to a higher-conversion pre-cancellation offer
- The system required no ongoing technical management from the client’s team after deployment and integrated fully with tools they were already using
- At-risk customers were identified early enough that the team had time to test different offer types and messaging, creating a feedback loop for retention optimization over time
Lessons Learned
The most consequential design decision was the risk score threshold. Set too low, the model flags a large proportion of the subscriber base and the retention offer loses its targeted value. Set too high, genuinely at-risk customers are missed. We worked with the client to define what a false positive cost them in offer discount versus what a missed at-risk customer cost them in lost subscription revenue, and calibrated the threshold accordingly. That economics-based calibration produced a much more useful model than optimizing for statistical accuracy alone.
The two-week prediction horizon was also a deliberate choice. A longer horizon would have produced more noise. A shorter one would not have given the team enough time to act. Testing different horizon lengths against the client’s historical data confirmed that two weeks was the point where predictive signal was strongest and operational lead time was sufficient, which is not always the same answer for every subscription business.