About Babylist & Manifold
Babylist makes the journey to parenthood easier and less overwhelming with its helpful content, curated store and universal baby registry.
Manifold is a full-service AI consulting company offering a complete range of engineering services, including machine learning, data science, data engineering, devops, cloud, and edge.
Babylist was looking for a way to make better and faster decisions and to be more customer-centric by leveraging the data they had been collecting.
“The customer prediction tools that Manifold created have significantly impacted our business. We are able to make quick and aggressive decisions with confidence and have a focus on quality instead of just quantity.”
— August Flanagan, CTO
Manifold accelerated development of predictive customer analytics tools that help optimize day-to-day decision-making by answering questions such as the following:
- Which of my customers are likely to churn?
- How likely are my customers to activate?
- How much revenue will each of my customers generate for the company? How much revenue will this month’s set of new users generate?
- What are the predicted results of my A/B test?
- What behaviors should I incentivize in my promotion? Is my promotion working?
The Product team can now prioritize and build the right customer-focused features, while the Marketing team has a quantitative basis on which to predict the effectiveness of new promotions.
To succeed in e-commerce, you must have a deep understanding of your site visitors so you can optimize product features and marketing strategies for your target customers. However, the several months it typically takes to run end-to-end marketing campaigns and analyze results for the next campaign is far too long in the current competitive landscape.
Babylist wanted to be able to validate marketing channels and product changes within a week. In order to address this problem, Manifold created a family of machine learning models that could estimate the total revenue a customer would generate and classify whether a user was going to add items to their registry (adopt) or generate revenue (activate). We created a custom decision framework that traded absolute accuracy for time, and allowed our client to make decisions based on estimates of CLTV and probability of customer activation.
We started the engagement by first assessing the data quality to ensure consistency throughout the training and testing phases. We also cleaned the data to take care of duplicate entries, missing data, and anomalous data (e.g., outlier users adding excessive numbers of items to their registry).
Intelligent Feature Engineering
We wrote complex SQL queries to tie together relevant information from multiple tables across the client’s database and to generate predictive features. Feature engineering included reconciling nuanced cases, such as how to include both cash gifts—with varying dollar values—and physical goods. The most important data features included the number of items and value of items added to the registry, as well as the number of unique sessions—defined as adding registry items outside of a 12-hour period. From our prior experience with RFM (recency, frequency, monetary) analysis, we aggregated time-varying features into relevant cumulative periods, creating features from daily activity as well as features from over the course of 7 days and multiple weeks.
We determined over 300 potentially predictive features with the client, including attributes such as area code, signup platforms, how customers added items to their registry, whether they entered a due date, and the number of independent sessions they had.
After detecting seasonal variation in the data due to consumer behavior around the holidays, as well as general trends in changing user behavior over several years, we decided to train the models on one full year of data. We were careful to exclude training data from anomalous months when Babylist was running a promotion, and to exclude features that had changed over time.
After experimenting with various models including logistic regression, random forests, and gradient boosting, as well as optimizing hyperparameters, we selected gradient boosting as the most accurate model for the client’s dataset, as it yielded the highest ROC-AUC (receiver operating characteristic—area under the curve).
We created different models for different use cases, and settled on a family of models that would allow the client to make marketing and feature decisions as soon as a day after user signup, but also provided higher-accuracy predictions for those users as more data was collected.
For the marketing department, we created monthly revenue forecasts based on features with 30 days’ worth of data, and trained on a year’s worth of data prior to the month of prediction. Using this approach, we were able to create high-accuracy models (ROC-AUC = .92) that estimated monthly revenue to within 5% of actual revenue. Once trained, the model accurately predicted who would save items for purchase, who would make eventual purchases, and how much revenue those would generate.
Manifold’s commitment to knowledge transfer ensures that our clients see value in our work well beyond the end of the engagement. As part of our training process, we helped Babylist vet and hire a data scientist, and worked closely with her to carry out experiments by running data, feature, and model evaluation in Jupyter notebooks. We walked through several examples, evaluating our models with five-fold cross validation, and explaining feature importances, separation of posterior PDFs, picking operating points on the ROC curve, and the implications for TPR (true positive rate) and FPR (false positive rate).
Among our deliverables was a private Docker repository with tagged production images. This contained a fully functional, out-of-the-box prediction engine that could be run with a single command. This Docker repository ensured that we could seamlessly deploy the engine to the client’s production infrastructure from Manifold’s production QA environment with guaranteed execution consistency. In addition, we delivered a GitHub repository that contained the source code and full Dockerfile, so that Babylist can accurately rebuild the production images if needed.
Once the prediction engine container was deployed, we worked with Babylist to implement a cron job that runs a daily prediction engine of 12 models, and generates predictions that are then written to their SQL database. We optimized this model and prediction cadence based on the client’s business needs—minimizing server load and computation cost, while providing the most accurate predictions.
Babylist now can use the real-time prediction data on a daily basis. The marketing department can use our A/B testing framework to regularly choose winning strategies, and to understand which acquisition funnels are yielding the best customers. They have already noticed better customer acquisition through rapid marketing optimization, as well as faster product improvement through rapid experimentation and validation. At the end of the engagement, Manifold has enabled our client to meet their business goals of becoming more customer-centric, providing differentiated customer service, and better serving their customers.