About Highfive & Manifold
Highfive is on a mission to transform every meeting room with modern, beautifully-designed technology that empowers people to get their best work done together.
Manifold is a full-service AI consulting company offering a complete range of engineering services, including machine learning, data science, data engineering, devops, cloud, and edge.
In 2015, the customer success team at Highfive realized that they needed to tap into insights about their customer’s needs and behavior buried in product usage logs. The opportunity was promising but the log files were massive, measuring in the terabytes. Highfive had prudently collected data on application usage across platforms, including phones, laptops, boxes in conference rooms, and across attributes, including audio/video quality, time and end-points, and more.
To wrangle the data deluge, the Highfive team started working with a team of data scientists from Manifold with the goal of creating a web-based dashboard of customer key performance indicators (KPI). The model underlying the dashboard would be trained on historical data and be applied to near-real-time product usage logs at the individual customer and at aggregate levels.
Highfive and Manifold chose Google Cloud’s BigQuery as the right big data warehousing tool to perform the needed analytical processing. There were a few key reasons for selecting BigQuery:
- Fully managed and NoOps: The team could could focus on the analysis of data, rather than worrying about infrastructure.
- Low cost: The total cost with the estimated data size and query load was easy to calculate and surprisingly low.
- Fully scalable: As product usage and new log data grew exponentially, scalability would not be an issue.
- Ease of analysis: Analysis could be done right in SQL without having to introduce additional tools.
Manifold did all of the exploratory analysis and most of the feature engineering right in SQL. It was easy to write identified features to new tables in BigQuery. At one point, the team needed to do advanced processing on the data beyond pure SQL. Fortunately, Python Pandas could also interact easily with BigQuery and write the final KPI metric to a BigQuery table. To keep the dashboard up to date—without manually running SQL queries and a local Python script—the team used Google Compute Engine to run a nightly shell script that stitched together the entire data pipeline.
Highfive’s customer engagement predictive model and dashboard now run seamlessly in the cloud. The team uses these to identify accounts to focus on and what metric targets to aim for. More importantly, the robust analytics pipeline built on BigQuery will allow the dashboard to scale with the rapid growth in users and product usage and will allow their internal engineering team to update the analytics with ease.
Manifold now uses BigQuery regularly in engagements that require the speed, flexibility and open APIs needed to process terabyte scale data quickly. Using BigQuery, the timeframe for similar engagements is now under two months with a two-person team.