Artificial Intelligence (AI) now enables us to organize the vast amounts of data most companies now generate and use it to yield actionable insights. At Manifold, we often work with organizations that are using machine learning and statistics to make decisions and become data-driven for the first time. For this reason, having a simple process that we can follow together is key. In the course of our client work over the last year or so, we've refined a process that helps us deliver business value quickly and efficiently. We call it Lean AI.
To arrive at our Lean AI process, we drew inspiration from many previous frameworks — including Human-Centered Design at IDEO, Agile Software Development, the Lean Startup methodology, and the CRISP-DM approach created by the data mining community.
Ever hear that joke about how there were 14 engineering standards, which was clearly too many? A bunch of engineers got together to write the One True Standard to make sense of the existing 14. So now there are 15 engineering standards...
Why does AI/ML as it exists today need its own set of principles? Why not just follow one of the methods we based our Lean AI approach on instead? Because the current best practices for learning from data require the coordination of so many different teams, many of which used to be individually siloed in most organizations. Modern AI/ML requires developers, engineers, business stakeholders, domain experts, and data scientists to work seamlessly together to identify key business questions and opportunities and then collect, process, analyze, and put into production the data needed to address them. Keeping everyone organized and aligned requires a specialized process.
So What is Lean AI?
Lean AI is a set of six steps that is carefully designed to address risk early on in the process and make a tangible business impact as efficiently as possible.
Of course, none of the individual steps is new. What's important about the process is the emphasis on performing each step in a carefully considered order. Many data scientists want to focus on the modeling step — which is the fun part where you get to do fancy math. But building and tuning models is only one piece of the puzzle. In fact, all the steps before and after the modeling is where much of the risk lies. We've learned to use the Lean AI process to tackle the big risks as early as possible.
Lean AI is not a perfectly linear process. We perform data modeling iteratively, and also use modeling to help us better understand your data and the engineering and ETL that has gotten us to a given point. In addition, user feedback after we've done some of the work often helps us clarify the business problem. Sometimes the process causes us to “short circuit” and go back to the beginning. And that's okay—that's part of what makes it the most efficient way to get to a polished end result that directly addresses the needs of your business.
There are two things we need to understand when approaching an AI project—first, we need to understand the business problem our client is hoping to address; then, we need to understand the data. We can reduce risk by pointing technology at the right business problem, and by ensuring that the quality of the data we're working with is high enough to solve that problem.
This step of our process often involves running workshops to help our clients determine what is a “solvable problem” with AI on a reasonable time scale. The output of this step is a set of AI specs, which include the problems we think we can solve with particular data sources — and, most importantly, the approximate investment of time and money required to create a v1 system to solve them.
Of course, in the end, solid AI isn't magic, it's software development. We use industry best practices to ensure a robust and flexible result. At Manifold, one of the most important practices we’ve implemented involves using Docker to take advantage of containerized data science. The resulting developer flow is cleaner and more collaborative, and it's ultimately far more productive.
Data engineers create the framework in which good ML work can be done. This work is a core competency for creating solid AI/ML systems. It's very difficult for one person to embody all the required skills across the spectrum from creating complex data pipelines to complex ML models, so most of our delivery teams include both data engineers and ML engineers. The team takes time at the beginning of a project to think ahead to deployment and engineer a robust architecture that will allow us to accomplish our goals together.
Only after the data engineering is solid do we begin to build data models. When we get to this stage, it's sometimes tempting for our clients to let their imaginations run wild. But we advocate taking Emmanuel Ameisen’s advice—efficient problem-solving happens at the most straightforward, basic level. We have found that baseline models will consistently deliver superior end products, especially for the user. To this end, we use explicit rules in our process to keep simplicity in mind.
We believe in really nailing a core set of features first; we can always add more later. A corollary to this viewpoint is that we should have shippable models at every stage—call it the “minimum viable model”—so that even in the simplest form we have something we can put into production. This approach is key to the Lean mentality.
Shiny AI models are great, but at the end of the day, it's humans who have to interact with and make sense of the machine's recommendations. Thats why it's important to get the results from the ML/AI system in front of users as fast as possible. To that end, we often run workshops with the final end-users, whether they're maintenance techs, product purchasers, marketing professionals, or health coaches.
There are two key threads in the user feedback step of our Lean AI process. First, raw predictions are often insufficient on their own. It’s necessary to build a user interface that allows post-processing and interpretation so users can use the AI output to solve the tangible business problem. This interface may be a search engine, app, or dashboard, but it needs to be human-friendly. Second, we have to build trust in the model. Many people are naturally wary of “black box” machine learning models, and a certain amount of early skepticism is helpful. The model has to earn the user's trust by proving itself to be accurate and reliable.
The ultimate goal of embarking on the Lean AI process is getting a model into production. If the model is still in the sandbox rather than having real-world impact at the end of an engagement, then we have failed. The key to the Lean AI process is to think about deployment from Day 1, so we don't end up having to reinvent the wheel later on in the process. What does this mean in practice?
We work with the software and operations teams, as well as other key stakeholders, to understand the key integration points, as well as the volume and velocity of data and predictions, so that we can set up clean interfaces for upstream and downstream integrations. We find that when we put in this work up front in the Engineer step, the Deploy step becomes simple and seamless.
The final step of our Lean AI process is about observing how real end-users interact with the final product we've built. Sometimes we're able to go back and make changes at this stage, although if we've done a good job in the User Feedback step then there are very few surprises left at this point. Rather than actively soliciting feedback in this step, we try to get out of the way and act like an ethnographer, watching how the tool we've created is being used “in the wild.”
In our client work, one of our main goals in this step is to offer their internal teams support in this phase — whether in the form of technical training, change management support, etc. To support them in maintaining the build, and to aid our own future learning, we often build instrumentation into the AI workflow interface to produce product metrics. We can then use data science and analytics on our own data science product to understand how users are employing it.
Why Use Lean AI?
At its core, Lean AI is a set of mental models that help teams work together smoothly to deliver business value faster when implementing AI in their organization. It's not just about aligning expectations between us and our clients; it's about aligning expectations across all the different teams within our client as well: IT, DevOps, engineering, business, etc.
Our clients receive not only a set of deliverables or Manifold's expertise, but also a transparent and efficient process with clear expectations attached to each step. As we work alongside them, our client teams often learn and continue using the Lean AI process for future projects after our engagement together. They trust the process to deliver impactful results and find it helps them to stay organized.
To read more about the results our Lean AI process delivers, please read some of our case studies.