OneReach.ai founder, Robb Wilson, shares his perspective on analytics and reporting in an excerpt from his upcoming book, Age of Invisible Machines.

When it comes to experience design, data and analytics are nothing new, but in the realm of conversational experience design, they take on a new immediacy. The old software creation cycle of “design, build, test, and iterate” doesn’t disappear when working with conversational AI, but it becomes massively accelerated and is an ongoing element of the continual evolution of automated solutions. Conversational AI requires orchestration across different technologies and modes. When you are sequencing technology behind a conversational interface and building new and improved solutions at will, you are doing something Gartner calls “hyperautomating” and thinks of as an “inevitable market state.” Hyperautomating means accounting for a wide variety of data points and information streams, adding to both the complexity and the possibilities.

Maximizing the impact of conversational AI requires tools that let organizations analyze and report on interactions between users and intelligent bots (I call them intelligent digital workers or IDWs) in real-time. This creates a powerful and integrated form of user research where analytics and reporting—operating somewhat independently—become inseparable. This rapid user research provides crucial information for establishing and evolving an ecosystem of automated solutions. 

Even the most basic user experiences with conversational AI produce massive amounts of data, which is a good thing from an analytics perspective: the more information, the better. Conversation interfaces yield unstructured data, however, and that requires analytics to give it shape and meaning. Combine that swelling pool of nebulous information with an organization’s internal data—specifically, the data that’s feeding the conversational experience—as well as information that’s available from the sea of connected devices in and around an ecosystem, and profound complexity emerges.

Analysts love data, and in these scenarios, they can test out hypotheses by exploring as much data as possible—running it through different filters and looking for patterns. There are various tools out there, such as machine learning, that let you crunch large amounts of data and find actionable patterns to exploit. These tools don’t need to work in real-time, but they need to be flexible so that analysts can explore data in many ways.

Let’s say you’re an analyst for a subscription-based cat litter company. You have a hunch that when customers move, they often cancel their subscriptions and restart them again later rather than just updating their address information. You run the data to see if that correlation is supported. You find a pattern of users who have canceled subscriptions and then created new ones with different addresses. This is valuable, actionable information that you then pass along to an experience designer.

When XDs are building conversational workflows for IDWs, they can implement algorithms built around the patterns provided by analysts. In our kitty litter scenario, the XD could add prompts when users first sign on and when they initiate a cancellation. The XD can design ways to communicate to users that updating their address is easy—easier than canceling and then starting up again.

As they make these kinds of adjustments, XDs can use real-time reporting to learn what’s working and what isn’t. In some cases, they might even improve interactions as they’re happening. This is the nature of reporting in hyperautomation scenarios. Reporting is always on and always being acted on. Tactically speaking, this requires a dashboard that allows experience designers to watch as users interact with IDWs, moving through a series of skills and flows. When experience designers can literally see how users are interacting with machines, they can be regularly fine-tuning interactions.

The data supply that comes with most of the machines that people release into the wild are black boxes with basic metrics (attrition, hang-ups, etc.) and are of little use when creating an ecosystem for hyperautomation. Unless you’re summarizing data in real-time, you’ll find it exceedingly hard to patch holes in your boat before the water pours in. Being able to retrace conversations in a timely fashion is the real value and the real-time reporting aspect needs to be baked into whatever tool you’re using.

What’s required is an ever-spinning feedback loop. As skills are deployed and utilized, those interactions can be analyzed and iterated on, often as they unfold. This isn’t a nice-to-have, it’s an imperative. If you can’t easily report on how people are using your system and experiences, you’ll find it impossible to fulfill the promise of rapid iteration, which is critical for hyperautomation. Hyperautomation allows us to create hyperpersonalized solutions, but that hyperpersonalization requires this deeper, ingrained relationship with analytics and reporting.

Not only is this the most efficient way to aggregate and actualize vast amounts of data within your ecosystem, but it’s also a boon to your team. In this scenario, data analysts get to focus squarely on what they do best: digging through data and identifying patterns. In a complementary fashion, your experience designers will be designing experiences that are truly dynamic, relying on an exciting new rate of iteration that should ignite the creative spirit of anyone passionate about the progression of experience design as a discipline.

In some ways, this is a dynamic new form of experience design that represents the process in its purest form. By designing experiences, watching them unfold in real-time, and taking opportunities to continually improve on them, you’re flowing through steps hourly or daily that would be weeks or even months apart in typical software development scenarios. This creates a feedback loop that lets you design and maintains incredibly powerful software at a rapid pace.

Keep in mind, however, that companies are not successful because they produce more software but because they produce more good software. This is where this new approach to experience design pays off. When experience designers no longer have to concern themselves with being developers—when they can create and evolve software at will without having to write code—they can focus squarely on designing truly good software.

__

Age of Invisible Machines cuts through the AI hype, revealing key ideas and best practices discovered through more than 2,000,000 hours of testing and using data from over 10,000+ AI applications. It is both a visionary manifesto and a practical guide written for AI strategists, technology and business managers, and leaders alike.

Gain insights and practical guidance on the processes, people, tools, and architecture for creating a hyper-automated ecosystem. Sign up for updates and pre-order your copy of the book here.