The Limitations of Data Analytics: A Conversation with a Data-driven CMO

Editor’s Note: The following is an edited transcript of a conversation with Jeff Weiser, CMO at Shopify. 

I’m an unusual messenger for the idea that there are limitations to what you can do with marketing analytics. I am, after all, a highly data-driven CMO with a background in data science, analytics and other quantitative disciplines. But changes in today’s marketing environment are shifting the way we work with data and our expectations of what it can do.

Over the last ten to fifteen years, marketing has become much more reliant on data. I saw the transition as it happened. At the time, I was running strategy and analytics departments—everything from simple marketing analysis and dashboarding to financial analysis, FP&A and big data. Slowly, marketers started coming into my quant departments for guidance on how to divvy up their budgets or develop analytically-derived audience segments.

The line between marketing and analytics blurred fairly quickly, and I became something of an “accidental” CMO, currently heading up marketing at Shopify. Still, I’ve always thought of myself first and foremost as an analyst. This is why it feels a little strange to be the one calling out the fact that we have, in some instances, reached a point of diminishing returns when it comes to viewing marketing and sales through a purely data-defined lens. We’re starting to realize that you can’t completely manage human decision making with an algorithm.

How We Got Here – Discovering PLG Roots in Direct Response

Let’s take a quick step back and look at how we got here. The current product led growth (PLG) trend didn’t come out of nowhere, and neither did the more general direct-to-consumer business model. If we time travel back about ten years, we can see the roots of both of these movements, which harness data to power marketing and drive growth.

At that time, about 2010, I joined an incredible Santa Monica-based company called Beachbody, which was transforming the health and wellness space selling exercise DVDs and meal replacement shakes via direct response TV and multi-level marketing. I’m not ashamed to say that being part of that organization at that time remains one of the best professional experiences of my life. From 2010 and 2016, Beachbody exploded, growing their revenue from about $400 million to more than $1.3 billion. They accomplished this Herculean feat by being very disciplined about running the numbers and striving to be the gold standard for analytics-driven direct response marketing.

And if we go back a little further into the history of direct response marketing, we can see how pioneers in the space laid the groundwork for contemporary direct-to-consumer brands like Bonobos and Dollar Shave Club. Back in the late 1990s and early 2000s, a company called Guthy Renker forged new marketing territory with products like Proactiv, cutting out the middleman and building their business on an in-depth understanding of unit economics, customer acquisition cost (CAC) and lifetime value (LTV).

Today’s product-led companies are building on those ideas, creating a hybrid entity that combines direct response analytics with a superior customer experience. This is the revolution Shopify powers. Our customers are the companies that are carrying the direct response torch into the future. Shopify provides them with everything they need to start, sell, market and manage their businesses in a single platform. Our own company has evolved alongside our customers, broadening our services and products to meet the growing needs of an expanded range of entrepreneurs.

Mostly, we believe we’ve succeeded because we take something very complex and make it simple. That’s a big advantage these days because marketing seems to get more complicated every day.

The State of Marketing Today – Adapting to a Different Landscape

It wasn’t that long ago that we mostly thought about marketing as existing in two discrete categories: brand and direct response. But now, it’s more accurate to describe marketing as a spectrum.

There are several factors driving this shift. For one thing, the evolution of the direct-to-consumer brand, which combines the best of both worlds—the math of direct response and the experience of a brand marketing operation. Then, there’s the rise of growth marketing, a trend that embraces the concept of creating something that’s more than the sum of its parts. It used to be enough to spend $1 on marketing and get $2 in return, but that kind of linear growth doesn’t cut it anymore. Growth marketers have hacked the system using viral marketing mechanics and gaming techniques to create viral loops and other growth drivers so that we’re no longer talking about one plus one. Now the equation is more like one plus one squared and becomes four.

On top of all these changes, marketers are also realizing that there are limitations to what we can do with measurement and attribution. For a while, we were thinking that if we just did enough math, eventually we’d be able to reduce everything to a perfect direct response model. But now marketers are coming to terms with the fact that there’s no silver bullet modeling solution for attribution or the effect of brand marketing on direct response metrics. All of which leads to more blurred lines as companies attempt to achieve some level of quantification in their marketing, but are also happy to settle for general improvement in place of a global solution for how to spend marketing dollars most effectively.

The convergence of brand and direct response marketing is also removing the traditional division of channels. Once, brand marketing was almost exclusively associated with radio, TV, and out of home while the direct response space usually occupied by B2B companies was primarily focused on digital channels. Now, however, there’s a lot more channel crossover, which is in part a natural correction to what was an overswing of the pendulum toward digital spending.

Shopify is getting more aggressive about looking at ways to use non-digital channels. While I’m still a strong advocate for precise and consistent use of data and analytics to inform marketing decisions, it’s possible to reach a point at which companies are willing to pay more than advertising is worth simply because it can be measured. Paying a premium just so you can measure something is counterproductive. I’d much rather achieve a better return in an uncertain and imperfectly measurable world.

Take word of mouth, for instance. That is a powerful marketing tool that can’t be boiled down to a formula. It’s something of a black box, which makes it very difficult to create intentionally. There is, obviously, not only a human element to word of mouth, there’s also a community element. One of the first steps to encourage users to share your product with their peers is to participate in the conversation. I’m often shocked by the extent to which brands avoid participating in the conversation people are having about their product. People will talk with or without you; why not join in?

Another approach to influencing word of mouth is to create viral mechanics that produce a reliable incentive for people to pass on information about your product. This taps into social gaming strategies and can work well when done right. The most effective method we’ve found, however, is simply to build a product that’s so good people can’t help talking about it. That’s the path Shopify took, and it has been very successful. As one set of entrepreneurs used Shopify to bring their business dreams to life, they told other entrepreneurs. All we had to do was build a better product for an under-addressed, but large, market segment.

The Internal Marketing Organization – Building around Real-life Sales Funnels

In addition to adapting to the changes in the external marketing landscape, organizations also need to adjust their internal operational structure and flow so that it aligns more closely with how customers actually buy. Lesson one: the funnel is never a straight line. Any attempt to take a complex process like convincing someone to adopt and use your product is never going to be as simple as MQL, SQL, SAL. You might be able to build a perfect data model in the world of analytics, but in the real world, things are a whole lot messier than that.

So, how do we succeed in this messier world?

The first step is to segment your customer base effectively. Instead of treating your customer base as a monolithic entity, segment it in terms of, for example, LTV or conversion propensity. Defining LTV will help you determine if a customer deal is lucrative enough to justify the expense of sales rep time and labor. Being able to score customers based on how likely they are to convert can help you refine your marketing efforts, though perhaps not in the way you would assume. Rather than focusing on the people who are most likely to convert, you might do better to work on the second or third decile of conversion propensity—people who are sort of open to buying your product, but not already sold on it. The people most likely to buy will probably convert on their own—without needing sales support—but that 20th or 30th percentile could use some sales support.

Whichever scenario you’re dealing with, you need to be pretty rigorous about understanding—in real time—who your customer or lead is so that you can put them in the right funnel.

Speaking of the funnel, who owns the funnel in a product-led organization?

In my experience, the best way to address any ambiguity is through a well-defined organizational structure and—so far—the one that I’ve seen work most effectively for product-led companies is a matrixed environment in which you have product lines or business units on one axis and services that support those product lines on the other axis. In this scenario, marketing is typically treated as a service because it takes on a supporting role in a product-led environment.

For Shopify, I embed a product marketer in each product line. These product marketers become the hinge that connects the product and services axes on the organizational matrix. In this position, they maintain important connections with both sides. On the product side, they are connected at the hip to their respective product manager, and on the services side, they all report to a centralized head of product marketing.

The partnership between the product marketer and product manager is key because they have joint ownership over a set of numbers that they agree to hit together. While some people will push back on this, saying that it’s critical to have only one person be accountable, our teams have done well with this model.

Having all product marketers report into one head of product marketing is also crucial because it helps to maintain consistency across multiple products. Without this throughline, you risk accidentally shipping your org chart, meaning you go to market as multiple products instead of a single, cohesive platform which is how the customer experiences your product.

Ultimately, the way it works in a product-led organization is that product leads, sales follows and marketing supports.

The Road Ahead – Making Peace with the Chaos

The marketing landscape continues to evolve, sometimes in anticipated ways, and sometimes in unexpected ways. And while data will always be the backbone of a lot of marketing strategy, people are starting to embrace the fact that you can’t measure everything or attribute every single dollar and every single action. Over several years, we’ve learned that attribution is something you improve, not something you solve.

That said, we’re definitely doing better on a lot of fronts. We may need to update John Wanamaker’s famous quote, “Half the money I spend on advertising is wasted; the trouble is I don’t know which half.” With access to data and the technology to analyze it, we have been able to reduce wasted ad dollars.

Still, we know enough now to realize we’ll never be able to perfectly model exactly where a sale came from. We have to get comfortable with the messiness of the real-life marketplace and get brave enough to try out different attribution schemes. We have to stop making the mistake of sticking with faulty attribution models that we know are wrong just because they deliver neat and tidy results—like the last-click models that fail to accurately address a purchase journey that has an average of six touch points.

Eventually, you’ll do something like use last-click attribution to give social media credit for a sale that actually came from search, and then increase your spend on social and fail to get the incremental sales you expected. That’s not a winning strategy.

So, don’t expect data to solve all your problems. Don’t deny the messiness that is modern-day marketing. But do test what you’re doing to see what works for your business. The more comfortable you are operating in a dynamic environment, the more successful you’ll be.