Marketing

Measuring the Impact of Your Developer Relations Team

February 5, 2020

It’s often said that you can’t improve what you can’t measure. But what if the thing you want to improve seems impossible to measure?

DevRel (Developer Relations) is definitely one of those things that’s challenging to measure, mostly because it’s sometimes hard to define, but also because some elements of DevRel don’t lend themselves to quantifiable measurement.

To begin with, DevRel means different things to different people. Depending on who you are and where you work, DevRel might focus on community development or content creation or conference sponsorship or writing code. And—just to keep things interesting—most DevRel efforts are comprised of a combination of strategies and tactics that can change week to week.

And then there’s the subtle differences between Developer Evangelists and Developer Advocates within a Developer Relations organization. For me, a Developer Evangelist is someone with a megaphone whose primary task is driving awareness by giving talk after talk after talk after talk. Developer Advocacy, however, implies community, conversation and collaboration. It’s about activation, retention, adoption and feedback—very different focus, very different approach. While Developer Advocates still give talks at meetups and conferences, it’s not the core and only aspect of their role. I’m a super huge fan of Phil Leggetter’s DevRelOMeter for understanding the difference between Developer Advocacy and Developer Evangelism.

The only thing all DevRel people have in common is exhaustion. As roles go, DevRel is pretty damn demanding. These folks wear all the hats: marketer, brand manager, developer, partnership manager, biz dev partner, event manager, content creator, technical writer, panelist…and the list goes on (and on).

So, DevRel is difficult to define, and—like relationships—difficult to quantify. Still, companies want to be able to measure its ROI. DevRel is expensive with all the travel requirements, swag and other costs. Company leadership wants to know it’s getting its money’s worth and how the work of this super hardworking Developer Relations team is impacting the growth of the product and revenue of the company.

What to Measure–Reverse Engineering Your Way to A Strong DevRel Measurement Plan

The first rule of measuring DevRel ROI is tying DevRel goals to company goals. This may sound like common sense, but you wouldn’t believe how often people get this wrong. If your DevRel goals are not aligned with larger company goals, people will be hard pressed to recognize the value of DevRel activities. Without context, they won’t be able to connect the dots between what you’re doing and ultimate business outcomes. This makes it hard for other functions to get behind DevRel. And if the DevRel team doesn’t understand the importance of their role within the organization, they can get off course pretty quickly.

The range of possible activities that DevRel advocates can use gives you a lot of different opportunities. Depending on what it’s doing, DevRel can provide valuable support to all the core business areas including sales, marketing, engineering and product. The most appropriate way to define your DevRel strategy is to identify which goals are most relevant to your business objectives, and then reverse engineer the specific DevRel activities (and corresponding metrics) that will help you meet those goals.

For example, let’s say one of your company goals is to increase your rate of growth of your open source project’s downloads by 1%. Your Developer Relations team could plan a set of tactics that they think will help influence that number such as attending and engaging with developers at key open source meetups and conferences, creating content or tutorials that help developers get started more easily using your open source project, creating a Slack channel to help interact with people in the open source community surrounding your project, and much more.

For the most part, goals fall into one of three broad categories: awareness, revenue and developer experience.

Awareness

      • Goal Alignment: Marketing and Product
      • General Objective: Get the word out, increase interest and inspire conversations
      • Possible Activities: Blog content, event sponsorship, conference booth, webinars and speaking events
      • Points of Measurement:
        • Content: Number of content pieces, traffic, views, social media likes and shares, email opens, comments and time spent on page
        • Events: Conference/panel/webinar attendance, social media mentions, booth visits, audience engagement, post-event engagement/follow-through with event-specific content/URLs/CTAs, post-event coverage, post-event connections/conversations

Revenue (Sales Driven)

      • Goal Alignment: Sales and Marketing
      • General Objective: Build positive relationships with developers at enterprise companies that are high on the sales teams’ prospect list
      • Possible Activities: Participating in pre-sales technical calls, collaborating with enterprise developers on Slack channels and online forums as they are trying to get started, fielding pre-sales technical inquiries if a sales engineering team doesn’t exist yet at your company, creating technical content that is uniquely useful to larger teams or enterprise customers
      • Points of Measurement:
        • Top Targets List: How many interactions occurred with developers at top target accounts
        • Content: Number of pieces of content, traffic, views, social media likes and shares, email opens, comments, time spent on page
        • Sales calls: Number of sales calls attended when technical members of the prospect need to learn more about the technical aspects of the product

Revenue (Self-serve Driven)

      • Goal Alignment: Marketing and Product
      • General Objective: Drive top-of-funnel activity and funnel conversion
      • Possible Activities: Anything that drives acquisition and activation by encouraging developers to sign up for the free tier or convert to a paid self-serve account. This might include writing reference or how-to documentation, developing quick-start apps, creating tutorials, delivering webinars, etc.
      • Points of Measurement:
        • Content: Number of pieces of content, traffic, views, social media likes and shares, email opens, comments, time spent on page
        • Documentation: There’s an interesting program here that can be created so that Developer Relations can work with support to look at the top questions repeatedly asked by developers and create documentation and content around those topics. The measurement is around the rate of repeat questions in the support channels decreasing over time or number of articles created.

Developer Experience

      • Goal Alignment: Product and Engineering
      • General Objective: Build, nurture, and sustain a happy, engaged, loyal developer community
      • Possible Activities:
        • Developer-centric: Anything that helps developers get started including how-to guides, quick-start apps, tutorials, support, other documentation
        • Product-centric: Reference documentation, library development, capturing developer feedback, Alpha/Beta programs
      • Points of Measurement: Signup-to-Active ratio, NPS of the community and more

There is clearly a fair amount of crossover in terms of which activities can be applied to which goals. For instance, depending on the subject matter and target audience, a webinar might be used to create awareness, drive acquisition, encourage activation or improve retention. Likewise, support activities can aid activation or retention depending on where the user is in the customer journey. The main thing to remember is that all of this is an exercise in reverse engineering. You start with aligned goals, then drill down to relevant tactics and identify the most appropriate metrics.

It’s worth calling out that DevRel activities are not typically considered to have a direct correlation to revenue outcomes. There are just too many external factors (pricing and packaging not the least). DevRel activities are generally believed, however, to have a positive effect that can influence revenue, albeit not always in easily tracked ways. And, in some cases, you can loosely connect the dots to show how DevRel played a definitive role. For instance, say part of your DevRel strategy is to have meaningful conversations with 4 to 5 developers within each of your top 15 enterprise prospects. If you have those conversations and several developers within one or more of those organizations signs up for a trial or an account, it’s not that difficult to see the relationship between DevRel and a revenue opportunity. But frankly, revenue shouldn’t be the thing you’re measuring when you’re thinking about Developer Relations. It’s about building a strong community and strong product which generally happens to result in move revenue.

Goals–Setting Them and Keeping Track of Progress

Once you have defined your big-picture goals and determined which kinds of activities your DevRel team will employ, it’s time to set some specific quarterly goals so you know exactly what you’re striving toward.

For example, if your focus is on content, you might have goals around both delivery and performance:

      • Delivery: Number of pieces, frequency of publishing, content mix
      • Performance: Views, shares, likes, comments, backlinks

Eventually, you can pull all of this together into a single goal statement such as, “We will earn 500 interactions on 10 pieces of content including 3 thought leadership pieces, 3 tutorials, 2 community stories and 2 sample apps.”

If you’re focusing more on events, you might set your sights on attending 10 events with a combined reach of 3,000 attendees and land speaking gigs on 6 panels.

Whatever you choose as your North Star, it’s important to reevaluate and update goals on a quarterly basis so you can adjust for DevRel burnout (it’s a very real thing), shifts in company goals and previous activity results.

To keep your team on the right path, it’s helpful to track not only specific activities and KPIs, but also the connections and conversations that come out of those activities. For instance, if your presentation at a conference results in an introduction to a key person at one of your company’s prospects, you’ll want to have a record of that and any following conversations as the relationship develops. You might keep notes in a spreadsheet, a journal, a shared doc, a weekly report or even in some kind of marketing software like Marketo or Salesforce.

It may feel a little strange at first, but documenting your relationships is a great way to track your own progress and help your colleagues understand exactly what you do. It also gives you the ability to point out when your conversations and interactions have contributed to specific outcomes such as a new hire, a guest author for your blog, an integration opportunity or even a demo request.

Results–What to Do With Them Once You Have Them

Now that we’ve covered some ground about what to measure and how to measure it, we get back to the crux of the question: why are we measuring it in the first place? There are three main reasons.

First of all, support for DevRel needs to come from the top. Your leadership team needs to be behind you 100%—backing you up and making it clear to the rest of the organization that DevRel is a key part of the company’s overall growth strategy. They need to help everyone—from sales and marketing to product and engineering—understand the value DevRel brings to the table. To do this effectively, and in a language everyone can understand, leadership needs some quantifiable results. They need success stories corroborated by cold, hard facts.

The second reason we measure things is to improve performance. Like we said in the beginning, you can’t improve what you can’t measure. Once you have some data on what’s working and what’s not, you have the information you need to optimize your strategies and tactics so you can do even better next time.

Finally, measurement of deliverables and performance ties back to DevRel finances—how it’s funded and how DevRel professionals are compensated. DevRel’s budget typically rolls up under the department with which it’s most closely aligned. So, if DevRel is focused primarily on awareness goals, funding will come out of Marketing. If DevRel is doing more developer experience work, the budget may come out of Product. Most of the time, because of key activities like event sponsorship and developer programs, the lion’s share of DevRel budget comes from Marketing.

As for compensation, baseline targets can be determined based on performance against established goals, and increases set to align with certain milestones. In addition, DevRel team members can be incentivized with bonuses for “stretch goals.” A stretch goal might be generating or commissioning a piece of content that goes “viral,” landing a byline or interview with a key influencer or putting together a really strong technical tutorial that serves a specific audience need.

Caveat–DevRel Professionals Are Only Human

All this talk about measurement and metrics, budgets and compensation can get a little heady. Part of running a healthy and productive operation is remembering that DevRel professionals are only human. In addition to their multiple responsibilities—outreach, content creation, writing code, social media, support and so on—there is usually a massive travel requirement to keep up with all the conferences, meetups and other events that are integral to the relationships at the center of strong DevRel.

DevRel professionals have impressive multitasking skills and stamina, but they are not immune to burnout.

It’s just not realistic to expect someone to be on the road for twelve months at a time. It’s even less realistic to expect that person to travel that much and still crank out content and provide support and do all the other things that fall onto DevRel’s plate.

When you are setting goals, focus on creating balance and living within the constraints of reality. Look at your efforts on a quarter-by-quarter basis so that you have built-in points at which to reevaluate your plan. Be strategic about how you break up certain activities. If one quarter is chock full of conferences, maybe you designate the following quarter for work on podcasts that can be done from home base.

You also need to be aware of work rhythms on a smaller, weekly scale. For instance, many events (like hackathons) take place over weekends. In addition to giving people time to recover from crazy travel schedules, make allowances for days off during the week when there’s time spent on DevRel activities over the weekend.

Overall, it’s important to be selective about your activities. Once you get the ball rolling, you will receive invitations to a lot of events. A lot. The knee-jerk reaction is to accept every offer that comes your way (More exposure! More opportunities!), but know that it’s okay to say, “No.” No is your friend. It could save your life (or at least your sanity). Cherry pick the events and invites that make the most sense and have the greatest potential for your specific situation. Measure each event’s worth against your goals. Is there alignment?

Even when you have to say no, you can leave the door open for future collaborations. DevRel is a long game. You can’t (and shouldn’t) feel like you have to get it all done in one day, or even one quarter. The trick is to have a plan, to know what it’ll take to execute well on that plan, and then pace yourself for the long haul.

And my final point here, most Developer Relations team members are engineers or have to be super technical and product-focused. Even if they are hired within the marketing organization, you should be compensating them on engineering and product pay scales.

A Final Word–What This All Looks Like IRL

While this entire piece has been about the importance of metrics and measurement, I’d like to close out with a caution to avoid getting too caught up in the numbers. Measuring performance is always helpful, but with a practice like DevRel, you don’t want to let the pendulum swing too far to the purely quantitative. At its roots, DevRel is about relationships and building community, and—as we’ve already noted—the ROI of a community and relationship building is hard to measure.

And even though you do measure against deliverable and performance goals, keep in mind that DevRel’s efforts do not stand on their own. They only exist in the larger context of the entire company. DevRel’s primary responsibility is not the final outcomes that move the needle on things like revenue, referrals or retention. DevRel’s primary responsibility is to build the relationships that lead to opportunities that support those company objectives.

The most effective way to measure the ROI of DevRel is to find a happy medium that provides a framework and some feedback while allowing DevRel the freedom to focus on what’s most important: connecting with developers in authentic ways, having meaningful conversations and helping them solve their problems. The DevRel way is all about building affinity, camaraderie and trust. You never know what benefits might grow out of even the smallest interaction, so you want to be careful not to let rigid metrics devalue non-measurable activities.

Successful DevRel has a lot to do with strategy and planning, but it also has to do with following your instincts. There will be times when you act based more on intuition than intel, and that’s okay. If you’re doing a good job measuring the things you can measure, it’ll all balance out in the end.