Developer User Research extends your runway, with Ana Hevesi

Hi, everyone. You're listening to Scaling Dev Tools, the show that investigates how dev tools go from 0 to 1. I'm joined today by Anna Haversi, who is a dev tools UX researcher and developer ecosystem consultant. Anna has a ton of experience in DevRel, open source, community management at companies like Stack Overflow, Nodejitsu, and MongoDB. The very observant of you will notice that I also worked at Stack Overflow.

And we were there at the same time, but we unfortunately did not meet. Anna, thank you so much for joining.

Jack, absolutely. It is great to be here. I'm sorry that, you know, we didn't meet while we were actually officially colleagues, but, like, better late than never. Right?

Absolutely. And I'm really excited about this episode because I think that messaging, user research, just really understanding your customer, your developer is just so important. So thank you really for your time. Anna, what drew you to start doing user research on developers?

I have been in open source community management and developer relations for a long time over a couple of cycles, a lot of different companies and ecosystems. And what I found over time was that a lot of the efforts that we employed, that me, my colleagues, my teammates, the direct reports employed to reach users, we were not always as clear as I felt we could have been about why a given thing was the right thing to do and why it was having the impact that we wanted to have. And so somewhat recently, I was in the community team of a premature public organization, and we were working on what exactly to do next and, you know, how the community team fit into the subset of of developer relations, activities and so on. And I said, hey. I think there's really an opportunity to get specific with our users, especially, you know, given the maturity of this product.

Let's sit down. Let's actually do some developer user research with our users, and what came out of that was all of these very specific opportunities that weren't really surfacing via, you know, existing marketing processes or weren't necessarily on the radar of the developer advocacy team, but that were, for example, really exciting use cases that we hadn't even known but that might have made sense to continue to support, or it got to be very clear that folks were eager to start start learning from from each other on a certain area of the product. And I I I shared the findings with everyone, and folks said, this is great. Why haven't we been doing this sooner? And at that point, I felt like I was probably onto something.

And that's where I went out, as an independent consultant, and have been offering this ever since.

And, Anna, how do you usually conduct developer user research at start ups?

So I really like to start with a question that I think of as, like, the anchor question. And you want this to be big, open ended, and crucial to the business. And so examples of questions, very often, we're just starting off together, me and my clients, with, in what ways does this product solve meaningful problems for our users? But it can also be, you know, depending on where things are, a little bit more fine grained than that. It can be things like, why do people get involved in our support community?

Or why aren't we attracting more open source contributors, or is this pivot the right evolution for our product? And so when we're early as so many of my early stage clients are in this practice and getting to know our users, I recommend I like to have us do a broad swath of different types of users. I like to kind of go as wide as possible, And so we will optimize for people who, have varying educational backgrounds, have spent really different amounts of time in the field, who have different roles within their orgs, who have different representations of race, gender, so on and so forth. And I tend to find that going wide that way means we surface the most opportunities to find the things that we should be drilling into further later. They're gonna be useful down the line.

So, you know, from there, we we reach out. We we ask users for their time. For once where they take that piece very seriously because when users are giving us their time, like, that is an absolute gift. I'll write a script which uncovers 2 things together, and it's both where they are in their journeys as technologists so we can understand what their goals are in their careers next, in their lives next, and then we also get into a lot of nitty gritty about how they are using the product. Typically, we will either analyze we we will either take recordings of the conversations or I will have someone riding shotgun and taking notes.

From there, I wind up sitting down and I do I dive deep into an analysis phase. I'm a big fan, by the way, of the user research tool Dovetail. I frequently use that for this step. But there are sort of 2 axes or information types that I look for, and one is things that are sort of very easy to observe. Like, okay.

Well, this person has spent 5 years in the industry, or this is someone's title in their current job, etcetera, etcetera. After I've done a pass for those sort of clear and observable things, I will dig deeper, and that's when I start to subs out, oh, that comment about their experience with this product, that was I'm getting frustration or I'm getting. And that piece, having the sort of cut and dry objective information is really necessary to know the product and the person well enough to to contextualize that emotion. But from there, these start to become patterns. Right?

They become themes. And so what happens is I will begin layering these instances of these different themes, reactions, etcetera, on top of each other. And then what we have is quotes that I will pull out that me and other folks within the company can refer to, and we can we can look around together and say, this looks like this, especially around this point in the product. That making sense to everybody? Absolutely.

Great. Okay.

So you're kind of bringing those lots of interviews down into actionable things. How do you use those findings that you pull out?

What I will try to do here is give you some specific enough pet pictures or pads inward while being clear for you, me, and everyone who's listening. The details vary so widely based on the people, the situation, the product at hand? And it's really that variation that we're using as a point of power and leverage, but nonetheless. So we've conducted these user interviews. We, as I said, have started out with a very deliberate anchor question.

And some of the things that we are going to know after this process, I spoke to this already, but we're we're we're going to have a demographic level and and sort of very clear cut picture of what kinds of users are currently having the best time with the product. And so to speak to some of the attributes that can be revealed there, sometimes we get clear on, oh, okay. Well, there's, for example, a lot of TypeScript usage that goes along with our product. Okay. Cool.

Or alright. There are a lot of people using the product right now who are tech leads and sort of straddling this this place between, management and individual contributorship. Just as a couple of examples, we're gonna understand who is being impacted by the tool and by the product better. From there, we are also going to have as I mentioned, we're going to have discussed where the product is giving people superpowers and where it's kinda onerous getting to be onerous for them. And so we're gonna have a much better understanding of the gaps, and that could be what we identify as gaps in how the product itself is working.

Maybe it's gaps in how clear or direct error messages are. Maybe it is gaps in the documentation. We're gonna have to start to understand where our model of this tool and how it works and how it will be received, where there's a delta between that and our users. So from there, we can start to infer a lot of things about what we do next and how we spend our time. That can mean we're going to focus on, improving documentation for some more specific use cases past just the get up and running stage of the product, or it can point to some really good other tools or products to spend time doing integration with.

We could also have it revealed to us that there are a lot of our users who want to better understand how to improve in one specific area, and, you know, that can be an area of this tool and this product, but it's sort of the tip of the iceberg of a larger scale that they want to improve at. And so maybe there's an opportunity to start getting folks together in, you know, you and your dev rel team getting folks together in a sort of structured or semi structured, you know, lightning talks or some other sort of pure learning opportunity. Some things like that.

Anna, one of the things that you've spoken a lot about is how user research can extend our runway at a start up. How can user research extend our runway?

So it's a matter of conducting developer user research, making it possible to gain insights that we don't otherwise have the ability to gain, which allows us to run much more deliberate experiments. And if you, as a founder, are looking at time spent headcount, the the different directions you go in and effort you put in. This is going to allow you to draw way more specific pictures of what you're doing over what amount of time and why, and all of that is going to enable you to basically just chart a much clearer navigation path. One thing that comes up sometimes I I wanna mention is some folks will will you know, I have a chat with them, and they'll say, well, I'm I'm actually talking to my users all the time. And, so I I don't think we we really need to to do this.

In fact, you know, we hang out in Discord together all the time, and sometimes people will, you know, drop a feature request, and we'll implement it on the spot. And I always wanna pause at that moment because while that sort of in the weeds and on the front lines perpetual rapid interaction with our users and the very, very high responsiveness with and to our users that I'm describing here, that can look and sound from the outside the same as the developer user research practices that I'm talking about. However, they're actually distinct because what I've what I spoke to a little bit in the practices when I work with a client is we're basically putting a box around things, and we're making a little laboratory. And that makes it possible to isolate the variables, which is what in what allows you to see how one problem connects to one potential solution you wanna experiment with and how much of your cycles you're going to spend doing that. Whereas if you are having tons of ongoing interaction with your users and maybe implementing things based on their request in a very unstructured fashion, it makes it incredibly hard to isolate the variables.

So to be clear, I'm absolutely not saying don't interact with your users, don't build relationships with your users. I'm not for a moment saying that, but I am saying that taking a step back, implementing these practices deliberately, and then deciding what the next best and right thing to do is based on the tangible evidence that we've come to is going to allow you to chart a deliberate course in pace with your runway. So there are gonna be so many fewer surprises.

That makes so much sense. It can feel so often at start up like a lurching from one direction to another based on random pieces of feedback or conversations. And if you're making big business, pieces of feedback or conversations. And if you're making big business decisions, directional decisions, it completely makes sense that it's worth doing this properly in a in a lab, as you say. You know, a lot of that is understandable, though.

Right?

Because the the rate at which people in the world are becoming software developers has just been accelerating and accelerating and accelerating. And if we go back several decades, someone being a programmer, someone being a software engineer meant sort of one reasonably clear thing. And now, I was looking up these numbers the other day. The market research firm SlashData said in 2019 that there are 25,000,000 developers in the world right now and that that number is expected to increase to 45,000,000 by the year 2030. So between boot camps, between more traditional paths, between the wide variety of different jobs in tech and subsets of the industry, it makes sense that it is difficult to figure out which action connects to which form of impact in our product experience and in our user adoption efforts.

But it is actually possible to create you know, draw this box, to create this lab, as I said, and we can calibrate our instruments so much more finally. And I'm really excited about what that can do for people and how that can help start ups get further. So what I would like to leave folks with is that this can help you make better trade offs, and you're already doing developer user research, whether or not you plan to. The question is how expensive is it, what is it costing you, how much sooner could you have this information, and what could you do with the differential in your runway if you had this information now. So basically developer user research is a means of taking your navigational instruments and calibrating them much more finely and thoroughly so everyone in your org knows where they're going and so that you as a founder can make an informed decision about how much of your fuel you're using to get to your next destination.

Amazing. Where can people learn more about you and contact you if they want help?

You can find my website at anahevisi.com. That'll get you everything you need to know about my approach to reach out to me. Once again, that is anahevesi.com.

Amazing. And, Anna, thank you so much for joining. And, everyone, thank you so much for listening, and we'll see you again very soon.