Episode 111
· 40:01
Was recently talking to Alana who is the founder of Basecase. She invested in like super base, browser base, resend lots of cool dev tools and I asked her what new dev tools she's excited about and she said Braintrust. It's an eval platform for LLM so to see how well a response did based on the prompts. And they've had just crazy enterprise adoption. So they're working with Figma, they're working with Vercel, they're working with Stripe, Airtable and they just raised $36,000,000 in series a from Andreessen Horowitz.
Jack Bridger:So doing really well. So I was like, yeah. That sounds great. Do you know Anko? Could you put us in touch?
Jack Bridger:And she said, yeah. He's my husband. Dev Tools family. So thank you, Alana, for making this happen. It's a very interesting episode because I think Ankur kind of peels back how they've been able to do this and it really seems to me that the secret to their success has been the user research.
Jack Bridger:How heavily involved they've been with talking to the enterprises, really understanding what their actual burning problems are and then delivering a product that solves that. So I think it's a really good episode. Enjoy.
Ankur Goyal:Focus is really hard. I think as an entrepreneur or an investor or or, you know, whatever your role is in starting a company, I think, actually, one of the most important things you can do is create the framework by which the company can focus. And that's very difficult, because you end up leaving a lot of things on the table that you might wanna do. And I have failed many, many times, through my, embarrassingly long career of attempting to do things, at providing focus for for folks, and and suffered the consequences, you know, in a variety of different ways. So when we started thinking about Braintrust, what what was quite relieving is that there actually weren't that many companies that had done anything interesting in AI.
Ankur Goyal:Our first few users were Zapier, Airtable, Coda, who are all customers that had released AI products, like, 6 months ahead of anyone else. So, basically, Elad and I wrote down a list of 50 companies. It wasn't exactly 50. It was around 50. And these were, like, the only people that we thought were either already doing stuff with generative AI or were about to do stuff with generative AI.
Ankur Goyal:And, and that was our target list. And and we got, now we're we're almost all the way through, I think, with all of those companies getting them to be customers. But we got pretty much everyone on that list to be either a customer or an investor. Some of them had already built internal tooling. Some of them for some of them, the problem was maybe 6 months away or something like that.
Ankur Goyal:So not everyone immediately became a customer. But enough did that we sort of formed this early nexus that helped us build the product, you know, for a very small group of people. And that that kind of got the the wheels going. And I think one of the things that, has been really nice, because we've had a fairly narrow, target customer, is that we don't do a lot of product management or prioritization or anything like that. We sort of assume, you know, if, Zapier is asking about something or if Notion or whatever is asking about something, it's probably something that other companies are going to want too.
Ankur Goyal:And maybe they'll want it, like, a month or 2 from now. And that's pretty much, you know, it's pretty much held to be true. So, having that concentration is also very relieving because it allows you to really build what your customers are asking for and and do that knowing that it'll be repeatable.
Jack Bridger:Yeah. I think that's, like, a really, actually, powerful point that you're like it seems like you're going after the kind of people, the front runners, the kind of trendsetters, and then it's like, if you just make them happy, everyone else is gonna catch up and want the same thing.
Ankur Goyal:Yeah. Yeah. That's that's kind of our one of our core hypotheses around around the company.
Jack Bridger:To go after the, like, customers who are the front runners, like, good companies.
Ankur Goyal:Yeah. And and I think in particular, like, you know, AI is a new way of building software. I I've had the, you know, maybe, at this point, 8 years of accumulated wounds and scars working on it. But we for a lot of people, especially those of us without a statistics background, this is maybe the 1st year that you've, you know, seriously built AI software. And there's a lot of things that go into building it well, that by working with the companies that are at the very forefront of it, we're able to codify into the product.
Ankur Goyal:And I think, for me, the most rewarding thing about BrainTrust is when we meet a customer who starts using our product, and especially starts investing in evals. And then, like, a few weeks later, they're like, oh my god. This has completely changed how I build AI software. And I I can't even imagine what it was like before this. Like, a good example, the Vercel folks released a blog post about how they do evals.
Ankur Goyal:They use Braintrust, and they've used Braintrust, you know, since the very beginning of that journey. They've been incredible partners for us. They've given us great feedback. We're also Vercel customers, so it's a lot of fun to work with them. But it's just very cool to see how how much they've embraced the idea of building AI software around doing good evals and just how far it it's taken it's taken them and and their product.
Ankur Goyal:So, yeah, I I find that if we work with the innovators and we kind of codify their learnings into the product, then, we're able to build this exceptional product that, you know, the next as people sign up, they just, like, instantly accelerate their journey to to where the innovators are.
Jack Bridger:This episode is brought to you by WorkOS. At some point, you're gonna land a big customer, and they're gonna ask you for enterprise features. That's where Workhorse comes in because they give you these features out the box. Features like skin provisioning, SAML authentication, and audit logs. They have an easy to use API, and they're trusted by big dev tools like Vercel as well as smaller fast growing dev tools like Nock.
Jack Bridger:So if you're looking to cross the enterprise chasm and make yourself enterprise ready, check out Work OS. We've also done an episode with Michael, the founder of Work OS, where he shares a lot of tips around crossing the enterprise chasm, landing your first enterprise deals and making sure that you're ready for them. Thanks WorkOS for sponsoring the podcast and back to the show. Yeah. And and how, like, for Vercel in particular there, like, how did you start working with Vercel, and how like, what were the kind of initial conversations like?
Ankur Goyal:Yeah. I think it started the way that it does for many companies that work with Vercel, which is, they get connected to Guillermo, and then Guillermo, shits all over their product and gives them a click with feedback. And then and then they come back asking for more, because his feedback is great. You know, he is an awesome guy. And so we started our journey that way as well.
Ankur Goyal:You know? Guillermo, had lots of fun things to say about Braintrust. But, in the process, there's this incredibly sharp guy named Max who is a beast and, built, you know, a large fraction of v zero and, was working on their eval stuff at the time, and and Max was also very skeptical. But as v zero evolved and brain trust evolved and we got to know each other better and all this other stuff, at some point, what happened, and this happens with most people, that use our product, is they sort of have this moment where maybe they shipped something and it broke, or they switched a model and it broke everything, or they had a production issue. And they're like, I you know, I okay.
Ankur Goyal:I I I really need to to to get better here. And so I think it was one weekend or something. Max just kind of installed Braintrust and got the the the the, you know, the basic loop plugged in. And then it just started going really quickly from there. I think a few other engineers on the team started embracing it.
Ankur Goyal:It became part of the PR workflow. And, actually, I remember working with Max sent me a very, strongly worded message at some point about how, challenging it is to, copy paste results from BrainTrust into their GitHub PRs. And so we built a GitHub action that makes that really easy. And, and and, yeah, I mean, I think at this point now, they they sort of, I don't wanna put words in their mouth, but I think the blog post is really great. But, you know, as they as they create new features in vzero or, iterate on the product in in any way, they they sort of do it in an eval first way.
Ankur Goyal:And and I think, basically, once you it it's a little bit like unit testing, but it's more extreme because in AI, it's when you manually verify something, you can't necessarily assume that it'll work for, you know, the next data point. But, once you sort of embrace the flow of building around evals, you just start going way faster, and it feels like you're making systematic progress. And so I think it you just start to accelerate. And I I think they experienced that at some point a few months ago, and it's it's just been very cool to see.
Jack Bridger:Yeah. That that's really interesting, and, it's funny that you mentioned twice that there was, like, you know, a strongly worded email, and Guillermo shitting all over the products. And it feels like that's probably quite actually, like, a crucial thing and and a positive thing that, like, you're actually you were putting it you were having those conversations and, like, you were putting things out there, like, and getting that feedback.
Ankur Goyal:Yeah. I I love the, Kari from linear. I love how he his framework or sort of, rejection of product metrics. He's, like, the first, you know, successful founder or whatever I've heard that's been so clear about that because I've I've always agreed but not had the courage to speak up about it. And I I think, my kind of comparable thing is I I really I feel energized when people are complaining about your product.
Ankur Goyal:And and the reason is, to me, I know that that means that they're using it. Anything any other type of information that you get that it complained, it's not per it's not totally clear that they're actually using the product. There's a lot of products that people sort of vibe with, and they're like, oh, yeah. I like I I'm, like, so into this product. And then you ask them, and they've they've literally never used it.
Ankur Goyal:Yeah. And yeah. So to me, when people are are complaining about your thing, 1, obviously, every little ounce of feedback you can get is, incredibly valuable because it's an opportunity to improve your offering. But that that that is the, it's kind of like the heartbeat that I tap into, to know that someone is is actually, deriving some value from the product.
Jack Bridger:Yeah. I I think it's a really, really good point. I think there's so there can be so much, like, to just get, this is cool, and people on Product Hunt saying, great. Like, looks looks good. I'll start it.
Jack Bridger:And it's kinda meaningless, but, actually, someone's saying, oh, but can it do this? Or, oh, I looked and it can't. I don't these sorts of things are, like, so much more valuable because the person spent, like, a 100 x amount of time, like, looking at this thing and, like, cared enough. For sure. And and in your case, actually using it and, like, having specific issues with, like, their own workflow is, like, so much more valuable.
Ankur Goyal:I totally agree. Yeah. Kind of going back
Jack Bridger:to this a little bit more, like, you mentioned Elad Gil, like, a few times, and it sounds like their relationship is a lot more than kind of he's like a he you said he incubated you. Could you talk a bit about, like, about that?
Ankur Goyal:Yeah. I mean, look. It's it's it's I I would say it's genuinely difficult for me to point to, someone outside of Atlanta who's been more impactful, in my life over the past decade. Yeah. Elad is a very special person.
Ankur Goyal:We met at a pizza shop back in, I think, like, 2016 or something. And we started chatting, and then he invested in Impira. And he was always always more helpful than the dollar value or, you know, equity ownership or timing or any of these things kind of would suggest. And that is a very unique quality that it sounds simple to not be transactional or to always be helpful or whatever, but very few people actually put it into into practice. You know, n might equal 1 or maybe 2 or 3, in this case.
Ankur Goyal:And so, Elad and I, just kept, you know, chatting and getting closer and closer and closer. And when I got Impira acquired, he was, like, really, really helpful and involved. I think, you know, acquisitions are a very complex and not always fun process. You know, you you really understand when you're going through an acquisition who actually gives a shit about you and and who doesn't. It's it's sort of one of those experiences.
Ankur Goyal:And Ilhan was incredibly, incredibly helpful. And so, you know, we, when when when I was at Figma, we just just not with any particular agenda, but we just always are, you know, chatting about stuff. And one of the things we were chatting about is how we had to basically build this internal tooling, at Figma, and then it's exactly the same thing that we had built at Impura. And that sort of Elad, that was kind of a light bulb moment for him. Like, you know, why did you build this?
Ankur Goyal:Why couldn't you use the other tools that were already available? And we'd sort of talked about that in-depth. It's one of those things where, I actually thought the idea was really stupid because I had done it twice already, and so it felt really boring, and and and maybe too obvious or, small. But that's where you sort of have the benefit of perspective, and I think Elad is also, world class at thinking about markets and all this other stuff. But Elad, was like, you know, actually, I think this is a pretty interesting and big idea.
Ankur Goyal:And I think one of the key insights into brain trust is that, you know, if you assume AI is gonna be a thing, I was a little bit jaded having worked on AI and ML for a while and and thinking about how small the market is. But now that, GBT, you know, at this point, 3 had come out and all this other stuff, AI is, applicable to a much wider audience. In fact, the market size of, you know, developer tooling for AI goes from being stats PhDs and data scientists to all software engineers, which is an enormous market. And that was kind of a that was a really important insight, I think, that we we sort of chatted through. And, you know, Braintrust started out as something we were going to incubate, for fun, and we talked to, we made that list of 50 people, and then we reached out to a bunch of them and did kind of open ended interviews.
Ankur Goyal:And, I I've never experienced anything like doing that. One, it was just great to sort of learn a little bit from Elad about how to do those interviews really well, outside of pitching or already being committed to an idea. A few people, especially, Brian from Zapier, who's really been an exceptional, partner and and customer and and so on for us. He reached out after we chatted with him and was like, we need this now. You know, we're and, again, they were really far ahead.
Ankur Goyal:I think they still are really far ahead. But he is like, you know, it was just me building AI stuff. Now we have another team, and we wanna have more teams. And we're just totally we have no dev workflow. We we we don't know how to how to organize this stuff really well.
Ankur Goyal:Please help us. And so that that doesn't happen every day. And that's kind of what got me incredibly excited about about the opportunity. I I love building stuff, and I love, you know, building stuff for working with people that, you know, actually care about it. And so, that was that was sort of the conviction that I needed to to jump into it.
Ankur Goyal:And that's kind of how how Braintrust got got started there. Yeah.
Jack Bridger:Yeah. That's that's pretty amazing when you've got, like, companies like Zapier reaching out to you and, actually, you know, wanting wanting what you've got at such an early stage.
Ankur Goyal:Yeah. I mean, I I I think one of the things I've learned over time is that, rather than thinking about it as, I have an idea and it's super exciting that x company is excited about my idea, I think it's actually better to invert it and say, I am a, you know, good person, and I'm going to work on ideas that are so important that they are a top end priority for, executives at my target customer. And if you think about it from that perspective, then you sort of, you go from I'm solving this problem or I'm building this software to solve this problem to I wanna find a problem that is actually so meaningful that that happens. And I think if you work on a problem that has that kind of significance, then these things follow very naturally, and it's sort of an implementation detail to get in front of the right people, which is not to minimize the act of doing that. It's just that, you know, Brian is not a he doesn't discriminate.
Ankur Goyal:You know, if you're if if you're some random person who has a great idea and could potentially solve it, I think he'd probably wanna talk to you. But, yeah, I I it's just something I've learned. You know, prior to Braintrust, I would always say, like, I'm doing this. What do you think? And I I just think it's it's useful, especially if you're trying to start a company, to search for the problem that actually, one, you are passionate about solving, but, 2, you know, sort of inspires or motivates people like that.
Jack Bridger:Yeah. Maybe this is obvious to you, but I think it's quite hard for a lot of people, including me. It's like, how how what kind of how are you asking these kind of questions? Like like, how are you kind of teasing that out of someone like Brian?
Ankur Goyal:Yeah. I would say another thing I've learned from Elad is that there are a few things that are, somewhat counterintuitive that are very valuable to spend a lot of time on. And I think in general, you could say that, preparation is one of those things. But in this case, like, let's say you're doing user interviews. I would say prior to this exercise, I would just wing it and say, oh, hey.
Ankur Goyal:Let me jump on the call and sort of have a free form discussion with someone about my idea. And I I just think that's a terrible idea. What what we did is Elon and I spent, like, probably a week or 2 going back and forth on a Google Doc where we, wrote out a bunch of questions, and we were sort of debating with each other about how to sequence the questions and, sketching out my answers based on my own experience. And we sort of boiled it down to a few assumptions. One of the interesting things early on is we thought BrainTrust would have to be open source, because, it's a dev tool and a lot of you know, I I'd been working in databases before.
Ankur Goyal:You know, databases are now all open source. And so I remember even in the intro email that we sent to people, we were like, hey. And it's probably gonna be open source. And a few people viscerally, negatively responded to that. And they said, no.
Ankur Goyal:Please don't make it open source because that suggests that I'm gonna have to put in more work to run this thing than I actually want to. I just want infrastructure that's, you know, like Datadog. It's just I just turn it on, and I I never have to think about it again, and it solves these problems for me. Or GitHub. Right?
Ankur Goyal:They're neither of these products is open source. I I don't think you necessarily want them to be. I mean, there are, of course, great open source equivalents, but, I I could tell you, I don't wanna use an open source, GitHub or Datadog. It's not not the problem I I I I I really wanna solve. And so, yeah, so anyway, we boiled down these assumptions, and we actually spent a lot of time working on the interview structure before we did the interview.
Ankur Goyal:And there's all the classic stuff, like, don't be biased, don't ask leading questions, try to ask questions that have, structured answers so that you can correlate information across answers. One of the things that we learned, for example, is that not that many people were building AI or wanted to build their AI software in Python. Braintrust, one of the things we've done really well is we have a really fantastic TypeScript SDK. And we're I would definitely consider us to be TypeScript first. But in AI at the time, that was incredibly counterintuitive.
Ankur Goyal:And because we asked the question of what programming languages are you using to build the software, we learned that very quickly in a structured format. So I I don't think any of this is rocket science other than the fact that you should actually spend a lot of time working on the interview, before you start just, hey. You know, can we grab coffee and chat? Also, I think the user experience for the person you're interviewing is way better if you are prepared and you have a very structured set of questions. They feel like it's a better use of their time.
Jack Bridger:Yeah. Do you remember any of the questions that you asked?
Ankur Goyal:Yeah. For sure. I mean, I think we asked questions like, what programming languages are you using? What cloud services are you using? What databases are you using?
Ankur Goyal:What, are how are you thinking about prioritizing internal versus external projects? Are you working with text, images, or other modalities? How are you logging and monitoring performance today? How would you like to do that tomorrow? What is the area of the workflow that gives you the most anxiety?
Ankur Goyal:What's stopping you from, rolling this out to more users? Is it cost, availability, team members on your side, or something else? Like, questions like that. And and, actually, when we asked these questions, it became incredibly obvious in almost all cases that evals were the bottleneck. Really?
Jack Bridger:That's Yeah. Yeah. Those are such good questions. I feel like and the anxiety one, I don't know if I've heard that. That's a really good question.
Ankur Goyal:Yeah. Yeah. Yeah. Yeah. I mean, I think the key is just to ask questions that, are not leading and, sort of give you, emotional insight into how someone is thinking about about about the problem.
Jack Bridger:And when you started talking about it, so so you ask these questions and they tell you, you know, they say, we're very anxious about evals. Like, we have no idea if we're doing a good job. It's all vibes. What is, like, the next step beyond that?
Ankur Goyal:Yeah. So we spent about 10 minutes at the end of these the calls were usually 30 minutes, and we spent about 10 minutes at the end, kind of pitching what we were thinking about. And I think when you pitch something, especially if you have a smile on your face, and or the person maybe knows you or whatever, people are generally gonna say, oh, yeah. That sounds great. So there's there's nothing to take away from that.
Ankur Goyal:I think the value in pitching them is to seed the idea so that if it is actually a problem, that they will be interested in exploring further with you. So we did that, maybe not with a super clear agenda, but just to seed the idea. And what happened is that multiple people followed up with us and said, please, we need this now, which, is quite I I think that was quite tell telling about the, severity of the of the challenge that the early adopters were were sort of having with, with Evals. Yeah. So that that's kind of that's that's sort of how we how we did that.
Jack Bridger:Yeah. That's, is is very interesting. And you'd already started kinda working on evals, at this point. Right? You'd already
Ankur Goyal:I mean, I I hadn't written any code yet, but I, I I had built the thing twice. And so I I I kinda knew the the where all the bodies are buried and how to do it, in a in a good way. And so it it I think we built the first prototype in, like, you know, maybe 3 or 4 days.
Jack Bridger:Yeah. I I guess this is, like, somewhat or this is a very hypothetical question, but do you think if they'd have actually not said, you know, evals, like, would you have just kind of tried to solve the problem of the thing that they were anxious about that came up? Would you have, like, switched?
Ankur Goyal:It's a good question. You know, I almost didn't even wanna work on it because I thought it was too boring even though they said they even though they said, evals. There was another idea I was gonna, thinking about and and other other stuff floating through my head, and I was very distracted at work. And I had you know? So, you know, I I don't know.
Ankur Goyal:It's it's it's it's hard to think about how these things play out, but, I I think it was the fact that people really wanted this thing that, that pushed me in the first place. And it it sort of helped me overcome the hump of like, okay. This seems very boring to build, but, will probably be useful. I think if it were something, you know, if it were something outside of my wheelhouse, then I probably would have, you know, collaborated with Elad and maybe tried to hire some people or find another avenue to kind of, give the insight a home. But the interesting thing about evals is that they are, at the end of the day, a data problem.
Ankur Goyal:And I have been working on data and databases for over a decade. And so it is a problem that I feel both very well equipped to solve and, very passionate about, you know, typing. I just I, you know, I I love working on things related to data.
Jack Bridger:Yeah. Yeah. And I guess, like, the kind of looking ahead. So you've been working with a load of companies, like, you the ones ones you've mentioned, and I think if anyone goes to your website, they'll see a lot of, like, household tech names, on your on your logos of customers. What are you like you just raised, like, a kind of pretty substantial series a, I think, 36,000,000, which is, like Yeah.
Jack Bridger:Pretty big, I think, even for SF, AI standards. What what are your kinda what are you thinking? Like, what are what are you, kind of focusing on now? Like, where do you see the the next areas of focus?
Ankur Goyal:Yeah. Yeah. So, I mean, I think there's 2 obvious things that any company like us should care about, you know, aside from company building and hiring people and all that stuff. It's what are we doing on the product side, and how are we making the product accessible to more people? So on the product side, what's interesting is that people have been pulling us into more and more and more interesting things.
Ankur Goyal:And and by that, I mean, once someone starts building their evals in Braintrust, they want to kind of use Braintrust for everything. Like, the first thing that happened is people said, okay. How do I get data for my evals? Can you build logging? And so we built logging.
Ankur Goyal:And I would say the unique thing about our logging is that it is really, really designed specifically to help you diagnose particular user issues and then take the that diagnosis and save it into datasets that you can use to do evals, and, you know, fine tune models or whatever else you wanna do with the datasets. But, really, I think the high high order base evals in the future. And that workflow, like logs, datasets, evals, that is the primary focus of or has been the primary focus for our logging and observability features. What's interesting about that is I think if you just started and said, I wanna do observability for AI, you probably wouldn't that's not probably what the product you would build. And I think, actually, what's interesting is that there are many products out there that just do observability for AI that are not that different from the workflow that you might have in, you know, a Datadog or something else.
Ankur Goyal:But we've really focused on that, making that sort of AI centric logs, datasets, evals thing work really well. Now, of course, once people start doing that, then they're like, hey. I already have all these logs in Braintrust. Now I also want to monitor everything in Braintrust because I I have a bunch of data here. So, you know, people have been pulling us in that direction.
Ankur Goyal:The other thing that's, really exciting, I would actually say potentially more exciting, and more strategic for us is that once people run evals and brain trust, they they wanna spend as much of their time in our UI as possible. It's much easier to work on AI in an environment like the BrainTrust UI, where you can, for example, run a prompt or test it out on, like, 10 data points rather than your IDE, where the prompt is, like, you know, in a jail cell of a format string in some file and very, very hard to access and actually work with. And so we have been basically inching towards these steps that make Braintrust more and more of an IDE like experience. And, I can't say everything that we're doing here, but I think we sort of see a world where code is, in AI is more of a scaffolding thing. So you kind of create the, integrations and auth and connect the vector database to, you know, like, just sort of set up the basic architecture and code.
Ankur Goyal:But a lot of the iteration, once you sort of create the scaffolding and code, happens in inside of BrainTrust. And that means, the prompt engineering, connecting prompts and tools and, other prompts and models together, testing, fine tuning stuff, evaluating stuff. It all happens in this kind of live and collaborative environment inside of BrainTrust. And I think pushing that road map is a non trivial thing because it basically means we're building a lot of just kind of the classic developer experiences, from the pre AI world into Braintrust. And so that is a an exciting and non trivial effort.
Ankur Goyal:But on the flip side, most of our customers who use us think of Braintrust as kind of, like, their developer platform for building AI software. And I think that is a really exciting place to be and and one we, you know, we we, sort of don't take the opportunity to do that lightly, and we want to invest seriously into it. On the on the go to market side, we just need more people. I mean, even the launch that we did, for the with the funding announcement a couple weeks ago has had a, you know, very nontrivial effect on our pipeline and and so on. And so we we we sort of we just need people.
Ankur Goyal:Like, if anyone is listening to this and you are a salesperson or a sales engineer or support person or or, you know, anyone who is somewhat technical, or can speak to technical people and wants to help them, you know, build better AI software, please reach out to me. We just we need more help and more people there. So, I think, you know, there's a, we have some really exciting ideas about how we wanna build the go to market out, but, you know, executing on that is another big area for us.
Jack Bridger:Yeah. Sounds like you're just getting a lot of inbound and just trying to keep up with the inbound.
Ankur Goyal:Yeah. I mean, I think we are getting a lot of inbound. We are also succeeding with outbound, which is a rare thing to see, at a company of our stage. And so, you know, there's a lot to there's a lot to, do there.
Jack Bridger:Very cool. Really a cool opportunity. Yeah. That's amazing. And okay.
Jack Bridger:So I have, 2 two last questions. What one is what advice would you give to another DevTool founder? And let's say an AI DevTool founder in particular, if if relevant.
Ankur Goyal:Yeah. I think I would have a very high precondition for the problem that you're solving. So first of all, don't make the mistake that I've made, you know, 5 or 6 times, which is I'm really I'm gonna build this thing. Like, I I'll give you an example of, like, a common thing people wanna build. It's like, hey.
Ankur Goyal:I've noticed that everyone wants to build a copilot into their product. I'm gonna build the SDK. It's like, you know, the Vercel AI SDK meets, you know, blah blah, whatever, into, like, a new sexy SDK for building copilots, and it'll have persistence and all this other like, you know, it's awesome that you like that idea, but everyone's thinking about that idea, and for some reason, it's not working. So why is that, and what actual problem are you solving? Another example of this is, something I talked to Swix about recently, and we had sort of a fun debate about it.
Ankur Goyal:I'm I'm probably not right, but I'll still make the point. I don't think fine tuning is a is a business. And the reason is that it is not a business problem. No company cares about fine tuning specifically other than fine tuning companies. They care about, achieving optimal performance for their AI models with optimal cost.
Ankur Goyal:And fine tuning is one of k methods that you might use to achieve that. So if you're overly, attached to one method, then, you're not you're not actually solving a business problem. And if your company is not solving a business problem, it's not a company. So I think the really the only thing that matters is figuring out what is the actual problem that you're solving, and being clear about it. And I think a really good, heuristic for this is if you can hold a conversation with a customer where the only thing you talked about is the problem, like, that's it.
Ankur Goyal:You just talked about the problem. Then that it's a good sign that you're working on a problem rather than a solution. And so I would just have a very high bar for that.
Jack Bridger:Yeah. I I love this. This is this is really good advice. Thank you. That's amazing.
Jack Bridger:And, okay, final question is, are there any other dev tools that you're really excited about right now?
Ankur Goyal:Well, we we talked a little bit about BrowserBase before this call. I I I think Paul and BrowserBase, and the team are are they're exciting because I I I think one of the most exciting things about software, and infrastructure is that it lets you do, things at scale that, like, a single workstation can't do. Actually, maybe I'd say this is the benefit of infrastructure software. Right? Like, when I was at MemSQL, I remember the first time we ran a distributed database at Zynga, which had a 100 nodes.
Ankur Goyal:And I opened I SSH'd into all 100 nodes on my laptop with a script, and I pulled up htop and watched the CPUs. And seeing a 100 CPUs go crazy, at once, it was just really cool. Or 100 sets of CPUs or, you know, 800 CPUs or whatever. And I think what they're doing is kind of, you know, creating the tools, the fundamental infrastructure that allows you to do that for the web. And, I think, you know, in particular, LLMs are the core unlock that, allow the web to be sort of manipulated and navigated in this kind of headless way.
Ankur Goyal:And I just think that's that's very exciting. I'm always a big fan of, what the Vercel folks are doing for for a few different reasons, but maybe one of them is that I think, you know, in this case, I would say Guillermo in particular and and Jared, have have really pushed on this excellence bar in, in in dev tools, and and they've kind of created a bunch of conventions, which maybe pre Vercel didn't seem to be conventions. Like, you know, in enterprise software and dev tools, quality was never and polish was never a thing that you felt like you could prioritize. It was always like, I I need SOC 2 or some SSO thing or what whatever. And, you know, the product can be ugly.
Ankur Goyal:Who cares? You know, all this, of course, no thanks to the advice that you get from VCs, who literally generally don't give a shit about the quality of your product. But I think I think, they they've really created a positive example for the ecosystem of caring about quality and polish and, and and stuff. And and I think they've also navigated a bunch of challenges around pricing and, you know, what does a free tier mean? And I know they they sometimes get slack for it.
Ankur Goyal:But in my opinion, they're really good people who are who have a very genuine, set of objectives that they're trying to achieve. And they're sort of navigating through this and creating a bunch of good case studies for the rest of us. So I just I kind of applaud what they do, and I I appreciate it. And I think, it sets a good example for for others who are building dev tools and care about quality. Yeah.
Ankur Goyal:So those are 22 that I I I think, about quite a bit. Yeah.
Jack Bridger:Yeah. There there was a tweet going around today of, Guillermo a picture of Guillermo from, like, more than 10 years ago when he was doing socket. Io, and, and then someone else commented with, like, all the thing like, saying, like, people forget how cracked Guillermo was, is, and all the things that he's done. And, yeah. Vercel is is pretty extraordinary company.
Jack Bridger:Yeah. Yeah. That's really cool. Okay. Browser based in Vercel.
Jack Bridger:And, and make sure you understand the problem and that you're really focusing on the problem that your customers have.
Ankur Goyal:For sure.
Jack Bridger:Yeah. Amazing. Well, thank you so much, Ankur. Thanks for coming on the, on the show, and Of course. Very excited about, Braintrust.
Ankur Goyal:Yeah. You're a great interviewer. I really appreciate you having me. It was a lot of fun.
Jack Bridger:Thank you. Thank you. And also shout out to Alana for making it happen. Yes. Thank you.
Jack Bridger:As always. As always.
Listen to Scaling DevTools using one of many popular podcasting apps or directories.
