Podcast: True Observability in a Real-time World with Charity Majors

May 19, 2020 in Eventador Streams Podcast



Podcast: True Observability in a Real-time World with Charity Majors

In this episode of the Eventador Streams podcast, we had the chance to look a little outside the traditional Apache Flink, Apache Kafka discussions that we’ve been having to chat about another side of data systems—observability. Kenny and I were joined by Charity Majors, Co-founder and CTO at Honeycomb.io for a fantastic and fun conversation about how and why observability is something that all organizations should have as an integral part of their data systems.

Learn more about Honeycomb, observability and the fun of building a business from the ground up in this episode of the Eventador Streams podcast:

Want to make sure you never miss an episode? You can follow Eventador Streams on a variety of platforms including Soundcloud, Apple Podcasts, Google Play, Spotify, and more. Happy listening!

Episode 07 Transcript: True Observability in a Real-time World with Charity Majors

Leslie Denson: If you’re in any way involved in developing, optimizing or responding to incidents for your streaming data systems or even your overall data systems, you’ve likely heard of our next guest and the company that she co-founded. Kenny and I had the pleasure of chatting with Charity Majors of Honeycomb.io about the intricacies involved with system observability and the ones involved with founding a company, on this episode of Eventador Streams, a podcast about all things streaming data.

LD: Hey, everybody. Welcome back to another episode of Eventador Streams. Kenny and I are really excited today to have Charity Majors, who’s the co-founder and CTO of Honeycomb on the line with us today. How’s it going, Charity?

Charity Majors: Oh, you know, COVID, the pandemic, the world’s on fire so everything’s pretty great as far as I’m concerned.

LD: Surviving.

CM: Yeah.

LD: Yeah. We know the feeling. We know the feeling for sure. The work from home, the dogs, the kids, the everything. Well, we appreciate you taking the time, especially with as busy as I’m sure everything is to chat with us today and based on the guests that we’ve had in the past and the conversations that we have we think this one will be super interesting for everybody ’cause it’s a little different than what we’ve done because of what Honeycomb does. So with that, why don’t you give us a little bit of background for those who may not know on you and Honeycomb and what you guys do?

CM: Sure, now I wanna look at your lineup to see who you’ve had in the past to see what’s different about it. Well, I’m an infra-nerd, sometimes database, but by default not like Kenny, I didn’t really study it, but I’ve been on call since I was 17, like well over half my life, and for the past four, four and a half years… Gosh, time flies… I’ve been working on Honeycomb, which is kind of next generation systems observability. Observability is… Is this really grew out of our experience at Parse, which is the company I worked at before Honeycomb and Parse was a very before its time, we were doing mobile apps as a service basically. And I was the first infra hire, which is really my sweet spot. I love doing that, I love joining as the first infra person with a group of software engineers and helping it kinda grow up and become a real company, right?

CM: And that’s what I was doing with Parse. Around the time we got acquired by Facebook, I was just coming to the horrified conclusion that we had built something that was basically undebuggable by some of the best software engineers in the world doing all of the “right things.” But every day there were either Parse would go down or somebody would be complaining and they’d be like, “Parse is down.” I’d be like, “Parse is not down. Behold my wall full of dashboards. They’re all green, everything is fine.” ‘Cause maybe it’s an app like a Disney or something, who’s doing four requests per second, which local traffic isn’t huge, volume-wise. And but we’re doing 100,000 requests per second on the back end, and it just, it literally, they could be completely down and it would never show up in my graphs.

CM: And this is kind of characteristic of the monolith versus microservices, where it used to be that your systems could be 99.9% up, and you’d be like, “Great, a 10th of a percent of a time, somebody’s getting an error.” And it was pretty evenly distributed ’cause everybody was always using the same app, the same database, and now it’s like you could have 99.9% up time and that means that everybody whose last name begins with S-H-I thinks you’re 100% down or it’s like all these little nooks and crannies throughout the system where you could have a consistent subset of people who are just like, “We’re just f*ed.” And yet your top level stuff looks the same, and it also gets to the problems of, if you didn’t predict that you were going to need to ask the question, then you can’t ask it, because you have to do these custom metrics and you’d want to, basically you’d be in a situation where, ’cause we had a million mobile apps, and what I wanted was to pre-generate a set of dashboards for every single one of those apps for every single one of the questions anyone would ask, which is cost-prohibitive and storage-prohibitive and energy-prohibitive and just in all these different ways you couldn’t do it, right?

CM: So I was running up against all of these just like, which I didn’t understand at the time, because we were doing microservices before microservices were cool, though we just didn’t have the language, we didn’t have the toolset, we didn’t have all this stuff, and I was just trying tool after tool, and the first glimmer of hope that we got was this tool at Facebook called Scuba which is, it is aggressively hostile to users, it is not a fun tool to use. But we started streaming some data sets into it.

Kenny Gorman: Some new UI paradigm right there, aggressive towards users.

CM: Aggressively hostile to users, yeah.

KG: I like that, that’s perfect.

CM: Yeah, but it let us do one thing really well, which was slice and dice on the benches of high cardinality in real time. And so suddenly we were able to start breaking down by, okay, one in a million mobile apps, okay, just breakdown by the app ID. Cool, now break down by end point, now calculate the latency, see which ones are slow. Oh, it seems to be the right endpoints that are slow. But is it all of them? Or what do they have in common? Oh, it seems to be the ones that are writing to Mongo. Was is it all of them? No, it’s just the ones that looked to this replica set. Is it all of them? No, it’s just the… So you could just take step by step and follow the trail of breadcrumbs to the answer every time without having to start by predicting what the answer was going to be in advance and then checking for it. And this was just revolutionary because suddenly we went from, it was open-ended amount of time that it would take us to answer one of these questions. It could be days, it could be never, and it just dropped like a rock to seconds, not even minutes.

LD: Wow.

CM: So it wasn’t even engineering problem anymore, it was a support problem. We just, we knew we could find the answer every time within a few clicks. And that just blew my mind. And I’m in ops. I didn’t actually dwell on this too long, I fixed the problem and I moved on ’cause that’s what I do, but when I was leaving Facebook, I suddenly kind of stopped and I went, “Oh, I don’t know how to engineer anymore without this kind of tooling, it’s become so core, not just to how I solve problems or when the site’s down but this has become so integral to how I think about engineering, this is how I build things now, by hypothesizing, by instrumentation first, sort of observability driven development where you don’t start by building the thing, you start by building instrumentation, so that you know whether the thing’s worth doing or not.

KG: It sounds like you guys were ahead of the curve anyway, right? So if I had a three-tier app kind of… We would do what you said anyway, but it would take five minutes and we’d be done with it. The RCA would be longer than the debugging steps. But you’re saying microservices changed the world, right? I think I’m hearing you right…

CM: Yeah. Microservices changed the world, yeah.

KG: And because of the microservices are so prevalent, and why wouldn’t they be, right? Everybody just wants to write some little snippet or piece of logic or whatever and deploy it so easy, right? Quote, unquote, Easy. But then you’re screwed because you can’t observe it later, you can’t debug it…

CM: Well, microservices made this absolutely overwhelmingly obvious to everyone, but I think that it’s always been the case that this would have been the better way to do things. It’s just the historical pathway of… Well, Etsy and StatsD, and then we all kind of… The entire industry went down this evolutionary fork of metrics where it’s the number with the tags appended, which is absolute, you cannot have high cardinality with that. And so we all just kind of just bent our brains in the direction of thinking that this was the only way to do things, which is why I find that kids who are fresh out of college or software engineers who have never done the ops stuff find it much easier to adopt the observability sort of mental framework and paradigm than people who have been doing ops for 20 years.

KG: So I’ve got one for you, I’ve got one for you. So way back in the day ’cause you said ops for 20 years, and I can identify, but way back in the day, we used to work at eBay, and we were in the NOC and the NOC was like, imagine, Cape Canaveral with these tiered desks, and there were SysAdmins and DBAs.

CM: Yeah, yeah, yeah, totally.

KG: It was totally like you would expect it to be… That’s not necessarily… It’s kind of the thing of the past now, although I think EBay does still have the NOC exactly the same way it was whatever many years ago, but we used to have the TV up, and we would have CNN on, and that was a monitoring point. So, CNN’s like, “eBay’s down! No one can get in.” We’re like, “Really? Oh, where?” That was actually a monitoring endpoint for us. Think about how far we’ve come since the days of watching the news channel.

CM: How far we have and haven’t come ’cause when healthcare.gov rolled out their site, they did the same thing. They literally had TV up, and they’d be like, “Oh, guess it’s down again.” [laughter]

KG: Right, yeah, wow. [laughter]

CM: Yeah, I think the biggest struggle that we have in trying to show people the light is, first of all, convincing them that there is a better way ’cause every vendor says this, and so we really have to let our users talk for us. Our users have to go out there and say, “Hey, I adopted observability, and I cut the number of paging alerts I have by 90%.” Right, ’cause every single one of them does, but we can’t say that, we have to let them say that. Just believing that there’s a better way and believing that it’s accessible to them. That’s the other thing, so many people, they hear these things, but they feel like it’s not for them, right? Especially if they don’t work in Silicon Valley, or whatever, they just don’t think that they’re good enough, or that it’s accessible or that… And the thing is, it’s easier to write code if you can put your glasses on first, if you can see what you’re doing, if you can watch it roll out, if you can watch users interact with it. It is so much easier than trying to build these mental sand castles in the sky, right? That’s what takes super geniuses. Real people like me, we need to see what we’re doing. So, part of our struggle has just been convincing people this is actually easier, and it is available to you, it is accessible to you, it is not just for the 1%.

LD: It makes a lot of sense to us too. I can understand what you’re saying ’cause I know what our team goes through as they’ve been trying to debug and figure out problems with people, our customers’ systems.

CM: Oh, it’s so hard!

LD: It’s so hard!

CM: It’s so hard when you can’t see what’s going on, when you get in the habit of just starting by adding instrumentation first, then you can see what’s going on, you can see what’s coming out the other side. It always… I always laugh my ass off when I see software engineers try to debug their code by looking at it. I’m just like, what do you think you’re gonna see the 20th time you read this s*t? You’re not gonna see it. You’ve gotta look…

KG: My linter’s not good enough, yeah.

CM: Your code is not the source of truth, your production is the source of truth.

KG: That’s a good point. So tell me, Kubernetes is obviously been crazy as of late, in terms of adoption. We use it internally. I’ve had my challenges with it. Now I just say, “Hey, send me this or that.” I don’t even wanna log into the database anymore. I used to just say, “I’ll just poke into the database, I’ll figure that out.” Now, Kubernetes is so abstract that I’m just like, “Guys, help me out here.” I don’t even know how to get in there anymore. And for all the great things that it brings to the table in terms of scalability and containerization and our ability that distributed systems and scaling, they work great, they work great together.

CM: Yeah, but most people don’t need it.

KG: Yeah, so tell me about your thoughts there. How does that play into Honeycomb.io? And how does that play into your thinking kinda going forward? Is it opportunity, is it terrible, is, what?

CM: There’s a lot of self-inflicted pain that’s being caused right now by people adopting Kubernetes. It’s kind of brilliant what they’ve done, and also it kinda makes me sad because this is resume-driven development. It’s become something that engineers feel like they need to know in order to get jobs. So then they bring it in, but everybody looks… We all know that you don’t actually need it, but the individuals need to know it so that they can get the good jobs. So that’s sort of unfortunate, but whatever, it’s not the first time, and it won’t be the last. It’s sort of fun that you asked that because I’ve seen recently a couple of leaked pitches for people who are trying to raise money to do observability startups, like Honeycomb for Kubernetes.

CM: And I’m just rolling my eyes because… But it’s not about the… Kubernetes is just a… It’s not where the logic that you need to debug lives, it’s like the new syslogd as far as your code is concerned, right? It’s just another high cardinality dimension. You tag the event with the name of the kubelet, and that’s all that you need. There’s no… You should be running them as dumb things that you don’t need to debug, right? All the logic that you should be changing on a daily basis lives in your code. So, observability for Kubernetes is just observability. It’s just part of the environment. It’s just part of what you gather, and it never changes, but this is something that seems to click with almost nobody, so maybe they’ll raise money on it. I don’t know, so I don’t know…

KG: Interesting, interesting.

CM: It’s like saying observability for AWS instances or observability for Docker, it’s just the thing that runs the thing that you need to care about, yeah.

KG: It’s a characteristic, yeah. In fact, it makes it more abstract. It makes your problem solving more opaque and ultimately I think it makes your product more interesting, right?

CM: Yes. Hopefully, you can configure your Kubernetes so that it is as dumb as possible because the goal is not to be doing things with Kubernetes, that’s not what your differentiator is, it is not what your… Presumably what your business is about, unless you’re one of the many, many, startups that is doing Kubernetes things.

KG: That’s right, yeah, right, exactly. So tell me about so where… The listenership to this podcast are data nerds for the most part in one way or another, tell me about your data problem. Is it a streaming data for event streaming events as some people call it type data problem? Tell me a little bit about how you stream events from your end points, and how you capture them, and you don’t have to give us anything too deep, but it’d be interesting to hear your take on that kind of space.

CM: Yeah, sure, I actually, I don’t think I’ve kept up on what the kids mean by streaming events. And specifically, we use the term event, but what we mean is the fundamental building block of observability is an arbitrarily wide structured data blob. It needs to be arbitrarily wide… And it needs to not have to care about indexing or anything like that, because of the fundamental tenets of observability are that you need to be able to gather data, just throw more details in at any time without having to worry about any sort of schema or anything like that, just throw something in when you see it, and don’t throw it in when you don’t see it, right? Which is why anything that has any sort of schemas is very anathema.

CM: If you’ve installed this is a library in your code, like the Honeycomb Beelines, what they do is they handle all the stuff like exponential backoffs and graceful retries, and bundling up the data in the right format. So when your request enters a service that’s been instrumented with Honeycomb Beelines, we initialize an empty Honeycomb event, and then we pre-populate it with anything that we know or can guess about the event, any parameters that were passed in, any details about the environment that you’re running in, like the container, or the instance, the Amazon instance, or your IPs, or the language internals, anything that we know or can guess, we just pre-populate it with. And then while your code is executing, you can also… It’s basically like doing a print-off to a log, but instead you print off the bit of data to the Honeycomb event, and you just stuff in anything that you think might be interesting as your code is executing. And this is ’cause we can’t guess what you’re gonna name the variables in there, right?

CM: We can do things like we automatically wrap any HTTP calls out, so we’ll capture what you sent out, what the timing of that event was, what the latency was. We’ll wrap any database queries, we’ll store the normalized query, and the raw query and how long it took to execute and all that stuff. But any variables that… You might have a shopping cart ID, you would wanna stuff that in. And then when the request is ready to exit, or error, then we ship it off to Honeycomb as one arbitrarily wide structured data blob. And we also do batching and stuff for very high frequency stuff, and we also have some helpers who are sampling there. But I don’t think that that’s what you mean when you’re talking about streaming events.

KG: Well, it is. Well, ultimately I was curious about, yeah, what is… And you dove right into it, what is the payload? What are some of the attributes that would be interesting? Are you inserting to a database serialized or are you sending these events off to some buffer or… And then how do you aggregate them? And I’m even interested in, do you use Protobuf or Avro or is it just raw JSON or kinda what’s your plan of attack, or what was your plan of attack when you designed this?

CM: Yeah, so we don’t care, by the time it hits Honeycomb, we have an API where we just expect there to be this arbitrarily wide structured event, we don’t care how it gets to there. And I have come to think of the library’s instrumentations being a third of the magic, but fundamentally, you can just run a curl request from your shell, and it’s the same f*ing data blob. But we do have, I think for Golang we do use Protobufs because we use that ourselves, so that’s very efficient.

KG: Yeah, makes sense.

CM: But you can do whatever you want fundamentally, we’ve always meant for these more to be helpers, because people who go deep in observability, inevitably have their own stuff that they care about that they want to do to make it nice for their environment. I’ve been a little surprised that people have not wanted to build much themselves. I always kind of felt like instrumentation was a thing that people would do but…

KG: For you. 

CM: So we’re investing more into that side, it’s cool.

KG: Makes sense.

CM: Yeah. But once the event reaches our API, there’s a thin little API and then we drop it into Kafka ’cause we do want them to be… We wanna be able to partition them by and shard them by user, distribute them, ’cause on our backend, we have this great very distributed data store, which we borrowed a bunch of stuff from the Scuba white paper, so we’re like, “Aha, some stuff that we do not have to reinvent from scratch.” And it’s basically an append-only columnar database. And we recently… There’s a great blog post about this. We recently moved most of it to serverless, actually, both the running of queries is done by serverless functions now, and we age it pretty rapidly out to S3. We were afraid that it would be much slower, and we have very high… In order for you to do observability you have to be able to ask an infinite amount of questions very quickly, it has to be stay in your state of flow. You have to be able to ask a question and a question and a question and a question. So our 95th percentile, we strive for one second for those queries, so we were afraid that we wouldn’t be able to support that in S3, but in fact, I would say that nothing on average got slower. The distribution changed somewhat, but those serverless snippets and S3 functions are badass. 

KG: So the users typically care about the… It’s some sort of now minus some time variable generally, right?

CM: Yeah.

KG: So you must have something where like, “Hey, data a month ago is not as interesting as data two seconds ago.”

CM: Yeah, exactly.

KG: ‘Cause I’ve got a production problem right now.

CM: Exactly. We try to keep the most recent couple a hours or so, depending on the data set. For some people it’s a day, for some people it’s an hour, but yeah, we keep the most recent…

KG: That’s cool.

CM: In what we kind of think of as RAM, which are just the SSD volumes, and then we age it out to S3.

KG: Right. And so you mentioned, so maybe not everybody knows, but you mentioned a column store database. And what’s interesting, and append only, so what I’m hearing is, obviously, with a column store and append only, those go together pretty nicely because you don’t need indexes and so you can look up any…

CM: Well, they are indexes. [laughter]

KG: Yeah, right.

CM: They’re all indexes.

KG: Each one of them is an index. That’s right. That’s the trick.

CM: Exactly.

KG: And so if you must have a dimension like you maybe you’re looking up your data by, AWS instance type you mentioned earlier, and you want to see all the M5.xlarges or something, and how are they performing, then that would be a dimension that would not need an index and you can probably group by those. I’m just assuming that that’s how your design was laid out.

CM: Yeah. You can group by any of the dimensions that you’re dripping into Honeycomb.

KG: That’s a powerful paradigm when you go to a column store.

CM: It’s really awesome. Yeah.

LD: Yeah, cool.

CM: So if you if you’ve got like a million mobile apps, you can group by that one in a million and then group by… It’s amazing when you remove that constraint of, ooh, I can’t have high cardinality, or ooh, I can’t have eventuality, and you just start stringing together, “I can have this, and I can have this, and I can have this, and I can have this,” and you just start to be able to ask these incredibly fine grained questions on the fly.

KG: Okay, so you’re using a column store, how do you handle late arriving data and updates? Or do you update?

CM: We don’t, we do not update.

KG: So, instead there’s a timestamp?

CM: There’s a timestamp. Yeah.

KG: Gotcha. Okay, cool.

CM: Yeah. And, there’s a timestamp that you can give the data and a timestamp that we can give the data. And so there’s some subtleties in there and I don’t remember exactly what the nitty nitty-gritties are ’cause I don’t do that anymore. [laughter] But we’ve gone back and forth about which is going to be considered the canonical source of truth and there’s a bit of a buffer there where we reconcile in memory. And yeah, so there’s bits that are fun, but not.

KG: Yeah, like event time versus like capture time or something like that, right?

CM: Yeah.

KG: Yeah. Gotcha. Okay. Makes tons of sense. And I think most of the folks listening to this, that’s gonna resonate. It’s obvious that when you put an event in Kafka, it gets a timestamp at that point. But for your system, that may not be the interesting point. It may be, at the client side, where it said, “No, this event happened 10 minutes ago and I was so busy, I couldn’t get this event out to the bus at the right time or whatever.”

CM: Right. And then you’ve got people who will be like, “Cool, I’m just gonna bulk upload my last week’s worth of logs all at once.” And you’re like, “Okay, [laughter] cool. Let’s rate limit this a bit.” [laughter]

KG: Got it. Got it. So that’s awesome. We never got the… I’m aware of your platform and we’ve obviously chatted a bunch in the past and we’ve known each other for a while, but I really never got the nitty gritty around how you guys manage data and how it works and how you designed it in your head. So that’s super cool.

CM: Yeah, yeah, it is. I really, I’m surprised that there haven’t been more columnar stores.

KG: Right.

CM: Whether it’s open source or whatever. We didn’t want to start by writing a database. We would have given anything to not spend our first two years not giving anything to customers, not showing anything to our investors, right? But we had to because there was nothing out there. Druid was the thing that was closest that we considered doing, but it ultimately wasn’t flexible enough. It was too much like a metric storm, and it didn’t allow for, we would have had to rewrite a third of it just to have flexible schemas.

KG: Well back in the day, we used to use Infobright and it was fantastic, except it was just a single database. So, as soon as you tapped out that largest box you could buy from them way back in the day, you were screwed. And I always wondered, why is there not a better ecosystem from a distributed database standpoint that, and I think you’re right, you jumped right in there. It’s really the index that matters. Everything’s indexed, and I mean compression is fantastic.

CM: Exactly. In the future, nothing matters if you can’t search for it.

KG: Right. Exactly. Exactly. So, I always wondered about that. And there’s obviously solutions out there and some of them, by the way, are very, very expensive, and so there’s kind of the proprietary angle, but, I don’t know, open source never really kind of caught up to where I would have… If I would have pulled out a crystal ball 15 years ago, I would have said, “Oh, yeah, column stores are gonna take over.” But it really has been slower than I thought. I agree. It’s interesting.

CM: Yeah. Yeah, I don’t really have any theories there. I don’t know. I guess MySQL has been good enough for most people for so long. And then once they have to jump the chasm to something much better, they all go proprietary, they all go in-house, is what it feels like. But it’s kind of a shame. I feel like the entire open source ecosystem of databases is withering, like there are the ones that are too big to fail but where are the new entrants? That’s not where the money is. That’s not where anyone’s going. All the idealists have kind of graduated.

KG: Yeah, there was a huge influx around when MongoDB came out and then Riak and…

CM: Which was around the last recession, wasn’t it?

KG: I think so. I think so. That roughly… Yeah, here we go. There’s gonna be more, there’s gonna be more.

CM: Fingers crossed. We figured it out.

KG: Yep. Causality. There’s a recession, databases come out.

CM: There you go.

KG: You heard it here first.

CM: We can hope. Well, honestly, we’ve talked about open sourcing our data store, and I would do it in a heartbeat if we get out of survival mode, and if we can find a cost paradigm that works. I would love to open source it. I’m a big believer in open source, but I’m an even bigger believer in survival, and that’s where we’re at right now.

KG: Yeah. So tell us a little bit more about entrepreneurship. That’s something that I think over the last few years, both us and you have, it’s been somewhat new. Tell us how that’s going, give the folks some color on, is it what you thought it would be? Is it harder, easier? What, give us some color.

CM: I gave approximately zero thought into it. I’m gonna say some things that I’m really not supposed to say, because whatever. ‘Cause data nerds and investors don’t listen to these podcasts, right? But I always assumed that we would fail. I, 1000%, I was just like this tool must exist because I will be a sh*y engineer without it. I can’t go back to not having this tool, so I’m gonna build it. It will fail and then I’ll open source it and that’ll be fine. [chuckle]

KG: Right. We’ll go from there. Right.

CM: We’ll go from there. I 1000% expected that to be the outcome, and we keep not failing. Now, the other unfortunate thing that happened, was we started, there were three of us, and I was not supposed to be the CEO, and unfortunately, we parted ways with our third founder. And then I had to be the CEO, and that I never would have done this if I had expected that to be an outcome, ’cause I know myself and my strengths and…

KG: Now we’re on the same page.

CM: And they are not…

KG: Now we’re on the same page, Charity.

CM: They are not CEO. And that, I did it for three and a half years, we finally reached product market fit the last quarter that I was CEO, and I’m still recovering honestly. It’s a year later, and I’m still recovering…

KG: Traumatic.

CM: From the aftermath. I say that with tongue in cheek, but also not. It was traumatic. I burned relationships, I burned health, it was not really great for me. Christina’s a much better CEO than I was, and I’m relieved and grateful that we have survived it and I think that we’re gonna be okay, ’cause our customers are our best, they are huge advocates for us, and they are really great at spreading the word. And I think that we’ve really made an impact for them. I think we’re over that hump where it depends on me, I don’t think it depends on me anymore, thank God. But it’s one of those things that you look back and you go, “Yeah, I never would have done that if I had known.” And I will only do it once and I’ve done it. [laughter] And I won’t do it again.

KG: Right. Well, that’s the thing, right? It takes a serial entrepreneur, and it’s like, “Man, are you crazy?” And I think, you mentioned Christina, I share a great fondness for my co-founder, Erik. And you know Erik, and he’s a great guy. But, I’ll tell you what, we were just talking the other day, and startups take twists and turns you never expect.

CM: Yeah, they do.

KG: And when we look back, we’ve been doing this for a number of years now, and I just go, “Holy sh*t, man.” I just think back about when we first started there was three of us, we were trying to do this and that. We had this vague idea of what was broken, and that we knew we had to fix it. And it’s just like, “Well, are you sure you’re gonna be successful?” And I agree, we never looked ahead and said, “Oh yeah, we’re gonna kill this, and we’re gonna this or that.” Mostly it was like, “No, no, can we convince someone to see it like we see it? Can they actually… ” And most of the time I walked away saying, “No, we’re too early or we’re dumb or they don’t get it or they’re dumb or we’re both dumb or whatever.”

CM: Yeah, but it’s like a religious conviction. You have this religious…

KG: That’s right.

CM: You’ve seen this thing, you can’t un-see it, you know everyone’s gonna see it too, eventually, and you just wanna make their lives better.

KG: I said this thing on the last podcast, which I don’t know in order if you’ll hear it first or second, but there’s this thing called product-market fit, everybody knows that, but there’s also founder-market fit, or founder-product fit if you wanna call it that, and it’s like, “Are you gonna wake up every day and totally care more than anything else to try and see your dream through and see it through?” And it sounds like you had that good fit, because it was a problem you had,” and you wouldn’t give up until it was solved. And then you were screwed, because you didn’t know how to do it in either way, and I think that’s hilarious.

CM: Yeah, no, I was not. I can see being a serial entrepreneur, but I think that one thing about starting company is not to get too all airy fairy Silicon Valley. But you learn so much about yourself. Like you go through the fire and it’s just like, “Okay, now I know. I know that I’m not meant to be CEO, I know that that’s not my thing,” but yeah, I think there’s also the… If you’re lucky, you can accelerate things that the world is destined to see and make them see it sooner. I think that when we started talking about this, everybody was just like… Literally I had so many people tell us… Roll their eyes at us and just say, “It’s a solved problem. Datadog’s about to go public. What are you even doing here? You’re late to the party. There’s nothing to do here, go home.”

CM: And we just kept stubbornly being like, “No, but there’s this huge gap between what they’re capable of and what the world needs. And it’s hurting you, and if you can’t see that… ” And it took three years of me traveling, 50% of the time saying this over and over and over. And slowly other people started it… There were a lot of people who could see it, but they are not typically the people that are listened to. And it kinda took a while for it to percolate up and for it to gain this critical mass. And then that last fall, the last fall that I was CEO, there was a QCon track that was observability, and that’s when I knew, “Oh, this is starting to pop into the mainstream.”

KG: That’s right.

CM: Of course, then all of the big companies went and tried to steal our message and go, “Ooh, we do observability too,” and now I’m just like…

KG: Yeah, now it’s a buzzword. So walk me through this, Charity, walk me through this because I gotta know. So you have these detailed thoughts about how this should be. You’ve got sort of an engineering… You obviously have a gigantic and massive and wonderful engineering background, and you’re pitching to Silicon Valley, and you’re in that room, in that boardroom or whatever, and you’re pitching this. Tell me some color on how that went.

CM: Oh, well the first round, our third co-founder was a guy. And so, mostly they just talked to him.

LD: Oh, God.

CM: I just remember that one now and then, and it was fine, it was fine.

KG: Really?

CM: Then we had to fire him, and then people were… Took the second lick, and they were just like, “What the f* is going on here?” No, I honestly… I’ll give some of these guys more credit than that. I was speaking from a deep, whole other experience, right? And I think that they saw our conviction and our passion and our certainty, and they kinda just went, “Well, there might be something here. It might not be what they think it is, but they seem really determined.” And lots of companies end up pivoting, lots of companies end up doing not exactly… We thought we were an ops tool to start with, it took us two years to realize we were software engineers, so they weren’t wrong.

KG: That’s interesting. Yeah, we’ve had a similar kind of path, and I remember, if you think back how different that very first pitch is compared to the ones that you might give now, today, right?

CM: Yeah. Oh my God, yeah.

KG: And you said your own growth. For me, that’s been the whole thing. I know databases and I know data, that’s never been the hard part, it’s always been about, How do you actually communicate to that, to someone in a believable way, especially when it sounds like to them it’s a lie, or it sounds like to them it’s a trick, or whatever. They’re like, “This just sounds ridiculous.”

CM: And they have 16 others like this today.

KG: Yeah, and they’ve got four advisors that say you’re stupid too. And you’re like, “Well okay, I probably am stupid, but this is working for me and I’m gonna have to just keep going forward.” I think that’s an interesting insight that you brought up there. That’s cool.

CM: Yeah. Well, I’ve never really doubted for confidence in myself and my own beliefs.

LD: I love it.

CM: But, there is a meeting halfway that you have to go through, right?

KG: Yeah, for sure.

CM: You have to make it so that other people can consume it, and if you can’t do that, then you’re screwed, no matter what. And so VCs are kind of like your first customers. If you can’t explain it to them in a way that will make them want to give you money, how do you expect to explain it in a way that will make anyone want to give you money?

KG: That’s a good point.

LD: And I love that you don’t doubt in confidence in yourself because you shouldn’t, and I think that is something that all co-founders need and I think we have two. Erik and Kenny are great, I think that’s something that…

KG: Mostly Erik.

LD: A lot of co-founders… Did you say, “Don’t say Erik”?

KG: No, I said it’s mostly Erik.

LD: I thought you said, “Don’t say Erik.” I was like, “He shouldn’t listen to this podcast then.” No, but I think that’s something even the best struggle with, and to your point, I think you’ve gotta have that confidence in yourself in order to sell it to anybody. Investors, customers, whatever it is, because you gotta know your s*t’s rad.

CM: Yeah. Well, it’s also just a function of the amount of feedback that you have coming in, and not all of it is going to be positive at any given time. It’s a delicate dance of when do you decide to listen to the naysayers, because sometimes you need to listen to them in order to get better. And sometimes you need to not listen to them in order to keep going, and that is the trickiest needle to thread.

KG: I agree. Yeah.

LD: We actually talk about that quite a bit internally when it comes to stuff, because if you ask a hundred customers, you’ll probably get a hundred different responses as to what they think should be next on the road map, or what they think your product should do. But what’s actually accurate, maybe zero of those. You just have to be able to infer from what they’re saying what they’re actually thinking.

CM: This is where I feel like, if your product doesn’t have a north star, if there isn’t something you know you believe in. I’m a big believer in opinionated products. I think they make it easier for everyone, but you have to know what that opinion is. And it’s not like every startup has to follow this pattern. I’m advising a startup right now who is much more… They’re a couple of guys who… They want to start a company, and they don’t really care so much what it’s about, they have background, they have experience, and so they’re just trying things to see what fits. And that is another completely valid way to do things, but it is not the way that I started out, it is not the way that you started out, Kenny.

KG: Right, yeah.

CM: We started out with a north star that we were gonna do or bust, and so I think you just need to know what kind of company you are and keep it along the way.

KG: Not give up, yeah.

LD: Well, I think I say this too often, but I mean it every time when I say I think that was a lot of fun.

CM: That was great.

LD: I enjoyed that. Again, and I know I said this earlier, we’ve talked a lot with people who are very steeped in the streaming data industry, ’cause that’s what we are, that’s what podcast is, but this goes hand-in-hand with… Observability, monitoring, all that stuff goes hand in hand with what we’re doing. So again, I think it’ll be really interesting to people, and it was a great, different perspective of this whole thing.

CM: Good.

KG: For sure.

CM: I loved it. Thank you, Charity, so much for coming. I appreciate it.

CM: Absolutely.

KG: Yeah, thanks Charity. Good to chat again.

LD: Huge thanks to Charity for joining us on the podcast today. Really enjoyed that conversation. And while it may have been a slight departure from our normal Flink and Kafka programming, it’s definitely one that’s important to anyone relying on streaming data. If you haven’t done so yet, check out the great work Honeycomb is doing at Honeycomb.io. And while you’re at it, also check out Eventador.io, to learn more about how the Eventador Platform can help you unlock more value from your streaming data. As always, you can reach out to us at hello@eventador.io, with any questions or comments that you have. Happy streaming.

Leave a Reply

Your email address will not be published. Required fields are marked *