Sentry with David Cramer

Hello, and welcome to podcast.init,

the podcast about Python and the people who make it great.

I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable.

For details on how to support the show, subscribe, join our newsletter, check out the show notes, and get in touch. You can visit our site at pythonpodcast.com.

Linode is sponsoring us this week. Check them out at linode.com/podcastinit,

and get a $20 credit to try out their fast and reliable Linux virtual service for your next project.

We are also sponsored by Sentry this week. Stop hoping your users will report bugs. Sentry's real time tracking gives you insight into production deployments and information to reproduce and fix crashes.

Check them out at gitcentury.com

and use the code podcast in it at signup to get a $50 credit.

You can also join our community at discourse. Pythonpodcast.com

for your opportunity to find out about upcoming guests, suggest questions, propose show ideas, talk to other listeners of the show, and follow-up with past guests.

Your hosts, as usual, are Tobias Macy and Chris Patti. And today, we're interviewing David Kramer about Sentry, which is an open source project as well as a hosted service for capturing and tracking exceptions in your applications.

David, could you please introduce yourself?

Yeah. Absolutely. Thanks, guys.

So my name is David Kramer.

I'm the founder

and CEO and original author of Sentry. It's an interesting thing, because I've been involved in the Python scene for a very long time, and Sentry was just an open source project there, which we, I guess, recently have turned into a real business.

And so that's my day to day now.

So how did you get introduced to Python?

Long, long time ago.

I don't even remember what version of Django,

but we were a PHP company, and I think I saw

some Rails screencast or something, you know, where they're quickly adding comments to a blog post. And I'm like, that's pretty cool.

Wasn't really captivated by the Ruby language.

But then right around that same time, Django was sort of,

I'd I'd say,

it was sprouting out of the ground. It was still very young, but I it was also doing some similar things. I'm like, okay. Python's kinda cool. I looked at it before, and that was, like, a decade ago now and,

kind of haven't looked back since then.

What led you to choose Python when you got started writing Sentry, and would you make the same choice again?

Yeah. I mean, like, I'm pretty much, of the belief that complexity is bad

almost a 100 percent of the time. You can make arguments that there are situations for it, but I'd make the arguments that,

you can get away without it. So for me, Python has, like, always been a great solution that it's a practical language.

For things like the web, it's fast enough, so you don't have any performance concerns.

It's actually very fast in some situations,

specifically ones that Sentry caters to,

that allow kind of run time introspection and things like that, just the way you use the, kind of the exception flow and everything.

So for me, like, when I when I got my start, it was purely about the accessibility of the language. I got I'm not a,

I'm self taught, so I'm not, like, academic.

Therefore, I've never been a big fan of,

any of that,

that thinking, I guess. Just because it was never ingrained in me. So I'm like, okay, I wanna learn something. I wanna get stuff done, which is why PHP was great.

And I did that over the years and, you know, when when we started

Sentry, it was you know, I was working at Disqus, and we were a Django Python company already, so it, you know, just made sense.

So the client side of error reporting needs to be super lightweight in order to be useful. What were some implementation challenges you faced around this issue?

Yeah. So it's interesting because we do SDKs in pretty much every language and every platform these days, and

the challenges are significantly different in each of them. So in Python,

I'd say the challenge is a lot about compatibility,

and it's also a cultural thing. So we've faced the challenge as a company for a long time. Like, how do we convince people that, like, error tracking is different than logging? Just like you you know, you wouldn't send New Relic performance reports to log files on disk, and you wouldn't send your, I don't know, slow query logs into New Relic, at least not, like, just drop in.

So we always face that challenge, and Python is probably the biggest,

culprit of this where it's like, oh, you can drop in some logging stuff,

and then all of a sudden it creates a poor experience.

So we face a lot of issues around that, but

I'd say equally in other languages, some of the challenges are the same. And it's like, how do we deal with ensuring that you don't accidentally send us credit cards? How do we ensure the the latency doesn't affect you? How do we ensure that, you know, the crash reporter doesn't crash your app? And Python has been pretty easy. I I'd say, like, we have a phenomenal team.

1 of my colleagues, Armand Ronak here, has been doing Python as long as I have, and

we're pretty,

pretty well versed there at this point. So I guess we have the foresight to solve a lot of problems before they become problems.

That's great. So just out of curiosity, I realize it's a Python podcast, but I I'm a big polyglot, you know, whatever works kind of person. What was the most challenging platform to implement the client side on for you guys or for you?

By far, browser JavaScript.

Say what you will about JavaScript,

whether you're a fan or not.

The complexities of the browser and the limited support there has made life very challenging for us. We've got a somebody else works on it at this point,

but even just getting a stack trace out of the browser is very hard.

And then you have, like, browser extensions playing their part, and you have just a mess of, like, different browsers reporting things differently. And it's even modern browsers that mess it up. For example, like, Firefox

has a different style of stack trace, which is just plain text than, like, Chrome. And I think in Internet Explorer, I don't even know if we support it, and I think it's because we can't or something along those lines. And then you have source maps and minified code on top of it, which are an entire another level of complexity.

So we we spent a significant amount of time making browser JavaScript work, I would say, fairly well, but it is it is not a fun problem.

Yeah. And then also add into that the localization settings that different people apply, you know, with different time zones, different languages, and that just compounds the problem. And I'm sure that that also affects your ability to aggregate those errors, particularly when it's coming in with different languages.

Yeah. So interestingly,

the way we do aggregate aggregation is,

we don't even care that much about the error. And that matters because in JavaScript, there aren't real errors. So in Python,

I'm gonna take Django, for example. If you, like, query a database row and it's not found, Django, by default, will raise an exception and an exception has a type, it does not exist, and then it has a clear value. That value is not always useful, in terms of aggregation because it contains, unique data, But the type is super important. And then most importantly is the stack trace. And fortunately, there's no obscure localization on stack traces in JavaScript,

and that's our primary,

grouping,

algorithm.

So often, we can group JavaScript errors together. The challenge becomes,

that it's almost like coroutines or like evented code in that the stack trace changes even if the code that's being run isn't necessarily different, if that makes sense. So part of the stack is the same, but part of it is,

I don't wanna say dynamic, there's just many kind of parents that would lead to the same

child. So there's definitely a lot of problems there that we have yet to solve, but, work in progress.

So we've talked a little bit about your implementation and language choice. But could you dig a little deeper into what Sentry is and a little bit more of its history?

Yeah. For sure. So I actually think Sentry has a really compelling

history. And while lately our space has gotten crowded,

Century's kind of been this long time project. We're about,

our 8th birthday is actually

just around the corner. It's, Thursday,

and, it's 8 years as an open source project. That's, like, when I made the first commits to, like, a public repository.

And we built it, like, from day 1 as this open source thing, which is a really cool part of our history. And more importantly, there was no business, you know, 8 years ago or even 4 years ago, I don't think. That was kind of added on later where we we said, hey. Like, a lot of people are asking if we would do, like, like, a hosted SaaS offering out of this. And, you know, we kinda said yes. But along the way, it was it was, very much scratch our own itch. Right? Like,

Sentry started out as a a drop in kind of extension for Django.

And that that was cool and, like, a lot of people were into Django at the time, and then there was a lot of other Python people that we are involved with and they're like, hey, it'd be nice if I didn't have to use Django to use this. Right? So, like, okay. Yeah. That would be nice. That'd be cool. You know? So then, you know, obvious next step, we make it work for Python outside of Django. And then, you know, Python developer and I think this is actually the story that happened. It was, like, Python developer goes to Ruby company. It's like, hey. I would really like to use Sentry for Ruby. You know, 1 thing led to another.

Here we are 8 years later in a, you know, massive support everywhere. But, like, the main differentiators beyond just, like, open source, I think are, like,

we're a pretty mature product, and I think especially in the Python world, most people know who we are. It's been interesting, like, we've started doing exhibits at, other trade shows, like, we were at RailsConf last week. And it's always interesting me the perspective that different ecosystems have.

Like, in Python, it's often people don't understand

the idea of what Sentry is. So that is we we instrument your application,

and when there's an unhandled exception, that is if if you have a website and it throws a 500 error, we wanna tell you, you, the developer, what that error is as fast as possible and with as much information as possible.

But, like, us web developers, we're so used to having log files that we'll often reason ourselves out of having good tools. We're like, well, I can already, like, look at the log files. Not that I'm going to, but I can.

Whereas, like, we talked to the Ruby community, and they, there's a very old tool called Airbreak, which was a couple other tools before, and it's been around as long as century.

And but they've had this for so long that it's, like, the idea of, like, this exception tracker is ingrained in what they do. And it's very interesting to see, like, the cultural and community differences there. But yeah. So, like, fundamentally,

we look at it and we say, like, you know, if you have Firefox,

this is my favorite example to use. No. No stabs at Firefox, but,

it crashes and it presents you with this dialogue. Right? It's like, do you wanna notify the developers that something's broken here? And we just, like the way we look at what we do is we do that, except we don't ask permission, and we do it on every single platform,

web, mobile, and desktop.

And then I think a lot of the differences

are we try to make we try to make use of the massive amount of error data and noise. So Sentry customers are some of the biggest technology companies out there, and it's it's pretty hard when you have a lot of errors, some of them constant, some of them you're never gonna fix,

to know what matters.

So a lot of our focus

is on then, yeah, you have a lot of errors. We've sort of told you how to fix the error if you wanna fix the error, but what ones matter to you? Right? And so we do that by saying, you know, like, hey. It's affecting a lot of users,

or this is new in this version of code and kind of things like that.

You've dug a bit into why somebody would want to use Sentry. Can you do a little compare and contrast from some of the other offerings that are available in the space?

Yeah. Sure. So we've seen in the last few years, there's a lot more of these. Like, I I feel like I discovered 1 a month. Most of them are single platform.

So there's a there's a few that do cross platform. Not that actual many. Like, I think it's 3 or 4. All of them are proprietary, which is an important difference. It's probably

externally, just looking at all the products, that's gonna be the first thing you'd notice and maybe the first relevant

thing to you. Outside of that, it's

it's really hard to say with a straight face that it's all that different. I think we're like, all of us are building different products, and I think everybody's brought some good things to the table.

But at the end of the day, a lot of what we're doing

is, like, not yet accomplished. So, like, a lot of the companies are still immature. Even though we're you know, we've been around for much longer than them, we didn't, like, I had a full time job

up until January of, last year.

So it's like we weren't really working on this like it was a company.

So I think we're still in this early stage of what kind of,

production crash monitoring really is.

But,

yeah, fundamentally,

the main differences between some of them are either they're cross platform or they're not. They're open source or they're not. And then down to that, like, for example, for us, we focus on a lot of workflow features.

Like, we're a big believer that

you shouldn't need to create an issue on GitHub for an error that's in Sentry

if you're just creating the issue so you can, like, reference it in your commit log. Right? I wanna be able to go in and say, like, I'm gonna commit this. It's gonna go through maybe a pull request, and I'm gonna just reference a Sentry error. And then we can do cool things, like, at that point, and we can say, hey, like,

within Sentry, it can be, like, okay, it's fixing this commit. That commits deploy to staging. So if the error happens in staging, we know that you did not fix it. Right? And a lot of it's really, like, how do we prove

if we look at, like, old school developer tooling I don't know if you guys have ever if you're familiar with Nagios, but it's this classic example of this, like, dinosaur application. It's not pretty. It's not fun to use. It's not fun to set up.

And we're like, you know, what if what if technology wasn't like that?

Why why are developer tools not just as important? Like, we pay developers ridiculous salaries,

but we don't have any tooling. And, like, you can contrast that with, let's take salespeople, and they have Salesforce, and they pay, I don't know, $100 a month

per,

salesperson at their company.

And then it's interesting to go look at, like, how much we pay developers. And, like, 1 of our challenges is how how do we convince a a company like Microsoft that might be willing to pay that for Salesforce, that they should pay that kind of money to developer tools? And not just Sentry, like, all developer tools so we can move away from the Nagios era onto stuff that actually is really important. And I think we're getting there. There's definitely an industry shift lately, but that's also why you're seeing a lot more of these kind of, these developer focused companies popping up even outside of the crash reporting space.

Yeah. There's definitely been a big surge in developer oriented SaaS products. So like you said, exception handling, but also things like monitoring with New Relic and its competitors

and log aggregation,

so things like Sumo Logic or Loggly or log entries and then also various other spaces.

Yeah. Absolutely.

And also continuous integration is 1 of the big ones there.

I'm definitely excited about this shift in technology. Like, we we think of this era as, like, the rise of the developer. It's kind of the same we have internally. And I think it's very much true. It's where, like, we're finally the I wouldn't say overvalued, but, again, the well paid developers are finally getting the choices in their tools,

which I mean, it makes a lot of sense when you think about it, but it's it's taken a long time

to the, to get the purchasing decisions off of, like, CIOs who have nothing to do with anything onto, like, allowing the developers sort of choice, I guess.

Yeah. And that's also come in conjunction with a lot more responsibility

being focused on developers and in a lot of places getting rid of some of the other facilities that were sort of tacked on. So QA and operations sort of merging into 1 overall department as opposed to them being discrete departments or discrete responsibilities within a department?

Yeah. Absolutely. Like, we we actually see that, 1 of the things Sentry tries to do and we haven't visibly shown a lot of these efforts yet, so you're getting a sneak peek into our inside plans. But we think it's important that bigger companies and smaller companies are able to, like, communicate effectively around building products. So I worked at Dropbox for a long time and, hopefully, none of them would get mad at me for saying this, but, like, the only time I really ever saw the support team interacting with product teams was when, like, it hit the fan. And that shouldn't be the case. It's just, like, the tooling and the systems we have in place aren't great. So 1 thing we do internally

is we have a strong tie in with

what are customers and how does that affect errors. So you can imagine there are errors affecting, say, me me, David Kramer on the site. I'm opening up a support ticket

and support is, like, okay, whatever. They're gonna they're gonna do whatever they normally do. Right? But if all of a sudden they now were, like, connected with Sentry and they could see, like, yeah. I see this error is happening to this user. Oh, also that error all of a sudden is happening to a lot of people. It helps bridge this gap between what really matters to companies, and that's, like, customers

and this closed product loop where you have silos of different teams not really communicating unless they have to. So we're super excited to, like, push that a little bit, but, still in the early stages.

Yeah. That's definitely a very important point to make is that things like Sentry and other logging services and metrics services really raise the visibility

of issues

at the customer level because it makes it much more evident to product owners and the business as a whole what kinds of issues the developers are facing, whereas I think historically, developers have been kind of pushed in the backroom. And we tell them what to build, they build it, and we don't really have much interaction beyond that where it now becomes much more of a broader dialogue where the developer is starting to gain more recognition and influence in the company as a whole.

Yeah. Absolutely. Especially with the rise of this idea where, you know, I, as a developer, should be able to push my own code and things like this pass and, even AWS

signal a lot of, like, we wanna be able to trust the responsibility of this, but often there's still that disconnect. So I I think it's there'll be even out way beyond Sentry, there'll be a lot of innovation in this area in the next few years.

Can you dig into the architecture of the Sentry project, both in terms of the software design and the infrastructure necessary to run it?

Yeah. Sure. So

at Sentry, we've always had this challenge because we were very big on, like, we wanna support the open source community. So our big challenge was, like, how do we keep it simple to run Sentry? How can you

on your VPS with no memory, no CPU run Sentry? And more importantly, that leads to how can we run it in development in a simple way, right? Well, then at the same time, we have big companies like like Uber and Airbnb and like Spotify, like running Sentry, and they send serious amounts of data to it. So how do we how do we take that same architecture and transform it to work at that scale as well? And so a lot of what we did, like, if you look at,

there's this trend, and I'm I'm not a I'm not on the same page as everyone else, but a trend of, like, microservices,

which is really just a new name for Soa.

And SOA

never caught on to small companies because it's such a complex and hard problem.

But I think the lessons of something like so are not that you have to deploy separate services,

like physical,

like this server over here is communicating with this server, but that just you should break up your systems. So for example, in Sentry, when an error comes into Sentry, the first thing it does is basic data validation and whatnot in Python. From there, we just immediately throw it into an ingestion pipeline, and that ingestion pipeline is either RabbitMQ or it's Redis. It's it's something that depends on your situation.

But the abstraction and that abstraction is common in Python, that's Celery. Right? It makes sense when you think about that. It's Celery that does this. But really what Celery has done is it's built a service that's handling

the push pop mechanism of a queue. And we take that same concept and we do it everywhere in Sentry. So once it goes through the ingestion pipeline, the workers pick it up, they do a bunch of stuff on it, and at some point they say, Okay, I need a I need a buffer, a counter, and that is like, I need to say this event has happened 1 additional time, I need to say that it's happened in this time series database, like, at this interval and stuff, and we, so we have an abstraction called buffers and the only implementation of that abstraction is Redis. It's either Redis or you don't use a buffer at all. And all that does is let us say, well, here's an interface we provided, it has like a, it's an anchor and a git or something like that, but you can swap it out with your own implementation that fits your own needs or your own infrastructure if you needed it, right? And while you might not do it in case because Redis is super simple, it doesn't really matter, we go farther and we get to the point where we store the data. Sentry operates on a lot of data. I think we are, I want to say above and beyond the largest product like this out there. And we do it in a way that we try to be efficient

and again we try to be scalable. So for us, the default in SolveSentry is primarily like a Postgres database.

So that is, you're going to have all the metadata in there. When you look at an error, it's a row in the database. There's another row that stores kind of the JSON representation of that error. And when you look at the data set, you'll find that a majority of the disk space used is in that JSON representation.

But if you look at the characteristics of that, all we do

we're not searching over that, we're not doing anything, we're just saying, I need to fetch this blob and I need to show it to the user or I need to add a new blob So there's a few very important characteristics 1, we never change the data there

and 2, we just fetch it by a key name and so if you look at that, it's very easy to be like, you could look and say, well, this kind of looks like memcache or something It's got like a git and a set, which it is kind of true Now, they won't really fit in memcache, but what they will do is they will easily fit in a simple interface that has a git, a set, and a delete. So we built another service called Node Store in Century, which has a SQL implementation that just runs on top of Django and is the default. For get century.com, we swap it out and we run React. And that, again, it's simple

service characteristics and it's very easy to operate, so it's great for us some other companies have swapped it out and they run Cassandra

and again, it's like fundamentally our thinking is like how do we keep things as simple as possible, which makes it really nice for us to develop against, so we don't have to run a production environment, but also makes it really nice for everybody to run it themselves. And there was actually a very long time where, we actually supported SQLite, Postgres, and MySQL. And, honestly, like, we sort of still do today. We try not to. And it has its challenges for sure doing things the way we do, but it's I think it's allowed us to

gain adoption in a way that's not common in the sense that, like, I I wish I could show you,

the actual list of customers we have, but it is it is phenomenal and it is every company.

And that's only been possible because of or the flexibility we've given

customers with using Sentry. But, yeah, fundamentally, like, if you look at it from a high level, there's a lot of these little services.

It looks kind of like a data pipeline, But realistically, it's just a SQL application on top of a typical, like, web stack.

And have you thought about adding the capability for swapping out some of the queuing with something more large scale like Kafka or 1 of the other data ingestion pipelines, maybe Kinesis?

Yeah. Absolutely. So it's funny you should mention that. I don't know when it's gonna happen, but we already have some prototype stuff and we have a spec, and we're absolutely going to swap it out. So our plan is we will natively, at the very least, we'll be supporting Kafka, and that'll be our recommended solution. But we've also written a Redis

implementation of Kafka.

So same characteristics as Kafka, which again, this is all part of the paradigm, like, build a simple interface, implement that interface, keep it small, keep it lightweight, and we were able to achieve it in in Redis very, very easily. So, yeah, sometime

soon we'll be doing that. It's kind of like surprisingly,

we have not hit any limitations yet. The only concerns today

are if you've ever run RabbitMQ, Rabbit cluster is not really great to work with. So your main issue is you don't really have any kind of availability. Like, if our broker goes down, like, we actually just have to, like, switch over to a new broker. There's, like, no magic. It just, like, literally pointed at a new server because the other ones whatever it's doing, it's not working anymore. And that's that's definitely an issue, but we've actually found it to not be that big of an issue, so we put it off for a long time.

It's really interesting to me

how pervasive

the microservices,

SOA,

meta pattern is becoming. And I really think, in a lot of ways, I realize you can get yourself in trouble with it certainly, and there are times not to use it. But I'm just kind of blown away, honestly,

by how many software projects and even how many organizations

are yielding really incredible results and having really positive things to say about having invested the time and effort into making something like that work for exactly the reason you specify. As you grow, as you scale, you can swap things out. And since everything has well defined interfaces,

it all just still works. I think that's really you know, speaking as someone who has been around for in this industry for a very long time, it's really incredible to see that become the norm.

Yeah. It is it's definitely interesting. I think there's a lot of things that need solved yet to make it truly accessible to people. I think just like, I don't wanna hate on anything, but just like MongoDB became successful back in the day, it was a little bit of a trap. Everybody dug themselves holes and they had to work around it in the end. And my fear is that there has been a lot of investment in some really cool technology. I'm I'm especially a fan of a lot of the Mesos tooling. Like, Twitter and some other companies have built some really cool frameworks on top of it. But my fear is that the state of production tooling, for example, in, like, many ways, including, like, Sentry, for example, is not nearly as good once you start saying why I have a 100 services to deal with versus, like, you know, 5 applications.

So I think we have a lot of distance to travel in terms of maturity of products around there. But it definitely seems like it's moving in that direction.

Yeah. And the truth is, you know, you mentioned we have a long way to go. We definitely do. But I think over the last couple of years, especially, like, the the ecosystem is responding. You see the growing pervasiveness

of things like Docker as a for instance. And you mentioned Mesos is another great example of sort of having meta infrastructure to help manage these services and allow you to really sort of deal with them in a in a reusable disposable kind of way that makes it all a little more sane. I totally agree there's a lot of work to do, but I also think it's a really exciting time for those of us who are watching to see what evolves and see what starts to fill that void.

Yeah. Absolutely.

So what made you choose Django versus another Python web framework, and would you choose it today?

Yeah. So this is this is kind of interesting because so, like, Django, the most mature framework in Python in terms of flexibility

and low I wanna say low commitment, but that's not entirely

true. And, honestly, when we chose Django,

it was before really most of these things existed. Like, today, like, I I have some colleagues that use, like, Pyramid and,

some of these other micro frameworks, I guess, you'd call them. Uh-huh. I have never really used them personally. Like, I'd say the reason we use Django is because we are experts in Django. We've been using it for so long

that if there's an issue, like, it's, like, second nature for us to resolve it. But it's also kind of interesting, like, we use Flask as well. And we've joked around, like, should we just swap out Django for Flask based on how little we do in Django anymore? I mean, also, Armin, 1 of my colleagues here is, like, the author of Flask. So it's, like, it's always something that comes up. But honestly, like, Django, it's, like, it kinda gets out of our way. It doesn't try to hide too many things. So if there is an issue, we can figure it out and we can understand it. Right? There's not a lot of magic behind the scenes. And it is, like, pretty loosely coupled, especially in modern versions of Django. That that's changed a lot over the years. For us, like, we don't use it too much anymore. So, like, we use the ORM still, though I don't know how long that's gonna last. And we use it for very simple routing, but, like, our our app is primarily, like, it's a React single page app. And then we have a kind of Django Python API, and then we have a Python kind of ingestion pipeline. And that's actually, like, interesting because the ingestion pipeline

arguably needs to be a lot faster for cheap scale in theory. It's actually worked fine for us so far, but you can imagine, like, we have this massive,

rate of events always coming into Sentry, and Python is not the greatest for this kind of stuff. So we'll see how long that lasts as well. But yeah. I mean, I think Django's it's still, like, a great a great way to get into Python, and I would argue no company is too big to build their app on top of Django.

Another benefit that Django has too is that it's got a very strong focus on keeping up to date with security patches, and their upgrade pathways

have historically been very well thought out so that when you do need to move to a newer version, it's not as big of a pain point as it could be in, you know, some of the other available frameworks potentially.

That's definitely true. We, we started supporting some PHP frameworks,

I'd say better than we used to recently. And I and I could be wrong about this, but I think 1 of them to upgrade from, like, 1 version to the next major version, it does a bunch of cogeneration, so you'd effectively have to, like, regenerate

or something. And I I'm not I don't even know if you can upgrade. But, yeah, Django has been good about that. Like, it's definitely not without its flaws in terms of upgrade processes. Like, we actually can't easily upgrade Sentry to a new version. But for normal users of Django, it's it's phenomenal and they do a great job with stability.

I I also agree with regards to this very little magic in Django, and I don't know it extremely well, but I've been learning it recently.

And I tried to learn Rails. This is no knock on Rails. This is a knock on me. For years, I I never could fully get my head around it because there were so many

black boxes that I just I had a very hard time

connecting the dots fully. And with Django, I really feel like I understand what's going on. And as I work through building my application, I feel like the

experience too.

Is, I think, a huge feature.

Yeah. That's actually been my experience too. You know, a number of years ago when I was first trying to learn Rails, I had the same experience of

not understanding

what was happening because there's too much spooky magic at a distance. And then I reimplemented the same application in Django, at which point everything started to click. And then from then, I was actually able to go back to Rails and understand what was happening because I had seen the guts of it in Django so that I could understand what the connections were under the covers.

Yeah. Like, my my favorite thing, especially, like, when younger developers ask me, like, how to do something. And even, like, when it comes to, like, documentation,

Django included is, like, just go look at what it's doing in the code. And it it's, like, for Python, almost all the time, it is so easy to read and understand the code that it's it's almost better than documentation.

But that's really hard to say the same about many other languages. Rails specifically is notorious for, like, hiding a lot of it and I know they follow some, like, patterns now. So if you know the pattern, it's a little bit easier. But I find myself every day whenever I'm looking at an API, I just straight up open the source code. And it just it's very smooth and simple.

Yeah. I can definitely agree with that. And particularly with the strong focus on making documentation

a first class citizen in the code itself really lends a lot of transparency to what's happening.

We've discussed how Sentry supports a number of different languages and platforms. I don't know if you can enumerate them a little bit and then just talk about what is involved in actually

getting Sentry integrated into an application?

Yeah. Absolutely. So what do we support? So I'm just gonna start I'm gonna list the important ones or from my perspective, the important ones.

So we have what I'm gonna call first class support, which is it works very well. Python, PHP, Ruby, Browser JavaScript, Node,

Java.

Alright. What other languages do people use? We have some c sharp stuff going on. We're about to, don't tell anybody, but we're about to launch some phenomenal iOS support as well as Unity and Android. And those will be native integrations. I don't know. We support a ton of frameworks and other stuff. Like, we actually, like you can embed our JavaScript stuff in a React native app, which is if you haven't looked at React native, it's super cool. React in general is super cool. We support, like, all these kind of, like, I don't wanna say odd, but, like, nontraditional use cases of languages. But definitely we're we've always been, at least historically, focused on, like, the web platforms

and specifically, like, the open source platforms. So dot net hasn't been our forte. Yeah. And then, our focus is always it should be 2 steps. It should be you declare dependency,

in Python that's, you know, added to setup dot py or added to your requirements dot text, and then you drop in effectively the API key. What that translates to in Django is you add an include path in your installed apps, and then you specify what we call a DSN, which is a data source name, which is just a really fancy way of combining

the host of your Sentry server, as well as your API key and some other information. So you'll set that, and then maybe you'll configure some other things. For example,

you might wanna tell Sentry the version of your application. That's a very simple, like, config call. You might wanna tell Sentry who the acting user is. Right? That often comes down to, like, maybe you have middleware or something where you can just say, here's the current user. There's a lot of things like that. But the the idea is that it's very low impact. And I don't know if, as a business, this is the best strategy,

but we've tried to make it so you could rip out Sentry without changing your code, as well as you can install Sentry without changing your code. It's been a very important thing for us. And most platforms work pretty much in an identical way. The 1 area, you know, I I talked a lot about how browser JavaScript is really hard. The 1 area that it kind of redeems itself is, like, you literally just include the script tag and you configure the, API key, and it's just magic after that. Like, we instrument everything. And we do a lot of really cool stuff from there. And we've tried to do that we tried to make it, like, I guess, as low friction as possible, but also not completely magic. So if you've ever set up New Relic, you kinda just run your app inside of it and it monkey patches a lot of things. We're starting to do a little bit more of that but we've avoided that as much as possible. And we're like, here, you just wrap this. You can understand what it is. You can look at the code to see what's going on. Like, kind of that that same, concept where, you know, we go simplicity over complexity. But, yeah, in general, it's, you know, dependency and buying an API key and you're done.

So just out of curiosity, what's the most obscure

either platform or platform variation? Like you mentioned sort of native JavaScript and something, you know, it could even be something like that that you folks support. What's something what's a client side language or platform support that made you just say, wow.

Yeah. Definitely some of the browser JavaScript has been there. But, like, we started doing stuff with tvOS. It's like, oh, yeah. We can do that. There's there were some people, like I don't even know what I can talk about, honestly. There's there's some crazy uses. I won't talk about any of those. I don't know if it still works, but, maybe a year or 2, but probably even more than that. Somebody wrote an R client for Century, which is super confusing to me because R is like a a, like, mathematical language, from my understanding, it's for doing, like, scientific computation and stuff. I'm like Statistics. Yeah. Yeah. Statistics. And I'm like, I don't understand why you need it, but that's kinda cool. But, yeah, definitely, like, the JavaScript stuff has been

the most recent innovative concepts. I remember back in the day. So a lot of us here are gamers. Like, historical note, Sentry is named off of a StarCraft unit.

Oh, nice. So, we've actually had a lot of gaming companies as customers, and I remember,

CCP is 1 of them. And they they make EVE Online and and they actually make some of the cool new VR demos that you might have seen. And,

we learned way back when, that their first console game, I think, which was a game called Dust, it was like an FPS.

I think it was running on the PS 3. That was actually running the Sentry SDK.

And I'm like, that's really cool. And we we always discover, like, new things like this. So it's like I don't know. Whenever we use something and we find out that that company behind it or that literal, like, actual thing is running Century, like, that's, like, really awesome. And I think that's the highlight for us. Yeah. And going back and speaking to your point about making it easy to

both onboard and offboard with Sentry, I think, is actually a big selling point because

while it potentially might not make business sense to make it easy for your customers to go somewhere else, it does help with them pulling the trigger on that decision making process because they don't have to worry about, okay, well, what if we put it in now and then we don't wanna use it down the road? Because they can just say, oh, well, if I don't wanna use it down the road, it's easy to take back out. So I think that would actually make it more likely for people to go with Sentry in the first place and then also stay with it in the long run because they don't have that worry or doubt over their heads as they progress further down their development cycle of wondering what's gonna happen if and or when they need to go to a different service or Sentry, for whatever reason, ceases to become viable.

Absolutely. And that's actually, like, kind of the same thought we have around the fact that we're open source. Like, honestly, most people it doesn't really matter that we're open source. Like, running it yourself is not cheaper than paying us. Like, there there are some companies with legitimate reasons to, run it themselves, but open source is just like there's no lock in with our SDKs. Open source is like, well, if I want to move off of the hosted service, I don't have to switch to another product. If I really like Sentry, I can just bring Sentry in house and I don't have to pay for it. It's it's free open source. It's truly open source. And I think that's really important to our customers even if they will never do that.

Yeah. Definitely agree there. I've actually been planning to set up an instance of the open source Sentry project and my own infrastructure at work, partly too because of the fact that we're gonna be going to an in house OpenStack deployment. So we actually won't have to pay any money to a cloud provider.

Yeah. It's always nice.

1 of the big discussions in open source these days is around maintainability, and a common approach is to have the hosted offering, which you guys support for paying the bills to keep the project moving forward. I'm just wondering what your experience has been with managing the open source community around the project in conjunction with providing a stable and reliable hosted service for it. Yeah. So this is interesting. So I've given a bunch of talks at, like, Python conferences and stuff over the years. And, 1 of my main points in the talk was that nobody contributes to Sentry. And this is kind of a lie, but it's also the secret truth. So if you look at Sentry, the server, like, the core of Sentry, it's it's Sentry, the comp like, it was primarily me for a long, long time. That's not to say we don't get contributions.

Like, we had, we've had, like, people from, like, Uber and Square, like, contribute awesome fixes for random things over the years. But we don't get people, like, building features and things

like that, like, true

substantial contributions.

However, we do get people building integrations and building SDKs and making it work better for their platform. And I would say part of like, there's a failure on our part, not bringing the community into contributing with Sentry, and we're gonna work to see if we can improve that. But I actually don't think it's that big of a deal. Like, we we promote the idea that you can contribute if you if you want to, but we're probably gonna be better at solving the problem. And, honestly, we've made a business out of solving the problem, so we're super happy too. And I think, yeah, like you mentioned, the the hosted thing has been great. The majority of our revenue comes from our our SaaS service, and I, like, I prefer that, honestly. It makes it much easier for us. But it it's interesting because, like, we actually, like, we build new features that only on premise use quite often, honestly. It's probably, like, every release, and we release once a month now. And I think it's, like, a very interesting take from I I couldn't tell you another company that operates in a even remotely similar fashion. Like, most most open source companies are, like, well, they build a proprietary product on top of somebody else's open source, or

they live long enough to see themselves go almost entirely closed source. Like, we call it open core, but it's more like

closed features.

So it's it's not without its challenges, but I actually think it's different than an open source project like Django realistically.

It's mostly like you can sort of solve your own problems if you want to because it's open source, but often you won't have

to. And you've alluded to this a little bit, but are there any benefits to using the hosted offering beyond the fact that you don't have to manage your own infrastructure for it?

That's the primary 1. And that's, like, actually a big deal because so there's, like, a few things around that. Like, a, we're probably better security than most companies. Like, straight up, like, we are

ridiculously paranoid about security and compliance. That's a minor thing, but I think it's significant. But the common thing is, like, we are absolutely going to be cheaper, and we're about to change pricing. I think it will get cheaper for almost everyone, with the new pricing model. There there is a big real cost thing there. Like, there's no time spent operating, no time setting up, no time dealing with, like, problems. Like, if you get a burst of data, it's really painful. We have enough servers sitting in front of Sentry

just to deal with the constant, like, denial of service that we see against us. And it's not it's not abusive. It's just that all of a sudden their application is throwing a lot of errors. Right? That's really hard to deal with for an internal service. And then the biggest, and I think this is the most important thing ignoring anything else, is like you don't have to keep your installation up to date. Like, we move so fast. We tag a new release every single month. And that includes, like, schema changes, that includes big new features,

that might include changes to SDKs that you need to upgrade those as well to support. And I think people underestimate how complex it is to keep an installation up to date, especially if you have, like, an SLA that it needs to be online. Like, we find often many of our customers are running 1 to, like, 4 year old versions of Sentry, which is just insane to think about.

Yeah. That's definitely reason enough to go with the hosted offering of not having to manage the upgrade cycles as well. So valid points.

So have you faced any performance challenges implementing Sentry server side? I know you mentioned the ingestion pipeline earlier. Are there any other aspects of its performance that you've had trouble with?

Yeah. So I wouldn't say performance. It's more about scale. I actually think performance matters less and less, in this day and age. So there's I'm gonna talk about 2. So the first 1, if you Google transaction ID wraparound, I think our blog post is probably number 1. So we use Postgres and we are I don't have numbers to compare with other companies, but we are probably 1 of the larger in terms of throughput to a single instance of Postgres

users of it and that just means, like, each instance of Postgres we have is doing a lot of effectively updates and inserts per second, at a very high level. And what this does in Postgres is there's this transactional counter that, increases

effectively, like, it's very complex to try to explain this and I don't even understand the true inner workings, so I won't, but I'm going to simplify it. So I'm going to say when you do an insert, a counter increases.

That counter is a 32 bit 32 bit number,

and a process has to run to basically reset that counter. And if that process does not run, the database cannot run. And again, I'm going to try to skim over the details, but effectively,

the reason that has to happen is because it's like garbage collection. So it soft deletes a lot of rows and you need to kind of you can't just wrap around and keep reusing the number or anything because you'll start overwriting data. Anyways, so what happens is a process runs

to sort of clean that up, to garbage collect that, and we've hit I think, over the course of I wanna say we've operated the hosted service for

4, maybe more years now. We've had 3

significant outages from that 1 problem, and it is really hard to deal with it. We've got it resolved now. We just aggressively vacuum and our databases have ridiculous hardware, but it's been a nightmare. Let's see, the other side Okay, so the ingestion pipeline is the majority of our concern. When we implemented source maps, so browser JavaScript in general is a nightmare for us. We have this map. We'll have a blog post about it pretty soon. But we have this map where we started showing like, we show dots whenever there's an error and we show it geolocated. So we also color the dots based on platforms. So, like, a yellow dot in it is a browser JavaScript error, and the entire map is yellow. And that just means, like, all the data is effectively JavaScript. Now the problem with JavaScript is to get source code, we have to go fetch from a website. If it's minified code, we have to go fetch a source map. So we have a system that allows you to upload that stuff to us,

but by default, you set up Sentry, you do nothing else, we will scrape your website for the data when it comes in. And we'll try to do it efficiently and cache it and stuff, but what that means is we're constantly having workers that are network blocked.

Like, we didn't really have issues before, and we're using normal threaded Python, but as soon as we started doing this it's like all the workers are blocked all the time and some of the times it would take a really long time to download these these could be like slow responding web servers or they could be overloaded because they're already erroring, and their source and source maps could be, like, 50 megs in total, or something, like, ridiculous like that. So we're constantly trying to download this. We're putting more stress on their web servers. Anyways, it caused us a lot of problems for a while. We've sort of alleviated in that we've hit, like, enough scale that it's not that big of a deal, and we have a lot of people pushing their source code to us instead, which helps a lot. But that same exact issue now all of a sudden exists for iOS. So iOS, you have to push your debug symbols to us. And there's there's many other problems that we have to deal with around symbols, but those debug symbols, even on the small end, are, like, greater than a 100 megs, and you have to push a new set of symbols to us every time your source code changes.

And some of those are, like, a gig in size. So now all of a sudden we have to, like, take these where our peak before was, like, 10

megs, now the peak is potentially multiple gigabytes that we have to, like, figure out how to deal with. And that means, like, we need massive on disk caches. We need to be able to handle doing a lot of work in memory. And so we end up with a lot of challenges of, like, how do we make sense out of the data you're sending us? I would say Python does not make a lot of those prompts easier, but we've also been able to work around it. So it hasn't been a limiting factor yet.

So with regards to the PostgreSQL

transaction wraparound issue, are you guys considering moving to something perhaps with more of an infinite scale by design, like 1 of the NoSQL databases

that can grow to vastly larger extents more easily?

Yeah. So we we talked a lot about this. Like, we considered moving to Elastic and I know some of our competitors are on an Elasticsearch infrastructure. And interesting, I I like, I've listened to some talks by a few of our competitors, and they're all smart guys, but they all talk about, like, scale. And I I, like, I always kind of, like, joke around with it. For us, scale is, like, we move from 1 Postgres machine to 2, and we just move tables to the second machine. Like, we have such high throughput numbers without doing any effort. Right? And then we started moving, like, can we move to Elasticsearch to make, like, kind of spreading out data a little bit easier? And then we're, like, well, the cost of Elasticsearch Elasticsearch is really high because it's Lucene backed and it indexes all this stuff that we really don't care about. We don't need relevant search. We always need date based search. And it does all this other stuff and then there's trade offs where it's very very scary what happens in certain situations with Elasticsearch. Scary enough that most people

have an actual backup of their data. And it's like, well, we don't wanna have a backup. Like, we don't wanna have a secondary, like, data store just to make sure our first data store works. So at this point, we've decided that I mean, we can move a lot more tables to a lot more Postgres clusters before we actually need to deal with it. But we're, at some point in time, almost certainly going to build a distributed

solution on top of Postgres. What that looks like and when we do that remains to be seen, but we're effectively gonna do what Facebook and Dropbox and Yahoo and Google old days did with MySQL, but we're gonna do with Postgres. And that is we are gonna build a graph abstraction on top of it. That said, we are still going to use something like React. We might move to Cassandra in the future for blob data and maybe more than blob data. So, like, ideally, the large disk usage will scale much easier, because, obviously, scaling up Postgres in any situation is is a non trivial operational thing to do. I am far from an expert in Elasticsearch, but I know just from sort of witnessing at a prior company that I worked at, the difficulties in running and maintaining

a mission critical Elasticsearch

cluster that it can be a real challenge to keep it running in a resilient way with no data loss over time. So I can definitely see why you guys are a little gun shy about that.

Yeah. And we've we've always been, like, so half of my team I worked with at Disqus way back when, and we've always been really, really cautious about adopting technology.

And I think it it has paid off for us over the years. Like, we're like, we use Redis. Like, React is the most, like, crazy thing in our infrastructure. Right? Like, we use Redis a lot. We use it in the most simple way. We use React. Like, we don't use any features of React. Effectively,

if we were to somehow lose our Postgres database, I could never tell you what's in React. Like, all of its index through SQL. And that's the conscious decision we've made over the years. Like, keep it using technology that's time tested, proven, and continually being, I guess, improved on. So, like, by the time we try to build this distributed Postgres stuff, maybe they've, you know, kind of made it easy for us. Yeah. There are actually a couple of solutions out there that have already started to provide something like that. Like, I know there's Postgres

Excel, and, there's another company that just open sourced their layer on top of Postgres to make it easier to scale horizontally.

The name is escaping me at the moment. Yeah. It's, Citus DB or Citus. Yes. That's the 1. Yeah. It's interesting. We'll see how those projects come out, but there's people that are much smarter than us trying to solve some of the problems. And hopefully, they can. But at the end of the day, it's kind of straightforward. Like, you just put data over here. You can't query across it, and it's, like, if you simplify the model, the implementation

is understandable. It's reasonable. Operationally, it becomes easier to deal with. So we'll see. But, yeah, hopefully, there's a lot more invested in that because MySQL for a long time has gotten the, the majority of attention there.

What kinds of advice can you give to people who are trying to get the most utility out of their usage of Sentry?

So I would say

don't stop with the setup instructions I gave you like it's Yes, you can install it as simply as just configuring the API key. But there's so much more that it can do. So let's just start with the core data. So I talked about, like, you can you can mark users.

You don't have to send us email addresses. You can anonymize it. You can have your own internal ID sent. Like, you can send email addresses and that's compliance. Fine. Like, it's great. You can send custom tags, and this is kind of a really powerful feature we have. So we automatically tag every error that comes in with things like the browser name, the the operating system,

the version of your application, the environment, like production or staging, and then you can search on those tags so you can find all issues that are in staging. And you can also see a distribution of

how many errors are happening in staging versus production for a specific issue. But you could also do things like, for example, in Sentry, we have these concepts of projects, and we actually tag issues with the project that is affected. And that's really useful when we need to go look at, like, data or something like that, or we need to go, like, see is it, like, just 1 project. And there's a lot of cool things you can do there. And then beyond that, like, we have this really cool new feature that hopefully you never have to do anything yourself called breadcrumbs.

And we've launched it with JavaScript.

I think I don't know if we released the Python version of it that has some stuff. We're about to launch some Rails support. We have some PHP support. But what it does is it captures the trail of events that led up to an error. So we have this really powerful idea coming soon that says, when

something happens in my process

that amounts to billing, so that is when I make a Stripe API call, and then any kind of error happens later in that sequence of events, I wanna know about it and I wanna escalate it. Like, that's a really powerful idea that is a little bit more complex than, a lot of solutions

debug information that you never had before, but also be able to use that information in a really powerful way. But hopefully, you don't have to instrument any of that. Hopefully, we do it for you. Time will tell there. And then 1 really cool last thing, and we just launched this too, is we have this thing called user feedback.

And, you know, going back to, like, the this Firefox crash dialogue, I could ask you if you want to send the report. We built this, like, JavaScript widget that you can put on your site when there's an error, and it'll be, like, hey, we're really sorry about this happening. Like, you can fill in your name and email address. We'll autofill it if we already know. And then you can put in, like, some feedback, and they're, like, hey, what were you doing when you triggered this? And honestly, like, we don't really need the customer to tell us what happened. We have enough information already, but what it actually is is it's just, like, intimate connection now with the customer. Like, the customer feels like you're actually listening to them. And then very soon, the next step is gonna be, like, do you want to, like, respond to that customer? Do you wanna let them know you fixed this? Because we actually go out, like, if you've ever hit, like, an error on Sentry that's significant, especially if it's, like, a billionaire, we will actually go out, like, we will very rapidly get a fix out, and then we will email you and let you know that it's fixed. And, like, that idea is so unbelievably powerful.

And, like, engineers are shocked when they see this, and they're using Sentry already. And you can imagine how it is for nontechnical

consumers if they're, like, wow, you fixed this. Hey, not I didn't even know you knew it was broken. Right? And so, like, that that to me is a really cool idea that we're still in the early stages of.

Yeah. That definitely

provides a much more powerful way for engineers and companies to get a much more high touch and sort of concierge

feel to their products and really encourage customer loyalty and really make people wanna continue using the service because, you know, it's very easy for a customer to run into an issue

or an error on the server and say, oh, this, you know, this product is garbage. I never wanna use it again. But if you then all of a sudden give them a way to provide immediate feedback of what they're feeling, what they were doing at that moment, then it right then and there brings them back into thinking, oh, maybe this isn't so bad. And then if you then actually do have the capability of following up and saying, hey, that error that you ran into, you know, yesterday, last week, whatever the case might be, we just fixed it. If you're, you know, if if you still need to do what you were trying to do, then go ahead and try again because that error won't trouble you anymore. And that's a great way to not only get customer loyalty from them, but also make them more likely to promote your service to anybody else that they know because they'll say, oh, I had this great experience.

You know, it started off as an error, but then they immediately turned around and made it a wonderful interaction.

Exactly. And, like, I think that and we we think about this a lot internally, not just for how we build Sentry, but how we, like, grow our audience. Right? And I think the greatest thing you can do is hasten the cycle of taking a user from a customer to an advocate of your product. And these ideas, a lot of them are designed to, like, promote that. As long as you're willing to do a little bit more than normal and ideally, it's low touch and we built the tools so it's easy. Like, it can greatly impact the, kinda, like, the success around your customers.

Yeah. And then also going back to your point about breadcrumbs,

having the context of an error is immensely useful when you're actually trying to find the source of the problem and debug it effectively.

Can you describe some of the other facilities that Sentry and Raven have for assisting developers and providing that contextual information?

Yeah. So we're of the mindset, capture everything, ask for forgiveness.

So we capture source code. We we capture,

user data. We capture request data. We capture all this stuff. Most of the time you actually are, like, happy with it. 1 cool thing we do in Python, and unfortunately, we don't do this everywhere, is we actually get the,

the stack locals. That is, like, we get the variables, the values of them in every frame in your stack trace. And it's unbelievable how useful that is. And that's what we started out with. We actually we 100%

stole Django's, like, debug page. Like, they have this page that shows up when you have a 500 error, has a stack trace. You can expand the stack trace. There's, like, sort of source code around that frame, and then there's the the local variables there. And we're like, wow. That's amazing. How do we get that in production? And so we built Sentry. Unfortunately, this is really hard to do in some languages. So we do it in PHP, but it's not the same thing. We do calling our reflections, so we can tell you what the arguments

were that were passed into the function, which is usually really good already. Actually, that might be the only other language we have in a public implementation of. We are exploring. Ruby 2.0 has a new API that I think with a c extension, we can make fast enough to do the same thing as Python, which if we do this, it will be phenomenal. And in theory, and we've yet to prove it will work, we can do it in all c based languages.

Though, we might have to be a bigger company before we can pull that off. So but that is, like, the number 1 thing we do. And it's just like all these little bits and pieces of information combined.

The end goal is that you never have to say, hey. Like, customer,

please help me understand what you did. Like, it that's a it's a terrible interaction when you're, like, going to ask the customer for help.

We've touched on this a little bit, but it can be pretty challenging to create an effective method or algorithm for aggregating incoming issues that they're sufficiently visible and useful while at the same time not hiding or discarding any important information at the face level view. Can you explain how you do that and what the evolution of that system has been like?

Yeah. So, like, aggregation hasn't been too bad. Like, it's interesting because a lot of people don't understand how Sentry works, and I actually think we work differently than anybody else, which probably also explains why we don't have scale concerns. So you said, areas of the century, we understand the structure of the data, we understand the language, and we do, we hash those issues in a number of ways to ensure they group ideally in the most accurate way possible without you telling us anything. That's actually really challenging in some languages, but for the most part, let's just assume it works. So once we've grouped it, there's a lot of there's definitely a lot of, like, user experience challenges here. So you, like, click into an issue, and it's like an inbox. And then you're viewing a sample as we call it. So you're viewing an aggregate issue with a set of data that's from 1 error. It may not be the latest error. It may not be the error you really care about, but it's it's an example

of what that error is. And this is where people

immediately get confused. They're like, well, why is it not, you know am I just using viewing, like, individual errors or, like, aggregates? And I think we've put a lot of effort into user experience and design here, so we've actually made it so most people can reason about it, but it's still definitely a challenge. And then when you look at that actual error report itself, even the sample and everything, even if you've got that down, there's so much information there. So 1 thing we do, for example, is we try to know, and in in most languages

what is your application code versus what is, like, something else. Right? And we we call that, like, system code, but it it doesn't really matter what it is. Right? Let's say you're using Django. System code is the Django framework, as well as it might be, like, the requests

library,

it might be Celery, kind of any other third party. Right? So by default, we only show you application code. And we it's a little bit more complex than that, but mostly we pared down the stack trace to be only what's relevant. That same issue kind of happens elsewhere. Like, we have a lot of data to show, like, the HTTP requests and stuff. I wanna say we've solved all of it. I would say, like, you can usually skim a report and the visuals make sense. 1 area that we're going to be improving is we we wanna kind of tell you a context of what something is. So for example, if I'm rendering a view in Django or a view in Flask or a,

I don't know what it's called in Rails. It's either a controller or a view, but 1 of those things. Mhmm. I wanna know that it's that view. Right? I wanna know it's kinda like this URL, but not this URL. I want it to know it's like this code that powers this URL. And so we're gonna do something really soon where we just like, we give transaction names. And we sort of have this today, but it's it's not great. But the plan is to, like, help you more clearly understand what something is at a glance. Because 1 of the big challenges is, like, you have the list of all the errors, and if, like, 3 rows are, like, the same kind of, like, they just look the same with the same title and stuff, it's, like, that's already a poor experience because they're not the same. They're not the same because we do such a good job at aggregating things that we know they're not the same. But, like, what can we surface to help you understand that they're different? That's definitely an ongoing challenge, but, I mean, we're we're big believers in iteration, so we'll figure it out at some point.

So I notice in Sentry's Python a lot of from future import. Does it support Python 3 and or what's the plan for getting there?

Guido is not gonna like me for this, but I don't believe people use Python 3, and I don't believe that we will ever use Python 3. So, no, we do not. We run on whatever the newest 2 7 is. And there's there originally was a reason for that, and that's because Python 3 didn't really work with anything. And I know the ecosystem's gotten better since then. Like, third party libraries often support it now, but, like, frankly, we have no reason to upgrade. It's still, it's still massively painful.

It's getting easier, and I think that's good. Like, I think 3.3 or 3.4 started to there was more, of kind of the the passive code that will help you upgrade without changing all of your code, so that's good. Maybe there will be a time, maybe Python 4 will come out and we don't have to change any code and we can just upgrade, but there are trade offs for upgrading. So, like, there's some cool new features, like, I don't even think we support this in Python, we do other languages, but Python 3 has, chained exceptions,

so you can reraise an exception and keep the context. And that's actually, like, really, really nice because it's really common to rethrow an exception in Python. But because we're on 2, we don't have that at all. And there's some other like the I actually think I've done some Python 3, and I actually didn't didn't find the Unicode changes, which everybody loves to be that much easier. It's still hard to reason about Unicode. But, mostly, 1 of the big concerns we have is that Python is not the fastest, and Python 3 is definitely not faster. So what's the value in upgrading for us besides some fancy new features? So we'll see. I don't know. Maybe there will be a day that it comes along and it's really easy to upgrade and there's really strong reasons to upgrade. But so far, they just there hasn't been a compelling thing there. I mean, the good thing is Python 2 is gonna be around for a long time yet. It just it's still painful to upgrade. So it was probably 2 years ago I tried converting a project to Python 3 and it actually mostly worked and I was I was actually very surprised at the time. There was something important that did not work. I don't remember which 1 it was. Anyways, yeah, there was definitely

some trouble a couple years ago but it was like 1 thing And I was actually pleasantly surprised that it had gotten that far. And I like, we've actually supported, Python 3 in our SDK, I don't even know, for, like, 4

plus years now. So I can definitely believe that the community is almost there. But I I still think it's a lot for existing projects to ever upgrade. And it's, like, the same reason why, you know, people still run old versions of Rails or Django or Ruby or anything really, or why people are on the old JVMs.

I I'm thoroughly happy with the direction Python 3 is going. Like, I remember, I think even, like, 32. I I would have strongly said it's too hard to convert things. But, like, they've made a lot of changes that make the the Unicode

transition easier.

They made a lot of changes that make a lot of the syntax transition easier. I mean, I think that goes a long way,

especially for library support, which in my opinion is the what or at least was 1 of the biggest blockers. And I agree. Yeah. Like, from what I've seen, that's mostly gone away. Yeah. I don't know if there will be a day where it's easy enough for a big project upgrade. Like, I think it would be exciting if there was. I think if like, PyPI is cool and everything. I don't think it's necessarily a solution to problems. But if there were, like, other strong arguments for people to, like, spend engineering efforts, so if there were big performance gains or the type annotations actually, that could be a trigger. Kind of some of these more compelling things that are not just, like, small features, I think that's when it would become something like, okay. Maybe we'll reevaluate. Maybe it would be worth what would likely be a considerable engineering effort to move to a new platform. Because, like, for stuff like Century, it's like we support 1 version of Python, period. Like, we don't run on multiple anymore. Like, we did once upon a time, but 27 is the standard now, so we didn't have to. But I I'd imagine if we ever did go the route, it'd be like, well, we're no longer supporting Python 2. It's only Python 3.

Yeah. For a project like Century, that would make good sense. It's really only library

code that needs to be able to support both Python 23 simultaneously

in in about 4 years. It won't be required anymore, though. I'm sure a number of people will still continue to support Python 2 in some measure while the, inevitable migration continues.

Yeah. And we'll see. I like, I am actually super excited. Dropbox has been doing some really good work on this. They have, 3 or 4 different people internally that are working on effectively Python VMs. And I think that's led to a lot of really good ideas. Like, they employ, like, the the author of Mypy. Obviously, they have Pieston or Piston. I'm not sure how they pronounce it. Yes. And then Guido, of course. And I think the collective of that, like, momentum there, whether their VM becomes useful, but it it's at least contributing things that are getting back into Python 3 with type annotations being 1 of the big ones. And that 1 that one's super important to me. Like, I not publicly, but internally at Dropbox. Like, I voiced a lot of opinions on that over the the couple years I was there, and I'm so happy to see that, like, go through.

Yeah. Another

interesting project that I hope to see

incorporated into the mainline code fairly soon is the, Pigeon project out of Microsoft. I think that's gonna be a pretty big game changer if and when that actually hits mainline and people can really start plugging in implementations

into CPython

without having to go to a completely different runtime? Yeah. That'll be really cool.

So looking back to the beginning of the project, what are some of the most interesting and surprising changes that have happened during the lifetime of Sentry? And how does it differ from its original vision?

Yeah. So well, the original vision was, the project started because, there used to be this IRC channel. I think it was called Django users or something like that. It might have been just Django. And I, like, kind of idled in there and, like, helped people out and answered questions and,

somebody asked how to do this 1 day. And so, a guy uploaded a little snippet that showed him how to do it. Right? And that's how it was born. No ambition. No nothing. You You know, a few years later, about 3 years later, I was leaving discuss.

Still no ambition, but we had gone cross platform. And somebody, from Heroku was like, hey. Why don't you create, like, a Heroku add on out of this? And he's like, you can probably make some beer money or something. Right? I'm like, okay. Sure. Whatever. Greatly underestimated what that required. But I did it over, like, a Christmas vacation or something. And then, still no ambition. I'm like, okay. We'll see what happens.

And

ambition. I'm

like, okay. We'll see what happens. And then, let's see. Maybe it was 2 years later. A little

little over 2 years later, which would have been January of last year, myself and my cofounder, Chris, who's a designer,

were still the only people, like, actually part of the the company, if you could call it that at the time. He was working at GitHub right before that, and I was working at Dropbox. And we said, you know what? There are too many people paying us money and too many people using this. Like, it it's hard to maintain, and we don't wanna kinda, like, let it, you know, fall over. So we made a decision to quit our jobs, and then the company is now 14 today. So for me, that is, like, the biggest change. And the cool thing about it is it's, like, the code that runs getcentury.com,

barring, like, the billing code, which is irrelevant to people, is the same code that's open source. So I think we've kept, like, our our humble beginnings in a lot of ways, and we're very strong about that. But, like, we have matured so much as a a group of people building a, a product. We're 13 or 14 now. We have we employ people that only write code in other languages, which is super cool. And we're no longer this little Django project. We're used by, you know, tens of thousands of, like, real businesses.

And to me, it's, like, exciting to see this evolution of of nothing. Like, it was it was, less than 50 lines of code kind of into this big massive project that so many people value and so many people use it every single day. And it it's been super exciting to see that transition. And then, like and I mean, we're a we're a real business now. And so the future is it's gonna change even more. But we're very focused on how do we really,

keep and appreciate, like, where we came from? How do we, like, appreciate the open sourceness of things? And, like, how do we push that so we don't become 1 of those companies that's just like, well, you know what? We need to cripple the the free version so you'll pay us money. Right? It's been a good, it's been a good trip, I guess, getting here.

Well, I would like to, commend you on your effort and your progress because it's definitely a very impressive project and 1 that I have used in the past and intend to use in the future. So thank you very much for that. Are there any other questions that we didn't ask that you think we should have or anything else that you wanted to bring up before we move on? No. I think it it was good. Like, I appreciate the the questions and, the conversation was really good. I always love talking about, I mean, my 2 favorite things beyond Sentry just itself would be, like, scale and infrastructure and then kind of like open source. And I think we we got into a lot of that, so it was fun.

Yeah. I agree. So for anybody who wants to follow you and keep in touch with you and see what you're up to, what would be the best way for them to do that?

Probably Twitter.

Zeeeg,

zieg on Twitter. And then yeah. I mean, we have we have a bunch of century support things going on, but that's definitely the best, best solution for me as a as an individual.

So with that, we will take it on into the PIX.

I'm gonna keep it short this week and just pick the b Python interpreter. I've started playing around with that and using it as my Python shell for a little while, and it's been quite enjoyable.

I had used it a couple of times in the past, but it's definitely a pretty great environment.

Being able to sort of go back and undo

commands is an interesting feature.

And the auto completion and

auto

servicing of documentation

is pretty awesome. So IPython has the capability of adding the question at the end of a function call to show the documentation, but having it just automatically

surfaced as you're typing is pretty great. So definitely recommend people check that out for, and if they're looking to try out some new Python shells. And with that, I will pass it on to you, Chris.

So this week, I'm going to pick a couple of podcasts.

The first podcast

is a podcast called Developer on Fire. And every week, the host interviews a developer,

which sounds like it might be like, you know, any number of other podcasts. But what's interesting about this 1 is that it it's completely cross discipline. It really sort of like he talks to Ruby folks, he talks to Python folks, he talks to Node folks, and he he gets into

what makes them interesting as developers. What gets them excited. And so each episode is more about the people in the sense that you get a sense of what gets them excited, and that might not necessarily be the language that they're known for or work in. So I really love it. Great stuff. The next podcast

is called

Song Exploder, and that is by PRX.

It is basically

every episode,

they go through a song made by a particular musician, like a recent episode was Weezer. They've had the guy who made the Game of Thrones theme. And they do, like, an in-depth making of for that song. If you really love music like I do, it's pretty incredible. And with that, we'll turn it over to you, Dave. What do you have for picks? Oh, this is, it's tough. That podcast, a developer on fire, sounds pretty cool. I've also used b Python in the past and definitely a a nice project. I don't know. Like, I I'm a pretty terrible programmer these days. So, like, what what is interesting to me? Like, I love

the React and Webpack. It's phenomenal. Like, I I'm not a big JavaScript fan, and I love writing JavaScript with these platforms. So I'd, like, definitely say if you have not yet had the opportunity to try those out, do it. You will not regret it. Outside of that, I don't really know. I've gotten into, like, alpine climbing and stuff recently. So I've become more disconnected once I, started running a company. But, yeah, there is,

I know this, 1 of my colleagues is actually building something that's cool and I really like what he's doing. I'm trying to get him to do it for Python.

It's called Percy. It's percy.io

and the idea is, like, I'm a big believer in testing. Like, if you ever see me speak at a conference, I'm all about, like, yes. Let's deploy code fast. Let's automate tests. And what Percy does is they, like, in a really clever way, they take screenshots,

sort of screenshots, but they take snapshots of your code, like, with a Selenium test suite, and they basically,

like, upload all that at the end and you can kinda, like, replay what happened because, like, they're basically just capturing the DOM. And I find it super fascinating. I would love if there was, like, an open source thing that did that. But it's something that we started looking at using just, you know, again, testing philosophy. That's I don't know. Books? Red Rising trilogy? Phenomenal. Just finished that. Just sci fi, Hunger Games ish kind of thing, but pretty good if you're into that. But I'm a pretty, low key person. All I do is work these days. So that's about all I have. That's great. No. So there's some interesting things to use. And where I'm working right now, we're actually using React and Webpack for most of our front end work. So I can concur with those choices for modern JavaScript development.

Yeah. They're they're phenomenal. They make things so much easier. Just like that they solve source maps for you, and you don't have to think about it. Alright. Well, we really appreciate you taking the time to talk to us and tell us more about Sentry

and its history and evolution and all of its myriad features. So definitely an interesting project and 1 worth keeping an eye on. So I appreciate you, telling us more about it. Absolutely. Thanks, guys.

The Python Podcast.init

Summary

Brief Introduction

Interview with Firstname Lastname

Keep In Touch

Picks

The Python Podcast.__init__