Summary
Healthy code makes for happy coders, and there are many ways to measure the health of a project. This week Andrew Mason talks about the Undebt project from Yelp!, as well as some of the other tools and practices that have been developed to make sure that the balance on their technical debt card stays low. Give it a listen to learn how and why to measure and address the painful parts of your software.
Preface
- Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
- I would like to thank everyone who supports us on Patreon. Your contributions help to make the show sustainable.
- When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at www.podastinit.com/linode?utm_source=rss&utm_medium=rss and get a $20 credit to try out their fast and reliable Linux virtual servers for running your awesome app.
- Visit the site to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
- To help other people find the show please leave a review on iTunes, or Google Play Music, tell your friends and co-workers, and share it on social media.
- Your host as usual is Tobias Macey and today I’m interviewing Andrew Mason about technical debt and refactoring with Undebt.
Interview
- Introductions
- How did you get introduced to Python?
- How do you define technical debt and why is it an important aspect of a project to keep track of?
- How would you characterize refactoring in general and when you might want to do it?
- What is Undebt and what was the problem that you were facing at Yelp when it was created?
- For someone who wants to get started with using Undebt what does that process look like and how does it work under the covers?
- What are some of the other tools and techniques available for refactoring Python code and how do they differ from what is possible in Undebt?
- What are some of the other tools and methods that you use to maintain the overall health of your codebase?
- What are some of the limitations and edge cases that you have experiemced working with Undebt?
- It is often a difficult balancing act when working in a team to determine how much time to spend paying down technical debt and building tools that will act as force multipliers vs doing feature work that will be visible to end-users. In your experience, what are some ways to manage that tension?
Keep In Touch
- Andrew
- GitHub
- Website
- @andrew_mason1 on Twitter
Picks
- Tobias
- Continuous Delivery by Jez Humble and David Farley
- Andrew
- XI Editor
- The Circle by David Eggers
Links
- Martin Fowler
- “Uncle” Bob Martin
- git-code-debt
- Undebt
- PyParsing
- Podcast.init Episode About Parsing
- Rope
- Pre-Commit
- PyLint
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
[00:00:14]
Unknown:
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. I would like to thank everyone who supports us on Patreon. Your contributions help to make the show sustainable. When you're ready to launch your next project, you'll need somewhere to deploy it, so you should check out linode at ww w.podcastinit.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your app or experimenting with something that you hear about on the show. You can visit the site at www.podcastinit.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. To help other people find the show, please leave a review on Itunes or Google Play Music, tell your friends and coworkers, and share it on social media.
Your host as usual is Tobias Macy. And today, I'm interviewing Andrew Mason about technical debt and refractoring with the tool Undebt from Yelp. So, Andrew, could you please introduce yourself?
[00:01:01] Unknown:
My name's Andrew. I'm a software engineer at Yelp. I've been working there for
[00:01:06] Unknown:
about 8 or 9 months, I think. And do you remember how you first got introduced to Python?
[00:01:11] Unknown:
Yeah. I actually dabbled a little bit at the end of my senior year of high school when I was taking AP Computer Science. But then not my I would say my first real introduction was sophomore year of university, when I took computer networks. We did a lot of assignments in Python, did, like, traceroute implementations and stuff of that nature.
[00:01:33] Unknown:
And so the main focus of the conversation today is around the tool, Undet, that you've done a fair bit of work on. And 1 of the primary purposes for that is to address technical debt in a code base. So I'm wondering if you can start off by sharing your definition of technical debt and why it's an important aspect of a project to be able to keep track of.
[00:01:53] Unknown:
Sure. So I would say that technical debt is any technical decision that you made in the past that is getting in the way of developing new things now. And I think that's different than what a lot of other common definitions are because you often see, like, oh, bad code or, like, poorly written code, code that's hard to understand. But that always implies some sort of malice or, like, incompetence, and I don't think that that is fair because you can make a perfectly correct decision given everything that you know now, and you can learn new things later, or, like, the world can change out from under you, and that would invalidate a previous decision and make a previously correct choice incorrect.
[00:02:52] Unknown:
Yeah. It's 1 of the unique aspects of software development is that there is no 1 set of requirements because even if you write the initial project to exactly match what the end user or the customer wanted, then they will inevitably come back with a whole new set of requirements that they want because, as you say, the world changes, and what may work at 1 point will suddenly stop working for whatever reason that may be. And so a software project is always a living, breathing thing that is always going to continue to change. And the only time that it is not going to be modified is when it stops being used.
So in some ways, the manifestation of technical debt is a positive signal because it means that the project is still viable and that there are still people who are gaining value from it. But as as you said, it is something that can inhibit, forward progress on the code base because of the way that it was originally written or architected. And so that leads us into the idea of refactoring. So how would you characterize the idea of refactoring in general? And what are some of the signals of when you might want to do some refactoring?
[00:04:02] Unknown:
So refactoring is a little more specific. So when I was talking about technical, that that doesn't necessarily have to be code. I know that that's what we're going to talk about. But, you know, technical, that can can cover, like, infrastructure choices, software architecture choices. It doesn't have to be code specific. But refactoring, I would say, is basically any change to the structure of code to make it quote unquote better for some definition of better that doesn't actually change the behavior of the code. So that's 1 of the key things is not not to break anything. I I forget who said this, but it was somebody like Martin Fowler, Uncle Bob, or 1 1 of those types of guys who said at 1 point, make make the change easy, and then make the easy change.
And that's where I that's that's my mental model of what refactoring is good for. When do you want to refactor? Part of me sort of is tempted to say always, but I understand that that's not really that feasible because you're not changing behavior. So if you just spend all your time refactoring your your users or your customers, they're gonna notice that there's no new features, and they're gonna move on to some other set of software that is giving them new stuff. But at the same time, doing a tech debt pay down like refactoring, there's a there's a big trade off between delivering features and making it easier to deliver features in the future, and both of those things are valuable both to you as the developer and as the customer, but there has to be some sort of balance met between the 2. And taking 1 step back before we dig into undebt,
[00:05:56] Unknown:
in order to be able to understand where you need to target a refactoring, also useful to understand what are the areas that are accruing technical debt. So I guess 1 is how do you identify where the technical debt is? And I also know that at Yelp, you've also got a tool for being able to keep track of technical debt as it accrues. So I'm wondering if you can talk about that a bit as well as how to identify where the technical debt is, present in a project.
[00:06:25] Unknown:
Sure. So technical debt identifying technical debt is sort of subjective. It's basically you have to look and say where where are our pain points when we try to make changes to this code base. Like, what what aspects of the code base are making it harder for us to change, or what areas do we not even wanna touch because they're so difficult to change. Or you could also have, like, for example, we did a lot of refactorings around database queries the way we wrote database queries via an ORM in Python, basically fundamentally changed over the lifetime of the project.
And so we wanted to go through and fix and sometimes there's performance implications. Like, we used to run a query this way, but, you know, for we use SQLAlchemy, and so this this new SQLAlchemy feature makes the query run way faster. So we wanna go through the code and update all of that. So the first thing, as I realize I'm not actually answering your question, is to identify all those places where the old pattern occurs, and we do that with a tool called Git CodeDebt. And basically, the way that that works is you point it at a repository and you define all these metrics just in pure Python.
And it's all configured via YAML, and you tell you tell gitcodet which Python metrics you want to run. And gitcodet will basically walk through the git history and hand off the text of the files to these various metrics, and then the metrics are responsible for stating whether or not that pattern appears in the text that they're looking at. And then you get these really nice graphs that can show trends in in code over time.
[00:08:22] Unknown:
A couple of other ways that technical debt can manifest as well is the if there are inconsistencies in the code style within a project because while different people may prefer different code styles, if there are multiples within a single code base, it can be somewhat jarring and it it can add to the overall cognitive load of trying to understand what's happening as data flows through a program. So, that's 1 of the potential targets for refactoring as well as, for instance, if there are things that are preventing you from being able to upgrade various dependencies in the program. You know, 1 of the most notable that Python developers would be familiar with is the idea of switching from 2 to 3. So identifying all of the print statements versus print functions, for instance, or, byte strings versus Unicode strings.
And so with that, I will ask you to dig a bit into what the Undet project is and some of the problems that were being faced in the code base at Yelp that led to its development?
[00:09:24] Unknown:
Yeah. So Undet, if I quote or read me here, it's basically a tool that allows you to perform massive automated code refactorings. And it was developed internally at Yelp because we were trying to do a bunch of migrations, like you were saying, like, 2 to 3 is a good is a really good example of that. And there were like, Yelp has a fairly large code base, and so there are obviously a lot of places we needed to touch, and nobody wanted to go through. And it's just a very boring mechanical task of going through and changing, you know, print from a statement to a function in every single file.
And so nobody really wants to do that. And so we created this tool with the idea that if other if other teams at Yelp had an easier way to do refactorings, then they would prioritize paying down that technical debt more readily. Because if you make something easier to do, people don't try to fight it as much, if that makes sense.
[00:10:36] Unknown:
Yeah. It's definitely, particularly when you're looking at the list of tasks that you need to do for a given sprint or whatever your development cycle happens to be, and you're trying to say, okay, do I wanna pick off this hairy refactoring, or do I wanna spend 5 minutes fixing up this other little piece of code or adding this little feature? It's generally a lot easier to pick up the smaller project if you're just finishing a larger task. So if a large refactoring is as simple as making a few line code change, then, yeah, it can definitely contribute to the overall velocity of those refactorings, I would imagine.
[00:11:13] Unknown:
Yep. Definitely. Especially when you consider from, like, a microservices point of view, if you have, like, 20 different codebases, right, you could or some number. I I don't know. Pick a number of code bases. And you need to apply this refactoring to all these code bases. With Undet, you can basically write the pattern once and then share that pattern with the owners of all those different services and repositories, and they can all do the updates simultaneously. And we actually do do that as we have we we provide a bunch of examples, and, similarly, we have patterns that we use and share with other developers.
[00:11:59] Unknown:
And so you're referencing the patterns for defining the refactoring targets. I'm wondering if you can explain a bit in more detail how undead actually does the refactorings and, how those patterns are defined.
[00:12:12] Unknown:
Yeah. So your patterns, you define in Python. It's built on top of a library called Py parsing, which provides this really nice syntax and uses a lot of the, the nice the nice language features of Python, taking advantage of done their methods to make it all look pretty, and it looks very, very similar to how grammars are written in language theory. So if you have any background in context free grammars, you're going to immediately recognize a lot of the syntax. And so all you really have to do is define your grammar, which is going to be used to see if a given bit of code matches.
And then you define another you define a function called replace, which takes in all the tokens that get matched by the grammar and then returns the string that you want to replace those tokens with. And that's pretty much all there is to it.
[00:13:15] Unknown:
And so there are some other existing tools for being able to do refactoring in Python, most notably ROPE. So I'm wondering what were the limitations of those existing tools that led to the necessity of creating UNDET? And, in what ways do you still use some of those other tools?
[00:13:35] Unknown:
Yeah. So when when I saw this question, I I did some googling because I don't I I've never really explored this space too much before. So other, rope was the 1 that I came across with it, but I wasn't really sure how good it is. And I'm not trying to, like, talk smack about a project or anything, but I just had never seen it before, and when I looked at the docs, it was just sort of like, here are all these refactorings you can perform. And I was having a hard time seeing how you actually, like, use rope to do it. So that could just be me not reading correctly. I'm not sure. But 1 of the 1 of the big advantages of Undet is Undet is super general, in a way. All it really does is it parses and then does transformations on text, and it's just coincidental that a lot of the text that you see in the documentation is Python. Like, in theory and actually in practice, we have some examples of this. You can use Undet to refactor anything that can be defined with a grammar. So that includes, you know, Python, Ruby, basically any programming language.
And so that's really useful, is we can we can also apply on that to, our JavaScript code as well.
[00:15:02] Unknown:
And so beyond just doing things like replacing a print statement into a print function, is it also used occasionally for, changing method signatures, for instance?
[00:15:13] Unknown:
Mhmm. Yep. So you would you could you could do something like that. You would define 1 grammar to match the function definition and change it. Actually, I think it would be easier to just go in manually and change the actual function definition and then write a grammar to match every call to that function and update its its signature accordingly.
[00:15:41] Unknown:
And once you do have that grammar defined, could you then use it to, for instance, update any documentation that references that method signature as well?
[00:15:50] Unknown:
I would lean towards saying yes. It's a little hard because we're not actually working with anything concrete here, but yeah, you would you would have to define a second pattern that took the original the 1 to match the function call, and made sure it applied to comments. And you can actually so there's there's 2 there's 2 there's actually 2 ways to define stuff in in Undet. 1 is with the grammar and replace pairs, where Undet will basically take a module and look to import a thing called grammar and a callable thing called replace. And otherwise, it can also take a it will also try to import something called patterns, which would be a set of grammar and replace functions.
So you can basically have 1 module look for define multiple things to to do find and replace on if you need to make sure that certain changes go together. So you could have a set of patterns, 1 of which does the actual calls and the other 1 to do the documentation.
[00:17:00] Unknown:
And 1 of the things that I thought was interesting while I was reading through the documentation for Undet was the fact that as part of the refactoring, once you've defined the grammar, it will actually tokenize the matched statements so that you can actually reuse those matched tokens in the definition of the replacement string.
[00:17:21] Unknown:
Yep. That's super cool and and really, really useful. Another another cool thing about Undet is that it plays really nicely with other Unix utilities. Like, it can read files, file names from standard in, which means you can pair pair things with GRAP or GitGrep or anything like that. And so therefore, you can, like, apply a set of changes to just a specific part of your code base, or if for some reason you need to refactor really, really fast, you can use it with XArgs and things of that nature.
[00:18:04] Unknown:
And 1 of the risks that's always present when doing refactorings is that it could potentially introduce new bugs or regressions in the code base. So it's generally necessary to have a fairly comprehensive set of tests available for the code base. So I'm wondering what are some of the tools that you use in conjunction with UNDET to ensure that you don't suffer any sort of code breakage as a result of its usage?
[00:18:31] Unknown:
Yeah. So just having having that test suite is, absolutely essential. Without without having tests, I don't I don't know how you have the confidence to to do refactorings at a scale that you can achieve with UNDET. You could certainly go through and do them all manually, but, you know, that that doesn't really scale when you have a a massive amount of of debt to to pay down. Yelp also has a thing that I think that we we recently blogged about called, Seagull, and that's our our test our testing infrastructure that runs the thousands and thousands of tests against the Yelp website.
Another thing I'll say is you were talking about this a little bit earlier, where if you have developers with different coding styles, it can create this cognitive overload, where you keep having to switch between reading different styles of code. And 1 of the nice things that we have is a tool called Precommit, which basically installs a bunch of Git hooks, mostly written in Python. And you can use those to prevent developers from ever even committing code that violates certain style restrictions. So 1 of the things that we do is we have we have 1 commit hook called reorder Python imports that takes a look at any files any Python files that you've touched and make sure that your imports follow the spec in, I don't remember the PEP number, but the 1 that says that they should go in 3 sections and alphabetized and all all of that sort of thing.
And so by by definition, no 1 can get can get code into the code base that violates some style that we've deprecated, which is nice because it's nice as a way to sort of stop the bleeding, as it were.
[00:20:32] Unknown:
And what are some of the limitations or edge cases that you've come up against when working with Undet that it wasn't able to address and you ended up having to turn to other solutions?
[00:20:42] Unknown:
So we don't have a whole lot of love for non Python languages. In fact, there was a there was a lot of confusion when we initially announced it where a lot of people seem think that they could only refactor Python code with it. So we cleared that up pretty quickly. But on the same token, if you, say, are a Go developer and you wanna do some refactorings, you could totally use on that to do that. But in order to write the grammar, you would have to know at least a little bit of Python Python and know how to use Py parsing. So that can be a little bit of a hurdle for for non Python language refactorings.
However, to that effect, we have inside of undead. Pattern. Lang, we the intention there is to have a bunch of grammars defined for various common languages so that even non Python developers can can jump right in and easily, stitch some stuff together. Another snag that I have run across personally is it can be tricky to get the white space or indentation correct when you're trying to replace things. But, you know, if you if you use something like pre commit with Auto PEP 8, then you're good to go, and you don't even have to worry about it. Like, I would just run a refactoring, not care about indentation at all, and just let pre commit fix it for me.
[00:22:20] Unknown:
So it's often a difficult balancing act when you're working in a team to determine how much of your time to spend paying down technical debt and building tools that will act as force multipliers versus doing feature work that's going to be visible to end users. So in your experience, what are some of the ways to manage that tension?
[00:22:36] Unknown:
Yeah. It can be really tough. I think 1 of the ways to help with that is to make the attitude of we have a healthy code base, make that an an organization wide priority where developers take pride in the quality of the code that they write, and not just shimming features. And I think if you can have a few key teams or owners that are sort of driving the effort to refactor and pay down some of that technical debt over time, that can really help with, progress. So, like, having a team lead a migration movement and getting getting other developers on board with setting aside some time during a sprint to clean up some of their older older stuff.
Now on the other side, you have to you have to take take a few moments and and make sure that you're still still getting things out at a reasonable pace. So I know you can do things like you can compute velocity, but
[00:23:41] Unknown:
Yeah. So this is a conversation that we've been having at my place of work as well recently. And so 1 of the solutions that we've come up with is that on our task board where we're tracking how many different things we're trying to get done in a given sprint, we have decided to allocate a certain number of tickets that are addressing technical debt so that we're working it into our regular flow of work so that rather than just letting things build up for a long time and then pay it all down en masse, we're trying to stay on top of things as they come up so that we're encouraged to, during the sprint, identify different pieces of technical debt, log it as a ticket to be addressed, and tag it as technical debt so that we can identify during planning, which are the elements that will help us make sure that we don't get stuck with a, you know, large burden of debt that we then need to pay down in a large effort and so that we can just sort of make it part of our regular flow.
[00:24:35] Unknown:
Yeah. That sounds awesome. 1 of the things that that made me think of is, when we do identify an area of technical debt, we immediately start adding in pre commit hooks, because once we've said this is a bad practice and we wanna stop it, the first thing we wanna do is stop developers from being able to to do the outdated thing more. So while it's not great if we can't get rid of all the old code, it's better that we can't allow any new occurrences of the old code to come up.
[00:25:15] Unknown:
And when you add new checks to your list of pre commit hooks, do you have any sort of automatic script that will, update the hooks across across developers' individual machines so that you don't, for instance, add a new hook that's going to prevent a particular code style from being entered into the code base. And then it happens anyway because a developer forgot to synchronize their hooks?
[00:25:41] Unknown:
So that's actually 1 of the really nice things about pre commit, is you check-in a YAML file into your repository that tells pre commit what it should run. And so we almost solved that problem because if a developer were to commit before pulling the the the new pre commit config, then they wouldn't they still wouldn't run the hook. But we have build servers that run builds on master that will that we we typically run pre commit not just as a git commit hook, but we also run it as a stand alone binary as part of testing for all of our various repositories and services. So even if I, as a as a developer, committed around a around a given hook I mean, I could even commit no verify if I really wanted to skip a hook. But as soon as I push to master, that build would immediately fail on a on a pre commit issue, and someone would presumably go fix it or roll it back or something.
[00:26:52] Unknown:
And 1 of the things that can potentially become a problem when you are dealing with pre commit hooks is if the various checks end up taking too much time to execute. So for instance, if you've got a set of unit tests that start that are starting to take a long time and you are mandating that before somebody can commit, it executes all of those unit tests, but they take 10, 15 minutes, it can definitely slow down developers. So what are some of the ways that you try to prevent that, from becoming a barrier that would then encourage people to skip running them locally and just push all of that burden to the build server?
[00:27:28] Unknown:
Yeah. So there's there's 2 there's 2 things at at play here. 1 is generally slow response times in which so so the question is whether we're talking about pre commit or or testing, Because pre commit is only gonna run on files that are touched.
[00:27:47] Unknown:
More on, pre commit. So, basically, how how you ensure that you are not blocking developers' progress when they're checking in code just because of the pre commit hooks? Because I know that it is possible to enforce that it executes a set of unit tests as 1 of those hooks.
[00:28:04] Unknown:
Oh, yeah. I would definitely not recommend having a a test suite run as a part of the hook, because that's actually a very a very hard problem. In an ideal world, you could say, I know because you touched this file, I need to run exactly this set of tests, where that is, like, the minimal set of tests that that code impacts, and that would be really great. But I am not actually sure that you can really do that. I view pre commit as a way to prevent introduction of deprecated code patterns and things of that nature, and less of a way to make sure that you haven't broken the build. Yeah. I don't know. I I've I I know that you can use pre commit to run test suites before git commit, but I have never actually seen that done in practice.
And I'm not sure how valuable that would be, especially as your test suite grows.
[00:29:10] Unknown:
Yeah. The keeping your test suite lean and mean is definitely 1 of the perennial problems in software development, which it's a good thing when it does start to grow because it means that people are actually keeping up with making sure that the code that they're writing is, I won't say provable because that is a much greater effort, but at least is, somewhat protected against regressions and that the Yeah. Obvious bugs that they can conceive of are being taken care of. But, yeah, it's definitely difficult to ensure that as your code base and your tests continue to grow, that they do continue to be able to execute in a short period of time. And that's another target for refactoring is when your test suite does start to grow out of bounds. And it's becoming increasingly rare that people actually run them locally.
And so that's that's usually a good signal that you need to pay down some technical debt in that area as well.
[00:30:03] Unknown:
Yeah. Definitely. And that can be 1 case where, you know, maybe it's time to split up repositories if if your code has has grown a bit too large. So that's 1 thing. The other thing that I'll say is, yeah, that that growing test suite is definitely a good sign because it means that developers are adding tests, versus just throwing code into the repo and saying, you know, I'm pretty sure this works. And while tests certainly don't prove that your code works like you were saying, it's definitely more confidence inspiring than taking somebody's word on it.
[00:30:40] Unknown:
So we've covered a fair bit of ground in a few different areas of maintaining the health of a code base. Are there any other topics that you think we should cover before we start to close out the show?
[00:30:53] Unknown:
I think we're pretty much good. I wanted to talk about pre commit. It it could for other other tools about code health, we talked about pre commit. We talked about get code debt. Yeah. The those those are our bread and butter.
[00:31:09] Unknown:
So for anybody who wants to keep in touch with you and follow what you're up to, I'll have you add your preferred contact information to the show notes. Okay. And so with that, I'll bring us to the pick section. My pick this week is going to be on topic. It's the book Continuous Delivery by Jez Humble and David Farley. And it is a very well reasoned book coming from people who have a lot of experience in the area of maintaining the overall health of a code base and a delivery pipeline. So for anybody who's interested in some of the best practices around continuous integration and development practices to ensure that your code is always deliverable and trying to increase its maintainability.
It's a good book to read. It's very well regarded, and Jez Humble and David Farley are both, very experienced technologists, so I highly recommend it. And with that, I'll pass it to you. Do you have any picks for us this week, Andrew?
[00:32:07] Unknown:
Yeah. I got 2 picks. 1 technical, 1 nontechnical. First, I'm gonna I should have figured out how to pronounce this. I don't actually know how to pronounce this pick, but last week, I came across an editor underneath the Google GitHub organization, called, Xi. I don't know if it's Kai or Sai, but it looks super interesting. It's written in Rust, and it's, it's a multi part editor. So the the the main x I repository just implements a an editor back end, and then you are free to implement your front end however you see fit. And you communicate over standard RPC based JSON.
And it's really great. And I've been working on a front end that I found also written in Rust called, xitui, because it's a terminal or text based UI. And, yeah, I've been having a lot of fun working on that. I like dabbling in other languages from time like in my in my free time. And then my other pick is I'm going to pick The Circle by, Dave Eggers. It's really it's probably 1 of the last really good books that I read. Dystopian sci fi fiction novel. There's a movie coming out this weekend with, Emma Watson, Tom Hanks. So I'm definitely looking forward to checking that out.
[00:33:41] Unknown:
But, yeah, definitely give it a read. Alright. Yeah. Definitely sounds like an interesting book I'll have to keep an eye out for. So I appreciate you taking the time out of your day to join me and tell me more about the ways that you're trying to keep your code base healthy. And I hope you enjoy the rest of your evening. Thanks. You too.
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. I would like to thank everyone who supports us on Patreon. Your contributions help to make the show sustainable. When you're ready to launch your next project, you'll need somewhere to deploy it, so you should check out linode at ww w.podcastinit.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your app or experimenting with something that you hear about on the show. You can visit the site at www.podcastinit.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. To help other people find the show, please leave a review on Itunes or Google Play Music, tell your friends and coworkers, and share it on social media.
Your host as usual is Tobias Macy. And today, I'm interviewing Andrew Mason about technical debt and refractoring with the tool Undebt from Yelp. So, Andrew, could you please introduce yourself?
[00:01:01] Unknown:
My name's Andrew. I'm a software engineer at Yelp. I've been working there for
[00:01:06] Unknown:
about 8 or 9 months, I think. And do you remember how you first got introduced to Python?
[00:01:11] Unknown:
Yeah. I actually dabbled a little bit at the end of my senior year of high school when I was taking AP Computer Science. But then not my I would say my first real introduction was sophomore year of university, when I took computer networks. We did a lot of assignments in Python, did, like, traceroute implementations and stuff of that nature.
[00:01:33] Unknown:
And so the main focus of the conversation today is around the tool, Undet, that you've done a fair bit of work on. And 1 of the primary purposes for that is to address technical debt in a code base. So I'm wondering if you can start off by sharing your definition of technical debt and why it's an important aspect of a project to be able to keep track of.
[00:01:53] Unknown:
Sure. So I would say that technical debt is any technical decision that you made in the past that is getting in the way of developing new things now. And I think that's different than what a lot of other common definitions are because you often see, like, oh, bad code or, like, poorly written code, code that's hard to understand. But that always implies some sort of malice or, like, incompetence, and I don't think that that is fair because you can make a perfectly correct decision given everything that you know now, and you can learn new things later, or, like, the world can change out from under you, and that would invalidate a previous decision and make a previously correct choice incorrect.
[00:02:52] Unknown:
Yeah. It's 1 of the unique aspects of software development is that there is no 1 set of requirements because even if you write the initial project to exactly match what the end user or the customer wanted, then they will inevitably come back with a whole new set of requirements that they want because, as you say, the world changes, and what may work at 1 point will suddenly stop working for whatever reason that may be. And so a software project is always a living, breathing thing that is always going to continue to change. And the only time that it is not going to be modified is when it stops being used.
So in some ways, the manifestation of technical debt is a positive signal because it means that the project is still viable and that there are still people who are gaining value from it. But as as you said, it is something that can inhibit, forward progress on the code base because of the way that it was originally written or architected. And so that leads us into the idea of refactoring. So how would you characterize the idea of refactoring in general? And what are some of the signals of when you might want to do some refactoring?
[00:04:02] Unknown:
So refactoring is a little more specific. So when I was talking about technical, that that doesn't necessarily have to be code. I know that that's what we're going to talk about. But, you know, technical, that can can cover, like, infrastructure choices, software architecture choices. It doesn't have to be code specific. But refactoring, I would say, is basically any change to the structure of code to make it quote unquote better for some definition of better that doesn't actually change the behavior of the code. So that's 1 of the key things is not not to break anything. I I forget who said this, but it was somebody like Martin Fowler, Uncle Bob, or 1 1 of those types of guys who said at 1 point, make make the change easy, and then make the easy change.
And that's where I that's that's my mental model of what refactoring is good for. When do you want to refactor? Part of me sort of is tempted to say always, but I understand that that's not really that feasible because you're not changing behavior. So if you just spend all your time refactoring your your users or your customers, they're gonna notice that there's no new features, and they're gonna move on to some other set of software that is giving them new stuff. But at the same time, doing a tech debt pay down like refactoring, there's a there's a big trade off between delivering features and making it easier to deliver features in the future, and both of those things are valuable both to you as the developer and as the customer, but there has to be some sort of balance met between the 2. And taking 1 step back before we dig into undebt,
[00:05:56] Unknown:
in order to be able to understand where you need to target a refactoring, also useful to understand what are the areas that are accruing technical debt. So I guess 1 is how do you identify where the technical debt is? And I also know that at Yelp, you've also got a tool for being able to keep track of technical debt as it accrues. So I'm wondering if you can talk about that a bit as well as how to identify where the technical debt is, present in a project.
[00:06:25] Unknown:
Sure. So technical debt identifying technical debt is sort of subjective. It's basically you have to look and say where where are our pain points when we try to make changes to this code base. Like, what what aspects of the code base are making it harder for us to change, or what areas do we not even wanna touch because they're so difficult to change. Or you could also have, like, for example, we did a lot of refactorings around database queries the way we wrote database queries via an ORM in Python, basically fundamentally changed over the lifetime of the project.
And so we wanted to go through and fix and sometimes there's performance implications. Like, we used to run a query this way, but, you know, for we use SQLAlchemy, and so this this new SQLAlchemy feature makes the query run way faster. So we wanna go through the code and update all of that. So the first thing, as I realize I'm not actually answering your question, is to identify all those places where the old pattern occurs, and we do that with a tool called Git CodeDebt. And basically, the way that that works is you point it at a repository and you define all these metrics just in pure Python.
And it's all configured via YAML, and you tell you tell gitcodet which Python metrics you want to run. And gitcodet will basically walk through the git history and hand off the text of the files to these various metrics, and then the metrics are responsible for stating whether or not that pattern appears in the text that they're looking at. And then you get these really nice graphs that can show trends in in code over time.
[00:08:22] Unknown:
A couple of other ways that technical debt can manifest as well is the if there are inconsistencies in the code style within a project because while different people may prefer different code styles, if there are multiples within a single code base, it can be somewhat jarring and it it can add to the overall cognitive load of trying to understand what's happening as data flows through a program. So, that's 1 of the potential targets for refactoring as well as, for instance, if there are things that are preventing you from being able to upgrade various dependencies in the program. You know, 1 of the most notable that Python developers would be familiar with is the idea of switching from 2 to 3. So identifying all of the print statements versus print functions, for instance, or, byte strings versus Unicode strings.
And so with that, I will ask you to dig a bit into what the Undet project is and some of the problems that were being faced in the code base at Yelp that led to its development?
[00:09:24] Unknown:
Yeah. So Undet, if I quote or read me here, it's basically a tool that allows you to perform massive automated code refactorings. And it was developed internally at Yelp because we were trying to do a bunch of migrations, like you were saying, like, 2 to 3 is a good is a really good example of that. And there were like, Yelp has a fairly large code base, and so there are obviously a lot of places we needed to touch, and nobody wanted to go through. And it's just a very boring mechanical task of going through and changing, you know, print from a statement to a function in every single file.
And so nobody really wants to do that. And so we created this tool with the idea that if other if other teams at Yelp had an easier way to do refactorings, then they would prioritize paying down that technical debt more readily. Because if you make something easier to do, people don't try to fight it as much, if that makes sense.
[00:10:36] Unknown:
Yeah. It's definitely, particularly when you're looking at the list of tasks that you need to do for a given sprint or whatever your development cycle happens to be, and you're trying to say, okay, do I wanna pick off this hairy refactoring, or do I wanna spend 5 minutes fixing up this other little piece of code or adding this little feature? It's generally a lot easier to pick up the smaller project if you're just finishing a larger task. So if a large refactoring is as simple as making a few line code change, then, yeah, it can definitely contribute to the overall velocity of those refactorings, I would imagine.
[00:11:13] Unknown:
Yep. Definitely. Especially when you consider from, like, a microservices point of view, if you have, like, 20 different codebases, right, you could or some number. I I don't know. Pick a number of code bases. And you need to apply this refactoring to all these code bases. With Undet, you can basically write the pattern once and then share that pattern with the owners of all those different services and repositories, and they can all do the updates simultaneously. And we actually do do that as we have we we provide a bunch of examples, and, similarly, we have patterns that we use and share with other developers.
[00:11:59] Unknown:
And so you're referencing the patterns for defining the refactoring targets. I'm wondering if you can explain a bit in more detail how undead actually does the refactorings and, how those patterns are defined.
[00:12:12] Unknown:
Yeah. So your patterns, you define in Python. It's built on top of a library called Py parsing, which provides this really nice syntax and uses a lot of the, the nice the nice language features of Python, taking advantage of done their methods to make it all look pretty, and it looks very, very similar to how grammars are written in language theory. So if you have any background in context free grammars, you're going to immediately recognize a lot of the syntax. And so all you really have to do is define your grammar, which is going to be used to see if a given bit of code matches.
And then you define another you define a function called replace, which takes in all the tokens that get matched by the grammar and then returns the string that you want to replace those tokens with. And that's pretty much all there is to it.
[00:13:15] Unknown:
And so there are some other existing tools for being able to do refactoring in Python, most notably ROPE. So I'm wondering what were the limitations of those existing tools that led to the necessity of creating UNDET? And, in what ways do you still use some of those other tools?
[00:13:35] Unknown:
Yeah. So when when I saw this question, I I did some googling because I don't I I've never really explored this space too much before. So other, rope was the 1 that I came across with it, but I wasn't really sure how good it is. And I'm not trying to, like, talk smack about a project or anything, but I just had never seen it before, and when I looked at the docs, it was just sort of like, here are all these refactorings you can perform. And I was having a hard time seeing how you actually, like, use rope to do it. So that could just be me not reading correctly. I'm not sure. But 1 of the 1 of the big advantages of Undet is Undet is super general, in a way. All it really does is it parses and then does transformations on text, and it's just coincidental that a lot of the text that you see in the documentation is Python. Like, in theory and actually in practice, we have some examples of this. You can use Undet to refactor anything that can be defined with a grammar. So that includes, you know, Python, Ruby, basically any programming language.
And so that's really useful, is we can we can also apply on that to, our JavaScript code as well.
[00:15:02] Unknown:
And so beyond just doing things like replacing a print statement into a print function, is it also used occasionally for, changing method signatures, for instance?
[00:15:13] Unknown:
Mhmm. Yep. So you would you could you could do something like that. You would define 1 grammar to match the function definition and change it. Actually, I think it would be easier to just go in manually and change the actual function definition and then write a grammar to match every call to that function and update its its signature accordingly.
[00:15:41] Unknown:
And once you do have that grammar defined, could you then use it to, for instance, update any documentation that references that method signature as well?
[00:15:50] Unknown:
I would lean towards saying yes. It's a little hard because we're not actually working with anything concrete here, but yeah, you would you would have to define a second pattern that took the original the 1 to match the function call, and made sure it applied to comments. And you can actually so there's there's 2 there's 2 there's actually 2 ways to define stuff in in Undet. 1 is with the grammar and replace pairs, where Undet will basically take a module and look to import a thing called grammar and a callable thing called replace. And otherwise, it can also take a it will also try to import something called patterns, which would be a set of grammar and replace functions.
So you can basically have 1 module look for define multiple things to to do find and replace on if you need to make sure that certain changes go together. So you could have a set of patterns, 1 of which does the actual calls and the other 1 to do the documentation.
[00:17:00] Unknown:
And 1 of the things that I thought was interesting while I was reading through the documentation for Undet was the fact that as part of the refactoring, once you've defined the grammar, it will actually tokenize the matched statements so that you can actually reuse those matched tokens in the definition of the replacement string.
[00:17:21] Unknown:
Yep. That's super cool and and really, really useful. Another another cool thing about Undet is that it plays really nicely with other Unix utilities. Like, it can read files, file names from standard in, which means you can pair pair things with GRAP or GitGrep or anything like that. And so therefore, you can, like, apply a set of changes to just a specific part of your code base, or if for some reason you need to refactor really, really fast, you can use it with XArgs and things of that nature.
[00:18:04] Unknown:
And 1 of the risks that's always present when doing refactorings is that it could potentially introduce new bugs or regressions in the code base. So it's generally necessary to have a fairly comprehensive set of tests available for the code base. So I'm wondering what are some of the tools that you use in conjunction with UNDET to ensure that you don't suffer any sort of code breakage as a result of its usage?
[00:18:31] Unknown:
Yeah. So just having having that test suite is, absolutely essential. Without without having tests, I don't I don't know how you have the confidence to to do refactorings at a scale that you can achieve with UNDET. You could certainly go through and do them all manually, but, you know, that that doesn't really scale when you have a a massive amount of of debt to to pay down. Yelp also has a thing that I think that we we recently blogged about called, Seagull, and that's our our test our testing infrastructure that runs the thousands and thousands of tests against the Yelp website.
Another thing I'll say is you were talking about this a little bit earlier, where if you have developers with different coding styles, it can create this cognitive overload, where you keep having to switch between reading different styles of code. And 1 of the nice things that we have is a tool called Precommit, which basically installs a bunch of Git hooks, mostly written in Python. And you can use those to prevent developers from ever even committing code that violates certain style restrictions. So 1 of the things that we do is we have we have 1 commit hook called reorder Python imports that takes a look at any files any Python files that you've touched and make sure that your imports follow the spec in, I don't remember the PEP number, but the 1 that says that they should go in 3 sections and alphabetized and all all of that sort of thing.
And so by by definition, no 1 can get can get code into the code base that violates some style that we've deprecated, which is nice because it's nice as a way to sort of stop the bleeding, as it were.
[00:20:32] Unknown:
And what are some of the limitations or edge cases that you've come up against when working with Undet that it wasn't able to address and you ended up having to turn to other solutions?
[00:20:42] Unknown:
So we don't have a whole lot of love for non Python languages. In fact, there was a there was a lot of confusion when we initially announced it where a lot of people seem think that they could only refactor Python code with it. So we cleared that up pretty quickly. But on the same token, if you, say, are a Go developer and you wanna do some refactorings, you could totally use on that to do that. But in order to write the grammar, you would have to know at least a little bit of Python Python and know how to use Py parsing. So that can be a little bit of a hurdle for for non Python language refactorings.
However, to that effect, we have inside of undead. Pattern. Lang, we the intention there is to have a bunch of grammars defined for various common languages so that even non Python developers can can jump right in and easily, stitch some stuff together. Another snag that I have run across personally is it can be tricky to get the white space or indentation correct when you're trying to replace things. But, you know, if you if you use something like pre commit with Auto PEP 8, then you're good to go, and you don't even have to worry about it. Like, I would just run a refactoring, not care about indentation at all, and just let pre commit fix it for me.
[00:22:20] Unknown:
So it's often a difficult balancing act when you're working in a team to determine how much of your time to spend paying down technical debt and building tools that will act as force multipliers versus doing feature work that's going to be visible to end users. So in your experience, what are some of the ways to manage that tension?
[00:22:36] Unknown:
Yeah. It can be really tough. I think 1 of the ways to help with that is to make the attitude of we have a healthy code base, make that an an organization wide priority where developers take pride in the quality of the code that they write, and not just shimming features. And I think if you can have a few key teams or owners that are sort of driving the effort to refactor and pay down some of that technical debt over time, that can really help with, progress. So, like, having a team lead a migration movement and getting getting other developers on board with setting aside some time during a sprint to clean up some of their older older stuff.
Now on the other side, you have to you have to take take a few moments and and make sure that you're still still getting things out at a reasonable pace. So I know you can do things like you can compute velocity, but
[00:23:41] Unknown:
Yeah. So this is a conversation that we've been having at my place of work as well recently. And so 1 of the solutions that we've come up with is that on our task board where we're tracking how many different things we're trying to get done in a given sprint, we have decided to allocate a certain number of tickets that are addressing technical debt so that we're working it into our regular flow of work so that rather than just letting things build up for a long time and then pay it all down en masse, we're trying to stay on top of things as they come up so that we're encouraged to, during the sprint, identify different pieces of technical debt, log it as a ticket to be addressed, and tag it as technical debt so that we can identify during planning, which are the elements that will help us make sure that we don't get stuck with a, you know, large burden of debt that we then need to pay down in a large effort and so that we can just sort of make it part of our regular flow.
[00:24:35] Unknown:
Yeah. That sounds awesome. 1 of the things that that made me think of is, when we do identify an area of technical debt, we immediately start adding in pre commit hooks, because once we've said this is a bad practice and we wanna stop it, the first thing we wanna do is stop developers from being able to to do the outdated thing more. So while it's not great if we can't get rid of all the old code, it's better that we can't allow any new occurrences of the old code to come up.
[00:25:15] Unknown:
And when you add new checks to your list of pre commit hooks, do you have any sort of automatic script that will, update the hooks across across developers' individual machines so that you don't, for instance, add a new hook that's going to prevent a particular code style from being entered into the code base. And then it happens anyway because a developer forgot to synchronize their hooks?
[00:25:41] Unknown:
So that's actually 1 of the really nice things about pre commit, is you check-in a YAML file into your repository that tells pre commit what it should run. And so we almost solved that problem because if a developer were to commit before pulling the the the new pre commit config, then they wouldn't they still wouldn't run the hook. But we have build servers that run builds on master that will that we we typically run pre commit not just as a git commit hook, but we also run it as a stand alone binary as part of testing for all of our various repositories and services. So even if I, as a as a developer, committed around a around a given hook I mean, I could even commit no verify if I really wanted to skip a hook. But as soon as I push to master, that build would immediately fail on a on a pre commit issue, and someone would presumably go fix it or roll it back or something.
[00:26:52] Unknown:
And 1 of the things that can potentially become a problem when you are dealing with pre commit hooks is if the various checks end up taking too much time to execute. So for instance, if you've got a set of unit tests that start that are starting to take a long time and you are mandating that before somebody can commit, it executes all of those unit tests, but they take 10, 15 minutes, it can definitely slow down developers. So what are some of the ways that you try to prevent that, from becoming a barrier that would then encourage people to skip running them locally and just push all of that burden to the build server?
[00:27:28] Unknown:
Yeah. So there's there's 2 there's 2 things at at play here. 1 is generally slow response times in which so so the question is whether we're talking about pre commit or or testing, Because pre commit is only gonna run on files that are touched.
[00:27:47] Unknown:
More on, pre commit. So, basically, how how you ensure that you are not blocking developers' progress when they're checking in code just because of the pre commit hooks? Because I know that it is possible to enforce that it executes a set of unit tests as 1 of those hooks.
[00:28:04] Unknown:
Oh, yeah. I would definitely not recommend having a a test suite run as a part of the hook, because that's actually a very a very hard problem. In an ideal world, you could say, I know because you touched this file, I need to run exactly this set of tests, where that is, like, the minimal set of tests that that code impacts, and that would be really great. But I am not actually sure that you can really do that. I view pre commit as a way to prevent introduction of deprecated code patterns and things of that nature, and less of a way to make sure that you haven't broken the build. Yeah. I don't know. I I've I I know that you can use pre commit to run test suites before git commit, but I have never actually seen that done in practice.
And I'm not sure how valuable that would be, especially as your test suite grows.
[00:29:10] Unknown:
Yeah. The keeping your test suite lean and mean is definitely 1 of the perennial problems in software development, which it's a good thing when it does start to grow because it means that people are actually keeping up with making sure that the code that they're writing is, I won't say provable because that is a much greater effort, but at least is, somewhat protected against regressions and that the Yeah. Obvious bugs that they can conceive of are being taken care of. But, yeah, it's definitely difficult to ensure that as your code base and your tests continue to grow, that they do continue to be able to execute in a short period of time. And that's another target for refactoring is when your test suite does start to grow out of bounds. And it's becoming increasingly rare that people actually run them locally.
And so that's that's usually a good signal that you need to pay down some technical debt in that area as well.
[00:30:03] Unknown:
Yeah. Definitely. And that can be 1 case where, you know, maybe it's time to split up repositories if if your code has has grown a bit too large. So that's 1 thing. The other thing that I'll say is, yeah, that that growing test suite is definitely a good sign because it means that developers are adding tests, versus just throwing code into the repo and saying, you know, I'm pretty sure this works. And while tests certainly don't prove that your code works like you were saying, it's definitely more confidence inspiring than taking somebody's word on it.
[00:30:40] Unknown:
So we've covered a fair bit of ground in a few different areas of maintaining the health of a code base. Are there any other topics that you think we should cover before we start to close out the show?
[00:30:53] Unknown:
I think we're pretty much good. I wanted to talk about pre commit. It it could for other other tools about code health, we talked about pre commit. We talked about get code debt. Yeah. The those those are our bread and butter.
[00:31:09] Unknown:
So for anybody who wants to keep in touch with you and follow what you're up to, I'll have you add your preferred contact information to the show notes. Okay. And so with that, I'll bring us to the pick section. My pick this week is going to be on topic. It's the book Continuous Delivery by Jez Humble and David Farley. And it is a very well reasoned book coming from people who have a lot of experience in the area of maintaining the overall health of a code base and a delivery pipeline. So for anybody who's interested in some of the best practices around continuous integration and development practices to ensure that your code is always deliverable and trying to increase its maintainability.
It's a good book to read. It's very well regarded, and Jez Humble and David Farley are both, very experienced technologists, so I highly recommend it. And with that, I'll pass it to you. Do you have any picks for us this week, Andrew?
[00:32:07] Unknown:
Yeah. I got 2 picks. 1 technical, 1 nontechnical. First, I'm gonna I should have figured out how to pronounce this. I don't actually know how to pronounce this pick, but last week, I came across an editor underneath the Google GitHub organization, called, Xi. I don't know if it's Kai or Sai, but it looks super interesting. It's written in Rust, and it's, it's a multi part editor. So the the the main x I repository just implements a an editor back end, and then you are free to implement your front end however you see fit. And you communicate over standard RPC based JSON.
And it's really great. And I've been working on a front end that I found also written in Rust called, xitui, because it's a terminal or text based UI. And, yeah, I've been having a lot of fun working on that. I like dabbling in other languages from time like in my in my free time. And then my other pick is I'm going to pick The Circle by, Dave Eggers. It's really it's probably 1 of the last really good books that I read. Dystopian sci fi fiction novel. There's a movie coming out this weekend with, Emma Watson, Tom Hanks. So I'm definitely looking forward to checking that out.
[00:33:41] Unknown:
But, yeah, definitely give it a read. Alright. Yeah. Definitely sounds like an interesting book I'll have to keep an eye out for. So I appreciate you taking the time out of your day to join me and tell me more about the ways that you're trying to keep your code base healthy. And I hope you enjoy the rest of your evening. Thanks. You too.
Introduction and Guest Introduction
Andrew's Journey with Python
Understanding Technical Debt
Refactoring and Its Importance
Identifying Technical Debt
The Undet Tool
Ensuring Code Quality During Refactoring
Balancing Technical Debt and Feature Development
Maintaining a Healthy Codebase
Conclusion and Picks
