Summary
Most developers have encountered code completion systems and rely on them as part of their daily work. They allow you to stay in the flow of programming, but have you ever stopped to think about how they work? In this episode Meredydd Luff takes us behind the scenes to dig into the mechanics of code completion engines and how you can customize them to fit your particular use case.
Announcements
- Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
- When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
- Your host as usual is Tobias Macey and today I’m interviewing Meredydd Luff about how code completion works and what it takes to build your own
Interview
- Introductions
- How did you get introduced to Python?
- Most programmers are familiar with the idea of code completion, but can you just give the elevator pitch to get us all on the same page?
- You gave a presentation recently at PyCon about how to build a code completion system. What was your approach to identifying what fundamental concepts needed to be addressed and how to fit that lesson into the available time?
- In the presentation you mentioned that you had built a more full-featured completion engine into Anvil. Can you describe what possessed you to build your own code completion tool?
- What are the core components required to build a completion engine?
- What are the benefits that can be realized by customizing the completion engine for a given language or task?
- Can you describe the feature set and implementation details of the full-fledged completion engine that is available in Anvil?
- Beyond the toy example, there are a number of considerations to address if you want to make the completion engine "production grade". Can you talk through some of the obvious edge cases and how to solve for them? (e.g. handling parsing of incomplete code)
- What are the inputs that you use to build up the list of candidate tokens for completion?
- Once you have a functioning baseline for offering completions, what are some of the signals that you hook into for ranking suggestions?
- In your presentation you leaned on the machinery available in the Python standard library. What are some of the ways that you might think about generalizing across languages vs. coupling to a given language?
- What design/architectural advice do you have for compartmentalizing logic in a full-featured completion engine?
- What are some of the complexities that become a factor when you are trying to scale across an entire code base?
- Beyond just being able to parse and process a body of code, there is also the question of integrating with the development environment. What are some of the challenges that get introduced when trying to access the appropriate set(s) of files and code through the editor interface(s)?
- What are the most interesting, innovative, or unexpected ways that you have seen code completion applied to developer experience?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on code completion for Anvil?
- When is code completion more effort than it’s worth?
- What do you have planned for the future of the Anvil code completion functionality?
Keep In Touch
Picks
- Tobias
- Meredydd
Closing Announcements
- Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@podcastinit.com) with your story.
- To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
- PyCon presentation about building a completion engine
- Anvil
- Nano
- Language Server Protocol
- Jedi
- Skulpt
- Parser
- Abstract Syntax Tree
- OpenAPI
- GitHub Copilot
- Halting Problem
- Parser Generator
- Python Language Grammar Definition
- Lezer Parser Generator
- Tree-sitter
- PyScript
- Grafana Tempo Tracing Service
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. When you're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it. So take a look at our friends over at Linode. With the launch of their managed Kubernetes platform, it's easy to get started with the next generation of deployment and scaling powered by the battle tested Linode platform, including simple pricing, node balancers, 40 gigabit networking, dedicated CPU and GPU instances, and worldwide data centers.
Go to python podcast.com/linode, that's l I n o d e, today and get a $100 credit to try out a Kubernetes cluster of your own. And don't forget to thank them for their continued support of this show. Your host, as usual, is Tobias Macy. And today, I'm interviewing Meredith Luff about how code completion works and what it takes to build your own completion engine. So, Meredith, can you start by introducing yourself?
[00:01:07] Unknown:
Hi. Thank you very much for having me back. My name is Meredith. Among other things, I'm 1 of the original creators of Anvil, a platform for building web applications entirely in Python, so no JavaScript or HTML required. I also enjoy poking around with programming language usability, and recently was in Salt Lake City giving a talk about code complete as a a Python because that's the thing I found myself building and discovered was really, really cool.
[00:01:35] Unknown:
And for folks who haven't listened to your previous appearances on the show, can you just quickly give a overview of how you first got introduced to Python?
[00:01:44] Unknown:
So oh, you're you're gonna make me open with 1 of the controversial things because, actually, I got into Python, originally via creating anvil. So we wanted a full stack environment for developing applications that instead of making you learn 5 different programming languages, let you actually build with a coherent set of abstractions. And that meant 1 programming language for everything from like your back end code, your front end, your UI driving. And we chose Python for this because it's so popular with, well, you know, everybody from someone taking their first steps into programming to machine learning engineers to the folks who run Instagram.
So because that wide appeal, we reckon that will be a good language to choose. And so I ended up coming in from that end from hacking on a compiler for it into the Python ecosystem, which is a fairly unusual route in, but it was quite fun. From that compiler perspective, I'm wondering how much of that experience
[00:02:42] Unknown:
translated into your current interest in code completion.
[00:02:46] Unknown:
Oh, I mean, huge amounts. If you are building a developer tool of any sort that needs to understand what your code is doing, whether that is a compiler, whether that's code completion, whether that's assistance within development environment, it's all sort of very much scratches the same kind of mental itch. And I imagine that as this conversation goes on, we'll be talking a lot about the overlaps and how that works. In terms of the idea of code completion, most programmers who have done any amount of development, particularly if they're using a full fledged IDE, has experienced it in 1 form or another.
[00:03:21] Unknown:
But for anybody who has been living under a rock for the past 30 or 40 years, can you just give the quick elevator pitch of what code completion is and why it is useful and some of the avenues in which it will help the developer do their job?
[00:03:36] Unknown:
Sure. Code completion is that little pop up box you get if you're using a development environment like PyCharm or Versus Code. As you type, it'll be giving you a little pop up box of things you might want to type next, you know, it'll help you autocomplete typically variable names, sometimes other identifiers in your code. I mean, it really is in any modern developer toolkit, and for very good reasons. These days, the average developer uses a huge number of libraries. And code completion gives us discoverability because it lets us see what functions are available in some library module, without having to tab over to the documentation, you can just hit, you know, mylib dot and see what it's got. It gives a speed obviously because you can hit type a few characters and hit the tab key rather than taking the time to spell initialization correctly for the 3rd time this function.
It gives us confidence that what you're doing is right because there are whole classes of bugs that aren't necessarily easy to spot with the naked eye immediately, but the code complete will tell us what's wrong. And so we'll find and fix certain classes of mistakes without ever hitting the run button. For all those reasons, it feels good to use. It lets us stay in that flow state that's really where you want to be as a developer. So, yeah, code completion is awesome. You've probably used it. If you haven't, I'm really interested in what language, what environment you are working in. And if you are still hanging out in Vim with no plug plug ins, at least try the Jedi plug in. Code completion is available to you. You do not have to leave your 80 character text window to enjoy it. Yeah. And with the introduction
[00:05:09] Unknown:
of the language server protocol, let's see, become even more approachable and accessible without having to conform your development environment to what everybody else might be using, and it actually just makes it much easier for you to use VIM or emacs. Or at this point, it's probably available at nano. I don't know.
[00:05:27] Unknown:
Oh, like, I will bet you the next beer that if I Google nano language server protocol client, you'll find it. I think standardization of that has been good, although it has kind of standardized everyone onto the lowest common denominator, which I think we'll talk about in a bit. And so as you mentioned, you gave a presentation
[00:05:45] Unknown:
at the most recent PyCon to talk about some of the elements of code completion the just talk to your overall approach for identifying what the core and fundamental concepts are that needed to be addressed to make it digestible and how to fit them in the time that you had available.
[00:06:11] Unknown:
So it's a fairly big subject. And obviously, in a half hour talk, I couldn't cover all the intricacies, really was trying to talk about the basic mechanism, because code completion is 1 of those things that seems like absolute magic, until you actually work out how it works, at what point you realize it's blindingly simple in the middle, and then you can add extra sort of block complications onto the outside. So I reckon half an hour is just about long enough to explain the basic mechanism how a code completion engine works. And then to prove that it really was that simple by building 1 on stage in about 5 minutes.
And so the real challenge there, I think, was to find like a subset of functionality that's recognizably a code completer that like hits the high notes that demonstrates like what a completer does, but is still simple enough that I could write the skeleton of it on stage while chattering without boring the audience to tears. So I ended up with writing a completer that understands it was variable assignments and function definitions, and that's about enough to get through the idea that your code completer is gonna have to look at the code you're writing, work out what's going on in that programme, build up some kind of idea, in this case, some kind of model of what variables are defined at which points in the program. And then when it finds where your cursor is, it's going to have to offer you the right set of choices. If you're inside this function, then you offer local variables that are inside this function. If you're out in global scope, then you offer the global variables, etcetera.
And, you know, it was pretty successful at that. And, I mean, as I said in the presentation and I would hold to now, like all the extensions between that basic kernel and most of a full fledged completer are actually like pretty mechanical. Okay, right. I understand the assignment statement. Now I need to understand every other piece of syntax in Python, as opposed to any kind of big conceptual leap. So, yeah, the goal was to find what the core information is and the core of building a code completer, which is what's going on in a program, find out where the cursor is, and what valid moves the programmer might make, and then find a sort of subset of it that was small enough to build and demonstrate on stage.
[00:08:28] Unknown:
In the presentation and earlier in this conversation, you mentioned that you had actually built a completion engine for use in the anvil environment. And I'm wondering if you can just talk to what led you to attack this problem in anger and actually subject yourself to building a full featured completion engine and how naive you are going into it and how much you regret that decision now.
[00:08:51] Unknown:
Alright. So, okay. I feel like we're definitely teasing the audience here because we're talking around the edges without diving into properly what a completion engine is. But I suppose it's useful to set the scene. So Anvil is an web framework, it's open source, you can run it headless, just download your code and launch the standalone server. But Anvil's editor is an online IDE. And so you are building your user interfaces, writing your client side code, writing your server side code, configuring your database, and running and testing in this browser based environment.
Obviously, if you're going to try to make an IDE, then you're gonna need good code completion. And there are, as we previously alluded to, off the shelf code completion systems available. So the language server protocol is this network protocol for talking to a standalone programme that's offering code completion. And there are a couple of LSP engines for Python. There is JEDI, which is at the core of 1 of the Python LSP engines, which is an open source project for Python specific auto completion. There are a couple of other things out there. And we could have grabbed 1 of those off the shelf and used it. But Anvil has a couple of specific requirements.
1 is that Anvil knows about a bit more than your typical ID because it's full stack framework, because you've got this drag and drop editor built to build your user interface. It knows about what components are on your screen. And so we want the autocomplete to help you with that, to help you interact with those UI elements. And so it's gotta know what's in your drag and drop designer. It's also full stack server and client, so it's gotta know what's going on on the server so that when you call some function on the server, it actually knows what data you're getting back and can offer you helpful order completion and it knows the schema of the built in database. So if you get a row from the database, it can auto complete what columns you're pulling out and that kind of thing. And sure, okay, like that's a challenge, but we could have hacked that into something like JEDI. The real challenge is that it's a web based editor, which means that there's just not a lot of time between keystrokes, the programmer is typing at full programmer speed, and you don't have a few 100 milliseconds to like send your code back to the server and run it through a completion engine.
So all those reasons for us pointed towards building 1, actually running live in the browser that we had full control over so we could make it smart in the ways we needed it to be smart. And so what I did, because I was the original author of the subsystem, is I picked up the Python to JavaScript compiler that's used to run your applications because in Amble, you're writing all your code, including your client side code in Python. And, obviously, if you're running that in the browser, then we need to translate it to something the browser understands. We use a JavaScript compilers called sculpt, s k u l p t. And, basically, what I did is I cracked it open and pulled out its front end, the bit that is used for understanding code before it compiles into JavaScript and use that for understanding the user's code. And then builds the logic of working out what the user's program is doing and working out what the valid next moves were, what auto completion suggestions to present sort of by hand. It was a challenge. It was a fun challenge. I say, as a developer, it taught me an awful lot.
I think that it's 1 of those things, which is 80 20. So the initial results came very quickly and rounding out the corner cases is an ongoing project. And if you're in a position to draft off somebody else's work, I'd probably say, yeah, that was worth doing that. On the other hand, a lot of the things we're doing with Anvil, it is very difficult to do still with an LSP based off the shelf order completer. So I think we probably made the right choice for us, but we're kind of an idiosyncratic case. So, you know, no regrets, but also not necessarily the right choice for you if you're listening to this.
[00:12:42] Unknown:
And so in terms of the actual process of building a completion engine, you mentioned that you ended up having to build this from scratch because you had very bespoke requirements that were customized to your execution environment. I'm wondering if you can just talk to the actual core elements that are necessary to be able to tackle such a problem
[00:13:01] Unknown:
where you don't have prior art to be able to build on top of. I think it will be a bit grandiose to claim this isn't prior art. People have done this before. But let's zoom right out. If you're building a code completion engine, start by thinking about what it's got to do. Because you're writing a program and its job is to be presented with a half written program from the user's text editor, that's got a cursor somewhere inside it. And it's got to sort of answer what might the programmer type next, it's got to work out sort of what the valid next moves the programmer might make are. And to do that, it's got to understand that half written programme.
And this is obviously this is a huge challenge if you're just thinking about this thing as a bag of bytes you've just been handed. But this is also a bit of a challenge for something like a Python interpreter, if it just thought of the program as a bag of bytes that's been handed. But actually, if you run a program in Python, the Python interpreter opens your dotpy file, it reads it in, but it doesn't read it in by byte, execute each command as it goes. What it does is it reads it in all at once, and it's got a piece of code that turns this text into a more tractable representation. So there's already a program for understanding programs.
It's called a parser. And you can sort of think about it as a black box that you load a bunch of bytes representing a program text into the front, and then it will spit out at the other end, something called an abstract syntax tree. So this is a set of objects that represent your program. So for example, you might have a function definition object. It will have some objects representing the arguments, what names those arguments have. And then it might have a list of statement objects for what statements are in the function. And if there's an if statement inside there, well, there will be an if statement object with statements for 1 end of the statements, what's in the else block and so on. And that set of objects is much easier to walk over in code.
So what you can do is you can take the half written program the user has given you, you can feed it to the parser, ideally, you'd replace the cursor with some special symbol, and then you would feed this half written program to the parser and get your syntax tree out. At which point, you can walk through the program, building up a representation of what's going on. So, every time you see in Python, an assignment to a variable named x, Well, now you know there is a variable named x in this scope. And so as you wander through the program, when you encounter the cursor, if that cursor's in a position where it might want to type a variable name, well, you know that x is 1 of the options that the user might want to type next.
And this basic process is pretty simple. But it's even simpler in Python because we have access to the AST module, which is part of stat Python standard library, and it gives you access to the Python parser. So you can call AST dot par, then you give it a string of Python code, and it will give you that bunch of objects that represents your Python code. And so what I actually did in the talk as a demonstration was to take a program that I already had written and running with a code editor where every time I triggered the code completion, it would call this call this function with text. And I fed that text into AST dot parse, I walked through the statements that come out of that AST built up a representation every time I saw a variable assignment, I went, Oh, well, that variable must be in scope. And then when I saw the magic cursor symbol, I just offered all the variables that were currently in scope as potential completion. So say that took less than 5 minutes to build end to end.
So the basic components to build a completion engine, to get back to your question, a parser. That's pretty much it. If you have something that will turn the bag of bytes you're getting out of the text editor into a structured set of objects that you can loop over, then 90% of the battle has already won. And it's just your job to walk through the program thinking, you know, for the computer you are building, what information the program is going to want to need to know at the time you hit the cursor.
[00:17:17] Unknown:
In terms of the opportunities for improving the overall capabilities and kind of enhancing the developer's experience beyond those core elements, what are some of the other opportunities for customization and improvement and some of the ways that somebody might think about extending an existing completion engine to fit a particular domain or specific use case, like, for instance, if they're trying to add some DSL component to a development environment to address, you know, the needs of their end users?
[00:17:49] Unknown:
So I think I've already talked about this from Anvil's point of view is that, you know, what sets Anvil apart is that it's full stack. And so the code computer wants to know about more. It wants to know in Anvil's case about what's on in your user interface. But really, this is sort of extensible to anywhere where there is structure that's not necessarily part of the language that you might need to know about. And really, the opportunity that you get from customizing your completion engine or building 1 from scratch is that you get to build that knowledge into the completion engine. So just as an example, the Anvil code completer knows about the schema of your database. And so Anvil has a sort of object database that you can access here, it's all in Python code, you don't generate SQL because generating SQL in Python is almost as bad as generating JavaScript from Python, not quite, but almost.
And so the types, if you're searching a database, the keyword arguments you might have depend on what columns are in your database. And that's not information that is readily available to a code completer that can just see your dotpy files, a typical code base. So the opportunity for us there was to inject knowledge from outside just the Python source code. For example, what arguments that function takes what what keys are in the dictionary it returns. There's a sort of analog actually for this. So okay, well, I'll go for the runtime itching to. So the big problem with the web, that code completion really makes a sort of brings to a point is that if you're building a typical web application today, you're building effectively 2 programs, 1 program that runs on the server with like maybe Python and Django, fast API, or whatever you're using. And then 1 that runs in a user's web browser built with JavaScript and react or reducts and bootstrap on whatever else in fashion this week.
And those 2 programs have to talk to each other. Now, they pass data to each other an awful lot of the time. That's the main data path of your application. But if you're just analyzing that dot JS source code of your front end code, you don't really know what shape of data is coming back from 1 of your HTTP requests that goes to the server. And so there's actually a limit to how much your code completion can really offer. Your code completer really needs a bit more information about what's going on on the other end of that HTTP request. It would ideally, when you fetch something from API endpoint, it should be able to autocomplete the attributes on that object. Now, currently, the best the world has got to which is I mean, this is better than it used to be better than it was 5 years ago, certainly, is there's a specification called open API, which is a way of encoding like basically, type information for HTTP endpoints.
And this is a the analog JSON file, that if you're using certain frameworks like fast API, you can actually pull a lever and just get it out of your server code. These are the arguments that all these HTTP endpoints are expecting, this is the shape of data they will return. Sometimes you have to write that up by hand, but you can get this machine readable specification. But unfortunately, really the current state of the art is you then feed that JSON file into a code generator like Swagger, and then Swagger will spit out a bunch of dotpy files as a sort of library on your disk. And then your code completion engine will walk over that library. And then it can offer you some kind of code completion on what's coming back from your API endpoints, which it works. And it's better than not having code completion on your applications main data path. But really, you could say that whole really quite laborious complicated process that requires a ton of automation, that's really mostly a workaround from the fact that your code completer doesn't know what's going on on the server. And if that code completer could have knowledge injected into it about what's coming off those HTTP endpoints, then you could write the simple code with just like the requests library or something like that.
And you would still get the quality of code completion you'd get off the huge amount of Python generated by Swagger. So, you know, what's the advantage of a code completer being smarter in the ways that building your own currently allows you to be smarter? Well, it lets you provide the programmer with assistance without having to go jump through hoops of code generation.
[00:22:19] Unknown:
Taking that a bit to the extreme is the example of what GitHub recently released with Copilot where they say, we're going to analyze all of the source code that we can find and generate entire, you know, program fragments or function definitions for you so that you don't have to know about it. And, of course, you know, in some cases, great. That sounds amazing. And in other cases, oh my god. That sounds awful.
[00:22:41] Unknown:
Yeah. So, I mean, I think Copilot, it's a different sort of beast to like parser based code completion. Copilot is basically running your code for a great big language model, GPT 3, I think, or derivative of it. And that's like, it's spooky how good it is. Sometimes it has some kind of semantic understanding, understanding in square quotes here, but some kind of semantic understanding of what's going on in your code and it can suggest sort of, you know, whole functions of code all at once, but it's not really the same thing as what are the valid moves I might make next as a programmer. Like, parser based code completion is really good at, you know, I've typed x dots, and it knows all the attributes the X object has. And so it's giving me a kind of complete list of the valid next moves, whereas something like copilot gives you an option chosen with a certain amount of inspiration and a certain amount of chance, which is, I mean, it's useful, but it's a very different experience. It's not like Copilot is not gonna like burrow itself back into your brain stem like a code completer is. It's not going to become
[00:23:49] Unknown:
part of, I'm not even thinking about using this. It's a different sort of thing. It's really cool, but it's not the same thing as classic code completion, I'd say. Digging a bit more into what you're talking about of the various signals that you can use to be able to understand what are some of the valid inputs. Obviously, in Python, there is the abstract syntax tree. I can analyze the source code in the file that I'm editing. I can spider across the repository to understand what are all the other source code files that I might be dealing with. You You know, I can understand the import mechanisms to be able to say, okay. Now I'm going to see that I need to understand what is being imported so I can spider my virtual environment to understand where all the tokens that are in there. But if I don't have that virtual environment populated on disk yet, then I have no way to understand what those imports are actually bringing in. And then in another direction, there's what you're saying about some of the API endpoints that I might be pulling to draw in different JSON payloads that have their own structure. And, oh, now I need to go and find the JSON schema that hopefully exists somewhere so that I can load it in and tell my editor this is where it lives so that it can understand that it needs to add that to my code completion signals. And then there are opening up another potential can of worms here, the aspects of type hinting in Python to be able to say, okay. This variable x is actually of type foo, and I know that foo has attributes, you know, bar, baz, and quoks. And so I'm going to say, if I have x, then I know that I'm going to provide, you know, a suggestion for bar. So just wondering if you can talk through some of those other different elements of the types of signal that you might need to bring in and some of the extra machinery beyond just being able to parse the file that I'm editing right now?
[00:25:27] Unknown:
I mean, I think you offered a pretty good survey of them, really. It's about pulling other types into the module you're currently editing. And, I mean, honestly, I think you've gestured towards some of the big class of solutions for each of them. Really, I think that the core of all of this is the ability to pause this go through pause a piece of Python code, a walkthrough and learn some things about it, like, you know, what type of thing does this function return? And absolutely, you know, 1 of the ways you can learn that is by inferring it from code at what to do when the type annotations disagree with the code is another of these fun questions. There was a rather vituperative article a couple of weeks back about somebody getting very disillusioned with Python typing because in their experience, it was leading them wrong as often it was leading them right. I didn't think that's a majority opinion, but, like, you do have to answer questions like, I'm pretty sure that, you know, you declare this thing as colon int, but you just assigned a string to it. What am I gonna do now?
Although, actually, that does lead us onto 1 of the other functions of code completion, which is that it's not just about working out what the programmer might type next and popping up that autocomplete box. Because if you're scanning over the code, then already doing the parts of understanding what's going on, which means that you can often like highlight sort of syntactically valid, but did you really mean to do that moments? Like for example, you've declared this variable as type int, but you just assigned a string to it, are you sure? That's valid Python, but are you sure? Or you've just defined a function inside a class definition, and you haven't called its first argument self.
Are you sure you meant to do that? Valid Python, but it's questionable practices. And I think that's 1 of the great things about it that you can do all sorts of analysis. I mean, there are programs that, you know, scan your code and try to detect bad security practices, and they do very similar things as well. Once you've got your code as a syntax tree, you can really go to town on it. But yes, I think you did a pretty good survey, honestly, of the sources of information you might want to be bringing in. There's a bit of a challenge there when you're using sort of large libraries of stuff, because obviously, you don't really want to be re parsing everything on every keystroke.
And so you end up doing what's classically referred to as the index. So you'll build some kind of data structure that represents the things you know about the other pieces of code that you're not editing right now. And then you will use those to answer questions in the code computer. So, you know, you would be pause all of the request library every time someone edits a piece of code that uses it, but you will know that they've done import requests and they've got requests dot, and you know what the possible answers there are. You know that if they type request dot get, and then they want to, you know, come on, click on that to jump to code, you know, where that function was defined and so on. But it's, again, I think something like indexes runs into 1 of the classic problems with doing code completion, especially a language dynamic as Python, which is that all of this is always an approximation.
It is a perfectly valid move in Python to import requests and then go requests dot my new function equals foo. At which point, every technically speaking, every other piece of code in that interpreter has a new function in the requests package if they've imported it. And your autocomplete has to work out, well, am I going to support that? Because sure, that example was quite malformed, but your typical class in Python, we only know what attributes are in that class because the dunder init method assigned a bunch of attributes to self. There was no declaration of what was coming. So it's all very dynamic and we do have to infer a lot and we kind of have to work out, well, if I see somebody, abusing another module or assigning an attribute of another module or variable in another module, do I update my index for everybody who's using that module? Do I say, well, you're being naughty, so I'm just not gonna help you with that naughty thing you're doing?
It's all approximation, really. On a fundamental level, in a language as dynamic as Python, it's always mathematically impossible to know for sure what your code is going to do before you execute it. That's the halting problem. That is true of every programming language. But in Python, because it's so dynamic, that means you don't know for sure even what functions are going to be defined in this file. An autocomplete is always guessing. It's always putting a wet finger in the air and going, well, I'm pretty sure this is what's going to happen. And that's kind of okay.
Because code completion, I like to say it's a little bit like building the graphics for a computer game. You do not need perfect physical realism, perfect physical realism isn't the point. The point is that there is a human in front of that keyboard, and you want to keep them happy. And so when you encounter something funky, like someone monkey patching an extra function onto the requests library, It's a pretty rare case when that happens. So as long as you handle most of the things correctly, most of the time, most of the people are going to be happy most of the time. It's 1 of the reasons why I said earlier, it's a sort of 8020 game, that actually the first bits of work get you a lot of benefit, because most of the time, people don't do silly things. And so most of the time, the straightforward strategies for all the sources of data you were suggesting basically work.
[00:30:51] Unknown:
Once you have that initial 80% of this does the majority of what I want, it will pull in the different signals that I need, and now I want to add that extra few layers of polish to actually make it an enjoyable experience and provide real utility to the end user. What are some of those elements of taking a, I have a basic completer working to, I have a basic completer working that I am confident as labeling production grade and, you know, is generally useful thinking in terms of things like I have malformed Python that is not parsable, but I still need to be able to give some input, or I have multiple different signals of where this symbol might be coming from or, you know, multiple possible completions of this, you know, text fragment? How do I understand priority ranking of which 1 to suggest first and things like that? I think this question is really back to basics. This is not about, you know, how I'm gathering signals and everything in my virtual environment. This is about, you know, that you will encounter this problem, even if you are completing, you know, 1 simple Python script, and not referring to anything outside the text editor. And the big problem there you alluded to is
[00:31:55] Unknown:
an awful lot of the time, the programs we are in the middle of writing are syntactically invalid, because we haven't finished writing them yet. And if you feed a syntactically invalid program to a parser, the typical result you will get is allowed explosion and an exception race. This is obviously not great. I mean, it does help. Again, it allows that in line, you know, help in your editor, you can put a little red squiggle underneath a particular line and say, a syntax error on line 4. But what you really want is to nevertheless be able to provide some useful completions.
And this gets into the thorny area of error recovery in parsing. So the idea here is that you can write a parser that blows up on a syntax error, or you can write a parser that sort of guesses it knows that it's hit something syntactically invalid, then it goes, well, there's a syntax error somewhere around here, but can I sort of squint and still pass the rest of my program? So this could be as simple as like, I am just going to forget that the line of code on which this error occurred even existed and see if the program passes if I get rid of it. Or you can go into basically arbitrarily complex stuff about, oh, I think they've forgotten the comma here. If I inserted a comma, you know, at column 14, then the rest of this code parses. So I'm gonna cross my fingers and parse this code as if they had that comma there, and then later, I'll tag this with the syntax error saying, by the way, this is a best guess, but it's always a guess. You have some partial AST. And what you really hope is that there is enough information in that AST that you can still gather the information you want for code completion about things like, what variables are available.
Like, that's roughly how you cope with errors is you spot them, you try to recover from them by fair means or foul. Because again, this is an approximation, you only need to keep the computer happy. There is no correct answer. It's a syntactically invalid program. You just have to come up with an answer that you can get away with. And then you pause what you've got, and you ignore the syntax error token and serve the user your best auto completion. And sometimes it will be nonsense, and your hope is that most of the time, it's better than just giving up. So that's handling partial code.
You also asked about a sort of interesting 1 about how you prioritize the results you get back, how you'd rank suggestions. And I can see how that sort of makes sense as a question because it feels almost like a search engine when you want to rank suggestions. But the thing about a a parser based code completion, classic code completion, is that it's part of the editor. It's kind of being used sort of, you know, halfway to the brain stem by the programmer. They're not thinking about your code completion, which means that it's really quite important for your code completion not to try and be too clever. If the programmer is using code completion for discoverability, they are typing requests dot to find out what functions are in this module they've just imported.
Sure, yes, it will be It's useful to show them the stuff they might be interested in first. But most of the time, a program will be typing reqtab.getab. And your code completers job is to ensure that that actually produces the code they want. And if you're trying to be too clever about it, you're going to trip them up because you will no longer be predictable. If they typed request of GA, they're confused, they're at the wrong place in the listing, they want to know it's alphabetical. So they'd know they go down in the list of possible completion to get to get. So I think really prioritizing completions, except for sort of broad things like you probably are not interested in the Dundas. So sort of sort the Dundas towards the end of the list.
Apart from broad things like that, prioritization, we restrict prioritization to like, we get 1 guess. 1 guess of, you know, if you types of self dot, or anvil dot, you know, what first completion should we start out highlighted in that little drop down box of possibilities. But beyond that, it's an alphabetized list. You know, it completes prefix matching first. And if you hit the tab key, it will complete what you expect. That's what the program is going to want, hitting a new key. In some cases, it's like in almost single digit milliseconds. They don't really have a lot of time to interpret what your code can be just pulling up, so you want it to be predictable.
[00:36:28] Unknown:
In terms of that timing aspect, there's also the question of this is probably more of an editor question than a completer question, but, you know, how soon do you want that book window to actually pop up and give me a suggestion? Like, if I'm typing and I know what I'm typing and you pop up a suggestion, I'm just gonna get angry.
[00:36:43] Unknown:
Oh, boy. Yes. It's an excellent question, isn't it? I'm not gonna pretend that I have the answer. That's again, like, that 1 is 1 of the questions that illustrates how far we are into the land of your job is to keep the human happy, there is no objectively mathematically correct answer. Typically, answers you see are things like, if they pause for a certain number of milliseconds, you pop it, or if they hit the tab key or control space, whatever their hot key is, then you pop it instantly. Or if there's only 1 completion left at that point, you do the tab, even then, like that causes edge effects. Like, that means that, you know, typically, the tab key will insert the currently selected completion, if there is 1 popped up. So they can be, you know, 1 millisecond, either way and the tab key either pops up little list or it causes you to complete the first thing on that list that just popped up. And again, get the wrong 1 of those and you make the programmer angry.
Yeah. I'm not gonna pretend I have any hard and fast rules there. The answer to that 1 is just use a test. Sit somebody down in front of your program and see exactly what say shade of purple they turn as they fight with your autocomplete.
[00:37:50] Unknown:
Going back to the question of indexing and, you know, understanding how frequently you want to rebuild the index for which pieces of the overall space that you're trying to work with, you know, that that gets into the question of there are 2 hard problems in computer science, naming things, cache invalidation, and off by 1 errors.
[00:38:09] Unknown:
Yep. And auto completion deals with both of them. Yes. Yes. Absolutely. I mean, yes, this depends on your strategy for how you're dealing with the dynamism. There's also like all sorts of things. We actually, we encounter some problems shaped like this quite a bit in Anvil, which is that if you have 2 components with like a mutual dependency or mutual recursion, or typically the thing you'll find in Anmol app is that we have like 2 forms to like the views, chunks of UI. And 1 of them is embedded inside the other. There's this list of components on the screen, you've initialized it with a bunch of rows you've just got back from a database.
And so obviously, when you update the code that reads from the database and plugs the result into that list, you want to make sure that you propagate that information so that when you're editing the template for each item in that list, where you get all the nice code completion for, hey, what columns are available in the item I'm displaying. But also when you click a button on that item, it opens another form to edit that is simple, that list is my to do list. And when I click, I can open a detail page that edits that particular item from my to do list. Well, now the type of that item that's been passed through is also now subject to change because I've changed something like 3 modules away. And if I jump straight editing module number 1 to editing module number 3, am I gonna have stale information from my index Because we haven't worked out, we need to reposs the 1 in the middle that transfer that data around. So yes, hard problems, all mostly heuristics.
Honestly, if I want to think about, you know, what surprised me in building a code completer, probably the biggest surprise is how well cheating works. So you're approximating, you can't rebuild everything anytime something that might touch it gets edited because you'll be reindexing your entire app on every keystroke. But it really is remarkable how far you can get with pretty simple heuristics about, for example, when to rescan what, or even guessing what the program is going to do, or even like backward references. So, you know, if you're editing something further up the file, but, you know, you're using a function to find further down that file, well, you know, how exactly do you sort of know about what's later that early in the past and it or that early in the scan. And again, it turns out really simple heuristics, like, you know, just pass it twice. I mean, they don't get you all the way there, but they really do get you very close for most users.
So, yeah, I'd say, to answer your question about thing for you to scan things, how do you deal with x computationally intractable, theoretically impossible to solve perfectly problem? The answer is you cheat, and often you cheat dirtier than you might think.
[00:40:53] Unknown:
Digging more into some of the implementation details, you know, in your presentation and in the conversation we've been having so far, we've been talking a lot about using some of the built in machinery in Python to be able to parse the syntax and turn it into that AST, which is part of the standard library. Different languages might have different AST representations or different ways of structuring their trees. And I'm wondering if you can talk to some of your thoughts about being able to generalize across the space of, I want to write a code completer that works for different languages and how heavily to lean into the machinery for a given language runtime that you're trying to target. And then as part of that, how you think about the architectural elements of designing the actual code that you're writing to write the completer for other people's code?
[00:41:39] Unknown:
Yeah. Roughly, if you're writing a computer, you know, are you writing a single language or a multi language system or something you're expecting to, like, plug in multiple languages to? So when we talk about parsers, you know, I talk about the Python parser. It's not actually usually a handwritten piece of code that grubs through bytes and uses the algorithm to turn this into an AST. Usually, instead, what you get is you use a parser generator. So if you look into the Python documentation, you'll find something called a grammar file. And this is a textual representation of every structure that could make up a valid Python program. So for example, you can find the definition of an if statement.
This is the file that says the definition of an if statement in Python is the word if and then an expression and then a colon and then a block of code and then maybe an else statement, another block of code, maybe an else statement and another expression, another block of code, maybe nothing at all. And you can drill into that and you will find the definition of what exactly makes up an expression and, you know, or it will describe to you all the symbols you can combine and how to make a valid Python expression. And then this exhaustive text file, which goes on for pages and pages and pages, you can then feed that to a parser generator.
And that is a program that reads in this grammar file and spits out a parser. So it spits out the code for a programme that reads in your programme and spits out an AST. It's it's all extremely meta. And there are absolutely frameworks for sort of parser generators that are explicitly aimed at like syntax, hierarchy and code completion tasks. There's 1 called laser, which is l e z e r, which is code mirrors version. And there is 1 tree sitter, which is 1 that's built in rust. And these are projects that are designed to take a syntax description of almost any language, and then spit out a syntax tree.
Now, the syntax trees they spit out are less nice to work with than something like Python's AST, usually because they are like a concrete syntax tree, which is to say that the actual structure of the objects that's produced is derived very mechanically from the grammar description. And unlike the way that Python's parser works, most ways to specify the grammar of a language in a generic way will produce some rather ugly syntax trees to parse, which means that if you're writing the completion engine that crawls over them trying to understand what's going on, then you're in for a slightly more painful time. But of course, however, this is done, the tree at the other end is the shape of that tree is going to depend on the language you're building. So you can't really abstract out understanding of the language. The language has its own rules for things like variable scoping, Things like where you can use a function, how?
How arguments work? What syntax is even available? And I can't speak to how people like JetBrains have done it because they build a very good code completion system that's used across their fleet of IDEs, you know, PyCharm for Python, IntelliJ for Java. They have 1 for almost every language under the sun these days. And I imagine that they share some components, but my wet finger in the air guess, and I'm waiting for Paul to email me as soon as this episode goes live and tell me I'm wrong, My wet finger in the air guess is that there's actually quite a lot of language specific code, because you need something that really that code that's walking through a program, understanding what it's doing, kind of needs to understand what that programming language is about.
Otherwise, like, you're going to have sort of approximations caused by being generic, and that's gonna read as quite unpythonic if somebody is writing a Python program. So my guess is that there's actually relatively little you can do to try and handle multiple languages at once. I think that probably the only bits that you could abstract out are things like the parser, which is why there are projects like laser and tree sitter. But I think that building a good computer certainly is a more artisanal process of trying to actually understand the program and working out exactly how much you can infer from a statement in any given language. And I don't think you can really replicate that automatically.
[00:46:18] Unknown:
To your point of, you know, every language environment has its own vagaries that need to be accounted for in the completion engine. I'm sure that there are also elements of the ways that you think about indexing those semantic structures in a way to make them accessible to those completion engines that raise the bar for being able to have any sort of common standard for that. I know that for at least some subset of languages, there is the c tags approach, which started for c as an indexing structure. You know, some other completion engines try to lean on that generally with varying levels of success, but, you know, to the point also of in order to be able to index the entire list of tokens that I might want to complete in my program, I have to first download all of the associated libraries into a virtual environment so that I can then crawl all of those and cause my fans to go into turbo mode for an hour to build up those indexes, you know, wondering about the viability of being able to even within a given language community, say, okay. This is the standard index format for being able to represent all of the tokens that might go into a completion engine. Those now get compiled into those packages so that I can download those as a metadata file or, you know, fetch them from an API so that my completion engine can give me those tokens without actually having to download all of those files, rerun all that index, and, you know, waste all of that compute energy across any number of thousands of machines, you know, turning that into some sort of standard reference so that I can have a completely empty virtual environment directory, start typing. I'm going to build a Django application and have all of that code completion in my editor, you know, from the moment that I start.
[00:47:51] Unknown:
So I do know that, again, the JetBrains family, tip of the hat once again, their Java IDs do have like prepackaged indexes for some libraries, I believe. But generally speaking, I think what you'll be asking there for like, getting different completion engines to interoperate is kind of asking them to standardize on their internal data structures, which I think is a bit of a big lift. I think that's probably best done, like, within 1 completion engine, which, you know, is more feasible for a giant. So I could absolutely imagine Microsoft's completion engines for common languages to do that and have some kind of central repository.
In general, I'm not sure that would be useful. Certainly, as someone who wrote a completion engine for idiosyncratic reasons, I guess we would make use of such a repository where it available. But the very first thing we would have to do is basically translate it into our world because this is the thing we know from things like language server protocol or or rather the existing LSP servers out there. There are ways that most completers think about code that aren't particularly handy for some of the bits of information that Anvil wants to inject into the computer's knowledge. And I can hardly imagine we're unique here. Here. You know, if you're writing a new completion engine, it's probably for a reason. Therefore, probably don't want to share your central implementation with the previous guy. Well, in terms of
[00:49:17] Unknown:
your experience of building a completion engine, working with end users to understand the utility of that and where you've gone wrong, you know, your work as a developer experiencing other people's completion engines. What are some of the most interesting or innovative or unexpected ways that you have seen code completion used and applied to enhance or augment the developer experience?
[00:49:39] Unknown:
I guess it's probably safest to toot my own horn here and say something that I'm quite proud of was that integrating like code completion into the Anvil UI editor so that you actually, when you're also saying, okay, how do we set up a property, a particular component on the screen, that's actually a Python expression and that's got like a full code completion engine with context from the rest of your app attached to it. That's something I'm pretty proud of. I've heard some really fun things from companies with absolutely gargantuan code bases, where, you know, just like, you know, downloading all of the code in a, you know, Anagu faced soft repository, just to index it when you're using common libraries is, you know, that would be a bit of a challenge.
Even downloading the indices for all of those will be a bit of a challenge. And I have heard of some quite fun things being done in those environments to try to provide a good auto completion experience despite it not being a sensible thing to do to drink the ocean and parse everything that's on your Python path. So, yes, some of the stuff I've heard I've heard from there is pretty cool as well. I think really, anything that gets you
[00:50:52] Unknown:
just like an incrementally better understanding of what the program might do, what options are available in front of you is always gonna be a win. In your own experience of working in this space and building a completion engine and shipping it to end users, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?
[00:51:10] Unknown:
I think honestly, I think I've already answered this. By far, the biggest surprise is how well cheating works. You can you can use some remarkably simple heuristics, and it will give you remarkably high fidelity code completion.
[00:51:23] Unknown:
What are some of the situations where either building a completion engine or applying code completion to a problem space is more effort than it's worth, and you're best suited to just read the docs?
[00:51:34] Unknown:
Almost never. I do think that a code completion engine is the best index into your API docs anybody will ever use. But I do say API docs. This was in fact the subject of my last year's PyCon talk. But there are different sorts of documentation for a project. If you are using code completion, you're using effectively API documentation. Like you are answering questions, what functions are in this module? If I grab this object, what attributes will it have? What arguments does this function take? And that is good, but that's not all of the documentation of a project. If you're building a project for yourself, you're going to want some tutorial, some high level, like from the beginning, teach somebody the ropes, show them around, you might want some how to guides to accomplishing specific, you know, real world tasks, you might want some reference documentation, there's not actually API documentation, people confuse the 2 a lot, but is sort of more of a description of the machinery and how to operate it. That isn't necessarily information that can reasonably conveyed, as you know, a little bit of description part of some big machine. So the example I use here is, if you've got some unit testing library, it has some concept of a code fixture. So you run some code to set up your test environment, you run a test, you run some code to tear it down, that piece of machinery needs reference documentation, what it does needs to be fully specified.
But you can't actually get all the information about how that library works by looking at the API docs for you know, the at test dot setup at test dot test at test dot teardown decorators, you need some high level reference or not higher level, but some reference that describes that whole piece of machinery. And so, that's an example of information that you can't really get out of a code completer. You've really got to go jump to the docs. But for the docs that are shaped like that, for API documentation, for describing individual code objects, there's really nothing better than a proper code completion.
[00:53:44] Unknown:
As you continue to support and iterate on the completion engine that you have found yourself maintaining and owning? What are some of the things you have planned for the near to medium term or any interesting areas of exploration that you have slated?
[00:53:58] Unknown:
There's always the ongoing maintenance and, oh, you know, the completion engine should have inferred that. The big most fun thing that's happening at the moment is that there is a rewrite ongoing of sculpt's parser. So sculpt, that's the Python to JavaScript compiler. And we use sculpt's parser as the front end for the fanboys auto completer. And that parser is being rewritten. I'm 1 of the maintainers of sculpt. So I'm involved in this effort. And a couple of goals. 1 is, of course, going to be to give better fidelity to new versions of Python, as the language grammar changes. But the thing I'm really excited about is giving it better error recovery, so that our code computer can be better at sort of seeing through syntax errors and giving the user a little bit of help regardless.
[00:54:42] Unknown:
Just to kind of be devil's advocate, why not just decide to use PyScript and let it do the parsing for you?
[00:54:50] Unknown:
I mean, the PyScript parser would have been a legitimate choice. If people listening aren't familiar with PyScript might as well give the thumbnail. So PyScript is the full CPython interpreter compiled to web assembly and then used to drive an HTML page. So it's basically using Python instead of JavaScript. So you open a Pyscript file, your browser downloads like all of CPython. It's fairly chunky, which is 1 of the big problems with it for some applications, obviously not for others. And then it runs a full actual, like the same Python that runs if you type Python on your command line, running the same thing in your browser. And then, you know, it's got APIs for driving the HTML elements in your web page and so on. So you ask why not use its parser? I mean, actually, perfectly reasonable, something like the Anvil IDE, it doesn't really matter if it's downloading 16 megabytes of Python runtime and had PyScript existed back then, that that would have possibly been a reasonable answer. I mean, as it is, we kind of benefit from having overlap between the compiler that executes the code of Anvil applications, which is pretty lightweight. I think the core compiler is under 300 kilobytes on the wire and we can break down the standard library and download it in chunks when the user wants it. So can load faster than the WASM version of Python. So there's good reasons for using that for the front end of apps built with Anvil. And once we're doing that, there's really a lot to gain from sort of reusing that component and gaining familiarity with the 1 component rather than trying to be deeply familiar with both. Well, for anybody who wants to get in touch with you and follow along with the work that you and your team are doing, I'll have you add your preferred contact information to the show notes. And so with that, I'll move us into the picks.
[00:56:27] Unknown:
This week, I'm going to choose Weird Al Yankovic. I recently kind of rediscovered him and have been enjoying listening to some of his songs with my kids. So if you're looking for some lighthearted fun and some interesting takes on different popular songs, always worth giving Weird Al a listen. So with that, I'll pass it to you, Meredith. Do you have any picks this week? And he is such a good musician
[00:56:47] Unknown:
as well. Like, he's really technically well executed. Yeah. Full marks to him. My pick for this episode is going to be Timescale DB. So generally speaking, I'm an advocate of keeping your infrastructure environment simple, Like, if you're maintaining a zoo of different sorts of infrastructure elements, and the chances that 1 of them is going to bite you keep going up. And historically, I've tended to, if a problem can be solved with Postgres, use Postgres for it. There's this line people use about Python is that it's the second best choice for everything. And I kind of like that is the way that learning Python gives you a good all around capability.
And in the same way, Postgres is kind of the 2nd best choice for everything involving data storage. It's not a NoSQL database, but like it's JSON store is close enough that it might as well be. And 1 of the few areas where that wasn't followed through was like high density time series data, which Postgres historically coped fairly poorly with. And then Timescale DB came along and made a Postgres extension that turns it into a pretty good time series database that you can use with all the Postgres tooling that you've become familiar with. So, yes, we are big fans. We use it for a whole bunch of logging and time series work in Anvil, and it will, I think, remain a tool I reach for many years to come. Even better, they have just released, I haven't had a chance to properly get to grips with it. But they've just released out a prom scale, which is using Timescale DB for, like, monitoring application monitoring and tracing rather than bringing in, like, we use the Grafana stack, so things like Tempo, storage engine, so on. But if you can store your traces in Postgres and, like, query your application tracing in Postgres, again, like, everything gets better. I like really like using the 1 tool for everything, and Timescale DB
[00:58:43] Unknown:
drastically expands the set of problems I can use 1 of my favorite tools on. So, yeah, that's my pick for the week. Alright. Well, definitely, second, your commendation of the Timescale DB folks, definitely a good product that they've built. I actually had them on a couple episodes of my other podcast, so I'll add links for folks who are interested. And so definitely want to thank you very much for taking the time today to join me and share your learnings and experience of digging into the space of code completion. It's definitely an area that is obvious as a developer of when you're using it, but can often be mysterious and appear to be a black box if you don't dig into it. So thank you for helping to shed some light on that, and I hope you enjoy the rest of your day. Thank you once again for having me. It's been great.
Thank you for listening. Don't forget to check out our other show, the Data Engineering Podcast at dataengineeringpodcast.com for the latest on modern data management. And visit the site of pythonpodcast.com to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email host at podcastinit.com with your story. To help other people find the show, please leave a review on Itunes and tell your friends and coworkers.
Introduction and Guest Introduction
Meredith's Journey into Python
Understanding Code Completion
Building a Code Completion Engine
Challenges and Solutions in Code Completion
Extending Code Completion for Specific Use Cases
Signals and Type Hinting in Code Completion
Handling Malformed Code and Prioritizing Suggestions
Indexing and Cache Invalidation
Generalizing Code Completion Across Languages
Innovative Uses of Code Completion
Lessons Learned and Future Plans
Closing Remarks and Picks