Summary
In order for Python to continue to attract new users, we need to have an easy way for people to get started with it, and Windows is still the most widely used operating system among computers. Steve Dower is the build maintainer for the Windows installers of Python and this week we spoke with him about his work in that role. He told us about the changes that he has made to the installer to make it easier for new users to get started and how modern updates to the packaging ecosystem for libraries has simplified dependency management. He also told us about how the Visual Studio team is building a set of tools to make development of Python code more enjoyable and how Microsoft’s adoption of open source is making Windows a more attractive platform for developers.
Brief Introduction
- Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
- I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable.
- Linode is sponsoring us this week. Check them out at linode.com/podcastinit and get a $20 credit to try out their fast and reliable Linux virtual servers for your next project
- We are also sponsored by Sentry this week. Stop hoping your users will report bugs. Sentry’s real-time tracking gives you insight into production deployments and information to reproduce and fix crashes. Check them out at getsentry.com and use the code podcastinit at signup to get a $50 credit on your account!
- Visit our site to subscribe to our show, sign up for our newsletter, read the show notes, and get in touch.
- To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
- Join our community! Visit discourse.pythonpodcast.com for your opportunity to find out about upcoming guests, suggest questions, and propose show ideas.
- Your hosts as usual are Tobias Macey and Chris Patti
- Today we’re interviewing Steve Dower about Python on Windows
Interview with Steve Dower
- Introductions
- How did you get introduced to Python? – Chris
- You are currently the release manager for Python on Windows. How did you end up with that responsibility? – Tobias
- While Python has supported Windows for a long time, the overall experience has historically been rather poor. Can you give a bit of the background of why that was and tell us about some of the work that you and others have been doing to make it better? – Tobias
- Given that a large percentage of users are still on Windows, having a good story for getting started with Python on that platform is important for adoption of the language. What are some of the areas where the current situation needs to be improved? – Tobias
- What is the most difficult part of building a distribution of Python for a Windows environment? Has it gotten easier in recent years? – Tobias
- When we were speaking at PyCon you mentioned that the most frequently downloaded version of Python from the python.org site is the 32 bit version for Windows. Do you think that is an accurate and useful metric? What other statistics do you wish you could capture or improve? – Tobias
- How does Python Tools for Visual Studio compare with other Python IDEs like Pycharm? – Chris
- What are some unique features that Python Tools for Visual Studio offers that other tools don’t? – Chris
- Are there any compelling aspects of developing Python on Windows that could convince users on other platforms to make the switch? – Tobias
- Could you give our listeners a whirlwind tour of the underlying implementation of PTVS? How does Visual Studio provide such in depth introspection for your Python code? – Chris
Keep In Touch
Picks
- Tobias
- Chris
- Steve
Links
- Windows compilers
- Visual C++ Build Tools (for Python 3.5 and later)
- Visual C++ Compiler for Python 2.7
- PEP 514
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
[00:00:14]
Unknown:
Hello, and welcome to podcast. Init, the podcast about Python and the people who make it great. I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. Linode is sponsoring us this week. Check them out at linode.com/podcast tonight and get a $20 credit to try out their fast and reliable Linux virtual servers for your next project. We are also sponsored by Sentry this week. Stop hoping your users will report bugs. Sentry's real time tracking gives you insight into production deployments and information to reproduce and fix crashes. Check them out at get sentry.com and use the code podcast in it at sign up to get a $50 credit on your account. To help other people find the show, you can leave a review on Itunes or Google Play Music and tell your friends and coworkers. You can also join our community at discourse.pythonpodcast.com for your opportunity to find out about upcoming guests, suggest questions, and propose show ideas.
Your host as usual are Tobias, Macy, and Chris Patti. And today, we're interviewing Steve Dauer about Python on Windows. So, Steve, could you please introduce yourself?
[00:01:09] Unknown:
Yeah. Hi. My name is Steve. As you can probably tell straight away, I'm Australian. I've moved to the US, about 4 or 5 years ago now, to start work at Microsoft. So I've been working there for that time on a lot of the the Python tooling and Python libraries and stuff that you will have seen coming out of Microsoft recently. And in that time, I also got involved, directly working on CPython. So I've been a core contributor for a couple of years there, maintaining and just really trying to support the Windows platform.
[00:01:42] Unknown:
So how did you get introduced to Python?
[00:01:45] Unknown:
It was initially a, summer vacation job that I had in between, in between studies. So I had a few months off and had a friend, set me up with a job which he knew about. And this job was designing medical devices. They're actually a startup. They're in early stages of, setting up demonstrable prototypes of their what were they doing? They had these little cards that were sort of gonna be prefilled with all the the medical testing stuff you needed and you'd be able to inject, a sample into this card, stick it in device and it would pump everything around and eventually 1 part of it would change color and we could detect the color change and give you the results. So the idea was that you send, you know, an HIV test. You could send a 1, 000 cards, each of which is an individual HIV test and someone who's not a medical professional could do it. So I was working there for a couple of months.
They were building a prototype system which was being automated with Python. So I had to learn Python at that point, this is about 7 years ago now. I learned Python to automate this system and build that, and from there went straight into a PhD and on the first day my advisor comes to me and says, hey, I have all of this code that you can use but it's in Python. And I just went, oh, great. I know that now. So let's go.
[00:03:09] Unknown:
So you're currently the release manager for Python on Windows. How did you end up with that responsibility?
[00:03:15] Unknown:
Basically, I volunteered and everyone said, hooray, we have someone who wants to work on Windows. So so that came out of going to Python which, so all my work at Microsoft has been around Python. We've been able to get to go to PyCon for a number of years and and sponsor the conference and have a booth there and just get to meet a lot of people. Couple of years back, there was a big there was a lot of discussion going on about packaging, and I went to 1 of the panels on that. And at that panel, Nick Coghlan, who's very heavily involved in in the packaging side of things for Python, basically called for more help, called for more people to come join the distutilsigmailing list, and just be available to discuss and and figure out some of the problems that we were facing at that time. So I volunteered for that and said, you know, I'm familiar with Windows. It sounds like I'm more familiar than a lot of the people who are in the room at the moment.
I'm happy to come and help out. Fast forward a year to the next PyCon. I've been doing that. And essentially I did the same thing, and said, I've noticed the installer has a few issues. Is anyone gonna mind if I come in and and, you know, submit a few patches or, you know, update it a bit? And everyone was sort of like, yeah. Yeah. Sure. Go ahead. Talk to the guy who's currently maintaining it. The next day, the guy who was currently maintaining it sends an email saying, I think I'm done. If you can find someone to take over for good that would be great because it's time for me to step away. And so I stuck my hand up again and got the job.
And since then, I have rewritten the installer completely. I've rewritten the build completely on Windows and gone back and fixed some of the older, more obscure bugs that have just not been fixed in that time.
[00:05:08] Unknown:
Yeah. I know that for quite a while, the overall experience of getting Python installed on Windows and being able to set up your path and environment was rather difficult and fraught with peril for certain use cases. And from what you're saying, doing the complete rewrite of the installer has greatly improved that experience. I know that I recently installed Windows on a laptop for my son to start teaching him Python. And it was much easier to do that recently than it was when I first started trying to use Python on Windows about 6 years ago. So I'm wondering if you can give a bit of background on some of these some of the improvements that you made there and some of the issues that you came across in the process.
[00:05:48] Unknown:
Yeah. So 1 of the big differences between, I I guess, the Linux world and the Windows world is there's a really big cultural difference between hacking software, I guess. So on on Linux, basically all software, there's there's a few sort of device driver exceptions. But essentially all software comes down as source code which you'll then build on your own machine, and have a result that fits perfectly. Windows, the culture has always been you download the binaries or you get a CD or even a floppy disk at 1 point with the binaries on and you can just run them. And maybe there's an installer, but all that's really doing is copying them onto the disk. There's no compilation step taking place. Because of that, developer tools almost always come with a compiler but not everyone gets a developer tool. So 1 of the big issues that Windows has been facing for a long time is when you get Python, you aren't actually getting things like a c plus plus compiler or a c compiler even and as a result, when you then need 1 because you're in the developer tools world now, you need a c compiler, you don't have it and so a huge huge majority, I would estimate, of people with Python on their machine don't have a c compiler and you just have this disjoint culture. So so the Linux package developers are saying I can publish source code. When people install the source code, it will compile and they'll be good to go. And that's really baked into Python through the distutils library and, setup tools. I just have been assuming that for so long that installing packages, well, of course, you'll have a c compiler because everyone has a c compiler and and just Windows has not had that. So people will often hit packages that can't be installed because they're missing a compiler.
There's a number of other things. The the sort of culture of living at the terminal, living at the console is not the same on Windows. It's a point and click environment and has been for, I guess, 20 years now. More than 20 years. It's you work with a mouse. Occasionally, like, keyboard is for input into a program, not for starting the program itself. So things like environment variables, things like the current working directory don't have as much of an importance on Windows as they do on other platforms. As a result, things like Python that heavily rely on those, global settings don't necessarily behave the same way, and and don't work as as well or as smoothly or cleanly as they do on a platform where environment variables are front and center, everyone uses them all the time, everyone knows how to set them, and knows how to adjust things like the path, which on Windows doesn't make such a big deal. So as for fixing that, honestly, I haven't really done much to fix that.
The biggest change the installer has made recently, so since 3.5 when I rewrote it, is being able to do a proper installation for a user who is not an administrator on their machine. Because previously that was completely impossible. You had to be an administrator, you had to be able to change system level files to install Python on your machine, and this was excluding a lot of students who had machines issued by their by their schools. A lot of research departments were given computers by their schools and not given permissions in a lot of cases. That was that was my situation until I replaced the machine my university gave me with my own 1. A lot of businesses cannot install, like, won't let their employees install things on their own machines because that's I I get it's the policy, it's the culture around it, and it's how it's used. So being able to make the Python installer still install in those situations and give someone a working copy of Python is I think the biggest change that that I've made so far for 3.5.
And frankly, it makes a lot of terminal usage harder to do it that way. It doesn't really make it more like Linux. It's making the Python installation more like a Windows program should be. And I've also made a few sort of more technical changes within the core for software that wants to embed Python, which is a fairly common thing. There's a lot of software out there on Windows that you install it through a normal installer and in the background it's running Python. It's got the Python DLL. It's loading code and running it. And there's just a few changes in 3.5 that should make that easier for the developers who want to do that and distribute Python as a library of their application on Windows.
[00:10:28] Unknown:
Just being able to install Python to a user's account rather than requiring the admin privileges, like you said, greatly increases the overall reach and availability of Python for people in a large number of situations. And also, you know, maybe kids who have parents who are savvy enough to be able to lock down the machine and not give them complete, you know, admin control over everything. So it definitely broadens the range of people who can now potentially start experimenting with Python whereas previously it just would have been out of the question.
[00:10:58] Unknown:
And just to add to that, since the Python installer was last majorly updated like this, Windows changed the model so that by default you're not an administrator. So you don't even have to be that savvy these days to lock down your child's account on a Windows machine. By default it's locked down and that is just such a change. Like it's it's a big change in culture. It upset a lot of people when it happened, because it changed everything. But that is fundamentally new since the installer was last done and so it needed to be resolved.
And
[00:11:33] Unknown:
going back to what you're saying earlier too about the fact that most Windows machines don't have a c compiler available for being able to install packages with c extensions. Some of the recent updates to how those packages are distributed, namely wheels, has made that a lot easier as well. I don't know if you can speak to that a bit.
[00:11:52] Unknown:
Yeah. So wheels were just coming about, at the that they were, I guess, gaining acceptance at that point where I started joining in on Digital SIG. They've been under development for a long time. A lot of people trying to solve this problem, and it's not just a Windows problem even. It's a lot of places, a lot of people want to build once on Linux and keep reusing the build because they know they have a compatible machine and so they can use wheels and they do use wheels to set up maybe an internal package server, and then maybe on a 1000 machines instead of installing and compiling a package over and over again, they'll pull down that wheel which is essentially just a zip file with all the pre built packages inside it.
They'll pull that down and install it, still using the same pip install command, and it's a lot quicker than running it through the compiler would be. So it's it's been a it's been a problem that's being worked on for a long time, but it's but it's hit a point now where I don't I don't have numbers handy. I'm sure someone does, or can get them, but so many packages on the Python package index are coming with wheels now, at least for Windows. There's the option now to put up highly compatible Linux packages that will work on most distributions that are out there, and certainly packages for Mac, can be put up there which just means that installing packages doesn't require a compiler in so many cases now that you can almost forget that you don't have 1. Simultaneously, 1 of the great things about working at Microsoft is I get to go and chat to the people who manage things like the compilers And so I can go and sit in their offices and chat with them about here's why we should make it easier for people to get access to the compilers.
So in the same time that that's been going on, I've organized putting out a copy of the old compiler that people need for Python 27, and so that's available from a link which we'll get in the show notes, and I've also been able to help encourage the compiler team to put out a much simpler bundle of the most recent compiler, which is called the Visual C plus plus build tools 2015, I believe. Again, I'll give you a link to the show notes, which is not a full Visual Studio install. It's not the same licensing things. We want people to have the C plus plus compilers because that means they're building things for Windows that's good for the ecosystem.
So we want to make the compilers easy to get and we've been able to improve that. So now it's actually not that hard to get the compilers you need to build packages on Windows, but with so many developers having or publishing wheels for their packages, it's less necessary.
[00:14:38] Unknown:
What an incredible paradigm shift that is. Just wanna sort of take a moment to appreciate that. I mean, for so long, those of us who've been around the industry for for a while remember that, you know, vendors were so incredibly guarded about releasing compiler technology of any kind, and and this idea that end users should should should be, empowered to build their own binaries was almost, heretical, you know. So it's it's it's a really really wonderful change that's been been happening, especially at Microsoft and other vendors too. But it's it's most most heartening to see happening there because Microsoft used to be kind of 1 of the seats of binary culture. Right?
[00:15:28] Unknown:
Yeah. Oh, absolute. Absolute. And I I would say that it still is. I mean, it's it's certainly not moved to the point where regular end users are expected to be able to compile stuff on their machine. I I again, it's very easy to to have a small view of who your end users are. The the range of people that are capable of getting machine running Windows versus 1 getting machine running even Ubuntu, but some of the more obscure Linux distros that are just a little harder to get set up and running and keep running, there's a huge huge base of Windows users who cannot and should not be compiling anything at all. They should not have the c compiler on their machine because they need those few gigabytes of space for photos, for videos, for music, and they're simply not interested in building things. So while there is a shift for the enthusiasts and and it and I'll call them the amateur developers but it's it's anyone who wants to build things for the platform without having the backing of a large company behind them. Things things like Visual Studio Community Edition are specifically for supporting those and you can see that from the license. It's a full copy of Visual Studio, the same thing we charge 1, 000 of dollars for, to companies.
But if you're not a big company or if you're working on open source software, you can have it for free. There's no restrictions. It is simply the case that there are now, there's there's a big movement of amateur developers out there who are not a company making 1, 000, 000 of dollars every year off their software and we want to encourage and support them as well.
[00:17:09] Unknown:
And given that a large percentage of overall computer users are still on Windows, having a good story for them to be able to get started with Python is definitely important. And we mentioned a few improvements that have already been made, wondering what are some the areas where the current situation still needs to be improved, and are there any areas that would be, ripe for listeners of the show to potentially contribute?
[00:17:34] Unknown:
So I am still very 1 of 1 of the things that I am sort of constantly working on in the background as part of my day job is speaking to the people we need to speak to about how do we make, you know, opening a command prompt on a brand new install of Windows and typing Python do something useful. Because right now that gives you an error message. See, on the operating system in in popular use, even unpopular use, that's gonna do that, just give you a flat out error for, oh, you type Python, that's not a real thing. So I'm certainly working on trying to come up with a better approach to that, a better response to that, but as far as other things that Python needs on Windows right now, migrate to Python 35, please.
It's a considerably better experience on Windows now than 27 is and even 3334. So 35 is gonna be much better. 36 hopefully will be better again. And and the other thing is encourage package developers to produce wheels and if that means you have you have a Windows machine and you go to them and say, I am happy to build the wheel for you on my machine. I'll run the tests and give it to you so you can publish it if that is simply being there to test and give that response. And again, it's the enthusiast developers who have a C compiler on their machine who can do this and start contributing to a package just to get a wheel up there so that all the other developers who, simply using Python and want to be able to use that package are able to install it and just use it.
[00:19:11] Unknown:
Yeah. Having build hardware for platforms that the developers aren't generally working on is usually a bit of a pain point for making a package generally available to other platforms. So I know that, for instance, the Kivy folks have had some issues occasionally with getting builds available for OSX and for Windows because most of them are developing on Linux, for instance. So I don't know if that's an area for people to potentially contribute having Windows build servers. Or I know that with the advent of cloud computing, that's sometimes easier to do where they can just rent a Windows instance for a little while to build their packages and then distribute them.
[00:19:50] Unknown:
Yeah. That's certainly a good option that plenty of people are taking these days. There's a lot of work to be done there just as far as making that simpler for people to do which is, again, also something that is kind of constantly buzzing around in my head. How how can we make this really easy for people who don't use Windows regularly, who aren't familiar with it, but need to be able to build something, need to be able to diagnose when the build fails, why the build fails, do that testing, do that analysis, which again, if you jump into an a native debugger on Windows, you're going to see very different things than a native debugger on Linux. And how do we make that entire workflow easy enough that we can legitimately start going to these projects and say, plug into this and you'll get your Windows build and here's how to deal with things that don't work.
[00:20:35] Unknown:
Another question that comes up is you're saying how the way that Python is installed on Windows and referenced is a little bit different than how it's set up in Linux, you know, in terms of path and environment variables. And I'm wondering if that poses any difficulties for people who are trying to build cross platform projects.
[00:20:53] Unknown:
My hope is that it would not. So it's largely and this is this is a great opportunity for me to talk about my my first PEP, which was ever accepted. I recently had PEP 514, was approved and is is now an active PEP, which is about configuring the registry on Windows for locating Python installed. So 1 of the things that we discovered, through some of the telemetry that Visual Studio has is that most of our users on Windows have 2 or more copies of Python installed, which is totally legitimate thing to do. I have about 16 copies of Python installed on most of my dev machines because that's what I do. But we definitely found that the majority of our users had 2 or more, and locating each of those by simply typing python at a command prompt is not actually useful. It's you're gonna get 1, you're gonna get the other. It will be consistent on your machine. It may be different on someone else's machine who installed things the same way. There's just no reliable way of doing it, relying entirely on path. So what PIP 514 defines is the set of registry entries. So the Windows registry, for those who aren't familiar, is essentially a hierarchical configuration file.
It's, it has a global per machine section and a per user section that are independent, can be read independently. So Python has always registered itself in either the the per user 1, if it's installed per user or the the per machine 1, if it's installed for everyone. Which means that anyone on that machine can look in the registry and say, oh, where is Python 35 installed? Here it is. Now I can run it. And again, because most Windows users are not going to the console and just typing Python, they'll be going through some shortcut, they'll be going through some other program. It's able to use the registry to find it and execute the right 1. Where that ran into problems is if you have collisions in that, you can only have 1 value under a certain key. So if that key says this is Python 27 and you install, somehow 5 different versions of Python 27, which is actually entirely feasible, then you get collisions, you can't find all of them. PEP 514 is designed to disambiguate that, to make it possible to have each of those, identified separately. So it's a different model from what you see on Linux.
I have seen pyenv which will do, some clever path searching on Linux and give you that free choice of I have all of these different ones installed and I wanna choose which 1 to use right now. That doesn't fit the Windows model. The Windows model is this is the 1 that I want, look it up, run it explicitly, and not change the the sort of ambient state of which 1 I'm using right now. But that should make things no harder for package developers in theory, and with proper tooling around it can make it easier because it makes it a lot more feasible to have many copies of Python installed on the 1 machine and be able to accurately choose which 1 you want to use right now.
[00:23:54] Unknown:
And back on the user side, how does that tie into things like virtualenv?
[00:23:59] Unknown:
So virtualenv, could potentially take advantage of that information that's there for letting users choose which Python to work from. Typically, most people are going to simply use run virtualenv out of the version that they want to inherit from, but you can specify any version of Python to base it on. So you can type virtualenv, I believe it's dashp, and give the full path to the Python executable you want to base it from. So it doesn't matter if it's not the same version, it will go off and run against that and give you that environment based on a different version of Python. So that could potentially make it so you don't need to know the full path of where it's installed.
And given that the defaults default install path changed recently from a very short CPython 35 or CPython 34, Python dot exe to something more like c users, my username, app data, local programs, Python 3, Python Python 35, Python dot exe, not having to remember that full path could could be very handy. So that's the kind of thing that that information can enable and certainly when you get into GUI applications like the IDEs, Python, Python Tools for Visual Studio, PyCharm, Wingware, they can just display a graphical list of everything that's registered on your machine and you can simply click the 1 that you want right now and you'll get it.
[00:25:25] Unknown:
So what is currently the most difficult part of building a distribution of Python for Windows and has that gotten easier in recent years?
[00:25:34] Unknown:
I believe it's gotten easier. I I am a compulsive automator, so when I hit a problem that I have to do more than once, I will find a way to script it. So I have a a collection of scripts, some floating around, some actually checked into the repository for CPython that that will do the build. So I have a script and a big bundle of files that will set up my build machines for me. And there's there's no there's no magic there. It's just simply I can't redistribute an entire copy of Visual Studio. So that's my private script that will I can pull up a clean VM and have everything installed very quickly. But then actually building the distribution of Python for Windows is it's 1 batch file. So it's 1 1 shell script that I can run that will do everything, with a few with only a few options and, again, I optimize them for my use cases. It will code sign all the binaries. It will run installation tests. It will do the 32 bit, the 64 bit versions of Python and I can kick that off and walk away for an hour and come back and it's all done. And then I have 1 more batch file to upload the official release. Those are both checked in and it's considerably easier now, I believe, than it used to be. But again, that's it's my automation. I understand it. So far, I'm aware of 1 other group that's made use of it and has made a few tweaks to the installer and run and and is using these batch files to produce their own distribution of Python 35.
So it has helped other people. I believe it's easier. The most difficult part is getting hold of the tools but even that with the c plus plus build tools which it which are sufficient for building Python, I just did it recently on a new machine, it is easier than it's ever been in the past, I believe, to to get the tools you need onto a machine. I haven't tested this explicitly, but I believe all you need is a copy of subversion and a copy of the c plus plus build tools, and everything else will be installed as part of the build process. So that should be all that's required to get started building Python on Windows is Subversion and the c plus plus build tools.
[00:27:46] Unknown:
That's definitely a big improvement because I know that at least as of a couple of years ago, being able to build the Windows distributions for Python was a bit of a black art that had a fairly low bus factor. So having it automated is definitely a good step in the right direction. When we were speaking at PyCon, you mentioned that the most frequently downloaded version of Python from the python.org site is actually the 32 bit distribution for Windows. I'm wondering if you think that that's an accurate and useful metric and wondering if there are any other statistics that you wish you could capture or improve.
[00:28:18] Unknown:
Yeah. So so we don't do a great job of capturing the statistics from there. I managed to, get access to the the raw logs from, someone there who trusts me, which is very kind of them. So so I've been able to pull those down and do some basic analysis, just to to get an idea of what people are actually downloading. So it turns out that since since I last spoke to you about that, the the downloads of Python 3 5 2 are blowing away everything else. They're about double so so per average per day is roughly double what Python 27 is getting at the moment or 2712 is getting. So as far as people downloading the latest versions of Python, there there's about 2 thirds going for Python 3 versus Python 2 which is awesome to see, really exciting to see that. There there's a few odd versions of Python 27 that's still being downloaded. 275 for some unknown reason is really popular. 273 is still up there as well. But the downloads of the installers for Windows are an order of magnitude higher than the next set of downloads which is the installers for Mac and some of the source distributions available there. So it's there's clearly a huge interest and we're talking something like 5 to 8000 downloads a day roughly.
Again, I don't have precise stats but that's ballpark area. There's there's a number of thousands of downloads a day of each distribution. So it's very popular on Windows. Most people, I believe, are coming to python.org to get it and the the the breakdown within versions of Python, I think, is heavily biased by the first download button that you see on the site. So when you go to the python.org site, certainly on Windows, I assume it changes on other platforms, and hit download, it presents you with download Python 35, download Python 27 and that link takes you straight to a download. So that's obviously going to be the most popular download because it's the fewest clicks to get there. I've thought about experimenting with getting that changed to point to something else and see how things go, but really there's there's no point in that. Python's a 20 meg download which is not that big these days for most people. If you click through, there are smaller downloads. There's a web installer that will let you deselect options before you've even downloaded them, which if you're not interested in Tickle, if you're not interested in the documentation, if you're not interested in the test suite, you can get it down to a 7 meg download for the entire thing. And people are using that as well, which is great to see because I put a lot of work into making that actually work.
But but yeah. As for other stats that I'd like to see, I I can probably already get them. I just don't actually know what I should be interested in. It's not really fair to compare downloads for Windows to other platforms because other platforms have other ways of getting it. We know that we know and and again, Windows users are quite often going to Continuum Analytics or Nthorpe to download their distributions. There's they get a lot of Windows downloads as well, certainly more than their other platforms. And but but again, for Mac users, a lot of them have Python already. For Linux users, they've all got Python already. No 1 needs to come to Python dot org to download it. So there's no sense comparing those statistics.
The 1 the 1 comparison that I do like, and I have the data to analyze this, I just haven't actually done it, is how quickly old versions of Python are fading from the downloads and certainly we're still seeing downloads every month of Python 2.0, Python 2.1 which are still there and apparently people are still clicking on them. So I'm interested in the trends of how quickly the old versions die off which they certainly seem to be. Definitely the latest versions are the ones that are dominating. Including Python 3 6 is very popular as well.
[00:32:06] Unknown:
The degree to which, you know, sort of like government contracts and other environments where really seriously legacy software is used should never be underestimated. Right? These contracts take decades sometimes to age out. So it's it's kinda not surprising when you take that into account.
[00:32:25] Unknown:
No. Absolutely. Though though there's certainly a number of critical infrastructure projects out there that are using very old versions of Python, which is a little bit The The the IDEs are are a fairly similar cluster of tools. There's there's quite a big gap between editors and editors with plugins from there's there's quite a big gap between that sort of cluster of tooling and the IDE's. The IDE's, which which include all of those tend to be, aiming to be the 1 stop shop for once you get into the IDE, you can do everything there. There's never a need to alt tab out, or alt tab or equivalent shortcut to change to another window.
We consider that every time you have to go to a normal console to get something done, we failed. And and that's certainly the view that all the IDEs take. You should not have to open a regular console, type Python Myscript dash anything to make it work. If so, then the IDE has failed. So that's that's kind of the aim that all the IDEs have. And and for for the most part, we all can't we all tend to get there. So how they compare against each other tends to be in the very specific narrow cases. So there's certain areas where, Visual Studio is going to be ahead of PyCharm. There's certain areas where PyCharm is definitely ahead of Visual Studio, and and the other IDEs are going to be better in their little areas. Spyder is 1 that has quite a quite an interesting niche right now where they're they're doing a lot of cool data science stuff that the other IDEs simply aren't doing.
And so if that's an area that you need, if that's the tooling that you need, that's gonna be an a good IDE for you. Visual Studio's debugging support is often cited as the reason people are using it. I think we're the only IDE out there that has mixed Python and C debugging. GDB can certainly do it. We have it in a graphical form which lets you set visually set breakpoints in the editor and say this is my C file, I want a breakpoint here, this is my Python file, I want a breakpoint here. And I can start running and it will stop at each of those breakpoints. I can inspect the native variables, I can inspect the Python variables, and have a really sort of mixed together view of the world there, which is something that we keep hearing especially from a lot of big companies that that is the reason they're using Visual Studio because they have that functionality and no 1 else is offering it to them. Otherwise, feature wise, we're all largely around the same area with, you know, the little little tweaks within our margins. None none of the IDEs are as simple as text editor with plugins, but I don't think there's many areas where any really really jump out as as especially unique.
[00:35:21] Unknown:
That's interesting. I was thinking that, 1 of the things I always value about Visual Studio, admittedly, I haven't used it extensively in in quite a while is, the IntelliSense syntax. To call it highlighting is is damning it with faint praise syntax, tools, basically, that allow you to do some really interesting rich introspection things with your code. And so I was wondering if, perhaps, Visual Studio had anything interesting to bring to bear, for Python with regards to that.
[00:35:54] Unknown:
Yeah. Absolutely. The latest 2 version well, so the most recent version of Visual Studio 2015 has, has the option to install our Python support when you're installing it. Earlier versions, we've been a separate extension. Newer versions of Visual Studio, which there are a couple of previews out now, have Python even more front and center. It's a very easy option to select and that brings in a lot of that really awesome editor support. So you'll get nice syntax highlighting for all of your code, which which includes recognizing things like, oh, you this is a class that you've aliased or that you've passed into a function and we know it's a class still. So we'll highlight it differently even though you did an assignment, even though you put it into a dictionary and then put it out later.
There's there's a lot of clever work flowing that information through that lets us visually display how your code is working, what values go where, and and that also helps when it gets to completions. So when you type a word and hit a dot after it again, if you're at the basic text editor with plugins level, maybe you'll get some information if it's really obvious what the value is or maybe you'll just get a list of things that you've ever typed before, which is going to be a large list and not entirely relevant. With the code analysis that we run-in Visual Studio entirely statically without executing any of your code, we can very often give you exactly the type of that variable, exactly what type of values it has in it, and show you a list. We can give you a list and say these are all the members of these 3 classes that this variable could be and optionally here's the intersection of those members.
So these are the 3 functions that exist on all of them. These are the only ones you should call. We can display that to you and once you get into each of those, now we can tell you what parameters they have. Here's the docstring and bring all that information from where it's originally written and put it right in front of you where you're currently typing.
[00:37:53] Unknown:
And is it able to leverage any of the features of the typing module or tools like MyPy for being able to surface some of the information that they expose?
[00:38:04] Unknown:
Currently, no. And that's simply because we haven't gotten to supporting that yet. It's certainly something that we've put a lot of thought into. It's an interesting thing to approach. So our our model for figuring out all of the the types predates the typing module, predates function annotations and typing annotations. So we have a model that works and the the function annotations is a little bit of a little bit of a spanner in the works because it puts us in a position where we now have known information but we also know the information that it's meant to be telling us. So if someone puts a type annotation on a function parameter and we know that that function parameter could be something else, we could you know, how do we deal with that? If someone says this must be an integer, we know that it's being used as a string everywhere.
Who's right? Do we assume that, oh, okay, you said integer, guess it's an integer, everyone else is wrong or, you know, this is called a 100 times already and it's a string every time and now you're saying it's an integer. You're wrong. So it's something that we've put a lot of thought into and haven't yet settled on and implemented a method to take advantage of those types. I suspect when we do, it will be along the same lines of how PyCharm does it. If you specify type annotation, that is correct and everyone else is wrong.
[00:39:26] Unknown:
Yeah. Just being able to surface the fact that there is a conflict would be useful, I think, regardless of what the end decision is as to which set of information should be correct. Because just knowing that there's a conflict will provide enough context to the developer to know that something has to be done, and then they can at least make the decision as to which direction they wanna go with it. So are there any aspects of developing Python on and or for Windows that would be compelling enough to convince users on other platforms to make the switch, whether that's platform features or tools that are available, for instance, in Visual Studio?
[00:40:02] Unknown:
So, yeah, Visual Studio is definitely convincing to a lot of people. There's a number of comments on Reddit and Twitter that members of my team have screenshotted and repeatedly send up our management things like I reinstalled Windows just so I could use this. So so we're definitely seeing some of that. I think the most compelling feature coming up at the moment for people that are looking to switch, and especially people who are currently on, say, macOS because they like that there's a terminal that they're familiar with and there's good tools and and a good graphical interface, and it's gonna be compatible with a lot of other devices, but it's also got that sort of basic POSIX layer is in the next update coming for Windows 10. I believe it's coming out this month. I don't actually remember the exact date. I don't know if the exact date has been announced publicly either, is a Linux subsystem, which there's been some talk about. It was announced a little while back. There is now the ability in Windows to run a complete Linux kernel with within Windows. So it's not a virtual machine. It is it it's actually a device driver inside the kernel that is running a full Linux kernel. So you can run a complete Ubuntu session on there or potentially any distro that you want to be working with and not have to reboot the machine. It's not dual booting. You don't have switch into an isolated virtual machine. It's right there. It's integrated with the Windows file systems that you have so you can directly share files. Over time, the tooling is definitely going to get better for that.
So my my hope my my sort of pie in the sky dream, which as soon as I can, I'm going to do, is the ability to be working in Visual Studio, working on Python code, developing it, and then when you hit f 5 to start debugging, it will run on Linux in this subsystem and be letting you debug code that's running literally on Linux without having to switch to another machine, without needing more hardware, without needing separate isolation magic with no extra steps. It's really Linux on Windows right there just so developers can be comfortable with the tools that they have, with the target platform that they're working on, so that packages can be installed. And and yes, when you're running this Linux subsystem, you have a c compiler because GCC is there.
So you can certainly install whatever packages you need on that. But that's certainly 1 of the most compelling things, I think, for developers that's coming out soon.
[00:42:26] Unknown:
Yeah. I think that once that actually hits general release that a lot of people are going to make the jump back to Windows because of the fact that they'll have access to everything in both platforms. The implementation I heard briefly referenced as sort of the inverse of wine though I don't know how accurate that
[00:42:43] Unknown:
is. Yeah. So the implementation is essentially a kernel shim. So anywhere that normal Linux distro would make a syscall, that gets intercepted and translated by by a device driver that's in in Windows and transformed into the the native syscall behavior. So it's it's a shim at very low level. I believe they said there's only 300 and something syscalls they needed to implement, which is a much lower number than something like wine. Wine is implementing a layer at the user layer. So it's intercepting every single call into the operating system and redirecting that. This is a much lower level. So you get a genuine Linux. Not not the genuine Linux kernel, but everything above that is exactly the same. So you can install a full distro and you'll get every tool, exact behavior, complete binary compatibility because everything is still running on the system it thinks it is. It's just once it hits the syscall layer, it's being transformed into the equivalent of a Windows system call rather than going through the Sys call it would normally go through.
[00:43:47] Unknown:
I'd be interested to see what desktop environments end up looking like in that context.
[00:43:52] Unknown:
I'm interested as well. I don't think you can put a desktop shortcut directly into a a Linux program yet, but I'm sure that's on its way.
[00:44:01] Unknown:
So could you give our listeners a whirlwind tour of the underlying implementation of Python tools for Visual Studio? How does Visual Studio provide in-depth introspection for your Python code?
[00:44:11] Unknown:
This is gonna be quite the whirlwind. We have a a 1 hour recording internally for new members of our team describing this. Let me try and give you the 32nd version. Essentially, we have an implementation of Python which we've rewritten that executes your code but instead of using values, it uses types. So anywhere you have a variable that you assign a value to, we remember the type. And then we basically try and flood fill that throughout all the code that it reaches. So anywhere there's a function call, passing that as a parameter will flow the type through and just try and keep flowing it through until we hit the the boundary of where nothing else passes that variable on and say, okay. Now we know what the types are up to here. The that is actually a ridiculously huge problem, as I'm sure you can imagine, normal code has loops.
A lot of normal code has infinite loops, and there's conditions there that that would break out of that that we simply can't assume that we can't understand when we're only dealing with the types. So there's a lot of the magic is knowing when to stop. And typically, we do have to stop early. So typically, you won't get a 100% perfect information because to do so would take forever and you'd definitely run out of memory. You'd probably run out of, you know, the sun before it's a 100% complete. But we do have the the heuristics are are, I guess, the secret source for when is this information good enough that we can stop trying to evaluate it from here. But essentially, we execute the we simulate executing the code and instead of passing around the values, we just pass around the types and end up with a model of the code where each variable has its set of types associated with it. As for the rest of Visual Studio, so Visual Studio is this amazing platform.
It's very much like an operating system. It provides a huge set of services, a whole lot of generic user interfaces. Everything is very extensible. There's 1 team that we call platform team that does the core piece and every other part of Visual Studio, every other language, every other extension, every basically, every feature up to and including the text editor is an extension on top of this core base. So it's a hugely extensible platform which makes it really fun to develop for in a lot of ways because if you see that someone has done something so if you see that someone has made the editor do something clever, there's a way for you to do it as well. If you see that someone has added a window with certain functionality, there is also a way for you to do that. There's nothing that Visual Studio does that can't be reproduced by someone else, that you can't reproduce in your own extension, which is really which is really cool. Though it's not necessarily easy because unlike most operating systems, it's frankly not as well documented and it's not anywhere near as stable as an operating system would be. So there there's challenges either way but it is it's definitely a platform to develop on top of. It's very amenable to that. There is plenty of good documentation examples.
And and for us working internally, we get to look at some of the internal examples. And then when we take our code and make it open source, that helps other people come and see how did they do this. Oh, now I can reproduce this in my in my own code, which is everything from supporting project systems, extending the debugger, extending the profiler, extending publishing support, if you want to publish code to another machine or to the cloud. All the IntelliSense tricks that we do, all the completions, tool tips, it's all extensible.
And the fact that our code gets to be an example of that to the rest of the world is really cool.
[00:47:37] Unknown:
Absolutely. I guess 1 thing that I was wondering about when you were discussing previously the sort of internals of PTVS with the the typed version of the person's code. Where is the line in that implementation between the c sharp and the Python?
[00:47:53] Unknown:
So the original implementation of that was in Python. That got translated to c sharp. In large part, I believe it was because of the way garbage collection works. Python's garbage collection was not efficient enough for the amount of sort of dangling references we were dealing with. This this predates my time on the project. So I it's always been written in c sharp as far as I'm aware. Now that I'm controlling the direction a bit more strongly, I'm trying to move what makes sense into Python code and sort of minimize adding new c sharp code, in large part because I wanna, you know, encourage contributors. And when we can say 90% of the debugger is implemented in Python code, if you found a bug, you can probably fix it and then you can send a poor request which we're allowed to accept, then that's really cool. But certainly, the code analysis is largely c sharp for performance. So there's sections there that are deliberately marked as unsafe so that they'll execute quicker. There's parts that rely heavily on tag and sweep garbage collection rather than cycle detection, simply because there are thousands and thousands of cycles all the time. And if it was written in Python, the cycle detector would spend all its time trying to detect them and probably not even be able to break up many of them. So, yeah, we we went to C Sharp for that.
[00:49:05] Unknown:
So are there any questions that we didn't ask that you think we should have or any other topics you'd like to bring up before we start to close out the show? No. I think that's fine. So for anybody who wants to follow you and keep up to date with what you're doing, what would be the best way for them to do that?
[00:49:20] Unknown:
Probably the best way is to follow me on Twitter. So my name there is Zouba, z o 0 b a. Don't don't ask how I came by that name. I was 13. I keyboard mashed and it stuck around ever since then. And and we so everything that that my team does goes up on GitHub. So github dotcom/microsoft. You'll find the organization. And if you search for Python in that, then you'll find a lot of my team's work. There's also github.com/azure. And again, if you type Python, you'll find a lot of that. And typically, I go by my real name everywhere else. So steve.dauer is going to be me on the Python bug tracker and and forums like that. Great. So with that, we'll move us on into the PIX.
[00:50:00] Unknown:
I've got 2 PIX this week. My first 1 is a tool called kdiff 3. That's what I use as my merge tool when I'm dealing with Git Conflicts, and that's the best 1 that I've come across so far. I've tried a few different ones, but for visual merge tools, it does a really good job with through a merges. Has some it exposes easy ways to set the key bindings that you want. I'm a user of e max for, you know, a few years now, So my fingers always want to use emacs key bindings, so I set it to use control n and control p to jump to the next and previous diffs. And it's easier to automate than some of the other ones I've tried to use. And my next pick is something called the Spyderco Triangle Sharp Maker. It's a sharpener for knives and scissors. And, basically, anything that you have that has an edge, you can use this to sharpen it, and it just does a really good job. It makes it easy to add an edge to your kitchen knives or a pocket knife. It even handles serrations, does scissors. So definitely recommend taking a look at that for your kitchen or anywhere else that you have need of bladed instruments. And with that, I'll pass it to you, Chris. Thanks, Tobias.
[00:51:06] Unknown:
My 1 pick today is audiobooks. Specifically, I'll use Audible because that's the that's the vendor that I'm using. I I love reading fiction in actual print form, but I, for some reason, have realized that nonfiction, trying to read it in print form puts me to sleep. But audiobooks, I can cruise through with no problem at all. So, I'm really enjoying that. That's all I have for picks this week. Steve, what do you have for us for picks?
[00:51:31] Unknown:
Yeah. So I went ahead and picked a couple. The first is gonna sound like a product placement, and it's not because no one's paying me for this. I picked up the SanDisk Extreme Portable SSD Hard Drive a while back and almost immediately went back and got another 1. This thing is, like, 500 gigabytes SSD USB 3, so I get over a 100 megasecond transfer. And and the thing is a quarter the size of my phone. It's like I put in my pocket and I can forget that it's in there. So that that is really cool. If you if like me, you end up transferring huge amounts of files around physically rather than sending them all up on the cloud and downloading them elsewhere, that's really great. That's Sandisk has not paid me for that. If you wanna send me more of the hard drives, don't bother. I have enough. But that's that's a really cool hard drive. My my other 2 picks are both kind of light entertainment. Most of the time, once I get, you know, out of work and just wanna hop on the Internet, it's it's sort of relaxing time, and just light entertainment. So Saturday Morning Breakfast Cereal is a webcomic that I've been reading for years. This stuff is hilarious. So the website is smbchyphencomicsdot com. The guy who writes at Zach Weenor Smith is very smart man, knows a lot of stuff and is able to turn it all into really of often intelligent jokes. Occasionally butt jokes but often intelligent jokes to do everything from philosophy, technology, computer science, physics. Very entertaining.
Always get a good laugh out of it. And and my last pick is my favorite YouTube channel right now which is called Random Encounters. The channel name is youtube.com/randomencountersent for entertainment, e n t. And these guys make gaming musicals. So they they aim to make a musical each month, about a video game or based in a video game. They tend to be short. So it's like 1 song out of a musical but they've got some really great ones based on Pokemon, really great Zelda ones, Team Fortress 2, 5 Nights at Freddy is a recent game that they finished doing as short series of musicals on. Very entertaining stuff, very well performed. A lot of cool people from around the the YouTube coming on and singing and performing with them. It's great light entertainment. I really enjoy it.
[00:53:30] Unknown:
Well, thank you very much for taking the time out of your day to join us and tell us all more about Python on Windows and some of the different tooling around that. Definitely excited to see what happens once the, Linux subsystem becomes generally available. I'm sure I'll probably end up experimenting with it a bit. I appreciate all the work you've done to simplify installation of Python. It's made it easier for me to get it up and running to work on it with my kids. Alright. Thanks for having me on. Thank you. Bye bye.
Hello, and welcome to podcast. Init, the podcast about Python and the people who make it great. I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. Linode is sponsoring us this week. Check them out at linode.com/podcast tonight and get a $20 credit to try out their fast and reliable Linux virtual servers for your next project. We are also sponsored by Sentry this week. Stop hoping your users will report bugs. Sentry's real time tracking gives you insight into production deployments and information to reproduce and fix crashes. Check them out at get sentry.com and use the code podcast in it at sign up to get a $50 credit on your account. To help other people find the show, you can leave a review on Itunes or Google Play Music and tell your friends and coworkers. You can also join our community at discourse.pythonpodcast.com for your opportunity to find out about upcoming guests, suggest questions, and propose show ideas.
Your host as usual are Tobias, Macy, and Chris Patti. And today, we're interviewing Steve Dauer about Python on Windows. So, Steve, could you please introduce yourself?
[00:01:09] Unknown:
Yeah. Hi. My name is Steve. As you can probably tell straight away, I'm Australian. I've moved to the US, about 4 or 5 years ago now, to start work at Microsoft. So I've been working there for that time on a lot of the the Python tooling and Python libraries and stuff that you will have seen coming out of Microsoft recently. And in that time, I also got involved, directly working on CPython. So I've been a core contributor for a couple of years there, maintaining and just really trying to support the Windows platform.
[00:01:42] Unknown:
So how did you get introduced to Python?
[00:01:45] Unknown:
It was initially a, summer vacation job that I had in between, in between studies. So I had a few months off and had a friend, set me up with a job which he knew about. And this job was designing medical devices. They're actually a startup. They're in early stages of, setting up demonstrable prototypes of their what were they doing? They had these little cards that were sort of gonna be prefilled with all the the medical testing stuff you needed and you'd be able to inject, a sample into this card, stick it in device and it would pump everything around and eventually 1 part of it would change color and we could detect the color change and give you the results. So the idea was that you send, you know, an HIV test. You could send a 1, 000 cards, each of which is an individual HIV test and someone who's not a medical professional could do it. So I was working there for a couple of months.
They were building a prototype system which was being automated with Python. So I had to learn Python at that point, this is about 7 years ago now. I learned Python to automate this system and build that, and from there went straight into a PhD and on the first day my advisor comes to me and says, hey, I have all of this code that you can use but it's in Python. And I just went, oh, great. I know that now. So let's go.
[00:03:09] Unknown:
So you're currently the release manager for Python on Windows. How did you end up with that responsibility?
[00:03:15] Unknown:
Basically, I volunteered and everyone said, hooray, we have someone who wants to work on Windows. So so that came out of going to Python which, so all my work at Microsoft has been around Python. We've been able to get to go to PyCon for a number of years and and sponsor the conference and have a booth there and just get to meet a lot of people. Couple of years back, there was a big there was a lot of discussion going on about packaging, and I went to 1 of the panels on that. And at that panel, Nick Coghlan, who's very heavily involved in in the packaging side of things for Python, basically called for more help, called for more people to come join the distutilsigmailing list, and just be available to discuss and and figure out some of the problems that we were facing at that time. So I volunteered for that and said, you know, I'm familiar with Windows. It sounds like I'm more familiar than a lot of the people who are in the room at the moment.
I'm happy to come and help out. Fast forward a year to the next PyCon. I've been doing that. And essentially I did the same thing, and said, I've noticed the installer has a few issues. Is anyone gonna mind if I come in and and, you know, submit a few patches or, you know, update it a bit? And everyone was sort of like, yeah. Yeah. Sure. Go ahead. Talk to the guy who's currently maintaining it. The next day, the guy who was currently maintaining it sends an email saying, I think I'm done. If you can find someone to take over for good that would be great because it's time for me to step away. And so I stuck my hand up again and got the job.
And since then, I have rewritten the installer completely. I've rewritten the build completely on Windows and gone back and fixed some of the older, more obscure bugs that have just not been fixed in that time.
[00:05:08] Unknown:
Yeah. I know that for quite a while, the overall experience of getting Python installed on Windows and being able to set up your path and environment was rather difficult and fraught with peril for certain use cases. And from what you're saying, doing the complete rewrite of the installer has greatly improved that experience. I know that I recently installed Windows on a laptop for my son to start teaching him Python. And it was much easier to do that recently than it was when I first started trying to use Python on Windows about 6 years ago. So I'm wondering if you can give a bit of background on some of these some of the improvements that you made there and some of the issues that you came across in the process.
[00:05:48] Unknown:
Yeah. So 1 of the big differences between, I I guess, the Linux world and the Windows world is there's a really big cultural difference between hacking software, I guess. So on on Linux, basically all software, there's there's a few sort of device driver exceptions. But essentially all software comes down as source code which you'll then build on your own machine, and have a result that fits perfectly. Windows, the culture has always been you download the binaries or you get a CD or even a floppy disk at 1 point with the binaries on and you can just run them. And maybe there's an installer, but all that's really doing is copying them onto the disk. There's no compilation step taking place. Because of that, developer tools almost always come with a compiler but not everyone gets a developer tool. So 1 of the big issues that Windows has been facing for a long time is when you get Python, you aren't actually getting things like a c plus plus compiler or a c compiler even and as a result, when you then need 1 because you're in the developer tools world now, you need a c compiler, you don't have it and so a huge huge majority, I would estimate, of people with Python on their machine don't have a c compiler and you just have this disjoint culture. So so the Linux package developers are saying I can publish source code. When people install the source code, it will compile and they'll be good to go. And that's really baked into Python through the distutils library and, setup tools. I just have been assuming that for so long that installing packages, well, of course, you'll have a c compiler because everyone has a c compiler and and just Windows has not had that. So people will often hit packages that can't be installed because they're missing a compiler.
There's a number of other things. The the sort of culture of living at the terminal, living at the console is not the same on Windows. It's a point and click environment and has been for, I guess, 20 years now. More than 20 years. It's you work with a mouse. Occasionally, like, keyboard is for input into a program, not for starting the program itself. So things like environment variables, things like the current working directory don't have as much of an importance on Windows as they do on other platforms. As a result, things like Python that heavily rely on those, global settings don't necessarily behave the same way, and and don't work as as well or as smoothly or cleanly as they do on a platform where environment variables are front and center, everyone uses them all the time, everyone knows how to set them, and knows how to adjust things like the path, which on Windows doesn't make such a big deal. So as for fixing that, honestly, I haven't really done much to fix that.
The biggest change the installer has made recently, so since 3.5 when I rewrote it, is being able to do a proper installation for a user who is not an administrator on their machine. Because previously that was completely impossible. You had to be an administrator, you had to be able to change system level files to install Python on your machine, and this was excluding a lot of students who had machines issued by their by their schools. A lot of research departments were given computers by their schools and not given permissions in a lot of cases. That was that was my situation until I replaced the machine my university gave me with my own 1. A lot of businesses cannot install, like, won't let their employees install things on their own machines because that's I I get it's the policy, it's the culture around it, and it's how it's used. So being able to make the Python installer still install in those situations and give someone a working copy of Python is I think the biggest change that that I've made so far for 3.5.
And frankly, it makes a lot of terminal usage harder to do it that way. It doesn't really make it more like Linux. It's making the Python installation more like a Windows program should be. And I've also made a few sort of more technical changes within the core for software that wants to embed Python, which is a fairly common thing. There's a lot of software out there on Windows that you install it through a normal installer and in the background it's running Python. It's got the Python DLL. It's loading code and running it. And there's just a few changes in 3.5 that should make that easier for the developers who want to do that and distribute Python as a library of their application on Windows.
[00:10:28] Unknown:
Just being able to install Python to a user's account rather than requiring the admin privileges, like you said, greatly increases the overall reach and availability of Python for people in a large number of situations. And also, you know, maybe kids who have parents who are savvy enough to be able to lock down the machine and not give them complete, you know, admin control over everything. So it definitely broadens the range of people who can now potentially start experimenting with Python whereas previously it just would have been out of the question.
[00:10:58] Unknown:
And just to add to that, since the Python installer was last majorly updated like this, Windows changed the model so that by default you're not an administrator. So you don't even have to be that savvy these days to lock down your child's account on a Windows machine. By default it's locked down and that is just such a change. Like it's it's a big change in culture. It upset a lot of people when it happened, because it changed everything. But that is fundamentally new since the installer was last done and so it needed to be resolved.
And
[00:11:33] Unknown:
going back to what you're saying earlier too about the fact that most Windows machines don't have a c compiler available for being able to install packages with c extensions. Some of the recent updates to how those packages are distributed, namely wheels, has made that a lot easier as well. I don't know if you can speak to that a bit.
[00:11:52] Unknown:
Yeah. So wheels were just coming about, at the that they were, I guess, gaining acceptance at that point where I started joining in on Digital SIG. They've been under development for a long time. A lot of people trying to solve this problem, and it's not just a Windows problem even. It's a lot of places, a lot of people want to build once on Linux and keep reusing the build because they know they have a compatible machine and so they can use wheels and they do use wheels to set up maybe an internal package server, and then maybe on a 1000 machines instead of installing and compiling a package over and over again, they'll pull down that wheel which is essentially just a zip file with all the pre built packages inside it.
They'll pull that down and install it, still using the same pip install command, and it's a lot quicker than running it through the compiler would be. So it's it's been a it's been a problem that's being worked on for a long time, but it's but it's hit a point now where I don't I don't have numbers handy. I'm sure someone does, or can get them, but so many packages on the Python package index are coming with wheels now, at least for Windows. There's the option now to put up highly compatible Linux packages that will work on most distributions that are out there, and certainly packages for Mac, can be put up there which just means that installing packages doesn't require a compiler in so many cases now that you can almost forget that you don't have 1. Simultaneously, 1 of the great things about working at Microsoft is I get to go and chat to the people who manage things like the compilers And so I can go and sit in their offices and chat with them about here's why we should make it easier for people to get access to the compilers.
So in the same time that that's been going on, I've organized putting out a copy of the old compiler that people need for Python 27, and so that's available from a link which we'll get in the show notes, and I've also been able to help encourage the compiler team to put out a much simpler bundle of the most recent compiler, which is called the Visual C plus plus build tools 2015, I believe. Again, I'll give you a link to the show notes, which is not a full Visual Studio install. It's not the same licensing things. We want people to have the C plus plus compilers because that means they're building things for Windows that's good for the ecosystem.
So we want to make the compilers easy to get and we've been able to improve that. So now it's actually not that hard to get the compilers you need to build packages on Windows, but with so many developers having or publishing wheels for their packages, it's less necessary.
[00:14:38] Unknown:
What an incredible paradigm shift that is. Just wanna sort of take a moment to appreciate that. I mean, for so long, those of us who've been around the industry for for a while remember that, you know, vendors were so incredibly guarded about releasing compiler technology of any kind, and and this idea that end users should should should be, empowered to build their own binaries was almost, heretical, you know. So it's it's it's a really really wonderful change that's been been happening, especially at Microsoft and other vendors too. But it's it's most most heartening to see happening there because Microsoft used to be kind of 1 of the seats of binary culture. Right?
[00:15:28] Unknown:
Yeah. Oh, absolute. Absolute. And I I would say that it still is. I mean, it's it's certainly not moved to the point where regular end users are expected to be able to compile stuff on their machine. I I again, it's very easy to to have a small view of who your end users are. The the range of people that are capable of getting machine running Windows versus 1 getting machine running even Ubuntu, but some of the more obscure Linux distros that are just a little harder to get set up and running and keep running, there's a huge huge base of Windows users who cannot and should not be compiling anything at all. They should not have the c compiler on their machine because they need those few gigabytes of space for photos, for videos, for music, and they're simply not interested in building things. So while there is a shift for the enthusiasts and and it and I'll call them the amateur developers but it's it's anyone who wants to build things for the platform without having the backing of a large company behind them. Things things like Visual Studio Community Edition are specifically for supporting those and you can see that from the license. It's a full copy of Visual Studio, the same thing we charge 1, 000 of dollars for, to companies.
But if you're not a big company or if you're working on open source software, you can have it for free. There's no restrictions. It is simply the case that there are now, there's there's a big movement of amateur developers out there who are not a company making 1, 000, 000 of dollars every year off their software and we want to encourage and support them as well.
[00:17:09] Unknown:
And given that a large percentage of overall computer users are still on Windows, having a good story for them to be able to get started with Python is definitely important. And we mentioned a few improvements that have already been made, wondering what are some the areas where the current situation still needs to be improved, and are there any areas that would be, ripe for listeners of the show to potentially contribute?
[00:17:34] Unknown:
So I am still very 1 of 1 of the things that I am sort of constantly working on in the background as part of my day job is speaking to the people we need to speak to about how do we make, you know, opening a command prompt on a brand new install of Windows and typing Python do something useful. Because right now that gives you an error message. See, on the operating system in in popular use, even unpopular use, that's gonna do that, just give you a flat out error for, oh, you type Python, that's not a real thing. So I'm certainly working on trying to come up with a better approach to that, a better response to that, but as far as other things that Python needs on Windows right now, migrate to Python 35, please.
It's a considerably better experience on Windows now than 27 is and even 3334. So 35 is gonna be much better. 36 hopefully will be better again. And and the other thing is encourage package developers to produce wheels and if that means you have you have a Windows machine and you go to them and say, I am happy to build the wheel for you on my machine. I'll run the tests and give it to you so you can publish it if that is simply being there to test and give that response. And again, it's the enthusiast developers who have a C compiler on their machine who can do this and start contributing to a package just to get a wheel up there so that all the other developers who, simply using Python and want to be able to use that package are able to install it and just use it.
[00:19:11] Unknown:
Yeah. Having build hardware for platforms that the developers aren't generally working on is usually a bit of a pain point for making a package generally available to other platforms. So I know that, for instance, the Kivy folks have had some issues occasionally with getting builds available for OSX and for Windows because most of them are developing on Linux, for instance. So I don't know if that's an area for people to potentially contribute having Windows build servers. Or I know that with the advent of cloud computing, that's sometimes easier to do where they can just rent a Windows instance for a little while to build their packages and then distribute them.
[00:19:50] Unknown:
Yeah. That's certainly a good option that plenty of people are taking these days. There's a lot of work to be done there just as far as making that simpler for people to do which is, again, also something that is kind of constantly buzzing around in my head. How how can we make this really easy for people who don't use Windows regularly, who aren't familiar with it, but need to be able to build something, need to be able to diagnose when the build fails, why the build fails, do that testing, do that analysis, which again, if you jump into an a native debugger on Windows, you're going to see very different things than a native debugger on Linux. And how do we make that entire workflow easy enough that we can legitimately start going to these projects and say, plug into this and you'll get your Windows build and here's how to deal with things that don't work.
[00:20:35] Unknown:
Another question that comes up is you're saying how the way that Python is installed on Windows and referenced is a little bit different than how it's set up in Linux, you know, in terms of path and environment variables. And I'm wondering if that poses any difficulties for people who are trying to build cross platform projects.
[00:20:53] Unknown:
My hope is that it would not. So it's largely and this is this is a great opportunity for me to talk about my my first PEP, which was ever accepted. I recently had PEP 514, was approved and is is now an active PEP, which is about configuring the registry on Windows for locating Python installed. So 1 of the things that we discovered, through some of the telemetry that Visual Studio has is that most of our users on Windows have 2 or more copies of Python installed, which is totally legitimate thing to do. I have about 16 copies of Python installed on most of my dev machines because that's what I do. But we definitely found that the majority of our users had 2 or more, and locating each of those by simply typing python at a command prompt is not actually useful. It's you're gonna get 1, you're gonna get the other. It will be consistent on your machine. It may be different on someone else's machine who installed things the same way. There's just no reliable way of doing it, relying entirely on path. So what PIP 514 defines is the set of registry entries. So the Windows registry, for those who aren't familiar, is essentially a hierarchical configuration file.
It's, it has a global per machine section and a per user section that are independent, can be read independently. So Python has always registered itself in either the the per user 1, if it's installed per user or the the per machine 1, if it's installed for everyone. Which means that anyone on that machine can look in the registry and say, oh, where is Python 35 installed? Here it is. Now I can run it. And again, because most Windows users are not going to the console and just typing Python, they'll be going through some shortcut, they'll be going through some other program. It's able to use the registry to find it and execute the right 1. Where that ran into problems is if you have collisions in that, you can only have 1 value under a certain key. So if that key says this is Python 27 and you install, somehow 5 different versions of Python 27, which is actually entirely feasible, then you get collisions, you can't find all of them. PEP 514 is designed to disambiguate that, to make it possible to have each of those, identified separately. So it's a different model from what you see on Linux.
I have seen pyenv which will do, some clever path searching on Linux and give you that free choice of I have all of these different ones installed and I wanna choose which 1 to use right now. That doesn't fit the Windows model. The Windows model is this is the 1 that I want, look it up, run it explicitly, and not change the the sort of ambient state of which 1 I'm using right now. But that should make things no harder for package developers in theory, and with proper tooling around it can make it easier because it makes it a lot more feasible to have many copies of Python installed on the 1 machine and be able to accurately choose which 1 you want to use right now.
[00:23:54] Unknown:
And back on the user side, how does that tie into things like virtualenv?
[00:23:59] Unknown:
So virtualenv, could potentially take advantage of that information that's there for letting users choose which Python to work from. Typically, most people are going to simply use run virtualenv out of the version that they want to inherit from, but you can specify any version of Python to base it on. So you can type virtualenv, I believe it's dashp, and give the full path to the Python executable you want to base it from. So it doesn't matter if it's not the same version, it will go off and run against that and give you that environment based on a different version of Python. So that could potentially make it so you don't need to know the full path of where it's installed.
And given that the defaults default install path changed recently from a very short CPython 35 or CPython 34, Python dot exe to something more like c users, my username, app data, local programs, Python 3, Python Python 35, Python dot exe, not having to remember that full path could could be very handy. So that's the kind of thing that that information can enable and certainly when you get into GUI applications like the IDEs, Python, Python Tools for Visual Studio, PyCharm, Wingware, they can just display a graphical list of everything that's registered on your machine and you can simply click the 1 that you want right now and you'll get it.
[00:25:25] Unknown:
So what is currently the most difficult part of building a distribution of Python for Windows and has that gotten easier in recent years?
[00:25:34] Unknown:
I believe it's gotten easier. I I am a compulsive automator, so when I hit a problem that I have to do more than once, I will find a way to script it. So I have a a collection of scripts, some floating around, some actually checked into the repository for CPython that that will do the build. So I have a script and a big bundle of files that will set up my build machines for me. And there's there's no there's no magic there. It's just simply I can't redistribute an entire copy of Visual Studio. So that's my private script that will I can pull up a clean VM and have everything installed very quickly. But then actually building the distribution of Python for Windows is it's 1 batch file. So it's 1 1 shell script that I can run that will do everything, with a few with only a few options and, again, I optimize them for my use cases. It will code sign all the binaries. It will run installation tests. It will do the 32 bit, the 64 bit versions of Python and I can kick that off and walk away for an hour and come back and it's all done. And then I have 1 more batch file to upload the official release. Those are both checked in and it's considerably easier now, I believe, than it used to be. But again, that's it's my automation. I understand it. So far, I'm aware of 1 other group that's made use of it and has made a few tweaks to the installer and run and and is using these batch files to produce their own distribution of Python 35.
So it has helped other people. I believe it's easier. The most difficult part is getting hold of the tools but even that with the c plus plus build tools which it which are sufficient for building Python, I just did it recently on a new machine, it is easier than it's ever been in the past, I believe, to to get the tools you need onto a machine. I haven't tested this explicitly, but I believe all you need is a copy of subversion and a copy of the c plus plus build tools, and everything else will be installed as part of the build process. So that should be all that's required to get started building Python on Windows is Subversion and the c plus plus build tools.
[00:27:46] Unknown:
That's definitely a big improvement because I know that at least as of a couple of years ago, being able to build the Windows distributions for Python was a bit of a black art that had a fairly low bus factor. So having it automated is definitely a good step in the right direction. When we were speaking at PyCon, you mentioned that the most frequently downloaded version of Python from the python.org site is actually the 32 bit distribution for Windows. I'm wondering if you think that that's an accurate and useful metric and wondering if there are any other statistics that you wish you could capture or improve.
[00:28:18] Unknown:
Yeah. So so we don't do a great job of capturing the statistics from there. I managed to, get access to the the raw logs from, someone there who trusts me, which is very kind of them. So so I've been able to pull those down and do some basic analysis, just to to get an idea of what people are actually downloading. So it turns out that since since I last spoke to you about that, the the downloads of Python 3 5 2 are blowing away everything else. They're about double so so per average per day is roughly double what Python 27 is getting at the moment or 2712 is getting. So as far as people downloading the latest versions of Python, there there's about 2 thirds going for Python 3 versus Python 2 which is awesome to see, really exciting to see that. There there's a few odd versions of Python 27 that's still being downloaded. 275 for some unknown reason is really popular. 273 is still up there as well. But the downloads of the installers for Windows are an order of magnitude higher than the next set of downloads which is the installers for Mac and some of the source distributions available there. So it's there's clearly a huge interest and we're talking something like 5 to 8000 downloads a day roughly.
Again, I don't have precise stats but that's ballpark area. There's there's a number of thousands of downloads a day of each distribution. So it's very popular on Windows. Most people, I believe, are coming to python.org to get it and the the the breakdown within versions of Python, I think, is heavily biased by the first download button that you see on the site. So when you go to the python.org site, certainly on Windows, I assume it changes on other platforms, and hit download, it presents you with download Python 35, download Python 27 and that link takes you straight to a download. So that's obviously going to be the most popular download because it's the fewest clicks to get there. I've thought about experimenting with getting that changed to point to something else and see how things go, but really there's there's no point in that. Python's a 20 meg download which is not that big these days for most people. If you click through, there are smaller downloads. There's a web installer that will let you deselect options before you've even downloaded them, which if you're not interested in Tickle, if you're not interested in the documentation, if you're not interested in the test suite, you can get it down to a 7 meg download for the entire thing. And people are using that as well, which is great to see because I put a lot of work into making that actually work.
But but yeah. As for other stats that I'd like to see, I I can probably already get them. I just don't actually know what I should be interested in. It's not really fair to compare downloads for Windows to other platforms because other platforms have other ways of getting it. We know that we know and and again, Windows users are quite often going to Continuum Analytics or Nthorpe to download their distributions. There's they get a lot of Windows downloads as well, certainly more than their other platforms. And but but again, for Mac users, a lot of them have Python already. For Linux users, they've all got Python already. No 1 needs to come to Python dot org to download it. So there's no sense comparing those statistics.
The 1 the 1 comparison that I do like, and I have the data to analyze this, I just haven't actually done it, is how quickly old versions of Python are fading from the downloads and certainly we're still seeing downloads every month of Python 2.0, Python 2.1 which are still there and apparently people are still clicking on them. So I'm interested in the trends of how quickly the old versions die off which they certainly seem to be. Definitely the latest versions are the ones that are dominating. Including Python 3 6 is very popular as well.
[00:32:06] Unknown:
The degree to which, you know, sort of like government contracts and other environments where really seriously legacy software is used should never be underestimated. Right? These contracts take decades sometimes to age out. So it's it's kinda not surprising when you take that into account.
[00:32:25] Unknown:
No. Absolutely. Though though there's certainly a number of critical infrastructure projects out there that are using very old versions of Python, which is a little bit The The the IDEs are are a fairly similar cluster of tools. There's there's quite a big gap between editors and editors with plugins from there's there's quite a big gap between that sort of cluster of tooling and the IDE's. The IDE's, which which include all of those tend to be, aiming to be the 1 stop shop for once you get into the IDE, you can do everything there. There's never a need to alt tab out, or alt tab or equivalent shortcut to change to another window.
We consider that every time you have to go to a normal console to get something done, we failed. And and that's certainly the view that all the IDEs take. You should not have to open a regular console, type Python Myscript dash anything to make it work. If so, then the IDE has failed. So that's that's kind of the aim that all the IDEs have. And and for for the most part, we all can't we all tend to get there. So how they compare against each other tends to be in the very specific narrow cases. So there's certain areas where, Visual Studio is going to be ahead of PyCharm. There's certain areas where PyCharm is definitely ahead of Visual Studio, and and the other IDEs are going to be better in their little areas. Spyder is 1 that has quite a quite an interesting niche right now where they're they're doing a lot of cool data science stuff that the other IDEs simply aren't doing.
And so if that's an area that you need, if that's the tooling that you need, that's gonna be an a good IDE for you. Visual Studio's debugging support is often cited as the reason people are using it. I think we're the only IDE out there that has mixed Python and C debugging. GDB can certainly do it. We have it in a graphical form which lets you set visually set breakpoints in the editor and say this is my C file, I want a breakpoint here, this is my Python file, I want a breakpoint here. And I can start running and it will stop at each of those breakpoints. I can inspect the native variables, I can inspect the Python variables, and have a really sort of mixed together view of the world there, which is something that we keep hearing especially from a lot of big companies that that is the reason they're using Visual Studio because they have that functionality and no 1 else is offering it to them. Otherwise, feature wise, we're all largely around the same area with, you know, the little little tweaks within our margins. None none of the IDEs are as simple as text editor with plugins, but I don't think there's many areas where any really really jump out as as especially unique.
[00:35:21] Unknown:
That's interesting. I was thinking that, 1 of the things I always value about Visual Studio, admittedly, I haven't used it extensively in in quite a while is, the IntelliSense syntax. To call it highlighting is is damning it with faint praise syntax, tools, basically, that allow you to do some really interesting rich introspection things with your code. And so I was wondering if, perhaps, Visual Studio had anything interesting to bring to bear, for Python with regards to that.
[00:35:54] Unknown:
Yeah. Absolutely. The latest 2 version well, so the most recent version of Visual Studio 2015 has, has the option to install our Python support when you're installing it. Earlier versions, we've been a separate extension. Newer versions of Visual Studio, which there are a couple of previews out now, have Python even more front and center. It's a very easy option to select and that brings in a lot of that really awesome editor support. So you'll get nice syntax highlighting for all of your code, which which includes recognizing things like, oh, you this is a class that you've aliased or that you've passed into a function and we know it's a class still. So we'll highlight it differently even though you did an assignment, even though you put it into a dictionary and then put it out later.
There's there's a lot of clever work flowing that information through that lets us visually display how your code is working, what values go where, and and that also helps when it gets to completions. So when you type a word and hit a dot after it again, if you're at the basic text editor with plugins level, maybe you'll get some information if it's really obvious what the value is or maybe you'll just get a list of things that you've ever typed before, which is going to be a large list and not entirely relevant. With the code analysis that we run-in Visual Studio entirely statically without executing any of your code, we can very often give you exactly the type of that variable, exactly what type of values it has in it, and show you a list. We can give you a list and say these are all the members of these 3 classes that this variable could be and optionally here's the intersection of those members.
So these are the 3 functions that exist on all of them. These are the only ones you should call. We can display that to you and once you get into each of those, now we can tell you what parameters they have. Here's the docstring and bring all that information from where it's originally written and put it right in front of you where you're currently typing.
[00:37:53] Unknown:
And is it able to leverage any of the features of the typing module or tools like MyPy for being able to surface some of the information that they expose?
[00:38:04] Unknown:
Currently, no. And that's simply because we haven't gotten to supporting that yet. It's certainly something that we've put a lot of thought into. It's an interesting thing to approach. So our our model for figuring out all of the the types predates the typing module, predates function annotations and typing annotations. So we have a model that works and the the function annotations is a little bit of a little bit of a spanner in the works because it puts us in a position where we now have known information but we also know the information that it's meant to be telling us. So if someone puts a type annotation on a function parameter and we know that that function parameter could be something else, we could you know, how do we deal with that? If someone says this must be an integer, we know that it's being used as a string everywhere.
Who's right? Do we assume that, oh, okay, you said integer, guess it's an integer, everyone else is wrong or, you know, this is called a 100 times already and it's a string every time and now you're saying it's an integer. You're wrong. So it's something that we've put a lot of thought into and haven't yet settled on and implemented a method to take advantage of those types. I suspect when we do, it will be along the same lines of how PyCharm does it. If you specify type annotation, that is correct and everyone else is wrong.
[00:39:26] Unknown:
Yeah. Just being able to surface the fact that there is a conflict would be useful, I think, regardless of what the end decision is as to which set of information should be correct. Because just knowing that there's a conflict will provide enough context to the developer to know that something has to be done, and then they can at least make the decision as to which direction they wanna go with it. So are there any aspects of developing Python on and or for Windows that would be compelling enough to convince users on other platforms to make the switch, whether that's platform features or tools that are available, for instance, in Visual Studio?
[00:40:02] Unknown:
So, yeah, Visual Studio is definitely convincing to a lot of people. There's a number of comments on Reddit and Twitter that members of my team have screenshotted and repeatedly send up our management things like I reinstalled Windows just so I could use this. So so we're definitely seeing some of that. I think the most compelling feature coming up at the moment for people that are looking to switch, and especially people who are currently on, say, macOS because they like that there's a terminal that they're familiar with and there's good tools and and a good graphical interface, and it's gonna be compatible with a lot of other devices, but it's also got that sort of basic POSIX layer is in the next update coming for Windows 10. I believe it's coming out this month. I don't actually remember the exact date. I don't know if the exact date has been announced publicly either, is a Linux subsystem, which there's been some talk about. It was announced a little while back. There is now the ability in Windows to run a complete Linux kernel with within Windows. So it's not a virtual machine. It is it it's actually a device driver inside the kernel that is running a full Linux kernel. So you can run a complete Ubuntu session on there or potentially any distro that you want to be working with and not have to reboot the machine. It's not dual booting. You don't have switch into an isolated virtual machine. It's right there. It's integrated with the Windows file systems that you have so you can directly share files. Over time, the tooling is definitely going to get better for that.
So my my hope my my sort of pie in the sky dream, which as soon as I can, I'm going to do, is the ability to be working in Visual Studio, working on Python code, developing it, and then when you hit f 5 to start debugging, it will run on Linux in this subsystem and be letting you debug code that's running literally on Linux without having to switch to another machine, without needing more hardware, without needing separate isolation magic with no extra steps. It's really Linux on Windows right there just so developers can be comfortable with the tools that they have, with the target platform that they're working on, so that packages can be installed. And and yes, when you're running this Linux subsystem, you have a c compiler because GCC is there.
So you can certainly install whatever packages you need on that. But that's certainly 1 of the most compelling things, I think, for developers that's coming out soon.
[00:42:26] Unknown:
Yeah. I think that once that actually hits general release that a lot of people are going to make the jump back to Windows because of the fact that they'll have access to everything in both platforms. The implementation I heard briefly referenced as sort of the inverse of wine though I don't know how accurate that
[00:42:43] Unknown:
is. Yeah. So the implementation is essentially a kernel shim. So anywhere that normal Linux distro would make a syscall, that gets intercepted and translated by by a device driver that's in in Windows and transformed into the the native syscall behavior. So it's it's a shim at very low level. I believe they said there's only 300 and something syscalls they needed to implement, which is a much lower number than something like wine. Wine is implementing a layer at the user layer. So it's intercepting every single call into the operating system and redirecting that. This is a much lower level. So you get a genuine Linux. Not not the genuine Linux kernel, but everything above that is exactly the same. So you can install a full distro and you'll get every tool, exact behavior, complete binary compatibility because everything is still running on the system it thinks it is. It's just once it hits the syscall layer, it's being transformed into the equivalent of a Windows system call rather than going through the Sys call it would normally go through.
[00:43:47] Unknown:
I'd be interested to see what desktop environments end up looking like in that context.
[00:43:52] Unknown:
I'm interested as well. I don't think you can put a desktop shortcut directly into a a Linux program yet, but I'm sure that's on its way.
[00:44:01] Unknown:
So could you give our listeners a whirlwind tour of the underlying implementation of Python tools for Visual Studio? How does Visual Studio provide in-depth introspection for your Python code?
[00:44:11] Unknown:
This is gonna be quite the whirlwind. We have a a 1 hour recording internally for new members of our team describing this. Let me try and give you the 32nd version. Essentially, we have an implementation of Python which we've rewritten that executes your code but instead of using values, it uses types. So anywhere you have a variable that you assign a value to, we remember the type. And then we basically try and flood fill that throughout all the code that it reaches. So anywhere there's a function call, passing that as a parameter will flow the type through and just try and keep flowing it through until we hit the the boundary of where nothing else passes that variable on and say, okay. Now we know what the types are up to here. The that is actually a ridiculously huge problem, as I'm sure you can imagine, normal code has loops.
A lot of normal code has infinite loops, and there's conditions there that that would break out of that that we simply can't assume that we can't understand when we're only dealing with the types. So there's a lot of the magic is knowing when to stop. And typically, we do have to stop early. So typically, you won't get a 100% perfect information because to do so would take forever and you'd definitely run out of memory. You'd probably run out of, you know, the sun before it's a 100% complete. But we do have the the heuristics are are, I guess, the secret source for when is this information good enough that we can stop trying to evaluate it from here. But essentially, we execute the we simulate executing the code and instead of passing around the values, we just pass around the types and end up with a model of the code where each variable has its set of types associated with it. As for the rest of Visual Studio, so Visual Studio is this amazing platform.
It's very much like an operating system. It provides a huge set of services, a whole lot of generic user interfaces. Everything is very extensible. There's 1 team that we call platform team that does the core piece and every other part of Visual Studio, every other language, every other extension, every basically, every feature up to and including the text editor is an extension on top of this core base. So it's a hugely extensible platform which makes it really fun to develop for in a lot of ways because if you see that someone has done something so if you see that someone has made the editor do something clever, there's a way for you to do it as well. If you see that someone has added a window with certain functionality, there is also a way for you to do that. There's nothing that Visual Studio does that can't be reproduced by someone else, that you can't reproduce in your own extension, which is really which is really cool. Though it's not necessarily easy because unlike most operating systems, it's frankly not as well documented and it's not anywhere near as stable as an operating system would be. So there there's challenges either way but it is it's definitely a platform to develop on top of. It's very amenable to that. There is plenty of good documentation examples.
And and for us working internally, we get to look at some of the internal examples. And then when we take our code and make it open source, that helps other people come and see how did they do this. Oh, now I can reproduce this in my in my own code, which is everything from supporting project systems, extending the debugger, extending the profiler, extending publishing support, if you want to publish code to another machine or to the cloud. All the IntelliSense tricks that we do, all the completions, tool tips, it's all extensible.
And the fact that our code gets to be an example of that to the rest of the world is really cool.
[00:47:37] Unknown:
Absolutely. I guess 1 thing that I was wondering about when you were discussing previously the sort of internals of PTVS with the the typed version of the person's code. Where is the line in that implementation between the c sharp and the Python?
[00:47:53] Unknown:
So the original implementation of that was in Python. That got translated to c sharp. In large part, I believe it was because of the way garbage collection works. Python's garbage collection was not efficient enough for the amount of sort of dangling references we were dealing with. This this predates my time on the project. So I it's always been written in c sharp as far as I'm aware. Now that I'm controlling the direction a bit more strongly, I'm trying to move what makes sense into Python code and sort of minimize adding new c sharp code, in large part because I wanna, you know, encourage contributors. And when we can say 90% of the debugger is implemented in Python code, if you found a bug, you can probably fix it and then you can send a poor request which we're allowed to accept, then that's really cool. But certainly, the code analysis is largely c sharp for performance. So there's sections there that are deliberately marked as unsafe so that they'll execute quicker. There's parts that rely heavily on tag and sweep garbage collection rather than cycle detection, simply because there are thousands and thousands of cycles all the time. And if it was written in Python, the cycle detector would spend all its time trying to detect them and probably not even be able to break up many of them. So, yeah, we we went to C Sharp for that.
[00:49:05] Unknown:
So are there any questions that we didn't ask that you think we should have or any other topics you'd like to bring up before we start to close out the show? No. I think that's fine. So for anybody who wants to follow you and keep up to date with what you're doing, what would be the best way for them to do that?
[00:49:20] Unknown:
Probably the best way is to follow me on Twitter. So my name there is Zouba, z o 0 b a. Don't don't ask how I came by that name. I was 13. I keyboard mashed and it stuck around ever since then. And and we so everything that that my team does goes up on GitHub. So github dotcom/microsoft. You'll find the organization. And if you search for Python in that, then you'll find a lot of my team's work. There's also github.com/azure. And again, if you type Python, you'll find a lot of that. And typically, I go by my real name everywhere else. So steve.dauer is going to be me on the Python bug tracker and and forums like that. Great. So with that, we'll move us on into the PIX.
[00:50:00] Unknown:
I've got 2 PIX this week. My first 1 is a tool called kdiff 3. That's what I use as my merge tool when I'm dealing with Git Conflicts, and that's the best 1 that I've come across so far. I've tried a few different ones, but for visual merge tools, it does a really good job with through a merges. Has some it exposes easy ways to set the key bindings that you want. I'm a user of e max for, you know, a few years now, So my fingers always want to use emacs key bindings, so I set it to use control n and control p to jump to the next and previous diffs. And it's easier to automate than some of the other ones I've tried to use. And my next pick is something called the Spyderco Triangle Sharp Maker. It's a sharpener for knives and scissors. And, basically, anything that you have that has an edge, you can use this to sharpen it, and it just does a really good job. It makes it easy to add an edge to your kitchen knives or a pocket knife. It even handles serrations, does scissors. So definitely recommend taking a look at that for your kitchen or anywhere else that you have need of bladed instruments. And with that, I'll pass it to you, Chris. Thanks, Tobias.
[00:51:06] Unknown:
My 1 pick today is audiobooks. Specifically, I'll use Audible because that's the that's the vendor that I'm using. I I love reading fiction in actual print form, but I, for some reason, have realized that nonfiction, trying to read it in print form puts me to sleep. But audiobooks, I can cruise through with no problem at all. So, I'm really enjoying that. That's all I have for picks this week. Steve, what do you have for us for picks?
[00:51:31] Unknown:
Yeah. So I went ahead and picked a couple. The first is gonna sound like a product placement, and it's not because no one's paying me for this. I picked up the SanDisk Extreme Portable SSD Hard Drive a while back and almost immediately went back and got another 1. This thing is, like, 500 gigabytes SSD USB 3, so I get over a 100 megasecond transfer. And and the thing is a quarter the size of my phone. It's like I put in my pocket and I can forget that it's in there. So that that is really cool. If you if like me, you end up transferring huge amounts of files around physically rather than sending them all up on the cloud and downloading them elsewhere, that's really great. That's Sandisk has not paid me for that. If you wanna send me more of the hard drives, don't bother. I have enough. But that's that's a really cool hard drive. My my other 2 picks are both kind of light entertainment. Most of the time, once I get, you know, out of work and just wanna hop on the Internet, it's it's sort of relaxing time, and just light entertainment. So Saturday Morning Breakfast Cereal is a webcomic that I've been reading for years. This stuff is hilarious. So the website is smbchyphencomicsdot com. The guy who writes at Zach Weenor Smith is very smart man, knows a lot of stuff and is able to turn it all into really of often intelligent jokes. Occasionally butt jokes but often intelligent jokes to do everything from philosophy, technology, computer science, physics. Very entertaining.
Always get a good laugh out of it. And and my last pick is my favorite YouTube channel right now which is called Random Encounters. The channel name is youtube.com/randomencountersent for entertainment, e n t. And these guys make gaming musicals. So they they aim to make a musical each month, about a video game or based in a video game. They tend to be short. So it's like 1 song out of a musical but they've got some really great ones based on Pokemon, really great Zelda ones, Team Fortress 2, 5 Nights at Freddy is a recent game that they finished doing as short series of musicals on. Very entertaining stuff, very well performed. A lot of cool people from around the the YouTube coming on and singing and performing with them. It's great light entertainment. I really enjoy it.
[00:53:30] Unknown:
Well, thank you very much for taking the time out of your day to join us and tell us all more about Python on Windows and some of the different tooling around that. Definitely excited to see what happens once the, Linux subsystem becomes generally available. I'm sure I'll probably end up experimenting with it a bit. I appreciate all the work you've done to simplify installation of Python. It's made it easier for me to get it up and running to work on it with my kids. Alright. Thanks for having me on. Thank you. Bye bye.
Introduction and Guest Introduction
Steve's Background and Work at Microsoft
Getting Introduced to Python
Becoming the Release Manager for Python on Windows
Challenges and Improvements in Python Installation on Windows
User and Developer Experience Enhancements
Future Improvements and Community Contributions
Building Python Distributions for Windows
Download Statistics and Trends
Cross-Platform Development Challenges
Visual Studio and Python Tools
Compelling Features of Python on Windows
Implementation of Python Tools for Visual Studio
Picks and Recommendations
