The Python Podcast.__init__

Share on social media:

[00:00:13] Unknown:

Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. When you're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it. So take a look at our friends over at Linode. With 200 gigabit in private networking, node balancers, a 40 gigabit public network, fast object storage, and a brand new managed Kubernetes platform, all controlled by a convenient API, you've got everything you need to scale up. And for your tasks that need fast computation, such as training machine learning models or running your CI and CD pipelines, they've got dedicated CPU and GPU instances.   Go to python podcast.com/linode,   that's l I n o d e, today to get a $20 credit and launch a new server in under a minute. And don't forget to thank them for their continued support of this show.  

As a developer, maintaining a state of flow is key to your productivity.   Don't let something as simple as the wrong function ruin your day. Kite is the smartest completions engine available for Python, featuring a machine learning model trained by the brightest stars of GitHub.   Featuring ranked suggestions sorted by relevance, offering up to full lines of code, and a program in Copilot that offers up the documentation you need right when you need it.   Get Kite for free today at getkite.com   with integrations for top editors, including Atom, Versus Code, PyCharm,   Spyder, Vim, and Sublime.  

And you listen to this show to learn and stay up to date with the ways that Python is being used, including the latest in machine learning and data analysis.   For even more opportunities to meet, listen, and learn from your peers, you don't want to miss out on this year's conference season.   We have partnered with organizations such as O'Reilly Media, Corineum Global Intelligence, ODSC,   and Data Council.   Upcoming events include Picon US in Pittsburgh.   Go to python podcast.com/conferences   to learn more about these and other events and take advantage of our partner discounts to save money when you register today. Your host as usual is Tobias Macy. And today, I'm interviewing   Antonio Cuny about HPY, a project aiming to reimagine the c API for Python.  

So, Antonio, can you start by introducing yourself?  

[00:02:16] Unknown:

Yes. Hello to everyone. I am Antonio Cooney.   I am mostly known in the Python community   because of my work on, PyPI. I have been a core developer of PyPI since 2006.   And do you remember how you first got introduced to Python? Oh, yes. It was long time ago. I think it was 2, 001. Just before I joined the university,   I was working on the Italian C plus plus news group.   And there was a guy, Alex Martelli, who is very well known in the Python community as well, who kept writing about a wonderful list Python as a language. So I gave it a try and never looked back.  

[00:02:55] Unknown:

And so   now you're working on this HPI project, which I understand is at least partially motivated by some of the pains that you felt as a maintainer of PyPI. So I'm wondering if you can just give a bit of an overview about what HPY is and how it got started. Right. You are correct. So by by  

[00:03:14] Unknown:

1 of the most painful point of pipe I is using the c extensions.   Most of the c extensions for the in the Python world are written targeting the c Python c API,   and, in the past years,   we, as by by developers,   have been trying to   support all of these extension in PyPI, but it's very painful and slow and cumbersome because the CAPI   is,   too tied to the CPython internals.   And,   so after years years of pain, we started to think about a better way of doing the job.   And,   we started to think that a newly designed API could be   better. The problem   of designing a new API is that then you need to rewrite all the extensions. So it was not clear that it was   a good idea.  

But that's basically   how and when   this idea became   stuck in our minds.  

[00:04:14] Unknown:

And so as far as the goals for this project, I'm wondering what the overall scope is and sort of when you can   state that it is done or successful?  

[00:04:26] Unknown:

Good question.   It the the goal is is for it to be an incremental project.   We can't fix   every problem   at once.   And,   so in a way, it will never be finished. The goal is to to start having a foundation in which you can start to experiment with, to have different back ends for better implementation,   and, and to start   rewriting   some extension or part of some extensions using HPI so that we can see what are the benefits.   Hopefully,   if the benefits are as good as we hope, then people will become interested in reporting the extension to H by   writing new back ends, for example, for Cython so that Cython emits code which is compatible with h pi and maybe even to start experimenting with   improving or   evolving CPython   to try new things that are currently impossible because of the current CAPI.  



[00:05:24] Unknown:

And in terms of the people who are involved in this project, I understand that Pypi being 1 of the main intended consumers, there are obviously people working from it there. Are there any other projects or people within the CPython core contributors or the steering council who are either contributing to it or who you have had conversations with about determining an appropriate direction for it? Yes. HPI is a project which started from  

[00:05:52] Unknown:

the the pipe by developer, at least in terms of thinking, but, other people are involved,   which include the many projects which are important in the Python community. Actually, PyPI started for real,   almost,   8 months ago at, EuroPython in, Basel   when, I met Victor Stinner, which is who is a a core C Python developer and is now part of the Python steering committee. Talking to him, I realized that we both are were concerned about the shortcomings of the current c API,   and, we started to to talk about this new API which could solve all the problems. And,   so we we decided to sit down and   and start to to do something more concrete. At the EuroPython,   we   we involved also other people who are there, and in particular, Stefan Benel,   who is the main site on developer, and Mark Shenon, who is another CPython developer. And on the PyPI side,   there were me, Armin Rigo, and then Rolandami. And so hpy started, not as a pure PyPI project, but, really as a community driven project so that, it it can be useful for all the parties involved. And for the CPython maintainers, what are some of the pain points that they're dealing with with the current implementation  

[00:07:15] Unknown:

that they are interested in working on HPY as a solution to that? So the c pi there are many c pi to maintainers.  

[00:07:23] Unknown:

So as far as I know, not not everybody is concerned about the current problems of the c API. But, for example,   as I said,   some of this Python developers are worried about the fact that it is harder to,   improve   and to   innovate   the the internals of CPython.   Because, for example,   given the current state of affairs, it's very hard to remove the gil, the infamous gil, or to use,   a different garbage collector,   which is not a reference counting.   In a sense, HPI will will help see Python itself to to try new things,  

[00:08:04] Unknown:

which might or might not be useful in the future. And for other consumers of the c API, such as people who are writing   performance critical extensions like NumPy or SciPy?   What are some of the benefits that they might be able to gain by having this   reimagined interface to the CPython implementation or something that's a little bit more portable between run times? Well, the the most obvious,  

[00:08:30] Unknown:

advantage is that,   you will be able to use, these extensions   also on alternative runtimes, for example, PyPI. As of now, PyPI   for certain, code patterns is by far the fastest Python implementation   with which is around, but you can't use it effectively with the c extensions such as NumPy, which are needed for performance   for other in in other areas.   So the most obvious   use case is to use, for example, NumPy and SciPy together with PyPy   and, and   have a complete   stack of software which is passed. And,   more in the long in the longer term, the the best benefit of HPI will be the fact that it will be possible to innovate, to try new things, to   to develop a new Python implementation, which which we can't think, think of right now, but it will   be much, much   easier for them to to try,   to write   a new implementation without all the base that we had   for PyPI.  



[00:09:34] Unknown:

And so for people who are currently consuming the c API, what'll be the process for them to be able to take advantage of this new interface?   And from the maintainer side of the CPython runtime,   what do you see as being the path forward for them to be able to   deprecate   this old c API or at least migrate people primarily to HPI so that they can feel confident with changing what the internal API looks like without having to worry about breaking a number of extensions that are dependent on those implementation details?  

[00:10:08] Unknown:

So, for, for extension   users, basically, they like a NumPy user, they will need to wait until,   NumPy is made compatible with HPY. It's not something   that the user and consumers of this extension can do. For the extension writer, they will need to,   rewrite their extension   using HPI instead of the c API, the currency API,   API,   the currency API functions. We are designing a p, HPI,   making sure that this transition which be as easy as possible and not, not too hard. It it must be say that currently,   some extensions are written manually in c, for example, NumPy. So for for it, the only way of immigrating to HPY is to rewrite   parts of it,   but most of other extensions   are written, for example, using Cython. And so it means that it will be enough for Cython to meet HPY code,   and, all these extensions will be automatically taking advantage of HPY.  

It also must be said   that,   performance is not the only   advantage of using hpy.   Another selling point of it is that there will be, debug mode, which, will introduce many   many check at the run time. For example, to to check that we are not leaking object or that we are not using an object after it has been freed and,   and these kind of,   checks. And,   it will make make debugging and development   much more easy   easier and much more robust. And, this this debug mode will be enabled.   It it will be possible to enable the debug mode at run time.   So it will not it will not be necessary to recompile the extension to enable it, which is a pain point of this currency API.  



[00:12:06] Unknown:

And so for the HPI project itself, can you talk through how it's structured and just the overall development process of being able to iterate on wrapping the existing CA API and some of the design challenges that you're facing in terms of determining what the user facing interface is going to look like and something that will be clean and maintainable going forward?  

[00:12:29] Unknown:

Well, 1 of the goal of HPI is, is to make make it possible to to do an incremental porting.   So current extension will not be forced to do as a port to rewrite all the code at once. It will be possible to start   porting,   porting things, 1 function, 1 method, or 1 class at a time. This is, very important because it makes it make it possible to experiment, to try to see how it looks like, to check whether the performance   is,   is good, and, and to become interested in in trying more, basically. And, this is related   to to what it is, the biggest issue and, the biggest   difficult   in, developing HPI, which is to find   the right sweet spot   in the trade off between writing a nice, new, modern API,   and, to write an API, which makes it easy   for people to migrate to migrate their existing extensions.  

And,   every day in developing API, we we see this trade off, and we are trying various,   various things to to make it easy to to integrate and to   to use the best practice   and to design an API which is modern and safe. And so in terms of the interface  

[00:13:52] Unknown:

that is consumed by the run time, so specifically   the c Python or Pypyruntimes,   what's involved in integrating   HPY into those projects or any other alternative runtimes, such as Rust Python that might want to be able to provide this same outward facing interface to, extension developers?  

[00:14:14] Unknown:

So,   every Python implementation   will need to implement   their own HPI interface. But   usually,   it should be easy because,   for example,   1 1 function will will be HPI add, which which is does that exactly the same, semantics as the plus operator in, in Python. So if you have if you're if you're implementing the Python implementation, you need to to have this logic inside. And the the HPI   is just a way to call this logic from c.   So, ideally, HPI will be a thin layer on top of the existing semantics, which makes it possible to call it from c. Currently, we are developing 3 different back ends at the same time, and,   2 of these back ends are for CPython, and 1 is for PyPI.  

Why? Why do we have the need for 2 different back ends on CPython?   Well, the point is that,   1 of the goal of HPY is,   that extension should not be slower on CPython. We don't want the NumPy people to support the extension to   HPI and then discover that it is lower on the Python   because I'd say it's not going to be useful. So 1 of the backend that we have is for Spartan is, is basically a set of c macros and, start taking line functions, which translate   the HPI calls into all the c API calls at compile time. So from the point of view of C Python, these extensions will be exactly as before. And then another possibility   is to compile,   an HPI extension in what we call   the universal mode. So it produce   a DLL, a dot   s o on Linux, which is,   universal, and it's in theory, loadable and executable by   any compatible   Python implementation   without recompiling.  

And, we also have, a universal backend for CPython and then universal backend for PyPI. And this is the only backend that will, will, will ever be needed for PyPI. And,  

[00:16:23] Unknown:

is another advantage of HPI, the possibility of distributing 1 binary which works on every Python implementation if you don't care too much about that from us. So for extension developers who want to start porting some of their code to use the HPY interface, is it something where they would include the HPY library in their source code, and then during compilation, it would then translate to whichever back end they're targeting? Or is it something that would need to be compiled into the Python runtime to provide those interfaces that would then be consumed by the library developers?  

[00:16:57] Unknown:

So this is work in progress. Currently,   you you need to manually include a specific set of header style to compile an extension for HPI. But, eventually,   we want to provide,   an ISP   installable package,   which you can use to,   develop and compile your HPY compatible extensions.   So ideally and eventually, people will just do something like   from h pi dotdevilimportextension,   and use and use this extension   class in their setup dot pi file, and this class will,   take care of all the logic needed to compile the extension.   And, users will be able to decide at the compile time whether to target 1 specific implementation,   for example, C Python, or to target the universal  

[00:17:46] Unknown:

mode so to produce a file, which will be compatible with, everyone. And you mentioned that there are 2 different back ends for CPython because of some of these performance considerations.   So I'm curious what you have seen as some of the observed or potential impacts   on performance and runtime overhead for using   HPI in that universal mode?  

[00:18:08] Unknown:

Yes. That's a very good question. First, let me explain why we expect the universal back end to be slower than the c Python specific back end. For example, in c Python, you are using reference counting, which means that every time you pass an object around, you need to call in graph and then decrypt on it. And, in CPython,   these calls are usually are usually   macros, and so the C compiler can optimize it very well. In the universal mode, there are no longer in graph and the graph. We we are using the concept of handles,   which are   more opaque. And since since the implementation   is universal, it cannot be written as macros. It it needs to be function calls, which are resolved at a run time when they are loaded   into the proper interpreter. So, for example, it means that every incorrect and the graph inside the extension module   will become a function call instead of a macro. That's why we expected it to be slower in the universal mode instead of   the CPython specific mode. But, actually, we we have a benchmark which shows that this impact   is much lower than we thought. So the first benchmark that we tried is to port the UltraJson library to HPY, and,   we measured that the CPython backend   has no impact over at all. So it user JSON HPY is as fast as the normal user adjacent on c Python. And, on c Python using the universal backend, this benchmark is just 2%   lower or so, which means that for most projects, it is might be   good enough. And, the   the advantage of being able to distribute a single   binary for all the implementation might be bigger than the disadvantage of having a 2% slowdown. Of course, it must be say that this is just a single benchmark. We need more benchmarks, more extensions,  

[00:20:05] Unknown:

and more measurement before knowing precisely what it would be the the actual impact. As you have gotten further along in this project, what are some of the ways that the initial vision and scope of the project has been updated or any assumptions that you had going into it that have been changed or modified in the process?  

[00:20:23] Unknown:

So far, I think that we are mostly on track in the sense that the project is going more or less in the direction that we,   imagine when we started. 1 of the thing that we realized   is that the CAPI   is very broad. We we already knew it, but,   we we knew from the point of view of people that wants to implement   its semantics. Now we are also seeing it from the point of view of people who wants to use their API for doing stuff. So we are realizing there are very   there are many different use cases for which people needs to use the the API   and,   the existing API. So we need to design a new API, which makes it possible to   implement all these use cases.  



[00:21:09] Unknown:

So in terms of the timeline for this, what do you anticipate as being the amount of time that's required to get this to a point where library developers   should start using it as the target for building their own work?   And what do you see as the path forward for the c Python implementation   as far as adopting this as at least an optional interface for the core runtime?  

[00:21:35] Unknown:

So for now,   the the project is is still in its very early stages.   We,   we are still busy implementing the very basic stuff. So, for example, as of now, it is not yet possible to write a custom type. We can you can just write functions,   so it it's clear that it's not   immediately useful and usable for   for writing the real extensions.   The how much it will take to to put it into a state which is useful?   But I don't know. It's mostly dependent on how much time we can spend on it, which also depends on how much money we have to to to work on it, basically. I hope that in the in the following, in the next months, we we should be be able to arrive at the point in which we have,   a small,   NumPy like library,   which shows   all the interesting and hard   parts, and, and then we can use this as a benchmark and as a starting point to go forward and, for example, to start experimenting with the real NumPy. What was the other question? Sorry.  



[00:22:43] Unknown:

Just curious about the timeline for the project and any plans or projected timeline   for HPY to be available   in the core CPython runtime as at least an optional interface.  

[00:22:56] Unknown:

Yes. I don't think that this is going to happen anytime soon. I can imagine,   an ideal world in which   all the current   c extension will be written using hpy. And, so CPython   will will be able to   decide   to to use hpy as their official interface. But I don't think it's going to happen anytime soon because it will involve   rewriting   many parts of the,   of the internal core. What could happen and what I hope it will happen   is that   people could try to fork, say, Python,   to experiment. So for example, I can imagine that,   once HPI is stable and usable,   and, and,   and you know how it looks like, I can imagine to fork C Python to remove   many unnecessary   well, not unnecessary, but not vital modules and, start   to reimplement the day internal   CPython objects in terms of HPI API. And then once you have this, you can start to experiment with   with improving,   the the virtual machine of CPython itself. For example, you can try to use a a garbage collector instead of reference counting and, and things like this. But as I say, the I think this is more in the medium or long term  

[00:24:19] Unknown:

than the short term. And as you have been working on HPI, what have you found to be some of the most challenging or contentious aspects of the project   or any   long running design questions that you have had to go back and forth with the other members of the project?  

[00:24:37] Unknown:

As I said before, the biggest challenge is to find   the sweet spot between, make having a nice API and make it easy   to to port existing extensions. So,   many times, we   we sit down in front of the Blackboard   or in front of a minotaur, and we try to sketch how a c extension,   looks like,   how how to implement a type, how to pass   pointers, which can be moved in memory by the GC and etcetera. And,   every time we spend hours, really, real hours discussing,   maybe 2 or 3 different   possibilities   and,   possibility   a is   closer to the current Python solution, but maybe a bit more inefficient for PyPI.   Possibility   b is maybe better for PyPI, but harder for c users to write, and then there is a lot of   trying around until we find something which   which respected all of the requirements, basically.  



[00:25:41] Unknown:

And when you're commenting too about the possible transition period of HPI,   it starts to bring up echoes of the Python 2 to 3 transition. So I'm sure that there are a lot of concerns   on that line as far as not wanting to fork the community   because of this potential new interface even though it will likely have significant benefits downstream?  

[00:26:05] Unknown:

Yes. Yes. You are completely right. We want to avoid any   split in the community, any fork, and anything like that. That's why earlier I said that 1 of the goal of HPY is to be incremental.   You will be able to take, for example, NumPy   and start   rewriting just 1 method of it using the HPY interface,   and keep using the old API   for all the all the all the rest of the extension.   And this new method will be fast on by pi, for example, and there will be no difference for CPython.   So   with this starting point, it means that,   you you can always improve   and make it make things   better, for example, for PyPI   without making it worse for CPython.  

And, and the more you do, the better things become. And then, hopefully, at some point, h 5 will get enough traction  

[00:27:00] Unknown:

that people will want to use it to get to the benefits. And what do you see as some of the potential risks to the success of HPI or any blockers that might come up that will prevent any forward progress on it? The biggest risk is that,  

[00:27:16] Unknown:

many projects don't want to change. So, for example, from the point of view of NumPy   developers,   why do they want to accept   maybe our poor request to migrate slowly to Edge by   when they their existing   code is perfectly   fine for the existing users? I can't answer to this question.   I hope that the community as a world   will, will see that, HPI is needed to to make it possible to innovate   in the Python world. Maybe I didn't I didn't make it clear earlier in the interview, but, the biggest problem of the current CAPI is that it makes it hard or impossible to innovate. Python as a Python as a language is, doing a good job because of its big ecosystem   with all the libraries, and people expect all the libraries to be available when they when they use Python. But all these libraries are very tied to the current internals of CPython. For example, the fact that they have a reference counting and that   at   offset 8   of of an object, there is a field,   which is the reference count,   and, and and so on. This   low level details,   which makes it impossible   for new   virtual machines to innovate and to try things to try other things. If if Python,   as a as an ecosystem,   do not try doesn't does not try to evolve and to and to innovate.  

It will be the risk is that it will be,   become less and less relevant in the future. The gil is already a perfect example.   There have been many attempts of killing the gil, removing the gil,   but, reference counting is all is a big,   issue to it. And, it is not possible to rewrite the C Python without it because of all the issues that I mentioned before. So from side point of view, h pi will be is good and needs to to happen   for the world by an ecosystem, not just for PyPI. PyPI is just   the the main benefit immediately, the main,   immediate   project, which can benefit from it. But, generally speaking,   the possibility of having more   virtual machines   and run times is,   is something that you really need as a language, as an ecosystem. And I can definitely see that as being a potential path  

[00:29:48] Unknown:

forward as well for solving some of the issues that we have in being able to target   mobile platforms   or the web as well. Because   as you said, the majority of Python is tied to the c Python runtime where once we get to a point where we can innovate on the VMs or have multiple different deployment targets for it, it will unlock some of the potential for being able to run on some of these platforms that we're currently unable to target. Yes. Yes. I completely agree, basically. Yes. And so in terms of the near to medium term for HPI, what do you have planned?  

[00:30:21] Unknown:

Most immediate target is to be able to support to support,   custom c types in extensions So we can experiment with,   NumPy like extensions   and and see how it looks like and what is the performance. This is the here's something which is going to happen in the next weeks, basically. And then the the other big,   plan that we have in the short period is to   to write a site on back end for h 5 so that, people writing extension in site on will be will be able to immediately,   benefit of it without having to rewrite anything manually. In terms of any help or contributions  

[00:30:59] Unknown:

or support for the project, what are some of the most useful skill sets or ways that people can help to move the HPI project forward?  

[00:31:08] Unknown:

That's a that's a very good question.   Of course,   any help is, is welcome.   I think it will be very welcome to have someone who who is a a consumer   of the CAPI,   someone who who tries c extensions   daily   or who has lot of experience in writing it because   it could,   it could be very it could be very useful for us to be warned beforehand   that this direction that we are taking is wrong or that direction is better and and things like this. And,   and then it's it's already   nice to see that, HPI is attracting the interest from various   projects and parts of the community   because,   we started in a few few people, but, immediately,   we got   the manifestation of interest from,   developers of Rust Python so so that they could load their,   HPI extension on their runtime.  

Growl Python, which is, another Python implementation   based on Java,   and developed by Oracle. And, recently,   there is also a sub project to be able to use,   HPY with Rust so that you can write Python extension in Rust and decide whether to target CPython or HPY. So,   we have   users and, people involved in or from both, ends of of the of the line. From in 1 end, you have   people writing extensions, and on the other end, you have Python implementation.   And, any people who have experience in these 2  

[00:32:48] Unknown:

topics is very welcome to help. And are there any other aspects of the work that you're doing on hpy or the goals that you have for it or some of the potential impacts and benefits that we didn't discuss that you'd like to cover before we close out the show? Sorry. Could you repeat, please? I'm just wondering if there's anything else that you think we should talk about in terms of the work you're doing on HPI or any of the potential benefits or impacts that it might have or any of the, related projects that will,   benefit from it? Not not not more than I just than I already said. Okay. Well, for anybody who wants to follow along with you or get in touch or contribute to the HPY project, I'll have you add your preferred contact information to the show notes. And so with that, I'll move us into the pics. And this week, I'm going to choose the poetry tool. I've started using that more heavily in some of my projects, and I've been enjoying   that as a way to get things set up quickly and managing dependencies. So definitely recommend taking a look at that for any new projects you're working with. And with that, I'll pass it to you, Antonio. Do you have any picks this week?  



[00:33:51] Unknown:

Yes. I would like to suggest a book which I read recently, which is Collapse by Jared Diamond. And, I think it's a book which is interesting and terrifying from some point of view, but, I think it's good to read it to have an understanding of what's going on in our world, keeping in mind what happened already in the past. I'll definitely have to take a look at that. So thank you for the recommendation,  

[00:34:14] Unknown:

and thank you for taking the time today to join me and share the work that you're doing with HPI. I'm definitely excited to see how it goes and see some of the downstream benefits from that. So I appreciate all of your work on that front, and I hope you enjoy the rest of your day. Thank you very much, and goodbye.   Thank you for listening. Don't forget to check out our other show, the Data Engineering Podcast at data engineering podcast.com   for the latest on modern data management.   And visit the site of python podcast.com   to subscribe to the show, sign up for the mailing list, and read the show notes.  

And if you've learned something or tried out a project from the show, then tell us about it. Email host at podcastinit.com   with your story.   To help other people find the show, please leave a review on Itunes and tell your friends and coworkers.