The Past, Present, and Future of Twisted with Moshe Zadka

Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. When you're ready to launch your next app, you'll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 200 gigabit network, all controlled by a brand new API, you've got everything you need to scale. Go to podcastinit.com/linode to get a $20 credit and launch a new server in under a minute.

Go to podcastinit.com/gocd

to learn more about their professional support services and enterprise add ons, and visit the site at podcastinit.com

subscribe to the show, sign up for the newsletter, and read the show notes. Your host as usual is Tobias Macy, and today, I'm interviewing Moshe Zadka about Twisted, the original multifunction tool for asynchronous operations and network protocols in Python. So Moshe, could you start by introducing yourself? So my name is Moshe Zadka. I've been using Python since 1998.

I've done some core contributing,

and I've been part of the Twixys project since its its inception around 2, 001. And do you remember how you first got introduced to Python?

I was, looking around for a language

which had, a good XML support with,

validation for schemas,

and Python was the only language that had actually good libraries that I was willing to use, so I decided to run Python.

And for anybody who isn't familiar with the Twisted Library, can you share a brief overview of what it is and,

what it does, and what is some of the things that people use it for? It is a library for doing event based programming, and that turns out mostly to be about network events, although not,

uniquely. So most of the things people end up doing with Twisted

are, network

programs.

That includes both network clients and servers of various sorts all the way from various web servers to, things that

send SNMP

commands to routers.

And so you mentioned that you were

part of the, team at the point when Twisted was first being created. So can you share a bit about the origin story and how you got involved with the project?

So Glyph wanted to write a library for, game programming

that involved,

networks,

and he was,

talking about it in, channel Python

on, IRC,

what is now free node.

And,

I got involved mostly because I had some interesting ideas about how to implement, protocols,

and I I want really wanted a place to get a chance to to to try that out. Do you know if Glyph was ever successful in actually creating the game, or did he get too far down the rabbit hole of building Twisted?

That's actually a funny story. So the original name for, Twisted

was because the game framework was supposed to be Twisted Reality,

and so Twisted dot reality

was, the subpackage that was supposed to be the game.

If you look at Cambridge, you'll see that, reality has been deleted because it never actually got to the, a functional point. So not only did we did we not end up, writing a game,

we didn't end up even writing the proper game support library. Just the networking part of it. As is the as is the way with such things. You, set out with the best of intentions and then get waylaid and instead

successful networking library.

I Gliff keeps threatening that 1 day he lights a a game. I think the code name for that is Divinal.

So if you look at his, blog, he actually wrote something about it a while ago. As somebody who's been a maintainer of the twisted library

for such a long time, what is your current role in the team and how many people are currently involved in keeping the, project moving forward? So I don't have a specific role, although

I've kind of done quite a bit of preparing teaching materials on that. But, occasionally, I'll try to review patches, and, occasionally, I'll help people,

fix bugs

that, bother them. I think

there's, like

depending on how you count, anywhere from,

5

to 15 people who kind of semi regularly work on it. It's a very changing crew depending on who has time and who wants to do something specific.

And

so 1 of the things that I heard a lot when I first, learned about Twisted was that it can be difficult

to learn and get started with and that people sometimes have difficulty wrapping their head around the programming model that it uses, particularly

given the fact that it is so heavily asynchronous. So what are some of the

resources

or materials that you found to be most useful for people who want to learn and get started with using Twisted?

So first, I'll I'll I'll say, like, what is the easiest way to get started with Twisted because this this has changed, and I think it's it's changed a lot for the better.

So if you are starting now in 2018,

just start using Python 3 6 with Twisted,

and this means you can use the modern syntax support with async and await functions,

and that will make a lot of things just a lot easier to get started with. So 1 resource that,

Glyph and I prepared, which actually doesn't use the async and await, but using uses a very similar approach to kind of ease people up, is a tutorial we gave in PyCon 2017,

which is obviously on PyVideo.

We can put a

link in the show notes.

And we also,

had, like, the notebook that we use to give this tutorial as a as a resource for people who don't want to watch

a 3 hour long video but just want to go through the exercises can do that. I think this is 1 of the nicest thing. It's kind of a pre it it it tries to,

teach from a very modern approach twisted. And I guess the other thing,

is that somehow a lot of other people are using async programming nowadays because Node. Js

is 1 of the most popular platforms,

and browser JavaScript is even more popular than that.

So I think some of it is just maybe people just

heard that and and and kind of are a bit too scared, but it's actually easier than it looks,

and literally everybody else is doing that. And as 1 of the early movers

in terms of building an asynchronous

library and doing event driven programming, do you

ever sort of,

start shaking your fist at people who are praising the newfangled async frameworks

and, telling them that you've been doing it since they were in diapers? Or Well, so I I I I you know, a lot of people expected that recent people to react badly to that, but we're very happy. It feels like after, like, literally 15 years of shouting at everybody that AC is not that hard, suddenly everybody gets a message and everybody's doing that.

So we're actually very happy to kind of finally have, proven that, this this is this is a good way to go forward.

You have vindicated

in your,

in your beliefs. Enough to say yes. Definitely.

And I know that at least

originally, both Twisted and a number of the other asynchronous

style libraries

were heavily built around the notion of callbacks,

which a lot of people will bemoan occasionally because of the fact that they can become very deeply nested, and it becomes difficult to reason about the order of operations and when things are going to fire. And I know that with a number of frameworks have moved more towards using,

futures or promises

that are

a little bit easier to reason about. So has Twisted followed along on that path as well? In many ways, Twisted has led along that path. I think Twisted had deferred,

since a lot of these did not have them. And you know what they say that, the pioneers have the arrows in their back. So deferreds were very early. And so they're pretty nice, but, definitely,

a lot of the promises in futures, I have to say, are a little bit nicer than the diff twisted deferreds just because we had a chance to learn from the deferreds and not make some of the mistakes we made with them. But still, deferreds are pretty nice, and they really avoid the whole nesting thing with callbacks.

And we've been doing that

for, like, the last 15 years, so at least they're very stable. And with the recent addition

of the asynchronous support in Python 3.5, I think was the first inclusion

and more improvements

with the async await

and yield from syntax

in the more recent versions of Python 3. Has that eased your

workload in terms of adding new feature support to Twisted?

And what does the integration

between the Twisted event loop and then some of the other async engines in the Python ecosystem look like? So,

I guess these these are 2 different questions and but but it's good that you have them together because people often confuse the 2 issues. So while ACK08 was originally invented to help the ACKO library,

Twisted supported it since it since it began. So now I think even when Python 3.6 was in beta, we already had, some async await support. So async await await works pretty well in Twisted just as well as it works in,

in asyncio,

and it really has made wizard programming really nice. So in in many ways, I said that we were happy about other libraries,

coming along. And so the the biggest way ACKO has helped us is because it's developed by Python Python Core. They're adding,

special syntax support

for, async programming, and we get to take advantage of it that in Twisted. So this is actually pretty nice.

As far as asyncio integration,

we've had the ability to run the twisted event loop on top of the asyncio event loop for quite a while,

which means you can actually fully intermix it. We don't have the ability to do the other thing to run the asyncio

event loop on top of the Twisted event loop. And, mostly, that means that you don't get certain features in asyncio because you can get every feature in Twisted by running it on top of the asyncio event loop. But if you, say, wanted to integrate asyncio

with, GTK,

then the easiest way would actually to be to have ACK running on top of Twisted and take advantage of Twisted supports for GTK. I think people are working on that. That's definitely something we plan on supporting, but we don't have that yet.

And 1 of the things that

I am often surprised by, and I know others are as well, is the breadth of

protocols that are built into Twisted and that you can use for being able to tie together very disparate,

types of operations

within the 1,

library of twisted. So what are some of the most

so 1 pretty cool thing that they think,

is now defunct, unfortunately, but, company was using Twisted to basically do router health management. So they were using,

Twisted's ability to support SNMP to send SNMP,

commands to a lot of routers and and and figure out what SMP is a simple network management protocol, and that's basically the native protocol most commercial routers use to report health and to manage themselves.

And they had, like, this device that basically would manage a whole fleet of routers. And, actually, Twisted was so efficient. It would occasionally kill the Cisco routers because it would send them, requests too fast. And

this is actually interesting because some of the support we have interested is, like, you you don't want to obviously slow down your stuff, but you want to make sure you don't kill

anything behind you. So we have actually pretty sophisticated,

support to make sure that you rate limit how much you send to a specific point or to a specific service. And this is always the story I tell when I introduce those tools. I I I I tell the stories of the dying Cisco routers. For the internals of Twisted, is it pure Python,

or do you have any sort of siphonized

or

c bindings for

performance improvements?

So

most of it is pure Python.

There are a few little bits of c that

basically the plan is to spin them out to a dedicated library so that we can do an optional dependency on them. I think 1 is for performance,

but it's it's kind of an edge case. It's for a protocol very few people use, which is 1 of the reasons we want to make it an optional dependency.

And 1 is actually for bindings

to see functions that the usual Python library doesn't bind. I think some of the more,

esoteric,

unique system calls that you need to,

do funny things with unique domain sockets to send file descriptors on.

They're not supported by by Python natively, so we had to write our own binding. But, again, that's kind of, like, a use

case. So the plan is to spin some of those out into,

a dedicated library that just does the binding

and then have Twist and do an optional optional dependency on them. 1 of the things that I've seen in a few places

is that people recommend using Twisted

as the

WSGI layer for web servers in place of some of the other options such as gunicorn

or micro wizgi because of its performance capabilities and its asynchronous nature?

And are there any other

places

that you tend to use Twisted either in your own work or that you recommend others use it for where it,

isn't necessarily

the first thing that people reach for?

Sure. But first,

about the WSGI support, I actually have something to say about that. I don't think it's more performance than you WSGI, but I think I think it still has benefits because very few people will run into the performance issues. But it's actually nice to have a pure Python library,

doing the web serving. That means that a lot of problems, sometimes people wanted to use with how it's built. For example, we have a problem at work with how it's built against SSL, and it's doing something weird. And using using Twisted for that just avoids these problems, and because Twisted was started out as a regular web server with the movie capabilities,

it has built in ability to do SSL termination

and to do file service. So often, you don't need to do this, thing where you put NGINX in front of your gunicorn or in front of your, USB. You can just use Twisted for what you would have used NGINX for as well as what you

as as well as the Willgi server, and that simplifies operational complexity.

The other thing I like to do with Twisted

is I like to use it instead of Supervisory.

Supervisory is certainly a respectable async support that Python 2.7 had that was so bad we had to write,

Twisted, and that's kind of a universal acceptance that was really bad. Asyncio was written because

async chat and async core were not very good, and, supervisor is written on top of basically a deprecated

thing in Python 2.7 that is gone from Python 3. So instead of that, I like to use n Colony,

which is a twisted based, process manager. And it has a bunch of nice things like how it, dynamically

stops and starts services based on detecting

files getting created and deleted.

So it's pretty easy to, do dynamic things. I like to put it inside my Docker containers. So when I dock exec into the Docker container, I can literally just remove a file and have the main service shut down so I can start my own with debugging,

on or something like that. Obviously, it's not great

to have to do it, but in a situation where you have to do it, it's it's pretty cool to be able to do it. And what are some of the cases where Twisted is the wrong choice of library to use or examples of places that you've seen people misuse it or abuse it where they would have been better served by some other piece of technology?

So, first of all, the simplest thing to write always is just a plain synchronous program

that,

has 1 thread of execution,

starts up, does something, then does the other thing. And

if you're using

for no good reason, it'll be harder to debug

than just using something with 1 thread of execution. And the other place

that Twister would not shine in is if you have something very processor heavy, then you're probably better off just

either,

using some,

some c extensions and then using threads and releasing the gill in the c extensions or using multiple processors.

But, definitely, twisted is not for the case where where your main bottleneck is a CPU because it just just like always with Python, you can only use 1 core at a time, so it's not gonna be good use of your machine if you're CPU bound.

And what are some of the ways that Twisted has evolved and changed over the number of years that it's been around?

So, obviously, we've added features, removed bugs, but I think the biggest,

evolution we've had is,

adopting much more rigorous

development process. So

we've sort of gone from 1 extreme to the other. In the beginning, it was kind of cowboy, and everything got checked into right into master without any thought.

And we've developed a very rigorous and

exception free process called,

the ultimate quality development system

that

I think by now is pretty much what everybody does.

But back in

2004, it was kind of an innovation because we had this strict policy of branch based development. So you always

do any feature development on a branch, and

you never start a feature unless it's a corresponding ticket.

And you don't,

you're not finished until the the unit tests that checks that you've done it correctly,

and nothing gets merged without a peer reviewer.

And I think most people, like, now in 2018,

follow some approximation of that, but basic back in 2004,

with the very limited branching capabilities that we had there,

it was really weird to do that. And I'm very happy we did it because it made that Twisted has gotten

much more solid over the years.

And

as part of the life cycle and evolution of Twisted,

Python 3. And from casual conversations

with you and some of the other members of the team, I understand that that was a bit fraud at times. So I don't know if you wanna talk a bit about the,

efforts that you had to go through to get twisted to work with Python 3, particularly given the Unicode

versus bytes issues at the network layer that is often

problematic.

Yeah. So I I think among other things, I think Twisted put a very good pressure on Python itself

to

have better transitional

plans. For example, adding the u prefix to strings

in Python 3 so that you can have code that's 2 and 3 compatible

will definitely

be a big part of it. And in general, the ability to write code that is 23 compatible.

In the beginning, the plan was that you'll write your code

in 2, and then you'll you'll you'll automatically,

do 2 to 3, and you'll publish 2 libraries. And that's both operationally very complex, and I don't think anyone actually managed to follow that plan. And kind of part of a a lot of libraries, but the moment we had a hard time dealing with that. And and, eventually, I think around 3

Python 33, Python 34, core Python started adding enough capability to make writing

code that is both

compatible, more realistic, as well as, the 6,

library coming along and making it easier. And I think at that point, yes, we had a lot to think about the unicode and bytes. I think a lot of that thought process was really good. Unfortunately,

the Internet is really complicated.

A lot of the protocols that we support are, like,

really, really old.

IRC in particular has some very, very interesting,

thoughts about the semantics of bytes. So kind of reconciling them with the Python model of bytes are for lower level things and strings are for human visible things,

was nonprevious.

But in general, I think that this this distinction was very good for twisted

because

a networking library does have to make the distinction between what is a string that is supposed to be shown to human people

and what is

the sequence of bytes that is supposed to be sending over the network and

when the encoding

information and all of that. So, overall, I think Python 3 was good for Tristan, but I think it took time for core Python to

really think about what a realistic plan for library transition looks like. And as somebody who's been working with Twisted for so long,

what are some of the things that you have learned in the process

that were unexpected

or most beneficial?

So I think 1 of the biggest lessons I've learned

is how useful

really, really good unit tests

are. In in in many in many aspects of Twisted,

I've I've kind of learned this lesson over and over again. And, actually, the last point about this, the transition to Python 3 is really interesting.

Part of transition into 3 was actually writing a lot of unit tests

to make sure that we actually do run correctly on Python 3. So all the things that were in the past, especially the old code, it wasn't very well covered by unit tests,

writing unit tests for that and finding a lot of bugs. I I I really think

that Twist is really

hammered

well in me the concept of unit tests are really, really important for code quality.

And 1 of the things that I found interesting in a previous episode that I did with Glyph about the

event machine library that he wrote as a,

corollary to the work you're doing with Twisted is that,

by virtue of adding that event machine capability, you assessed out a few more bugs that didn't get caught by the unit tests. So are there any other,

additional libraries

or tangential projects that have been spawned either as a result of the work on Twisted or to support the work that you're doing? So I think 1 of the things that has been spawned, but I think it it never really it it so far, it hasn't really solidified.

I keep hoping that someone will have time to work on it. It's a pure

Python SSL implementation

that is based on very declarative ideas. Automatt would be a part of it, but a big part of it would be how you declare the structure, the various structures that SSL

lot of the historical

a lot of the historical security flaws in open SSL,

come from the fact that 2 parts, in the code don't don't reveal the interpretation of a certain sequence of bytes. I I think we actually had something that,

started

that, but it turns out to be a really hard project. And that actually spawned, like, a bunch of us kind of going off

and kind of researching

how how to how to do protocols correctly and how to implement protocols correctly

and kind of it sparks a lot of interest in things like TLA plus, you know, the, to this community.

So I I I guess it's it kind of relates to the

to to unit tests,

in the sense that traditionally in computing, there are 2 ways to,

improve the correctness of something. You can test it a lot, and unit test is 1 way of testing it, or you can prove it correct. And automatic in many ways is part of the other

the other tradition of of proving things correct. And I think we've we've kind of gotten so good at unit tests. We've reached the limit of how well they can make your code higher quality, and now we've started to look into

what kind of things we can get from the, from the other traditional stream, from the from the side of proving things correct, from the making sure that things can't be,

can't be incorrect by definition.

And your discussion about doing protocols correctly also brings up some of the recent movement

of decoupling

protocol implementations

from the IO aspects. So I'm wondering how much of that is present and twisted where the protocol implementation

can be taken

on its own without necessarily

having to tie into the asynchronous event loop that is the core of Twisted.

So there's a lot of, side to that. So when we started writing the,

protocol, I think the the big the big plan was to actually make them what would now be called San Diego. And some of the simpler protocols are actually written in San Diego.

But, unfortunately, it's just too easy, the protocol implementation, to kind of reach into the event loop and do something, and sometimes that makes, you know, it it it it's very at the moment, it looks like a good idea. So a lot of the protocols

unwisely,

reach into the event loop. So we do have we we do have some plans around that.

Among them, there's, cubes,

which is a library that's,

written kind of on top of, but you could actually decouple it and use it, however you

want, which is supposed to be this hierarchical

protocol framework,

that is really supposed to just,

be bytes in, bytes out and not think about event loop. And the other thing is that we are really happy to throw away

all the protocol parsing code we have. We have a really good library.

So we're pretty much in the process of throwing away a lot of our HTP implementation

in favor of the,

projects from the,

hyper Python,

Python project. So h 2 and h 1 1.

I think we we never had an implementation

of, h t p 2, and we just decided to not have an implementation of h t p 2. We use,

h 2, and we the way that's visit supports h h t p 2 is just by using the San Diego library.

And we're in the process of basically removing the h t p 1 1 implementation,

again, in favor of just using the, h 1 1 library.

So I think the biggest plan is that these things just don't belong in Twisted,

and,

Twisted will happily use any San Diego library that it can in in favor of instead of writing its own implementation.

I I think in the old days, it was harder, but, Python packaging has gotten a lot better nowadays, so we don't have any problem just depending on whatever libraries we need. Given how long Twisted has been around,

it's easy to sometimes

allow it to fade into the background

and forget that it is there and has played such an important role in the Python community for a number of years. So what are some of the places that people

are using Twisted

that aren't necessarily

the first thing that people think about or that are maybe not quite as visible?

So it's it's always these open source projects.

Tracking usage is is a little bit tricky. So often, the only way we know that people use it is,

either they report the part of the mailing list or they,

hire 1 of us to help them.

So I don't do security.

Pretty much use uses Twisted for all their web facing stuff, so including, like, the, actual security stuff. And,

Apple uses Twisted for their,

calendar server. So if you have an iPhone and your iPhone has a calendar on it and it's correct, then Twister is involved in that story.

And the other the the the newest,

user of Twister is Metrics, which is supposed to be the kind of new replacement to IRC and and, like, really more than I am, decentralized, and all the good stuff. And among other things, the French government,

have decided to to standardize the metrics to be the internal governmental,

communication tool. So now Twist, I guess, is gonna be in charge of, securing all of France's government's con con communications.

It's a big responsibility.

Yeah.

And what are some of the ways that people can support

the twisted project and the people working on it or get involved

and help to continue to bring it forward?

So there's a few things. 1st, we're always,

seem to be bottlenecked

on, having good reviews.

As I said, we don't allow any codes,

into the master branch without without the peer review, and that is a good way to get in because often people are a little bit worried about writing code for the project. But even if you're completely new to Python, just making sure that the code is clear and that that it's obvious that it's doing the right thing is is is great. So just

hoping a review

is is is super useful to the project.

Of course, if you want to fix bugs and twisted, we're more than happy to to help. The the mailing list and IRC channel

was always,

some of us happy to to guide if there's an issue. And the other thing is if you work for a company that use Twisted

and they would like to support us financially,

Twisted has a record of using financial contributions very efficiently.

A few tens of 1, 000 of dollars a year

is usually enough for us to manage to find,

someone to work, on Twisted

for a year. And, basically, when we get that kind of funding,

JP Calderon

and Amber Brown have done it in the past, and they've done, like,

really a lot of stuff on cleaning up,

twisted and making it much better. So if you if you do have a if you are working for a company that use Twisted and and you they want to become a Twisted sponsor, we use money very efficiently, and we are tax deductible.

And what are some of the plans that you have for the future of Twisted?

So some of the plans are really, really good integration with asyncio.

We've already shown some,

some proof of concept about, like, literally running the a AIO HTTP server on top of the recent event loop, which is useful for some use cases.

But we want to do,

much deeper integration, so a lot of people are working on that. And then 1 of the other things where we keep working on is actually taking

bits from

like I said, Twisted, in the old days, packaging was really hard,

so Trisid became very big because,

we didn't want to break it into multiple packages because every extra dependency was a huge pain.

Now with modern PIPI,

dependencies

are are a breeze. So we try to take

bits of Twisted that could stand alone as a library

and break them out into their own library. We move the code from Twisted

and and and have Twisted depend on them.

So a couple of the things we've done it with is the URL support.

Now it's, a hyper project called hyperlink, and now it really doesn't have any URL support. It just use a hyperlink URL support, and we hope to basically, eventually, not have anything that's really cold twisted. Maybe it's gonna be a meta package that just depends on a lot of packages,

but, that's gonna be a slow process. But every single bit we can break out and make into its own library,

end up being a huge game. So that's kind of 1 of the long term plans for, how this is going to evolve.

And are there any other aspects of Twisted

or your work with the project

or anything related to it that we should discuss further before we start to close out the show? No. I think we covered everything I had to say. Alright. Well, for anybody who wants to keep in touch with you and follow the work that you're doing and that is happening with Twisted. I'll have you add your preferred contact information to the show notes.

And with that, I'll move us on into the picks. And this week, I'm going to choose the Leatherman wave plier tool. I've been eyeballing it for a while now, and I finally picked 1 up. And it's a pretty great little multi tool.

It's pretty compact, has a lot of great little

tools built into it, and I've been enjoying using that. Actually,

came in handy a few times last week when I was building a deck, so, it's great to have on hand. So with that, I'll pass it to you, Moshe. Do you have any picks for us this week?

Yes. So I wanted to,

recommend 1 of my favorite,

online books,

called unsung book. It's kind of hard to describe, but the easiest way I found to describe it, it's urban fantasy,

but based on, Jewish mysticism

as the fantasy layer. And

it's a really great book, and I think, not to spoil it too much, but I think that's a really good bit there for programmers.

So, I highly recommend if anyone finds that kind of stuff fun, they should check out the book. Alright. Well, thank you very much for taking the time to join me today and discuss the work that you're doing with Twisted. It's a very interesting library and 1 that has been a crucial piece of the Python ecosystem for a long time. So thank you for that, and I hope you enjoy the rest of your day. Yep. Thank you. Too, Jackie. And, bye.

The Python Podcast.init

Summary

Preface

Interview

Keep In Touch

Picks

Links

The Python Podcast.__init__