Summary
Real-time communication over the internet is an amazing feat of modern engineering. The protocol that powers a majority of video calling platforms is WebRTC. In this episode Jeremy Lainé explains why he wrote a Python implementation of this protocol in the form of AIORTC. He also discusses how it works, how you can use it in your own projects, and what he has planned for the future.
Preface
- Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
- When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. With 200 Gbit/s private networking, scalable shared block storage, node balancers, and a 40 Gbit/s public network, all controlled by a brand new API you’ve got everything you need to scale up. Go to pythonpodcast.com/linode to get a $20 credit and launch a new server in under a minute.
- And to keep track of how your team is progressing on building new features and squashing bugs, you need a project management system designed by software engineers, for software engineers. Clubhouse lets you craft a workflow that fits your style, including per-team tasks, cross-project epics, a large suite of pre-built integrations, and a simple API for crafting your own. Podcast.__init__ listeners get 2 months free on any plan by going to pythonpodcast.com/clubhouse today and signing up for a trial.
- Visit the site to subscribe to the show, sign up for the newsletter, and read the show notes. And if you have any questions, comments, or suggestions I would love to hear them. You can reach me on Twitter at @Podcast__init__ or email hosts@podcastinit.com)
- To help other people find the show please leave a review on iTunes, or Google Play Music, tell your friends and co-workers, and share it on social media.
- Join the community in the new Zulip chat workspace at pythonpodcast.com/chat
- Your host as usual is Tobias Macey and today I’m interviewing Jeremy Lainé about AIORTC, an asynchronous implementation of the WebRTC and ObjectRTC protocols in Python
Interview
- Introductions
- How did you get introduced to Python?
- Can you start by explaining what the WebRTC and ObjectRTC protocols are?
- What are some of the main use cases for these protocols?
- What is AIORTC and what was your motivation for creating it?
- How does it compare to other implementations of the RTC protocols?
- Why do you think there haven’t been any other Python implementations?
- What are some of the benefits of having a Python implementation of the RTC protocol?
- How is AIORTC implemented?
- What have been some of the most difficult or challenging aspects of implementing a WebRTC compliant library?
- What are some of the most interesting or useful lessons that you have learned in the process?
- What is involved in building an application on top of AIORTC?
- What would be required to integrate AIORTC into an existing application built with something such as Flask or Django?
- What are some of the most interesting uses of AIORTC that you have seen?
- What are some of the projects that you would like to build with AIORTC?
- What are some cases where it would make more sense to use a different library or framework for your WebRTC projects?
- What are your plans for the future of AIORTC?
Keep In Touch
- jlaine on GitHub
- Website
- @JeremyLaine on Twitter
Picks
- Tobias
- Jeremy
Links
- AIORTC
- WebRTC
- Electrical Engineering
- [C](https://en.wikipedia.org/wiki/C_(programming_language)?utm_source=rss&utm_medium=rss
- C++
- PHP
- Ruby
- STUN (Session Traversal Utilities for NAT)
- TURN (Traversal Using Relays around NAT)
- ICE (Internet Connectivity Establishment)
- TLS (Transport Layer Security)
- RTP (Real-time Transport Protocol)
- Zencastr
- Jitsi
- RawRTC
- AsyncIO
- AIOICE
- Cryptography
- OpenCV
- PyAV
- FFMPEG
- Edge Detection
- Asterisk
- Raspberry Pi
- Datagram Transport Security
- Mozilla
- Augmented Reality
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
[00:00:12]
Unknown:
Hello, and welcome to podcast thought in it, the podcast about Python and the people who make it great. When you're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it, so check out Linode. With 200 gigabit private networking, scalable shared block storage, node balancers, and a 40 gigabit public network, all controlled by a brand new API, you've got everything you need to scale up. Go to python podcast.com/linode today to get a $20 credit and launch a new server in under a minute. And to keep track of how your team is progressing on building new features and squashing bugs, you need a project management system designed by software engineers for software engineers.
Clubhouse lets you craft a workflow that fits your style, including per team tasks, cross project epics, a large suite of prebuilt integrations, and a simple API for crafting your own. Podcast.init listeners get 2 months free on any plan by going to pythonpod cast.com/clubhouse today and signing up for a free trial. And visit the site at python podcast com to subscribe to the show, sign up for the newsletter, and read the show notes, and keep the conversation going at python podcast.com/chat.
[00:01:25] Unknown:
Registration for PyCon US, the largest annual gathering across the community, is open now. So don't forget to get your ticket, and I'll see you there. Your host as usual is Tobias Macy, and today I'm interviewing Jeremy Linne about AIORTC, an asynchronous implementation of the WebRTC and object RTC protocols at Python. So, Jeremy, could you start by introducing yourself?
[00:01:47] Unknown:
Sure. I'm nowadays, I'm a software engineer even though that's not what I initially trained as. I initially trained in electrical engineering and mathematics. But nowadays, well, my day job and my hobbies seem to have converged towards software development.
[00:02:03] Unknown:
Yeah. It's funny how many people who start off in some form of the sciences or engineering all end up converging into software at some level. Yeah. And I have to say that
[00:02:13] Unknown:
open source was a big factor in this for me because I was I I mean, I was tinkering with open source software since my university days. And well, I discovered that that's really kind of what I wanted to do on a day to day basis. And that's how I became a software engineer. Just like every company is ultimately going to be a software company, whether that's their main, business concern or not. Yeah. I guess that applies at personal level too.
[00:02:40] Unknown:
And do you remember how you first got introduced to Python? I do.
[00:02:44] Unknown:
Well, this was back in 2007. I was as, until then, I had been developing mostly either in low level c or c plus plus languages or for web for the web, I was using the dreaded PHP. And in 2007, when I joined a company, we were looking for at some higher level languages. And so at that time, what I did was that we were looking to, implement a webmail. And so well, I went and implemented 1 in Python, which I was discovering at the time, and 1 in Ruby, which I which I was also discovering at the time. And, well, once I'd implemented both, I kind of took a step back and decided that, well, Python was definitely the language I wanted to stick to. And so
[00:03:37] Unknown:
at some point along your journey, you have obviously come up against the WebRTC protocol. So I'm wondering if you can discuss a bit about what the WebRTC and ObjectRTC protocols are and some of the major use cases for that protocol.
[00:03:53] Unknown:
Okay. Well, WebRTC is both a set of protocols, and an API. So I'll explain this a bit more. It's a set of protocols in the sense that the basic building blocks for WebRTC are mostly well established, protocols such as well, the first thing you want to do is to find a path between 2 peers who want to exchange media or data. And so you will rely on what's called interactive connectivity establishment. On top of that, you're going to have a cryptographic layer, but with initially a detailless handshake. And on top of that, you're going to have sort of 2 stacks. 1 is going to be for media, so audio and video, and this is going to rely on encrypted RTP.
And on the other side, you're going to have the data channels. So the ability to exchange arbitrary binary data between your 2 peers. And all this is tied together. This is common to both WebRTC and ObjectRTC. And the the difference between these 2 is how the API is exposed to the to the to the programmer. Both WebRTC and ObjectRTC primarily target browsers. So they're what the specs define are JavaScript APIs. WebRTC kind of takes the, everything is going to revolve around a sort of central object called the peer connection. When as Object RTC kind of breaks this down into smaller, smaller objects, For instance, you'll get an object which represents the the encrypted transport or you'll get an object that represents a data channel, and you can instantiate all these, all these objects in a standalone fashion.
So the main I mean, the the use cases for WebRTC, I mean, there's a pretty wide range of of applications, but I think the key element is that it's really designed to be peer to peer. So you can exchange media or data directly from 1 browser to another, or you can exchange, again, media and data between a browser and a server or server to server. The the it gives you quite a wide, I mean, it gives you quite a wide range of of applications. You can you can be do you doing things like controlling robots, let's say, with data channels, or you could be, collecting video from webcams and bringing this either to a central server or giving access directly to anyone who has, who has a web browser.
There are really quite a wide range of exciting applications which you can, which you can build with WebRTC. I think 1 of the very interesting parts of WebRTC is that browser vendors have gone through the trouble of building this fairly complex stack and building it into all the into in straight into the browsers, which means that as a programmer, you get to interface with these fairly complex, protocols in a relatively simple, let's say, API.
[00:07:08] Unknown:
And in terms of some examples of applications that listeners might be familiar with that are implemented on top of WebRTC, I know are things such as Google Hangouts or the modern reimagining of Skype. And in fact, the application that you and I are using right now to record this conversation is implemented on top of WebRTC, and I believe takes advantage of, a wide swath of its capabilities in terms of being able to do peer to peer exchange of audio. And then at the end of our conversation, it uses the, binary transfer capability for being able to, upload the audio from your computer to a shared storage location. So as you mentioned, there's a wide variety of ways that it can be used, and there are, I'm sure, a number of applications that people use day to day that take advantage of this technology without them necessarily being explicitly aware of it. Right. I mean, if you're using any kind of
[00:08:06] Unknown:
web conferencing software, there's a high probability that it's being backed by WebRTC. Whether it's screen sharing or exchanging voice and video, these are all ideal use cases for WebRTC.
[00:08:21] Unknown:
And I'm sure we'll discuss a bit more of some of the nuance of the protocol and implementation details as we discuss the work that you've been doing on the aiortc library. So I'm wondering if you can spend a bit of time talking about what that is and your motivation for creating it in the first place. Right. So
[00:08:43] Unknown:
AIO RTC is, so as you mentioned, is, a Python implementation of WebRTC. So the full stack, including, you know, the low level networking stuff through the crypto layer and all the way up to actually handling handling media exchange. My initial use case for for my motivation for writing AORTC, was that I wanted a scriptable test tool for testing a production WebRTC server. WebRTC has, I mean, as I as we mentioned, is implemented in the browsers. So for interactive sessions, that's fine. But I really wanted to have something which I which I could script, run-in an automatic fashion. And that's how I really started tinkering, with the guts of WebRTC and
[00:09:38] Unknown:
implementing it in Python. Yeah. And I almost wish that I had come across your project maybe a year or 2 ago because I had an idea for an application that was going to leverage WebRTC. But at the time, all I could find were some fairly complex and, inconsistently documented frameworks for being able to implement it both in the browser or in the server side. So I'm wondering if you can spend a bit of time discussing how aioRTC compares to some of the other implementations of the RTC protocols that people may have come across.
[00:10:13] Unknown:
Sure. The most well established code base which relates to WebRTC is the 1 that's available at webrtc.org. This forms the basis both of Chrome's implementation of WebRTC and to some extent Firefox's. This is a codebase is written in c plus plus. It's been heavily battle tested because this is what most, most users, are, you know, are using when the when they're using WebRTC on the web. I think it serves that purpose well of being the engine that that runs WebRTC inside browsers, but it also it's also being used to create native applications using WebRTC, or server side applications.
And for that use case, I think it's not quite as quite as well as adapted because it's a very complex code base. It's got a lot of abstraction layers, which, implementers can tweak and tinker tinker with when they're when they're doing the embedding. And the end result is that if you're unless you're really familiar with the the the intricacies of the build system and how all this code is structured, this is a code base, which is I mean, integrating this code base into your own custom project is really a full time job. So I was trying to do 2 things. 1 thing was to gain a deeper insight into how this whole stack works. And the other 1, the other goal was to have a simpler implementation, which is more fun to hack on and to integrate into your own pet projects.
I think I think that, in this respect, a o AIO RTC it's a bit of a mouthful. It is. AIO RTC has served this purpose well. The project is now well, it's going to come up for its first birthday soon. So it's it's still relatively young project, but from the amount of feedback I've had from GitHub users, it's, it's obviously a niche which had been overlooked and which is now, now pretty well addressed. In terms of other WebRTC implementations, there's also raw RTC. So raw RTC, which is, an implementation, which is written in c, which is also fairly easy to, let's say, to integrate into a project, but it is focused exclusively on data channels. So it won't do things like audio and video for you. I've also spotted another young, let's say, WebRTC implementation, which is, I believe written in in Go. And what was kind of happy to see that there there are places in the code where they draw inspiration from the work I did on AIORTC.
[00:13:20] Unknown:
Why do you think it is that there haven't been any other Python implementations or at least none that have gained any measure of visibility?
[00:13:30] Unknown:
I think that the fairly recent availability of asyncio in Python was an important enabler because it allows you to, let's say, to to mix IO bound and CPU bound code in a simpler fashion than relying, you know, relying just on a multi in multi threading. So I think that's 1 1 explanation for why why we haven't seen Python implementation so far. Another reason is just it's it's a very deep stack, So it does take quite a lot of commitment to go, I mean, from the the the to to to build this from the ground up. 1 of the first things I had to implement, was just an the interactive connectivity establishment library, which I spun out into a separate library, which is called aioice.
And that was really the first the first block of code that I started working on. What this gives you is is the ability to, let's say, exchange a set of IP addresses between 2 peers and find, you know, test all the the different combinations between these sets of IP addresses, discover some additional, some additional IP addresses through the use of STUN, and, you know, just get a data path between 2 points. That's really the first step in implementing, the WebRTC, stack. Building upon that, there were some other blocks which were missing. I I used the cryptography, library, and I I really enjoyed interacting with the folks.
There were a couple of of missing building blocks in there, and they were they were very happy to review and eventually merge my my pool request. So really great project, extremely well documented, have the unit tested, just a kind of code I like.
[00:15:30] Unknown:
Sure. No problem. And so you said that your original reason for starting down this path of building AIO RTC was to have an easily scripted interface for being able to interact with other RTC implementations. And so I was just wondering if you, now that you have gotten the library to a state of usability, have actually completed that original goal or if that was lost along the way as you were going down the rabbit hole of getting this entire stack built up? It was definitely a risk of of abandoning the original purpose, but, no, it's actually completed,
[00:16:06] Unknown:
and it's where it's working. So that original purpose has been served since then. Well, I'm kind of enjoying being able to look at, at slightly more exciting use cases. I had not had a chance to to play much with, things like OpenCV, which really gives you some some exciting building blocks to to to do computer vision or image processing. So that's something I've been I've been spending more time on recently. Another area, let's say, of interest which has sprung up, along the way is, being able to read and record video, to media files.
And so along the way, I stumbled upon pyav, which is binding to, to the FFmpeg libraries. Well, that's been an amazing journey because, well, I liked it so much that nowadays, AIO, RTCs, video and audio frames, classes are, well, are directly those of, PyAV. So if you don't know this project, I would really recommend having a look at it. It it allows you to do things like, you know, multiplexing them. It's, what it really gives you the full power of FMmpeg
[00:17:37] Unknown:
in a really Pythonic API. Yeah. I'll definitely have to take a look at that because I have a script of my own right now that just uses a fairly low level binding, to FFmpeg for doing some simple processing. So I'll take a look at pyav and maybe I'll set my sights a little higher. And,
[00:17:54] Unknown:
if you're just looking at transcoding anything that the command line FFmpeg would do, just stick to whatever you're you're doing at the moment. But if you if you want to actually bring some, let's say, business logic into the the, into the actual processing of of the the audio, then IAV is really a very cool project. And a shout out to Mike Bors who runs the project. And your mention of being able to process
[00:18:25] Unknown:
aiortc, reminds me of as I mentioned, I had a project in mind a while ago that I wish this had been around for. And maybe I'll revisit it now, but I had had the idea of as part of a WebRTC conversation, being able to record the audio or video onto a server for later, redistribution or analysis or processing. So, this definitely seems like it would be very useful for anybody who has similar use cases. Yep. Absolutely. Well, AIO RTC,
[00:18:57] Unknown:
comes with a number of examples, and some of these examples do illustrate what you were mentioning, recording media to a file. There are also examples which will illustrate how to read media from a file. For example, recently, I was, streaming big buck bunny to my browser, from from another from another computer, and it's, I mean, it's it's it's really fun, all these, all these applications where you can really interact directly with the media. Some other examples which are included, feature, the use of OpenCV. So 1 of the examples you can run, the server example in AIORTC, will capture comes in 2 part. There's a back end part written in Python and a very thin part which runs in a browser. And so you can fire up the Python process and then point your browser, to to an HTTP endpoint.
And from there, you can start recording from your webcam and your microphone, send the these media streams to the Python process, which will at the moment, the the demo does edge detection on the on every frame, every video frame it receives, and it sends it back to the browser. So you can have real time edge detection on your video sent back to your to your browser. And
[00:20:25] Unknown:
that leads me to a question about the architectural design of a WebRTC application that implements the RTC protocol in the Python layer versus the JavaScript layer where I know that a number of, for instance, video or audio chat applications rely on implementing that logic in JavaScript. So I'm just curious how you would end up building an application that does use the browser, but implements the RTC protocol and processing logic at the Python layer? Well, any WebRTC
[00:21:00] Unknown:
project is is basically going to have 2 main components. 1 is going to be what you use to, I mean, what are the media endpoints? So that those will be the entities which are actually speaking WebRTC. And so the different setups you can have are you might have a browser talking to a browser or browser talking to a server, for instance, maybe asterisk if you want to reach out to the plain old telephony system. And, you're also going to have a signaling some kind of signaling server. 1 thing which the WebRTC specifications have purposefully left out is how you achieve signaling, which is how do you put these 2 parties in touch? How do you get them to exchange their, IP addresses, the the the codex they they support?
And if you if you search on the web for Python and WebRTC, most of the the hits you're going to come up with deal with signaling, which and so it's probably going to be something like WebSocket server, which, which just relays messages from 1 core participant to another. I feel I'm not exactly addressing your your your questions, but if you're talking to a browser, you're going to be using the browser's WebRTC stack no no matter what. So let's say you can leverage the fact that you're using maybe Python on the server side to do additional processing on the media. But, really, if all you want is to relay media untouched and without capturing the media or doing or, I mean, doing anything additional, you know, the most efficient setup is still going to be, having a browser talking to a browser and just using the the WebRTC stacks built into the browser. I think the interesting use cases for for using Python, if you want to do some additional processing on on on the media or on the the the data channels. There are some other use cases which are interesting.
For instance, if you want to talk to devices which do not feature a browser, Let's say, Internet of things applications. You could have, for instance, Raspberry Pi, which is running ao AIO RTC, which is maybe capturing video from a from a webcam and possibly you'd be controlling this little device using data channels. This is something that aiortc allows you to do. You do not need to have a browser on your your Raspberry Pi. You can just have your Python process.
[00:23:44] Unknown:
And in terms of the implementation of AIO RTC, I'm wondering if you can talk through the internal architecture and how that has changed as you've gone through different iterations and revisions go from when you first started to where you're at today?
[00:24:01] Unknown:
Sure. 1 of the first questions that that came up when writing this code was, what is the public API going to look like? I've really completely mimicked the the the JavaScript, API. So in that respect, I mean, you'll have camel case camel case names in the the APIs. It may not be the most Pythonic in that respect. I I favored familiarity over, you know, having something which was pure purely Pythonic at this stage. The general knowledge of WebRTC is not that great. So I thought it would be a good service to the API users to have an API which they recognize.
That being said, another, let's say, design decision was that the internal objects are going to look mostly like object RTC, ORTC. So as as I mentioned earlier, the the difference between WebRTC and ORTC is that ORTC gives you more elementary objects which you can manipulate. For instance, you might have an object which encamps encapsulates the transport layer and the encryption layer. And on top of that, you're going to, let's say, create a data channel or, create an RTP sender or receiver if you were exchanging media. The internals of AIO RTC went through a couple of iterations for for 1 simple reason, which was that I was still discovering asyncio at the time.
And I originally, well, I I had quite a bit of trouble finding efficient patterns, let's say, in using when using asyncio. 1 problem I had was that I was using too many queues in the code. There were several places within the code where I would take packets, put them into a queue, and then, in another coroutine, extract these these packets and do some some more processing. And the problem was this was, I had latency building up. So I revisited the these these choices. And nowadays, there is in the internal code, there is more callback based code, which may not be what you think of when you think of asyncio. But the user the user interface does use code the user facing API does use coroutines.
So you benefit from the simplicity of manipulating coroutines as opposed to callbacks. 1 of the the challenges in implementing aiortc was that the the standards upon which you rely, the the specifications are well, they're scattered all over the place because, as as I said, WebRTC builds upon some existing protocols such as, you know, interactive connectivity establishment, datagram transport security, and art the good old RTP real time protocol for for media. And so when implementing a WebRTC stack, you've really got to juggle between all these different standards and,
[00:27:18] Unknown:
well, understand how they fit together. And are there any sorts of test harnesses or test endpoints that you can use for being able to ensure that you're fully compliant with the WebRTC with the WebRTC specification? Or is the specification itself so broadly defined and flexible that that would be almost meaningless in terms of establishing full compliance?
[00:27:43] Unknown:
The best I mean, the the the best real world test, when implementing WebRTC is really just to to to put your code, face it, get it to a level where you can start talking with a browser, whether it's, Chrome or Firefox. Firefox tends to be more standards com compliant in my experience. The tester and the the automatic test suites are, you know, different difficult to to leverage because they're going to be JavaScript based. So they will take their the, the existing test suites out there are mostly to test a full browser against either itself or another browser, but there isn't really a ready made cross language, test framework you can rely on. Now that being said, once I'd lock down, you know, the the the behavior facing a well known endpoint such as another browser, I've invested a lot of time into unit testing, and AIO RTC.
[00:28:43] Unknown:
Well, nowadays, every single line of code in AIO RTC is exercised by at least 1 test case. And what have been some of the most interesting or unexpected or useful lessons that you've learned in the AIORTC and WebRTC oriented applications?
[00:29:02] Unknown:
That's a good question. I think well, I think 1 of the the 1 of the really positive lessons I I've drawn from from this exercise was that interacting with the Mozilla people is really fantastic is really fantastic. These, these are people who've who are both present in the standardization process and who've gone through the trouble of implementing WebRTC, and they're very open to discussion with other open source maintainers. So that's really I mean, for me, that's been 1 of the great parts of the this adventure. I also discovered, let's say, that, these standards are still pretty much living standards, and there's still ongoing standardization, what people are thinking about, what the next steps might be.
The experience from implementing WebRTC yourself does give you quite an insight into into the standardization process. That's also been a very valuable
[00:29:58] Unknown:
experience. And for somebody who's interested in implementing their own WebRTC oriented application, what would be involved in getting started with leveraging AIO RTC as part of that stack?
[00:30:11] Unknown:
Well, I'd say the the the best thing is to just pip install a I o r t c, and then head over to our GitHub repository. There are 6 or 7 fully featured examples. I think you're you're quite likely to find at least 1 example which looks like what you're what you want to do. So you can just fire up these examples. They come with documentation, and then I'd say start tinkering from there. And if somebody has an existing application,
[00:30:43] Unknown:
maybe built on top of Django or Flask or something like that, Are there is there any easy way or what would be involved in integrating WebRTC into an application like that? Or is that a fairly meaningless question?
[00:30:59] Unknown:
It's not meaningless because I think that if you have an existing web application, it's probably going to tie into the signaling layer. So this let's say the web application you're building might be, let's say, some conferencing software, and you'll have users who have sessions on your application. And you're probably going to want to use that to put your users in touch with each other. But you're probably going to have, another process running, which will actually handle the media streams or the the data streams. So I think you're really going to have 2 parts to your application. They can all be they can all be running Python, but you're probably going to have a split between the web part, which is probably going to be request and response based as opposed to a a long running a long running process, which is going to handle the actual flow of media and data. And what are some of the most interesting or unexpected uses of the a I o r t c library that you've seen so far?
I think 1 of the the the interesting use cases I saw was for, was for collecting video from stores, as in physical points of sale. What interested me in that project was the the scale of the project because we're talking about running aiortc on a fleet of thousands of devices. So I was pretty impressed that someone would actually, go and pick up this fairly young young project
[00:32:34] Unknown:
and roll it out at such a scale. And are there any projects that you have been pondering or that you would like to see others implement that would be able to make effective use of AIORTC?
[00:32:47] Unknown:
I think at the moment, 1 of the the, let's say, exciting areas of software engineering, which I'd like to see converge with the AIORTC, is related to augmented reality. I think there's, an opportunity to, you know, leverage the the big computing power you can get on a server side and which is not at the moment available on devices. I think you can offload some of the processing by sending by sending the video to to a server, which would be running AIO RTC. And I think there are some pretty exciting use cases in that area. I haven't personally had enough time to experiment with that, but,
[00:33:27] Unknown:
that would probably make for a very sexy project. Yeah. That would definitely be interesting. And particularly, if you you're running it on a local device, not necessarily an Android or an iPhone because deploying Python to those is still difficult and a a fraud path. But for maybe some other sort of, small form factor computer being able to input the video stream, process it locally, and then render that out to a display and have the AORTC essentially as a middleman in in that flow. I definitely think that that could be an interesting approach.
[00:34:02] Unknown:
Sure. As I said earlier, I think having this implementation which does not rely on a web browser does, open up use cases to more more kinds of devices, than than those we know which have an actual user interface on them. So connected objects, your whatever your favorite Raspberry Pi based project is. I think AIORTC fits in there quite nicely.
[00:34:31] Unknown:
And what are some of the areas that would be able to take advantage of WebRTC or ObjectRTC that would not be a good fit for the aioRTC library particularly?
[00:34:44] Unknown:
Well, I think that if you're if you're targeting a mobile device, you're probably going to to want to leverage either the built in browser or a native application because, well, you know, the cost of the the the complexity of deploying Python at this stage is, is probably a bit too much. Other than that, if you're looking at extremely extremely constrained devices, well, maybe there are, smaller libraries. For instance, if you want to just to do just to have the the data channel features of WebRTC, you're probably not want going to want to put in all the codex involved in the media part. And at the moment, aaortc doesn't allow you to, you know, pick just, some parts of it. It comes as a fully featured package, which currently depends on FFmpeg.
So in that respect, it does drag in some dependencies which are fine for a server laptop or, or even, you know, even something as powerful as a Raspberry Pi. But if you're talking about, micro controllers, you're probably not going to want to to to run a r t c.
[00:36:02] Unknown:
And what do you have planned for the future of AIORTC? Are there any particular new features or improvements or bug fixes that you're excited to release?
[00:36:14] Unknown:
Well, the last, published release of, AIO RTC came with 0 features. It was all about performance improvements and reducing the memory footprint. That's probably something I'm going to want to continue. Other than that, I think that just keeping up with the the evolving specs is a work in in itself, and I definitely want to stay on stay on top of that. For instance, the, at the moment, if you want to add, media streams or, you know, have a very complex setup where you have lots of different media streams, you're adding some, moving some. That's not going to work at the moment with a I o r t c, and it's, it's an area which I'm currently working on.
[00:37:04] Unknown:
And is there any particular type of help or contributions or community engagement that you're looking for or that would be most useful to you at this stage?
[00:37:15] Unknown:
I think that, including more examples into AIO RTC is, would be 1 area with which I would really appreciate help because, because, I mean, I've got some other ideas of of, applications, but I'm sure that users have plenty plenty of other creative ones. And I think that's 1 area where they could really help. Possibly possibly, getting started guide. At the moment, there's a lot of document a lot of API documentation, but I think there's a need for some more introductory material to get to get users who are maybe not familiar with our WebRTC to get them started on their first project, and that would also be that would also be really appreciated.
[00:38:08] Unknown:
And are there any other aspects of the aioRTC project or the WebRTC specification or anything else that we didn't cover yet which you think we should cover before we close out the show?
[00:38:21] Unknown:
Yeah. Well, 1 kind of fun aspect of implementing AIO RTC has been crashing browsers. As, you know, a lot of the browsers rely on the same code base. And it's interesting to see that, if you put the effort into developing your own WebRTC stack, you're going to find edge cases which have not been exercised so far. So I think there's room for more, more WebRTC implementations. I think for Python, we're quite well served by AIO RTC. But I think the more implementations there are, the the more robust WebRTC will become.
[00:39:09] Unknown:
Alright. Well, for anybody who wants to follow the work that you're up to or get in touch, I'll have you add your preferred contact information to the show notes. And with that, I'll move us into the picks. And today, I'm going to choose a band that I've been listening to recently called Tanger Cavalry, which is a very interesting cross section of music. They, the the way that they summarize themselves is that they are nomadic folk metal. So there's an interesting array of music, musical instruments and styles. It's worth taking a look at. I've been enjoying that. So with that, I'll pass it to you, Jeremy. Do you have any picks this week?
[00:39:46] Unknown:
For anyone who's interested in both Python and audio and video, I'd really encourage you to go and check out the pyav project. Mike Bors has done absolutely excellent work on it.
[00:39:58] Unknown:
Alright. Well, thank you very much for taking the time today to join me and discuss the work that you've done with AIORTC. As I said, perhaps I'll revitalize my old ideas for building some applications on top of it. So thank you again for your work on that and for your time, and I hope you enjoy the rest of your day.
[00:40:17] Unknown:
Well, thank you very much. I'm looking forward to your next pull request on AIO RTC.
Hello, and welcome to podcast thought in it, the podcast about Python and the people who make it great. When you're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it, so check out Linode. With 200 gigabit private networking, scalable shared block storage, node balancers, and a 40 gigabit public network, all controlled by a brand new API, you've got everything you need to scale up. Go to python podcast.com/linode today to get a $20 credit and launch a new server in under a minute. And to keep track of how your team is progressing on building new features and squashing bugs, you need a project management system designed by software engineers for software engineers.
Clubhouse lets you craft a workflow that fits your style, including per team tasks, cross project epics, a large suite of prebuilt integrations, and a simple API for crafting your own. Podcast.init listeners get 2 months free on any plan by going to pythonpod cast.com/clubhouse today and signing up for a free trial. And visit the site at python podcast com to subscribe to the show, sign up for the newsletter, and read the show notes, and keep the conversation going at python podcast.com/chat.
[00:01:25] Unknown:
Registration for PyCon US, the largest annual gathering across the community, is open now. So don't forget to get your ticket, and I'll see you there. Your host as usual is Tobias Macy, and today I'm interviewing Jeremy Linne about AIORTC, an asynchronous implementation of the WebRTC and object RTC protocols at Python. So, Jeremy, could you start by introducing yourself?
[00:01:47] Unknown:
Sure. I'm nowadays, I'm a software engineer even though that's not what I initially trained as. I initially trained in electrical engineering and mathematics. But nowadays, well, my day job and my hobbies seem to have converged towards software development.
[00:02:03] Unknown:
Yeah. It's funny how many people who start off in some form of the sciences or engineering all end up converging into software at some level. Yeah. And I have to say that
[00:02:13] Unknown:
open source was a big factor in this for me because I was I I mean, I was tinkering with open source software since my university days. And well, I discovered that that's really kind of what I wanted to do on a day to day basis. And that's how I became a software engineer. Just like every company is ultimately going to be a software company, whether that's their main, business concern or not. Yeah. I guess that applies at personal level too.
[00:02:40] Unknown:
And do you remember how you first got introduced to Python? I do.
[00:02:44] Unknown:
Well, this was back in 2007. I was as, until then, I had been developing mostly either in low level c or c plus plus languages or for web for the web, I was using the dreaded PHP. And in 2007, when I joined a company, we were looking for at some higher level languages. And so at that time, what I did was that we were looking to, implement a webmail. And so well, I went and implemented 1 in Python, which I was discovering at the time, and 1 in Ruby, which I which I was also discovering at the time. And, well, once I'd implemented both, I kind of took a step back and decided that, well, Python was definitely the language I wanted to stick to. And so
[00:03:37] Unknown:
at some point along your journey, you have obviously come up against the WebRTC protocol. So I'm wondering if you can discuss a bit about what the WebRTC and ObjectRTC protocols are and some of the major use cases for that protocol.
[00:03:53] Unknown:
Okay. Well, WebRTC is both a set of protocols, and an API. So I'll explain this a bit more. It's a set of protocols in the sense that the basic building blocks for WebRTC are mostly well established, protocols such as well, the first thing you want to do is to find a path between 2 peers who want to exchange media or data. And so you will rely on what's called interactive connectivity establishment. On top of that, you're going to have a cryptographic layer, but with initially a detailless handshake. And on top of that, you're going to have sort of 2 stacks. 1 is going to be for media, so audio and video, and this is going to rely on encrypted RTP.
And on the other side, you're going to have the data channels. So the ability to exchange arbitrary binary data between your 2 peers. And all this is tied together. This is common to both WebRTC and ObjectRTC. And the the difference between these 2 is how the API is exposed to the to the to the programmer. Both WebRTC and ObjectRTC primarily target browsers. So they're what the specs define are JavaScript APIs. WebRTC kind of takes the, everything is going to revolve around a sort of central object called the peer connection. When as Object RTC kind of breaks this down into smaller, smaller objects, For instance, you'll get an object which represents the the encrypted transport or you'll get an object that represents a data channel, and you can instantiate all these, all these objects in a standalone fashion.
So the main I mean, the the use cases for WebRTC, I mean, there's a pretty wide range of of applications, but I think the key element is that it's really designed to be peer to peer. So you can exchange media or data directly from 1 browser to another, or you can exchange, again, media and data between a browser and a server or server to server. The the it gives you quite a wide, I mean, it gives you quite a wide range of of applications. You can you can be do you doing things like controlling robots, let's say, with data channels, or you could be, collecting video from webcams and bringing this either to a central server or giving access directly to anyone who has, who has a web browser.
There are really quite a wide range of exciting applications which you can, which you can build with WebRTC. I think 1 of the very interesting parts of WebRTC is that browser vendors have gone through the trouble of building this fairly complex stack and building it into all the into in straight into the browsers, which means that as a programmer, you get to interface with these fairly complex, protocols in a relatively simple, let's say, API.
[00:07:08] Unknown:
And in terms of some examples of applications that listeners might be familiar with that are implemented on top of WebRTC, I know are things such as Google Hangouts or the modern reimagining of Skype. And in fact, the application that you and I are using right now to record this conversation is implemented on top of WebRTC, and I believe takes advantage of, a wide swath of its capabilities in terms of being able to do peer to peer exchange of audio. And then at the end of our conversation, it uses the, binary transfer capability for being able to, upload the audio from your computer to a shared storage location. So as you mentioned, there's a wide variety of ways that it can be used, and there are, I'm sure, a number of applications that people use day to day that take advantage of this technology without them necessarily being explicitly aware of it. Right. I mean, if you're using any kind of
[00:08:06] Unknown:
web conferencing software, there's a high probability that it's being backed by WebRTC. Whether it's screen sharing or exchanging voice and video, these are all ideal use cases for WebRTC.
[00:08:21] Unknown:
And I'm sure we'll discuss a bit more of some of the nuance of the protocol and implementation details as we discuss the work that you've been doing on the aiortc library. So I'm wondering if you can spend a bit of time talking about what that is and your motivation for creating it in the first place. Right. So
[00:08:43] Unknown:
AIO RTC is, so as you mentioned, is, a Python implementation of WebRTC. So the full stack, including, you know, the low level networking stuff through the crypto layer and all the way up to actually handling handling media exchange. My initial use case for for my motivation for writing AORTC, was that I wanted a scriptable test tool for testing a production WebRTC server. WebRTC has, I mean, as I as we mentioned, is implemented in the browsers. So for interactive sessions, that's fine. But I really wanted to have something which I which I could script, run-in an automatic fashion. And that's how I really started tinkering, with the guts of WebRTC and
[00:09:38] Unknown:
implementing it in Python. Yeah. And I almost wish that I had come across your project maybe a year or 2 ago because I had an idea for an application that was going to leverage WebRTC. But at the time, all I could find were some fairly complex and, inconsistently documented frameworks for being able to implement it both in the browser or in the server side. So I'm wondering if you can spend a bit of time discussing how aioRTC compares to some of the other implementations of the RTC protocols that people may have come across.
[00:10:13] Unknown:
Sure. The most well established code base which relates to WebRTC is the 1 that's available at webrtc.org. This forms the basis both of Chrome's implementation of WebRTC and to some extent Firefox's. This is a codebase is written in c plus plus. It's been heavily battle tested because this is what most, most users, are, you know, are using when the when they're using WebRTC on the web. I think it serves that purpose well of being the engine that that runs WebRTC inside browsers, but it also it's also being used to create native applications using WebRTC, or server side applications.
And for that use case, I think it's not quite as quite as well as adapted because it's a very complex code base. It's got a lot of abstraction layers, which, implementers can tweak and tinker tinker with when they're when they're doing the embedding. And the end result is that if you're unless you're really familiar with the the the intricacies of the build system and how all this code is structured, this is a code base, which is I mean, integrating this code base into your own custom project is really a full time job. So I was trying to do 2 things. 1 thing was to gain a deeper insight into how this whole stack works. And the other 1, the other goal was to have a simpler implementation, which is more fun to hack on and to integrate into your own pet projects.
I think I think that, in this respect, a o AIO RTC it's a bit of a mouthful. It is. AIO RTC has served this purpose well. The project is now well, it's going to come up for its first birthday soon. So it's it's still relatively young project, but from the amount of feedback I've had from GitHub users, it's, it's obviously a niche which had been overlooked and which is now, now pretty well addressed. In terms of other WebRTC implementations, there's also raw RTC. So raw RTC, which is, an implementation, which is written in c, which is also fairly easy to, let's say, to integrate into a project, but it is focused exclusively on data channels. So it won't do things like audio and video for you. I've also spotted another young, let's say, WebRTC implementation, which is, I believe written in in Go. And what was kind of happy to see that there there are places in the code where they draw inspiration from the work I did on AIORTC.
[00:13:20] Unknown:
Why do you think it is that there haven't been any other Python implementations or at least none that have gained any measure of visibility?
[00:13:30] Unknown:
I think that the fairly recent availability of asyncio in Python was an important enabler because it allows you to, let's say, to to mix IO bound and CPU bound code in a simpler fashion than relying, you know, relying just on a multi in multi threading. So I think that's 1 1 explanation for why why we haven't seen Python implementation so far. Another reason is just it's it's a very deep stack, So it does take quite a lot of commitment to go, I mean, from the the the to to to build this from the ground up. 1 of the first things I had to implement, was just an the interactive connectivity establishment library, which I spun out into a separate library, which is called aioice.
And that was really the first the first block of code that I started working on. What this gives you is is the ability to, let's say, exchange a set of IP addresses between 2 peers and find, you know, test all the the different combinations between these sets of IP addresses, discover some additional, some additional IP addresses through the use of STUN, and, you know, just get a data path between 2 points. That's really the first step in implementing, the WebRTC, stack. Building upon that, there were some other blocks which were missing. I I used the cryptography, library, and I I really enjoyed interacting with the folks.
There were a couple of of missing building blocks in there, and they were they were very happy to review and eventually merge my my pool request. So really great project, extremely well documented, have the unit tested, just a kind of code I like.
[00:15:30] Unknown:
Sure. No problem. And so you said that your original reason for starting down this path of building AIO RTC was to have an easily scripted interface for being able to interact with other RTC implementations. And so I was just wondering if you, now that you have gotten the library to a state of usability, have actually completed that original goal or if that was lost along the way as you were going down the rabbit hole of getting this entire stack built up? It was definitely a risk of of abandoning the original purpose, but, no, it's actually completed,
[00:16:06] Unknown:
and it's where it's working. So that original purpose has been served since then. Well, I'm kind of enjoying being able to look at, at slightly more exciting use cases. I had not had a chance to to play much with, things like OpenCV, which really gives you some some exciting building blocks to to to do computer vision or image processing. So that's something I've been I've been spending more time on recently. Another area, let's say, of interest which has sprung up, along the way is, being able to read and record video, to media files.
And so along the way, I stumbled upon pyav, which is binding to, to the FFmpeg libraries. Well, that's been an amazing journey because, well, I liked it so much that nowadays, AIO, RTCs, video and audio frames, classes are, well, are directly those of, PyAV. So if you don't know this project, I would really recommend having a look at it. It it allows you to do things like, you know, multiplexing them. It's, what it really gives you the full power of FMmpeg
[00:17:37] Unknown:
in a really Pythonic API. Yeah. I'll definitely have to take a look at that because I have a script of my own right now that just uses a fairly low level binding, to FFmpeg for doing some simple processing. So I'll take a look at pyav and maybe I'll set my sights a little higher. And,
[00:17:54] Unknown:
if you're just looking at transcoding anything that the command line FFmpeg would do, just stick to whatever you're you're doing at the moment. But if you if you want to actually bring some, let's say, business logic into the the, into the actual processing of of the the audio, then IAV is really a very cool project. And a shout out to Mike Bors who runs the project. And your mention of being able to process
[00:18:25] Unknown:
aiortc, reminds me of as I mentioned, I had a project in mind a while ago that I wish this had been around for. And maybe I'll revisit it now, but I had had the idea of as part of a WebRTC conversation, being able to record the audio or video onto a server for later, redistribution or analysis or processing. So, this definitely seems like it would be very useful for anybody who has similar use cases. Yep. Absolutely. Well, AIO RTC,
[00:18:57] Unknown:
comes with a number of examples, and some of these examples do illustrate what you were mentioning, recording media to a file. There are also examples which will illustrate how to read media from a file. For example, recently, I was, streaming big buck bunny to my browser, from from another from another computer, and it's, I mean, it's it's it's really fun, all these, all these applications where you can really interact directly with the media. Some other examples which are included, feature, the use of OpenCV. So 1 of the examples you can run, the server example in AIORTC, will capture comes in 2 part. There's a back end part written in Python and a very thin part which runs in a browser. And so you can fire up the Python process and then point your browser, to to an HTTP endpoint.
And from there, you can start recording from your webcam and your microphone, send the these media streams to the Python process, which will at the moment, the the demo does edge detection on the on every frame, every video frame it receives, and it sends it back to the browser. So you can have real time edge detection on your video sent back to your to your browser. And
[00:20:25] Unknown:
that leads me to a question about the architectural design of a WebRTC application that implements the RTC protocol in the Python layer versus the JavaScript layer where I know that a number of, for instance, video or audio chat applications rely on implementing that logic in JavaScript. So I'm just curious how you would end up building an application that does use the browser, but implements the RTC protocol and processing logic at the Python layer? Well, any WebRTC
[00:21:00] Unknown:
project is is basically going to have 2 main components. 1 is going to be what you use to, I mean, what are the media endpoints? So that those will be the entities which are actually speaking WebRTC. And so the different setups you can have are you might have a browser talking to a browser or browser talking to a server, for instance, maybe asterisk if you want to reach out to the plain old telephony system. And, you're also going to have a signaling some kind of signaling server. 1 thing which the WebRTC specifications have purposefully left out is how you achieve signaling, which is how do you put these 2 parties in touch? How do you get them to exchange their, IP addresses, the the the codex they they support?
And if you if you search on the web for Python and WebRTC, most of the the hits you're going to come up with deal with signaling, which and so it's probably going to be something like WebSocket server, which, which just relays messages from 1 core participant to another. I feel I'm not exactly addressing your your your questions, but if you're talking to a browser, you're going to be using the browser's WebRTC stack no no matter what. So let's say you can leverage the fact that you're using maybe Python on the server side to do additional processing on the media. But, really, if all you want is to relay media untouched and without capturing the media or doing or, I mean, doing anything additional, you know, the most efficient setup is still going to be, having a browser talking to a browser and just using the the WebRTC stacks built into the browser. I think the interesting use cases for for using Python, if you want to do some additional processing on on on the media or on the the the data channels. There are some other use cases which are interesting.
For instance, if you want to talk to devices which do not feature a browser, Let's say, Internet of things applications. You could have, for instance, Raspberry Pi, which is running ao AIO RTC, which is maybe capturing video from a from a webcam and possibly you'd be controlling this little device using data channels. This is something that aiortc allows you to do. You do not need to have a browser on your your Raspberry Pi. You can just have your Python process.
[00:23:44] Unknown:
And in terms of the implementation of AIO RTC, I'm wondering if you can talk through the internal architecture and how that has changed as you've gone through different iterations and revisions go from when you first started to where you're at today?
[00:24:01] Unknown:
Sure. 1 of the first questions that that came up when writing this code was, what is the public API going to look like? I've really completely mimicked the the the JavaScript, API. So in that respect, I mean, you'll have camel case camel case names in the the APIs. It may not be the most Pythonic in that respect. I I favored familiarity over, you know, having something which was pure purely Pythonic at this stage. The general knowledge of WebRTC is not that great. So I thought it would be a good service to the API users to have an API which they recognize.
That being said, another, let's say, design decision was that the internal objects are going to look mostly like object RTC, ORTC. So as as I mentioned earlier, the the difference between WebRTC and ORTC is that ORTC gives you more elementary objects which you can manipulate. For instance, you might have an object which encamps encapsulates the transport layer and the encryption layer. And on top of that, you're going to, let's say, create a data channel or, create an RTP sender or receiver if you were exchanging media. The internals of AIO RTC went through a couple of iterations for for 1 simple reason, which was that I was still discovering asyncio at the time.
And I originally, well, I I had quite a bit of trouble finding efficient patterns, let's say, in using when using asyncio. 1 problem I had was that I was using too many queues in the code. There were several places within the code where I would take packets, put them into a queue, and then, in another coroutine, extract these these packets and do some some more processing. And the problem was this was, I had latency building up. So I revisited the these these choices. And nowadays, there is in the internal code, there is more callback based code, which may not be what you think of when you think of asyncio. But the user the user interface does use code the user facing API does use coroutines.
So you benefit from the simplicity of manipulating coroutines as opposed to callbacks. 1 of the the challenges in implementing aiortc was that the the standards upon which you rely, the the specifications are well, they're scattered all over the place because, as as I said, WebRTC builds upon some existing protocols such as, you know, interactive connectivity establishment, datagram transport security, and art the good old RTP real time protocol for for media. And so when implementing a WebRTC stack, you've really got to juggle between all these different standards and,
[00:27:18] Unknown:
well, understand how they fit together. And are there any sorts of test harnesses or test endpoints that you can use for being able to ensure that you're fully compliant with the WebRTC with the WebRTC specification? Or is the specification itself so broadly defined and flexible that that would be almost meaningless in terms of establishing full compliance?
[00:27:43] Unknown:
The best I mean, the the the best real world test, when implementing WebRTC is really just to to to put your code, face it, get it to a level where you can start talking with a browser, whether it's, Chrome or Firefox. Firefox tends to be more standards com compliant in my experience. The tester and the the automatic test suites are, you know, different difficult to to leverage because they're going to be JavaScript based. So they will take their the, the existing test suites out there are mostly to test a full browser against either itself or another browser, but there isn't really a ready made cross language, test framework you can rely on. Now that being said, once I'd lock down, you know, the the the behavior facing a well known endpoint such as another browser, I've invested a lot of time into unit testing, and AIO RTC.
[00:28:43] Unknown:
Well, nowadays, every single line of code in AIO RTC is exercised by at least 1 test case. And what have been some of the most interesting or unexpected or useful lessons that you've learned in the AIORTC and WebRTC oriented applications?
[00:29:02] Unknown:
That's a good question. I think well, I think 1 of the the 1 of the really positive lessons I I've drawn from from this exercise was that interacting with the Mozilla people is really fantastic is really fantastic. These, these are people who've who are both present in the standardization process and who've gone through the trouble of implementing WebRTC, and they're very open to discussion with other open source maintainers. So that's really I mean, for me, that's been 1 of the great parts of the this adventure. I also discovered, let's say, that, these standards are still pretty much living standards, and there's still ongoing standardization, what people are thinking about, what the next steps might be.
The experience from implementing WebRTC yourself does give you quite an insight into into the standardization process. That's also been a very valuable
[00:29:58] Unknown:
experience. And for somebody who's interested in implementing their own WebRTC oriented application, what would be involved in getting started with leveraging AIO RTC as part of that stack?
[00:30:11] Unknown:
Well, I'd say the the the best thing is to just pip install a I o r t c, and then head over to our GitHub repository. There are 6 or 7 fully featured examples. I think you're you're quite likely to find at least 1 example which looks like what you're what you want to do. So you can just fire up these examples. They come with documentation, and then I'd say start tinkering from there. And if somebody has an existing application,
[00:30:43] Unknown:
maybe built on top of Django or Flask or something like that, Are there is there any easy way or what would be involved in integrating WebRTC into an application like that? Or is that a fairly meaningless question?
[00:30:59] Unknown:
It's not meaningless because I think that if you have an existing web application, it's probably going to tie into the signaling layer. So this let's say the web application you're building might be, let's say, some conferencing software, and you'll have users who have sessions on your application. And you're probably going to want to use that to put your users in touch with each other. But you're probably going to have, another process running, which will actually handle the media streams or the the data streams. So I think you're really going to have 2 parts to your application. They can all be they can all be running Python, but you're probably going to have a split between the web part, which is probably going to be request and response based as opposed to a a long running a long running process, which is going to handle the actual flow of media and data. And what are some of the most interesting or unexpected uses of the a I o r t c library that you've seen so far?
I think 1 of the the the interesting use cases I saw was for, was for collecting video from stores, as in physical points of sale. What interested me in that project was the the scale of the project because we're talking about running aiortc on a fleet of thousands of devices. So I was pretty impressed that someone would actually, go and pick up this fairly young young project
[00:32:34] Unknown:
and roll it out at such a scale. And are there any projects that you have been pondering or that you would like to see others implement that would be able to make effective use of AIORTC?
[00:32:47] Unknown:
I think at the moment, 1 of the the, let's say, exciting areas of software engineering, which I'd like to see converge with the AIORTC, is related to augmented reality. I think there's, an opportunity to, you know, leverage the the big computing power you can get on a server side and which is not at the moment available on devices. I think you can offload some of the processing by sending by sending the video to to a server, which would be running AIO RTC. And I think there are some pretty exciting use cases in that area. I haven't personally had enough time to experiment with that, but,
[00:33:27] Unknown:
that would probably make for a very sexy project. Yeah. That would definitely be interesting. And particularly, if you you're running it on a local device, not necessarily an Android or an iPhone because deploying Python to those is still difficult and a a fraud path. But for maybe some other sort of, small form factor computer being able to input the video stream, process it locally, and then render that out to a display and have the AORTC essentially as a middleman in in that flow. I definitely think that that could be an interesting approach.
[00:34:02] Unknown:
Sure. As I said earlier, I think having this implementation which does not rely on a web browser does, open up use cases to more more kinds of devices, than than those we know which have an actual user interface on them. So connected objects, your whatever your favorite Raspberry Pi based project is. I think AIORTC fits in there quite nicely.
[00:34:31] Unknown:
And what are some of the areas that would be able to take advantage of WebRTC or ObjectRTC that would not be a good fit for the aioRTC library particularly?
[00:34:44] Unknown:
Well, I think that if you're if you're targeting a mobile device, you're probably going to to want to leverage either the built in browser or a native application because, well, you know, the cost of the the the complexity of deploying Python at this stage is, is probably a bit too much. Other than that, if you're looking at extremely extremely constrained devices, well, maybe there are, smaller libraries. For instance, if you want to just to do just to have the the data channel features of WebRTC, you're probably not want going to want to put in all the codex involved in the media part. And at the moment, aaortc doesn't allow you to, you know, pick just, some parts of it. It comes as a fully featured package, which currently depends on FFmpeg.
So in that respect, it does drag in some dependencies which are fine for a server laptop or, or even, you know, even something as powerful as a Raspberry Pi. But if you're talking about, micro controllers, you're probably not going to want to to to run a r t c.
[00:36:02] Unknown:
And what do you have planned for the future of AIORTC? Are there any particular new features or improvements or bug fixes that you're excited to release?
[00:36:14] Unknown:
Well, the last, published release of, AIO RTC came with 0 features. It was all about performance improvements and reducing the memory footprint. That's probably something I'm going to want to continue. Other than that, I think that just keeping up with the the evolving specs is a work in in itself, and I definitely want to stay on stay on top of that. For instance, the, at the moment, if you want to add, media streams or, you know, have a very complex setup where you have lots of different media streams, you're adding some, moving some. That's not going to work at the moment with a I o r t c, and it's, it's an area which I'm currently working on.
[00:37:04] Unknown:
And is there any particular type of help or contributions or community engagement that you're looking for or that would be most useful to you at this stage?
[00:37:15] Unknown:
I think that, including more examples into AIO RTC is, would be 1 area with which I would really appreciate help because, because, I mean, I've got some other ideas of of, applications, but I'm sure that users have plenty plenty of other creative ones. And I think that's 1 area where they could really help. Possibly possibly, getting started guide. At the moment, there's a lot of document a lot of API documentation, but I think there's a need for some more introductory material to get to get users who are maybe not familiar with our WebRTC to get them started on their first project, and that would also be that would also be really appreciated.
[00:38:08] Unknown:
And are there any other aspects of the aioRTC project or the WebRTC specification or anything else that we didn't cover yet which you think we should cover before we close out the show?
[00:38:21] Unknown:
Yeah. Well, 1 kind of fun aspect of implementing AIO RTC has been crashing browsers. As, you know, a lot of the browsers rely on the same code base. And it's interesting to see that, if you put the effort into developing your own WebRTC stack, you're going to find edge cases which have not been exercised so far. So I think there's room for more, more WebRTC implementations. I think for Python, we're quite well served by AIO RTC. But I think the more implementations there are, the the more robust WebRTC will become.
[00:39:09] Unknown:
Alright. Well, for anybody who wants to follow the work that you're up to or get in touch, I'll have you add your preferred contact information to the show notes. And with that, I'll move us into the picks. And today, I'm going to choose a band that I've been listening to recently called Tanger Cavalry, which is a very interesting cross section of music. They, the the way that they summarize themselves is that they are nomadic folk metal. So there's an interesting array of music, musical instruments and styles. It's worth taking a look at. I've been enjoying that. So with that, I'll pass it to you, Jeremy. Do you have any picks this week?
[00:39:46] Unknown:
For anyone who's interested in both Python and audio and video, I'd really encourage you to go and check out the pyav project. Mike Bors has done absolutely excellent work on it.
[00:39:58] Unknown:
Alright. Well, thank you very much for taking the time today to join me and discuss the work that you've done with AIORTC. As I said, perhaps I'll revitalize my old ideas for building some applications on top of it. So thank you again for your work on that and for your time, and I hope you enjoy the rest of your day.
[00:40:17] Unknown:
Well, thank you very much. I'm looking forward to your next pull request on AIO RTC.
Introduction and Guest Introduction
Jeremy Linne's Background and Journey to Python
Introduction to WebRTC and ObjectRTC Protocols
Use Cases and Applications of WebRTC
Motivation and Development of AIO RTC
Internal Architecture and Design Decisions
Getting Started with AIO RTC
Future Plans and Community Contributions
Final Thoughts and Closing Remarks
