Summary
For any program that is used by more than one person you need a way to control identity and permissions. There are myriad solutions to that problem, but most of them are tied to a specific framework. Yosai is a flexible, general purpose framework for managing role-based access to your applications that has been decoupled from the underlying platform. This week the author of Yosai, Darin Gordon, joins us to talk about why he started it, his experience porting it from Java, and where he hopes to take it in the future.
Preface
- Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
- I would like to thank everyone who supports us on Patreon. Your contributions help to make the show sustainable.
- When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at www.podastinit.com/linode?utm_source=rss&utm_medium=rss and get a $20 credit to try out their fast and reliable Linux virtual servers for running your awesome app.
- Visit the site to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
- To help other people find the show please leave a review on iTunes, or Google Play Music, tell your friends and co-workers, and share it on social media.
- Your host as usual is Tobias Macey and today I’m interviewing
Darin Gordon about Yosai, a security framework for Python applications
Interview
- Introductions
- How did you get introduced to Python?
- What is Yosai and what is the problem that you were trying to solve when you started it?
- How does Yosai compare to existing libraries for web frameworks such as Flask-Security or Django Guardian and why might someone choose Yosai instead?
- In the documentation it mentions that Yosai is a port of the Apache Shiro framework from Java to Python. What was most difficult about exposing a Pythonic interface while maintaining the core principles of the original?
- Authentication and authorization are difficult problem domains and can cause significant issues if they are not implemented in a secure fashion. How do you ensure an appropriate level of quality in Yosai to be confident having people use it?
- To start can you describe how the framework is architected and what is involved in integrating it with a project?
- Outside of the context of web applications, what are some situations where someone should consider integrating authentication and authorization into their project?
- What have been some of the most challenging aspects of building the Yosai project?
- Tell us about the Rust extension you wrote earlier this year
- What do you have planned for the future of Yosai?
Keep In Touch
- Website
- GitHub
- @darin_gordon on Twitter
Picks
- Tobias
- Darin
- The Asphalt Framework. Asphalt is an asyncio-based microframework for network oriented applications.
Links
- Yosai Project Web Page
- Github Repo
- RBAC
- Apache Shiro
- TOTP
- Pyramid
- SOLID
- Builder Pattern
- POJO
- Corey Benfield
- Hyper HTTP/2 Library
- Passlib
- Hugo
- MKDocs
- YAML
- Middleware
- IoT
- Authz in Rust
- PyO3
- Snaek
- PyCon Canada
- PyCascades
- JSON Web Tokens
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
[00:00:14]
Unknown:
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. I would like to thank everyone who supports us on Patreon. Your contributions help to make the show sustainable. When you're ready to launch your next project, you'll need somewhere to deploy it, so you should check out linode at ww w.podcastinnit.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your app or experimenting with something that you hear about on the show. You can visit the site at www.podcastinit.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. To help other people find the show, please leave a review on Itunes or Google Play Music, tell your friends and coworkers, and share it on social media. Your host, as usual, is Tobias Macy. And today, I'm interviewing Darren Gordon about YoSci, a security framework for Python applications.
So Darren, could you please introduce yourself?
[00:01:00] Unknown:
Hey, everyone. It's, it's an honor to to be here and to speak, before such a great audience. I've been working for with Python for almost 4 years now. I do have a tech background, but working in financial services, for almost 10 years, I I went from a programmer position into something in product development and risk management and, margin financing and prime brokers. So I've done a variety of things, and, I'm back working again full time as a you know, with programming, for my own venture. And, I wanted to give back, by working on, the YoSai project. And, so I'm really happy to be here today and to tell everyone, more about the project.
[00:01:43] Unknown:
And do you remember how you first got introduced to Python?
[00:01:46] Unknown:
Well, for me, it was a strategic decision. 4 years ago, I had an important decision to make. What language will I use as my primary language to build the first at least the first version of the services for my venture? I'm doing something entrepreneurial. So, I consider Java, Ruby, Node. Js, and Python, and Python won, because basically with Python you can go further. It's a powerful general purpose language, I can use it to create services and I can use it for analysis. It's a really powerful analytical tool. It's got a great diverse ecosystem of open source projects that I could adopt for my work.
I can optimize it and extend it to run quickly. And I'd say the most important reason why I chose Python is because there's just a huge global community of people who love to use the language. They're passionate about it. And, they're willing to help other people like me use it better. So I've gotten a lot of help along the way because of that. And, again, you know, I felt indebted to somehow find a way to give back and I'm trying to do that now with, with this work.
[00:02:51] Unknown:
And so that's a good opportunity to describe a bit about what YoSai is and the problem that you're trying to solve when you first started it.
[00:03:00] Unknown:
Sure. So YoSight, it began as a proprietary project, you know, for internal use. But, I decided to open source this project. You know, I'm working on a venture now where application security, it's a it's a top priority. Once I decided that I'd use an authorization policy based on role based access control, I looked for an open source solution written in Python that I could adopt to enforce RBAC. The problem is no such library really exists in Python, but I did find 1 in Java. It's it's called Apache Shiro. Now, the problem is I didn't want to use Java in my work. I want to use primarily Python and I didn't want to extend Python with, with this library, so I figured, why not port Apache Hero to Python?
And so I did, and the result is YoSai. YoSai is a security framework written in Python that is based on a version 2 alpha implementation of Apache Shiro, which is written in Java. It offers authentication, authorization, and session management from a common API. In other words, Eosai helps a developer verify who a user claims to be. It helps the developer control what the user can do in the system, and it helps you set time limits on how long a user may use the system or idle for before requiring that user to reverify its identity. Aside from just those 3 core services, some some other features I'd like to highlight are 2 factor authentication using time based 1 time passwords.
You can throttle authentication attempts, locking accounts after certain number failures. YoSight produces a full audit trail of every major event that's happened through the system, structuring, and it's it's it's producing it through structured logs, formatted as JSON. And, with that, it could be analyzed further downstream by other systems. And, another really useful feature is the user impersonation tool, which is for administration use where a developer or an admin of some some type in operations would be able to perform activities and sit in the system as if they were that user. All of this was really compelling for me. I I found it worthwhile to build the, to build the OSI.
[00:05:20] Unknown:
And so you mentioned that when you were first, sort of auditing the available libraries to fulfill the need for having that role based access control that you didn't find anything that suited it. So I know that at least in the web arena, there are things like Flask Security and Django Guardian that have a fairly granular capability for permission control. But it sounds like some of the requirement was that you needed something that was not necessarily web based and still had that same level of granularity. Is that accurate?
[00:05:49] Unknown:
More or less. Yes. It's I didn't exactly have a web requirement and what I found was that the web framework that I had chosen is Pyramid, and Pyramid has a really great, native authorization policy, in place. But, again, when I the way I got started was by thinking about how to structure permissions, using a role based access control policy. So after I decided on the policy, then I looked for the technology that would enforce it. And, you know, I wasn't completely married to using Pyramid, and so I did look at Flask and I did look at Django and I saw their plug ins and what was available. And the thing is, YoSai still offers a very compelling value proposition, even today despite what's out there, there are some differentiating parts here that still make it unique even despite almost 17 years of recreating the wheel across all these different major frameworks. And so a couple of those are, YoSai, it's it's a framework that you can easily customize. It's a security framework. It's fast if you have caching enabled, which you should by default. It offers a granular permission level authorization in such a way that none of these other plug plug ins do, and you can customize that permission further, which I have even for my own personal use.
And it could be easily adapted to any kind of application. It's not just for web apps. Although, the the vast majority of the apps that we're creating today are web apps, you know, if people want to use this say for, like a desktop application, a client that has a client server connection and it's using, let's say, RPC, well, you know, you could just use the OSI again for that and just write an adapter that knows how to to communicate with the server without having to recreate an entire library like this from scratch.
[00:07:41] Unknown:
Yeah. The overall approach of having it decoupled from the underlying use case is definitely very valuable. And it's something that, I know is a concern that the Python community has started to tackle in the protocol space as well, most notably with the work that Corey Benfield has been doing with HTTP 2 and having that completely separated from the actual IO, whereas in a lot of the other protocol libraries, the IO and the protocol handling itself have been intermingled, which reduces the ability for reuse within other contexts. So having that capability for security of being able to extract it from the underlying concern and have it purely as a, you know, essentially a protocol implementation is definitely valuable because it does give you that flexibility of being able to use it in other projects and even for being able to span across projects while still maintaining the underlying security definitions.
[00:08:31] Unknown:
Yeah. Definitely. It's valuable to take that approach, but, it's also reasonable not to, at least initially, because you're trying to solve your own immediate problems and you're not thinking strategically. It was much easier for someone like me to identify a project that had already done and to port it over than to think from scratch about how I would write or architect such a thing. So I I can I can understand why people didn't take that approach initially, but now the now such a project like YOSA exists, and, it seems like, you know, I'm hoping that other people find it useful for that purpose, and it kind of fills a gap, if you will? You know, addresses a need for something like this, something universal, and, I hope to use it in that way.
Now, in addition to just, you know, what makes YoSight unique, I I thought a bit about, you know, what kind of people would benefit by using it, and I came up with a few. Teams that have used other types of authorization such as resource based ACLs, but they've kind of grown beyond. It's gotten out of their control. It's become cumbersome. It's tedious to manage it the way that they have using groups and so forth. And they've learned about RBAC and maybe they want to adopt in another project. Another example is, developers who are maybe creating a new type of framework, like another web yet another web framework, which is very possible because of all the async IO work that's been going on. Let's say you're 1 of those, 1 of those leaders in tech and you you haven't even begun to think about how to, you know or what type of middleware to create. Well, you can use YOSI and write an adapter for it. So that's another good candidate. You could be a team that's maybe working on a new type a new a whole new project.
No authorization decisions have been made yet, and, you know, this RBAC thing might interest you. And, also, I say, finally, if you're part of a team that's using Java today and you're using Apache Shiro, well, YoSai is a port of Apache Shiro, so it's providing functionality that you're familiar with. So you'd be a good fit for this too.
[00:10:32] Unknown:
And 1 of the things that I've experienced a lot for people who come from Java or are trying to copy things from Java is that they get stuck in sort of the idioms of the other language. And so I'm curious as to what your experience was trying to take the underlying concepts from Shiro, but add a Pythonic feel to the overall final product.
[00:10:56] Unknown:
This has been quite an exercise. So it was pretty brutal porting thousands of lines of Java code, you know, across, like, dozens of classes. It's it's a pretty large project. So it was painful just pouring going through line by line, trying to make you know, understand some of the things that were being done. You know, I I hadn't been weren't been working every single day with Java. So there is a lot of searching involved, and I had to really get acclimated in a bit, to move forward with each step. But the approach I took was fairly common. My goal was to first make it work, and then I would make it right, and then I would make it fast. So this is a fairly common approach in software development. So what I did was in the first 2 versions was I ported everything over to Python.
I refactored some along the way, but starting in the the latest version, I really refactored a lot more, reengineered a lot more based on feedback, you know, know, a lot of flak ahead. I'd gone on the first 2 releases, but people didn't really understand where I was coming from. I didn't share them with them that, hey. I just wanted to make this work initially, and then I would improve it. It. But it's it's such a huge project. I couldn't really get my arms around it until, that first release. So, I did, and I've been doing lots of refactoring and reengineering since then. And, making it fast, the thing is, the OSI already offers, uses caching, and it's pretty quick already. But I've I've been exploring other ways using, for example, using, Rust extension, to potentially make it even faster. However, on the other side, there are ways to make it quicker even using PyPI, which I explored as well while I'm working with Rust. So, but overall, when I was porting the, the project, 1 of the greatest challenges I had was that I chose to port an unfinished alpha version. So version 1 of Shiro features, like, many levels of inheritance using mixins, whereas version 2 was, using a compositional design to replace all of that inheritance and using a compositional design to replace all of that inheritance. And so in version 2, there are far fewer moments of head scratching when you're trying to figure out how code works.
The problem that I ran into is that version 2 was unfinished. The main author of Shiro, he's the CTO of a startup, and he wasn't really available to answer questions I had about the new architecture. So it was out there. He said, hey, check this out. Check out version 2. I know you're working on it. And I did, but I was coming pretty much on my own. I had to figure out what he was thinking and just try to rationalize ways to finish, you know, completing the bridge, if you will, to the other side. And so I I ultimately, it was up to me to grab the bull by the horns and finish what I you know, what was starting, and and I did. So, where it is today is yosei is it works, and many parts are idiomatic Python. And it's pretty fast, but, you know, again, it can go a lot further. Another thing about porting Apache Shiro and probably 1 of the things 1 of the reasons 1 of the green lights I had I had sensed, if you will, when I was reading through it and trying to assess whether I'd be able to do this was that what made it easier to port it over was that the authors of SHIRO had used a a solid design had followed solid solid design principles, solid as in the acronym. I don't know if that that interests or that that that rings a bell for you. Yeah. I'm familiar with the acronym, and I'll add a link in the show notes for somebody who wants to sort of dig into it further. Okay. Sure. We could probably spend an entire hour, a good hour, just talking about how Shiro and YoSai follow the solid design principles, but, you know, they they do. You might and if you're familiar with Solid, you may see you may understand some of that just by reading through Yosai's code. And, you know, a a little bit about the refactoring where, you know, you start off just because, you could do something in Python doesn't mean it's right or that it's idiomatic, and, it was a learning experience for me. Sometimes, I I did things knowing that there were better ways to do it in Python, but I just had to get things done. And going back to it and refactoring and reengineering was, like you know, it kind of created a bit of a brain rush for me. You know, like, I I got a bit of a coders high using it because you know, or doing that because it felt really good to relieve myself of a lot of this sort of debt with Java. Here are a couple examples. So in Java, you have these if else conditions everywhere. You have to check a condition before you move forward. Whereas with, Python, you use try except blocks. In the way that people describe this is it's Java, you look before in Java, you look before you leap. Whereas in Python, you it's easier to ask for forgiveness than permission. So I ported tons of if else conditions to try accept blocks and, so that was, and in order to take the, take advantage of the, of of using Python.
And, you know, there are some design patterns that just weren't necessary in Python that came from Java. A couple examples include the builder design pattern. A builder design pattern can be just replaced using keyword args and, handling them accordingly in a in a function. Let's see. Custom exceptions. I kinda learned this the hard way, you know, I I ended up with way too many custom exceptions, and I didn't take full advantage of the exceptions in the, standard library. So, I was, got into a bit of a trouble. You know, people had criticized why I was doing that. And, frankly, I mean, initially, I didn't put much thought into it. I just wanted to make it work. And, so I did go back and clean that up a bit, and, I did find that most of the exceptions that were customized could be addressed with the exceptions in the standard library.
If, I only took the time to think a little bit about, what was available and how to apply them correctly. And I'd say another another Java pattern, that wasn't necessary, with Python, was that in Java, you have these POJOs. You have these just plain old Java objects, being used to store values and and in Python, you don't really need to do that. You don't need classes. You can use these primitive types like you can use named, tuple, or dicts, to store values and to pass them between functions. And so I did a lot of that in version 0.3. I cleaned up a lot of that and tried to take advantage of take advantage of what was available in Python. In in terms of regrets, porting this project, the 1 thing that really sticks out, if I can go back and change anything, there are probably too many iterations of unit testing. I went through too many iterations. I was too focused early on in trying to achieve this sort of maximal unit test coverage, when I should have waited for the dust to settle a little bit, so to speak, before actually unit testing. The thing is I thought good unit test coverage would encourage a lot of people to give you a try because bad test coverage would steer people away rather quickly. I didn't wanna, like, scare people away because there's enough scary parts of this, and at least initially, and I was trying to reduce that a bit. But the thing is I released 3 times, within a year, and there are a lot of changes that had broken tests. And so I was going back and cleaning up and changing tests 1 too many times. So that that was a bit of a a waste of time. And so, you know, if I can go back and do it again,
[00:18:18] Unknown:
I would just wait until unit I would unit test later on. Yeah. TDD works best when you know where you're going, not when you're trying to figure out what the destination happens to be.
[00:18:29] Unknown:
Yeah. I didn't really understand how all the parts worked. I did I couldn't see the forest from the trees, unfortunately. Anyone who's out there and if you're porting a project from 1 language to another, take it easy with the unit testing early on. Wait a bit.
[00:18:42] Unknown:
Yeah. When you're doing something like that, it's best to do more outside in testing where you start at the outside to make sure, okay, the perimeter of my system works the way that it needs to. And then as you sort of solidify the interior bit by bit, then you can work the tests into a more granular level. Yeah. It's much easier and you're much,
[00:18:58] Unknown:
you're probably more likely,
[00:19:00] Unknown:
to not have to break or or to to fix all those tests, you know, week by week. And 1 of the questions that came out of your description of porting the project is because of the fact that it does have this other code base that it is using as inspiration. I'm wondering if the intent is for you to track its progress as it continues to evolve or if the point at which you started work on YoSai was essentially just a forking point where YoSai will have its own destiny separate from, well, where wherever Shiro happens to go. Well, it seems that at this point that the Shiro,
[00:19:36] Unknown:
the people who are supporting Shiro, you know, primarily, the main author has stepped away. He's a CTO at a at a company now. So, yeah, he's more hands off than other people who are still supporting the, version 1. And so because I've really ported version 2 and that's incomplete and, they don't really seem anywhere in the near future ready to, you know, turn things over to version 2. I've this is this is a fork. It's a permanent change, and, like I said, I had to fill in a lot of gaps. So YoSai is its own creation now, and, really, the way that it's going to evolve over time is going to depend entirely based on the feedback from the Python community.
I I don't intend to look to anything that you know, I I can't, unfortunately, reference anything that is happening now in Shiro. And it really doesn't make sense to it. YoSai has taken on, a life of its own.
[00:20:25] Unknown:
And given that authentication and authorization are such broad and difficult problems to solve, and any issues with how it's implemented can potentially lead to significant impact in the projects that are relying on it for security. So I'm wondering, what are some of the techniques that you're using to ensure that there's an appropriate level of quality in YoSci itself for for other people to be confident using it to protect their applications?
[00:20:49] Unknown:
Yeah. Sure. It's a perfectly valid concern. But let me first begin by saying that I have not contracted or a qualified third party to audit YOSI, but welcome anyone out there who meets that criteria to please come in and evaluate the project. And currently, I'm not aware of anyone conducting such an evaluation, but I do believe that a qualified third party audit would be helpful. We could spend at least an hour talking about why I'm comfortable with, or confident about YoSai and why anyone else should be confident with it, but let me just touch on a few points here. So regarding authentication, so YoSai facilitates authentication.
I didn't roll my own crypt though. YoSai uses Passlib, the very popular Passlib library for the cryptographic hashing and password verification. How a password is hashed and persisted into long term storage, it's a far more risk prone exercise and it's an exercise that's outside of the scope of YoSai. YoSai queries and verifies credentials against those provided by a user. The moment that a credential is no longer needed, it's cleared from memory. Passwords are stored as byte arrays for this reason. Regarding authorization, a few points. YoSai denies access by default. A user is granted access only when that user is explicitly assigned to a role or assigned permission. IoSci verifies that the roles and permissions from a data store imply the roles and permissions are required to perform an activity. Permissions are queried, not created by Yosai. The permission authorization logic works exactly as that which is implemented in Apache Shiro. There's there's no functional difference. And that logic, it's been battle tested and it's been adopted by major firms.
Regarding sessions, sessions are also treated with care. A session ID changes when a user logs in. In web applications, session IDs are stored in cookies and sessions time out due to inactivity and absolute timeouts.
[00:22:38] Unknown:
Despite everything I've mentioned and why I'm comfortable with using EOSI, I nonetheless encourage anyone who's interested in adopting it to consider auditing it prior to production use. And you mentioned too the, sort of test coverage that you have for it. I know that you said that you have had some trouble with being able to have the tests keep up with the work that you're doing. But at this point, is there a fairly substantial level of coverage on the actual project itself to prevent any regressions
[00:23:06] Unknown:
from the current state? With each release, I have tried to achieve maximal test coverage. So when I had shared my grievances about test testing, I nonetheless did test. So, with each release and in the ReadMe, I try to make it very clear, about test coverage, and it's you know, I would consider it It's still at a very a very great level. It's I'm not trying to achieve a 100%, but I'm trying to achieve at least 90% across all the major modules, and so it it it offers that. And you alluded to the read me briefly, and I know that when I was doing research for this interview,
[00:23:39] Unknown:
I was digging through the documentation, which I must say is quite substantial and fairly well written. So I'm curious, 1, how you managed to achieve that level of coverage of the overall documentation, and 2, what sort of inspired you to actually put out that level of effort to make sure that you had that documentation to begin with? Oh, thank you. I appreciate it.
[00:24:00] Unknown:
It was weeks of effort, if not months of effort. And, you know, all of that is really above and beyond what I would have ever done, if I wasn't open sourcing in this and I want other people to use it. But because it's such an elaborate system, I felt it necessary to document as thoroughly as I could, and I tried to imagine how people the questions that people would have, and I tried to put it all together in that web page. You know, there's a full project page for for YoSai, and I try to summarize that in in the read to me and I try to create visualizations and things like that to, you know, just make it easier for people to understand, what's what's going on. It took a lot of work. There's no way around it. Writing technical documentation wasn't something I've ever enjoyed doing, but I found that anytime you write software that someone other than you is going to use, and if you don't wanna be fully roped into everything, you've got to teach them somehow. And, so you do that through documentation and tutorials and demos. And, I I created all of that for YoSight because, you know, I do I have already moved on to other projects, and I want people to find it useful. So I I I saw no other way. And so it sounds like you did all of that yourself. You didn't hire a technical writer to farm any of that out? Yeah. That's all me. Yeah. Whatever it is in my life, I I just I figure things out. I I guess it you know, my greatest strength is that, I I guess I I eventually can solve most of the problems thrown at me. And I'm willing to give it a try, as anxiety inducing as it really is. I've written all the documentation. I did manage to copy, however, some of it from the original Apache Shear website. I copied some of it, but I I really try to paraphrase and I really try to make things unique. I don't like copying other people's work. I do sometimes when I get absolutely lazy, but then when I do, I try to cite, you know, the original author as best I can. I'm not trying to get credit for stuff like this. So it I I do believe I'm giving credit where credit's due, but I did a lot of the you know, I did a lot of it on my own. With the help with, static site generators, I used, Hugo. I used the Hugo set static site generator, which is written in Go. I I tried the ones written in Python and I tried Jekyll, but I found Hugo and the the the, templates that were available, most useful.
To create the documentation, I used a Python library called MK Docs. I used MK Docs. It was it was it was actually great, because I didn't have to do hardly anything other than write markdown. And, the static site created all the documents for me, and it did very nicely. However, I also used, for the tutorial, I used Hugo, and, but that was only to create the time based 1 time password tutorial. So you're going to see some difference, but, the general idea is to, write your content in markdown and then find a static site generator that'll convert that markdown into a presentable web page. And,
[00:26:56] Unknown:
There's certainly no shortage of options on that front having, explored a number of them myself for various other use cases, both for blogging and for different projects I've been involved with. That's great. So I'm wondering if you can dig a bit into the way that YoSai itself is architected.
[00:27:12] Unknown:
Essentially, you install YoSai and its extensions from PyPI, and then you configure this YAML, y a m l, YAML settings file. During your application's initialization process, you know, during the bootstrap process, what you want to do is instantiate a YoSai instance telling it the location of that settings file, and then you store that YoSai Instance somewhere that it's accessible during every application call that may require YoSai related functionality. If you're building a web application, what I'm talking about is the registry that your middleware references. The next step is you write a piece of middleware that opens a context using the IoSi instance as a parameter and that's the only thing that the middleware needs to do is it opens up a context.
Now, from that point on, wherever we want to authenticate or manage session state, what we do is we create a subject object and we use it to log in a user or access the user's session. The main API, Yosai's main API for your typical developer, it's going it's accessible through this subject object. And the next thing is, aside so use the subject object to authenticate and to manage session state. You could also use it for authorization, but you don't necessarily need to. So let me explain. Wherever you want to authorize access, you have 2 options. You can use imperative style access control or declarative style. Imperative, you're creating a subject instance and you're making explicit calls to the Subject's authorization API. With a declarative style access control, you're decorating functions with authorization logic and allowing YOSI to handle all that boilerplate authorization logic for you. If anyone's interested about, you know, how this works in detail, please go to the documentation on the project website and, you're gonna see exactly what I'm talking about. But this is how YoSai works in a nutshell. You you create the instance, you put it somewhere, you open up a context, and from that point, you could use YoSai for authentication, authorization, and session management.
[00:29:11] Unknown:
And when you're not using YoSai as part of a web application, which is where, I, in particular, am most familiar with managing access control. I'm wondering what are some of the other types of situations where a user should consider integrating that authentication and authorization capability into their project?
[00:29:29] Unknown:
There's kind of a cop out, but, if you wherever you when you write an application where user identity matters and you want to control what that user can do, then your application would benefit by Yosai. The the example I had raised earlier is another viable example. It's a it's a client. It's an old sort of client server, app, you know, setup, except you've got, like, maybe a thick client application. And and the server doesn't necessarily have to be a web server. Maybe it's you're using RPC, but you still bent you you still need to know who the who's making the calls, who the user is, and and you wanna control what that user can do. Yeah. Another type of example that somebody might use is a sort of IoT situation where you've got a sensor that wants to be able to send information to a back end, but you wanna make sure that people can't just spam the back end with
[00:30:16] Unknown:
arbitrary false information. So you need to have some method of authenticating that sensor to the back end and controlling the types of information that it's allowed to send based on, you know, a a unit ID or something like that. Oh, yeah. Absolutely. And, you know, that that's you're we're just warming up. And, I mean, when we talk about IoT,
[00:30:34] Unknown:
you know, if people are running home servers and people are trying to access, you know, physical locations, you know, trying to open doors and so forth, Well, that server is going to have to know who that person is that's trying to enter, and, there's gonna be access control, you know, involved. So, yeah, if it might be a web application that's doing that. It may not be. It might be something else. And, if you're writing in Python, hey, you know, maybe you should use YoSci for it. So what have been some of the most challenging aspects of building your OSI? And what are some of the things that you learned that you weren't expecting to come across in the process? Yeah. Well, I I I touched on some of that already. The whole project was very challenging, but it was a it was a huge learning opportunity for me. You know? I was really cutting my teeth on a huge project, with this. You know, cutting my teeth with Python and, you know, there are a lot of moving parts and so forth. So it was, you know, the sheer, it's just a ginormous effort, you know. So I I would say it wasn't a individual thing that was making it hard. It was just all of them combined. Right? The documentation was really hard.
So let's just leave it at that. The docs were the hardest. Yes. So please read them.
[00:31:41] Unknown:
Well, I I have read the I've read a fair bit of them, so you've got at least 1 person who's benefited from them. So, just to be clear, I think. Thank you. And are there any sort of, unexpected use cases that you've seen people put your site to that you would like to call out? Not yet. None that I'm aware of yet. So it sounds like the sort of most valuable community contribution you could have right now is just having people use it and provide feedback from their experiences?
[00:32:06] Unknown:
Oh, yes. Absolutely. Give it a try. If you wanna write, if you wanna write adapters for for any of your existing frameworks, I would help you do it. I guarantee that together we would have it written and tested, within just a few hours. I haven't done it because it's not something I wanna own and manage going forward because I I'm probably not gonna use it unless it's unless it's for pyramid, and I've already released the pyramid adapter if you wanna see how I did it. But, you know, because I would like to see people use it and I would value your feedback, I'll help you. You know, I'll facilitate this by showing you how to write it. And you mentioned that 1 of the approaches that you've taken to try and speed up parts of YoSai is to actually write a Rust extension for it. So I'm wondering if you can just spend a bit of time talking about your experience with that, particularly given that Rust as a extension mechanism isn't fully supported by Core Python yet. But the thing is it's really picking up steam. Since the beginning of this year, things have really been moving quickly on the Rust front.
So early this year, I read a blog post by Armin Rocha about extending Python with Rust. He wrote about his experience and then he open sourced his work. This inspired me to give Rust a try. So I spent a few weeks learning the basics of Rust, asking a lot of questions on IRC, and and I need and I learned how to write a Python Rust extension. I thought, what better way to learn it than with a practical use case? And so what I did was I chose the part of of YoSai, for authorization that had the potential to become more computationally expensive because, you know, if you're gonna write something in Rust, you know, and you you ideally, you want it to run faster. So I thought, well, this part of code would benefit by that. And so I ported the entire, like, you know, small library for authorization, and the end result is, I managed to port all of it. It works in Rust. I got to learn some Rust. I open sourced the work, and I even wrote a tutorial about it. But the thing is I haven't given a talk about how to extend Python applications with Rust, and and I might in the near future, especially now because, there's some other really interesting things happening on the Rust front, with respects to integrating with Python. 2 projects that come to mind are Pythonium, trioxide, which is Py 0 3. The author just made his first release on that, a week ago. It's 1 way to write Python extensions, Python Rust extensions, and it's a 2 way sort of library. So if you want to access Python, from Rust, you could do it, or you could access Rust from Python.
His library allows something like that. And Armin Roacher, since that first project, learned from his experiences and has created an even better way to extend Python with Rust, and he's calling that snake, but spelled a little differently, s n a k. And, you know, I've had some chat with corresponds with, Armin, and, he's still fleshing some things out, making some important decisions about things such as memory management, but, I think that, you know, with all these there's a lot of interest that and there are a lot of people who seem to be working on ways to extend Python to Rust, and I wanted to give that a try. And so, I did.
And, of course, at the end of this, I I really want to know, well, how much faster how much faster how much, you know, how much faster is this Rust version of my of what I wrote in Python? And so I tried as best I could to come up with an apples to apples comparison. Just running the exact, logic on both parts and not trying and trying to eliminate any other overhead. And what I found was that, unsurprisingly, the the rust ran approximately, let's say, 10 times faster than CPython, but here's the thing. Then I want to know, well, what happens if I run this through pypy? And so I did and the PyPy, the compiled Python actually ran faster than the Rust extension. And I I was like, my mind was just blown away when I saw it. I was like, wow, this is great. This is this code would actually benefit by using PyPI. You know, why would I do this in Rust? But the thing is I I didn't I wouldn't have known that unless I got I went through this whole exercise. So, that was revealing for me from this, experience.
[00:36:20] Unknown:
Yeah. It's definitely interesting. And, if you are looking for a venue to talk about any of your work, I don't know if you are aware, but the call for proposals for PyCon Canada and PyCascades are open right now. So Okay. Noted. Yeah. I I would love to share my experience with this. I I'll you know, maybe elaborate on this story too. And what do you have planned for the future of EOSI? I know that you mentioned that you're not actively working on it at the moment because your attention is elsewhere, but it sounds like you're still committed to maintaining it and supporting it going forward. So I'm wondering if there are any sort of additional features that you're looking to implement or would like other people to help you with adding in. Yes. Absolutely. So, I think that YoSai would benefit by more refactoring and reengineering and trying to make it even more Pythonic. It's a great experience,
[00:37:06] Unknown:
trimming off the fat. You know, it's great experience making code more efficient, and I think YoSci still has, rooms for improvement. And, so there's that. And I I usually do bits and pieces of that with each release, but I if anyone would like to give it a try, you know, read through the source and see what you can come up with. I I would gladly, approve of changes as long as you're not breaking anything. What I plan to do, at least in the very next version, which I expect to have done by the end of this year, hopefully, is, I've already written support changes on the master branch to support the Rust extension. I plan to modularize the OSI so that if people just want to use the authorization component, they can without having to install all of the OSI and all of his other parts.
So modularization is another thing another project that I I plan to accomplish, and I want to make sure that, each of the parts work through PyPI. Another thing is I I'd like to add support for JSON Web Tokens. I'm very cautious about using JWT because of
[00:38:08] Unknown:
the experiences other people have had. So I'm still looking and researching how to follow best practices, and I'm really receptive to people who have had experience with JWT and I may be interested in making that work with the OSI. Alright. Yeah. I've actually got a blog post on my blog about my experience using it, a couple of years ago, but I'm sure that the overall best practice has been updated since then. I'll check it out. So are there any other topics that you think we should cover before we start to close out the show?
[00:38:38] Unknown:
I well, I've covered a lot of things here. I think this is a good start. But, I encourage people to check out the the GitHub repo and to check out the project web page.
[00:38:47] Unknown:
And for anybody who wants to contact you directly or follow the work that you're up to, I'll have you add your preferred contact information to the show notes. Sure. And with that, I'll bring us to the picks. And my pick this week is the Brain On podcast, which is a show that I started listening to with my kids recently, which is a podcast from 1 of the NPR stations that is a science show focused at kids in the, you know, probably 8 to 14 time range. And it's just a lot of fun to listen to. So some of the shows we've listened to are things like how the moon affects the tides, debates between the deep sea and deep space as far as what, you know, what is cooler. So just surfacing a lot of interesting content about that and, fire versus lasers. So it's just a fun show with a lot of whimsy and a lot of really valuable educational content. And with that, I'll pass it to you. What do you have for us for picks this week? So what I'm picking is a really great,
[00:39:44] Unknown:
project. You can see it out on GitHub. It's called Asphalt. It's the Asphalt framework. Asphalt is an asyncio based microframework for network oriented applications. It's written by Alex Gronholm and, he spends a lot of time, creating, you know, creating Python software correctly, making sure it's idiomatic. And Alex has been very focused on using the latest and greatest from the asyncio front, and,
[00:40:10] Unknown:
I'm very optimistic about where this project is going. And I encourage people to check out Asphalt. Alright. And as a, meta comment here, are there any of your own projects that you'd like to call out or try and get people interested in?
[00:40:23] Unknown:
Well, I'm now working on a, my own venture. I was working in financial services for almost 10 years and, you know, I've been it's all part of why I was learning Python and why I'm building Yosai and all these other projects. I am working on something and, it's been pretty lonely, but, you know, I've been trying to just get the the work done as a solo founder and, it's really out of necessity. So, you know, I'm an entrepreneur who's an avid contributor to the Python community, and, I'd love to network with people and connect with people and share with them what I'm working on. This is not a place, you know, for me to do it. I didn't wanna take advantage of our our discussion, but it's what I'm very passionate about. It's what's driving me to do what I'm doing. So, it's a major part of my life, and I love to talk about it.
[00:41:06] Unknown:
So I'd like to say thank you for taking the time out of your day to share your experience of working on EOSI and porting it from the Shira project. Definitely looks like an interesting, library and 1 that I plan to take advantage of in any future projects that I work on that require that authentication and authorization capability because it looks to be fairly comprehensive and flexible. So I appreciate that and I hope you enjoy the rest of your day. Thanks, Tobias. Bye, everyone.
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. I would like to thank everyone who supports us on Patreon. Your contributions help to make the show sustainable. When you're ready to launch your next project, you'll need somewhere to deploy it, so you should check out linode at ww w.podcastinnit.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your app or experimenting with something that you hear about on the show. You can visit the site at www.podcastinit.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. To help other people find the show, please leave a review on Itunes or Google Play Music, tell your friends and coworkers, and share it on social media. Your host, as usual, is Tobias Macy. And today, I'm interviewing Darren Gordon about YoSci, a security framework for Python applications.
So Darren, could you please introduce yourself?
[00:01:00] Unknown:
Hey, everyone. It's, it's an honor to to be here and to speak, before such a great audience. I've been working for with Python for almost 4 years now. I do have a tech background, but working in financial services, for almost 10 years, I I went from a programmer position into something in product development and risk management and, margin financing and prime brokers. So I've done a variety of things, and, I'm back working again full time as a you know, with programming, for my own venture. And, I wanted to give back, by working on, the YoSai project. And, so I'm really happy to be here today and to tell everyone, more about the project.
[00:01:43] Unknown:
And do you remember how you first got introduced to Python?
[00:01:46] Unknown:
Well, for me, it was a strategic decision. 4 years ago, I had an important decision to make. What language will I use as my primary language to build the first at least the first version of the services for my venture? I'm doing something entrepreneurial. So, I consider Java, Ruby, Node. Js, and Python, and Python won, because basically with Python you can go further. It's a powerful general purpose language, I can use it to create services and I can use it for analysis. It's a really powerful analytical tool. It's got a great diverse ecosystem of open source projects that I could adopt for my work.
I can optimize it and extend it to run quickly. And I'd say the most important reason why I chose Python is because there's just a huge global community of people who love to use the language. They're passionate about it. And, they're willing to help other people like me use it better. So I've gotten a lot of help along the way because of that. And, again, you know, I felt indebted to somehow find a way to give back and I'm trying to do that now with, with this work.
[00:02:51] Unknown:
And so that's a good opportunity to describe a bit about what YoSai is and the problem that you're trying to solve when you first started it.
[00:03:00] Unknown:
Sure. So YoSight, it began as a proprietary project, you know, for internal use. But, I decided to open source this project. You know, I'm working on a venture now where application security, it's a it's a top priority. Once I decided that I'd use an authorization policy based on role based access control, I looked for an open source solution written in Python that I could adopt to enforce RBAC. The problem is no such library really exists in Python, but I did find 1 in Java. It's it's called Apache Shiro. Now, the problem is I didn't want to use Java in my work. I want to use primarily Python and I didn't want to extend Python with, with this library, so I figured, why not port Apache Hero to Python?
And so I did, and the result is YoSai. YoSai is a security framework written in Python that is based on a version 2 alpha implementation of Apache Shiro, which is written in Java. It offers authentication, authorization, and session management from a common API. In other words, Eosai helps a developer verify who a user claims to be. It helps the developer control what the user can do in the system, and it helps you set time limits on how long a user may use the system or idle for before requiring that user to reverify its identity. Aside from just those 3 core services, some some other features I'd like to highlight are 2 factor authentication using time based 1 time passwords.
You can throttle authentication attempts, locking accounts after certain number failures. YoSight produces a full audit trail of every major event that's happened through the system, structuring, and it's it's it's producing it through structured logs, formatted as JSON. And, with that, it could be analyzed further downstream by other systems. And, another really useful feature is the user impersonation tool, which is for administration use where a developer or an admin of some some type in operations would be able to perform activities and sit in the system as if they were that user. All of this was really compelling for me. I I found it worthwhile to build the, to build the OSI.
[00:05:20] Unknown:
And so you mentioned that when you were first, sort of auditing the available libraries to fulfill the need for having that role based access control that you didn't find anything that suited it. So I know that at least in the web arena, there are things like Flask Security and Django Guardian that have a fairly granular capability for permission control. But it sounds like some of the requirement was that you needed something that was not necessarily web based and still had that same level of granularity. Is that accurate?
[00:05:49] Unknown:
More or less. Yes. It's I didn't exactly have a web requirement and what I found was that the web framework that I had chosen is Pyramid, and Pyramid has a really great, native authorization policy, in place. But, again, when I the way I got started was by thinking about how to structure permissions, using a role based access control policy. So after I decided on the policy, then I looked for the technology that would enforce it. And, you know, I wasn't completely married to using Pyramid, and so I did look at Flask and I did look at Django and I saw their plug ins and what was available. And the thing is, YoSai still offers a very compelling value proposition, even today despite what's out there, there are some differentiating parts here that still make it unique even despite almost 17 years of recreating the wheel across all these different major frameworks. And so a couple of those are, YoSai, it's it's a framework that you can easily customize. It's a security framework. It's fast if you have caching enabled, which you should by default. It offers a granular permission level authorization in such a way that none of these other plug plug ins do, and you can customize that permission further, which I have even for my own personal use.
And it could be easily adapted to any kind of application. It's not just for web apps. Although, the the vast majority of the apps that we're creating today are web apps, you know, if people want to use this say for, like a desktop application, a client that has a client server connection and it's using, let's say, RPC, well, you know, you could just use the OSI again for that and just write an adapter that knows how to to communicate with the server without having to recreate an entire library like this from scratch.
[00:07:41] Unknown:
Yeah. The overall approach of having it decoupled from the underlying use case is definitely very valuable. And it's something that, I know is a concern that the Python community has started to tackle in the protocol space as well, most notably with the work that Corey Benfield has been doing with HTTP 2 and having that completely separated from the actual IO, whereas in a lot of the other protocol libraries, the IO and the protocol handling itself have been intermingled, which reduces the ability for reuse within other contexts. So having that capability for security of being able to extract it from the underlying concern and have it purely as a, you know, essentially a protocol implementation is definitely valuable because it does give you that flexibility of being able to use it in other projects and even for being able to span across projects while still maintaining the underlying security definitions.
[00:08:31] Unknown:
Yeah. Definitely. It's valuable to take that approach, but, it's also reasonable not to, at least initially, because you're trying to solve your own immediate problems and you're not thinking strategically. It was much easier for someone like me to identify a project that had already done and to port it over than to think from scratch about how I would write or architect such a thing. So I I can I can understand why people didn't take that approach initially, but now the now such a project like YOSA exists, and, it seems like, you know, I'm hoping that other people find it useful for that purpose, and it kind of fills a gap, if you will? You know, addresses a need for something like this, something universal, and, I hope to use it in that way.
Now, in addition to just, you know, what makes YoSight unique, I I thought a bit about, you know, what kind of people would benefit by using it, and I came up with a few. Teams that have used other types of authorization such as resource based ACLs, but they've kind of grown beyond. It's gotten out of their control. It's become cumbersome. It's tedious to manage it the way that they have using groups and so forth. And they've learned about RBAC and maybe they want to adopt in another project. Another example is, developers who are maybe creating a new type of framework, like another web yet another web framework, which is very possible because of all the async IO work that's been going on. Let's say you're 1 of those, 1 of those leaders in tech and you you haven't even begun to think about how to, you know or what type of middleware to create. Well, you can use YOSI and write an adapter for it. So that's another good candidate. You could be a team that's maybe working on a new type a new a whole new project.
No authorization decisions have been made yet, and, you know, this RBAC thing might interest you. And, also, I say, finally, if you're part of a team that's using Java today and you're using Apache Shiro, well, YoSai is a port of Apache Shiro, so it's providing functionality that you're familiar with. So you'd be a good fit for this too.
[00:10:32] Unknown:
And 1 of the things that I've experienced a lot for people who come from Java or are trying to copy things from Java is that they get stuck in sort of the idioms of the other language. And so I'm curious as to what your experience was trying to take the underlying concepts from Shiro, but add a Pythonic feel to the overall final product.
[00:10:56] Unknown:
This has been quite an exercise. So it was pretty brutal porting thousands of lines of Java code, you know, across, like, dozens of classes. It's it's a pretty large project. So it was painful just pouring going through line by line, trying to make you know, understand some of the things that were being done. You know, I I hadn't been weren't been working every single day with Java. So there is a lot of searching involved, and I had to really get acclimated in a bit, to move forward with each step. But the approach I took was fairly common. My goal was to first make it work, and then I would make it right, and then I would make it fast. So this is a fairly common approach in software development. So what I did was in the first 2 versions was I ported everything over to Python.
I refactored some along the way, but starting in the the latest version, I really refactored a lot more, reengineered a lot more based on feedback, you know, know, a lot of flak ahead. I'd gone on the first 2 releases, but people didn't really understand where I was coming from. I didn't share them with them that, hey. I just wanted to make this work initially, and then I would improve it. It. But it's it's such a huge project. I couldn't really get my arms around it until, that first release. So, I did, and I've been doing lots of refactoring and reengineering since then. And, making it fast, the thing is, the OSI already offers, uses caching, and it's pretty quick already. But I've I've been exploring other ways using, for example, using, Rust extension, to potentially make it even faster. However, on the other side, there are ways to make it quicker even using PyPI, which I explored as well while I'm working with Rust. So, but overall, when I was porting the, the project, 1 of the greatest challenges I had was that I chose to port an unfinished alpha version. So version 1 of Shiro features, like, many levels of inheritance using mixins, whereas version 2 was, using a compositional design to replace all of that inheritance and using a compositional design to replace all of that inheritance. And so in version 2, there are far fewer moments of head scratching when you're trying to figure out how code works.
The problem that I ran into is that version 2 was unfinished. The main author of Shiro, he's the CTO of a startup, and he wasn't really available to answer questions I had about the new architecture. So it was out there. He said, hey, check this out. Check out version 2. I know you're working on it. And I did, but I was coming pretty much on my own. I had to figure out what he was thinking and just try to rationalize ways to finish, you know, completing the bridge, if you will, to the other side. And so I I ultimately, it was up to me to grab the bull by the horns and finish what I you know, what was starting, and and I did. So, where it is today is yosei is it works, and many parts are idiomatic Python. And it's pretty fast, but, you know, again, it can go a lot further. Another thing about porting Apache Shiro and probably 1 of the things 1 of the reasons 1 of the green lights I had I had sensed, if you will, when I was reading through it and trying to assess whether I'd be able to do this was that what made it easier to port it over was that the authors of SHIRO had used a a solid design had followed solid solid design principles, solid as in the acronym. I don't know if that that interests or that that that rings a bell for you. Yeah. I'm familiar with the acronym, and I'll add a link in the show notes for somebody who wants to sort of dig into it further. Okay. Sure. We could probably spend an entire hour, a good hour, just talking about how Shiro and YoSai follow the solid design principles, but, you know, they they do. You might and if you're familiar with Solid, you may see you may understand some of that just by reading through Yosai's code. And, you know, a a little bit about the refactoring where, you know, you start off just because, you could do something in Python doesn't mean it's right or that it's idiomatic, and, it was a learning experience for me. Sometimes, I I did things knowing that there were better ways to do it in Python, but I just had to get things done. And going back to it and refactoring and reengineering was, like you know, it kind of created a bit of a brain rush for me. You know, like, I I got a bit of a coders high using it because you know, or doing that because it felt really good to relieve myself of a lot of this sort of debt with Java. Here are a couple examples. So in Java, you have these if else conditions everywhere. You have to check a condition before you move forward. Whereas with, Python, you use try except blocks. In the way that people describe this is it's Java, you look before in Java, you look before you leap. Whereas in Python, you it's easier to ask for forgiveness than permission. So I ported tons of if else conditions to try accept blocks and, so that was, and in order to take the, take advantage of the, of of using Python.
And, you know, there are some design patterns that just weren't necessary in Python that came from Java. A couple examples include the builder design pattern. A builder design pattern can be just replaced using keyword args and, handling them accordingly in a in a function. Let's see. Custom exceptions. I kinda learned this the hard way, you know, I I ended up with way too many custom exceptions, and I didn't take full advantage of the exceptions in the, standard library. So, I was, got into a bit of a trouble. You know, people had criticized why I was doing that. And, frankly, I mean, initially, I didn't put much thought into it. I just wanted to make it work. And, so I did go back and clean that up a bit, and, I did find that most of the exceptions that were customized could be addressed with the exceptions in the standard library.
If, I only took the time to think a little bit about, what was available and how to apply them correctly. And I'd say another another Java pattern, that wasn't necessary, with Python, was that in Java, you have these POJOs. You have these just plain old Java objects, being used to store values and and in Python, you don't really need to do that. You don't need classes. You can use these primitive types like you can use named, tuple, or dicts, to store values and to pass them between functions. And so I did a lot of that in version 0.3. I cleaned up a lot of that and tried to take advantage of take advantage of what was available in Python. In in terms of regrets, porting this project, the 1 thing that really sticks out, if I can go back and change anything, there are probably too many iterations of unit testing. I went through too many iterations. I was too focused early on in trying to achieve this sort of maximal unit test coverage, when I should have waited for the dust to settle a little bit, so to speak, before actually unit testing. The thing is I thought good unit test coverage would encourage a lot of people to give you a try because bad test coverage would steer people away rather quickly. I didn't wanna, like, scare people away because there's enough scary parts of this, and at least initially, and I was trying to reduce that a bit. But the thing is I released 3 times, within a year, and there are a lot of changes that had broken tests. And so I was going back and cleaning up and changing tests 1 too many times. So that that was a bit of a a waste of time. And so, you know, if I can go back and do it again,
[00:18:18] Unknown:
I would just wait until unit I would unit test later on. Yeah. TDD works best when you know where you're going, not when you're trying to figure out what the destination happens to be.
[00:18:29] Unknown:
Yeah. I didn't really understand how all the parts worked. I did I couldn't see the forest from the trees, unfortunately. Anyone who's out there and if you're porting a project from 1 language to another, take it easy with the unit testing early on. Wait a bit.
[00:18:42] Unknown:
Yeah. When you're doing something like that, it's best to do more outside in testing where you start at the outside to make sure, okay, the perimeter of my system works the way that it needs to. And then as you sort of solidify the interior bit by bit, then you can work the tests into a more granular level. Yeah. It's much easier and you're much,
[00:18:58] Unknown:
you're probably more likely,
[00:19:00] Unknown:
to not have to break or or to to fix all those tests, you know, week by week. And 1 of the questions that came out of your description of porting the project is because of the fact that it does have this other code base that it is using as inspiration. I'm wondering if the intent is for you to track its progress as it continues to evolve or if the point at which you started work on YoSai was essentially just a forking point where YoSai will have its own destiny separate from, well, where wherever Shiro happens to go. Well, it seems that at this point that the Shiro,
[00:19:36] Unknown:
the people who are supporting Shiro, you know, primarily, the main author has stepped away. He's a CTO at a at a company now. So, yeah, he's more hands off than other people who are still supporting the, version 1. And so because I've really ported version 2 and that's incomplete and, they don't really seem anywhere in the near future ready to, you know, turn things over to version 2. I've this is this is a fork. It's a permanent change, and, like I said, I had to fill in a lot of gaps. So YoSai is its own creation now, and, really, the way that it's going to evolve over time is going to depend entirely based on the feedback from the Python community.
I I don't intend to look to anything that you know, I I can't, unfortunately, reference anything that is happening now in Shiro. And it really doesn't make sense to it. YoSai has taken on, a life of its own.
[00:20:25] Unknown:
And given that authentication and authorization are such broad and difficult problems to solve, and any issues with how it's implemented can potentially lead to significant impact in the projects that are relying on it for security. So I'm wondering, what are some of the techniques that you're using to ensure that there's an appropriate level of quality in YoSci itself for for other people to be confident using it to protect their applications?
[00:20:49] Unknown:
Yeah. Sure. It's a perfectly valid concern. But let me first begin by saying that I have not contracted or a qualified third party to audit YOSI, but welcome anyone out there who meets that criteria to please come in and evaluate the project. And currently, I'm not aware of anyone conducting such an evaluation, but I do believe that a qualified third party audit would be helpful. We could spend at least an hour talking about why I'm comfortable with, or confident about YoSai and why anyone else should be confident with it, but let me just touch on a few points here. So regarding authentication, so YoSai facilitates authentication.
I didn't roll my own crypt though. YoSai uses Passlib, the very popular Passlib library for the cryptographic hashing and password verification. How a password is hashed and persisted into long term storage, it's a far more risk prone exercise and it's an exercise that's outside of the scope of YoSai. YoSai queries and verifies credentials against those provided by a user. The moment that a credential is no longer needed, it's cleared from memory. Passwords are stored as byte arrays for this reason. Regarding authorization, a few points. YoSai denies access by default. A user is granted access only when that user is explicitly assigned to a role or assigned permission. IoSci verifies that the roles and permissions from a data store imply the roles and permissions are required to perform an activity. Permissions are queried, not created by Yosai. The permission authorization logic works exactly as that which is implemented in Apache Shiro. There's there's no functional difference. And that logic, it's been battle tested and it's been adopted by major firms.
Regarding sessions, sessions are also treated with care. A session ID changes when a user logs in. In web applications, session IDs are stored in cookies and sessions time out due to inactivity and absolute timeouts.
[00:22:38] Unknown:
Despite everything I've mentioned and why I'm comfortable with using EOSI, I nonetheless encourage anyone who's interested in adopting it to consider auditing it prior to production use. And you mentioned too the, sort of test coverage that you have for it. I know that you said that you have had some trouble with being able to have the tests keep up with the work that you're doing. But at this point, is there a fairly substantial level of coverage on the actual project itself to prevent any regressions
[00:23:06] Unknown:
from the current state? With each release, I have tried to achieve maximal test coverage. So when I had shared my grievances about test testing, I nonetheless did test. So, with each release and in the ReadMe, I try to make it very clear, about test coverage, and it's you know, I would consider it It's still at a very a very great level. It's I'm not trying to achieve a 100%, but I'm trying to achieve at least 90% across all the major modules, and so it it it offers that. And you alluded to the read me briefly, and I know that when I was doing research for this interview,
[00:23:39] Unknown:
I was digging through the documentation, which I must say is quite substantial and fairly well written. So I'm curious, 1, how you managed to achieve that level of coverage of the overall documentation, and 2, what sort of inspired you to actually put out that level of effort to make sure that you had that documentation to begin with? Oh, thank you. I appreciate it.
[00:24:00] Unknown:
It was weeks of effort, if not months of effort. And, you know, all of that is really above and beyond what I would have ever done, if I wasn't open sourcing in this and I want other people to use it. But because it's such an elaborate system, I felt it necessary to document as thoroughly as I could, and I tried to imagine how people the questions that people would have, and I tried to put it all together in that web page. You know, there's a full project page for for YoSai, and I try to summarize that in in the read to me and I try to create visualizations and things like that to, you know, just make it easier for people to understand, what's what's going on. It took a lot of work. There's no way around it. Writing technical documentation wasn't something I've ever enjoyed doing, but I found that anytime you write software that someone other than you is going to use, and if you don't wanna be fully roped into everything, you've got to teach them somehow. And, so you do that through documentation and tutorials and demos. And, I I created all of that for YoSight because, you know, I do I have already moved on to other projects, and I want people to find it useful. So I I I saw no other way. And so it sounds like you did all of that yourself. You didn't hire a technical writer to farm any of that out? Yeah. That's all me. Yeah. Whatever it is in my life, I I just I figure things out. I I guess it you know, my greatest strength is that, I I guess I I eventually can solve most of the problems thrown at me. And I'm willing to give it a try, as anxiety inducing as it really is. I've written all the documentation. I did manage to copy, however, some of it from the original Apache Shear website. I copied some of it, but I I really try to paraphrase and I really try to make things unique. I don't like copying other people's work. I do sometimes when I get absolutely lazy, but then when I do, I try to cite, you know, the original author as best I can. I'm not trying to get credit for stuff like this. So it I I do believe I'm giving credit where credit's due, but I did a lot of the you know, I did a lot of it on my own. With the help with, static site generators, I used, Hugo. I used the Hugo set static site generator, which is written in Go. I I tried the ones written in Python and I tried Jekyll, but I found Hugo and the the the, templates that were available, most useful.
To create the documentation, I used a Python library called MK Docs. I used MK Docs. It was it was it was actually great, because I didn't have to do hardly anything other than write markdown. And, the static site created all the documents for me, and it did very nicely. However, I also used, for the tutorial, I used Hugo, and, but that was only to create the time based 1 time password tutorial. So you're going to see some difference, but, the general idea is to, write your content in markdown and then find a static site generator that'll convert that markdown into a presentable web page. And,
[00:26:56] Unknown:
There's certainly no shortage of options on that front having, explored a number of them myself for various other use cases, both for blogging and for different projects I've been involved with. That's great. So I'm wondering if you can dig a bit into the way that YoSai itself is architected.
[00:27:12] Unknown:
Essentially, you install YoSai and its extensions from PyPI, and then you configure this YAML, y a m l, YAML settings file. During your application's initialization process, you know, during the bootstrap process, what you want to do is instantiate a YoSai instance telling it the location of that settings file, and then you store that YoSai Instance somewhere that it's accessible during every application call that may require YoSai related functionality. If you're building a web application, what I'm talking about is the registry that your middleware references. The next step is you write a piece of middleware that opens a context using the IoSi instance as a parameter and that's the only thing that the middleware needs to do is it opens up a context.
Now, from that point on, wherever we want to authenticate or manage session state, what we do is we create a subject object and we use it to log in a user or access the user's session. The main API, Yosai's main API for your typical developer, it's going it's accessible through this subject object. And the next thing is, aside so use the subject object to authenticate and to manage session state. You could also use it for authorization, but you don't necessarily need to. So let me explain. Wherever you want to authorize access, you have 2 options. You can use imperative style access control or declarative style. Imperative, you're creating a subject instance and you're making explicit calls to the Subject's authorization API. With a declarative style access control, you're decorating functions with authorization logic and allowing YOSI to handle all that boilerplate authorization logic for you. If anyone's interested about, you know, how this works in detail, please go to the documentation on the project website and, you're gonna see exactly what I'm talking about. But this is how YoSai works in a nutshell. You you create the instance, you put it somewhere, you open up a context, and from that point, you could use YoSai for authentication, authorization, and session management.
[00:29:11] Unknown:
And when you're not using YoSai as part of a web application, which is where, I, in particular, am most familiar with managing access control. I'm wondering what are some of the other types of situations where a user should consider integrating that authentication and authorization capability into their project?
[00:29:29] Unknown:
There's kind of a cop out, but, if you wherever you when you write an application where user identity matters and you want to control what that user can do, then your application would benefit by Yosai. The the example I had raised earlier is another viable example. It's a it's a client. It's an old sort of client server, app, you know, setup, except you've got, like, maybe a thick client application. And and the server doesn't necessarily have to be a web server. Maybe it's you're using RPC, but you still bent you you still need to know who the who's making the calls, who the user is, and and you wanna control what that user can do. Yeah. Another type of example that somebody might use is a sort of IoT situation where you've got a sensor that wants to be able to send information to a back end, but you wanna make sure that people can't just spam the back end with
[00:30:16] Unknown:
arbitrary false information. So you need to have some method of authenticating that sensor to the back end and controlling the types of information that it's allowed to send based on, you know, a a unit ID or something like that. Oh, yeah. Absolutely. And, you know, that that's you're we're just warming up. And, I mean, when we talk about IoT,
[00:30:34] Unknown:
you know, if people are running home servers and people are trying to access, you know, physical locations, you know, trying to open doors and so forth, Well, that server is going to have to know who that person is that's trying to enter, and, there's gonna be access control, you know, involved. So, yeah, if it might be a web application that's doing that. It may not be. It might be something else. And, if you're writing in Python, hey, you know, maybe you should use YoSci for it. So what have been some of the most challenging aspects of building your OSI? And what are some of the things that you learned that you weren't expecting to come across in the process? Yeah. Well, I I I touched on some of that already. The whole project was very challenging, but it was a it was a huge learning opportunity for me. You know? I was really cutting my teeth on a huge project, with this. You know, cutting my teeth with Python and, you know, there are a lot of moving parts and so forth. So it was, you know, the sheer, it's just a ginormous effort, you know. So I I would say it wasn't a individual thing that was making it hard. It was just all of them combined. Right? The documentation was really hard.
So let's just leave it at that. The docs were the hardest. Yes. So please read them.
[00:31:41] Unknown:
Well, I I have read the I've read a fair bit of them, so you've got at least 1 person who's benefited from them. So, just to be clear, I think. Thank you. And are there any sort of, unexpected use cases that you've seen people put your site to that you would like to call out? Not yet. None that I'm aware of yet. So it sounds like the sort of most valuable community contribution you could have right now is just having people use it and provide feedback from their experiences?
[00:32:06] Unknown:
Oh, yes. Absolutely. Give it a try. If you wanna write, if you wanna write adapters for for any of your existing frameworks, I would help you do it. I guarantee that together we would have it written and tested, within just a few hours. I haven't done it because it's not something I wanna own and manage going forward because I I'm probably not gonna use it unless it's unless it's for pyramid, and I've already released the pyramid adapter if you wanna see how I did it. But, you know, because I would like to see people use it and I would value your feedback, I'll help you. You know, I'll facilitate this by showing you how to write it. And you mentioned that 1 of the approaches that you've taken to try and speed up parts of YoSai is to actually write a Rust extension for it. So I'm wondering if you can just spend a bit of time talking about your experience with that, particularly given that Rust as a extension mechanism isn't fully supported by Core Python yet. But the thing is it's really picking up steam. Since the beginning of this year, things have really been moving quickly on the Rust front.
So early this year, I read a blog post by Armin Rocha about extending Python with Rust. He wrote about his experience and then he open sourced his work. This inspired me to give Rust a try. So I spent a few weeks learning the basics of Rust, asking a lot of questions on IRC, and and I need and I learned how to write a Python Rust extension. I thought, what better way to learn it than with a practical use case? And so what I did was I chose the part of of YoSai, for authorization that had the potential to become more computationally expensive because, you know, if you're gonna write something in Rust, you know, and you you ideally, you want it to run faster. So I thought, well, this part of code would benefit by that. And so I ported the entire, like, you know, small library for authorization, and the end result is, I managed to port all of it. It works in Rust. I got to learn some Rust. I open sourced the work, and I even wrote a tutorial about it. But the thing is I haven't given a talk about how to extend Python applications with Rust, and and I might in the near future, especially now because, there's some other really interesting things happening on the Rust front, with respects to integrating with Python. 2 projects that come to mind are Pythonium, trioxide, which is Py 0 3. The author just made his first release on that, a week ago. It's 1 way to write Python extensions, Python Rust extensions, and it's a 2 way sort of library. So if you want to access Python, from Rust, you could do it, or you could access Rust from Python.
His library allows something like that. And Armin Roacher, since that first project, learned from his experiences and has created an even better way to extend Python with Rust, and he's calling that snake, but spelled a little differently, s n a k. And, you know, I've had some chat with corresponds with, Armin, and, he's still fleshing some things out, making some important decisions about things such as memory management, but, I think that, you know, with all these there's a lot of interest that and there are a lot of people who seem to be working on ways to extend Python to Rust, and I wanted to give that a try. And so, I did.
And, of course, at the end of this, I I really want to know, well, how much faster how much faster how much, you know, how much faster is this Rust version of my of what I wrote in Python? And so I tried as best I could to come up with an apples to apples comparison. Just running the exact, logic on both parts and not trying and trying to eliminate any other overhead. And what I found was that, unsurprisingly, the the rust ran approximately, let's say, 10 times faster than CPython, but here's the thing. Then I want to know, well, what happens if I run this through pypy? And so I did and the PyPy, the compiled Python actually ran faster than the Rust extension. And I I was like, my mind was just blown away when I saw it. I was like, wow, this is great. This is this code would actually benefit by using PyPI. You know, why would I do this in Rust? But the thing is I I didn't I wouldn't have known that unless I got I went through this whole exercise. So, that was revealing for me from this, experience.
[00:36:20] Unknown:
Yeah. It's definitely interesting. And, if you are looking for a venue to talk about any of your work, I don't know if you are aware, but the call for proposals for PyCon Canada and PyCascades are open right now. So Okay. Noted. Yeah. I I would love to share my experience with this. I I'll you know, maybe elaborate on this story too. And what do you have planned for the future of EOSI? I know that you mentioned that you're not actively working on it at the moment because your attention is elsewhere, but it sounds like you're still committed to maintaining it and supporting it going forward. So I'm wondering if there are any sort of additional features that you're looking to implement or would like other people to help you with adding in. Yes. Absolutely. So, I think that YoSai would benefit by more refactoring and reengineering and trying to make it even more Pythonic. It's a great experience,
[00:37:06] Unknown:
trimming off the fat. You know, it's great experience making code more efficient, and I think YoSci still has, rooms for improvement. And, so there's that. And I I usually do bits and pieces of that with each release, but I if anyone would like to give it a try, you know, read through the source and see what you can come up with. I I would gladly, approve of changes as long as you're not breaking anything. What I plan to do, at least in the very next version, which I expect to have done by the end of this year, hopefully, is, I've already written support changes on the master branch to support the Rust extension. I plan to modularize the OSI so that if people just want to use the authorization component, they can without having to install all of the OSI and all of his other parts.
So modularization is another thing another project that I I plan to accomplish, and I want to make sure that, each of the parts work through PyPI. Another thing is I I'd like to add support for JSON Web Tokens. I'm very cautious about using JWT because of
[00:38:08] Unknown:
the experiences other people have had. So I'm still looking and researching how to follow best practices, and I'm really receptive to people who have had experience with JWT and I may be interested in making that work with the OSI. Alright. Yeah. I've actually got a blog post on my blog about my experience using it, a couple of years ago, but I'm sure that the overall best practice has been updated since then. I'll check it out. So are there any other topics that you think we should cover before we start to close out the show?
[00:38:38] Unknown:
I well, I've covered a lot of things here. I think this is a good start. But, I encourage people to check out the the GitHub repo and to check out the project web page.
[00:38:47] Unknown:
And for anybody who wants to contact you directly or follow the work that you're up to, I'll have you add your preferred contact information to the show notes. Sure. And with that, I'll bring us to the picks. And my pick this week is the Brain On podcast, which is a show that I started listening to with my kids recently, which is a podcast from 1 of the NPR stations that is a science show focused at kids in the, you know, probably 8 to 14 time range. And it's just a lot of fun to listen to. So some of the shows we've listened to are things like how the moon affects the tides, debates between the deep sea and deep space as far as what, you know, what is cooler. So just surfacing a lot of interesting content about that and, fire versus lasers. So it's just a fun show with a lot of whimsy and a lot of really valuable educational content. And with that, I'll pass it to you. What do you have for us for picks this week? So what I'm picking is a really great,
[00:39:44] Unknown:
project. You can see it out on GitHub. It's called Asphalt. It's the Asphalt framework. Asphalt is an asyncio based microframework for network oriented applications. It's written by Alex Gronholm and, he spends a lot of time, creating, you know, creating Python software correctly, making sure it's idiomatic. And Alex has been very focused on using the latest and greatest from the asyncio front, and,
[00:40:10] Unknown:
I'm very optimistic about where this project is going. And I encourage people to check out Asphalt. Alright. And as a, meta comment here, are there any of your own projects that you'd like to call out or try and get people interested in?
[00:40:23] Unknown:
Well, I'm now working on a, my own venture. I was working in financial services for almost 10 years and, you know, I've been it's all part of why I was learning Python and why I'm building Yosai and all these other projects. I am working on something and, it's been pretty lonely, but, you know, I've been trying to just get the the work done as a solo founder and, it's really out of necessity. So, you know, I'm an entrepreneur who's an avid contributor to the Python community, and, I'd love to network with people and connect with people and share with them what I'm working on. This is not a place, you know, for me to do it. I didn't wanna take advantage of our our discussion, but it's what I'm very passionate about. It's what's driving me to do what I'm doing. So, it's a major part of my life, and I love to talk about it.
[00:41:06] Unknown:
So I'd like to say thank you for taking the time out of your day to share your experience of working on EOSI and porting it from the Shira project. Definitely looks like an interesting, library and 1 that I plan to take advantage of in any future projects that I work on that require that authentication and authorization capability because it looks to be fairly comprehensive and flexible. So I appreciate that and I hope you enjoy the rest of your day. Thanks, Tobias. Bye, everyone.
Introduction and Guest Introduction
Darren Gordon's Background and Python Journey
Choosing Python for Entrepreneurial Ventures
Introduction to YoSai
YoSai's Unique Features and Use Cases
Challenges in Porting Apache Shiro to Python
Ensuring Quality and Security in YoSai
YoSai's Architecture and Implementation
Non-Web Use Cases for YoSai
Challenges and Learnings from Building YoSai
Community Contributions and Future Plans
Exploring Rust Extensions for YoSai
Future Enhancements and Modularization
Picks and Recommendations
