The Python Podcast.__init__

Share on social media:

[00:00:12] Unknown:

Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great.   When you're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it. So take a look at our friends over at Linode.   With 200 gigabit private networking, scalable shared block storage, node balancers, and a 40 gigabit public network, all controlled by a brand new API, you've got everything you need to scale up.   And for your tasks that need fast computation, such as training machine learning models, they just launched dedicated CPU instances.   And they also have a new object storage service to make storing data for your apps even easier.  

Go to python podcast.com/linode,   that's l I n o d e, today to get a $20 credit and launch a new server in under a minute. And don't forget to thank them for their continued support of this show.   And you listen to this show to learn and stay up to date with the ways that Python is being used, including the latest in machine learning and data analysis.   For even more opportunities to meet, listen, and learn from your peers, you don't want to miss out on this year's conference season.   We have partnered with organizations such as O'Reilly Media, Corinium Global Intelligence,   Alexio, and Data Council.  

Go to pythonpodcast.com/conferences   to learn more about these and other events, and take advantage of our partner discounts to save money when you register today.  

[00:01:32] Unknown:

Your host as usual is Tobias Macy. And today, I'm welcoming back Thomas Hatch all the way back from episode 1 to talk about his work on the POP library and how he's using plug in oriented programming in his work at SaltStack. So Thomas, can you start by introducing yourself?  

[00:01:46] Unknown:

Yes, I can. Hello, my name is Thomas Hatch, I am the original creator of SALT and I'm excited to be on here today to be able to talk about plugin oriented programming, a new approach that I've got around software development. I'm also the chief technical officer for SaltStack. And do you remember how you first got introduced to Python? Yeah. I mean, this this is going back a long ways.   It was back in the days when there was a bit of a debate who's gonna win, right, between Python and Ruby. And I I spent a lot of   time, looking into Ruby. I'm still really impressed by Ruby. I think that they've got a lot of really fantastic language features. But in the end, I I lean towards Python because   it was   such a   it's a much richer   ecosystem,   and I felt that the language,   was a lot more pliable and a lot more usable.  



[00:02:37] Unknown:

And so as you mentioned, you've been working on Salt for a number of years now, and 1 of the outgrowths of that is your work on this PoP   library. So before we get too far into that, I'm wondering if you can just start by giving your definition   of how you view plug in oriented programming and some of your thoughts on the benefits that it provides.  

[00:02:58] Unknown:

You bet. It's it's been really interesting. I feel like I almost   stumbled   upon the concept of turning my plug in system into a programming paradigm,   until about a year ago if in my development when I realized that that's kinda what I've done.   The the core idea behind plugin oriented programming is that whenever we make large scale applications,   we almost always need   to make them pluggable to a certain degree, make them extremely modular.   And so I thought, wouldn't it be better if we just started building things in a pluggable way? And   pop allows us to build software that's not only   100%   comprised of plugins,   but it also allows us   to create applications   that are pluggable with each other,   which extends the paradigm   of libraries and applications into a more unified concept where we are able to create   entire plugin subsystems   that define   how an application is supposed to work,   not only the functionality which the application exposes through a library like medium. And so I felt that again that makes makes it so that it's a lot easier to create   smaller, portable,   compartmentalized   units of software.  

It also adopts a lot of concepts for microservices,   but instead of looking at things through   the microservices lens, it makes it a lot easier to create compartmentalized  

[00:04:28] Unknown:

units that run together in a single process even. And I understand that a lot of your thinking about how this should work and   some of the design features   of what pop provides   are influenced by the work that you've done on Salt itself, which is definitely very extensible and plug in oriented platform. So I'm wondering if you can talk a bit about some of the ways that   Salt has influenced your thinking on how to implement plug ins for software projects and some of the ways that those thoughts and your ideas of best practices have manifested in the PoP library   and some of the capabilities that it provides.  



[00:05:07] Unknown:

Yeah. And so when we come back and we look at,   Salt,   to be honest, in the beginning, I wrote a plugin system for Salt. And I and I didn't think that we would use it inside of SALT anywhere near as much as we did. And at this point, I think that there are 36 plug in subsystems inside of Salt, which abstracts from a PoP perspective something like 12 independent applications.   And when I started working on POP, I really was looking at the the plugin system inside of SALT and saying, hey, this is really useful. I wanna use this in other projects without having to import all of Salt to get it. And so my first iteration was just trying to make the Salt Loader standalone.   But as I iterated on the idea before actually   releasing it, I I rewrote   the the pop platform   numerous times. The initial releases release of it is actually the 5th iteration.  

And   so,   not only was it heavily influenced by Salt's plug in system, but I tried to solve a lot of the problems that arose   in Salt's plug in system. And, again, at first, I was looking at it purely from a technical problem perspective,   but as I kept working on it, it became really clear   that there were a lot of problems that we ran into and that most, honestly, the vast majority of software projects, especially open source projects run into these similar   restraints   and issues   mostly around   maintenance of the platform.   And   there were a few other things   from a from just a human perspective that I started to realize as I was working on this whole plugin oriented paradigm. And then again, I mean, I've been working on the paradigm for the last maybe 3 and a half years,   despite only releasing it and only feeling like it was mature enough to be released a few months ago and after literally   4 rewrites from the ground up. But 1 of the big things that I learned from Salt is just how much time   we spend reviewing pull requests and reviewing code. So I step back and I started to wonder how much time do   most open source projects and most open source developers have to spend   reviewing code, reviewing proposals   instead of just writing software.  

And I found   out that I was far from alone in this problem   where the vast majority of us who create these large platforms end up   having our lives consumed by reviewing contributions. Now I don't wanna say that to to say that I don't like contributions   or that, they aren't incredibly valuable.   But at the same time, I wanted my time back, and I wanted to be able to write software again instead of my time being so heavily consumed by, reviewing others' code. Linus Torvalds recently said in an interview   that he doesn't he's not a software engineer anymore. His job is to just say no.   His job is to just tell people, no. I won't let this code in. And so 1 of the big   things that I, that I put inside of plug in oriented programming is the ability for someone   to extend a piece of software   without having to necessarily push that code upstream   so that it becomes easier for someone to   write plugins and then deploy those plugins using the app merging concept instead of plugin oriented programming. So instead of coming back and saying, hey,   Tom, can you please review my code and merge it?  

Regardless of the maturity of the code or how well it's going to be adopted or used inside of this mainline platform?   It's a lot easier with plug in oriented programming for somebody to write,   an app merge plug in system that literally augments the original piece of software.   Then those augmentations to that original software can grow and be adopted on their own merit, just like writing an independent piece of software, which makes it a lot easier   to be able to be able to vet and verify   that the software that we're looking at is   something that we actually want to put into the main line, something that, we might want to support upstream.  

But it also puts the onus of support   to any piece of software more firmly in the original author's hands,   which almost universally leads to significantly better support and better maintenance of that software because of their own intent. So SALT not only influenced me technically   from the perspective of saying, hey. I love plug ins. I want more of these. I wish I'd made more systems inside of Salt Plugable. It would have solved a number of problems that we've run into over the years, but also that it creates a more human,   sustainable, and maintainable   piece of software.  

That means that,   myself   and the software developers on my team   are able to spend more of their time focused on engineering and less of their time focused   on reviewing people's code. And at the same time, our contributors,   contributors who are contributing to a pop project since they can use app merging,   they also don't need to wait on us to review their code.   They can release their code independently,   which again,   my hope and anticipation here is that it becomes a platform that not only accelerates software development, makes high level software development easier, but can also accelerate the development of open source software. Wow. Goodness. Sorry. I hope I hope that answer wasn't grossly too verbose.  

[00:10:42] Unknown:

No. That was fabulous. It's definitely helpful to get a lot of the perspective   of your experience of working, as you said, both technically and socially on Salt, and   And I understand also that some of your motivation for creating PoP as a standalone library   was a way for you to explore some of these concepts   independently and in isolation without having to try and factor them back into Salt itself and deal with all of the weight of everyone who is relying on Salt to be stable and reliable,   and give you a way to determine whether or not you could then incorporate these ideas or possibly even the PoP library itself back into SaltStack. And so I'm curious   how either the technical implementation of PoP or just the lessons that you've learned in the process  

[00:11:42] Unknown:

PoP inside of Salt,   it   who, I mean, that that's kind of a tricky concept as as I've stepped back   and and looked at things,   the overall landscape of managing   has changed a lot, and this is still the the core   goal   of salt. And   I I love salt, and and we've done really incredible things with with that platform. But as I think about it, 1, pops plug in system has has fundamentally diverged from how Salt's plug in system   works. It's mostly compatible. I'd say about 95%   compatible with the, the Salt Loader, but it can't   drop and replace it. Now as we step back and we look at Salt from a again, from a pop perspective, like I was saying before, Salt is really comprised of   many, many different   independent projects, independent use cases. And 1 of the challenges that we run into from a from a SaltStack perspective is just letting people know how much is in there. It's it's incredible talking to people. We just had SaltConf last week, and it's incredible talking to people as they come back and say, wow. I'm I'm always learning that there are more things inside of Salt. And this is 1 of those realizations that I had that   since I've put so much functionality into a single platform,   that platform owns that functionality, but it also doesn't expose it. It also doesn't let people see that, hey, we've got beacons and reactors. I mean, almost nobody knows what Thorium is inside of salt, for instance, despite it being an incredibly powerful platform,   which is used to do some amazing things. So if I step back and I look at salt from a pop perspective,   it's really easy to say, okay,   if I were to reimplement salt to run-in from a pop perspective, it would be broken out into   countless individual applications   that do highly specific things. So for instance, grains would be its own application.  

SaltCloud would be well, I originally thought SaltCloud would be its own application,   but I actually made the,   the state systems language its own application   called item.   And then,   the SaltCloud working group has been working with me to   actually port much of SaltCloud's functionality   directly in directly,   on top of item   as these app merge features.   So   that, each individual cloud ends up being its own project   so that it plugs directly into the language. And then that allows us to   manipulate and push the language forward at a much faster rate. This, this implementation of the salt state system on top of pop called item already has a number of features that,   given the restraints inside of the the less flexible   implementation   of the state system inside of solve.  

We're just really, really difficult to get in there, but we got them inside of item. So the state system becomes its own   application, remote execution becomes its own application,   grains becomes its own application.   But then at the same time, we get to solve a lot of other problems that we run into from a salt perspective. So inside of salt, we ship   support for all these different platforms all over the place. We test them and maintain them in the same location.   But using PoP,   the goal here is that all of these operating system support, cloud support, API support, this huge array of just things that Salt is capable of, interfacing with can be developed and tested   independently, can be released independently.  

So that if we wanna add some additional support for   some new interface inside of Red Hat or new APIs inside of a cloud. All of these things from a pop perspective will be able to be released independently. And so I don't look at pop as necessarily going into the salt code base. I look at pop as a potential   for creating   really the next generation   of salt so that it's again something more pluggable, more pliable. It's a collection of components that are more reusable   in other applications,   as well as,   being able to   solve many of the fundamental challenges that we run into in salt and finally also being able to take Salt   and   level it up, make it   purely Python 3. Take advantage of a lot of these these new things that we have access to like like asyncio for instance. Asyncio   completely changes how you can write software  

[00:16:22] Unknown:

evolution of Salt, And I'm excited to see some of these new incarnations   of some of the different subsystems   in their own independent implementations that can then be merged together so that you only incorporate the bits that you need rather than taking the entirety of this massive framework for everything that you might want to do in the automation space.   And similar to how Linux has a lot of drivers for systems and hardware that you're never actually going to use, but it's there in the kernel anyway,   Salt has sort of gone down that same road because of just sort of legacy issues and the way that software and infrastructure was managed at the time. And you incorporate all these drivers and capabilities for platforms and systems that you don't necessarily care about. So I'm I'm definitely excited to see the future there. And then going back to PoP itself, I'm wondering if you can talk through some more of the advanced patterns or paradigms that this plugin oriented programming model allows for, and the way that you have experimented with and implemented some of these things using the PoP framework. 1 of the big challenges that I've got is that I'm presenting this as a programming paradigm.  



[00:17:30] Unknown:

I kinda like to explain to people that I must be completely insane   to have tried to create a programming paradigm. So I have to I have to kinda start from the whole, you know, what new patterns and things can be expressed by just saying,   I'm not out to, you know, destroy object oriented parent programming.   Much of pop has been inspired by many of the features and functionality of object oriented programming as well   as flow programming, functional programming, etcetera, etcetera. And I wanted to be able to take a lot of the benefits from these different paradigms,   but then wrap them up in such a way that again hopefully makes software development more human. When we look at programming paradigms,   programming paradigms in and of themselves   tend to evolve out of expressing how a computer works   or expressing   math. And I wanted to look at this from the top level and say, what are some of these concepts that make a lot of sense to just humans naturally? What are the concepts in computers that have endured   for the last 40 years that, well, I shouldn't say 40 years, but for the last 10 years, 20 years, 30 years, 40 years, that can be incorporated   in such a way that can hopefully make software engineering   easier,   not only more powerful and more capable of just exposing   exposing a human being's idea in in engineering. And so I've tried to look at pop from a top down perspective, a human   to code perspective as opposed to a bottom up perspective, which is what I feel most existing paradigms express.  

And   so in doing so, I introduce a bunch of concepts inside of salt.   1 of them is,   plugin subsystems.   So when you create plugins, those plugins have to exist in a bag. Right? I mean, they've they've gotta follow some kind of pattern. And so instead of looking at software from the perspective   of object oriented programming or I should say purely object oriented programming, you can certainly do object oriented programming   inside of pop, but instead of looking at it from that perspective of   hierarchies   and inheritance   and is a has a sort of,   sort of perspectives. I instead thought, well, humans think much more in patterns.  

And so you can create a plug in subsystem inside of inside of pop, and then that plug in subsystem follows a pattern. And so you're able to say I'm gonna let's just say grains for instance. So so for anyone listening who's unaware, grains is a is a thing inside of Solve that scans   your,   operating system to just gather some information about that operating system. Inside of pop,   you would view grains as expressing a pattern, and that pattern would be a data gathering pattern. And so any plugin that would be added to this subsystem   is going to just extend how data is gathered in a collective way. Similarly, inside of Salt, we've got a plugin system   that, is just called modules because it was the first 1,   but we call it execution modules now and it exposes   a system   management   library. And so inside of Salt, you can make a pattern that we would call a library pattern. And so in that case, any   any plugin that you would put inside of this plugin subsystem   would just extend that,   that library of functions, which again is a very common situation.  

Another pattern, for instance, though, that we run into that we see inside of salt is seen in the returner system. And this is a very common pattern especially inside of salt where   we are expressing an interface. And so this pattern looks a lot like the creation of an interface inside of object oriented programming where we say, look, certain functions need to be   exposed to be able to fulfill the functionality   of this interface.   So for instance, again, if we look at a returner in SALT, a returner is used to take information that's happening inside of the platform and send that information off to different data stores, right, databases, etcetera. And so it's got a collection of 6 or 7 functions that need to be implemented to create a return. And so that interface idea is just another pattern. And so and so a lot of my thinking there is that, again, if we expose patterns, that's a more human way to approach programming,   and it's also much more extensible.  

You don't need to   understand an underlying   abstract base class to be able to expand a system.   It becomes really easy to be able to just say, hey, I'm just gonna make a plug in. If it has these functions, it's gonna work, and that's really all I need to know. And that's another 1 of those things that we learned from Solve   is that it's very, very easy for someone to extend an interface when they can look at an existing interface and say,   hey. If I wanna add package management support for a new operating system, here's the 6 functions I implement and it's just gonna work. It's just gonna extend that system, which again, I I feel based on our contributions is a really easy way to express that problem.  

[00:22:35] Unknown:

Yeah. Being able to   just copy and paste an existing implementation   and then either rewrite or extend the specifics of what you care about and just keep it in your own repository and synchronize it into the system is something that I've been able to take advantage of quite a bit to manage our infrastructure at my work. So I can definitely see how that would be a powerful paradigm just for software in general of I have this application. It doesn't quite do what I want,   Where right now, you would usually have to fork the entire project, make the changes in line in the code base, or figure out how to extract it out, and then go through the complicated process of building that interface so that you can then submit that back upstream or permanently maintain a fork of the project versus just being able to say, I only care about reimplementing this 1 subsystem.   I'm going to override it in this other repository and load it in, and then the person who's maintaining the core project doesn't ever have to know or care about it. I I can see that that would be very powerful and a much more enjoyable way of managing software projects going forward. Yeah. That's the goal. And it's and it sounds like you get what I'm throwing down. So so that that makes me feel better.  



[00:23:41] Unknown:

I I always worry about the communication aspect.  

[00:23:44] Unknown:

And so in terms of the pop library or framework itself, however you want to refer to it as, I'm wondering if you can dig into some of the ways that it's implemented, and some of the other languages or libraries or ecosystems that you've looked at for inspiration   as you have   iterated on the design and philosophy around the development pattern and the specifics of PoP itself? Yeah. So, I mean, 1 of my hobbies is to play around with programming languages.  

[00:24:10] Unknown:

I I I really love being able to just install and try out new languages. I honestly, now that I'm thinking about it, I haven't I haven't done any for almost a year now,   but I I love just sitting down and learning new languages. So   my first implementation of PoP   that was just a fork of the salt loader again was written in Python. And after I wrote it, I started looking at things and saying, I feel like there are certain limitations in Python.   I'm worried about performance. I'm worried about,   just how how to distribute software in Python. You know, just some of those some of those common issues that we run into when using Python. And so I step back and I thought, how about how about I try and implement this whole plugin system in different languages? And so I went back and I tried out Go and Rust,   and it's not surprisingly there's very specific challenges   in   implementing,   plugins in statically compiled languages. It's certainly possible. Don't get me wrong.  

But there are some specific challenges there. I tried implementing it in NIM as well as   OCaml.   I've always been a fan of OCaml, mostly from an from an academic perspective. I I probably wouldn't actually actually use OCaml.   But then I stumbled across a language called Julia, and I'm a huge fan of Julia. I I love what those guys are doing. I love the expression of a functional paradigm, the way that the way that they've got it. I also love how they compile it, how they use LLVM,   and how that whole statement by statement JIT compilation works.   I I'm probably not explaining it properly, but I mean, if anybody's interested, Julia is fantastic. I love Julia. And so I implemented this inside of Julia, and in doing so,   not only did I realize that there were a number of aspects of functional programming that I felt could be really helpful,   I also realized   that my fundamental approach from a salt plug in perspective   had a number of limitations   around how those plug ins were shared   and refreshed.  

And while working with Julia, I realized that   these things could be solved. So if we come back and we look at how the salt loader works,   the salt loader works   by well,   1 of the things that we've gotta do when you make a plug in system is that that plug in system dynamically changes   the name, route, and implementation of specific functions as they're exposed   relative to the underlying programming language. And so   being able to   to change those means that in salt, I've gotta take those raw functions, and then I need to,   embed them,   basically inject them, you set adder inside of Python   to inject those raw functions directly on top of the imported module object. Now when you put something on directly on top of the imported module object inside of Python, then that manipulates the imported instance of the module inside of sys.modules,   which means that if you ever want to reimport that module for any different use case, you're well, you're you're stuck with   all of these things already manipulating that namespace. So then inside of Salt to solve this problem, we had to come back and we had to move it so that we could so that we could import loader subsystems ended up in different physical locations under Sys modules. So we have multiple copies of modules that are imported, and and it's   kinda dirty.  

Whereas   what I realized when working with Julia   is it would be a lot more elegant   to instead make the   never never manipulate the module layer, but make it so that that module system   is   all abstracted   on a central,   on a central   hub into a hierarchical namespace. I mentioned earlier that there are certain concepts in computing that have endured   despite, frankly, lots of efforts to try and get rid of them. And 1 of those is a hierarchical namespace like like a file system. I mean, you can't get very far in computing without understanding how a file system works, and it's such an effective way of organizing data. And so inside of PoP, there's this concept called the hub. Now the hub is the root of your application's   hierarchical namespace. And so when you make a new plugin subsystem, that plugin subsystem   just exists   on that hub so that it is now accessible to any other piece of software that you've got inside of that process,   and   then that solves the problem that we had inside of the salt loader where   if I had 1 plug in subsystem inside of salt and I wanted it exposed to another plug in system, let me just say that there was some   finagling that had to happen under the hood to make that to make that work. But again, instead of pop, you put that thing on top of the hub just by saying, hey, I'm gonna make a new plug in subsystem and your entire application has access to, to those functions, just making that problem go away. And since we don't manipulate the module layer, everything is on the hub, then it's makes it much easier for us to be able to manipulate memory name spaces directly on this directly on this hub   and the hierarchical   objects   therein.  

So that that   allows us   to be able to reuse functions   for operation inside of their own kind of environments depending on how they're invoked, which goes back into giving us the benefit of having multiple   instances of objects that we get from object oriented programming   but then adds a benefit of all of the data associated with that object   being   namespaced to its when   you   when when you're looking at things from a paradigm perspective, it's interesting to kind of say what are some of the caveats that we run into? And 1 of the challenges that that   particularly annoys me with object oriented programming   is that you run into this thing that I call over encapsulation.  

We we didn't even think about it because we're just so used to writing software this way. But my argument is that over encapsulation is that once you start writing pieces of software and it get it get starts getting big at all, you end up with an instance of an object   in a file over in 1 part of your code, and then in another part of your code, you say, oh, I so wish I had access to that data. It's all the way over there.   How am I gonna get access to it? And then you can solve with a number of ways. 1, you can implement global. It's like, don't do that.   2, you can use language backdoors, which are almost just as ugly. Right? You you go into the languages internal hierarchical system and finagle it to find a backdoor to get to your object. And then, I mean, that makes incredibly   brittle code. Or you end up doing what most people do, which is you go up your call stack   to the point where that data was originally available and then you fork it down different call stacks. Inside of polyp, you just put that data on the hub, and it exists inside of this hierarchy. It's not a it's sure. It's globally accessible,   but the data exists inside of a hierarchical namespace.  

And, again, like I'm saying with file systems, hierarchical namespaces make so much sense. And so   then we don't have the problems of googles because you can look at where data is physically located and say, hey, that's not inside of my this   plug in subsystems namespace, so I shouldn't manipulate it, but it's right there. Right? I can make a copy of it.   I can read it and,   use it for the instantiation   of a new a new object or construct over here. And so I guess I guess what I'm saying is   I rethought how you expose memory to an application developer   relative to hierarchical namespaces   because Julia forced me to make a hub   when I was implementing it in Julia. It was it was just 1 of those, oh, mind expanding   kind of things. And so when I came back and reimplemented it again in Python,   Yeah. It it just became a lot easier. And that's also where I started to realize, oh, I'm expressing a paradigm, not just a really slick plugin subsystem.  



[00:32:17] Unknown:

And when I was reading through the documentation,   1 of the things that you   explicitly call out is that pop itself is just a single implementation of this over overarching idea, which we've been discussing, and that there's nothing that's Python specific about it. And I'm curious,   1, how large the actual POP framework itself is just in terms of code and complexity,   and some of your thoughts on   specifics of your implementation that wouldn't necessarily   translate well or would even want to translate to different languages or run times. So, I mean, that's that's a great question.  

[00:32:52] Unknown:

How big is pop? I haven't run w c on that or, you know, no, l o c on that for a while. Let me let me take a look.   Yeah. So if I run, lines of code on pop,   it says that we've got 4, 218   lines of code. So it's it's not   huge, and that's that's 1 of the goals of pop is that these individual pieces of software, they don't have to be big, And that goes right back into maintainership. I mean, how many lines of code can you fit in your head? Right? When I was originally developing Salt, I I still knew where everything was until we got up to about   somewhere between 5, 070, 000 lines. And   I can I can know where everything is across 15 projects that are smaller than 10, 000 lines   because they're all better compartmentalized, and I can transfer the knowledge over so much easier? I can introduce somebody to the entire pop code base in,   frankly, a week of sitting down and training an employee for instance, or sorry, training an engineer. And,   that compactness is really valuable.  

Now when it comes to other languages, 1 of the great things about Python is that it just it makes making these constructs so easy. Right?   Operator overloads in Python,   you know, being able to create a class that has,   a getattr and getattribute and setattribute,   and all this stuff is   so easy.   But with that said,   I I don't see anything that would restrict   one's ability to implement this construct inside of Julia, for instance, or even inside of c plus plus.   So any any language that   gives you a sufficient   a sufficient object oriented interface   should be expressible this way. And I mentioned earlier that it's it's kinda difficult to do this with   compiled languages. But at the same time when I was looking at that, I was not willing   to sacrifice the idea of being able to dynamically update your pluggable software while it's running. If you sacrifice that idea, again, it becomes much more plausible in in compiled languages. So I would love   to to see an implementation   of pop in something like Go or Rust.  

The other thing from a language perspective is that as I've sat down and worked with,   front end developers and people working in JavaScript   and expressed and, and gotten them on board with pop,   oftentimes, they've come back and said, wow.   This would solve a lot of the problems that we have in front end development,   because all this data exists on top of a hierarchy. It's easy to get access to.   The plugins are always accessible from any app from any location inside of your software.   It would make it so that extending something like JavaScript would be really easy. And so, personally, I would love to find some time somewhere to,   right, come back and try and implement this in a few other languages.  

But,   yeah,   I I I got a lot going on.  

[00:35:55] Unknown:

And 1 of the challenges   when you're dealing with applications   that are decoupled   at the interface level is just defining those interfaces and ensuring that any implementation to those interfaces adheres to that contract. And I'm curious   how you've approached that idea in PoP as far as being able to say, this is the stub of where you can add a new plugin.   These are the functions that it needs to define, or these are the data structures that it needs to be able to produce or accept. And then having a way to easily, on the other side of things, say, I'm going to write a plugin   that adheres to this interface and then be able to independently validate   that it's going to work when you go ahead and incorporate it into the broader runtime or the,   or you load it into the   application hub. And so I'm curious how you have thought about that, and if there are any sort of ways for you to be able to publish those interfaces   from the hub or in some sort of documentation format, and then be able to consume that interface specification   at the implementation side to be able to make sure that what you're writing is going to work when you actually go to push it to production. So 1 of the things that I love about talking to you, Tobias, is that you just gave me such a softball. Clearly, you've read the documentation.  



[00:37:11] Unknown:

So 1 of the big challenges that we run into in Salt is exactly what you're describing.   We basically say, hey. Here's some functions that implement this kind of pattern. Just or, yeah, here's some plugins that implement this pattern. Just kinda just copy them, and you'll be okay. Oh, by the way, you don't actually need function x.   So 1 of the first things that I implemented inside of pop is is this concept of contracts.   And   you can't make really pluggable   software   if you don't   have,   some way to enforce interfaces.   And so if we step back and and look at this,   contracts inside   of pop allow you to say that   any sort of plug in interface that exists   can be enforced down to a function   level. And when I say function level, I mean, all the way down to function signature validation.  

So that when,   when pop loads up a plug in, it's able to inspect that plug in,   compare it to a contract that defines all of the necessary functions and their function signatures so that it can come back and say, no. No. No. You you haven't implemented this correctly because you aren't accepting parameter Foo.   Therefore, I'm gonna I'm just gonna stack trace before I even try and run your code,   which means that we're able to avoid   having those situations where an improperly implemented plug in ends up blowing up on somebody long after it's been, it's it's been set up. And so the contract system wraps every 1 of the functions in every 1 of the plugins   that are expressed on on top of the hub. So what that means is that we can do   load time validation   so that as soon as that plug in subsystem gets loaded, it can come back and say these functions are good, these functions are bad, but it can also   wrap the actual execution of functions.  

So you can kind of think about it think of it like decorators,   but   since we're allowing people to create entire plug in systems, I felt really strongly that we needed to be able to make decorators   transparent.   So that if somebody wrote a plug in for a plug in for a pop program, that they would be able to then come back and say, look, The wrapping functions just magically work for them. And this is another thing that contracts gives you. Contracts allows you to define   pre, post, and call functions   so that you can say that anybody who implements function x before that function is executed, we're gonna execute a pre   function.  

And then that pre function, you can do things like input validation   or even modify,   what your arguments and core arguments are that are being passed into that. And then the post allows you to do things like modify the return. Modifying a return allows you to do   I mean, 1 of my favorite things that modifying the return allows you to do is to say, well, I'm gonna express this plug in subsystem as a front end to maybe an RPC or a rest interface. And so I always want that return to be a certain a certain object. Right? Well, now you don't have to put the creation of that object on your interface implementer.   You can just do it inside of your contract   once instead of having to instantiate your returns repeatedly as said object. Another thing that,   that, we've used this system for   on the pre and the post is that we've got a central location where all of the   role based access controls are implemented. So it makes it really easy to be able to say things like, yeah. Anybody who adds a function that accesses the database is going to automatically inherit an RBAC layer so that   somebody can say, yep. I've got a database interface, and I'm gonna use plug ins to implement,   all of the right functions to use a different database back end. But   the actual access to that database back end is still regulated by 1 unified RBAC system.  

And so   in in some of the projects that we've got inside of SaltStack, we've used this to implement RBAC, not only on the database layer, but also on the, RPC and rest layers. And so contracts   are incredibly valuable.  

[00:41:12] Unknown:

And another thing that you mentioned   is the ability to merge these different plugins or even   applications implemented using pop into larger applications   for   runtime   or for deployment.   And it seems that it would also be possible to take 1 of these larger applications and then decompose it into independently deployable microservices.   And I'm wondering,   what are some of your thoughts on the challenges that an engineer might face when designing and implementing a library   for being able to take advantage of these capabilities, either in the merging or the decomposing   aspect, and some of the,   sort of best practices around how to make sure that the plug in that you're building or the subsystem that you're implementing and defining is going to be able to work in these environments.  



[00:42:02] Unknown:

Now this is an area where I need more documentation   and also an area where I'm still learning. It's been really interesting to come back   and implement a bunch of little projects. So I've implemented a lot of little projects using pop. Most of them originally were just to try and   you ferret out these sorts of ideas. You really have to look at your application through a different lens when you're creating an application that's purely pluggable. You end up looking at it and saying, well, how is my instance data going to be expressed on a hub? What are the implications   of app merging?  

What   how what do I need to do to make sure that my system is still really flexible and pliable,   but also still simple to the point and gets the job done that I'm trying to that I'm trying to execute? And so this falls back on on an old kinda saying of mine that is, if you have a really complicated problem,   the best thing to do isn't to try and just   hit that problem head on,   but it's to minimize the impact   surface area of the problem. And then that minimized surface area becomes   manageable. So instead of looking at things and saying, I wanna solve memory management for every piece of software that's out there, 1 would come back and say,   how do I make it so that it's just really easy to keep track of my memory usage? What if I made it so that it's really easy to know where the where the bodies are and get alerts really quickly about,   performance and profiling and memory issues.  

And so if you take that approach, then your surface area becomes small and compact.   And   then your application can grow through application merging.   So when we look at application merging,   pop   currently, I   I don't know of any other ways of doing this yet, but we'll see. PoP currently expresses 2 types of application merging. 1 is called horizontal   application merging, another is called vertical application merging. Horizontal application merging   is a lot of what you were just describing that let's say, I've got 15 projects, and I wanna merge them all together so that all of their namespaces   really just end up existing on 1 hub, and they get operate and function inside of 1 process. And so as long as we follow the rules of not violating the namespace of your hub and making sure that your applications   follow these simple rules of make subsystems that are pluggable and, easily extends extensible,   then this becomes easy. 1 of the big challenges though with horizontal application merging   is the fact that each 1 of these applications that you merge together has merit and use in and of itself,   which means it's gotta have its own command line options. It's gotta have its own configuration   loading,   and then that configuration   loading needs to be expressed inside of all of these plug in subsystems.  

And so that's 1 of the reasons why pop has the configuration   loading system built in. It's just recognizing that part of an application   means you've got to be able to bring in configuration   data. And so Pob Express is a unified   hyper simplified way to bring in configuration data   and make that configuration data available to your project even when it's been merged together with lots of other projects. The other challenge that we're into is the difference   between   CLI invocation of a a program   and being able to call out that program from an app merge perspective. And so 1 of the little   little surface area issues that I've tried to minimize is to make it so that you can have all of your configuration options come in for an application,   and then you can have a really, really thin   function that just says, this is what it looks like if I invoke this from the CLI,   and   I'm still just expressing a bunch of functions that can be called from an app merge perspective. And so, again, it's all about saying, normally, these problems are really big, but let's let's make the surface area really small to make it so that big gnarly problems become manageable and that the user only needs to learn that, hey, as long as you keep a couple of these things in mind, you'll be okay instead of,   instead of having to think about   ongoing   really complex problems that have to be implemented and maintained   throughout a large code base, which is something that we we run into in in paradigm development in general. Does that answer your question?  



[00:46:27] Unknown:

Yes. It does. And moving on from some of the technical implications   of pop into some of the more social and communications oriented ones that we alluded to earlier, I'm curious   what you anticipate   as far as teams who are using this paradigm for developing their software, how that impacts the possible team structures and communication patterns that are possible,   where a lot of times   we have teams that are either centrally located or distributed, and so that influences whether we build monoliths or microservices.   It seems that this might unlock some slightly   more nuanced or evolutionary   ways to design and implement team structures and how that reflects in the software that we're building? That's an excellent question because  

[00:47:14] Unknown:

when when we look at building teams to develop large pieces of software, there's this this book that's,   kinda old. I wanna say I think it was released in 1974   called the mythical man month, And I love this book. I was surprised how few people have read it nowadays. I mean, when I started out in college, it was kinda mandatory reading. Right? Everybody had read mythical man month. Makes me feel old.   But the mythical man month was this book that was that was designed to be sent to managers who all of a sudden were trying to manage software engineering projects. And it's trying and 1 of the things that it's describing is that look, if you've got an assembly line and you wanna double production   for that assembly line,   then you buy twice as much equipment and you hire twice as many people.  

And for the most part, you're able to therefore double   double production.   And then mythical man month explains that this just doesn't work for software because you bring in   more people, and they have to understand   this this code base that you're working on so that it can be extended. And so with that in mind, I've spent a lot of time looking at how how we develop software, where the innovation comes from,   where the real guts and power of software comes from. And   there seems to be a strong correlation   between   small software projects   and rapid innovation,   rapid prototyping   relative to large projects that can become very, very difficult to prototype on top of unless you've got a strong compartmentalization   And   really successful   hyper large pieces of software   have strong compartmentalization,   like like the Linux kernel.  

Right? Somebody can come in. They can make a driver. They can make a file system. They can all they can get on with their lives.   But the majority of software really doesn't seem to be this way. And so 1 of my goals with PoP was to say, look.   If our application is broken up into lots of small services, which is, again, similar design to microservices,   then it becomes a lot easier to change what our team structure looks like as far as internal management. But I feel really strongly that this can help us with,   open source   software as well. Because when it comes to open source software,   I'm I'm I'm endlessly   surprised how difficult it is to convey the construction of certain aspects of   of, my code bases out into a community.  

So there are certain parts of Salt that I'm still the only 1 who's ever built anything inside of these components.   Whereas,   if we're able to break these things up, they're more conveyable, they're more understandable, they're more consumable,   how these things are actually built. And so it goes back to something I said earlier where   if if you have   if if you've got a code base that's a 1000000 lines of code, like solved, and I hire an engineer and say, hey. I need you to help me maintain this thing. It takes them I mean, sure. They're they're making contributions fairly   quickly, but until they really are able to grasp the big architecture, it can take months of them just being in there   until they're able to get that big picture. Whereas, when things are pluggable, I can sit down with somebody, and I can convey the whole application   in, I don't know, an hour, and they're able to own it and start running in a really deep way with that component   almost immediately.  

So for instance,   I just, released a piece of software built on top of pop called heist.   And I'm I've been able to sit down with engineers and, again, literally   in 45 minutes, they were able to repeat back to me the entire functional code base and pathways, and they're able to basically own that piece of software. And that is I feel   absolutely transformative   that somebody is able to just sit down and say, yep, I'm versed on these   8, 10, 15,   4   pop projects,   and it makes compartmentalization   of teams   a lot easier because then you can just say   who's been versed on what, who needs to be versed on something else. You know, you can draw some Venn diagrams, and you immediately   understand what your actual internal software coverage looks like. Right? What does your expertise look like? And then you can have rapid innovation because when you break something up like this,   the SALT's test suite takes hours to run because the the darn thing is so big. But I can make extensions to item, Right? The the pluggable implementation of salts configuration   runtime   very easily because its test suite runs in 4 seconds.  

Because it's only gonna test itself. It doesn't have to go and test 18 different operating systems. And so   the this whole concept of rapid development changes   and proper compartmentalization   and management changes and can defeat the mythical man mug where I feel like this has the potential   of creating   significantly   more rapid development cycles inside of companies.   And that's something that personally I'm I'm really, really excited about and that I've got really high hopes for. That we can prototype software faster,   we can get features out faster,   we can write components   once and not have to come back and say that, I've implemented say, well, yeah, I've implemented   a number of these components   multiple times. Even when you're using libraries, you're still gonna glow this stuff together. But if you're using app merging,   you can build that entire application component stack and reuse it without   any modification   over and over and over again. So, again, the   it's it's like everything just has super glue on the side of it instead of saying, hey, I've got a bunch of blocks and I've got to figure out how to glue them together. Yeah. This whole approach definitely seems like it does a good job of tackling some of the friction that occurs  

[00:53:05] Unknown:

when we try to reuse software, where that's generally the ultimate goal of write once, reuse many times. But then, as you said, particularly with libraries or, know, particular   applications that are intended to be incorporated into a larger whole or work alongside them, there's always some amount of rework or custom updates   or glue code that needs to happen in order for things to actually be reused.   And it seems that by having the entirety of the context encapsulated in this plugin subsystem,   it makes it much easier and more likely that you can actually implement these aspects of your domain logic in the subsystem and then be able to reuse it across multiple code bases that need to be able to handle some of those same problem domains.   So I'm definitely   excited to start experimenting with that on my own. But for   anybody else who has started drinking the Kool Aid at this point, I'm wondering,   the other side of things, what are some of the limitations   of pop, either in the implementation or the broader paradigm   or cases where you would recommend against  

[00:54:07] Unknown:

following this paradigm, and you would recommend for something more traditional or something that people are already familiar with? I don't feel like I've run into a whole whole lot of situations where I don't feel like this works really well, to be honest. And a lot of that is also that I've tried to put myself in a lot of situations of building different types of prototype software using it to fill out the paradigm.   Now most of what I've built has been   system and server side software. Most of what I've built so far has been,   things like management software, communication software.  

But when it comes to the flexibility of the platform,   1 of the pieces of software I wrote with it is called Umbra. Umbra is an artificial intelligence,   streaming   system. And using plugin oriented programming, I was able to very, very easily implement   a flow programming paradigm   so that I was able to do,   multiple data stream processes,   in parallel.   And implementing that on top of pop was   so much easier than implementing a flow system   otherwise because we're just able to use this hub and put, like, asynchronous queues on it, and it just works. And so   I'm really hesitant to sit here today when it's so young and say,   x y z just   isn't gonna work well with this because I don't know yet. And   and so far, everything I've tried has worked really well. Now with that, you you kinda have to think about your problem a little differently.  

It's   it's not like it's terribly obvious. And and some of the challenges that I've run into with people trying to build applications on-site on on top of pop,   is that they come back and they say, I can't imagine   needing to make my application this flexible.   And pop really pushes you to make really flexible software, really growth oriented,   big pieces of software   were   broken up into little pieces.   And so   if you're making something that is small that you can't possibly   imagine ever getting big. So 1 example is 1 of my 1 of my employees was just practicing   making a a an application that plays cards, And I'm trying to explain,   okay. I get it. You've made some classes that implemented   a deck of cards and poker and whatever, and I'm going pop from a pop perspective, you would look at it and say, okay. I wanna make a plug in subsystem that is able to flexibly create any type of deck of cards, which could then be applied to   any type of game   that could possibly be made with cards so that this platform could be used to   run a poker game as well as, like, Pokemon   or, like, Magic the Gathering or whatever. I'm trying to think at random card games. And it's it's not too terribly difficult to look at somebody who says, I just wanna write a poker game   and say, okay. You really don't need to think about your application   at this level of abstraction.  

So I guess that that's a good case of maybe this paradigm wouldn't work too well.  

[00:57:13] Unknown:

And   for anybody who is interested in getting involved in contributing to or experimenting with and giving feedback on POP, what are some of the areas where you're particularly looking for help? And what are some of the plans that you have for the future of the PoP library, either in terms of the specific implementation of it or some of the types of applications that you want to allow it to be used for?  

[00:57:38] Unknown:

So right now, I've made   and released   heist   item and umbra   on top of pop, and   I'm currently working on   another project,   I'm calling Takara.   It's really, really young, so I I don't know if it's early enough to talk about it. But from a contribution perspective,   I would say that most of what I've learned about extending pop has come from building applications using using the framework and the paradigm.   And so   if somebody wants to contribute,   I would say write whatever the devil you wanna write, but use pop and then come back and just ask questions like, hey. I'm trying to accomplish x. Have you run into this before? Does this merit an extension to the core platform? And then it becomes a lot easier to come back and say, yeah. That does merit an extension to the core platform. Let's take a look.   Or, hey. We've solved this problem this way a lot. Does that make sense? Do you think that we need to refine   how this aspect of the paradigm works? And so,   historically,   I'm so used to salt worth, like, you wanna contribute to salt? You're there's all these modules. Just add some modules. Pop itself is I mean, it's a paradigm. It's it's kind of a different animal, and we will learn more about it by using it more. And so if somebody ever writes a pop application,   please please let me know. I'm going to make a pop awesome   repository so that it's easier to keep track of, hey. Here's a bunch of applications that are written in pop. But beyond that, I mean, that's really the place to start with contributions.  

If we step back and we look at it from the perspective of some of these other projects,   item, for instance, already has a growing group of people who are building support for cloud management   on top   of item   so that,   we can port a lot of the functionality of Salt Cloud to be more standalone, more flexible.   And similarly on top of item,   we're we're trying to get more people who are interested in taking it as just a language. Because, again, it's it's basically salt states, but item allows you to run salt states decoupled from being part of a big salt ecosystem. And so a lot of what we wanna do there is actually port   stuff from Salt over to item or just allow people to write more,   interfaces to manage whatever they want,   on top of item. Whether that's an arbitrary   API somewhere like a GitHub API   or whatever. Right? A bamboo HR HP API. I don't know. So, I mean, those are some areas of contribution that are interesting. But, again, the reality is is make whatever you want, but try it out with pop and let me know how it goes. And so we can make this thing better. Are there any other aspects  

[01:00:19] Unknown:

of the pop implementation itself or the overall paradigm or some of the ways that you're using it that we didn't discuss yet that you'd like to cover before you close out the show? We have fantastic covering coverage. I'm also we forget contracts and, you made sure we got that too. So I I think we've got pretty good coverage here. Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing or contribute to some of the projects that you mentioned, I'll have you add your preferred contact information to the show notes. And so with that, I'm going to move us into the picks. And this week, I'm going to choose the show, the man in the high castle.   I, just finished watching the last season of it, and it was great all the way through. So I thoroughly enjoyed that. So if you're looking for a show to watch, definitely recommend that. Not as friendly for kids, but, all in all, a great story. So with that, I'll pass it to you, Thomas. Do you have any picks this week? I've really been meaning to,  

[01:01:10] Unknown:

to watch man in the high castle. I saw the first 2 seasons, and it was just phenomenal.   And and that did just come out. So that that has been   that's that's a really cool show. I think a new season is coming out or just came out on,   Tom Clancy,   Jack the Jack Ryan season   on Amazon Prime.  

[01:01:29] Unknown:

I'm really looking forward to that 1. Alright. Well, thank you very much for taking the time today to join me and share your thoughts and experiences   experimenting with and seeing some of the directions that you take Salt and associated projects with. So thank you for all of your time and efforts on that, and I hope you enjoy the rest of your day. Thanks, Tobias. It's always great to hear from you.  

[01:01:56] Unknown:

Thank you for listening. Don't forget to check out our other show, the Data Engineering Podcast at dataengineeringpodcast.com   for the latest on modern data management.   And visit the site of pythonpodcastdot   com to subscribe to the show, sign up for the mailing list, and read the show notes.   And if you've learned something or tried out a project from the show, then tell us about it. Email host at podcastinit.com   with your story.   To help other people find the show, please leave a review on Itunes and tell your friends and coworkers.