The Python Podcast.__init__

Share on social media:

[00:00:00] Unknown:

Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. When you're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it. So check out our friends over at Linode.   With their managed Kubernetes platform, it's easy to get started with the next generation of deployment and scaling powered by the battle tested Linode platform,   including simple pricing, node balancers, 40 gigabit networking, and dedicated CPU and GPU instances.   And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover.  

Go to python podcast.com/linode   today to get a $100 credit to try out their new database service, and don't forget to thank them for their continued support of this show. Your host as usual is Tobias Macy. And today, I'm interviewing Bruno Rocha about Dynaconf,   a powerful and flexible framework for managing your application's configuration settings. So, So, Bruno, can you start by introducing yourself?  

[00:00:55] Unknown:

Hi. Thanks for inviting me for this podcast. My name is Bruno Rocha. I live in Portugal.   Some things about me, I'm a member of the Python Software Foundation.   I'm a senior engineer on Red Hat Ansible.   I have a YouTube and a Twitch stream channel   targeting mostly Portuguese speaking audience where I talk about Python and programming.   And besides that, I like bicycles. That's all about me. And do you remember how you first got introduced to Python?   Oh, yeah. Let me remind this. It was around 2, 003.   I worked it in a factory,   and my job was maintenance of 100 of Linux desktops.  

And we use a Linux distribution   called   Linux, which is a Brazilian Linux distribution.   And 1 of the features of that Linux was called Magic Icons.   So it was like a GUI application   with buttons where user can click and run   bash scripts or things to automate Linux   stuff. And then I   work it written those scripts in bash and pearl for a long   time and adding more day to day off c activities to those icons like merging files,   printing and doing all the offsy day to day job using Pearl basically.   But at some point I started contributing. I think that's was actually my first open source project that I was a contributor.   And when I started contributing   back to the, magic icon system on on Linux,   I found that they were moving from Perl to Python. And then I had to learn Python to rewrite   those script bits, that distribution. Today, that distribution is no more tainted, but that was a good experience. And that's why I I first learned Python. And from that point, I stopped using Perl and then kept using Python for everything. Yeah.  



[00:02:51] Unknown:

That brings us to now the topic at hand, which is the DynaCon project. I'm curious if you can share a bit about what it is and some of the story behind how it got started and why you decided to build this project.  

[00:03:04] Unknown:

Well, so yeah. Dynacomf is is a library that I like to define it as a settings client.   So   settings usually are defined in multiple locations like   static files. You can have a py file, toml, yaml, JSON, or any of the other common formats   usually you do for application settings.   The settings is overridable   by environment variables   and even by external servers like in memory database or secrets vault.   And then at some point you need to gather all those information   in a single object. And this object is DynaConf like DynaConf is a I like to think it as a settings client   that I can configure   and point to multiple sources. And then it gives us a single settings object, a single single Python object where you can access and deal with those settings objects. So that is basically DynaConf and and if I want to if I can explain it simply, it's like it's a library to access settings.  

However, the settings are located,   then it brings you a a bridge for you to access and and get those settings together.   And the other question was the story behind it. So   I work at in a company that was running Django and Flask application on the Cloud on AWS.   It was 2014   and the main website in API, they use it load balance   and auto scaling.   So eventually,   there were 100 of EC 2 instances running on the cloud.   And they wanted to be able to share the same settings with all those instances that were running. So you can think like applications will start having   on the environment   or hard coded settings like in a constant file. But what they wanted in that project is to have a single place where the settings is stored, like an s 3 or any other shared file system.  

And all the instance could read from the same place, the settings. So this was like the first mission we we did.   And then   the second thing that we did in the project was like   turning the settings dynamic.   So for that time, we use it already   instance   and already is stored key values   and then all the instances read key values from the ready's instance   inside the AWS   network.   So that's how   the library started. So I just went to the main project and I created a folder called dynamic   config.   And then I started like coding using Python dynamic features like overriding get attributes,   trying to do like lazy evaluation.  

And then I ended up with something really nice which we were able to   use it to read files locally. If it was not found locally, it would fall back to this to the environment   and then to external services.   So we decided to like promote it to, to a package. And then   instead of dynamic config, we shorted a little bit to Dyna Conf and created a package. It started being used by all the projects, all the internal projects on that company.   So it was 2014   Heroku   released   a guide called 12 factor apps.   And 12 factor apps was like   our main drive for deciding how the library would be written.  

Because after this first version, we did many refactorings   and we base it a lot on 12 factor guide.   The 3rd topic on the 12 factor is the config   and that says that configuration must belong to environment and then we started like coding DynaConf to be like environment first.   And from that point,   we we added we added many other features. But that's how it started. It started, like,   just as a Python model inside a big monolithic project,   dynamic config, and I remember that we pulled that to   off that project and created the Dynacom.   And yeah. Now   it grow like it's no more like a single small library. It's more like you said on the beginning of the show. It's a framework for managing settings   together with this settings client. So, yeah, that's a little bit of the time.  



[00:07:36] Unknown:

Are there any particular   goals or kind of design philosophies   or   strong opinions about how to manage settings that you have for DynaCon?  

[00:07:47] Unknown:

The fact is that the project is evolving organically.   So it's based on the needs and feedback of the users and contributors.   So we are constantly   adding new things   and fixing things based on that.   So we don't have, like, a really strict guideline   on   on the style that we are following.   I can say that the goal as a project is to be reliable and simple to use.   So I all see that I want Dynacomf to be a go to choice when you think about how to manage settings on your Python application.   I already see that on the companies   and communities where I participate because I try to influence people sometimes to use the library and they say, yeah, we need to deal with settings. So it's gonna be like, just put Dynacom here. And to be able to be like a go to choice and simple,   you have to be very   like frictionless.  

So the project should be something you can plug into your project   and you can unplug from your project easily. So I think this is a thing that we try to keep on Dynacomf. You can take your project right now, add Dynacomf   and it works. And if you need or if you don't like Dynacomf anymore, you can just remove it. And it's not going to be a hard work because the way Dynacomf is implemented.  

[00:09:10] Unknown:

As far as the kind of types or categories of projects that you have in mind as you build and iterate on Dynaconf, are there any particular ecosystems   or use cases that you are more focused on or that tend to work better with Dynacon?  

[00:09:26] Unknown:

Yeah. Definitely.   I think web applications running on cloud is the main target.   So as we base it on 12 factor app, I think   whenever you have environment   and you have this environment varying   across multiple deployments,   I think Dynacomf is a very good fit for it. So it can be running on virtual machine, containers, Kubernetes,   everything about the 12 factor apps, which is like payment for SaaS applications.   I think it's a perfect fit for Dynacom. But we have users using Dynacom for other projects. So have people and contributors   running for machine learning pipelines. So they configure all the models   using Dynacomf   loading of of Dynacom files.  

We have CLI applications that people create and then the end user of the CLI application   like puts a settings file together,   in the folder they are running the CLI and DynaCon floats this the local settings. We have desktop applications and testing frameworks.   So   the first focus was cloud applications, but people users starts to be, like, more creative in using it, whatever you you imagine. So I think DynaConf can fit on many cases, but the main target   is when we are looking to, like,   change things, when we are looking to add new   functionalities, we always think about the cloud first. Yeah.  



[00:10:57] Unknown:

As far as the   category of settings management for applications, it's 1 of those things that is   seemingly very simple, but is deceptively complex as you dig deeper into the problem space where at the outset, it's just, oh, I just need a file that says to, you know, set these different values,   as you were describing, move into different environments, or you need to be able to support different run times or kind of application structures,   you start to realize, oh, now I need to be able to pull things from multiple places. I need to be able to manage things like overrides or, you know, how do I manage secrets? And I'm wondering what are some of the   kind of general philosophies that you've encountered about the so called, quote, unquote, right way to do it and some of the ways that you think about what philosophies are best suited for   applications that you're trying to kind of encode into how Dynacon is able to manage some of those things like overlays and environment specific attributes and secrets,  

[00:11:58] Unknown:

etcetera. I think that   settings is like an accessory for your application. It's not the main focus of any application.   So your application should focus on the things it should resolve.   And   as   less you see the settings the best because it will be, like, transparent   and it's not, like, being on your way from you and your code. So I think 1 of most complex thing   to do when you are creating your own settings management   is trying to do this in a more like frictionless way, transparent way. I see lots of settings objects where you just to read a single variable,   you need to call methods like   get, read, parse,   and you need to do your own specific type testing on every place.  

And then when you look to the code, you have more, like,   interaction with the settings than other things. So I think   what I learned at that is if you do that,   you are like coupling   your code with your settings library.   And the main philosophy philosophy for Dynacomf is to be like decouple.   We want Dynacomf to be decoupled and transparent from the project.   The Dynaconf API   behaves like a normal Python model   or a dict like object because a Python model with attribute access or a dict like object when you look up by keys   is the 2 ways people read settings. If you have a constant file, it's gonna be attribute. If you have a JSON dictionaries like after parsing. So Dynacom supports   both APIs and you can like treat Dynacom just as normal Python object. You don't have any kind of special that you need to to call or need to prepare your application. You just use as normal Python variables. I think this is important because, as I said, at any point in time, you might need to change your settings manager.  

Even if you have your own written settings manager, you might want to migrate to DynaConf. And if your   settings are really agnostic, like simple Python objects, it's gonna be easy to plug Dynacomf in your project. In the same way, the other way, like, if you want to remove Dynacom, it's just like removing the object created by Dynacom for placing to another object that behaves like a Python object and everything works. So I think this is important to decouple the settings from the   project.   So   there are challenges that you're gonna be having dealing with settings. Like,   if you   want, like, dynamic features like laser evaluation   and overload   access from the settings, like, you you want to put middlewares   between who is accessing your settings and where the settings is doing transformations.  

So this is gonna be easier if you do this in a transparent way because Python offers   lots of features for dynamic programming. You can override.   You can do laser evaluation. You can override attribute access.   So it's not that difficult to use Python to create a layer   of abstraction for assessing.   We see that with ORMs   for databases   and then a constraint to fit that for the settings. So, yeah. And   about the right way, what would be like the right way to treat the settings?   I still stick with the 12 factor apps. I think settings   belongs to environment.  

So   the application   must have reasonable   defaults.   It should be able to run without any specific setting.   Application could have, like, required settings, but it should be able to run with minimal settings.   But everything should be, like,   belonging to the environment where application is running. So the code is actually not. I think it's true   for cloud applications because you can run-in multiple cloud providers.   It's true for CLI applications if you distribute it to multiple targets and and multiple ways of running.   So,   yeah,  

[00:16:14] Unknown:

I think that's what I have to say. 1 of the things that you noted in there was the question of being able to   cast the different values into their appropriate types And with Python being a dynamic language and also the, I guess, not necessarily evenly distributed adoption of type annotations, etcetera,   that is an interesting question for being able to manage settings in Python applications.   And what you're saying about things like parse pool, parseInt, I've definitely seen that all over the place, both in widely used libraries as well as in some of the libraries that I've seen some of my teams kind of build in house.   And   I'm interested in understanding your thoughts on kind of how to   encode some of that type information in the settings that you're loading for your application as far as, like, what type should it be. And then particularly in the case where maybe it's a string, but it should only be 1 of a particular set. So you, you know, maybe you wanna have an enum and just being able to understand, like, what are the possible range of valid settings for this attribute and then being able to bring in things like   annotations   or documentation   around the purpose for a given setting so that as you come into a brand new project and it has the settings defined, you can say, okay. I understand what this value is supposed to do and why.  



[00:17:33] Unknown:

About typing, in the first version of DynaConf, we decided   to be transparent and frictionless.   I love this word like frictionless, like you just do settings dot key and you fetch the value you want   and with the type it's defined by Dynaculf.   So we wanted to avoid the need for you to call settings dot key dot cast   or settings dot get and then you put the key and then type that you want that to be tested. So we want to avoid this extra work and and give you the final data to use. So the first approach, which is already supported today but it's not highly used, we added type annotations to the variable itself. So as it's based on environment variable   and environment variables   as our first source of truth   on an application   environment.  

You can only export strings and Python reads all environment variables as strings. So what we did, we created some tokens, some markers.   It has nothing to do with type annotations.   It's just you put at JSON and then you can put the JSON string   and export it to your environment.   You can put, like, at int, at string, at boolean. All the basic types are we have a marker.   So when you are exporting environment variable, you mark and then   space and the actual value you want to be parsed. So the type is on the source, is on the data itself and DynaConf parses based on on that. So this was the first implementation.   As I said, it works. You can actually add your own transformations. So you can create your own markers, and then you put this in your own annotation.  

Like, you have, like, a parser for a phone number or email address. You can mark this as an email, and then Dynacom will try to parse this and include   and give you back when you access, like, a structure, an object, class, anything you want. So it's very customizable.   We call it converters,   inside the the source core   code. But then when we release the DynaCon 3.0,   we figure out that   most of the users   are were migrating to TOML. TOM obvious minimal language, which is used today   on many projects like by project dot toml, Rust uses on cargo dot toml.   We see this being adopted like as a very standard on on application configuration.  

So we decided to adopt toml as our default parser.   So that makes things like easier because you can export a variable as a string   and then we pass this when reading to the toml parser and if it's a valid toml   integer then a conf reads as an integer. If it's a valid toml   float, it reads as a float. So you can obviously replace this behavior. You can choose your dynaconf to use yaml as a parser, to use JSON as a parser. But the default 1 is Toml. So you can just export, like if it looks like a string, DynaConf will infer it's a string. So there is a type inference based on environment variables.   When you are working with files that has typing, for example, YAML has its own typing, toml, JSON, then DynaConf   respects the typing coming from the the source. Yeah. But for all the other string based   sources,   typing will be inferred unless you annotate it explicitly.  



[00:21:10] Unknown:

As I was mentioning   earlier,   basically every engineer who writes an application gets to the point where they say, Oh, I need to be able to change this setting. And so they start to implement their own settings management   system.   You know, oftentimes, it's very basic. But as you   evolve the project and iterate on it, your settings management becomes increasingly complicated. And I'm wondering what are some of the, I guess, antipatterns that you see people develop as they build their own homegrown systems for   addressing settings in their applications and some of the ways that it can lead to unmaintainable   code and overly complex   settings management   approaches or overly rigid or opinionated ways of being able to specify those values.  



[00:21:55] Unknown:

Dealing with multiple sources of data   is difficult,   especially if you will have layered environments.   For example, you have a special section for testing variables, production, development,   and this varies across all the environments you are going to be deploying. So   when you get all those data together,   there is a very hard decision to make. So let's say you read, like,   2 static files like toml or yml and you read multiple prefixed environment variables.   And now you have lot of data coming from different sources. How do you merge it? How do you if you have the same key on the settings file and the same key is overrided by an environment variable,   And this is not just a simple primitive type like an integer or a string. It's a data structure. So this happens a lot on Django projects. For example, You have like a key called databases, which is a dictionary,   and inside databases you have like nested keys. And then the user wants to override   only the password,   not the whole dictionary for that key. So how do you do that? So I see that in many cases you will be requiring the user to override the whole dictionary,   but we ended up adding a way for you to override only the nested keys. So you can traverse   using the variable names   to the next nested structures   and then merge data. But when we are talking about more complex data structures like lists,   and let's say you have a list of,   middlewares that your application will   use and you want to contribute with another middleware,   how do you decide if it's gonna be on the first item of the list, last item of the list? How do you ensure it's not duplicating?  

Like, you if you add the middleware twice, maybe you are, adding overhead in our application. So   those decisions are really hard because   there are multiple opinions on it. So there are people that thinks it should be more strict. Like, if you wanna   override something, you need to override the entire object.   We wanted to add more flexibility. So there are multiple merging strategies that we offer on DynaConf.   So when you are building your variable or your dictionary, you can mark it with some   special keys or markers, and this will tell Dynacomf   what is the merging strategy for that. So what I see when people starts doing   in house settings management is that they, like,   they don't think about it at the first time. And then when the application starts growing and having multiple environments,   they start needing   data merging   in multiple environments, and it becomes a challenge   especially if you have already running   environments that you don't want to change a lot. So   that's 1 of the things. Another thing is that based on Unix philosophy, I think environment variables first. Yeah. So you should always look to environment variables.  

We offer ways to define specific subset of environment variables you want to be overridable.   But based on the subset of environment variables you have, this should be the source of truth on your application.  

[00:25:16] Unknown:

And so digging into DynaConf itself, can you describe a bit about some of the implementation details and some of the ways that you designed the project to be able to make it sustainable and maintainable   and some of the   evolution that it has gone through from when you first started building it as a 1 off solution to a specific application to where it is now, where it is this generalized framework for settings management?  

[00:25:41] Unknown:

Yeah. So I think the first important thing on this topic is tests. So   from the time we decided to   promote DynaConf   to, package,   we decided that 100% of test coverage will be the target. So we have 100% of unit testing.   I know that unit testing is not enough for all the dynamic cases. So that's why we also have functional tests, which are growing.   For every issue, we add more functional tests. So tests   is, like, helping a lot for   keeping the library reliable.   Yeah. Because it's it's really important.   And yeah. In terms of implementation,   Dynacomf is just a class. Like, it's a Python class   and it's implemented using   lazy evaluation.  

So   there's a functional model inside Dynacomf. In this functional, we created some types that has lazy evaluation.   So the user declare   configuration.   For example,   paths, validators,   all the kinds of preferences they want when instantiate   the settings object from Dynacomf.   And at that point, nothing is really performed by Dynacomf. Dynacomf just creates a class with bunch of data in it, the declaration of how we want DynaConf to be behaving,   and   then you can, like, start your application without need to, like, have all the DynaConf   process. So you don't block the import time of your application   because of settings. This is also another thing I see people doing. Like, when you have a settings library that during our time, you need to load all the files and do all the transformations, then you can block   your application startup time just to read some settings files that is gonna be used only later. So that's why we implemented this as a lazy evaluation class.  

When the first variable is assessed in dynaconf,   in your application   lifetime,   then dynaconf starts building   its loaders.   So it works with a pool of loaders and you can add more loaders to it. It has a collection of loaders that are built in. For example, it has,   JSON loader,   dotini loader, toml loader, yaml loader, environment variable loader.   So those are included in a pool in a specific order. All of this is configurable and you can easily create your own loader and plug there.   And loaders   uses the strategy pattern   where you can take the base loader and then you pass your   file reader, your file writer, and the loader methods that will be returning data. So after the data is loaded, it's just a Python dictionary   that's coming from all the different   sources. So you see this makes it flexible because on the loader you can read from a database. For example, we have cases where people wrote a loader for MySQL. They go and do a select in MySQL, get some key pairs from MySQL, build a dictionary   and feed dynaconf with that data. So   after   every loader runs, we put each dictionary in a chain map.  

Chain map is a Python object that is a stack of dictionaries.   So you put your chain, lots of dictionaries together in a single object And then when you try to access a key,   if the key is found on the first dictionary, the last latest inserted, then   it returned. If not, then the chain map is gonna be trying all the other dictionaries in order until   it do, like, tribute error or key error. So   when DynaConf does it, it's loading. It puts in a in a chain map and then we do all the things we need in the chain map like merging, parsing.   All the transformations are inside the chain map.  

So the process of of loading   from source involves parsing. So because, as I said, we can define type annotations. So when you read an environment variable, if it's marked with int, bold, JSON,   etcetera, it should be, like, tested to the appropriate type.   Also, dynaconf   supports some lathe formatting.   So you can have, for example,   environment variable or a settings variable in a text file   with jing jett and plating.   So people usually use this to calculate how many processors you have in the machine   or to call some external thing and and build the settings on the go or to reuse some environment variable.   So this is the process. So this is the heavy process that Dynacomf does. So at the end of the day when your settings is loaded, what you have is a settings, which is   a class using all the dynamic Python features   such as   overloading of get attributes and all those methods that makes   it makes it   easy to get a key and look up that on the chain map that it has. And all of this is very quick because it's all using Python. On the chain map, we have almost constant time to access. So after the loading, everything is very very simple. So I just see the if you look the overview of the architecture is not like too much. It's just a simple class. But then we plug the loaders   and the strategies user on the loaders. Then we have the filters   and, especially, we have the validators so you can ensure that the settings look the way you want. So, yeah, that's more or less how it's implemented. And as I said, it growed,   evolved   mostly organic. So from that small project, it started becoming like a framework. And people   from different   backgrounds are contributing and bringing more features from sometimes these features in other settings libraries they bring to DynaConf.  

So we have different ways of doing.  

[00:31:37] Unknown:

Given the fact that it has these interfaces   for custom loaders and different strategies,   it definitely is very extensible.   And I'm wondering how you   thought about the process of deciding   which strategies, which loaders to incorporate into the DynaCon project   itself as it's packaged and released,   and what capabilities   you are   consciously deciding not to bring into the core because either it's a bespoke solution for 1 person or it's something that is not   a sort of core functionality, and you want to kind of push that more into   allowing people to do what they need to do without overloading the project and overloading your kind of maintainer capabilities   into some of the ways that people are able to hook into DynaConft to be able to add those customizations.  



[00:32:27] Unknown:

When I started, I got very excited about adding more features.   It was like my vision. Like, I want Dynaconf to be able to load all the possible   configuration   formats and all the possible   services   targeting for configuration like, etcetera, Redis, Vault,   etcetera.   And then at that point, we added too many features into the library. I know that. So all those loaders that we call the core loaders   for multiple file types   are not really necessary on all Dynacomf installations.   But currently, if you do pip install DynaConf,   you are getting all those together.  

So at some point, there was no extensive thinking about this.   When we released the 3 dot 0, I started having more contributions and we formed it like a board of co maintainers   and people started thinking about like, we don't want to add more things here. We need things to be added as a plugin.   So now we have like a initial plugin system. So for example, there is a pytest plugin to make it easier to use Dynacom for when you are mutating   settings during your settings.   And we decided to not accept this as a core feature and then the person that sent the PR   created that Dynacomp Pytest repository   with a plugin.  

So it was like 1 year ago or 2 years ago, I don't remember exactly, but   we started, like, rejecting   more features to focus on fixing and improving features that we have in the library.   And now we want to improve our plugging system,   and we have plans to have more pluggings in the future and even removing   some features from the core,   turning them into plugins. This is 1 of the things in our road map. Yeah. But right now, we are not accepting, like, new features, like new default core loaders. If you want to load something, for example,   for a new file format or a new external service,   we will   suggest you to create, for example, an plugin.  

There are documentation   on how you can use it   and yeah. Currently, that's our approach   to fix and prove things before we   we add more features. And try to get the library. It's small. It's very small, but we need to get it more small and and have only the core on the on the main repository.  

[00:34:55] Unknown:

And then as far as   promoting kind of reusability and collaboration   among the kind of ecosystem of people building on top of DynaConf, do you have any,   I guess, naming   convention for packages that plug into DynaConf to add custom functionality, such as if I wanted to add something that allowed for loading key value data from console, for instance? Or   is there any kind of GitHub organization that you have for being able to allow people to contribute their plugins to DynaConf   so that people can kind of discover all the options that are out there rather than just reinventing the wheel all the time?  

[00:35:32] Unknown:

Yeah. We have a GitHub organization. It's called Dynacomf. Github.com/dinacomf.   There is the website, Dynacomf.com,   and there, I think the 2 plugins that exists right now is listed somewhere there.   And we have the discussions   forum on GitHub.   So if   people want like to contribute with a new loader or new functionality, we discuss there or on our matrix channel   and we give the help needed to create this plugin. It's not a hard work because you can create a loader by subclassing the base loader and adding those 3 objects,   reader, writer, and loader.  

And also we have now support for hooks. So we got some inspiration from Pytest.   So you can file with hooks that Dynacomf will call on certain points of its loading process.   So it's possible to, like, customize lots of things in in Dynacomf right now. Yeah. But the main channel is GitHub. Github.com/dynacomf.  

[00:36:33] Unknown:

You mentioned a bit about the process for kind of onboarding onto Dynaconf as part of your project, where it's largely just PIP install and start using it. I'm wondering what are some of the ways that the   implementation   and usage of DynaConf scales with the complexity of a project or the complexity of the   environments   or kind of support matrix of a given project as it evolves?  

[00:36:57] Unknown:

The goal for the project is to be really simple. Like, you you really create your Python settings object.   Really,   the idea is that it's not going to be something   that you need to care a lot in your project besides the validators,   yeah, which is completely   the user   responsibility   to write. But everything else is really like   transparent and outside your project. And yeah. So the usage scale has been very good. I think there are lots of users right now. I don't remember right now, but I think in GitHub, we have more than   2, 000   repositories that depends on DynaConf right now.  

And recently,   PIPI market DynaConf has   critical project, which I think it means that it's on the top 1%   most downloaded libraries.   And   in terms of support, like,   we have few contributors   that are really active in the GitHub issues. So if you need something, if you need if you have questions,   it's really quick to get information.   On the matrix channel, we don't have lots of people because the channel is more like for devs,   but it's also a good source of information if you need implementing   DynaConf in your project.   And I would say that GitHub is our our main place in the discussion tab or issues.  

And usually, it's quick. Like, I tend to answer questions really fast and we have other people like helping. So   it's scaling good. As for example, the company I work on is, like,   sponsoring me to work on Dynacom for a certain time on my work.   I have, like, paid time to work on the library. So I'm trying to address lots of issues. So we work with open source, and DynaConf is 1 of the libraries in our open source. Yeah.  

[00:38:49] Unknown:

1 of the other topics that probably deserves its whole entire episode   that we touched on briefly earlier is the question of secrets management,   where   it's 1 of the things that, again, sometimes seems like it's straightforward   but can be   critical if you get it wrong because it, you know, has the potential to   expose your application, make it vulnerable to exploit or kind of malfeasance.   And I'm wondering, what are some of the ways that you have worked in DynaConf to be able to strike a balance between   that frictionless   use case that you're aiming for while providing appropriate guardrails   to prevent kind of credential leakage or prevent people   kind of circumventing   best practice for how to manage sensitive data in their application?  



[00:39:35] Unknown:

I think the best way to manage   secrets today   is using encrypted vault services, especially if you are working in the cloud.   So you're gonna have tooling that will   it will depend on the environment you are running. But for example, you have a w s secret manager, you have HashCorp Vault   and you can use encrypted environment variables depending on the on the place you are   deploying your application.   I think that's the best because you are delegating   responsibility   for   this where the secrets are stored and how it's encrypted to another service that is really good on doing this and well tested for this specifically.   And Dynacom supports   out of the box loading data from HashCorp Vault today.  

So you can like just pass the configuration,   the host, and your token for Vault, and then it you can pass, I don't remember exactly, but I think the name space and then it goes there and reads your secrets.   But you can add folder for other services. It's not hard to do.   But this is something that probably you're gonna do in a production environment when you have like a team or as a re or DevOps engineer that will provide the service for you as a developer.   During your development environment, I think you don't wanna have this work of, like,   deploying or running a vault service during the development. So we decided to make things easier when you're running in development environment. So Dynacom supports you to add a secret dot toml file. So using the same Dynacom structure when you are working, you have a settings dot toml or dot yaml. The format doesn't matter. You can create a file together   at any settings that is called dot secrets and this extension you are using. This is just a settings file,   but the first way it helps is that   it's added to the git ignore if you use dynamic CLI to initialize   your project.  

So you ensure you are not pushing it to your repository.   But it's a separate file   so you can ignore it from your project.   But you also can encrypt this file if you want. So you can use Dynacom CLI   to add new secrets and the secrets will be encrypted   using a private key.   And then your local vault is just a simple file with encrypted data inside it. This is not something that you should use in production system. I know people are using this encrypted secret file. In production, we can't control what users does.   So in production, I recommend using an external service, but this is something we provide for making development and testing easier.  

Another thing that we did is   Dynacomf identifies   when your settings comes from a secret.   And then it will not print it out in logs. It will not print it out in the standard outputs. And when you are using banner conf CLI to inspect your environment,   it will also just show the   part of the the secret, not the whole secret. So there's some kind of obfuscation on on the variable   to make it secure.  

[00:42:46] Unknown:

As users become more advanced or sophisticated   with their use of settings in their application and in their application of Dynaconf,   what are some of the kind of advanced or maybe underutilized   or overlooked capabilities   that are available in the framework?  

[00:43:03] Unknown:

Yeah. So besides the basic things, I think   yeah. As I said, on env vars, you can override,   you can traverse to nested keys.   That's very powerful, especially if you are working in an existing project. I see lots of Django users that is not using it, but it's very powerful   because we adopted the Django way of doing it, which is   you export a variable.   If it's a dictionary   inside your settings, you can use double underscore   to go nesting   1 level   in our data structure, and then you can keep using double underscore   to traverse to all the other nesting levels. And then you can specifically change that variable you want and that together with the validation   that DynaConf provides, you can have,   easy way to override settings. So this is an underutilized   capability   that I want to promote more and see people using this   better. Other thing is that Dynacom supports lazy values. You can actually create your own lazy formatter.  

Right now it has Jinja   and it has   the math   to do like, common math expressions like multiplication,   division, and so on. You can export the variable and mark it as a math or a jinga and put the template or the expression. And then during the load, then a conf will evaluate it. It also works together with validation to be more effective.   And besides the the loaders that we talked, like you can add new loaders to Dynacomf.   Now Dynacomf has a hook system. So you can   write functions that will be hooked into Dynacomf on specific parts of the loading process.   And from the hooks, you return a dictionary,   and that dictionary will have declarations that will contribute to the settings. So you can do like data transformation.  

You can do data validation.   You can do all sorts of custom things with this. And I think   the thing that is more underutilized   in Dynacomf   is validators.   And it's really important because by default, Dynaconf is schema less. You just create a settings   that equals Dynaconf class instance, and then you start looking for   keys inside it. But there is no definition of schema by default. It's very flexible.   But we provide a way to define a schema. So you have a validator class where you can provide   validator for each key you want and you can define the type of the key, the range of the key. I think you asked it before. How do you define a range if you have like a number, a port, for example, for a service?   So how do you define exactly which options are available there? So you can use the validators to define it.  

And another good feature of validator is that it has a text field, which is a doc for that settings.   So when you define,   collection of validators,   you put the rules that Dynacomf will validate and you put the description   of this. You can use Dynacomf CLI to generate a documentation   for the supported settings for your application.   So this is really under utilized it. I think that's my fault because I need to promote it more. We are focusing more on the other basic features, but it's very powerful.  

[00:46:20] Unknown:

And I guess 1   kind of side topic that I'm interested in personally is   another   project that I've seen that has some similar capabilities,   at least in terms of being able to   assign   schema and type information   to your settings and be able to do some validations   at load time is the Pydantic project.   And I'm wondering   if you see that as a complementary   or an alternative   solution for being able to approach settings?  

[00:46:51] Unknown:

Yeah. So the answer for these questions is on Dynacom pull request tab. If you go there, you see that there is a pull request that is going to be the Dynacom 40.   And on Dynacom 40, we are bringing by then take to Dynacomf.   So these validators   I told you now is we created the validation class. It works very well.   We want to keep it working,   but what we're gonna do is we're gonna provide a way for the users to create a schema,   use subclass dynaconf,   and then you create a schema and then use type annotations   and use bydentic   fields.  

And then Dynacomf will follow   bydantic,   style of schema. So we see that Python community   is moving to this thing called modern Python.   All the things are going to this way of programming.   So using more data class, more bydentic with type annotations.   So we are moving this way also. It's taking a time because we are not like lots of contributors, but this is in our roadmap for the next version. Yeah. In your experience  

[00:47:58] Unknown:

of building and using Dynacon for yourself and working with the community of people who are adopting it for their own projects, what are some of the most interesting or innovative or unexpected ways that you've seen it applied?  

[00:48:09] Unknown:

Oh, that's interesting.   I have been hired some months ago by a company to give a consultants   on a project they were using, DynaConf.   And it was a very old software. They had like this data collectors,   like, scan things and this generates a dot CSV file.   And   at the end of the day, the person collecting the data will select an upload field on the application,   select the CSV file or multiple files and upload to a back end. So they were able to   create a new back end using Flask   but they are not able to upgrade   this outdated   data collectors.  

So on the application they have on the client side, they have only a single upload field to send files to the server   and they wanted to gather more information   from the client parts. So they wanted to have a form with the name, the location, and all the information, but it was not possible because I think this is outdated. They can't, like, update that. So   what they did,   they added a toml   file in the mobile device with the information they want, like the name of the user   and all the serial numbers and things. And then the user, when sending the upload, they select the CSV file and select this toml file with the information from the the local place   and upload it to the Flask server. And Flask,   instead of parsing this   manually, they decided to use Dynacom for it, which is not like the go for Dynacomf, but work it because during upload, they drop this file in the configuration file   and then when running their pipeline, I think it was an airflow application,   a pipeline,   they read settings and Dyna conf would   load that toml file and know how to process that CSV file. So just to resume this,   what they did is instead of creating a new form in the application,   they replaced the form   with a toml file. They use a dynaconf to parse it. And dynaconf is not like a library for doing this. They use a dynaconf as,   ORM or something to process data coming from request   and not as a settings library,   but that was working. They didn't want it to change so   they got a problem where the end users   discovered   that they can   edit the toml file and override settings that they should not override.  

And then we ended like creating   a custom format, not toml,   and adding a new loader to Dynacomf with more validation   to prevent users to hack the system.   And it's running with Dynacomf today. So they have lots of collectors. They run multiple   pipelines on airflow,   and they use Dynacomf   like in unexpected   way. So DynaConf is their request   parsing library   and yeah. I don't think there is any other   personal or project using like this. And yeah.  

[00:51:21] Unknown:

Yeah. It's definitely,   interesting, the different ways that the kind of old trope of when you have a hammer, everything looks like a nail, some of the ways that that manifests.  

[00:51:31] Unknown:

Yeah. You can control what the user does. We try. Yeah. I learn with this and other projects I work on. We always try to, like, build fences   to prevent users for of doing this.   But, yeah, when you work in a project like this that grows organically,   when you find   the hole that you need to close,   all the users are already using that. So you can't just, like, remove this feature because everyone relies on the feature. So instead of a bug, you just document and it becomes a feature. Then life goes on, yeah, with lots of testing. So,   yeah, that's interesting.  



[00:52:07] Unknown:

In your own experience   of building DynaConf   and maintaining it and evolving it, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?  

[00:52:17] Unknown:

Okay. So well,   interesting.   I think   when I   started doing the project,   I thought that more documentation   would be better. If I documented   everything I did would be the best thing to do,   and that's not true. I found out that   sometimes   more documentation   is bad.   So the previous documentation of Dynaconf   have too many docs. We use it to   dock, for example, APIs and function calls and using that old style Python way of documenting   everything   on your library.   But as Dynacomf is more like a client,   not really a library.   You stunted as a client. You are the end users. You are not really   extending it a lot in terms of API.  

I learned   that today, and I think it's something that is changing   on software industry. If you look to the newest   software that is created on open source,   documentation   is becoming more like a set of tutorials   written in a book form. If you go to excellent documentations like fast API,   I can mention.   You see that the style of reading docs are more like a tutorial. You are guiding the person   to know workflows   on how to use your application   instead of documenting   every method or every single thing your application can do. You put everything together. So this is was an interesting thing I learned, a different way of writing docs,   and I'm trying to do this to apply this to Dynacomp. There is a lot of things we need to change there in this way. And this is more difficult because you can't just look to the code and document based on code.  

You need to think about use cases   and document   each use case   separately, but it's more powerful in terms of documentation.   An unexpected thing   that I figured out is that Django settings   is really difficult to customize.   While in Flask, you can easily subclass and override the config class because it's like designed for that.   On Django, the only way to customize the settings interface is by patching.   Yeah. So I discussed it with some Django core developers ways of making the Django settings   because Django has its settings manager when you do from Django dot conf import settings,   it does similar things as Dynacomf, like lazy loading and things like this.  

You can add plugins to it. You can customize it in a good   way. So what we did, we patch it. All the other libraries   for Django are patching the settings,   and we are constantly trying to get Django developers more involved   in a way to make it easier.   And the other thing was challenging, I think, is merging data from multiple sources. It's the biggest challenge right now. The source of our,   most of our bugs and requests and and confusion   is how do you take multiple data and merge conflicting data   and how to make the decision of the precedence,   which 1 wins and how the final data will look. So, yeah.  

We are still learning and trying to improve better ways of giving flexibility for people that wants to merge data from   different sources.  

[00:55:37] Unknown:

And so for   an individual or a team who's looking for   a way to manage settings for their application or their project? What are the cases where DynaConf is the wrong choice and maybe they're better suited with a different framework or building their own custom  

[00:55:55] Unknown:

solution? Well, I like to think that DynaConf is a good fit for any project that needs the couple settings from code.   Yeah? So if your project doesn't need to decouple the setting from your code, and you are good with just a constant dot py file and everything static there, I think then maybe DynaConf is not needed because you were adding the loading overhead and all the things, and you actually doesn't have the need. But I like to think that this is simple, small, and you can use it in any project.   From a simple project having only a single settings dot toml or dot yml file from   a project using multiple environments, thousands of files and external   services.  

However,   if your project is a library   or a plugin   to be added to another project,   I don't recommend you using Dynacomf. And I saw people trying to do this and coming to the issues saying that   there were some kind of bug or problem.   And actually, I think libraries   or plugins doesn't have to define settings.   Libraries and plugins   will   consume the settings from the application   that are consuming it. So I think   that   that will be the good choice. If you are doing a library or a plugin,   you   add interfaces that you can receive the settings from the application that you are plugging to and data conf will definitely not be a good solution. You're gonna be, like, frustrated because data conf depends on environment and you don't control environment when you are releasing a library or a plug  

[00:57:32] Unknown:

in. And as you continue to build and maintain and evolve the Dynacont project, you mentioned a little bit of the work that is kind of in process, but I'm wondering what do you have planned for the near to medium term or any areas of feedback or contribution that you're looking for help with?  

[00:57:49] Unknown:

Yeah. So first, I would like to thank the contributors and co maintainers of the project. We gather, like,   amount of people that helps me, like, taking decisions and giving the project to the right direction.   And we recently defined the road map for the version 4 0, and there are   3 big changes. We have small fixes,   improvements, but the 3 biggest changes is   we are implementing   that new way to validate settings. This is based on Pydantic and Python type annotations.   So we want to keep all the current way of doing things working without breaking any compatibility,   but we are adding this new way for you to define schema in the Pydantic style that will work good.  

The second thing   is we are splitting the code base and removing the core features and transforming in plugins.   So the Dynacomf package, when you install Dynacomf,   will keep being a bundle   with all the   parts installed together.   So it's not going to break any installation.   But then we are creating   external projects on the organization   that will be Dynacom Core,   Dynacom Vault, Dynacom Redis, Dynacom VM. So every single loader, every single functionality will be split into,   separate repository.   Not everything, but all the things that make sense to be separated.   So those can evolve like plugins, and when deciding which feature you want, you can pull only the feature you need to build your settings library. So this is like, technical depth we had on the first implementation,   and we are solving in the version 40.  

And I   think the other 1, which is the hardest 1, and I'm working a lot on it,   is the data merging algorithm. I said to you that data merging is the most difficult thing to do in a settings library   and data conf today has a part written in Rust. If you go to crates. Io, you have a project there called   HydroConf.   HydroConf is   the same philosophy, the same strategies of DynaConf   written in Rust for Rust web applications.   And when doing this in Rust, we found that the merging algorithm   is more efficient   in Rust.   So   what we're gonna do is we're gonna write this   as a separate crate on the Rust   and export that crate to a Python model.  

And then we use this in Python because it's something that today is really easy to do to create a Python plugin, Python model in Rust. And then we're gonna fix the problem of data merging to be really more efficient   using Rust. So those are the the 3 things that are on the road map for the version 4 0.   And, yeah, hopefully, it comes this year,   I hope.  

[01:00:41] Unknown:

Well, for anybody who wants to get in touch with you and follow along, I'll have you add your preferred contact information to the show notes. And so with that, I'll move us into the picks. This week, my pick is on the theme of secrets management for applications. So I've been using the SOPS project for a while for some of my work. It's a great utility for being able to encrypt data in flat files and do it in a manner that is   relatively secure because it's able to hook into things like either HashiCorp Vault or a different cloud key management solution so that you can use their secret key material to be able to manage the encryption. So it's a bit safer to be able to actually store it in your source repositories and version your secrets along with your code. So definitely recommend taking a look at that for anybody who's interested. It's a project out of the Mozilla group. And so with that, I'll pass it to you, Bruno. Do you have any picks this week?  

[01:01:35] Unknown:

Yeah. So first, a note on on your pick. I think it's interesting. I heard about it, never use it. I hope it becomes like a Dynacomps plugging in the future to use the SOPS, the encryption   stuff. We are using currently a thing called Jose, Jose. I don't know how to mention it, but it also does some kind of encryption.   And, yeah, with our plug in system, probably, we're gonna have subs integrated with Dynacomp and yeah. So now going to my picks.   Well, I don't have a pick in terms of software and as you said it can be anything.   So I recently   watched   the series, very interested, called Severance,   and I think it's really great. I just finished it and it's on my mind. I can't start thinking about what's happening there. So I recommend   for people who like a little bit of sci fi, but just a little bit and also computers. This is really interesting.  

Yeah. So my other pick is gonna be learn Rust. I think the future of Python is full of Rust. Yeah. We're gonna see more and more Rust in the future for doing better Python projects. So   it's gonna be a skill for every   Python person to learn a little bit of Rust. Yeah.  

[01:02:46] Unknown:

Alright. Well, thank you very much for taking the time today to join me and share the work that you've been doing on the DynaConference project. It's definitely a very interesting and full featured framework, and settings management is a important capability for any application. So I appreciate all of the time and energy that you and the other maintainers and contributors of that project have put into it. So thank you again for your time, and I hope you enjoy the rest of your day. Yeah. Thank you. Thanks for the invitation. It was very nice to participate and, yeah, have a nice rest of the day.   Thank you for listening. Don't forget to check out our other shows, the Data Engineering Podcast, which covers the latest on modern data management, and the Machine Learning Podcast, which helps you go from idea to production with machine learning. Visit the site at pythonpodcast.com   to subscribe to the show, sign up for the mailing list, and read the show notes. And if you learned something or tried out a project from the show, then tell us about it. Email host at pythonpodcast.com   with your story.  

And to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.