Summary
Any application that communicates with other systems or services will at some point require a credential or sensitive piece of information to operate properly. The question then becomes how best to securely store, transmit, and use that information. The world of software secrets management is vast and complicated, so in this episode Brian Kelly, engineering manager at Conjur, aims to help you make sense of it. He explains the main factors for protecting sensitive information in your software development and deployment, ways that information might be leaked, and how to get the whole team on the same page.
Preface
- Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
- When you’re ready to launch your next app you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to scale up. Go to podcastinit.com/linode to get a $20 credit and launch a new server in under a minute.
- Visit the site to subscribe to the show, sign up for the newsletter, and read the show notes. And if you have any questions, comments, or suggestions I would love to hear them. You can reach me on Twitter at @Podcast__init__ or email hosts@podcastinit.com)
- To help other people find the show please leave a review on iTunes, or Google Play Music, tell your friends and co-workers, and share it on social media.
- Join the community in the new Zulip chat workspace at podcastinit.com/chat
- Your host as usual is Tobias Macey and today I’m interviewing Brian Kelly about how to store, deploy, and use sensitive information in your applications
Interview
- Introductions
- How did you get introduced to Python?
- To begin with, how do you define a secret in the context of an application?
- What are the broad categories for solutions to secrets management?
- What are the different aspects of secrets management in the lifecycle of developing, deploying, and maintaining an application?
- How does the scale of a project or organization impact the strategies that are reasonable for secrets management?
- What are some of the most challenging aspects of secrets management at the different stages of usage?
- What are some of the common reasons that secrets management strategies fail?
- What are some of the vulnerabilities or attack vectors that development teams should be thinking about when working with credentials?
- What are your thoughts on versioning of secrets?
- Beyond storing and deploying sensitive information, what are some of the secondary concerns around secrets management that development teams should be thinking about?
- How does the use of multiple environments (e.g. dev, QA, production, etc.) affect the strategies used for secrets management?
- What are some of the most useful resources that you have found for anyone looking to learn more about this subject?
Keep In Touch
Picks
- Tobias
- Brian
Links
- Conjur
- CyberArk
- Datawire
- Transpiler
- IDL
- CSRF (Cross-Site Request Forgery)
- Hashicorp Vault
- Continuous Integration
- Continuous Delivery
- TLS (Transport Layer Security)
- RBAC (Role Based Access Control)
- Terraform
- SQL Injection
- Secretless
- MFA
- Duo Security
- Kubernetes
- Summon
- OWASP Top 10
- Configuration Management
- Puppet
- Chef
- Ansible
- SaltStack
- Immutable Infrastructure
- Conjur Blog
- Krebs On Security
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
[00:00:10]
Unknown:
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. When you're ready to launch your next app, you'll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 200 gigabit network, all controlled by a brand new API, you've got everything you need to scale. Go to podcastinit.com/linode to get a $20 credit and launch a new server in under a minute, and visit the site at podcastinit.com
[00:00:35] Unknown:
to subscribe to the show, Sign up for the newsletter and read the show notes. Your host as usual is Tobias Macy. And today, I'm interviewing Brian Kelly about how to store, deploy, and use sensitive information in your applications. So, Brian, could you start by introducing yourself? Sure. I'm the head of Conjur Engineering for CyberArk.
[00:00:52] Unknown:
Conjur is a, a vault or secrets management application, and CyberArk is a well established large, cybersecurity vendor. And, yeah, I've been in the software industry about 20 years, mostly managing engineers, and at times, sometimes they let me code as well.
[00:01:11] Unknown:
And do you remember how you first found Python?
[00:01:13] Unknown:
Yeah. I was a latecomer. I was working at a company called Datawire who does a lot of infrastructure for microservices. And, we were writing a transpiler, in fact, for, from a behavioral contract language into other target languages. So it's kinda like an ideal on steroids where you'd say, this microservice wants to talk to this other microservice, but it's not just parameters and, you know, paths and URLs. It's also retry logic or circuit breaking or other kind of behavioral semantics on top of the actual contractual parameters. So that was fun. That was my first major foray into Python. And sadly, now I'm, more in the Ruby world, but, I I really did love my time in Python for the those years. Yeah. Whatever gets the job done. Exactly.
[00:01:55] Unknown:
And so to begin with, I'd like to get your definition for what a secret is in the context of an application and, development environment.
[00:02:06] Unknown:
Sure. Basically, a secret is anything an attacker might want to broaden or further their attack surface, and they might use that secret to either get the value they want or to get closer to the value they want. So if it's a database password and it's the, password to the database that holds all of the bank account details or credit card numbers, they're pretty much done. Or it might be a very, very small innocuous secret that allows them access to impersonate someone else or do some Bitcoin mining on a small AWS account, you know, any of those things that where they can either move laterally and then move upwards in value or move upwards in in different, directions and, just keep moving. So that is a broad set. It could be a very small thing like a a, a password to something that you don't think is terribly sensitive, but stepping stone approaches can be used to make it more valuable later on. And as you mentioned, it might not necessarily be something that's directly exploitable, but it might be enough to
[00:03:09] Unknown:
lend credence to something like a phishing attack where they're trying to get somebody within the organization to perform some action that will then increase their foothold within the infrastructure or the organizational capabilities, or it might be something that they're able to use to decrypt or derive the values from some other set of information that they've already obtained. So maybe a salt for the hashing algorithm that you're using for your passwords or the Django secret keys so that they can then predictably generate a CSRF token or something
[00:03:42] Unknown:
like that. Exactly. And and would that everybody was using salts, would not be a great world. But, yes, all of those things are definitely, you know, maybe on their own, aren't terribly valuable. But if if used creatively or in conjunction with something else, suddenly make an attack, you know, really lucrative for that attacker.
[00:04:02] Unknown:
And so given the broad range of information that could potentially be classified as a secret within the context of an application or an organization, what are some of the broad categories of secrets management solutions that are available for people who are looking to go beyond storing everything in plain text?
[00:04:25] Unknown:
Yeah. So there's there's a variety. The 1 that people are probably most familiar with is a vault, and that's a software appliance that securely stores your secrets. And, those are you know, there there's plenty of of them out there with CyberArk, has has a a a very large enterprise vault, and also the 1 I work on Conjur is more focused on, DevOps flows and CICD systems. And, what that vault will do is just, you know, put it in a place that it's it's extremely hard to get at. The problems really occur when you deliver those. If you gotta open the vault, you know, banks are only useful if they open the vault and start doing something with their capital. So same with secret solutions. You store these secrets, but they've gotta go somewhere. So now you've gotta talk about delivery. And that's typically based on things like TLS as you'd expect, but it encompasses the authentication problem, which is these are not gonna be humans asking for these secrets. They're gonna be applications. So how do you how do you authenticate an application? You know, it's machine identity, not human identity. Even within the vault, though, there are things that some people are not aware of. It's a fairly mature space. So you have things like rotators, which on a per database or per service level, know how to contact that database or whatever it is and say, hey, you know, it's been 30 days. We need to change the password. They also have sophistication in terms of preventing the, interruption or intermittency of service when you rotate a password because you've gotta distribute the new 1. You've got a bit of a race condition. You know, how do you get the new 1 out there? See, there are strategies like dual accounts where you have 2 accounts always active and you're kinda flipping between them so there's a bit of overlap time. And then there's the whole aspect of authorization.
So once you've identified these applications that are coming to the vault, knocking on the door, and saying, I need the password for the customer database, please, then identifying them is only the first part that can see the authentication hurdle surpassed. But then you've gotta authorize them. Somebody, usually a human, has to say, is it appropriate for this microservice running over here to to get that database password? Some security staff member or officer of some sort has to know that that application should do that and should get it. And, also, at some point, maybe deauthorize it and and unenroll that application from the rights to that database. You know, maybe it's just grown out of that or maybe that database is getting decommissioned. But there's a whole life cycle around authorization, and, there's various solutions for authorization. We have an an RBAC, you know, roles based access control approach that there are other approaches used by other vendors. But we find the role based model good because it maps to human expectations around that, and you wanna really optimize for a human interaction with this system. So you say, okay. This is a microservice.
It is in this category or it's part of this layer of applications, you know, and and they all it's embedded within an auto scaling group. And whenever any of these come up, they need to have the same privileges. So there's a whole ecosystem of of products and and componentry in this space beyond just the secure vaulting of secrets.
[00:07:36] Unknown:
And beyond just storing and distributing the secrets, there are also a number of other aspects of secrets management throughout the life cycle of obtaining or generating the secrets and then deploying them to the application. So what are some of the other concerns that developers and operations engineers should be thinking about in the context of creating and storing and securing those secrets.
[00:08:01] Unknown:
Sure. It's unfortunate, but there are so many ways for your secrets to get leaked if you're not careful. And, the analogy I like to give is, you know, imagine this this big bank vault or or sort of Fort Knox kind of system where that's your, you know, the master storage system of everything. And then you you get your big delivery truck, you know, the big armored truck with with armed guards. You you take a couple of secrets and you're you're bringing them down to your application. And you hand them to your application and it's just you just leave them on the sidewalk and walk away. That's that's a good analogy for how a lot of secrets management is done in reality now. Very secure storage, secure transit, and then once they reach the application, an absolutely horrible situation, just a debacle of of, security problems.
For example, logging your, your secret, you know, logging your, whatever it is you received and just printing it out. It, you, it happens more often than not. In fact, if you run a Terraform, if you run Terraform, you probably know about, you know, how things end up in TF state files and whatnot. That will include your secrets. So you gotta be aware of that. There's a variety of other things that can happen. You probably wanna 0 that out in memory if you have an environment that allows you to, overwrite your memory section that had that. And a lot of interpret languages don't offer that, but that's good practice because leaving it in memory, beyond when you've set up your database connection or whatever is a good thing to do. But a lot of developers would consider that maybe too much. If it was injected into your process environment, you know, that might be difficult as well. So there's a whole set of problems that occur when you deliver a secret to an application.
And, really, what it comes down to is, you know, does the application and and this might be something the developers have written or it might be Jenkins or it might be Kubernetes or whatever. It's could be anything that's got anything to do with your application building, you know, delivery or or deployment. Really auditing and understanding how secrets get used outside of the vault is 1 of the biggest parts that people just forget. And they're all concerned about vaulting, and that's good. You need a vault. But beyond the vault, things are less sophisticated. And sadly, the hackers know that and they know, you know what? Do I wanna go after the vault that has had 10 years of development behind it and all kinds of pen tests? Or do we just wanna go and do a quick SQL injection attack on this this page? Because they probably just didn't even do the basics on, you know, on on database query management. Oh, look. There we go. We're in. And, that's kind of sad, but it's it is, it is the reality.
[00:10:32] Unknown:
And also, as you mentioned, auditability of the secrets also extends to creating the secret and whoever is actually obtaining the secret for distribution. So it's definitely important to keep a close eye on the entire path that a secret has taken from when it was first obtained through to actual delivery of the application so that if there is any compromise, you can identify at what point that may have occurred so that you can then try and prevent that in future situations.
[00:11:02] Unknown:
Yeah. Very much so. And auditing can be 1 of the noisiest parts. If you're if you're working a large enterprise, you'll have 100 of thousands of of pieces and and accesses, you know, every maybe every minute. And things are coming up and down. And when they come up, they've gotta request secrets. All those accesses, whether those are authorized or or not, and whether they if they were successful or not, they all have to be audited. That can create noise. And, and you need good auditing tools, I guess, and, and analysis tools to, to really figure out, you know, is there an attack going on right now? And then there's companies including ours that have solutions for that that say, you know, let's use some heuristics or machine learning or any kind of advanced technology to watch what's going on in the system and say, this, this is unexpected behavior over here. You know, we haven't seen this pattern in the last 6 months and all of a sudden there's a burst of activity. And so the point here is you you will need more than just humans to to deal with that level of auditing. And going into the actual use,
[00:12:03] Unknown:
by the application of those secrets, you mentioned the possibility of leaving the credentials around in memory after a connection has been established or or the fact that it may be present as an environment variable, which I know is 1 of the least secure ways to do it because if somebody manages to get a foothold on the system, they can fairly easily dump the environment variables of a running process to obtain those secrets. So what are some of the best practices for actually utilizing secrets once they're on a host to prevent them from then leaking to other processes or making them too accessible to somebody who manages to get some foothold on an instance? Sure. So there's a variety. And,
[00:12:44] Unknown:
the first thing I'll say is you can't leak what you don't have. And, therefore, 1 of the best approaches to preventing a secret leaking out of a process is for the process to never get it in the first place. So, we have a technology that does that and it's, you know, it's it's language agnostic. It's called secret list, and you can download it from secret list dot I o. And that's a fairly new novel approach to this. The secret is delivered, but it's delivered to a a process running side by side with the actual application. And, you know, I won't go into all the details on the podcast, but certainly there's there's a variety of architectural benefits to that. But let's just say you're doing the classic sort of, you know, application needs to get a secret pattern. The first thing to to do is make sure that the secret can only be used by that application so that if somebody steals it, the usage of that secret elsewhere should fail. Now how do you do that? Well, you can have a multi factor authentication model that, you know, we're all familiar with from our phones and and, Duo or whatnot, but that same model can be applied to applications.
So for example, if you have an application running in a, scheduled environment like Kubernetes, you can have certain factors associated with that application. For example, Kubernetes has an overlay network, and it could dole out IP addresses to applications. And an IP address or having an IP address assigned is not good enough for authentication by itself, but it can be a factor in the authentication handshake. So that's 1 approach, you know, and I know there there's other factors you could use like, hashing of the application code or, you know, a variety like that. And then, you know, the once that's done, the authentication is required to use that secret. So then if, somebody comes and and tries to do something else elsewhere, it it might not might not succeed. But generally, that's that's a high level of sophistication.
And usually, if an attacker is able to get a secret, they are going to be able to use it. So if there is any chance at all of 0ing out zeroing out memory, it should be done. If you can't do that, then narrowing the amount of time that a secret can be valid for is good. So rotating it so that, you know, if they get it, they just they can't just lurk for months using, you know, your secrets and and doing things. We have some customers that rotate very aggressively, like every 30 minutes. And, you know, we we have plenty of people looking into even faster ones than that. And, you know, it's strategies like that that that bound the the time or the scope of of validity of that secret. Those are generally the most successful. But we feel that, you know, this whole thing is kind of a fool's errand in a way. Yeah. Because application developers have proven themselves to be understandably more focused on feature development.
And then on security, security is hard and it's hard to sort of prevent holes or prevent flaws. So our, you know, thinking now is, well, let's just take it off their their plate. Let's just say it's okay. We'll do it for you. The secret can can, be delivered somewhere else, and you still get all the benefits of that. And you just don't have to worry about protecting it because honestly, you know, this is why we're having this discussion. It is really difficult to manage. And so in terms of
[00:15:58] Unknown:
somebody who is maybe a single developer working on a small application, what are some of sort of the hierarchy of best to worst ways of storing that secret for use by the application,
[00:16:26] Unknown:
So So I would say, starting at the whole, like, you know, post it note level, just running it down and and putting it somewhere statically. That's actually not as bad as, you know, putting it somewhere like GitHub in a public repo where all of a sudden I mean, I I just hope your listeners are aware there are bots watching every public repo check-in and scanning for predictable patterns like things that look like AWS keys. And if you put something out there like that, you will have a a a Bitcoin mining operation on your AWS account within minutes. And also the key will be changed, you know, so that you can't even go and revoke it easily. So that is like the the level 0. That's that's the place you wanna never end up in. Next up is is a good place to store the secret. So vaults, there's plenty of them. There there are open source options. There's no excuse for not having a vault, and it is absolutely a required part of this. So have a vault. It's a thing that I notice when I go to, you know, dev ops days or security conferences or or even just plain developer conferences. There's a lot of focus on the tooling and the infrastructure, but less on the developer experience. And the thing that makes or breaks a strategy for secrets management is how easy it is for a developer to deal with it. So if you require if you set up some know, Rube Goldberg machine that requires 17 steps and all kinds of machinations to make it work, developers are not going to adopt it. They're gonna sidestep it. They'll they'll pull out the password, and they'll just, you know, configure it statically somewhere or something really bad because they're busy and they have stuff to do and dealing with an an overcomplicated process is not gonna be good. So focusing on the developer experience from the start is good, and it can be a great developer experience with not so good security to start. And just as long as you maintain that developer experience as smooth and sort of invisible and as as provisioned as possible, then you can keep improving security as you go. And the the things that you would do to to your question about the hierarchy environment variables are not as bad as you might think. We, we offer a tool called summon, which injects them for you, and it also takes care of pulling them from your vault of choice.
And it will, you know, it'll do a good job of, managing all the security around that. But, yes, it does go into the environment. Beyond environment variables, I mean, then, you know, if you have an application language that allows for memory zeroing to occur, then, you know, like C or C plus plus then nuking that out. But if you don't, I mean, secrets have to get to your application. They have to be used. And at some point, they'll probably get rotated, and you'll have to go back and get the new version. If you can't rotate it and they have to be put into memory somewhere, then that is gonna be something you're just gonna have to live with. And then your next level of security is hardening your application so that it doesn't have, you know, basic OWASP vulnerabilities.
And then the next level above that is hardening where it runs. You know, if you have you got good I'm setup in Amazon or wherever your environment is. Then there's there's lots of pieces in between. Like, if you're running in Kubernetes, you could leverage Kubernetes secrets, which is fairly immature and and has come a long way. But it's it's it's not a really good security solution, but it's great for developers and things get injected or dropped into apps nicely by Kubernetes. So there's a whole journey you can take, but I would I would sort of emphasize the the human aspect that if you make something easy and invisible, it'll get used. If you make something difficult and super in your face for developers, they're gonna just get annoyed by it and stop using it or go around it. And in terms of the secrets themselves,
[00:19:58] Unknown:
there are things like database passwords that can fairly easily be rotated with some measure of regularity. And then there are more long lived static attributes, such as an API key for a service that you might need to interact with that you might rotate less frequently. But you may want to have some aspect of the versioning or history of some of those secrets. So I'm curious what your thoughts are on that area of secrets management, whether you should be able to keep multiple versions of a token, for instance, for historical purposes, or if you think that that is unnecessary and you should just focus more on continually rolling forward with those values.
[00:20:40] Unknown:
Yeah. So it's it's a really important point because it is something we support and most most valve vendors will support it. It's generally good practice because it, it makes things easy for developers and, and operators like it because it makes it less likely for things to be going down because, oh, we rotated our 1 our 1 true database password, and now everything is, unable to connect, and they don't have good logic to say, oh my god. I just got 401 or 403. I need to go back to the to the vault and get the new the new password. So versioning is definitely good. The only thing you gotta watch for is that you don't version so much and have so much flexibility that it becomes an attack vector in itself. So, you know, having a good small finite set of versions be valid is, is good practice. If you can avoid it, you should avoid it. You know, having multiple ways in, having multiple values for something might be a hacker's dream, but certainly for, you know, building that balance between security and developer flexibility or, or, or production operations flexibility is super important. I would just make sure it's an explicit decision on the team, not something that just grows on a Friday evening. Someone's like, oh my God, they did this deployment and, here, just turn on the versioning feature. Okay. We're good. Let's let's just leave that on now. That would then not be a good way to have that turned on. But as an architectural decision where people say, you know, we've examined this, and we think we need 5 versions of our s 3 keys in the application because of how many things are using it, how long it takes to rotate, and then the general notion and and awareness of how our applications work, we feel that is appropriate. Well, that's a good discussion. That's a good approach and 1 that comes with that sort of economic understanding of the balance between security and flexibility.
[00:22:33] Unknown:
And beyond the security concerns of secrets management, there are also some functional and operational concerns such as ensuring that the values you're using in an application are still current and still valid and that you don't accidentally revoke any credentials that are still in use by applications. So you have to have some aspect of inventory management of which secrets are in use and when. So I'm curious what your thoughts are on on some of the best strategies for ensuring that you are able to keep those secrets current and that you don't accidentally either revoke all of your secrets or let them expire or have them be too long lived?
[00:23:15] Unknown:
It's it's definitely a thing that requires diligence and human brainpower and time to do. And, and then we consider it almost like what we do with our calendar meetings, you know, in our job, which is nobody's allowed to set up, at least in our team, a never ending meeting series, you know, to, to pull people into a room every week. You can set it up for a certain amount of time. And then at the end of that time when it when it's, when it's over, if if you wanna renew that that meeting, you've gotta justify it again. And say, yeah. Let's do another 6 months of whatever meeting we were doing. Same approach for secrets. Things just just being let live forever, they become attack factors. And and, again, that could be almost always the innocuous ones that that people are are not thinking of. So bigger enterprises will have, you know, reviews scheduled, every whatever months, and they'll get together and and go through things. And if you have a good auditing system, you can just see, you know, what's the activity on it. If you have a good authorization system, you'll know who or what is accessing accessed in 3 months. Is that because the application that pulled it is so amazing? It has had no downtime in, you know, more than 3 months and it hasn't had to repull it. That may be the case, but it may also be the case that nobody needs it anymore and you can just get rid of it. But it is definitely a investment that humans have to make. And, it's, it's just 1 of those things. You, you just like you would go around your building and make sure that, you know, windows stay closed or or doors have have good locks on them. You know, if someone makes a little side door just for some temporary access while you're doing some construction, well, someone's gotta remember to go seal that up, and, same thing for secrets. And in terms
[00:25:04] Unknown:
of the delivery of those secrets, a lot of times, it'll be managed by some sort of automated process, whether it's configuration management for building and deploying servers and software, or it might be an agent on the application side that knows how to authenticate to the vault and pull the secret. So I'm wondering what your considerations are as far as the best practices for some of those different environments, particularly on the configuration management side. Yeah. So the the thing you wanna watch for in configuration management tools is the keys to the castle problem. And,
[00:25:39] Unknown:
a lot of these tools, you know, the big the big ones like Puppet Chef, Solid Stack, etcetera, they, they were architected initially with configuration management in mind. No surprise. Not as much, security. So some of them have challenges in that when you when you start to distribute secrets using those tools, that all of the secrets for the entire system get pushed down to, an application even though the application might only need 1 out of the 200 secrets that are sent down to it. So that's a significantly over privileging of that application. And least privilege, if possible, should be used so that the application only gets the secret that it should get. We've done integrations with Puppet, etcetera, that that allow for that to happen. But, you know, sometimes it does feel like we're we're fighting the tide a little bit because we're trying to implement a model that those tools don't innately support out of the box. It is possible and, you know, certainly can be done, but it can be it is extra work. What I would suggest people do is that they they really understand how secrets get distributed.
Because let's say you take an integration for some vault and and integrate with a Centimeters system. Does that mean that all the secrets go to the the master sort of we'll call it the master distributor or or the master configuration piece, whatever server is used to drive all the agents that are running, you know, elsewhere. Then if that thing, which is not a vault, gets all the secrets and has them stored in memory, well, then now you've just completely destroyed all the value of having a vault in the first place. So how are secrets getting there? How are they transiting through? Are they getting removed? And then how are you dealing with rotations?
So I'll be honest. This is 1 of the the big areas for vulnerabilities that secrets, you know, are generally vaulted well. Most people know, you know, oh, yeah. I should put my secrets in the vault. And then it's like, oh, well we have to distribute the more we have to, you know, get our infrastructure set up. Alright. Plug in the Centimeters system and and it just leaks like crazy or it puts them in places that are highly vulnerable and not hardened, appropriately.
[00:27:43] Unknown:
And then there are also the concerns of managing secrets across different environments all the way from a developer's local machine through QA environments to production. So what are some of the strategies that can be used for managing secrets across those environments to ensure that you mitigate the potential for any sort of security breach or,
[00:28:08] Unknown:
introducing vulnerabilities at any of those different stages? Yeah. It's a great question. Going back to that developer experience part, this would also incorporate into operator experience and making sure that it's seamless. So what you want to have is the experience for a developer who's building a small application or a microservice. You want that model, whatever is used for distributing a secret or managing it to be identical between development, staging, QA, production. And you also don't wanna make your problem more difficult by having 1 vault to try and serve all of those environments. Typically, what's done is you would have a vault per environment, and you would use a declarative immutable infrastructure approach, which is all very DevOps y and very, you know, good to say that the the the mode of access and the method of access is identical, but the value of secret that you return, each time is different. So when you're running a development, the development version of the app pulls the development secret for the development Oracle database. And there's no chance of that developer accidentally posting the the production Oracle database secret, on on their Twitter accounts or whatever. Now when that application flows to the other environments, and that's through its CI system, they the appropriate secrets are pulled for that environment. So that is a bit more infrastructure than people you're expecting. A lot of people think I'll just have 1 vault and just protect everything in there. But it might be easier to have, a a vault per environment. And certainly from a security perspective, that would be the appropriate way to do it because then you've got complete isolation. And if somebody does breach Evolts or does get, you know, elevated access in there, they're only getting the passwords for that environment. And if it's not production, then you're you're you're in a much better shape. So focusing on the kind of things that make an application successful from a, just from a non security perspective, like the developers already do this stuff. They say, oh, well, I'm gonna look for my environment here regardless of whether my application is running in dev, QA, or production. Well, guess what? Go look for your secret in the same place, regardless of whether it's running in dev, QA, or production. That same strategy works wonders here, and and it makes this kind of stuff just so much easier and more predictable.
[00:30:23] Unknown:
And 1 of the ways that you can enable that kind of interface too is with something such as service discovery layer where you can use a consistent DNS name across environments that will actually resolve to a different instance or a different service within those different environments. But from the developer and operator perspective, it's homogeneous regardless of where you happen to be running. Absolutely. That is exactly right. Yep. And what have you found to be some of the most useful resources for learning more about the different aspects of secrets management and application security for anybody who's looking to go deeper down this rabbit hole? Sure.
[00:31:04] Unknown:
The, the vendors in the space actually have some really good blogs, and, you know, share on their social accounts. So, you know, our blog dot conja.org, we write about this all the time. Certainly, other vendors do too. I would also encourage people to follow people like crabs, you know, crabs on security just to keep aware of the whole space, not just secrets management and any of those. There's plenty of sites that have like breach of the day, which is kind of such a sad statement about our industry that, that they now have to do multiple posts a day to to report, you know, all these different breaches and hacks of of our data. And, looking at the patterns and look at the the astonishing simplicity by which some of these attackers were able to get massive kind of value from their hacks is both depressing, but it's also important to know. I will say this is a bit of an un underserved space. And and, you know, beyond the vendors like ourselves and a couple of experts that it is seen as a, an adjunct or side or secondary topic. And it doesn't get the same kind love from the bloggers sphere. The bloggers sphere as as a immutable infrastructure or or DevOps culture or all the stuff that people just write constantly about. It's just this stuff is really important and it could use with some more writers. So generally, I I looked at Twitter. I've a I I follow a bunch of people on Twitter who are working in this space and and doing some really interesting stuff there.
[00:32:23] Unknown:
Yeah. And are there any other aspects of secrets management, either technically or culturally or organizationally, that we didn't discuss yet, which you think we should cover before we close out the show?
[00:32:36] Unknown:
Sure. There's there's 1 last thing to say, which is, you know, building a solution, a coherent solution for secrets management. This feels like a tools problem, but it's really a UX issue. You've got to solve for multiple persona. So like developers, operators, security staff, regulators, you've got to solve for different environments and what those persona expect to be appropriate in those environments. And as we talked about earlier, it's good to have 1 approach or 1 pattern for those. You've also got to solve for application identity instead of human identity. And the last thing I'll say is, you know, getting your your value pipeline, your delivery pipeline, your CICD system, providing secrets and getting secrets sort of in there very early in the process is 1 of the best things you can do. And it allows you to focus on the UX, the security UX, if you wanna call it that, for those different personas as you progress. Because if you end up with a a big master plan, a 2 year plan to to create a a secure pipeline, it probably won't ever get deployed. It's just gonna get deemphasized, and it's just gonna be more and more ceremonial and then more and more, risky to deploy because it's just this big thing that we're waiting to to pull the trigger on. Instead, make small changes, do them early, and go incremental.
It feels like a thing that you have to solve all the problems first, but really, it can be addressed, you know, piece by piece. And as long as it has that guiding directive of making developers' lives easier, making it predictable for operators, applying least privilege, all those things. You can do that, not be the world's greatest security, you know, solution or not have the world's best, secure pipeline, but you can certainly be on the path to being like that. And on the topic of incremental improvement,
[00:34:20] Unknown:
1 of the things that we didn't discuss much yet is the idea of some of these secrets management tools that are focused on creating gpg blobs in source control for distribution as sort of a first step towards having a more sophisticated full featured setup using a software or a hardware vault. So I'm wondering what your thoughts are on that stage of an organizational or project life cycle, or if there are better steps that can be taken with a similar level of complexity.
[00:34:53] Unknown:
I'm actually not familiar with the power. You're saying they they embed a GPG
[00:34:57] Unknown:
signature? So there's, a few tools that will let you add the public GPG key of different users in your development team so that when you encrypt a secret, it can be used, by those different people for deployment or for local development. And then the encrypted GPG, ASCII armored text goes into the source control system.
[00:35:18] Unknown:
Generally, I am not a fan of approaches like that just because, you know, the the the very last few words you said there were like, and then the secret goes into the source code control system. Even encrypted in there, it's like, it's just, you know, a little too hairy for me because now you've you've exposed the secret, not in plain text, of course, but a an attacker knows where to look and how to get it. And then their their next step will be, okay, how do we either socially engineer or otherwise wrangle our way to getting the key to decrypt it. And, you know, it's it's preserved, you know, history wise. It's it's, it becomes em emblazoned as this approach, and and that'll be hard to to pull out later on. I don't know how it deals with rotation and things like that. So I will say there's some sniff. It it there's some sniff tests that it's failing. However, as a, as an early stage approach, it might be okay. I would say 0 of the enterprises that we work with, and then we are skewed to dealing with, you know, a lot of the Fortune 500.
[00:36:19] Unknown:
0 of those would use that approach and would would approve it. So I don't know if that's a how helpful a data point that is, but it's, it it it's something to be considered. So for anybody who wants to follow the work you're up to and keep up to date, I'll have you add your preferred contact information to the show notes. And with that, I'll move us into the picks. And this week, I'm going to choose the series called the inheritance cycle, which is a fantasy series written by Christopher Paolini. And when he published the first book, he was still only a teenager, but it's just a very well written and engaging fantasy series. Lots of dragons and swords and sorcery, but just very well written, well thought through. There's a excellent, amount of world building that puts it somewhere in the echelons of Tolkien and CS Lewis.
So definitely worth reading. I I read them a while ago, and we just started listening through the audiobooks. It's a family, so worth checking out. Cool. So with that, I'll pass it to you, Brian. Do you have any picks this week? Oh, goodness.
[00:37:21] Unknown:
Alright. So my pick is a place most I am from Ireland originally, and a lot of people, go there, but they usually go to Dublin or Galway or Cork, and so they're very popular places. But there is a place that the further mo furthermost point from Dublin is, in the northwest called Donegal. And Donegal is a, very rugged, beautiful place in, in in Ireland, and it is familiar to those who watch Game of Thrones. Some parts of of the show are are filmed up in in Northern Ireland. Some of the Star Wars movies were or the recent Star Wars movies at least were filmed up there. But it is a place that is, unbelievably magical and defies expectations, and I wish everybody who who goes on a little trip to Ireland would go just those couple of hours extra
[00:38:06] Unknown:
and, go up to the wilderness of Donegal and get get blown away. Alright. Well, I'm hoping to make it to Ireland someday, so I'll have to be sure to, make a little jaunt out that way. So thanks for the recommendation. You bet. Alright. And thank you very much for taking the time out of your day to discuss the problems and, various solutions to secrets management for various applications. It's a problem that is critical to anybody writing software, so I appreciate that, and I hope you enjoy the rest of your day. You too, Tobias. Thanks for having me on.
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. When you're ready to launch your next app, you'll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 200 gigabit network, all controlled by a brand new API, you've got everything you need to scale. Go to podcastinit.com/linode to get a $20 credit and launch a new server in under a minute, and visit the site at podcastinit.com
[00:00:35] Unknown:
to subscribe to the show, Sign up for the newsletter and read the show notes. Your host as usual is Tobias Macy. And today, I'm interviewing Brian Kelly about how to store, deploy, and use sensitive information in your applications. So, Brian, could you start by introducing yourself? Sure. I'm the head of Conjur Engineering for CyberArk.
[00:00:52] Unknown:
Conjur is a, a vault or secrets management application, and CyberArk is a well established large, cybersecurity vendor. And, yeah, I've been in the software industry about 20 years, mostly managing engineers, and at times, sometimes they let me code as well.
[00:01:11] Unknown:
And do you remember how you first found Python?
[00:01:13] Unknown:
Yeah. I was a latecomer. I was working at a company called Datawire who does a lot of infrastructure for microservices. And, we were writing a transpiler, in fact, for, from a behavioral contract language into other target languages. So it's kinda like an ideal on steroids where you'd say, this microservice wants to talk to this other microservice, but it's not just parameters and, you know, paths and URLs. It's also retry logic or circuit breaking or other kind of behavioral semantics on top of the actual contractual parameters. So that was fun. That was my first major foray into Python. And sadly, now I'm, more in the Ruby world, but, I I really did love my time in Python for the those years. Yeah. Whatever gets the job done. Exactly.
[00:01:55] Unknown:
And so to begin with, I'd like to get your definition for what a secret is in the context of an application and, development environment.
[00:02:06] Unknown:
Sure. Basically, a secret is anything an attacker might want to broaden or further their attack surface, and they might use that secret to either get the value they want or to get closer to the value they want. So if it's a database password and it's the, password to the database that holds all of the bank account details or credit card numbers, they're pretty much done. Or it might be a very, very small innocuous secret that allows them access to impersonate someone else or do some Bitcoin mining on a small AWS account, you know, any of those things that where they can either move laterally and then move upwards in value or move upwards in in different, directions and, just keep moving. So that is a broad set. It could be a very small thing like a a, a password to something that you don't think is terribly sensitive, but stepping stone approaches can be used to make it more valuable later on. And as you mentioned, it might not necessarily be something that's directly exploitable, but it might be enough to
[00:03:09] Unknown:
lend credence to something like a phishing attack where they're trying to get somebody within the organization to perform some action that will then increase their foothold within the infrastructure or the organizational capabilities, or it might be something that they're able to use to decrypt or derive the values from some other set of information that they've already obtained. So maybe a salt for the hashing algorithm that you're using for your passwords or the Django secret keys so that they can then predictably generate a CSRF token or something
[00:03:42] Unknown:
like that. Exactly. And and would that everybody was using salts, would not be a great world. But, yes, all of those things are definitely, you know, maybe on their own, aren't terribly valuable. But if if used creatively or in conjunction with something else, suddenly make an attack, you know, really lucrative for that attacker.
[00:04:02] Unknown:
And so given the broad range of information that could potentially be classified as a secret within the context of an application or an organization, what are some of the broad categories of secrets management solutions that are available for people who are looking to go beyond storing everything in plain text?
[00:04:25] Unknown:
Yeah. So there's there's a variety. The 1 that people are probably most familiar with is a vault, and that's a software appliance that securely stores your secrets. And, those are you know, there there's plenty of of them out there with CyberArk, has has a a a very large enterprise vault, and also the 1 I work on Conjur is more focused on, DevOps flows and CICD systems. And, what that vault will do is just, you know, put it in a place that it's it's extremely hard to get at. The problems really occur when you deliver those. If you gotta open the vault, you know, banks are only useful if they open the vault and start doing something with their capital. So same with secret solutions. You store these secrets, but they've gotta go somewhere. So now you've gotta talk about delivery. And that's typically based on things like TLS as you'd expect, but it encompasses the authentication problem, which is these are not gonna be humans asking for these secrets. They're gonna be applications. So how do you how do you authenticate an application? You know, it's machine identity, not human identity. Even within the vault, though, there are things that some people are not aware of. It's a fairly mature space. So you have things like rotators, which on a per database or per service level, know how to contact that database or whatever it is and say, hey, you know, it's been 30 days. We need to change the password. They also have sophistication in terms of preventing the, interruption or intermittency of service when you rotate a password because you've gotta distribute the new 1. You've got a bit of a race condition. You know, how do you get the new 1 out there? See, there are strategies like dual accounts where you have 2 accounts always active and you're kinda flipping between them so there's a bit of overlap time. And then there's the whole aspect of authorization.
So once you've identified these applications that are coming to the vault, knocking on the door, and saying, I need the password for the customer database, please, then identifying them is only the first part that can see the authentication hurdle surpassed. But then you've gotta authorize them. Somebody, usually a human, has to say, is it appropriate for this microservice running over here to to get that database password? Some security staff member or officer of some sort has to know that that application should do that and should get it. And, also, at some point, maybe deauthorize it and and unenroll that application from the rights to that database. You know, maybe it's just grown out of that or maybe that database is getting decommissioned. But there's a whole life cycle around authorization, and, there's various solutions for authorization. We have an an RBAC, you know, roles based access control approach that there are other approaches used by other vendors. But we find the role based model good because it maps to human expectations around that, and you wanna really optimize for a human interaction with this system. So you say, okay. This is a microservice.
It is in this category or it's part of this layer of applications, you know, and and they all it's embedded within an auto scaling group. And whenever any of these come up, they need to have the same privileges. So there's a whole ecosystem of of products and and componentry in this space beyond just the secure vaulting of secrets.
[00:07:36] Unknown:
And beyond just storing and distributing the secrets, there are also a number of other aspects of secrets management throughout the life cycle of obtaining or generating the secrets and then deploying them to the application. So what are some of the other concerns that developers and operations engineers should be thinking about in the context of creating and storing and securing those secrets.
[00:08:01] Unknown:
Sure. It's unfortunate, but there are so many ways for your secrets to get leaked if you're not careful. And, the analogy I like to give is, you know, imagine this this big bank vault or or sort of Fort Knox kind of system where that's your, you know, the master storage system of everything. And then you you get your big delivery truck, you know, the big armored truck with with armed guards. You you take a couple of secrets and you're you're bringing them down to your application. And you hand them to your application and it's just you just leave them on the sidewalk and walk away. That's that's a good analogy for how a lot of secrets management is done in reality now. Very secure storage, secure transit, and then once they reach the application, an absolutely horrible situation, just a debacle of of, security problems.
For example, logging your, your secret, you know, logging your, whatever it is you received and just printing it out. It, you, it happens more often than not. In fact, if you run a Terraform, if you run Terraform, you probably know about, you know, how things end up in TF state files and whatnot. That will include your secrets. So you gotta be aware of that. There's a variety of other things that can happen. You probably wanna 0 that out in memory if you have an environment that allows you to, overwrite your memory section that had that. And a lot of interpret languages don't offer that, but that's good practice because leaving it in memory, beyond when you've set up your database connection or whatever is a good thing to do. But a lot of developers would consider that maybe too much. If it was injected into your process environment, you know, that might be difficult as well. So there's a whole set of problems that occur when you deliver a secret to an application.
And, really, what it comes down to is, you know, does the application and and this might be something the developers have written or it might be Jenkins or it might be Kubernetes or whatever. It's could be anything that's got anything to do with your application building, you know, delivery or or deployment. Really auditing and understanding how secrets get used outside of the vault is 1 of the biggest parts that people just forget. And they're all concerned about vaulting, and that's good. You need a vault. But beyond the vault, things are less sophisticated. And sadly, the hackers know that and they know, you know what? Do I wanna go after the vault that has had 10 years of development behind it and all kinds of pen tests? Or do we just wanna go and do a quick SQL injection attack on this this page? Because they probably just didn't even do the basics on, you know, on on database query management. Oh, look. There we go. We're in. And, that's kind of sad, but it's it is, it is the reality.
[00:10:32] Unknown:
And also, as you mentioned, auditability of the secrets also extends to creating the secret and whoever is actually obtaining the secret for distribution. So it's definitely important to keep a close eye on the entire path that a secret has taken from when it was first obtained through to actual delivery of the application so that if there is any compromise, you can identify at what point that may have occurred so that you can then try and prevent that in future situations.
[00:11:02] Unknown:
Yeah. Very much so. And auditing can be 1 of the noisiest parts. If you're if you're working a large enterprise, you'll have 100 of thousands of of pieces and and accesses, you know, every maybe every minute. And things are coming up and down. And when they come up, they've gotta request secrets. All those accesses, whether those are authorized or or not, and whether they if they were successful or not, they all have to be audited. That can create noise. And, and you need good auditing tools, I guess, and, and analysis tools to, to really figure out, you know, is there an attack going on right now? And then there's companies including ours that have solutions for that that say, you know, let's use some heuristics or machine learning or any kind of advanced technology to watch what's going on in the system and say, this, this is unexpected behavior over here. You know, we haven't seen this pattern in the last 6 months and all of a sudden there's a burst of activity. And so the point here is you you will need more than just humans to to deal with that level of auditing. And going into the actual use,
[00:12:03] Unknown:
by the application of those secrets, you mentioned the possibility of leaving the credentials around in memory after a connection has been established or or the fact that it may be present as an environment variable, which I know is 1 of the least secure ways to do it because if somebody manages to get a foothold on the system, they can fairly easily dump the environment variables of a running process to obtain those secrets. So what are some of the best practices for actually utilizing secrets once they're on a host to prevent them from then leaking to other processes or making them too accessible to somebody who manages to get some foothold on an instance? Sure. So there's a variety. And,
[00:12:44] Unknown:
the first thing I'll say is you can't leak what you don't have. And, therefore, 1 of the best approaches to preventing a secret leaking out of a process is for the process to never get it in the first place. So, we have a technology that does that and it's, you know, it's it's language agnostic. It's called secret list, and you can download it from secret list dot I o. And that's a fairly new novel approach to this. The secret is delivered, but it's delivered to a a process running side by side with the actual application. And, you know, I won't go into all the details on the podcast, but certainly there's there's a variety of architectural benefits to that. But let's just say you're doing the classic sort of, you know, application needs to get a secret pattern. The first thing to to do is make sure that the secret can only be used by that application so that if somebody steals it, the usage of that secret elsewhere should fail. Now how do you do that? Well, you can have a multi factor authentication model that, you know, we're all familiar with from our phones and and, Duo or whatnot, but that same model can be applied to applications.
So for example, if you have an application running in a, scheduled environment like Kubernetes, you can have certain factors associated with that application. For example, Kubernetes has an overlay network, and it could dole out IP addresses to applications. And an IP address or having an IP address assigned is not good enough for authentication by itself, but it can be a factor in the authentication handshake. So that's 1 approach, you know, and I know there there's other factors you could use like, hashing of the application code or, you know, a variety like that. And then, you know, the once that's done, the authentication is required to use that secret. So then if, somebody comes and and tries to do something else elsewhere, it it might not might not succeed. But generally, that's that's a high level of sophistication.
And usually, if an attacker is able to get a secret, they are going to be able to use it. So if there is any chance at all of 0ing out zeroing out memory, it should be done. If you can't do that, then narrowing the amount of time that a secret can be valid for is good. So rotating it so that, you know, if they get it, they just they can't just lurk for months using, you know, your secrets and and doing things. We have some customers that rotate very aggressively, like every 30 minutes. And, you know, we we have plenty of people looking into even faster ones than that. And, you know, it's strategies like that that that bound the the time or the scope of of validity of that secret. Those are generally the most successful. But we feel that, you know, this whole thing is kind of a fool's errand in a way. Yeah. Because application developers have proven themselves to be understandably more focused on feature development.
And then on security, security is hard and it's hard to sort of prevent holes or prevent flaws. So our, you know, thinking now is, well, let's just take it off their their plate. Let's just say it's okay. We'll do it for you. The secret can can, be delivered somewhere else, and you still get all the benefits of that. And you just don't have to worry about protecting it because honestly, you know, this is why we're having this discussion. It is really difficult to manage. And so in terms of
[00:15:58] Unknown:
somebody who is maybe a single developer working on a small application, what are some of sort of the hierarchy of best to worst ways of storing that secret for use by the application,
[00:16:26] Unknown:
So So I would say, starting at the whole, like, you know, post it note level, just running it down and and putting it somewhere statically. That's actually not as bad as, you know, putting it somewhere like GitHub in a public repo where all of a sudden I mean, I I just hope your listeners are aware there are bots watching every public repo check-in and scanning for predictable patterns like things that look like AWS keys. And if you put something out there like that, you will have a a a Bitcoin mining operation on your AWS account within minutes. And also the key will be changed, you know, so that you can't even go and revoke it easily. So that is like the the level 0. That's that's the place you wanna never end up in. Next up is is a good place to store the secret. So vaults, there's plenty of them. There there are open source options. There's no excuse for not having a vault, and it is absolutely a required part of this. So have a vault. It's a thing that I notice when I go to, you know, dev ops days or security conferences or or even just plain developer conferences. There's a lot of focus on the tooling and the infrastructure, but less on the developer experience. And the thing that makes or breaks a strategy for secrets management is how easy it is for a developer to deal with it. So if you require if you set up some know, Rube Goldberg machine that requires 17 steps and all kinds of machinations to make it work, developers are not going to adopt it. They're gonna sidestep it. They'll they'll pull out the password, and they'll just, you know, configure it statically somewhere or something really bad because they're busy and they have stuff to do and dealing with an an overcomplicated process is not gonna be good. So focusing on the developer experience from the start is good, and it can be a great developer experience with not so good security to start. And just as long as you maintain that developer experience as smooth and sort of invisible and as as provisioned as possible, then you can keep improving security as you go. And the the things that you would do to to your question about the hierarchy environment variables are not as bad as you might think. We, we offer a tool called summon, which injects them for you, and it also takes care of pulling them from your vault of choice.
And it will, you know, it'll do a good job of, managing all the security around that. But, yes, it does go into the environment. Beyond environment variables, I mean, then, you know, if you have an application language that allows for memory zeroing to occur, then, you know, like C or C plus plus then nuking that out. But if you don't, I mean, secrets have to get to your application. They have to be used. And at some point, they'll probably get rotated, and you'll have to go back and get the new version. If you can't rotate it and they have to be put into memory somewhere, then that is gonna be something you're just gonna have to live with. And then your next level of security is hardening your application so that it doesn't have, you know, basic OWASP vulnerabilities.
And then the next level above that is hardening where it runs. You know, if you have you got good I'm setup in Amazon or wherever your environment is. Then there's there's lots of pieces in between. Like, if you're running in Kubernetes, you could leverage Kubernetes secrets, which is fairly immature and and has come a long way. But it's it's it's not a really good security solution, but it's great for developers and things get injected or dropped into apps nicely by Kubernetes. So there's a whole journey you can take, but I would I would sort of emphasize the the human aspect that if you make something easy and invisible, it'll get used. If you make something difficult and super in your face for developers, they're gonna just get annoyed by it and stop using it or go around it. And in terms of the secrets themselves,
[00:19:58] Unknown:
there are things like database passwords that can fairly easily be rotated with some measure of regularity. And then there are more long lived static attributes, such as an API key for a service that you might need to interact with that you might rotate less frequently. But you may want to have some aspect of the versioning or history of some of those secrets. So I'm curious what your thoughts are on that area of secrets management, whether you should be able to keep multiple versions of a token, for instance, for historical purposes, or if you think that that is unnecessary and you should just focus more on continually rolling forward with those values.
[00:20:40] Unknown:
Yeah. So it's it's a really important point because it is something we support and most most valve vendors will support it. It's generally good practice because it, it makes things easy for developers and, and operators like it because it makes it less likely for things to be going down because, oh, we rotated our 1 our 1 true database password, and now everything is, unable to connect, and they don't have good logic to say, oh my god. I just got 401 or 403. I need to go back to the to the vault and get the new the new password. So versioning is definitely good. The only thing you gotta watch for is that you don't version so much and have so much flexibility that it becomes an attack vector in itself. So, you know, having a good small finite set of versions be valid is, is good practice. If you can avoid it, you should avoid it. You know, having multiple ways in, having multiple values for something might be a hacker's dream, but certainly for, you know, building that balance between security and developer flexibility or, or, or production operations flexibility is super important. I would just make sure it's an explicit decision on the team, not something that just grows on a Friday evening. Someone's like, oh my God, they did this deployment and, here, just turn on the versioning feature. Okay. We're good. Let's let's just leave that on now. That would then not be a good way to have that turned on. But as an architectural decision where people say, you know, we've examined this, and we think we need 5 versions of our s 3 keys in the application because of how many things are using it, how long it takes to rotate, and then the general notion and and awareness of how our applications work, we feel that is appropriate. Well, that's a good discussion. That's a good approach and 1 that comes with that sort of economic understanding of the balance between security and flexibility.
[00:22:33] Unknown:
And beyond the security concerns of secrets management, there are also some functional and operational concerns such as ensuring that the values you're using in an application are still current and still valid and that you don't accidentally revoke any credentials that are still in use by applications. So you have to have some aspect of inventory management of which secrets are in use and when. So I'm curious what your thoughts are on on some of the best strategies for ensuring that you are able to keep those secrets current and that you don't accidentally either revoke all of your secrets or let them expire or have them be too long lived?
[00:23:15] Unknown:
It's it's definitely a thing that requires diligence and human brainpower and time to do. And, and then we consider it almost like what we do with our calendar meetings, you know, in our job, which is nobody's allowed to set up, at least in our team, a never ending meeting series, you know, to, to pull people into a room every week. You can set it up for a certain amount of time. And then at the end of that time when it when it's, when it's over, if if you wanna renew that that meeting, you've gotta justify it again. And say, yeah. Let's do another 6 months of whatever meeting we were doing. Same approach for secrets. Things just just being let live forever, they become attack factors. And and, again, that could be almost always the innocuous ones that that people are are not thinking of. So bigger enterprises will have, you know, reviews scheduled, every whatever months, and they'll get together and and go through things. And if you have a good auditing system, you can just see, you know, what's the activity on it. If you have a good authorization system, you'll know who or what is accessing accessed in 3 months. Is that because the application that pulled it is so amazing? It has had no downtime in, you know, more than 3 months and it hasn't had to repull it. That may be the case, but it may also be the case that nobody needs it anymore and you can just get rid of it. But it is definitely a investment that humans have to make. And, it's, it's just 1 of those things. You, you just like you would go around your building and make sure that, you know, windows stay closed or or doors have have good locks on them. You know, if someone makes a little side door just for some temporary access while you're doing some construction, well, someone's gotta remember to go seal that up, and, same thing for secrets. And in terms
[00:25:04] Unknown:
of the delivery of those secrets, a lot of times, it'll be managed by some sort of automated process, whether it's configuration management for building and deploying servers and software, or it might be an agent on the application side that knows how to authenticate to the vault and pull the secret. So I'm wondering what your considerations are as far as the best practices for some of those different environments, particularly on the configuration management side. Yeah. So the the thing you wanna watch for in configuration management tools is the keys to the castle problem. And,
[00:25:39] Unknown:
a lot of these tools, you know, the big the big ones like Puppet Chef, Solid Stack, etcetera, they, they were architected initially with configuration management in mind. No surprise. Not as much, security. So some of them have challenges in that when you when you start to distribute secrets using those tools, that all of the secrets for the entire system get pushed down to, an application even though the application might only need 1 out of the 200 secrets that are sent down to it. So that's a significantly over privileging of that application. And least privilege, if possible, should be used so that the application only gets the secret that it should get. We've done integrations with Puppet, etcetera, that that allow for that to happen. But, you know, sometimes it does feel like we're we're fighting the tide a little bit because we're trying to implement a model that those tools don't innately support out of the box. It is possible and, you know, certainly can be done, but it can be it is extra work. What I would suggest people do is that they they really understand how secrets get distributed.
Because let's say you take an integration for some vault and and integrate with a Centimeters system. Does that mean that all the secrets go to the the master sort of we'll call it the master distributor or or the master configuration piece, whatever server is used to drive all the agents that are running, you know, elsewhere. Then if that thing, which is not a vault, gets all the secrets and has them stored in memory, well, then now you've just completely destroyed all the value of having a vault in the first place. So how are secrets getting there? How are they transiting through? Are they getting removed? And then how are you dealing with rotations?
So I'll be honest. This is 1 of the the big areas for vulnerabilities that secrets, you know, are generally vaulted well. Most people know, you know, oh, yeah. I should put my secrets in the vault. And then it's like, oh, well we have to distribute the more we have to, you know, get our infrastructure set up. Alright. Plug in the Centimeters system and and it just leaks like crazy or it puts them in places that are highly vulnerable and not hardened, appropriately.
[00:27:43] Unknown:
And then there are also the concerns of managing secrets across different environments all the way from a developer's local machine through QA environments to production. So what are some of the strategies that can be used for managing secrets across those environments to ensure that you mitigate the potential for any sort of security breach or,
[00:28:08] Unknown:
introducing vulnerabilities at any of those different stages? Yeah. It's a great question. Going back to that developer experience part, this would also incorporate into operator experience and making sure that it's seamless. So what you want to have is the experience for a developer who's building a small application or a microservice. You want that model, whatever is used for distributing a secret or managing it to be identical between development, staging, QA, production. And you also don't wanna make your problem more difficult by having 1 vault to try and serve all of those environments. Typically, what's done is you would have a vault per environment, and you would use a declarative immutable infrastructure approach, which is all very DevOps y and very, you know, good to say that the the the mode of access and the method of access is identical, but the value of secret that you return, each time is different. So when you're running a development, the development version of the app pulls the development secret for the development Oracle database. And there's no chance of that developer accidentally posting the the production Oracle database secret, on on their Twitter accounts or whatever. Now when that application flows to the other environments, and that's through its CI system, they the appropriate secrets are pulled for that environment. So that is a bit more infrastructure than people you're expecting. A lot of people think I'll just have 1 vault and just protect everything in there. But it might be easier to have, a a vault per environment. And certainly from a security perspective, that would be the appropriate way to do it because then you've got complete isolation. And if somebody does breach Evolts or does get, you know, elevated access in there, they're only getting the passwords for that environment. And if it's not production, then you're you're you're in a much better shape. So focusing on the kind of things that make an application successful from a, just from a non security perspective, like the developers already do this stuff. They say, oh, well, I'm gonna look for my environment here regardless of whether my application is running in dev, QA, or production. Well, guess what? Go look for your secret in the same place, regardless of whether it's running in dev, QA, or production. That same strategy works wonders here, and and it makes this kind of stuff just so much easier and more predictable.
[00:30:23] Unknown:
And 1 of the ways that you can enable that kind of interface too is with something such as service discovery layer where you can use a consistent DNS name across environments that will actually resolve to a different instance or a different service within those different environments. But from the developer and operator perspective, it's homogeneous regardless of where you happen to be running. Absolutely. That is exactly right. Yep. And what have you found to be some of the most useful resources for learning more about the different aspects of secrets management and application security for anybody who's looking to go deeper down this rabbit hole? Sure.
[00:31:04] Unknown:
The, the vendors in the space actually have some really good blogs, and, you know, share on their social accounts. So, you know, our blog dot conja.org, we write about this all the time. Certainly, other vendors do too. I would also encourage people to follow people like crabs, you know, crabs on security just to keep aware of the whole space, not just secrets management and any of those. There's plenty of sites that have like breach of the day, which is kind of such a sad statement about our industry that, that they now have to do multiple posts a day to to report, you know, all these different breaches and hacks of of our data. And, looking at the patterns and look at the the astonishing simplicity by which some of these attackers were able to get massive kind of value from their hacks is both depressing, but it's also important to know. I will say this is a bit of an un underserved space. And and, you know, beyond the vendors like ourselves and a couple of experts that it is seen as a, an adjunct or side or secondary topic. And it doesn't get the same kind love from the bloggers sphere. The bloggers sphere as as a immutable infrastructure or or DevOps culture or all the stuff that people just write constantly about. It's just this stuff is really important and it could use with some more writers. So generally, I I looked at Twitter. I've a I I follow a bunch of people on Twitter who are working in this space and and doing some really interesting stuff there.
[00:32:23] Unknown:
Yeah. And are there any other aspects of secrets management, either technically or culturally or organizationally, that we didn't discuss yet, which you think we should cover before we close out the show?
[00:32:36] Unknown:
Sure. There's there's 1 last thing to say, which is, you know, building a solution, a coherent solution for secrets management. This feels like a tools problem, but it's really a UX issue. You've got to solve for multiple persona. So like developers, operators, security staff, regulators, you've got to solve for different environments and what those persona expect to be appropriate in those environments. And as we talked about earlier, it's good to have 1 approach or 1 pattern for those. You've also got to solve for application identity instead of human identity. And the last thing I'll say is, you know, getting your your value pipeline, your delivery pipeline, your CICD system, providing secrets and getting secrets sort of in there very early in the process is 1 of the best things you can do. And it allows you to focus on the UX, the security UX, if you wanna call it that, for those different personas as you progress. Because if you end up with a a big master plan, a 2 year plan to to create a a secure pipeline, it probably won't ever get deployed. It's just gonna get deemphasized, and it's just gonna be more and more ceremonial and then more and more, risky to deploy because it's just this big thing that we're waiting to to pull the trigger on. Instead, make small changes, do them early, and go incremental.
It feels like a thing that you have to solve all the problems first, but really, it can be addressed, you know, piece by piece. And as long as it has that guiding directive of making developers' lives easier, making it predictable for operators, applying least privilege, all those things. You can do that, not be the world's greatest security, you know, solution or not have the world's best, secure pipeline, but you can certainly be on the path to being like that. And on the topic of incremental improvement,
[00:34:20] Unknown:
1 of the things that we didn't discuss much yet is the idea of some of these secrets management tools that are focused on creating gpg blobs in source control for distribution as sort of a first step towards having a more sophisticated full featured setup using a software or a hardware vault. So I'm wondering what your thoughts are on that stage of an organizational or project life cycle, or if there are better steps that can be taken with a similar level of complexity.
[00:34:53] Unknown:
I'm actually not familiar with the power. You're saying they they embed a GPG
[00:34:57] Unknown:
signature? So there's, a few tools that will let you add the public GPG key of different users in your development team so that when you encrypt a secret, it can be used, by those different people for deployment or for local development. And then the encrypted GPG, ASCII armored text goes into the source control system.
[00:35:18] Unknown:
Generally, I am not a fan of approaches like that just because, you know, the the the very last few words you said there were like, and then the secret goes into the source code control system. Even encrypted in there, it's like, it's just, you know, a little too hairy for me because now you've you've exposed the secret, not in plain text, of course, but a an attacker knows where to look and how to get it. And then their their next step will be, okay, how do we either socially engineer or otherwise wrangle our way to getting the key to decrypt it. And, you know, it's it's preserved, you know, history wise. It's it's, it becomes em emblazoned as this approach, and and that'll be hard to to pull out later on. I don't know how it deals with rotation and things like that. So I will say there's some sniff. It it there's some sniff tests that it's failing. However, as a, as an early stage approach, it might be okay. I would say 0 of the enterprises that we work with, and then we are skewed to dealing with, you know, a lot of the Fortune 500.
[00:36:19] Unknown:
0 of those would use that approach and would would approve it. So I don't know if that's a how helpful a data point that is, but it's, it it it's something to be considered. So for anybody who wants to follow the work you're up to and keep up to date, I'll have you add your preferred contact information to the show notes. And with that, I'll move us into the picks. And this week, I'm going to choose the series called the inheritance cycle, which is a fantasy series written by Christopher Paolini. And when he published the first book, he was still only a teenager, but it's just a very well written and engaging fantasy series. Lots of dragons and swords and sorcery, but just very well written, well thought through. There's a excellent, amount of world building that puts it somewhere in the echelons of Tolkien and CS Lewis.
So definitely worth reading. I I read them a while ago, and we just started listening through the audiobooks. It's a family, so worth checking out. Cool. So with that, I'll pass it to you, Brian. Do you have any picks this week? Oh, goodness.
[00:37:21] Unknown:
Alright. So my pick is a place most I am from Ireland originally, and a lot of people, go there, but they usually go to Dublin or Galway or Cork, and so they're very popular places. But there is a place that the further mo furthermost point from Dublin is, in the northwest called Donegal. And Donegal is a, very rugged, beautiful place in, in in Ireland, and it is familiar to those who watch Game of Thrones. Some parts of of the show are are filmed up in in Northern Ireland. Some of the Star Wars movies were or the recent Star Wars movies at least were filmed up there. But it is a place that is, unbelievably magical and defies expectations, and I wish everybody who who goes on a little trip to Ireland would go just those couple of hours extra
[00:38:06] Unknown:
and, go up to the wilderness of Donegal and get get blown away. Alright. Well, I'm hoping to make it to Ireland someday, so I'll have to be sure to, make a little jaunt out that way. So thanks for the recommendation. You bet. Alright. And thank you very much for taking the time out of your day to discuss the problems and, various solutions to secrets management for various applications. It's a problem that is critical to anybody writing software, so I appreciate that, and I hope you enjoy the rest of your day. You too, Tobias. Thanks for having me on.
Introduction to Brian Kelly and Conjur Engineering
Brian's Journey with Python and Secrets Management
Defining Secrets in Application Development
Overview of Secrets Management Solutions
Challenges in Secrets Management
Best Practices for Utilizing Secrets
Hierarchy of Secrets Storage Methods
Versioning and Rotating Secrets
Operational Concerns in Secrets Management
Configuration Management and Secrets Distribution
Managing Secrets Across Different Environments
Resources for Learning About Secrets Management
Final Thoughts on Secrets Management
