The Pulp Artifact Repository with Bihan Zhang and Austin Macdonald

Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. When you're ready to launch your next app, you'll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 200 gigabit network, all controlled by a brand new API, you've got everything you need to scale.

Go to podcastinit.com/linode

to get a $20 credit and launch a new server in under a minute. To get worry free releases, download Go CD, the open source continuous delivery server built by Thoughtworks. You can use their pipeline modeling and value stream app to build, control, and monitor every step from commit to deployment in 1 place. And with their new Kubernetes integration, it's even easier to deploy and scale your build agents.

Go to podcast in it.com/gocd

to learn more about their professional support services and enterprise add ons, and visit the site at podcast init.com to subscribe to the show, sign up for the newsletter, and read the show notes. Your host as usual is Tobias Macy. And today, I'm interviewing

Austin McDonald and Behan Jang about Pulp, the platform for hosting and managing software package repositories.

So, Austin, could you start by introducing yourself?

Sure. I'm a software engineer at Red Hat.

I've been on the Pulp project for almost 4 years. In my spare time, I play the horn, the French horn, although that's not technically correct.

Okay. It's not technically correct that it's the French horn or that you play it?

I guess you could say either 1, but, it's technically not French. Okay. And, Bi Han, how about yourself? I'm Bi Han Zane. I'm a software engineer also with Red Hat. I've been on the pulp project for about 2 years now. I've been knitting a Legend of Zelda Link costume in my spare time for Dragon Con. That's pretty fun. I'm about 1 5th of the way through. So, you know, most of it to go, but we'll get there. Sounds like quite the project. And, Austin, going back to you, do you remember how you first got introduced to Python?

Yeah. I used to have well, I still do. I have lots of terrible ideas constantly. And before I knew Python, I would pitch these ideas to my friend who knew Python and he finally got tired of it and told me that I should learn it. And then when I was in school, I found myself an unexpected Bi

Han, do you remember how you first got introduced to Python?

Yeah. I first heard about it in xkcd.

This is about 7, 8 years ago. I was in high school and I haven't really got into programming yet. I mean, I did do a little bit of like HTML, JavaScript stuff on the side, but, you know, I was a bit of a nerd already.

Found out about xkcd.

Read a whole of it in, like, 1 night. And, Randall Munroe is the creator of it. Loves Python. Like, any chance he can get, he he just mentions it. Like, puts down Pearl. Just, you know, Pearl sucks. Python's awesome. So that's how I got exposed to it initially.

And I didn't really use it until college when I finally decided to read up on it. Decided it was awesome and used it for my scripting needs ever since. And so

we're here to talk primarily about the Pulp project. So can you

describe a bit about what the project is and

what the motivations were for starting it in the first place? Sure. So when we refer to Pulp, we're referring to the whole project. There's multiple components. It's a plugin based open source content manager,

and Pulp core does most of the grunt work, and

then plugins built on top of it allow you to manage any kind of content.

So Pulp initially started as a Red Hat project for basically managing RPMs.

And, initially, we only had 1 stakeholder and that was Red Hat. It was only in the past

4 or 5 years that we started leaving that behind us and trying to grow it, grow a community around it. So, like, a lot of this effort was just going to more like open communication. So instead of making decisions in a

Red Hat employee only meeting,

we, started,

like, using the mailing list. It was slower and it was a little bit of a culture shock, but it's been really successful.

We've recently gotten our 1st, like, community contributed plugin. Someone,

Misa from

SaaS actually, wrote a, Debian plugin, and,

it works. It's really great.

He just wrote the whole plugin, and it's 1 of our most requested content types. In the future, we have a,

community only

run boost at Frostcon in Germany.

So, we went from this small

employee only project to a community, and that's kind of the gross of pulp.

And there are a few other content systems or artifact repositories

that are available in the marketplace. Some of them are open source. A number of them are closed source.

But what are some of the use cases or benefits

of hosting your own artifact repository as opposed to just relying on publicly available repositories

for the various types of artifacts that you might need to use for building a system? A lot of times, it's kind of by necessity.

Many companies have, like, a VPN and, like, really wants to control what goes over their firewall. It was actually a Python I was talking to this, government contractor

and their company, it blocks off all PyPI.

So the way this guy would download software is he would go into PyPI, get the wheel or source, install it, see if there's any dependencies. If there is, just go back and download it. So if he hosts his own, like, you know, repository manager on-site, then he can just go pip install from that repository manager. Austin, do you want to talk about, like, reproducibility?

Sure. This is 1 of my favorite topics. So before I came to Red Hat, I started at the Center For Open Science, and I learned quite a lot there. But 1 of the things that we talked about all the time was how to improve,

reproducibility

of science. And 1 of the things that kept coming up was that you need to make sure that your software is absolutely

the same in the new environment that you're trying to reproduce. So that means the software that you're using, but it also means the dependencies.

And that can be pretty challenging to wrangle all on your own. And so we're building Pulp 3 to consider the reproducibility

use case. And actually, we're gonna be, presenting at SciPy next month to talk about that. Yeah. And to talk at SciPy isn't just going to be on using Pulp. It's gonna be on, like, a lot of using a lot of different tools like Ansible or like Python, like, requirements dot text or PIP file,

PIP file dot lock.

And referring to Pulp 3, I know that currently

it's in the beta process and you've been working on building that, and it seems that there's a fairly significant

architectural shift between versions 2 and versions 3. So could you talk a bit about what's changing and what your goals are with the update to Pulp 3? Sure. So the architecture

is gonna be mostly similar in form. We're gonna have a web server. We're gonna have a database, etcetera. 1 of the bigger changes is that we're switching from MongoDB

to Postgres,

but, we really don't anticipate

those,

kinds of changes

having much effect on the the users other than being a little more stable. On the other side of things, the rest API will be changing, and we're introducing a new CLI. These things will,

affect the user experience, but not be too impactful on the workflows that they do with Pulp. And,

luckily, most of the data should be 1 to 1, so it'll be easy to bring forward. And we plan to have a migration with no downtime. That's the goal anyway, from Pulp 2 to Pulp 3. And another thing to note is that we're going to semantically version our plugin API. So in Pulp 2, so our plugins just it's not semantically versioned.

It could change anytime. But in poke 3, we do separate out the, poke core aspect and the plugin API aspect. So Pulp core can continue development where also plug ins are still guaranteed to work with a system plug in API.

And can you talk a bit more about the

overall architecture of the system in terms of how

the plug ins interface with Pulp core and

what types of data are stored in the database versus a file system, or if there are integrations with other types of storage mechanisms for the artifacts that you're publishing?

Sure. So,

Pulpcore will handle

all of the stuff that's generic,

between plugins. So Pulp core will store information about your repositories,

and it also provides classes that can be subclass by plugins.

And that's pretty much what we mean by the plugin API. We've also got a couple of tools, but for the most part, we're talking about an inheritance model. And so the plugins might override a class for

their content unit. So for PyPI, we've got a Python

I think we call it Python package content.

On that model, you just add all of the fields that are unique to PyPI.

There's some other classes that you'll make too. 1 that tells Pulp how to interact with PyPI,

1 that tells it how to generate a publication

so that other people can use your software.

So at a high level, for somebody who's first getting started with using Pulp 3,

what does the workflow look like for getting set up with the system and being able to publish and consume packages and how that data is propagated throughout the system? Sure. So let's take, like, a pipe, a Python package, for example. So pope has this concept called repositories

which is a collection of

software content over time. So every time you add and or take away content from repository, it creates a new version of the repository. So, if you're ever unhappy with your changes, you can always, you know, go back to a older version.

So first you could you can create 1 of these empty repositories

and then you can create like a we call it a remote, but it's it's basically like a pope. Like, it's,

it's a configuration

of,

something like up in the cloud. Right? So, Austin, you wanna say some words?

Sure. So

a remote is really a configuration for how to interact with an external repository. That's It's a really good way of putting it. Alright. So you can configure remote.

So in this case, it could be PyPI or test dot PyPI. You would sync content from the remote repository to your local repository. And, once the stuff is synced, version 1 of the repository is created with that initial

list of content. And Tobias, I think you asked about storage. So also metadata is stored in databases

and the file itself is stored in a

Postgres file field, I believe. So it sounds like it's relying on the Django storage's project for being able to

allocate where the file is actually placed, whether it's on disk or s 3 or some other types of type of object store? That's right. That was 1 of the big reasons that we wanted to move to Django

in the first place. And so is

the version

2 of the application,

different web framework? And then in version 3, you're moving to use Django, and I believe you're also switching to Python 3 in the process?

So version 2, we actually write the files on disk and create some links to avoid duplicates, and we serve, like, all of this local files on disk with Apache.

It's not

the fastest way of doing things, which is, like, a big reason why, like, we did the rewrite in the first place. But it does allow us to have 0 downtime on migration because all the static files will still be there.

And can you do a bit of comparison to how Pulp fares against some of the other artifact repositories,

most notably things like Artifactory

or some of the more single purpose repositories

like some of the various app repositories

or devpy for Python packages, things like that? Yeah. Definitely.

So compare, comparison to Artifactory, like, we're open source. We support less content types because we're like a development shop of 10 people plus a community of, you know, less than 20 people. And Artifactory is this big multimillion dollar company that has a lot of employees. So we definitely have less content types. But like of the content types we do support, we know them very, very well. So Austin is actually a contributor to warehouse, which really cool. So as far as Python support goes, like we got some metadata, Python metadata stuff down. As far as other things go, like we know our RPM. We have like we know a lot of people who know RPM. Docker wise, like, we just we have people who work on the open source container initiative.

Like helping us get the Docker plug in done. And just we we know our stuff. We have less stuff, but, like, of the stuff we do support, we support it very, very well. Oh, Austin, did you wanna add something to that? Yeah. The big addition with Pulp 3 is that we're designing it around writing plugins.

And so 1 of the big goals is to make plugin writing as easy as possible, and I think we've come pretty close to that. And we're getting our our docs in line, and I think it's actually quite simple to write a new plugin to support a new content type. And so we're just entering the beta recently.

So as we go forward, we expect to see a lot of new plug ins coming in from the community. There's also the added benefit of not having to run the JVM to get it up and running.

Yeah. That too.

But it's also great because, like, people who have, like, their custom content,

which, you know, Artifactory is never gonna help you support that. But, they can spin up pope and, like, make some changes to an existing plugin or just spin up their own because it's really, really easy and just support that themselves. And you asked about

the devpy. So pope is great because it can it supports a lot of different types of

content, right? And it manages all it manages all of it was like pretty much 1 API. So you can you can manage your Python content along with your Docker images, along with your like Ansible roles. All with 1 set of API commands. But if you were to spin up and host like devpy yourself and host your own like Ansible Galaxy and like host your own like Fedora,

repository. Like that takes a lot of effort and a lot of, like, different,

like, for assist admin to be able to support all those. They need, like, really deep understanding of all of them. Whereas with pope, you can slurp 1 API and be able to support all of those. Another thing to add to that is that we really have focused a lot on our API, our REST API. Well, it's REST ish, but we've focused on our HTTP

API

and that's given us a lot of benefits because

now we're working on the CLI. We're working on bindings. There's other projects.

There's a community contributed module for Ansible that allows you to use

Ansible

to manage Pulp.

And all of these interfaces

just rely on the REST API. So it's a pretty powerful

system,

and it can be controlled in pretty much any way that you want. And on the client side, do you have compatibility

layers for things like PIP or the Docker CLI

or RPM or apt for being able to natively interface

with Pulp as a source of those various content types, or is it necessary

for the user to build some sort of compatibility layer to download the packages and then independently install them from disk?

So this is what the Pulp publications do. So publications

serve as a, way of getting something out there that can interface with PyPI and that can interface with, the Docker COI. At least PyPI wise, like, we can we currently, for Popes, we publish

simple API. So, like, you can just point pip straight to that, and you can just do, like, pip install.

Forgot to exact command, like dash dash. Austin, do you remember? I think it's dash I. But the easier way to do it is and this is the reason I don't remember the command, is just to add the URL in your pip conf.

So, like, pip can definitely interface with PulpStream right now. Docker is a little bit more complicated because, like, it does require some live sync serving it. For poke 2, we have this project called crane, which kind of acts as a,

it acts as a Docker registry.

So that's how we're able to do in Pulp 2. For Pulp 3, we'll we would probably have something similar. Although we are not restricted to that, Pulp 3 allows plugins to register endpoints at any endpoint at all. And so

it really opens the door for implementing an entire live API

and doing whatever your plugin needs. On the Python side of things, we were considering using this live API to accept uploads from Twine, for instance. And as far

as the milestone

for a final release of Pulp 3, what are some of the major pieces of work that you have left along that path?

And do you have any sort of expected timeline of when you hope to reach that point?

So Pulp Core is

probably going to be in our seas this fall, so September, October ish. There's a couple of pieces of work left for that, and that's we need to get our Ansible installer done. We need to get our COI squared away and we have,

a little bit more like minor rest API, plug in API changes left to do. But,

also, like, we know what to do, like, the the scope is clear. But once core is in RC,

we start, like, really diving into plugins.

And, if during the course of any plugin development we find that there's some architectural issues with core. We'll have to, you know, account for that. But we don't really know until we start diving into plugin development. Hopefully next year at some point. We'll go like, GA was everything. 1 of the things that we'd like to do is have a really long

RC process

because we take semantic versioning very seriously, and we wanna leave a lot of room to make sure that we get everything right. And do you feel that the current implementation

is at a state where early adopters can start installing and working with the system to help you find some of those sharp edges

and polish some of the cases where you need to add improvements or possibly

bug fixes or raise things like that? Yeah. Definitely. We're at a point now where it is it's very clean. It's not changing very fast.

We tend to get a little worked up over small changes sometimes, but that's because we're all taking it very seriously. Go ahead, Bi Han.

Oh, okay. Yeah. We definitely are.

We've had, like, a couple of plugin development workshops for PulpStream just to, like, have other people test our plugin API and those have been really successful.

Someone wrote a, like, the initial

start of, like, a Maven plugin, so that's really cool.

But it's definitely ready for testing and for like all kinds of feedback. And like, if if you want, like, to have your feedback considered and like, active poll for me, really quickly, now is the time. If you ever want to use pulp, try it now. See if there's there's anything missing. If there is, let us know and we'll, like, get it in. I would actually say that the Python plugin is past where it was for Pulp 2. Oh, yeah. So,

the the Python plugin for Pulp 3 is

very feature rich and is will be useful already. Yeah. We kind of, like, stayed up all Python working on that.

What have you found to be some of the most interesting or challenging or unexpected aspects of building and maintaining the Pulp project? So

a project like Pulp is

inherently difficult because, well, first of all, content management is a complicated problem,

and it always is. And it always doesn't seem like it should be, but it always is.

And you can look at the history of Python packaging, and you can look at the history of RPMs, and it's just it's always messy. And that's by necessity because it's a messy domain. But 1 of the problems that we run into frequently in Pulp is that we are trying to work on something that can work for all of these

different answers to the same

difficult, complex questions.

And so

we often see Pulp through the lens of

the plugins that we spend the most time with. For me, I always see it through the

Python lens,

but others on the team may see it through the RPM lens or the Docker

lens. And we tend

to want to pull it in different directions because of that. But in the end, we're usually pretty well able to come to a good consensus

about what the best

thing to do across the board.

And 1 thing that I think is worth doing a bit more,

of a deep dive on is you mentioned briefly the idea of

versioning

the repository

and being able to

roll back and roll forward based on

as you publish new releases

or new

sets of packages

to make it easier to maybe move forward or backward in time to reproduce various states of a system. So can you talk a bit more about how that works and some of the use cases that it enables?

So 1 of the main use cases is just rollback. Right?

A Sysadmin publishes a piece of content. A user finds out it doesn't work and they need to go back. So with pope, like, because the publications are stored, like, publications are also version. But because, like, the repost and publications are versions, they could just roll back pretty much instantaneously to the previous version. That's also what enables the reproducibility

use case to work. Like, once you create a publication, you know exactly what's in that repository

can't ever change. Can't ever change is a little bit strong. Like, if you published a password in a publication just by accident, like, there are ways you can change it, but it's it's fairly difficult. Well, you can you can delete the publication, but the publication itself will never change on you, and so you can rely on it. Another thing we should probably mention is that versioned repositories is 1 of the big draws for Pulp 3. It's a completely new feature in Pulp 3, and we've built everything around it. And

it I think it really will make the whole thing

much easier to go through your workflows.

And I think that it's definitely 1 of the more appealing features, and it would also help with

being able to do staged releases

of dependencies.

So you're able

to point your QA system, for instance, at the newest version

and

your production instances

at the prior version,

and being a be able to

determine with a fair amount of certainty that even if you need to redeploy your production system before you are done with your QA testing. You're not going to accidentally pick up new versions of packages.

Exactly.

And also,

worth noting is that

because we create these objects called publications,

enabling them or disabling them at a particular place

is

a synchronous action. So in a single REST call that can't be interrupted,

it will,

automatically

forward

your request

to the old publication if you roll back. And so you don't need to worry about

a moment of downtime while you're republishing or anything like that

because we're using the same publication

in both places.

Yeah. It definitely makes a software development life cycle a lot easier. I know that Python in particular doesn't

exactly support anything like that, like PyPI in general.

So I mean, there's test PyPI, but there's really no way to promote from test PyPI

to PyPI PyPI. But with Pulp, you could roast through those stages like you said. And what are some of the future plans that you have for Pulp as you move to the version 3 release and then beyond that?

So 1 of the features that we haven't talked about, which

is not technically a part of Pulp core, but it is part of the Pulp project is

what we call lazy sync. We'll definitely be getting that

going at or before the GA, and what that is is it's a deferred download system.

So for PyPI, for instance, you could sync all of the metadata from PyPI, but not download the bits

until they're requested from a user. And the reason for that is obvious. You save a lot of disk space, you save a lot of bandwidth, and

you can can still sync the whole thing. And in that case, it's used as more of a read through cache without having to reach out all the way to PyPI to determine whether or not a package exists. But then if you actually do want to install it, you wait until that point to actually pull the package so that you, as you said, can lazily evaluate it and only

store the packages that you're actually using.

Exactly. And so we we also have a streamer that's a part of this. So pip doesn't need to know anything about this. Pip will retrieve the bits from Pulp and have no idea that it it came from somewhere,

from, like, PyPI.

Yeah. It's worth noting that that's already a feature in Pulp 2. Just we're bringing we're going to be bringing Pulp 3. And that'll be available for all of the plugins. For Pulp 2, it's only available for Pulp RPM. Yeah. That's true. And for Pulp Python in particular,

I really

want users to be able to Twine upload to Pulp and then I want users to be able to Pulp export to PyPI. So like once they've gone through like the testing life cycles,

the, UI life cycle, like, all that stuff, like, they should be able to push it to production in PyPI. So I I definitely want that. 1 of my personal goals is to make Pulp

really, really easy to use and really, really easy to develop and especially to develop new plug ins. And so I think we've got very good documentation today, but I expect it to continue to improve

and just become more and more user friendly because we know that the best way to build a community is to have good docs.

And are there any particular

areas of contribution

that would be most helpful from the community as you continue to build and develop the version 3 release? Well, the the easiest way to get started is just to be a Pulp user, and that's

a huge thing for us. If you're using Pulp and filing bugs,

asking questions, or helping anyone else on free note or on the mailing lists,

these things are all very valuable contributions.

And if you find something that doesn't fit your needs, that's even more valuable. We'd love for community members to add features. We're always accepting pull requests. We we actually kind of go out of our way to accept pull requests from the community. Sometimes someone will implement a feature that we didn't really quite want, but since they did it and they need it, why not? So we we take a lot of pull requests from the community. We also

really want to encourage,

plugin development.

And so if there is a content type that Pulp doesn't cover, I think it's actually very simple to write a plugin, and I think we've got very good docs to support that. And are there any other aspects of the project or anything related that we didn't cover yet that we should discuss before we start to close out the show? I do wanna say something about why

I see hope is kind of amazing and, like,

why,

why it's just really great. And this is paraphrased from 1 of my coworkers who's very, very eloquent.

So software is becoming more and more complicated. Right? There's more dependencies. There's more dependencies from a lot of places.

And these dependencies are always changing.

And what people need is so there is definitely a need for a repository manager, and that need is currently being fulfilled by either closed source software or software that only does 1 content type. And our goal in the long run is for everyone to have access to free and open services

that's necessary for the creation itself. Great. Well, for anybody who wants to get in touch with either of you and follow the work that you're doing with Pulp, I'll have you each add your preferred contact information to the show notes.

And with that, I'll move us into the picks. And this week, I'm going to choose the book Soonish,

which is a very humorous and informative look at some of the various

areas of science that are seeing a lot of exciting developments that may or might not end up panning out to produce,

very valuable results for us. It's written by a comic artist and his wife who is the

scientist, and they interviewed a lot of the people who are doing this research.

So there are a lot of humorous comics interspersed throughout, and it's a very enjoyable read. So for

definitely recommend that. And so with that, I'll pass it to you, Austin. Do you have any picks this week? I do. But before I tell you about it, I have to tell you that I am wearing a science t shirt that I got from a Kickstarter

from the same author as Soonish.

So for my pick,

I've chosen the Shostakovich

string quartet number 8. Emerson quartet has a very nice recording of that. It's

classical music, but not the kind of classical music that you might fall asleep to. Alright. And, Bi Han, do you have any pics this week?

I have a YouTube channel that I'm

watching a lot of. This is the AOPA. It's the aircraft owner, pilot something.

They have, a YouTube channel called Air Safety Institute,

and on it they have a bunch of

accident case studies.

So I'm getting or I'm going to solo for my private pilot licenses Sunday, wasn't permitting.

And I literally cannot stop watching people get into accidents on airplanes.

But it's a really good series and, like, they analyze the cause and effects and, like, what people did wrong very, very well. Alright. Well, thank you both for taking the time out of your day to join me and discuss the work that you're doing with Pulp. It's definitely a very interesting project

and 1 that I am planning on using for my own infrastructure

and most likely developing some plug ins for. So

I appreciate the work you're doing on that. I'm excited to see how the version 3 progresses, and I hope you enjoy the rest of your day. Thanks for having us. Yeah. Thanks.

The Python Podcast.init

Summary

Preface

Interview

Keep In Touch

Picks

Links

The Python Podcast.__init__