Sandstorm.io with Asheesh Laroia

Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable.

Linode is sponsoring us this week. Check them out at linode.com/podcastinit

and get a $20 credit to try out their fast and reliable Linux virtual servers for your next project.

We are also sponsored by Rollbar. Rollbar is a service for tracking and aggregating your application errors so so that you can find and fix the bugs in your application before your users notice they exist.

Use the link rollbar.com/podcastinit

to get 90 days and 300, 000 errors tracked for free on their bootstrap plan.

Hired has also returned as a sponsor this week. If you're looking for a job as a developer or designer, then Hired will bring the opportunities to you. Sign up athired.com/podcastinit

to double your signing bonus.

You can also visit our site to subscribe to our show, sign up for our newsletter, read the show notes, and get in touch at pythonpodcast

dotcom.

To help other people find the show, you can leave a review on iTunes or Google Play Music and tell your friends and coworkers.

You can also join our discourse community at discourse.pythonpodcast.com

to find out about upcoming guests, suggest quote questions, and propose show ideas.

I'd also like to mention that the organizers of PyCon Zimbabwe are looking to the global Python community for help in supporting their event. If you'd like to donate, the link will be in the show notes.

Your host as usual are Tobias Mason Chris Patti. And today, we're interviewing Ashish LaRoya about sandstorm. Io, a project that is trying to make self hosted applications easy and secure for everyone.

So, Ashish, could you please introduce yourself?

Yeah. Hi. My name is Ashish Laroya.

And, can you tell us a bit about your background and how you got introduced to Python?

Yeah. I'll start with how I got introduced to Python.

I believe that I was using Netscape 4.78

on,

Mandrake Linux desktop in my bedroom in my parents house in high school.

And somewhere, through some of the links

that I found, was an article by Eric s Raymond who was writing about

how Python helped him

write this program called fetch mail very easily, and I thought, that's interesting.

And then I

searched some more, probably on hot bot because Google wasn't very popular yet, and I found this programming language called Python. And it was a it was really eye opening for me because

I remember

I remember when I was in middle school I guess I'm doing that thing where people who've been programming for a long time

make other people feel small for not having programmed for a long time. So let me just say, I just enjoy thinking about when I was, like, 12. It's not that you had to start programming when you were 12. Far from it. Yeah. So I remember when I was in middle school, I thought c plus plus was the programming language that 1 day when I'm a real programmer, I will learn and do work in, and then finally I'll be, like, a real

programmer. And then I learned that interesting people write interesting software in other languages too. And this was pretty reassuring because writing anything in C plus plus took forever.

So later on, for my high school newspaper my senior year,

I got elected as the web editor.

And

I decided that I wanted to make the process of turning the Adobe Pagemaker based high school newspaper

into a website much faster. Back in Wackenhof, in high school as a senior, the way that it worked was that

somebody would carefully painstakingly copy and paste

all of the

articles

into a web editing program, and I thought this is exactly what a programming language like Python is for. So

what I do is I wrote a small

program called pm, numeral 2, HTML,

which converts from PageMaker to HTML. And it does so by this long, drawn out process that involves asking you lots of questions like, hey. I think this is the continuation of that article that we found on page 1. Is this right? And then it'll stitch them together automatically.

And then if it doesn't crash by the time it finishes, it outputs a nice directory of HTML files.

And I learned an enormous amount

struggling,

clawing my way through writing this program over the course of my senior year.

But I can definitely say that if it were in a program language that weren't Python

at the time it would have taken me a lot longer.

And I did manage to finish it well enough that I put it on source forge. So,

that's my history with Python anyway.

And can you start off by telling everybody what the Sandstorm project is and how you got involved with it? From a user perspective,

Sandstorm is an online productivity suite that is self hostable, open source, and focused on security.

And the way that it achieves that

is by, under the hood, being a package manager for web applications that puts individual users in charge rather than putting admins in charge. So the short answer is I got invited to the right party. About 2 years ago, I was I was at an event that I was that was a casual Sunday night get together of free software fans in the San Francisco Bay Area that we called Bill of Cluster,

And

Ken Kenton Varda and Jade Wong showed up and began telling me about this thing called Sandstorm, which they were working on. And they asked me for feedback. They knew that I cared a lot about server self hosting.

And they kept coming back, and they eventually invited me to

a hackathon at their house

where

we would all work on packaging apps for Sandstorm. And I wasn't super convinced yet that Sandstorm was something that I would find useful. To me it seemed like I know how to run a server,

I don't need this like cool easiness thing, but sure I'll come because you invited me so nicely. And while I was there I figured out that Sandstorm isn't just about making running a server easy, it's about letting

anyone choose

the web software

that they run for themselves, just as easily as you choose a software you run on your phone or on your own personal computer. And with a security story, that means that that choice is actually safe no matter what you end up choosing.

Yeah. I remember when I first came across the Sandstorm project, I had a similar sentiment where I thought to myself, I already know how to set things up on a server, so it didn't seem terribly interesting to me. But as the project continued to

pick up steam and add more capabilities,

it made me keep coming back and taking a second look.

So can you describe some of the reasons that an individual would want to self host their own applications rather than using some of the comparable services that are available through third parties?

So the short answer is control.

And that means a few things. It means, the user gets to choose what apps they use.

It means the apps don't go away.

And, if the app is running in Sandstorm, it means security also. So,

in terms of choice Well, so in terms of apps going away, 2 years ago,

1 of the stories that showed up on the Sandstorm blog is the story of Google Reader going away.

And

this really struck me because

I've certainly used websites and web services that go away. And I live in San Francisco,

and in San Francisco

well, so there's a Tumblr, I think it is. I think there's a Tumblr blog called Our Incredible Journey.

And Our Incredible Journey is a blog that contains the going away final blog posts of companies

that make products and services online

when they decide to do something else, for example, typically when they get acquired. And so the joke about Incredible Journey is that however much you think they cared about you,

they were much more excited about getting acquired

and that's 1 sort of sarcastic take on

on why services go away in the case of Google Reader it's not that

Google got acquired or something like that. The story of Google Reader is Google

couldn't figure out

so I I don't know the internal story of Google Reader actually,

but it seems to me as an outsider that what happened is that

Google probably

there was no 1 in management in particular

who anyone at Google could figure out who should be responsible for Google Reader

they have so many of so much other stuff to do these managers have so much other stuff to spend time on and so no 1 could figure out where it should live and so in the end being homeless it went away and this is just my guess as someone who's worked at organizations

if somebody though has worked at Google I know this story I hope you will please tell us but

but it's just such a funny thing that

you're using people including me sometimes are using these services at the whims of the nice people or company that choose to offer them to us at whatever price they think including free

and so there's no real

obligation

for that to continue and in fact it sometimes doesn't

so,

so 1 reason to sell post applications

is

just that

other people's whims won't disrupt your own habits your own workflow

Another reason is the ability to really carefully pick and configure what you want,

So, well, let me come back to that maybe. I mean, the aspect that drew me to self hosting in the beginning, when I had a server in my parents basement, I guess it was 2 things. First of all, it

was the ability to provide

myself and people that I cared about with a service that I could feel proud that I was providing.

And

another is that I wasn't limited by any sort of arbitrary limits of any other third party service.

So the first thing I did with the server at home is I set up Internet sharing in a time before

wireless routers were a thing you could to the store and buy, and they would just share the Internet at home.

So you don't need Samsung to do that, but

the second thing I did was I set up file access so

that I could access the files that I stored at home while I was at high school and even though your listeners

might not be at high school, you might have some similar issues where you have some files of yours that you want to be able to access on multiple computers

and you can use services like Dropbox

or Box or a long list of online hosted services that do that. And when you do, you'll possibly run into limitations like storage space even though you have a plenty good computer somewhere else

and

so it's nice to feel free

though there's no arbitrary limits being put on that another aspect is privacy the information that I store on a computer that I control is just between me and that computer and I think this is where

these issues start to become interesting to organizations as well There's an investigative journalism organization called the North Star Post run by Jason Hernandez

and the North Star Post does investigative journalism on the use of surveillance technologies by law enforcement.

So they really

want to be pretty sure that they're the only ones

reading their drafts of their writing until they're ready to publish them. Of course when they're ready to publish them they want everyone to be reading those drafts, they want everyone to read those final work, but

they wanna make sure they're keeping their sources safe. They wanna make sure that they have the sort of freedom to

research the material they have and understand it as well as possible. And so, for that reason, the North Star Post runs Sandstorm on a computer owned by Jason Hernandez, safe in a physical place that he controls,

and no 1 else can interfere with

what he's doing and what his colleagues are doing on that computer.

Yeah. So in summary, I guess, control can mean the choice of what apps you use. It can mean

those apps not going away, and it can mean security and privacy also.

And how does Sandstorm try to make the experience of hosting the various applications simple and enjoyable for the broadest variety of people?

The experience of hosting an application yourself on a on sort of a normal hosting provider

is typically that you find the files needed for the app. You copy them to that server somehow, you somehow make sure all the dependencies are on that server,

and then you start the app and you go through some lengthy setup process so that it knows where to find its database, then then you log in to the app, and then you're done, and now you can do things. We just help people skip most of those steps. So all the dependency stuff and all the service finding stuff, you know, finding the app's database, that's all done at the time the app is packaged.

So when you click inside Sandstorm to install an app, what you get is an app that's ready to use immediately. It's ready for you to log in. So you click on make a new Grain,

or make a new document, or make a new chat system, or whatever kind thing that the app provides. And what you find is that you're automatically logged in, and you're ready to use it. That's because Sandstorm

handles

sign on for all apps,

so the app knows who you are. And because you're the 1 who clicked to create the new instance of that app, then

it stands to reason that you should be allowed to use it. If somebody else tries to visit it, they aren't able to use it. And this relates to the sandboxing and access control that we do.

But when we talk about the broadest variety of people making these apps enjoyable for the broadest variety of people, that means people who don't know much about technology. That means

people who insist on total privacy. That means organizations that are anywhere in this range. And

the the vision is that it should be

just as easy to use as any other online service so that anyone who doesn't know much about technology

can click and use it.

And, yet, if you include the broadest variety of people, people all over the privacy spectrum, they can enjoy the very same maps too.

And you asked how we

make that experience

possible. I mean, I guess we can talk more about the specifics of of how the sandboxing works. But self hosting plus sandboxing,

plus the fact that everyone's using the apps, sort of whether or not they care about privacy,

means that any improvements to the apps help everyone,

not just those who care about privacy.

And can you dig a little bit into the system architecture for Sandstorm?

Yeah. Sandstorm is a program that runs on a Linux server,

and it's actually a collection of programs. At the bottom is what I'll call at the bottom, there's a sort of core sandstorm layer, which is written in c plus plus And this core sandstorm layer speaks to the Linux kernel,

in terms of making sure the containerizing

system calls get executed correctly, making sure that new apps can launch. I'll come back to that. So there's this bottom it all bottoms out with this this core Sandstorm layer. And it's just that and your Linux kernel and the dependencies that it bundles. So it doesn't need other things from your system. And so that as that core Sandstorm element runs containerizes itself just before it runs. And when I say containerizes,

I mean, it does so by calling the Linux kernel assist calls, because we don't assume you have Docker, so we just speak Linux to Linux. So that's the very bottom. Then there's,

Meteor JavaScript app that is the user interface for people clicking around inside Sandstorm, doing things like listing what documents do I own, how much space do they take,

granting access to those.

That's an HTTP

web service that is written, as I said, using Node. Js, using this

JavaScript framework called Meteor, which has the cool fact that

you can easily you can there's a straightforward way to share code between the client and the server. That speaks to the Sandstorm back end that I described over a UNIX socket. And I'll

or at least it speaks CAPM proto. I should check that it speaks over a UNIX socket. Let's say for now it speaks over a UNIX socket. I'll double check. So the METEOR front end, what we call the shell in Sandstorm, speaks to the Sandstorm back end that I mentioned before over this remote procedure call system called capnproto.

That Meteor system is itself the thing that buy that listens on HTTP or HTTPS, so you don't need another web server like Nginx on top of all this. So we've talked about a shell and the Sandstorm back end. When you click around

inside

that Sandstorm shell, eventually you might find yourself opening an Etherpad document. And at the point where you click on that, the shell asks the Sandstorm back end to please run the processes,

run all the code for Etherpad

in a context where that code is confined and only the data for this 1 particular document is available. And so it does that by using find mounts and mount namespaces,

which is, and then some other containerizing features that are, again, core Linux features.

And now when you

additionally, your personal session to this Etherpad document

is assigned a unique

temporary subdomain of your server.

And to avoid that being ugly and showing up at the interface, we show the app inside an iframe. So what you see at the top of the address bar is your server slash grain slash grain ID, and

the document lives inside an iframe below that that seamlessly shows you the actual HTTP

response from the iframe. 1 other aspect worth knowing, though, is that that Grain, as I mentioned, it runs totally isolated from the network. And this means that it doesn't even receive the direct HTTP request that you make. That request gets

filtered by

that Sandstorm back end and turned into,

captain proto message

inside the Grain, which is basically pre parsed

HTTP. So that if there's some HTTP parsing related security issues, those are neutralized before the request ever gets to the Grain. And when I say Grain here, I mean 1 particular unit of particular application. So in Etherpad, a grain will represent a document.

In something like RocketChat,

which is a web based chat system, it's the whole chat system, there's multiple chat rooms inside that rocket chat grain.

So we've talked about the Sandstorm back end, the shell, the launching of a grain, and it's being containerized as well, and this communication protocol so that you don't speak directly to the grain, but all network traffic gets parsed and delivered to the grain by an intermediate process. I think that's the core of the architecture.

So I noticed that, Sandstorm requires a very, very recent, version of the Linux kernel. What motivated that choice and how does it affect adoption?

So Samsung does require a sort of recent Linux kernel version. It requires Linux kernel version 3.13,

which was released on January 19, 2014.

That's a 2 and a half year old version at this point.

It was really first released in the Ubuntu distribution with Ubuntu 14 0 4, but this is somewhat new. The reason we need this is that for that security isolation, transform requires

kernel features to exist to do that. So 1 of the main feature that we need to work great is username spaces, which is a way for a program to run as a different virtual

user ID than the user ID that started it. So we always use this in an unprivileged sense. We map whatever user ID Sandstorm is running as into a generic user ID 1000

in the Grain. So the Grain can't perceive any difference between

running inside

running on different people's computers. So so I don't know. Let me push back. Is 2 and a half years old really very new, Chris? Maybe it is. I don't know. I'm curious for your take.

I agree that that's not very new. I have a a VPS that I run,

and I noticed that

it it wasn't supported. I think it might be CentOS 7 or something like that. Okay.

So maybe that's not new enough. But in any case,

I I guess, I had thought that that version was

capable of running containers, and so I I I wasn't aware of

what new features you folks might have been leveraging that would require something newer. But in any case, what you said makes perfect sense.

Yeah. And

the CentOS is actually really interesting.

CentOS or RHEL 7 is actually a really interesting platform to be targeting. They do they did recently ship a new kernel version in

RHEL and, therefore, CentOS 7.2,

where

RHEL 7 has been based on 3.10 kernel for a long time, and usernamespaces

were first released in Linux 3.8.

But they've stabilized over time. And so, actually, Red Hat didn't enable usernamespaces in the original

rel 7 kernel. But they did in a more recent build that's about 6 months to a year old, and I did actually attempt to experiment with

that particular rel 7.2

kernel that has user name spaces enabled once we figure out the problems with the other Sys calls we're using. Sorry to be a little vague. But we haven't dialed that back yet.

That's that's quite alright, and it's completely understandable. You're right. CentOS does tend to be extremely conservative

in terms of the kernel updates that they take. And I think that makes perfect sense. I would presume that a CentOS 8 is in the works, and that probably has all the right

kernel nubbins. So

I certainly appreciate why you folks made the choice that you did, and I'm sure going forward, people will have

necessary

operating system shortly.

Yeah. But you you bring up a good point about the the question

of if this affects uptake. And we certainly have seen people report to us that they want to run Sandstorm on either RHEL 7.2 systems or,

more interestingly,

every once in a while, somebody wants to run Sandstorm inside another containerization system like open v z or v server. That's because those systems can be used to run a really cost effective hosting provider. There's a lot of hosting providers like using OpenVZ or Vserver. But that ends up typically meaning that Sandstorm can't and in fact, openvz and vserver at least openvz,

I believe,

does use usernamespaces. I believe Openvz is actually the origin of that feature, that usernamespaces and the other Linux namespaces are kind of the Openvz code base linearized

slowly into patches into the Linux kernel over the course of years years until finally we can do the same things that OpenVz did but with a little bit of new architecture. So, anyway, so I believe that openvz uses usernamespaces,

and when Sandstorm wants to use it to protect the user from apps and to ice to make sure that apps can't perceive any difference between person 1's computer and person 2,

it has the problem that usernamespaces

have been disabled for everyone except the admin, typically.

But Sandstorm wants to

switch to a non root user ID and then use this new username space feature, which results in this conflict. And in fact, when I say with the admin in the case of OpenVZ, I mean sort of the person selling the access to the servers, not even based on your user ID.

And 2, I imagine that given the fact that

I imagine most people who are using Sandstorm are going to be availing themselves of multiple

of the different applications that can be installed there. And so performance capabilities would likely be a problem on those kinds of hosting providers as well.

Yeah. It's possible. I think it's also totally possible

to have

a to spin it the other way around. Things like OpenVZ and Vserver can actually help providers offer a higher performance service because don't have to preallocate the amount of RAM to each user like you do in virtual machines.

So you could give each user, I don't know, 2 gigs of RAM instead of 1 because

most of the time not all the people on the machine are using all 2 gigs, and then you'll have better performance,

except you still can't use sandstorm because it doesn't run inside of a d v. I just like to I'm just charitable to the open vz dz situation because the nice people who file bugs against Sandstorm who would like it to work, like, it's not their fault. Sure. Yeah. I used OpenVz a while ago, and it's definitely

a technology that was ahead of its time

and has somewhat faded from recent memory because of the recent advent of Docker. But it definitely pioneered a lot of different capabilities that have proven to be valuable to quite a number of people.

When you look at what Sandstorm

does,

perhaps through the the eyes of an of a somewhat more naive end user, you're not thinking in terms of the awesome,

security

architecture that Sandstorm provides

and what that might require in terms of kernel

niceties.

You're thinking in terms of, oh, you know, hey. I have hosting. I wanna run this thing, make it go kind of thing. So I can see where people might be confused,

about that. Yeah. I agree with that. And, you know, a few months ago, our website used to say, Sandstorm is an open source operating system for personal and private clouds. And now Sandstorm, the website says, Sandstorm is a self hostable web productivity suite.

And you would never particularly think that a web productivity suite

will be, like, calling Linux's calls directly. You'll imagine that it'll be less intense. But and so so the more that we try to help people realize what Sandstorm is useful for, I think the less that we end up giving them a sense of

the the fascinating security architecture. But 1 other thing about adoption, most of the users of Sandstorm, in my experience, seem to

spin up a new, like, virtual machine on a provider like EC 2 or DigitalOcean or Linode

or install it on a computer that they own personally.

And so to a large extent, I think that this hasn't massively hampered adoption. Although, I do think that the RHEL 7.2 issue is pretty important, and we'll be looking into it again soon.

As you mentioned, the

security model of Sandstorm is rather notable. I'm wondering if you can explain a bit about the capability based authorization model and how that enables Sandstorm to ensure,

good amount of privacy for the end users.

Yes. You talked about the capability based authorization

thing,

and I wanna zoom out for a second away from this word capability and talk about what it means for end users and then talk about the how. So I mentioned that if you click and launch

a new document within etherpad inside sandstorm,

it's automatically logged you in. And that's because

Sandstorm, considered in the middle between you and Etherpad,

it can know who you are, and it can check that you're supposed to be looking at that particular document. And then it can go tell Etherpad

who you are as by adding headers to the HTTP request. And Etherpad,

the ether the Etherpad code base that lives inside the Sandstorm package, it knows that whenever an HTTP request comes in, it's definitely, definitely, definitely coming from Sandstorm.

So if Sandstorm says who the user is, then

the app ought to pay attention to that. And

this idea that if all the pieces are connected, the knowledge of what it's allowed can flow through smoothly

is, I think, the real core

aspect of how our Sandstorm single sign on works, how the sharing model works, and really how capability security works in general. So, Tobias, we had chatted a little before this, and you told me that you had gone to Drew Fisher's Pycom talk, which explains

why you're asking me about capability security. It's a concept in security

that is the idea of a a capability

in security, is the concept of a communicable,

unforgeable

token of authority. So a communicable, unforgeable token of authority, that sounds a lot like an API key.

And there's some other elements to it that are important, which are that it refers to a particular object in another system,

and it also

embeds

the access rights somehow

so that someone who holds this capability

can only use that remote object

according to the access rights that are granted.

But a different way to think about it is to start from isolation. So if you were to run some software in a world where it woke up and it could save as much data as it wanted, but there were no 1 around to talk to,

then you'd have some software that was isolated. Now interestingly, this means that it couldn't do things like go look at the list of processes on the system. It couldn't

go changing directory into

directories owned by other people or other programs,

it's totally isolated. All it sees is the stuff it makes itself. If you have an isolated program, you might find yourself wondering,

how can I give it the ability to do things with this other data that I have?

And 1 way to do that is what we call object capabilities.

So I mentioned that it's kind of like an API key. It doesn't have to be this sort of dry dead bit of data that an API key is, because if you can if the 2 if this isolated program 1 and this isolated program 2 are connected,

imagine in your head there's a line between them, and you imagine that whatever drew that line, it embeds the permissions that are part of that line, and it is is to a particular remote object in a particular other program, then the, you know, then the first part can call methods on that object on the right hand side. And that's sort of really what how capabilities

how I like to think about capabilities, which is that they are these these safe connecting lines

between things where

somebody has made sure that the connecting lines have permissions associated with them, and that they grant access to a very specific other thing. That was like,

a 1 hour lecture in computer science in an operating systems class that I attempted to digest into,

a 5 minute discussion on a podcast. So it's a lot to keep in one's head. But this is a pretty familiar concept in a lot of ways. So Google Docs, for example

if you use Google Docs, you can make a sharing link to a document. You can give that sharing link to a friend over instant messaging, and then your friend clicks that link. The thing that happens is that Google Docs checks that the link is authentically really from Google Docs, and then it gives your friend the ability to view the particular

object, in this case, document,

with a particular set of permissions that you made when creating that sharing link. And so because Sandstorm sits in front of all these apps,

Sandstorm can apply that kind of sharing model to any app even if it doesn't have particular support for having a sharing dialog box. Sandstorm makes the sharing dialogue box. Sandstorm can check that token, and then it can route the user to that 1 particular grain. So

that's definitely

something. There's some other aspects of capabilities we use inside of Sandstorm in terms of getting access to

to privilege resources, like the ability to really listen on the network, but

I think that's a separate a separate question to answer.

And what are some of the most difficult challenges facing your you and your team in terms of the software architecture and design of Sandstorm, or even in terms of additional features that you're trying to enable? A big challenge for us in terms of software architecture and design is making sure that our security model stays easy to use

and keeps the security properties we want. And even achieving isolation

that I described before is actually really tough in the web environment,

And it actually takes a lot of work on the server side too. So, you know, we talked about user usernamespaces

before, but we wanna protect users not just from from apps trying to reach other's data in the normal ways, but we also, where possible, wanna try to protect users

from apps that exploit security vulnerabilities in the the Linux kernel. So we use a system called setcomp

with a filter set designed by Andy Budimirski

to make sure that works. And that filter set has to get threaded through this this back end

model that I described, and yet we have to make all this be joyously effortless.

And I think we've largely succeeded by making this sharing dialog box that David Renshaw pioneered last year, but there's more to do in terms of connecting the apps to each other and making that still easy to use

and still just as safe. I mentioned that we take this isolation aspect to a bit of an extreme. And at the end of the day, we want the answer to be yes when somebody asks, is each instance of each app isolated from each other? Android and iOS have done a lot of great work in terms of building a really good interface for a lot of similar concepts in security. They don't take the isolation necessarily as far. In Android, for example, they have to do get full network access by default, and there's no way to turn that off. But we're trying to follow in their footsteps in terms

of some of the security architecture

and in terms of thinking through how to expose that to people. So that's 1 architectural

difficulty we face. Another 1 is building development tools that make sense to software developers.

So so I told you that the Grain earlier in the discussion of architecture, I told you that each each HTTP request comes in already parsed,

parsed by an external bit of software.

But there's no Python Django integration for our particular binary wire format for HTTP. So what we end up doing is we have a little bit of software that runs inside the Grain that decodes this binary thing and spits out text HTTP. So we can pass that to uwhiskey, so we can pass that to Django so that,

people can have their issue few responses. So in the development tools, we try to paper over some of these implementation details. And Drew Fisher and I, a little over a year ago now, wrote a sort of new Sandstorm software development kit called vagrant dash spk,

where we wanna give people a Sandstorm app development environment

that works

on Windows, Mac, and Linux because we know we need to meet people where they are and not tell them to just install Linux. And so getting the dev tools to work in a way that is easy for developers to understand and figuring out which aspects to pay for Rover, which asks at what point we need to tell app authors about the different aspects of the platform. Getting that right is an ongoing challenge.

For people who are looking to install Sandstorm on their own servers, what's involved in getting that set up, and what kind of

are some of the different use cases particularly in terms of different scales?

You you can't see, but I'm grinning ear to ear because I maintain the install script, and,

the install script

runs on any Linux computer with a suitable Linux kernel version, for example, a Linux virtual machine with Ubuntu 1404 or 1604.

And,

it asks you a couple of questions. And it also asks you if it's okay if you get a free from us subdomain of sandcats dot I o. And you say, sure. I want example dot sandcast dot I o. And then it asks you, do you want free HTTPS certs with a wildcard cert so that you can access all these 1 time use subdomains that we create? And you say, sure. I don't know. Yes. Great. I just wanna go forward with this thing, Ashish. And then you wait about 20 seconds,

and then your Sandstorm server is ready. It downloads a compiled bundle from us and checks its signature and installs it, and the thing self containerizes

and starts via your

by creating a new Syspy init or system d service. So

it takes about, I don't know, 0 to 2 minutes to make a new VM somewhere. And there's a screencast of me, which I'll put in the show notes with y'all's permission.

The screencast of me going through that process over 30 seconds and then having a working server. So, that's what's involved. You have to download a script and run it, and you can optionally verify the script via GPG.

You can check the signature of the script against a key that is called the Sandstorm release signing key, which which you can check against my personal PGP key, which you can check against the devian keyring.

So if you need a trust path, that's the best way to get that.

I just have to say, when I realized that I I could not install Sandstorm on my VPS for the reasons we discussed previously,

I took a look at the install script and and I commend you, sir. There's 1

beautifully written piece of bash script and and,

I was impressed. It does certificate management and open SSL and HTTPS,

and just it it does an awful lot for a a bash script.

Yeah. Thank you.

The the install script is something that I it that became my thing at Sandstorm

when I kept having opinions about it, probably. Kenton Varda

initially wrote it, and he based it on the Meteor install script, I believe. And there's some links in there to resources that we found helpful. To go on a little bit about this install script, actually, 1 of the Python

tie ins for Sandstorm is that I wrote,

automated software testing tool

in Python to test this install script because I've never had to maintain

this much shell ever in my life. And I introduced I found out I was introducing bugs, and I understand that regression testing helps find out when this happens so you can avoid giving the buggy software to users. So there's a a collection of files, under the installer tests directory in the Sandstorm repo. Each 1 has a dot t

extension.

And these are

the the transcript of having a successful install or an unsuccessful install in some cases that we wanna make sure is successfully unsuccessful,

but we spin up a virtual machine

to run the install script with the right arguments inside the right virtual machine with the right kernel version, which we test

at least daily that we make sure that nothing we've done breaks the install process.

So, sure, it looks like it's a well written bash script, but I assure you that a

if you introduced a bug, it would still look like a well written bash script. So it's nice that we have this test suite. And, anyway, thank you. I appreciate the the compliment, in all seriousness.

Yeah. I can definitely imagine that having a pretty robust set of tests for the installation process is pretty valuable, particularly given the fact that a lot of the target audience is people who are generally non technical and are just looking for something that they can, you know, fire and forget and just have it work.

Yeah. And I think along those lines,

the main thing that Sandstorm

adds to the world is that it

offers people the kind of privacy that self hosting provides

without asking from them very much at all. And so that means that people who don't have the technical skills to carefully install a server

and think hard about securing it and think hard about limiting what apps they use can still get those privacy benefits. And that's true not just for for people, but for

organizations.

And

so there are a large number of different applications that are available for the end users to install once they do have their Sandstorm server up and running. I'm wondering what's involved in making a project compatible with the Sandstorm runtime environment. And are there any limitations in terms of the languages or application architectures

for the people who are targeting your platform?

Okay. So let me answer those in reverse order so I think I can remember them. First of all, are there any limitations in languages or app architecture?

And the answer is that any code that runs on Linux

can be made to work on Sandstorm. The way that you make a

project compatible with Sandstorm is you download our dev tools called the Vagrant SDK that I mentioned that Drew Fisher and I co maintain.

And this thing is in Python, by the way.

And you run it in the directory that your app lives in, and it makes a little dot sandstorm directory

and then tells you, okay. When you run vagrant spk

vm

up, what will happen is that

you will get a fresh Linux virtual machine that runs your app that also has Sandstorm inside and

where the current working directory is mounted back into the virtual machine using Vagrant and VirtualBox file sharing. It does work with libroot, by the way, for those who prefer LibriVirt over virtual box. And so

once you've done that, you can run a command to launch your app inside of Sandstorm, and then what you need to do is click around, make sure that your app works, and then stop this process by doing control c. And what that will do is save a list of all the files that your app accessed,

and then you can run the vagrant spkpack

command

to pack up all those files into, app package along with your app's code, and that's the core of it. So there are a few wrinkles as in anything else that sounds easy. 1 of them is that since I mentioned that the packaging tools watch your app as it runs to determine what files it needs, this does have this bizarre property that you need to exercise all the code paths of your app, maybe, which is a lot to ask. And so what we recommend doing instead is using this always include directive inside the sandstorm packaging

file that is part of this dot sandstorm directory

to tell the app packaging system about directories that definitely need to be included. And this should definitely include the Python virtual end that your app uses,

so then all of your Python dependencies are definitely inside the app package. There's another wrinkle, which is that because of the sandboxing we like to do, we don't make

slash proc available, which is a Linux feature, to the app. And so some

some programming languages,

some interpreters find this a little bit shocking and appalling and then crash on startup.

But if it's Java, we have it but we have a document written up for how to make Java find itself so that it can actually launch itself.

And

for other any other language runtimes, if you find that they fail to launch, then I'm sure that we can help you overcome those those obstacles.

Those obstacles used to be pretty common about 2 years ago, but the introduction of ignorant

SDK occurred sort of for the purpose of embedding what we knew, what we learned about running apps inside the Sandstorm sandbox

into an executable tool so people could

just have all the knowledge

executed

rather than having it in their brains.

And are there any limitations in terms of,

for instance,

requiring

certain additional dependencies, like a particular

kind of database like whether it needs a, you know, elastic search instance or postgres or some of the other different data persistence technologies that somebody might use,

thinking particularly in terms of when an application might require some measure of distributed

system as a supporting

service?

Let me answer this question by saying

no and yes. No. There's no limits on the code that you wanna bundle into your little app package. If you wanna bring Elasticsearch, great. Bring a little Elasticsearch.

Modify a file in dot sandstorm so the app can install Elasticsearch search, or you get it however you like, and then make sure you launch it in your dot sandstorm launcher script.

And then we'll make sure we find it in the tracing, and

there you go. You've got an Elasticsearch.

The the question about database technologies,

I think, comes from the architecture of applications

where

somebody at the company is the database person, and so they give you, the application person, a database.

And then you have to carefully put all of the different data the application needs for all of your different users, for all the different documents your application stores,

all inside that 1 database connection because the nice person who gives you database connections will only give you 1.

But in Sandstorm,

you can have as many as you want. You can just click the new the new Grain button, which launches a new instance of the app, and you're responsible

for starting

your code is responsible for starting your code is responsible for starting any of the services you need, so you get as many connections as you print for yourself. But this does relate negatively to that distributed systems question you mentioned. Because these apps run totally isolated, if your

vision is to build an app that when it's launched

connects to a global decentralized

bookmark sharing cloud

shared across all the people running your own bookmark sharing program,

like this guy, Andrew Wonsley, who's working on a Sandstorm app called Mark, then you will discover that your app runs sandboxed and can't reach any of its friends that are part of the bookmark sharing collective. So to make that work,

the best thing you can do

is to request network access from the user,

which at the moment will show

a

totally version 0

tracer interface that allows them to say yes if they're an admin

and just doesn't give them any results at all if they're a non admin.

And then the app can get permission to make outbound network calls. And then when it makes those outbound network calls, it kind of needs to do so using this Kapton Proto remote procedure call language rather than raw TCP against the Linux kernel. This gives us the ability to to route the

TCP traffic through Sandstorm itself before it hits the real network. So

the Sandstorm Kapton proto system, there are Python bindings for it, so what it amounts to is

you write a little bit of Python code that connects to a unix socket inside the grain and then

instead of calling socket dot connect, you call something akin

to sandstorm

dot sockets dot connect except it's a little longer than that, but that's basically the idea. There is a way that you can

work around that right now, which is a screen call that I should mention because the app's not truly, truly isolated yet because we did introduce a way for apps

to make outbound HTTP get requests. And so you can do that again using the same

Python tech and proto system.

You get a unique connection a unique socket connection

to a file inside the green, and then you do something akin to sandstorm.hdpgetopenpren

remote URL you wanna fetch. But that's only for the HTTP verb called get. And in the future, when you when your app calls that captain proto method, it'll create a permissions chooser, and this should be pretty familiar to people who use who develop apps for Android or iOS. In the long run, what we'd like to see is the introduction of a concept called drivers.

So right now and probably forever, only the server admin can grant this raw outbound network access

to an app because

it's a really dangerous thing. If we allowed any app to request it from any user, then the users would just say yes, and

all of the technical assurances

would be would go up in a puff of user excitement smoke.

And so what what we wanna do instead is to have specific drivers in Sandstorm that are regular old Sandstorm apps that can that do request network permission from the admin,

and then they themselves export

the ability to do other particular things

that are safe. So you can imagine a driver that gets the ability to connect anywhere on the network,

and what it offers inside to other Sandstorm apps is the ability to connect to a particular IRC service speaking the IRC protocol

where the person sitting at their laptop using the Grain

has to type the name of the IRC server into a box that is controlled by Sandstorm, not controlled by the app. And this kind of brings us back to capabilities, object capabilities thing. Because then the app can never accidentally or buggily or maliciously

connect to the wrong TCP port and speak the wrong protocol.

What it can do is get the particular connection to a particular

driver app where authorization has been granted to talk to a particular

RC server and port speak in the RC

protocol. And so that seems like it's safe enough to allow apps to request. And,

also, when users wanna find out what is this app doing, they can get an extremely clear answer,

clearer than ever before. Or they can say, oh, no. This app I must have given it IRC access a week ago. That's weird. X, and that's that. So,

the driver system doesn't fully exist yet. In particular, you don't have a way for these these apps to offer

capabilities to other apps within Sandstorm. But you can request

full outbound network access and full inbound network access on a particular TCP port, but only the server admin can say yes at the moment. Oh, and in terms of of this guy, Andrew Wonsley, who's writing this mark app, what Andrew Wonsley is going to do for now, as I understand it, is to speak HTTP get outbound because he can still for now without getting a permission chooser. And in the future, he'll still speak HTTP get outbound, but there'll just be a permission prompt for the end user. And to get traffic back into

your little distributed

social network bookmarking app, he'll ask the user to copy and paste a special API token that allows the app to receive traffic in. And this API token is like all these other capabilities type things that I mentioned where it's specific to this 1 app. It's specific to a particular permission level.

And you mentioned that there are a few different areas

of Sandstorm that take advantage of Python. I'm wondering if you can dig a bit deeper into

how much of the overall code base is written in Python

and what are the main responsibilities

for that usage.

Sandstorm, the user facing shell, is not at all written in in Python,

and the Sandstorm back end that I mentioned is not at all written in Python.

But

the unit tests for this install script are a new Python product that I created for this particular purpose. I was really hoping to find

that someone else had the crazy need that I had to spin up new virtual machines and

run text based programs and capture their output and then validate that against what they were expecting. I didn't find anything quite like that, so I made something. So that's 1 aspect where Python is used. Another way is that Python is used as the language of writing what you might call the Sandstorm

SDK,

which is a program called vagrant dash spk, spk being short for sandstorm package. And so this thing, in order to start your app inside a Linux virtual machine, no matter what operating system you run,

and to make sure that sandstorm is running inside that virtual machine,

it uses a tool called Vagrant. We chose Python in part because it was a programming language that Rufesh and I both knew, and we also chose Python because we

had a vision for how we could distribute it to Windows users. And so we're using a tool called pyinstaller at the moment so that there's a windows exe that people can double click and get an installer

and click next and then this thing is on their path and they can open up command prompt and run vakernetspk

And when they press enter,

Python from inside the vagrant spk installer tool

is the Python that interprets the vagrant spk Python script. So basically we picked Python because it's a language that's easy to use

and has a pretty good cross platform story.

And are there any other topics or questions that you think we should cover before we start to close out?

1 reason to self host that I didn't mention before is if you are creating something and you want to share it with others,

then if you're a developer, you're gonna have to host it somehow.

And running something on Heroku, or running something on EC2, or running something on Linode, or running something on a Python oriented

hosting service or running something on Sandstorm

these are all ways to

to give people your app but interestingly if you're the author of 1 of these apps

and you basically made a web app for yourself, you don't really want, necessarily,

to store other people's data.

You don't necessarily want to be on the hook and carry a

pager in case your service goes down when you're asleep. If you're asleep, you don't care the app is down because you don't need it. You're asleep.

But other people might if you're running a service for them. So

1 of the interesting things about self hosting

is not so much why you would self host, but it's why app authors benefit from self hosting, which is they can create an online interactive experience for other people

where the other people run their app. And in the long run, and even we're seeing this now, there's apps in Sandstorm that only exist because of the economics of self hosting, which are that the app author

distributes the app and then goes away for a while and isn't on the hook. So there's a there's a sort of drop off alternative

in Sandstorm called Davros,

And Michael Nutt wrote this thing specifically targeting Sandstorm

because he didn't wanna run an online service to host other people's data, but he was pretty happy to write some code that other people might use. So, I bring all that up because I was just talking about contact Otter by Philip James

who distributes contact Otter

to Sandstorm users so that there's an easy way for other people to use it who want more privacy

and then he doesn't have to run some kind of crazy high assurance data center for his hobby project. Contact author is written in Python, it's written in Django.

Maybe my favorite

Python app is an app by Daniel Kroll

which lets you

upload a Wikipedia backup

and then get an interactive offline capable view of that Wikipedia data, and so it uses another tool to display the Wikipedia data, but there's just a tiny Python app in front that

is there so that when you that there's something that you can upload

the Wikipedia backup. Well, the Wikipedia data export

too. And then Nginx checks, is there a Wikipedia export? And if so, then it runs QX, the tool that can actually render those. And there are a few other

apps in Python. Let me just, go through a few of them to make sure that people get a real sense that Python is a big part of the ecosystem. The first the the tool that I use for making slide presentations is an app called Hacker Slides, which takes markdown and renders it in the HTML, and that's a program written by Jack Singleton. It was, I think, the first app that just showed up out of nowhere in the community

where someone was like, hey. I wrote this thing. You don't really have any packaging tools, but you have this hard to use core tool called SPK,

and I made it work. Here's a Python app. PermaNote is another. RetiCal is a

a Python based contacts

and calendar management program that integrates with Sandstorm so that you can synchronize

your phone's calendar and contacts to a service you control so that no 1 else has access to your calendar

or so that you can create a shared calendar with friends with a sharing model that sandstorms

rather than requiring them to have a Google account, for example. And

Alexander Bogdanov

is the 1 who put that together. So those are my favorite transform apps in Python, but I'm sure there's a bunch

more, and I'm really happy for all of them.

So for anybody who wants

to follow what you're up to and keep in touch, what would be the best way for them to do that? So it depends on your style. It depends on and it depends on how much information you want per unit time.

So if you like reading Twitter, Twitter is actually a really good way to get updates about what we're working on.

You can also subscribe to the sandstorm blog, and you'll get just a little slower

version of a similar information stream. You'll hear about the important announcements. For even fewer announcements and to make sure you really read them, I personally recommend that you sign up for our email newsletter.

You can get to that by visiting sandstorm.

Io,

and scrolling to the bottom, and entering your email address in the form at the bottom. We try to keep that to a maximum of 2 posts per month, and we're usually closer to 1. I personally can be found by email at ashesh, that is

ashesh@sandstorm.

Io, and if you're really good at googling, I bet you can find my phone number, so feel free to call me there is 1 other Python app that

I I'm really excited about that I can't help but mention which is a media sharing app called GNU Media Goblin. It's

an effort that aims to integrate federation,

the idea that if I leave a comment on your photo gallery, I can do so with my user account that belongs to my own photo gallery and authenticate in a way that means that neither of us is running the central service that everyone uses.

Instead, we can each run little photo galleries and have the basic features we expect from a social group to keep working.

Yeah. There's a great Lost Weekly episode where they interviewed the creator of Media Goblin that I'll link to in the show notes. That's great. That's Chris Webber, and he's great. And, yeah, he and I have overlapped job wise. And so it's always great to put electronically or in person run into him.

And,

so with that, we'll take it on to the picks.

For my pick this week, I'm gonna choose the service Opsgenie,

which is

an operations alerting service,

along the same lines as Pingdom. And that's what I've been using for work. They're well put together service. They've got a good interface, a lot of really great integrations.

They're and they're easy to use, and they keep adding new capabilities while still keeping their pricing pretty sane. So, for anybody who works in the operation space or need some way of have managing an on call alerting,

rotation, I definitely recommend checking out Opsgenie. And with that, I'll pass it to you, Chris.

Thanks, Tobias. I have 3 this week, so I'll keep them quick. The first 1 is the Viking Godfather safety razor. I was inspired by your pick of the Viking Chieftain, and I gave,

safety razors a try, and I totally love it. It has been a revelation, and I actually really enjoy shaving for the first time pretty much ever. So thank you.

My next pick is a book by Paul Cornell. I'd recommended him before, but it's his latest in the Shadow Police series. If you like kind of dark urban fantasy, you really need to read these books. It's called Who Murdered Sherlock Holmes, and it's it's so far, it's excellent. I'm only, like, halfway through. My last pick is a beer.

It is the Petrus

aged red.

It is a Flanders red ale.

It's delicious. It's got notes of cherry and oak. It's just really rich, complex,

tasty dessert beer stuff. So I really enjoyed it.

Ashish, do you have any picks for us?

Yeah. So I have 3. The first 1 is Amtrak,

which is just to say that I have had a great time recently traveling around by rail. I recently

took the train from New York City to Rochester, New York. Rochester is where my parents live. It's actually where I'm recording this interview from because I'm visiting them and my grandmother. And I had just not looked out the window on that trip before I noticed how beautiful it is. It goes through all these beautiful,

nearly forest vistas, there's beautiful water running,

I really was having a bad day when I got on that train, and thought nothing could save it, and then I stood out the window and discovered the world's actually really nice. So that's Amtrak. Another is Tim Wu's book, The Master Switch. I've been reading this book recently about the history of telecommunication

systems

as they oscillate between close and open, and it talks about what's at stake in that that oscillation,

and it talks about

the specific

personal,

interpersonal,

economic, and political forces that

are part of those transitions

and I've learned a great deal about radio,

the history of television in the US, the history of movies in the US, and how other countries

have and haven't necessarily sometimes

had the same kind of transitions, so it's been really eye opening.

The third is

a sandstorm app called RocketChat.

You can use RocketChat even not on sandstorm, but I bring it up because it's just such a pleasure to use and is a huge part of my work day nowadays

I when I'm wondering what to do I log into chat and I ask my boss And,

if I want to embed a document within the sandstorm, I click this plus button, and

1 of these magic capabilities

choosers comes up. And well, I click the plus button, and I see a list of all the things I have access to, and I can click and embed permission to view that. It has emoji.

I upload photos. So that's rocket chat. And I I wish that I could tell you that I had some Viking things to recommend to you, but maybe not this time.

Well, we really appreciate you taking the time out of your day to join us and tell us more about the work you guys are doing at Sandstorm and how that ties into the Python ecosystem.

And, I definitely think that there's a lot of value in what you're doing, and I'm sure that a number of people are gonna start checking it out after they get done listening to this episode. So thank you, and I hope you enjoy the rest of your evening. Thanks. And I actually do wanna pause for 1 moment to say there are a couple of other Python apps that are really worth mentioning.

So for 1, there's IPython notebook, which many of you have seen. This is you click a button and you have a notebook, and you can share this notebook with Sam's store with build and access control. Anyone else on the server can make their own notebook,

which is totally isolated from yours, so even if there were an IPython notebook security bug, that wouldn't allow you to

read the data of your colleagues. So I I guess that's the most exciting 1 that I'll say. Alright. Well, thank you again, and I hope you enjoy the rest of your evening.

Yeah. Thanks so much. I really appreciate you both taking the time to do this interview. And if there's anything else you 2 need from me or you wanna try about anything else, I'd be happy to. This was a a real pleasure. Alright. Good night.

The Python Podcast.init

Summary

Brief Introduction

Interview with Asheesh Laroia

Keep In Touch

Picks

Links

The Python Podcast.__init__

Summary

Brief Introduction

Interview with Asheesh Laroia

Keep In Touch

Picks

Links

The Python Podcast.init