Summary
The breadth of use cases that Python supports, coupled with the level of productivity that it provides through its ease of use have contributed to the incredible popularity of the language. To explore the ways that it can contribute to the success of a young and growing startup two of the lead engineers at Wanderu discuss their experiences in this episode. Matt Warren, the technical operations lead, explains the ways that he is using Python to build and scale the infrastructure that Wanderu relies on, as well as the ways that he deploys and runs the various Python applications that power the business. Chris Kirkos, the lead software architect, describes how the original Django application has grown into a suite of microservices, where they have opted to use a different language and why, and how Python is still being used for critical business needs. This is a great conversation for understanding the business impact of the Python language and ecosystem.
Preface
- Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
- When you’re ready to launch your next app you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to scale up. Go to podcastinit.com/linode to get a $20 credit and launch a new server in under a minute.
- Visit the site to subscribe to the show, sign up for the newsletter, and read the show notes. And if you have any questions, comments, or suggestions I would love to hear them. You can reach me on Twitter at @Podcast__init__ or email hosts@podcastinit.com)
- To help other people find the show please leave a review on iTunes, or Google Play Music, tell your friends and co-workers, and share it on social media.
- Join the community in the new Zulip chat workspace at podcastinit.com/chat
- Your host as usual is Tobias Macey and today I’m interviewing Matt Warren and Chris Kirkos and about the ways that they are using Python at Wanderu
Interview
- Introductions
- How did you get introduced to Python?
- Can you start by describing what Wanderu does?
- How is the platform architected?
- What are the broad categories of problems that you are addressing with Python?
- What are the areas where you chose to use a different language or service?
- What ratio of new projects and features are implemented using Python?
- How much of that decision process is influenced by the fact that you already have so much pre-existing Python code?
- For the projects where you don’t choose Python, what are the reasons for going elsewhere?
- What are some of the limitations of Python that you have encountered while working at Wanderu?
- What are some of the places that you were surprised to find Python in use at Wanderu?
- What have you enjoyed most about working with Python?
- What are some of the sharp edges that you would like to see smoothed over in future versions of the language?
- What is the most challenging bug that you have dealt with at Wanderu that was attributable in some sense to the fact that the code was written in Python?
- If you were to start over today on any of the pieces of the Wanderu platform, are there any that you would write in a different language?
- Which libraries have been the most useful for your work at Wanderu?
- Which ones have caused you the most pain?
Keep In Touch
- Matt
- @matthewwwarren on Twitter
- Chris
Picks
Links
- Wanderu
- Northeastern University
- C++
- Perl
- Microservices
- PostgreSQL
- MongoDB
- Django
- Node.js
- Go-lang
- AWS
- ETL (Extract, Transform, and Load)
- Data Warehouse
- Graph Database
- Twisted
- Gevent
- Scrapy
- Virtualenv
- Ruby
- Rbenv
- Boto3
- PyMongo
- Ansible
- Pip
- TLS
- Cryptography
- Setuptools
- Openstack
- Requests
- PyCountry
- SOAP (Simple Object Access Protocol)
- XML
- Jinja
- OpenSSL
- pytest
- Bandit
- Gang of Four
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
[00:00:13]
Unknown:
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. When you're ready to launch your next app, you'll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 200 gigabit network, all controlled by a brand new API, you've got everything you need to scale. Go to podcastinit.com/linode to get a $20 credit and launch a new server in under a minute. And visit the site at podcastinit.com to subscribe to the show, sign up for the newsletter, and read the show notes. Your host as usual is Tobias Macy. And today, I'm interviewing Matt Warren and Chris Kurkos about the ways that they are using Python at Wanderoo. So, Matt, could you start by introducing yourself? Sure.
I'm Matt Warren. I lead the tech ops team at Wanderoo. Been there about 5 years.
[00:00:59] Unknown:
And Chris, how about you? Hey. Yeah. I'm Chris. I am a computer scientist, and I've been at Wander for close to 5 years as well. And we've both Matthew and myself have seen, longer than most of its growth, from small small cell company into,
[00:01:14] Unknown:
the scale that we are today, a profitable business. And going back to you, Matt, do you remember how you first got introduced to Python?
[00:01:20] Unknown:
Yeah. I do. My, senior year at Northeastern University, I was taking a I don't know. It was like a data sciency kind of class maybe, but we were writing a sort of a web scrape algorithm thing and I decided this seems like a really great task to do in Python so I gave it a shot and it failed horribly And I went to the TAs, and they said, well, we all know Pearl. So can you just reread it in Pearl? I said, I I mean, I guess. So that was that was my first experience. But but then when I when I started at Wandrew, there was a bigger knowledge set to to pull on there. So, you know, Wandrew is really where my my background in Python was was solidified.
[00:02:09] Unknown:
And, Chris, do you remember how you first got introduced to Python? Yeah. I was in I was in an r and d setting, and, you know, 1 of the 1 of the subcommunities of Python, before it was really big language was, the scientific community. And there were lots of tools that scientists were using that were all based on Python. So a lot of I got introduced to it through through r and d. I was writing a lot of c plus plus at the time, and I was trying to be as productive as possible, but there were a few things that I couldn't I couldn't do. There weren't there weren't libraries for things. I was doing a lot of things manually. So I so I stumbled on Python as having all of these these modules that are supporting different, different things. I can just import a module, and all of a sudden, I got this this whole world of functionality. Whereas before, I was writing c plus plus I had to write all that by hand. And it was just as good and just as fast. So I ended up ended up using a lot of Python along with my c plus plus to, to speed things up. And 1 of the earliest applications that I that I wrote in my software engineering career was a c plus plus based application, but I embedded a Python interpreter into it. I took the Python DLL, loaded it into the, loaded into the program, and then started using Python modules from from c plus plus. And I thought that was really, really cool about me to move move a lot faster than it would if I had to build all this functionality on c plus plus myself. So, so that was really That's funny. Most of the time you hear it going the other direction where somebody is using Python to call in c plus plus libraries, but you sort of flip that on its head to just call Python from your c plus plus.
It's not often that I've heard people do it that way. Yeah. And I don't think it's done too much. What I ended up doing was, I used it as a way to add scripting support into a into a library that I was already using. So if you need to perform it was performing data transformations. And so if you want to write a data transformation, you can just write it in Python and then you can build that into the application. All of a sudden, you have a new data transformation available. And so it would just we would load the the modules that it found and it can just call into those. It was just a much easier workflow than we need to have to write that, that module in c plus plus and recompile it all the time, and just allow us to move faster. So Yeah. That's definitely very cool. And so now you're both at Wanderoo,
[00:04:26] Unknown:
and I'm wondering if you can just start by describing a bit about what it is that Wanderoo does and how the overall technical platform is architected.
[00:04:36] Unknown:
So Wanderer is a metasearch company for, focusing on ground transportation. So think, buses and, trains, and ferries and and other things too. We're we're we're incorporating flights and hotels as well, but, but it most it's mostly traditionally focused on ground transportation. So if you wanna get from point a to point b and then you go to Wandreu, you, you do a search for how do I get from point a to point b, and we show you your ground transportation options. You know, take a take a bus, take a train, put 2 things together, take a bus and then a train, and it's all using using ground transportation. And we're slowly continuing to build out the platform to to be a really reliable point to point search. And, we started in the United States, and we've we've grown to encompass most of the United States in terms of coverage.
And now we're also expanding out to Europe and where we're where we've been building out a lot of our coverage in Europe as well. And then besides besides the metasearch aspects, we, we also post current travel related content on our blog. And so we'll do we'll do data posts and we'll do, travel posts and things like that and and try to deliver some good travel content to our to our subscribers. The site itself, the platform is architected. It's a micro service architecture. We use Python, Node, and Golang, as well as a combination of databases, MongoDB, Postgres, Redis.
We host the whole thing on AWS, and, and we started out as just a really small Django application where as a start up, and slowly that that grows. And and the most interesting thing is essentially we're we're doing the same exact thing as that we did when we were just a few months old. But now at scale, it changes the entire picture of the architecture. So, so how how we're hosting hosting on AWS in multiple regions and what kind of message buses we're using to to communicate between processes
[00:06:34] Unknown:
and, and how we scale to 100 of users, you know, simultaneous users on the site and and all of that. And so given that you are now using more languages beyond just Python, now that you have split it out into these various services. I'm wondering what are some of the broad categories where you are using Python and the sort of litmus test of where it fits best and when it's best to use a different language? Yeah. So broad categories for for Python
[00:07:00] Unknown:
include DevOps for monitoring, provisioning, and deployment, lots of command line tools and scripts, but we also use it for web services and middle tier microservices. So like I said, like I mentioned before, we've had, Django as a a web tool and then also TwistIt. So we've used both as web frameworks. And the other the other way we use it is data warehousing, and pipeline processing. So ETL kind of pipeline processing. So we use we use Python mostly for for that as well. So it's all over the stack at Wander. It's only the only places where we don't end up using Python are, places where we've had to have achieve a high level of concurrency where we've decided to pull in, node and, and places that we've had to be compute intensive.
That we've we've decided to pull in options like Golang or or a database or something to help us out with with those situations.
[00:07:53] Unknown:
And have you found that particularly in the compute intensive areas that it was faster and easier for you to implement solutions using Go versus writing a c or c plus plus or Rust binary that you can then call into via Python for tying everything together?
[00:08:10] Unknown:
Yeah. I I've gone back and forth on this so many times with with some of the performance aspects of our system. The timeline is really important when you're at a at a company. How fast you can deliver this, the the thing that you need is is very important, especially as a small company, especially at a start up. And we tried building it out in in node and that was that implementation failed us at scale because the memory requirements and we couldn't really and not how to optimize it properly. And we also did some testing in Python. We also used graph, you know, graph database and other kinds of databases that were supposed to be supposed to be good at at those tasks. And, ultimately, it came down to knowing that you could build a thing a certain way in a certain timeline, and that's why that's why I chose Go for for a particular particularly to do some computer intensive processing in in our stack. It was, basically because I knew that I I could just build the thing we needed and, instead of going through more of the prototype r and d things and potentially wasting time and so on. So we chose it because we knew we can we could go for
[00:09:16] Unknown:
the most reliable in the timeline. And just briefly curious which version or versions of Python you're relying on in your technical implementations?
[00:09:26] Unknown:
We have to use anything older than Python 279 for, talking with some of our partners because that's the version that introduced the TLS issues? Yep. TLS 1.2. Gotta have that in there. Yep. So that's sort of the minimum, but, you know, we have stuff I think we just rolled something that uses Python 37. Right?
[00:09:48] Unknown:
Yeah. I think that was well, what is it? Our 1 of the systems that it's using, what is it doing? It's now it's processing emails. Right? Where do you have an email processing system? Right. Using async IO and Python 3.7. It's it's also I'm very jealous because I'm still working on a system that's using Python 2.7, and I am itching to upgrade
[00:10:09] Unknown:
boot. Yeah. And in those services where you do require that high concurrency and high throughput, were those implemented at a time where the async story for Python wasn't yet at the state where it is now? And if you were to start over again, do you think that you would change the decision of what the implementation strategy was for those? Yeah. We chose we chose Node as a,
[00:10:35] Unknown:
as a technology tech for for concurrency specifically for that scenario. It's it's really fast and it's, it's concurrency model is is excellent. It's evolved over time as well. And it was at a point in time when Python didn't have the same kind of currency that that Node did. We didn't have async IO, but we had other tools that that existed. Right? We had TwistIt. We had g event at the time, and those things were were excellent. But I think the people who were deciding on the technology at that time had more more influenced by the scalability of Node than by the scalability of of Python. But well, I think 1 of the best things that has come out of Python 3, especially hitting Python 3.6, is asyncio. I think that brought Python into into the future. I would not start another project that relied on synchronous Python. I would only start a project that would rely on on async. Io from this point this point forward because it is so highly performant. So bringing that into the core the core library and to into the core of Python was 1 of the best moves. So if we had to redo it all over again, if we had to build something new, I would not hesitate at all to to put Python right in the front where we would normally put it in more process. The, the twisted stories
[00:11:49] Unknown:
sort of, like, just a big accident for us. We, we started off, you know, ingesting a bunch of data from various carrier websites, and there's a project out there called Scrapy that uses twisted libraries under the hood. And when we needed to scale it out to be a more distributed system backed by some, you know, persistent queue that we can do some inspection on. Chris sort of headed up this project, but, like, I think that's where we really fell into Twisted as a bigger framework that we could build a whole system on top of. And I the last few weeks, I've been working to introduce new improvements to that system, and it is nice to have at least something that translates into a Python 3.6 async, you know, style of writing functions and whatnot. It's a little bit easier than trying to do that full brain shift where you're writing, you know, a project that's using Python 3.6 or 3.7, and you're doing all these async things. And then you go back to it, you know, purely 2 7 code base, and you think, oh, wait. I can't do any of that stuff. Well, you can. You know, syntax will be a little bit different, but your your basic structure can can be somewhat the same.
[00:13:10] Unknown:
And, Matt, as somebody who's responsible for the deployment and platform of the overall application architecture. What has your experience been in terms of being able to deploy and update and manage the Python components of your platform? And what other ways are you using Python in terms of being able to manage that overall platform? Yeah. I I mean,
[00:13:35] Unknown:
there there are so many there are so many options when you're trying to deploy Python code, the first thing that came to mind is, you know, you just have virtual environments. And before I was writing a lot of Python at Wanneroo, I was sort of really getting into Ruby, but there was only very outside support of running different versions of Ruby and isolating libraries and things like that. So, you know, being able to use virtual environments and isolate packages and things was was really nice, you know. In addition to that, the bit that really helps us deploy out any solution is just the, I guess, ecosystem of Python.
You have all of these outside packages that talk to anything in AWS. Like, there's an officially supported, package for AWS. We use MongoDB as our, you know, backing data store for a lot of our data. And the Pymongo drivers, wonderful. The other, you know, big tool that we use is Ansible, and we've done some custom tooling, inside of Ansible and to be able to do that in all 1 programming language is just a nice thing to to have.
[00:14:53] Unknown:
Yeah. I use SaltStack for all of my infrastructure management, which is also implemented in Python, and all of the applications that we build are primarily Django. So as you mentioned, being able to keep everything with with 1 language and 1 syntax reduces a lot of the cognitive overhead of shifting between the application infrastructure and the technical infrastructure, and being able to just sort of keep that all in the same space and the same way of thinking is helpful. Yeah. Totally. And so, Chris, you had mentioned that some of the limitations that you've come up with, when trying to use Python for implementing the application logic is some things around, high concurrency and computational throughput.
So I'm curious, what have you what are some of the other limitations
[00:15:41] Unknown:
or difficulties that you've encountered while working at Wanderoo that are related to Python? Yeah. We've had a great experience with Python thus far. I think some of our difficulties has been, you know, the async async support. We fortunately had, as Matt was saying, found Twisted and had it as part of our infrastructure, and and we were able to go from an async synchronous code base to an asynchronous code base because of it. And the path from from Twisted to async IO looks very seamless, as well. And so so we're very fortunate in that respect. So definitely the the Python 3 upgrade, it's hard to upgrade legacy systems. And we started on Python 2.7 a few years ago, and I am determined to get us on Python 3 as soon as soon as we can, but we've got a lot of momentum. And and that momentum, it has to it has to go along with with the momentum of of the company. So so that's tough. The other things that have been kind of rough in terms of Python was, packaging.
Obviously, it's been it's been a big deal for a long time, but we've tried so many different things. I tried using namespace packages at 1 point, but then PIP was was was, messed up with namespacing. So, so, you know, we've got a top level namespace of Wanderoo, and we put all all our packages and and then some namespace of that, Wanderoo dot data, Wanderoo dot core, stuff like that, so we can segment our, our functionality into packages. And, and they're just dealing with little hiccups that come along the way with with trying to package and deploy, deploy our code. So that's been that's been a little challenging, a little bit of a test. You know, as Matt was saying, the SSL issues were definitely in fight and we had to do a lot of debugging to figure out what exactly it was doing, why it wasn't negotiating TLS 1.2 when that was necessary, in different scenarios, and just getting development environments up and running and piled against the records of bulk and SSL and all of that.
Those problems have seemed to go away over time, but we definitely we definitely bumped into that, a bunch. I think the other things that that are that have been important to us have been logging and that we've we've spent a lot of time trying to get our logging into a place that that is very useful for us. We can put it into a database and and dig into what's going on, have structured logs and things like that. And, and we've got it to a good place, but it's still not perfect. So I think spend, you know, probably inordinate amount of time on just trying to get logging right and, and overall, just tracing, debugging, things like that in in our systems.
[00:18:11] Unknown:
And, Matt, have there been any other particular edge cases or difficulties that you've encountered in terms of your use of Python at Wondering? The 1 place where it's really sticks out to me is funny because, you know, talked about the benefits of using Scrapy, and
[00:18:27] Unknown:
it helped us a lot when we were starting out. Unfortunately, we were using such a new version of it at the time that there were some breaking changes in the versions immediately after that. And we've had a heck of a time trying to upgrade that or isolate it or change it or what have you. And at 1 point, there was an unrelated change in oh, gosh. Was it twisted or no. It was in cryptography, and the version of setup tools that cryptography needed bumped heads with this version of setup tools that Scrapy needed. We just got into this weird place for a little while there, But, it all it all sort of worked out and and actually the community of people supporting those projects, we were able to to, you know, open bugs and and get some of those things resolved. So, like, at the end of the day, things have been pretty good, really. And have there been any places
[00:19:27] Unknown:
that Python has used at Wanderoo where you were surprised that that was the choice where you would have thought that some other either language or service or platform plays
[00:19:39] Unknown:
a very critical role to our entire entire stack. And it plays a very critical role to our entire entire stack. And, for the number of lines of code and the complexity of the project, you would assume that something like Java was was being used to power it. There's there's a little more structure of, need to to the language that they're using. I think traditionally people people consider Python good for data science or scripting or know, these these other isolated smaller, you know, old web frameworks. But we have a pretty large system that that is all built in Python. It's performing really well. And like I said, we we've we've we've founded a lot of that on on Twisted, and and so it's it's very concurrent serving serving a lot of a lot of requests and, and playing a really big really big role in our entire stack. I think 1 of the projects that gave me hope that we could achieve that kind of scale with Python was OpenStack. Like, I remember looking utilizing OpenStack for a while and just looking at the size of that of that code base and what they were accomplishing with with the microservice architecture and all of how it was all working together. That was the first example where I saw a really big cohesive code base all built on Python, all working really well. And I think that that example gave, gave me some hope that we can actually take our start you know, our our scrappy start up code and turn it into a into the scalable system that's, that's, like, credible to our to our business.
So I think it surprised me in its ability to to scale and accomplish what it's helped us accomplish.
[00:21:21] Unknown:
So as you have scaled the company and the platforms that you need to be able to support the business, what are some of the areas that have been most stressed and have required the most amount of work to be able to scale them out? Yeah. I think the compute intensive processes that we have, those are those have been the hardest
[00:21:41] Unknown:
when, you know, like you're saying, the being able to to find routes efficiently and things like that. It helps when you have a really fast, solid database and people who understand databases, to to offer a lot of a lot of that kind of processing. But sometimes you just need that that speed and that parallelism. And in those scenarios, that's been the tough parts for us. Yeah. To
[00:22:04] Unknown:
to give you a specific example, and it's this wasn't necessarily Python's fault, but we we were trying to iterate quickly over 1 particular Django application, and we had a bunch of data in MongoDB that this application was, you know, querying through an API and then running a huge, like, merge over multiple documents to then come up with the page to present to an incoming user and it was 1 of those things where this this Django application predated me and and we just kept swapping out the data source and it it you know it just got to a point where that was no longer going to be scalable And we finally rewrote it in in React and changed the underlying database and all those kinds of things. But, like, there was so much stuff going on inside of that application. It was, it was crazy.
[00:23:07] Unknown:
And as you have been working at wanderoo and adding new features and building out the platform, what have you found to be some of the most useful libraries or language features that enable you to get your work done? Yeah. I think, you know, Twisted has been
[00:23:25] Unknown:
pretty critical for us. I wish it was a little bit more supported nowadays. I think the matter was going with async, I don't know all that, but, but that's been pretty pretty critical critical for us. Requests has been a big deal as well. That's probably 1 of the most used libraries out there. And I think all the drivers, the database drivers, have been incredibly important. Just having really strong database drivers and and support for that kind of stuff has been pretty critical for us. And how about for you, Matt? Yeah. I'm I could name the big ones.
[00:23:58] Unknown:
Boto 3 and, you know, stuff like that. I think more recently I've been doing a lot with PI country. I'm migrating from you know sort of 1 data source to another and and PI country has provided a lot of glue where I can query information without trying to statically grab it myself. There's always just the 1 off thing that that will occasionally fill in a gap for you. And you're just like, oh, this is great. I'm so glad somebody wrote this.
[00:24:25] Unknown:
And, conversely, which libraries or language features have you found to be the most painful as you tried to get things done?
[00:24:33] Unknown:
Yeah. We've bumped into a bunch of issues with XML and SOP parsing and generation, and that's that's never that much fun generating valid XML documents, but they're not being accepted and and things like that. That's been that's been a little bit of a headache. So we we do a lot of XML templating with Jinja and and all of that instead of instead of using some some XML sort of libraries and all of that. So, OpenSSL, like Matt was saying, and straightvf both cause us a bit of pain with, dependencies and and upgrading and process offerings.
[00:25:08] Unknown:
Matt, if you could share some of the pain points that you've experienced as well. I think I'm with Chris on this 1. I'm not sure I don't think we've had too much that's been really catastrophic. If something doesn't work, there's probably another package that'll do it a different way. So apart from the things that are really integrated into our into what we do and and how we do it,
[00:25:31] Unknown:
I I can't think of much. And in terms of your development processes for Python in particular, but also in any of your other code bases, what are some of the ways that you're using Python in terms of the tooling or unit tests or things like that? We try to stick with the standard Python
[00:25:50] Unknown:
unit test frameworks and and everything built into the language, as much as we can. But we use Pytest, and we have code style standardization. And, we do unit tests as as much as we can, mass integration tests, and try to maintain a clean separation between what's a unit test and what's an integration test and when we run a unit test versus an integration test. So, the build a system that Matt has put together. We utilized Jenkins and some other some other processes, but we, we set we set to make sure our unit tests are are run before any of our deployments are run. So our unit tests, if they fail, block deployments, in our in our automated infrastructure and that we rely on that kind of stuff all the time. And really beyond that, it's for us, it's mostly about setting some standards and making sure that everybody understands them and follows them up with them in terms of how they do unit testing, how they do mocking, and how how they do integration testing, and what what the differences between between those are and setting standards for that kind of stuff. That's the most But we've had great experience so far with with just unit tests and and and mocks, and they've done they've done a lot for us.
[00:27:07] Unknown:
Yeah. Pytest is is pretty awesome. I'll I'll also say, something I recently found, and and we need to or I'd like to see us do a little bit more, with is is a thing or a library called Bandit, which is a security vulnerability sort of scanning tool. We don't have a lot of external facing Python code. You know, a lot of our code makes calls to other things but nothing or not a lot calls into it. Bandit does a good job of you know scanning for known Python things to avoid and and any published, vulnerabilities in the libraries that you're using as well. They can look some of that stuff up. And are there any other aspects
[00:27:56] Unknown:
of the work that you're doing at Wanderoo or the platform that you're building or some of the business challenges that you're facing or ways you're using Python to achieve any of those things that we you think we should cover in greater depth before we close out the show?
[00:28:10] Unknown:
Sure. Yeah. I can just say a few other things about about our technology choices and and why these use Python. I think the most the most important thing for us has been our ability to, work with each other in the company, and that that seems a lot on on all of our knowledge of of Python. Everyone in the company has an understanding of of Python and or on the tech side of that, and, it's really good to be able to, as a small group, cross teams and work with each other technically. And so so having a language that's both amenable to beginners and experts alike is really important for us.
So beginners can can jump in and use Python, and and it's it's also good for for experts who really know what they're doing with that. Right? It's, you know, a c extension or a Rust extension, or they want to optimize it, in a certain way, you can you can do that with Python. So it's really it's really very flexible in that respect. So that's been probably 1 of the most important things for us as well as as well as a rapid prototyping. It's it's easy for us to rapidly prototype a tool in a week, or a feature and then, take it to production after that. We use reuse a lot of the code that we've built in, you know, prototype and and and flush it out and make it production ready. So having having the ability to go to the rapid rapid prototype and and build it out into the production system is really important for us as well. So as a small company, speed and flexibility is really important. I think 1 of the challenges, though, as you scale with Python ends up being this this is probably true for most dynamic languages. They don't impose too much structure on you. They assume that you you know what you're doing and you know where to impose your own structure. That's that's a part of of using a dynamic language. And so so knowing where you should you should create structure is important, where a a piece of code is gonna be repeated multiple times. You're gonna you're gonna create a pattern. You create, you know, a set of models or a set of controllers or a set of classes, and you're gonna you're gonna establish a pattern and then know that that pattern is gonna be communicated and copied over and over again, you know, repeated to in order to achieve scale in that in that part of the system. But you have to put that structure in place with, with the language of Python. You don't you don't necessarily get a lot of the standard tools that you get with with Java, where where you know you're gonna you're gonna use the language. You're gonna use the design patterns from the day before book. You're gonna use the, the debugging support and and Eclipse and all of these tools that you've got. The set ecosystem that you've been set up for yourself and and you build.
You don't really do that with Python. You have to build your structure as you go. So it's important to be cognizant of that as as you scale. The point about, you know, having people
[00:31:08] Unknown:
across the, you know, tech group all know Python is kind of cool because it's not like we've gone and hired people who had a very strict Python background. We've had a number of people who came in not knowing any any Python, and they've been able to pick it up and learn very quickly and come up to speed with what we're doing with Python. The structure, you know, helps with that as well as you can sort of repeat something, and once you understand it, you can extend it and all those kinds of things. So, Python's been
[00:31:43] Unknown:
paying off in in really nice ways there as well. Well, for anybody who wants to get in touch with either of you or follow the work that you're up to, I'll have you add your preferred contact information to the show notes. And so with that, I'll move us into the picks. And this week, I'm going to choose the data grip tool from JetBrains, which is their general purpose database client, which has support for a large number of drivers and database platforms, and I found it to be quite useful for my day to day work. So, I've gone through a number of different database clients over the years, and data grip has been 1 that's had a had a pretty good amount of sticking power. So, it's definitely worth checking out if that's something you're in the market for. And so with that, I'll pass it to you, Matt. Do you have any picks this week? Yeah. You know what? We we moved offices recently, and I will say being,
[00:32:32] Unknown:
very close to Chuckararo is super nice, so I'll I'll pick that. If you ever if you ever been over over to Chacoraro, it's, you know, meat, beans, and
[00:32:42] Unknown:
tomatoes, I think. And, Chris, do you have any picks this week? I've been on a deployment kick, like, lately. So I'm gonna give some props to, to PDB, IPDB, PUDB, and, the 1 that that that ships with Versus Code. What is it? I forget the 1 that ships with Versus Code, but I've I've been integrating I've been using Versus Code lately and having that with an integrated debugger and being able to attach to the remote process or to to my local programming program and and figure things out, has been incredible. I I cringe at the use of of print statements, so most of my guys here would would definitely, tell you that I get pretty upset about print statements. But, I love I love the debuggers and the debuggers support, and I just hope that Python keeps keeps progressing in that in that front with tracing and all that kinda all those kinda tools. So so mad props to IPDB, PUDD, and,
[00:33:39] Unknown:
the Versus this 1 I said with Versus Code as well. Alright. Well, thank you both very much for taking the time out of your day to talk about the work that you've been doing and the ways that you're using Python to do it. I hope you enjoy the rest of your day and, thank you again. Thank you. Yeah. You too. Thanks, Tobias.
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. When you're ready to launch your next app, you'll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 200 gigabit network, all controlled by a brand new API, you've got everything you need to scale. Go to podcastinit.com/linode to get a $20 credit and launch a new server in under a minute. And visit the site at podcastinit.com to subscribe to the show, sign up for the newsletter, and read the show notes. Your host as usual is Tobias Macy. And today, I'm interviewing Matt Warren and Chris Kurkos about the ways that they are using Python at Wanderoo. So, Matt, could you start by introducing yourself? Sure.
I'm Matt Warren. I lead the tech ops team at Wanderoo. Been there about 5 years.
[00:00:59] Unknown:
And Chris, how about you? Hey. Yeah. I'm Chris. I am a computer scientist, and I've been at Wander for close to 5 years as well. And we've both Matthew and myself have seen, longer than most of its growth, from small small cell company into,
[00:01:14] Unknown:
the scale that we are today, a profitable business. And going back to you, Matt, do you remember how you first got introduced to Python?
[00:01:20] Unknown:
Yeah. I do. My, senior year at Northeastern University, I was taking a I don't know. It was like a data sciency kind of class maybe, but we were writing a sort of a web scrape algorithm thing and I decided this seems like a really great task to do in Python so I gave it a shot and it failed horribly And I went to the TAs, and they said, well, we all know Pearl. So can you just reread it in Pearl? I said, I I mean, I guess. So that was that was my first experience. But but then when I when I started at Wandrew, there was a bigger knowledge set to to pull on there. So, you know, Wandrew is really where my my background in Python was was solidified.
[00:02:09] Unknown:
And, Chris, do you remember how you first got introduced to Python? Yeah. I was in I was in an r and d setting, and, you know, 1 of the 1 of the subcommunities of Python, before it was really big language was, the scientific community. And there were lots of tools that scientists were using that were all based on Python. So a lot of I got introduced to it through through r and d. I was writing a lot of c plus plus at the time, and I was trying to be as productive as possible, but there were a few things that I couldn't I couldn't do. There weren't there weren't libraries for things. I was doing a lot of things manually. So I so I stumbled on Python as having all of these these modules that are supporting different, different things. I can just import a module, and all of a sudden, I got this this whole world of functionality. Whereas before, I was writing c plus plus I had to write all that by hand. And it was just as good and just as fast. So I ended up ended up using a lot of Python along with my c plus plus to, to speed things up. And 1 of the earliest applications that I that I wrote in my software engineering career was a c plus plus based application, but I embedded a Python interpreter into it. I took the Python DLL, loaded it into the, loaded into the program, and then started using Python modules from from c plus plus. And I thought that was really, really cool about me to move move a lot faster than it would if I had to build all this functionality on c plus plus myself. So, so that was really That's funny. Most of the time you hear it going the other direction where somebody is using Python to call in c plus plus libraries, but you sort of flip that on its head to just call Python from your c plus plus.
It's not often that I've heard people do it that way. Yeah. And I don't think it's done too much. What I ended up doing was, I used it as a way to add scripting support into a into a library that I was already using. So if you need to perform it was performing data transformations. And so if you want to write a data transformation, you can just write it in Python and then you can build that into the application. All of a sudden, you have a new data transformation available. And so it would just we would load the the modules that it found and it can just call into those. It was just a much easier workflow than we need to have to write that, that module in c plus plus and recompile it all the time, and just allow us to move faster. So Yeah. That's definitely very cool. And so now you're both at Wanderoo,
[00:04:26] Unknown:
and I'm wondering if you can just start by describing a bit about what it is that Wanderoo does and how the overall technical platform is architected.
[00:04:36] Unknown:
So Wanderer is a metasearch company for, focusing on ground transportation. So think, buses and, trains, and ferries and and other things too. We're we're we're incorporating flights and hotels as well, but, but it most it's mostly traditionally focused on ground transportation. So if you wanna get from point a to point b and then you go to Wandreu, you, you do a search for how do I get from point a to point b, and we show you your ground transportation options. You know, take a take a bus, take a train, put 2 things together, take a bus and then a train, and it's all using using ground transportation. And we're slowly continuing to build out the platform to to be a really reliable point to point search. And, we started in the United States, and we've we've grown to encompass most of the United States in terms of coverage.
And now we're also expanding out to Europe and where we're where we've been building out a lot of our coverage in Europe as well. And then besides besides the metasearch aspects, we, we also post current travel related content on our blog. And so we'll do we'll do data posts and we'll do, travel posts and things like that and and try to deliver some good travel content to our to our subscribers. The site itself, the platform is architected. It's a micro service architecture. We use Python, Node, and Golang, as well as a combination of databases, MongoDB, Postgres, Redis.
We host the whole thing on AWS, and, and we started out as just a really small Django application where as a start up, and slowly that that grows. And and the most interesting thing is essentially we're we're doing the same exact thing as that we did when we were just a few months old. But now at scale, it changes the entire picture of the architecture. So, so how how we're hosting hosting on AWS in multiple regions and what kind of message buses we're using to to communicate between processes
[00:06:34] Unknown:
and, and how we scale to 100 of users, you know, simultaneous users on the site and and all of that. And so given that you are now using more languages beyond just Python, now that you have split it out into these various services. I'm wondering what are some of the broad categories where you are using Python and the sort of litmus test of where it fits best and when it's best to use a different language? Yeah. So broad categories for for Python
[00:07:00] Unknown:
include DevOps for monitoring, provisioning, and deployment, lots of command line tools and scripts, but we also use it for web services and middle tier microservices. So like I said, like I mentioned before, we've had, Django as a a web tool and then also TwistIt. So we've used both as web frameworks. And the other the other way we use it is data warehousing, and pipeline processing. So ETL kind of pipeline processing. So we use we use Python mostly for for that as well. So it's all over the stack at Wander. It's only the only places where we don't end up using Python are, places where we've had to have achieve a high level of concurrency where we've decided to pull in, node and, and places that we've had to be compute intensive.
That we've we've decided to pull in options like Golang or or a database or something to help us out with with those situations.
[00:07:53] Unknown:
And have you found that particularly in the compute intensive areas that it was faster and easier for you to implement solutions using Go versus writing a c or c plus plus or Rust binary that you can then call into via Python for tying everything together?
[00:08:10] Unknown:
Yeah. I I've gone back and forth on this so many times with with some of the performance aspects of our system. The timeline is really important when you're at a at a company. How fast you can deliver this, the the thing that you need is is very important, especially as a small company, especially at a start up. And we tried building it out in in node and that was that implementation failed us at scale because the memory requirements and we couldn't really and not how to optimize it properly. And we also did some testing in Python. We also used graph, you know, graph database and other kinds of databases that were supposed to be supposed to be good at at those tasks. And, ultimately, it came down to knowing that you could build a thing a certain way in a certain timeline, and that's why that's why I chose Go for for a particular particularly to do some computer intensive processing in in our stack. It was, basically because I knew that I I could just build the thing we needed and, instead of going through more of the prototype r and d things and potentially wasting time and so on. So we chose it because we knew we can we could go for
[00:09:16] Unknown:
the most reliable in the timeline. And just briefly curious which version or versions of Python you're relying on in your technical implementations?
[00:09:26] Unknown:
We have to use anything older than Python 279 for, talking with some of our partners because that's the version that introduced the TLS issues? Yep. TLS 1.2. Gotta have that in there. Yep. So that's sort of the minimum, but, you know, we have stuff I think we just rolled something that uses Python 37. Right?
[00:09:48] Unknown:
Yeah. I think that was well, what is it? Our 1 of the systems that it's using, what is it doing? It's now it's processing emails. Right? Where do you have an email processing system? Right. Using async IO and Python 3.7. It's it's also I'm very jealous because I'm still working on a system that's using Python 2.7, and I am itching to upgrade
[00:10:09] Unknown:
boot. Yeah. And in those services where you do require that high concurrency and high throughput, were those implemented at a time where the async story for Python wasn't yet at the state where it is now? And if you were to start over again, do you think that you would change the decision of what the implementation strategy was for those? Yeah. We chose we chose Node as a,
[00:10:35] Unknown:
as a technology tech for for concurrency specifically for that scenario. It's it's really fast and it's, it's concurrency model is is excellent. It's evolved over time as well. And it was at a point in time when Python didn't have the same kind of currency that that Node did. We didn't have async IO, but we had other tools that that existed. Right? We had TwistIt. We had g event at the time, and those things were were excellent. But I think the people who were deciding on the technology at that time had more more influenced by the scalability of Node than by the scalability of of Python. But well, I think 1 of the best things that has come out of Python 3, especially hitting Python 3.6, is asyncio. I think that brought Python into into the future. I would not start another project that relied on synchronous Python. I would only start a project that would rely on on async. Io from this point this point forward because it is so highly performant. So bringing that into the core the core library and to into the core of Python was 1 of the best moves. So if we had to redo it all over again, if we had to build something new, I would not hesitate at all to to put Python right in the front where we would normally put it in more process. The, the twisted stories
[00:11:49] Unknown:
sort of, like, just a big accident for us. We, we started off, you know, ingesting a bunch of data from various carrier websites, and there's a project out there called Scrapy that uses twisted libraries under the hood. And when we needed to scale it out to be a more distributed system backed by some, you know, persistent queue that we can do some inspection on. Chris sort of headed up this project, but, like, I think that's where we really fell into Twisted as a bigger framework that we could build a whole system on top of. And I the last few weeks, I've been working to introduce new improvements to that system, and it is nice to have at least something that translates into a Python 3.6 async, you know, style of writing functions and whatnot. It's a little bit easier than trying to do that full brain shift where you're writing, you know, a project that's using Python 3.6 or 3.7, and you're doing all these async things. And then you go back to it, you know, purely 2 7 code base, and you think, oh, wait. I can't do any of that stuff. Well, you can. You know, syntax will be a little bit different, but your your basic structure can can be somewhat the same.
[00:13:10] Unknown:
And, Matt, as somebody who's responsible for the deployment and platform of the overall application architecture. What has your experience been in terms of being able to deploy and update and manage the Python components of your platform? And what other ways are you using Python in terms of being able to manage that overall platform? Yeah. I I mean,
[00:13:35] Unknown:
there there are so many there are so many options when you're trying to deploy Python code, the first thing that came to mind is, you know, you just have virtual environments. And before I was writing a lot of Python at Wanneroo, I was sort of really getting into Ruby, but there was only very outside support of running different versions of Ruby and isolating libraries and things like that. So, you know, being able to use virtual environments and isolate packages and things was was really nice, you know. In addition to that, the bit that really helps us deploy out any solution is just the, I guess, ecosystem of Python.
You have all of these outside packages that talk to anything in AWS. Like, there's an officially supported, package for AWS. We use MongoDB as our, you know, backing data store for a lot of our data. And the Pymongo drivers, wonderful. The other, you know, big tool that we use is Ansible, and we've done some custom tooling, inside of Ansible and to be able to do that in all 1 programming language is just a nice thing to to have.
[00:14:53] Unknown:
Yeah. I use SaltStack for all of my infrastructure management, which is also implemented in Python, and all of the applications that we build are primarily Django. So as you mentioned, being able to keep everything with with 1 language and 1 syntax reduces a lot of the cognitive overhead of shifting between the application infrastructure and the technical infrastructure, and being able to just sort of keep that all in the same space and the same way of thinking is helpful. Yeah. Totally. And so, Chris, you had mentioned that some of the limitations that you've come up with, when trying to use Python for implementing the application logic is some things around, high concurrency and computational throughput.
So I'm curious, what have you what are some of the other limitations
[00:15:41] Unknown:
or difficulties that you've encountered while working at Wanderoo that are related to Python? Yeah. We've had a great experience with Python thus far. I think some of our difficulties has been, you know, the async async support. We fortunately had, as Matt was saying, found Twisted and had it as part of our infrastructure, and and we were able to go from an async synchronous code base to an asynchronous code base because of it. And the path from from Twisted to async IO looks very seamless, as well. And so so we're very fortunate in that respect. So definitely the the Python 3 upgrade, it's hard to upgrade legacy systems. And we started on Python 2.7 a few years ago, and I am determined to get us on Python 3 as soon as soon as we can, but we've got a lot of momentum. And and that momentum, it has to it has to go along with with the momentum of of the company. So so that's tough. The other things that have been kind of rough in terms of Python was, packaging.
Obviously, it's been it's been a big deal for a long time, but we've tried so many different things. I tried using namespace packages at 1 point, but then PIP was was was, messed up with namespacing. So, so, you know, we've got a top level namespace of Wanderoo, and we put all all our packages and and then some namespace of that, Wanderoo dot data, Wanderoo dot core, stuff like that, so we can segment our, our functionality into packages. And, and they're just dealing with little hiccups that come along the way with with trying to package and deploy, deploy our code. So that's been that's been a little challenging, a little bit of a test. You know, as Matt was saying, the SSL issues were definitely in fight and we had to do a lot of debugging to figure out what exactly it was doing, why it wasn't negotiating TLS 1.2 when that was necessary, in different scenarios, and just getting development environments up and running and piled against the records of bulk and SSL and all of that.
Those problems have seemed to go away over time, but we definitely we definitely bumped into that, a bunch. I think the other things that that are that have been important to us have been logging and that we've we've spent a lot of time trying to get our logging into a place that that is very useful for us. We can put it into a database and and dig into what's going on, have structured logs and things like that. And, and we've got it to a good place, but it's still not perfect. So I think spend, you know, probably inordinate amount of time on just trying to get logging right and, and overall, just tracing, debugging, things like that in in our systems.
[00:18:11] Unknown:
And, Matt, have there been any other particular edge cases or difficulties that you've encountered in terms of your use of Python at Wondering? The 1 place where it's really sticks out to me is funny because, you know, talked about the benefits of using Scrapy, and
[00:18:27] Unknown:
it helped us a lot when we were starting out. Unfortunately, we were using such a new version of it at the time that there were some breaking changes in the versions immediately after that. And we've had a heck of a time trying to upgrade that or isolate it or change it or what have you. And at 1 point, there was an unrelated change in oh, gosh. Was it twisted or no. It was in cryptography, and the version of setup tools that cryptography needed bumped heads with this version of setup tools that Scrapy needed. We just got into this weird place for a little while there, But, it all it all sort of worked out and and actually the community of people supporting those projects, we were able to to, you know, open bugs and and get some of those things resolved. So, like, at the end of the day, things have been pretty good, really. And have there been any places
[00:19:27] Unknown:
that Python has used at Wanderoo where you were surprised that that was the choice where you would have thought that some other either language or service or platform plays
[00:19:39] Unknown:
a very critical role to our entire entire stack. And it plays a very critical role to our entire entire stack. And, for the number of lines of code and the complexity of the project, you would assume that something like Java was was being used to power it. There's there's a little more structure of, need to to the language that they're using. I think traditionally people people consider Python good for data science or scripting or know, these these other isolated smaller, you know, old web frameworks. But we have a pretty large system that that is all built in Python. It's performing really well. And like I said, we we've we've we've founded a lot of that on on Twisted, and and so it's it's very concurrent serving serving a lot of a lot of requests and, and playing a really big really big role in our entire stack. I think 1 of the projects that gave me hope that we could achieve that kind of scale with Python was OpenStack. Like, I remember looking utilizing OpenStack for a while and just looking at the size of that of that code base and what they were accomplishing with with the microservice architecture and all of how it was all working together. That was the first example where I saw a really big cohesive code base all built on Python, all working really well. And I think that that example gave, gave me some hope that we can actually take our start you know, our our scrappy start up code and turn it into a into the scalable system that's, that's, like, credible to our to our business.
So I think it surprised me in its ability to to scale and accomplish what it's helped us accomplish.
[00:21:21] Unknown:
So as you have scaled the company and the platforms that you need to be able to support the business, what are some of the areas that have been most stressed and have required the most amount of work to be able to scale them out? Yeah. I think the compute intensive processes that we have, those are those have been the hardest
[00:21:41] Unknown:
when, you know, like you're saying, the being able to to find routes efficiently and things like that. It helps when you have a really fast, solid database and people who understand databases, to to offer a lot of a lot of that kind of processing. But sometimes you just need that that speed and that parallelism. And in those scenarios, that's been the tough parts for us. Yeah. To
[00:22:04] Unknown:
to give you a specific example, and it's this wasn't necessarily Python's fault, but we we were trying to iterate quickly over 1 particular Django application, and we had a bunch of data in MongoDB that this application was, you know, querying through an API and then running a huge, like, merge over multiple documents to then come up with the page to present to an incoming user and it was 1 of those things where this this Django application predated me and and we just kept swapping out the data source and it it you know it just got to a point where that was no longer going to be scalable And we finally rewrote it in in React and changed the underlying database and all those kinds of things. But, like, there was so much stuff going on inside of that application. It was, it was crazy.
[00:23:07] Unknown:
And as you have been working at wanderoo and adding new features and building out the platform, what have you found to be some of the most useful libraries or language features that enable you to get your work done? Yeah. I think, you know, Twisted has been
[00:23:25] Unknown:
pretty critical for us. I wish it was a little bit more supported nowadays. I think the matter was going with async, I don't know all that, but, but that's been pretty pretty critical critical for us. Requests has been a big deal as well. That's probably 1 of the most used libraries out there. And I think all the drivers, the database drivers, have been incredibly important. Just having really strong database drivers and and support for that kind of stuff has been pretty critical for us. And how about for you, Matt? Yeah. I'm I could name the big ones.
[00:23:58] Unknown:
Boto 3 and, you know, stuff like that. I think more recently I've been doing a lot with PI country. I'm migrating from you know sort of 1 data source to another and and PI country has provided a lot of glue where I can query information without trying to statically grab it myself. There's always just the 1 off thing that that will occasionally fill in a gap for you. And you're just like, oh, this is great. I'm so glad somebody wrote this.
[00:24:25] Unknown:
And, conversely, which libraries or language features have you found to be the most painful as you tried to get things done?
[00:24:33] Unknown:
Yeah. We've bumped into a bunch of issues with XML and SOP parsing and generation, and that's that's never that much fun generating valid XML documents, but they're not being accepted and and things like that. That's been that's been a little bit of a headache. So we we do a lot of XML templating with Jinja and and all of that instead of instead of using some some XML sort of libraries and all of that. So, OpenSSL, like Matt was saying, and straightvf both cause us a bit of pain with, dependencies and and upgrading and process offerings.
[00:25:08] Unknown:
Matt, if you could share some of the pain points that you've experienced as well. I think I'm with Chris on this 1. I'm not sure I don't think we've had too much that's been really catastrophic. If something doesn't work, there's probably another package that'll do it a different way. So apart from the things that are really integrated into our into what we do and and how we do it,
[00:25:31] Unknown:
I I can't think of much. And in terms of your development processes for Python in particular, but also in any of your other code bases, what are some of the ways that you're using Python in terms of the tooling or unit tests or things like that? We try to stick with the standard Python
[00:25:50] Unknown:
unit test frameworks and and everything built into the language, as much as we can. But we use Pytest, and we have code style standardization. And, we do unit tests as as much as we can, mass integration tests, and try to maintain a clean separation between what's a unit test and what's an integration test and when we run a unit test versus an integration test. So, the build a system that Matt has put together. We utilized Jenkins and some other some other processes, but we, we set we set to make sure our unit tests are are run before any of our deployments are run. So our unit tests, if they fail, block deployments, in our in our automated infrastructure and that we rely on that kind of stuff all the time. And really beyond that, it's for us, it's mostly about setting some standards and making sure that everybody understands them and follows them up with them in terms of how they do unit testing, how they do mocking, and how how they do integration testing, and what what the differences between between those are and setting standards for that kind of stuff. That's the most But we've had great experience so far with with just unit tests and and and mocks, and they've done they've done a lot for us.
[00:27:07] Unknown:
Yeah. Pytest is is pretty awesome. I'll I'll also say, something I recently found, and and we need to or I'd like to see us do a little bit more, with is is a thing or a library called Bandit, which is a security vulnerability sort of scanning tool. We don't have a lot of external facing Python code. You know, a lot of our code makes calls to other things but nothing or not a lot calls into it. Bandit does a good job of you know scanning for known Python things to avoid and and any published, vulnerabilities in the libraries that you're using as well. They can look some of that stuff up. And are there any other aspects
[00:27:56] Unknown:
of the work that you're doing at Wanderoo or the platform that you're building or some of the business challenges that you're facing or ways you're using Python to achieve any of those things that we you think we should cover in greater depth before we close out the show?
[00:28:10] Unknown:
Sure. Yeah. I can just say a few other things about about our technology choices and and why these use Python. I think the most the most important thing for us has been our ability to, work with each other in the company, and that that seems a lot on on all of our knowledge of of Python. Everyone in the company has an understanding of of Python and or on the tech side of that, and, it's really good to be able to, as a small group, cross teams and work with each other technically. And so so having a language that's both amenable to beginners and experts alike is really important for us.
So beginners can can jump in and use Python, and and it's it's also good for for experts who really know what they're doing with that. Right? It's, you know, a c extension or a Rust extension, or they want to optimize it, in a certain way, you can you can do that with Python. So it's really it's really very flexible in that respect. So that's been probably 1 of the most important things for us as well as as well as a rapid prototyping. It's it's easy for us to rapidly prototype a tool in a week, or a feature and then, take it to production after that. We use reuse a lot of the code that we've built in, you know, prototype and and and flush it out and make it production ready. So having having the ability to go to the rapid rapid prototype and and build it out into the production system is really important for us as well. So as a small company, speed and flexibility is really important. I think 1 of the challenges, though, as you scale with Python ends up being this this is probably true for most dynamic languages. They don't impose too much structure on you. They assume that you you know what you're doing and you know where to impose your own structure. That's that's a part of of using a dynamic language. And so so knowing where you should you should create structure is important, where a a piece of code is gonna be repeated multiple times. You're gonna you're gonna create a pattern. You create, you know, a set of models or a set of controllers or a set of classes, and you're gonna you're gonna establish a pattern and then know that that pattern is gonna be communicated and copied over and over again, you know, repeated to in order to achieve scale in that in that part of the system. But you have to put that structure in place with, with the language of Python. You don't you don't necessarily get a lot of the standard tools that you get with with Java, where where you know you're gonna you're gonna use the language. You're gonna use the design patterns from the day before book. You're gonna use the, the debugging support and and Eclipse and all of these tools that you've got. The set ecosystem that you've been set up for yourself and and you build.
You don't really do that with Python. You have to build your structure as you go. So it's important to be cognizant of that as as you scale. The point about, you know, having people
[00:31:08] Unknown:
across the, you know, tech group all know Python is kind of cool because it's not like we've gone and hired people who had a very strict Python background. We've had a number of people who came in not knowing any any Python, and they've been able to pick it up and learn very quickly and come up to speed with what we're doing with Python. The structure, you know, helps with that as well as you can sort of repeat something, and once you understand it, you can extend it and all those kinds of things. So, Python's been
[00:31:43] Unknown:
paying off in in really nice ways there as well. Well, for anybody who wants to get in touch with either of you or follow the work that you're up to, I'll have you add your preferred contact information to the show notes. And so with that, I'll move us into the picks. And this week, I'm going to choose the data grip tool from JetBrains, which is their general purpose database client, which has support for a large number of drivers and database platforms, and I found it to be quite useful for my day to day work. So, I've gone through a number of different database clients over the years, and data grip has been 1 that's had a had a pretty good amount of sticking power. So, it's definitely worth checking out if that's something you're in the market for. And so with that, I'll pass it to you, Matt. Do you have any picks this week? Yeah. You know what? We we moved offices recently, and I will say being,
[00:32:32] Unknown:
very close to Chuckararo is super nice, so I'll I'll pick that. If you ever if you ever been over over to Chacoraro, it's, you know, meat, beans, and
[00:32:42] Unknown:
tomatoes, I think. And, Chris, do you have any picks this week? I've been on a deployment kick, like, lately. So I'm gonna give some props to, to PDB, IPDB, PUDB, and, the 1 that that that ships with Versus Code. What is it? I forget the 1 that ships with Versus Code, but I've I've been integrating I've been using Versus Code lately and having that with an integrated debugger and being able to attach to the remote process or to to my local programming program and and figure things out, has been incredible. I I cringe at the use of of print statements, so most of my guys here would would definitely, tell you that I get pretty upset about print statements. But, I love I love the debuggers and the debuggers support, and I just hope that Python keeps keeps progressing in that in that front with tracing and all that kinda all those kinda tools. So so mad props to IPDB, PUDD, and,
[00:33:39] Unknown:
the Versus this 1 I said with Versus Code as well. Alright. Well, thank you both very much for taking the time out of your day to talk about the work that you've been doing and the ways that you're using Python to do it. I hope you enjoy the rest of your day and, thank you again. Thank you. Yeah. You too. Thanks, Tobias.
Introduction and Guest Introduction
First Experiences with Python
Overview of Wanderoo
Python's Role in Wanderoo's Architecture
Challenges and Benefits of Using Python
Scaling Challenges and Solutions
Useful Libraries and Tools
Development Processes and Testing
Final Thoughts on Python at Wanderoo
Picks and Recommendations
