Mike Driscoll And His Career In Python

Hello, and welcome to podcast.init,

the podcast about Python and the people who make it great.

When you're ready to launch your next app, you'll need somewhere to deploy it, so you should check out Linode. With private networking, shared block storage, node balancers, and a 40 gigabit network, all controlled by a brand new API, you get everything you need to scale. Go to podcast in it.com/linode

to get a $20 credit and launch a new server in under a minute. And to get worry free releases, download Go CD, the open source continuous delivery server built by Thoughtworks.

You can use their pipeline modeling and value stream app to build, control, and monitor every step from commit to deployment in 1 place. Go to podcastinit.com/gocd

to learn more about their professional support services and enterprise add ons.

And visit podcastthenit.com

to subscribe to the show, sign up for the newsletter, and read the show notes. Your host as usual is Tobias Macy, and today I'm interviewing Mike Driscoll about using Python to test c plus plus QT UIs for embedded platforms. So, Mike, could you start by introducing yourself?

Sure. I'm Mike Driscoll.

I use Python full time for an egg company,

and I've been using,

Python itself for over 12 years.

And do you remember how you first got introduced to the language? Yeah.

So I started a job with, the local government here in Iowa, And,

my boss was really into Python,

and he just required any new hire to learn Python and start using it as part of our full time job.

And also for people who might be familiar with your blog, you've been running mouse versus Python for quite a while now, and I know that you've had some popular series there. So I'll definitely add some links to the show notes for anybody who wants to check that out in greater depth. And so today, we're focusing mainly on the ways that you're using Python at work. So I'm wondering if you can just start by describing

some of the ways that you're leveraging it in your day to day and how you got involved in that.

Yeah. So

I've been using Python professionally pretty much since I got out of

college. And,

my current job, I'm working for a company called Ag Leader. My job is to test their app that was written in c+ and Qt,

with Python.

And

so,

right now I'm the subject matter expert on the automated test system and on Python for their company. And

so the the basic idea is that we take Python

and we use it, paid application called Squish

that you can download,

and it can actually inject into any QT application

and get access to the public methods of the QT app so that you can press the buttons, spin the widgets, do pretty much anything to the QT app as well as access any public methods that are defined in the C plus plus

both on the QT side and if you like create custom public attributes.

So if we want to add special states to our c plus plus code, we could do that and squish can actually grab those

those methods and attributes and we can access them with Python

through Squish

and test it that way.

And do you know if that's leveraging something like CFFI bindings or c types for being able

to expose those methods from the c plus plus code to the Python runtime?

Well, Squish actually works with several different languages like, Tcl, JavaScript,

and, Ruby in addition to Python. So I don't think it works that way. My understanding is that,

some of the people from the original QT team

decided to stop working on QT and create a form a company called FrogLogic and then release a product called Squish

that they can use to,

make money,

for testing,

say, QT applications.

And what benefits

does Python provide for writing and running these tests,

particularly given the fact that the application is written in a different runtime?

I think that 1 of the biggest

benefits is that,

you can iterate quickly to see if your code is gonna work. You don't have to compile,

you know, at every step to see that this test

work. You can just run it and see. As my my team has discovered,

writing UI tests is tedious, and I'm assuming, but

but if you do enough of them, you start catching bugs as you write the code and, you get that feedback loop going faster, so you don't end up releasing

bad things to the end user. And it's also a good way to help prevent against regressions, particularly

in a UI oriented application where it's not as easy to sort of intuit how the data is gonna flow through the system without actually manually going through all the different steps. Yeah. That's correct.

1 of the big

pushes for getting this automated test system in place is we had a regression team,

and we still have a regression team that it'll take more than a month

to do a full regression.

And before the automated test, it could be up to 3 to 6 months to do a full regression. And now that we have a system to do the do it automatically,

even though it's not anywhere close to full coverage, it has reduced,

the full regression down by, you know, a a lot.

I would say at least 30 to 50%.

And given that it's a UI oriented application,

are you able to adopt any patterns or approaches from the Selenium community and how they use that tool for doing testing of web applications?

I think 1 of our other teams uses Selenium to some degree,

but, I actually didn't get to I wasn't in at the ground floor when the the framework was custom built. So I think it just kinda followed the initial

architect's,

dreams

and, you know, so it's all kind of how you thought it should work, and it's very functionally oriented instead of,

object oriented. And so there's a lot of steps you have to go through to get to certain pages. And

right now, I'm actually trying to refactor the code to make it simpler for our user base to use

so that test can be created more quickly and the code is a little bit easier to understand.

And it sounds like the primary consumer of these test cases is your regression team rather than it being exclusively executed via something like,

CI server. So has that

informed or affected any of the way that you create the tests or

create the interface for being able to execute them?

Actually, these tests do run on on a CI build system. For whatever reason, the regression team doesn't really pay too close attention to that other than to make sure that they are passing. They haven't learned Python yet.

So they don't actually write the test. That's that's kinda what the developers do. And so, you know, we write the tests, and the developers are and myself are supposed to keep an eye on our CI build system and say, is it is it working well well enough to release, basically?

And what are some of the drawbacks

or limitations

that come up as a result of using Python for interfacing with the c c plus plus applications

for these UI driven apps? So I don't really see any huge drawbacks other than, Python is very different than c plus plus to code in. So other developers,

including myself, when we switch back and forth between the 2 languages, we have to, like, stop and get our head back into that language so that we can use it appropriately.

And are there any particular

tools or techniques

or additional libraries that you pull into the Python

that you found to be particularly useful for building these tests

and creating reusable patterns to help with creating future tests?

Yeah. We've been using, the Pillow project, which is the fork of the Python imaging library. We use it to do image comparison diffs and determine if a screen on the display renders correctly or not, and that's that's been really helpful.

We also have utilized OpenCV,

for certain things,

in the display.

Particular stage, and I'm assuming that you have stored snapshots from previous runs so that you can do that diff. Is that correct? Yes. That's how it works. And how much of the

patterns or tooling that you're working with

was

preexisting

or available as sort of reference material or prior art? And how much of it was just stuff that you had to work through as a team to figure out what was working well and what wasn't and what was potentially sort of leading you into a dead end? I would say that most of the framework has been custom built from the ground up. I mean, yes, we're leveraging, like like, Pillow's ability to do diffs with, diff I think it's image chops. It actually has a way to diff images built into it. And we and we also use, opencv to, you know, find borders or

find something on the screen that's that's obvious that you can that it can see that isn't easy to easy to describe in code. So we leverage stuff that's, you know, built into those packages,

but the functions that we write around them are pretty custom.

And what are some of the most interesting or difficult aspects of this type of testing where you're working with these graphical interfaces

in a compiled runtime?

1 of the biggest problems that we found is,

when you run it on different machines,

the controls or widgets will get drawn on the screen at different rates. So we have to write your test to to say is this button here yet or is this widget here yet? Yes, it is. Okay. You can continue, hit next or okay or cancel or back

or what have you. So there's a lot of timing issues we have to work around because of that. If the machine

renders too quickly, we have problems with that too where, oh my, we've we zipped through the through the wizard

and something has loaded before we expect it to load and now the test is broken.

So we have both too fast and too slow issues and so we have to add sleep, so we have to have retries.

And that that's probably, like, the the biggest issue that's been hard to overcome.

And is there a lot of churn in the user interface itself between different releases that you have to deal with, or has it stayed fairly stable

and the majority of the UI testing is just increasing coverage or addressing

new features or new screens that get added in?

Most of the current, I would say most current development goes towards the new features.

If we're touching something that's never been tested before as part of adding a new feature, then we'll try to add coverage to that particular part, for example. So you get a little bit of legacy coverage added as a bonus when you're adding a new feature. And do you have any means of

determining how much coverage you have over the UI components in terms of the tests? Or is it mostly just sort of best effort and you just use some heuristics to determine how much of the application

is covered by tests?

Well, I've talked to our testing department. And, you know, if you look at the regression sheet and compare it to what's in the automated tests,

they think we have somewhere like 20% coverage or thereabouts,

from a UI standpoint, but that's much higher from a unit test standpoint.

And have there been any surprising or unexpected aspects of the problem space that you've discovered or new things that you've learned as a result of this type of work? I think the the biggest thing is just,

you know, I never thought about how how screens will render differently on different laptops

or, you know, different servers even and how if you have a really, really, really awesome machine, that rendering can change dramatically versus,

you know, a really awesome machine without a graphics card. Because when you don't have a graphics card, it gets rendered in software which changes

how the tests behave too. Just just stuff like that you never think about when you're designing your UI, and then you discover that really matters and that's really hard to test.

And did you have a lot of background

in test engineering

or UI development before you started this job?

Not automated testing. No. I have done a little bit of user interface,

work when I worked for, the local county government.

They did a lot of greenfield development in creating applications

and or converting applications from, visual basic for applications

to Python.

So I did get to use DBX Python for that and create

Python UIs, but I've never really done any kind of other,

user interface.

And as you mentioned, you've been working with Python and with the community for quite a number of years. So what are some of the other ways that you

have engaged with the language and community? 1 of the big ways is is writing on my blog. That drew in a community of of readers that have encouraged me to continue writing, not just on my blog, but also writing books. But, beyond that, it also encouraged me to, write on the,

Python Software Foundations blog, which I did for a couple of years. It encouraged me to

volunteer at PyCon,

which I also did for, I think, 3 PyCons at least, maybe 4. I've also, you know, worked with some of the the Python community like Naomi Cedar. I helped, work with him, for DZone on a a, Python

cheat sheet. So it just has opened up a lot of doors with the community and helping them and giving back and, you know, answering questions on COMPLAIN, PY, or Stack Overflow,

or even Reddit. You know? And just I'm working with different people in different spaces and helping them learn Python, and they're helping me figure out stuff that I didn't even know Python could do as well. And so you mentioned that in the process of working with the community and working with Python, it's led you to writing a few books. I know 1 of them was a compilation of some of the interviews that you've done with, various people throughout the community that you wrote an article form in your blog and then compiled into this book. So were there any notable conversations

that you think were particularly interesting, or were you learned something, new that you weren't expecting in the process? Yeah. When I, when I talked to Brett Cannon, he was telling me how, people ask why Unicode wasn't part of Python from the beginning and he points out that, Python predates unicode and that was something I'd never thought about. So that was that was just something, wow I never thought about that. That that makes

a whole lot of sense. Right? It took so long to get that in there. Another good example is I think it was again Brett Cannon who mentioned, you know, Google had asked,

Guido, how much Python do you know on a scale of 1 to 10? He was like, oh, maybe an 8. And, you know, you think the creator of Python is gonna say a 10 or maybe at least a 9, but, you know, he takes kind of the the b root and goes with an 8. Stuff like that. I I really enjoyed talking with Carol Willing about, the Jupyter project, how the Jupyter notebook works because she's on that core team. You know, just how she does just stuff with that team and makes it makes it work. That that was really interesting to me as well. And were there any insights

in terms of ways that these people engage with the community or engage with their work that inspired you

to bring any of those practices back to your own work? They're all very encouraging to,

other pea to new people that are coming into the language. And, you know, I try to do that with my team and the the new people that we hire too. I'm like, this is how you do something in our work environment, and this is how you get into Python.

And if you don't know, you can ask me or I can give you a list of resources.

And I was already doing that kind of stuff, but having the positive feedback loop from some of these

people who work with Python as core developers

is just, like, really awesome to see that they they are also doing that, and it helps encourage me to keep doing that kind of thing. And what are some of the projects that you're most interested in working toward in the future, whether at your day job or in your personal projects?

1 thing that I've been trying to get better about is creating talks to give about Python. I'm kind of a, introvert, so it's a little bit hard to get out there and talk about it in front of people. So this year, I've been trying to get better at that, and

I recently did a talk on Python decorators at a conference called Nebraska code. Actually, I did that last week, and I thought it went pretty well. And I'm hoping to do that again internally at my company

and possibly at a local Python user group. Besides that, I'm also I have all these ideas for different books about Python or Python third party Python packages,

and I'm I'm trying to put those ideas down and get them all kinda outlined to see if I think that'd be worth continuing to pursue

or not.

And 1 of the books that you recently completed was on report lab for PDF generation, and I know you've also written and reviewed stuff related to wxPython.

And are most of these

projects things that you work with in your day to day, or are they things that you set out to learn in greater detail for the purpose of writing the books? In the case of wxPython

and report lab, I use those on my 2 previous jobs. I don't use them at my current job. Probably used ReportLab most extensively in my previous job, and I noticed

that it is I have an article that's kind of an intro to report lab on my blog and that article has already stated in the top 5 for the last few years. And since I know there are no other books about ReportLab, I decided I'm gonna write about it. There's not a lot of good documentation besides,

the ReportLab user guide that the ReportLab company has put out. So I decided to just, you know, take what my knowledge is, learn a bunch more about it, and then put out a book on the topic. And was there anything

interesting in particular that you discovered in the process of writing that book or any practices or techniques as far as being able to

structure or research the content that you plan on leveraging in future efforts?

ReportLab is a a huge

beast of a tool, but it that's also yeah. Probably its biggest, feature is that it can do pretty much anything with a PDF. There were stuff in there that I didn't realize it could do, like, how do you realize you could use it to encrypt a PDF

or decrypt a PDF. It had its ability to create, custom widgets

was impressive. That I didn't realize it could do all those things.

So that was fun to research and try out new examples with. Some of the tools that I decided to to add in the second part of the book,

were really interesting, like, PDF RW, which you can use to,

scale pages from from preexisting PDFs

and then integrate them with your Port Labs so that you could actually, like, take a page out of something that already exists,

rotate it, scale it, make it into a 4 page,

put 4 pages on 1 page. Those kind of things

that I didn't realize were were available to me.

So I I just learned about some tools that I had heard of and I've even used a little bit, but I learned so much more about them. I was able to create really interesting tutorials that I think my readers have really enjoyed so far. And are there any other

topic areas

or work that you've done that you think we should discuss more? I think, so I saw a,

a talk for, again by Brett Cannon at PyCon recently

where he gave the keynote, and he was talking about how it's really good for,

not but not just the core developers, but also new contributors

to just be kind to each other. Because in the open source community, you can get into these hostile,

firefights

where you just say, your code is useless.

Forget you. I'm not gonna, you know, I'm not gonna submit this or and then the person who who should put up that PR is like, fine. I'm never gonna contribute to your your project ever again. Or, you know, you just get the the silly

bitterness if you just had a bad day. Whereas I think, hey. You could, you could make this so much better. And Brett was just pointing out, if we could just be kind in how we interact with each other, not only will you not be tearing down the new people that wanna contribute to your project,

but you'll also be keeping the people who are already doing it positive about it because if they get a lot of negative feedback, you know, those core developers might leave. And I can totally understand that because, you know, my own

programming life is if someone coworker comes up to me and says, oh, man. That code is crap that you put up today for review or, you know, they say anything like it really hurts. Or, you know, if I have a reader that says to me, your examples don't work and they're they're wrong because they're using the wrong Python version or the wrong, you know, w x Python version, That can hurt too because you're thinking, you know, I wrote that. I spent a lot of time writing that book or writing that article, and you're getting this negative feedback that can really kinda knock you down. And 1 thing too that a lot of people don't really think about in these engagements is,

well, the Python community and the software community are global in

a lot of ways. It's also a fairly small world, and you never know when you might end up working next to or with that person at your next job or something. So it's always best to try and make sure you're building bridges instead of burning them. Yeah. That's that's very true. And I actually saw something recently.

Some employers were talking about this on, I think, Hacker News or perhaps Reddit

where they were saying, we go to people's GitHub accounts and we go and check to see what your comments are to people who wanna contribute to your project or how you interacted with other people, you know, when you try to contribute to a project. And if you're acting like a jerk, we're probably not gonna hire you because that's all, you know, that's all out there. That's public information. And if you give us your GitHub account, you can expect us to research that. Well, for anybody who wants to get in touch with you or follow the work that you're up to, I'll have you add your preferred contact information to the show notes. And with that, I'll move us to the picks. And this week, I'm going to choose draw dot io, which is an open source,

diagramming tool that's available both as a

hosted website, as well as, something that you can install as a local web app. And it's

very useful for being able to create all kinds of flowcharts or network architecture diagrams or various things like that. And I just recently used it for being able to put together the network architecture diagram for a talk I gave at the Open edX conference. So it's, you know, freely available, easy to use, and, gives you nice SVG outputs, and you can export to PNG, PDF, etcetera. It has good integration with Google Drive. So for anybody who's looking for a tool like that, highly recommend it. And with that, I'll pass it to you, Mike. Do you have any picks this week?

I've been following the QT for Python project that was recently announced a little over a month ago. I think that's really interesting. It'll give, PyQt a run for its money now that the QT team itself is, creating a a Python binding. I also used the the Jupyter notebook,

for my talk that I gave last week on decorators.

And I just think it was really awesome. It was really helpful for creating an interactive presentation that I could run code in. And, you know, I I just think it was great great experience all around. Alright. Well, I want to thank you for taking the time to join me today and discuss the work that you've been up to and how you use Python in various ways for your work and your personal projects. So thank you for that and I hope you enjoy the rest of your day. Well, thank you for having me.

The Python Podcast.init

Summary

Preface

Interview

Keep In Touch

Picks

Links

The Python Podcast.__init__