Test Engineering with Cris Medina

Hello, and welcome to podcast. Init, the podcast about Python and the people who make it great.

I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable.

Linode is sponsoring us this week. Check them out at linode.com/podcastinit

and get a $20 credit to try out their fast and reliable Linux virtual servers for your next project.

We are also sponsored by Sentry this week. Stop hoping your users will report bugs. Sentry's real time tracking gives you insight into production deployments and information to reproduce and fix crashes. Check them out at getsentry.com,

and use the code podcast init at sign up to get a $50 credit.

Hired has also returned as a sponsor this week. If you're looking for a job as a developer or designer, then Hired will bring the opportunities to you. Sign up athired.com/podcast

in it to double your signing bonus.

The O'Reilly Velocity Conference is coming to New York this September, and we have a free ticket to give away. If you'd like the chance to win it, then just sign up for our newsletter at python podcast.com.

To help other people find the show, you can leave a review on iTunes and tell your friends and coworkers. And please join our community. Visitdiscourse.pythonpodcast.com

for your opportunity to to find out about upcoming guests, suggest questions, and propose show ideas.

Your host, as usual, are Tobias Massey and Chris Patti. And today, we're interviewing Chris Medina about test engineering for large and complex systems.

So, Chris, could you please introduce yourself?

Hi, everyone. My name is Christian Medina.

I'm a computer engineer, and I've been working in

test or some form of test engineering for the,

majority of my professional career.

I've done everything from

simple software tools to complicated hardware bugs and, all the back end

infrastructure

and

planning required to,

have a large project,

go through

a multi month cycle

of, testing.

So Chris, how did you get introduced to Python?

Actually, that was pretty,

interesting situation. So I've been running and talking about things that may not wind up doing this test management framework, and I was, working on some back end stuff. And I was

sitting around the office talking to some coworkers about it, and this, like,

Perl versus Python battle ensued.

My code was in Java at the time. And, so I knew I needed to do some something a little more elaborate. So I've been looking at other things and I knew enough about Pearl that I wanted to try something different. And,

yeah, and so then after that

interesting exchange, I was like, oh, yeah. Yeah. Maybe I should give Python a look here.

And, never looked back since.

It's pretty cool.

So to get us started, can you share your definition of test engineering and how it differs from the types of testing that your average developer is used to?

So testing in general is is a pretty wide field that encompasses a whole bunch of stuff, like all kinds of different methodologies and ideas. Everybody has a different way of going through everything.

But, you know, if you think of things that you're at the very minimum where there's

a a single developer that's focusing on a particular function, they've written,

doing things like, say, unit testing,

which for the most part is is going through and verifying, like, all the all the specific,

methods or whatnot that they've developed.

Things start getting more complicated once you move beyond that step and start looking at how that piece of code

interacts or integrates with other pieces

across your system. And so

it it's important

to,

make that distinction between,

you know, just testing just to, like, poke at everything that you have,

which, like, I'd say I'd call that kinda like a brute force. And, like, the actual

engineering approach where you have to, like, make a decision of, like, how do you how you go through things. So, like, in in my opinion, most recently, a lot of focus has been around

unit testing and continuous integration

mostly because of, like,

the benefits we've been able to get from

from automation. And all that stuff has a great value, especially when you're doing,

lots of builds.

So you can find out where your code breaks and all that kinda stuff. But, you know, when you when you step back and you look at the larger process of how your how your product interacts with different things in your environment that your users may be involved with, that's when you start getting into,

actual engineering. You have to to me it involves

the the larger process by which you you analyze the system,

its design goals, its objectives,

the intended use.

You contrast that against the actual patterns that

your users are are following when interacting with it and the side effects of all of those functions.

And then you, as the engineer come up with recommendations on how to improve the assist upon the system, whether it's by, you know, writing bugs or just communicating with development itself or to change your actual

goals of where you think you could take your product.

Right? And then having figuring out how to do all that stuff is like a large process that involves

looking through design specs, reading industry standards,

identifying your marketing requirements, looking at compatibility goals,

and your intended user experience,

which then, you know, puts everything together for you to come up with, like, a number of tests usually in the 1, 000,

likely in the tens of 1, 000,

that verify the behavior or the function that you're looking for. So to me, the the actual

the actual engineering step

is is is the creative step of actually designing a test, Right? You're you're not

writing code,

you're you're not,

you know, as as a base function, you're you're designing you're trying to figure out based on how

someone's

already existing code or

or system

functions, how best to approach testing it such that

you can you can understand

its its intricacies and how it ties into other aspects of your product and its environment,

how to, you know, how to

define your suite such that it's,

you have an individual test that's repetitive as it needs to be. Because sometimes you want things to repeat a little bit, but obviously repeating things a lot is a bad thing. How to

and how to figure out how to how to not get sucked into all the

possibly millions of permutations that your product might be supporting, especially if you have like open standards that

involve

allowing you to

to, like, say on,

on some systems,

run through a, compatibility matrix of things that you're supposed to work with. And when you start thinking operating systems and drivers and things like that,

you know, you gotta figure out how to

split things up so that such that you can focus on

on certain areas specific to the time frames that you have allotted for your for your testing. And and and when you do that, you gotta kind of,

you know, some people do this formally, but you gotta

figure out how to make it so that your tests

find,

failures

quickly, sooner

rather than later because

the sooner you find them, the cheaper it is to fix them.

You have

to then like with with with an idea of what you're gonna do, then you gotta look at what what infrastructure and hardware and what your test bed looks like so that you can then

take those,

those resources,

you have available and, apply them the most efficiently

to the testing you gotta do. And then and and efficiently,

you know, actually means efficiently. It doesn't mean, you know, just maximizing it because

sometimes you want like actual representative workloads where

you don't necessarily

want

to

maximize every single

use of of whatever your product might be.

And when you're,

you also gotta look at debugging

and, you know, writing,

writing bugs and identifying where those failures should be. Which bugs should you write? What should the severity of these be? What are the impact to my customers?

Given this list of bugs, which ones have priority

to, get fixed? Which 1 should I just, you know, given my time frame, which 1 should just become a limitation?

And things like that.

Yeah. And also with all the different

moving parts in any system, the understanding the various failure modes and how those failure modes can interact is

incredibly complex and just becomes exponentially difficult to comprehend. So figuring out where the different touch points are between the systems that interact with those various failure modes to make sure that you have those cases covered is critically important to making sure that you have a robust

and fault tolerant system as well.

Great. And and then like some your failure modes have failure modes too, right, when you when you when you start getting into something really, really complicated. I have spent, quite a bit of time doing,

you know, reliability testing, availability testing. Testing. So, you know, you where you gotta verify that the system you've designed actually fails in a designed way such that your customer can properly

manage all that. And then getting usually when you when you run into folks writing code, you you you don't really write the code to to handle you know, you handle your exception fairly quickly. But, you know, how are you going to communicate that to the user? How is the user going to recover? A lot of people kind of forget about that part. And then, when you get into these weird situations, you wind up with,

some interesting exchanges of, well, that only happened

1% of the time. There's a probability.

Should I bother fixing this? You know, it gets it gets pretty interesting.

Yeah. Are there any industries where this kind of testing does not become necessary very often?

Well, when you're making something really small, right,

that might not be a thing. But when you're, you know, when you're involved in a in a system that has

a high level of integration with some sort of third party module. And that third party doesn't have to be like somebody in a different company or organization. It might just be somebody in

different team, where you don't really control the release cycle of that piece of code, and you you're just kinda

you're just kinda using it. Right?

You know, you write

the the a simple thing a simple way to think about it would be you write an application, and you're just just using whatever Linux kernel version you're on. But if that Linux kernel revs,

then you're gonna have to go back and do some work. Right?

And if you if you if you run through and do and do some testing on your application and you realize that there's a bug in the Linux kernel that requires you to to make a change, you can't expect the folks that that, you know, that wrote the kernel to go do that just for you. Right? There's a whole other,

a whole other system that you're, you know, just a user of.

Maybe at some point in time they'll integrate it and pick something up, but then you're gonna have to handle that use case. And so identifying that type of stuff is is fairly important. Like a

a lot of folks,

you know, it's

it's you don't really think about it ahead of time,

especially when you're talking about something like an operating system. And my my experience has been

the majority of my time has been in servers,

you know, where you have

this this huge

system where it's not it's not just hardware.

Right? You have you have processors, you have memory,

you have

you have buses in which you

you can plug. You know, there's it's a standard slot, so you can plug adapters from dozens of manufacturers.

Some of them comply to the actual standards. Some of them do not.

Some of them just have bugs that don't that, in in their implementations

of things. And so you need to,

your stuff needs to be able to identify those systems, you know, and that's, you know, that's what gets into compatibility

and permutations because, you know, when you when you start adding things together, you end up with this this huge mess of stuff that even the mechanicals of where you put things matters. Like, you know, can can your server fit

into the proper,

you know, proper measurement, like use the units, height of your, racks?

Like, is the

can any card fit into the area that I made for this slot? And, you know, there are some things some cards that might have,

you know, a few extra things in their PCB that make it just a few millimeters too high and you can't use it.

You don't want the customer to find that.

And, and then there's, like, all the, like, underlying things that that, like, complicate that even even more. So, like, a server, there's that's just the hardware, but, like, a server is functions

through a bunch of firmware,

which, you know, UEFI these days, which used to be BIOS. Right? And there's usually some form of diagnostics or systems management back end that, enables the whole thing. And the systems management takes care of, you know, should my server continue running when,

you know, the temperature is too high? Or this this adapter,

or more likely this memory didn't generate a default. Should my server

reboot?

Should I just and then these days,

processor architecture and memory controller architecture gets even more complicated to where,

you know, a failure shows up in a DIMM and there's, you know, that gets detected by a memory controller, which then gets forwarded to a piece of

firmware running in UEFI, which then,

puts entries in a certain

location to notify your operating system, which is then running some driver that's verifying all of these,

checking all of these things. And then maybe you allow the operating system to make the decision. And you just, as the

designer of the platform in which dot OS is running, you just say, well, it's up to you what you want to do with this. So like a lot of times, you can you can design for a lot of that ahead of time, but most of the time, you wind up finding all these like weird

interdependencies

and intricacies while you're actually running tests.

Yeah. And another thing too is that

most people tend to think of testing as something that you do

once your code is written and before you release it. But there are also runtime tests that you can do to make sure that your that the environment the application is running in fits your expectations so that when you first release it, you're not then surprised when, you know, 2 weeks down the road, it it actually hits 1 not often used subroutine and all of a sudden that subroutine blows up because something that you were expecting to be there didn't actually exist.

Right. And once you get things sufficiently complex, a lot of times, you just kinda have to have a method in place for dealing with that, with those weird exceptions that you just find later because it's just almost physically impossible to be able to go through all of it. Right?

You know, there's

a when you when you, like, multiply all the possible things that you could do in some of these systems, you put,

different OSes or different drivers or different memory or different processors and all the different cards that you can stick in there, and then all the different environments in which you can use it. And then, you know, if it's too cold, maybe the the,

electronics,

created had a race condition because of some weird

design flaw that you didn't even notice. And so, obviously, there's there's testing specific to to those extremes, and and, you know, there's testing that goes into,

vibrations

and,

drop tests and all kinds of stuff as well when you get into, like, physical systems. Right? But,

you know, if you stay

on on the software, a lot of it is just like how all these things interact and the the more and, you know, as we progress

and things become more and more open, the more and more open APIs that you use, the more and more weird situations you can get into. Like if you write some software that uses the Twitter API to go look at your timeline, for example, that doesn't necessarily mean that that piece of code is actually gonna function if you, like,

put it against,

the fire hose where it's, you know, getting a ping,

100 of things a second. Right? So

I also think testing can can be a guardian

against

unintended

complexity and the negative effects that that brings to bear. Right? Like, you think I'm writing just writing this little script and then suddenly that little script is has grown and it's evolved into a whole, you know, production application or subsystem. And and, you know, you think, I don't need to test this. It's just it's just a tiny thing, but

really it almost never is that way. Right? Like, if any code that works

is destined to grow or evolve at least. And so testing helps you guard against that going in a really bad direction.

And and that's where the engineering steps is is really important. Right? If you,

I can't tell you how many situations I've been in where, like, we received 1 piece of firmware that's, like, oh, we just changed these couple of bits here. Don't worry about it. You don't need to test anything else. And we're, like, well, but those couple of bits get, you know, involved in this function here and that function there and that function over there. So we're gonna have to test all this stuff and give me 2 weeks. And then they're like, no. No. No. You don't have 2 weeks.

So

you get into

an interesting situation where you have to make

a choice, right? And a lot of it is an engineering decision

of

what are what's the risk? What's the likelihood? Right? And in order to do that, you need to be able to have like a good picture of

of what it is that that you're testing. And the more and more complicated things get, the harder and harder that is.

So how and where does Python fit into the kind of testing that becomes necessary when you're dealing with these kinds of complex systems?

I've

used, seen,

or have heard of Python being involved in almost

every aspect. Right?

You know, you can everything from, you know, designing the systems. Right?

If you're talking about, you know, a server or, you know,

a game or something like that, you know, a lot of these,

modeling tools

have direct ties and pipelining through Python to move things around to come to, like, process the data, convert it from 1 thing to another.

I know that given behavior driven development and test driven development,

you know, Python will be a great asset for

for,

putting design definitions and specifications

together.

I haven't actually seen that happen, but,

you know, things like Cucumber and although that doesn't specifically have a a Python module as is, is a, you know, is a is a good step forward of of of a way of defining

a

a a machine readable specification

that you can then later maybe write some

more

elaborate AI systems to kinda decide

for you,

based on history,

how to

properly test a function. Right?

There's

there's items involved in in

you know, I've I've done test management tools,

which involves everything from, like, recording what a test is and its procedure, the expected results

to, you know, what, you know, what what the actual execution of the test was and what configuration you used to do it and what the result was and keep that whole history there for future reference

so that you can then make decisions and for current reference so that you can,

you know, figure out where you're at,

with your process.

And then in order to do to execute tests, if you're on the software side, there's,

you know, there's Pytest, there's NoTest, there's Unitest,

to help with script execution.

Right?

If you get into more elaborate stuff, there's hypothesis, which you guys had

had the designer of that package on. Right?

That's pretty cool. I've actually hadn't had a chance to fiddle with, but that's that's a great thing to to manage,

different configurations

using the same, you know, the same type of test and things like

that, going into edge cases.

You can look at message queuing using using RAB and salary task queue

to,

go in and and

and do automated execution. You can look at builds.

You actually wind up using

a lot

of a lot of specific

modules that come with a standard library like OS, subprocess,

threading, and multiprocessing

to, like, interact with a lot of the things that you actually wind up,

executing. Because,

especially if you get into

low level things.

Sometimes Python is not the best tool. So you wind up writing something in c. You wind up with c types.

You do

heck, I I wrote a DHCP server once in Python

just to prove I could do it. I wound up being pretty cool. It gave me this whole sideband interface where I can just go in and and change,

the the my network infrastructure

fairly quickly without having to deal with anything else and then set my systems up for, OS deployments.

There's

you get involved into

in customer in simulations.

There's a

there's a few tools, different companies

based on, you know, the the products they make will will have some sort of simulation system. A lot of them will use Python scripting to help you

get into that. Like, you could get,

like, Intel, for example,

if I remember correctly,

is,

has a

a way of, like, sim simulating at least processor behavior. And so you'd be able to, like, write some software that would run on your processor, and you don't even have to actually physically have it. Right? There are you can obviously, restful frameworks, anything that you you can put in front of your, your other tools like

like Flask and Cherry Pie and Django and that type of stuff is is fairly helpful to, move things around in a very complex,

systems. In my environment,

we we moved around a little bit, and we wound up in this

new site

where we had to consolidate,

like, a whole bunch of different

test teams and test tools into,

you know, 1 1 set of infrastructure

running across multiple labs. And so we wind up picking

REST

communications,

using

JSON

and data exchange and that type of stuff to be able to talk to all the different individual systems.

So so, you know, the DHCP server, you can tell it what IP address to give a particular MAC address, and that can be driven by an OS OS deployment system that might be written in,

Pearl or whatever,

just using rest. And then when when your OS boots, you can notify something else to

to update

your list of resources that are available for testing, for example,

through through similar types of systems.

There's performance monitoring. Monitoring is a is a another thing that you can use, but mostly by driving some other tools through Python.

And then, you know, you can

after you've done all your stuff, you you have

you have Python is great for analyzing results and

incorporating

other

capabilities

to see throughout history.

Is this test always passing? Is this test always failing?

What what does my

what what does my my bug list look like? Which

when when this piece of when I got this new build, did this generate a whole bunch of new bugs?

How many, you know, how much how much time did I spend do I spend every time I I get a new build? All kinds of stuff.

Yeah. There's definitely a huge breadth of concerns when you're dealing with any sort of large scale testing. Basically, any small system can quickly become a large scale testing if you are in an organization that puts proper emphasis

on the fact that having well tested systems

are beneficial to their bottom line.

Right. And that's always an interesting thing. Right?

It it is it is a balance to try because

if you just let the test people loose, you'll you'll wind up with a big test budget.

Right? Because, you know, we wanna test everything. What do you what do you mean we gotta risk assume? What is this? But, yeah, it's always an interesting balance trying to figure out, you know, where where do you where do you draw the line? Like, how do you split it up? What how do you how do you decide to move from 1 phase to another to another to final product release with, you know, as small as budget as possible? Right?

Yeah. And how do you figure out which areas of the system are most critical to test? And are there any areas in that decision making process that Python can feed information into that process?

A lot of that is highly dependent on what it is that you're doing. Right? And a lot of it comes from actually,

being able to have a good picture of all of your interactions.

If you're talking about like and so an an interesting thing as I'm thinking about here, if you're writing python code, like if you're writing a a complicated application of python code, you can you can write Python code to detect, like, all of the different things that might be, you know, important to you

that's already,

written for you using the standard this module and and looking through the the AST.

Looking at,

figuring out what to focus on, a lot of it is data analysis. Like, you can look at, say, you can use you can use Python to look at,

say, social media and pull information out of that and see where most of your users are having issues. And, you know, just run it, use the standard Twitter stuff, run it through NLTK,

do a quick analysis of, you know,

words that come out more often and relationship between them and things like that. And you can say, well, hey, you know, my product's already out, but obviously, I need to focus on these things over here because a lot of people are complaining about this. And then you can look at,

performance metrics,

as well if you're if if that's a thing for your

for your for the software that you're delivering. And of course, well, yeah, it kinda covered that. And pandas and scipy and all that type of stuff is very useful to be able to look through all that.

So you've listed a pretty large number of different tools and libraries.

If you can just pick a couple of your favorites that you use pretty regularly. And then also, what are some of the areas where the available tooling that exists in Python falls short for your purposes?

Actually, I use cherrypi a lot. A lot of people aren't aware of it, but it's kind of like Flask. I'm not sure if it was there before Flask, but,

it's pretty much

a way of mapping HTTP requests to functions. I use it quite a bit for putting a quick REST interface in front of something.

And, Celery is great

for,

getting things automated and executed fairly quickly. And so that I guess those are the the major things I've been using most recently. Since I've been doing a lot of work in in the actual management and status reporting tools, I've also, like,

been involved with like the post grade modules and and SQL alchemy

and and all that type of stuff. I I find that I keep reinventing the wheel a lot with ui development.

And I'll say that's that's 1 of the things where I think Python is still kind of needs to move forward a little more. There's quite a few libraries and they all do 1 specific thing. But if I'm

trying to put together a site that needs to be used,

by

like a whole bunch of different browsers and,

needs to like back end load a whole bunch of

other

not so easily accessible

tools

or

or things that go off in my test

structure

and queuing and things like that, it starts getting interesting. And and, of course, keeping track of all the data and being able to show it in some pretty way, it all gets interesting. And I always wind up going to web

instead of, like, what would probably wind up being some better if I could just, like, write it all in Python and just get it over with. I wind up using a lot of JavaScript in,

in the back end.

And, sure, you can put Django in front of all that stuff, and and that'd be great. But you still gotta write all your JavaScript to, like, make your charts or to,

interact with your systems.

There are some offerings

that let you use Python to generate the data that gets shipped up to the front end. But at the end of the day, yeah, if you wanna build a widely available

UI, you basically, your default choice these days is the web. As far as using Python for doing that visualization,

basically, there's Bokeh as 1 option or there's,

Vincent if you wanna translate it to Vega and eventually to d 3. But yeah.

Yeah. I found myself recently.

I've gone through different organizational changes lately, and, I've I've had to, like, write very similar websites over and over again. And so I I finally got tired of it a couple months ago. I actually started writing a Python module to write websites for me. So

so, yeah, I actually

have a couple blog posts on that. I might actually wind up, posting it up soon on GitHub.

It it could be interesting. I don't know. I just wound up I wound up wrapping Bootstrap into,

a little Python module and then putting,

using WebSockets to communicate

with the UI, and then, like, all your back end could just be in Python. That sounds pretty cool. We'll see how that works out. It's I was, at Python,

with Russell,

the with the you had him a couple of shows ago with the,

Russell Keith McGee.

Yes. He's done a great job with all this UI stuff. It's just very interesting

and

how the deep dive that he's done to try to get all all all these common interfaces going and and whatnot. So hopefully that pans out for him. He's done some pretty remarkable work there. Listening to him dive deep on how it all fits together, I was pretty awestruck.

Yeah. It is definitely

a lot of work.

I think that is 1 of the underappreciated

projects

in Python open source today. Just simply because of its depth and and what it makes possible. I I really wish that it got more exposure and more love. Yeah. For sure. And given the breadth and depth of concerns that are encompassed with testing the various components of these large systems, what are some of the ways that a test engineer can get a good high level view of the overall state and what the sort of historical trends have been in test successes or failures

and, getting a good snapshot of the current state of everything that is under test?

So so that's a very interesting question. There's a there's a lot of tools out there, actually. Some of them really old, some of them fairly new that are meant to do this.

But nobody really kind of agrees ahead of time what it is that you're counting. They just know that you're counting something. So, some some tools wind up counting bugs. Some tools wind up counting test attempts

or, some tools count only successful test attempts, things like that. And then some of them are just so customizable

that you wind up having

to have an engineer with the sole purpose of maintaining the tool that you bought from somebody else.

And then only that guy really actually knows what it is that you're monitoring. It's kinda interesting how that whole thing has played out. I've I've written some tools myself,

and it's highly dependent on what your test organization

actually values. And so again, like, for a while there,

I was in an org where everything was about test attempts. Like, are you 100% attempted? And and that's good. That's that means you've gone through all of your testing. But in the grand scheme of things, it doesn't really mean anything.

Because, you know, if you had, you know,

a 1000 tests executed and you executed 999 of them, say 999

of them passed. So So you're 99%

passed, 9.9%

passed, and you're missing

1 test. And that 1 test is, can my can my server power on? It doesn't you need some qualification

behind the numbers. Right? So,

different I've been in other organizations where where they do,

defects. And defects is interesting.

I think it's it's a bit better.

So that way you track

you track, are you complete with your testing?

And then where are you at with your bug status? Are you are all your bugs disposition? Meaning,

you know, do you have

are they closed?

Or, or do you have an action plan on what you're gonna do with them over time? Right? And, that's an interesting metric, but that works mostly for

what you've discovered. Right? Which if you have proper testing, then that's theoretically good enough. I think over the years, I've developed like, in my head mostly because I haven't been able to convince to convince many people of this,

it kinda like a hybrid approach where you wind up with you report your status based on what it is that you're actually

testing. Like, so if you had like 10 tests and your product is really, you know,

5 functions, you report status based on per function. So like the status goes for the function. So you have like an overall

idea of where your product is but you can also drill down into the individual

things that you're supposed to verify and where where those are at. And then that way, you're qualifying

the numbers.

And then, you know, obviously, on top of that, you add

defects and

and where that's going. But it's very interesting. There's not really a standard.

Different people like doing it different ways. It's a matter of figuring

out what works for your organization.

But the important thing in figuring that out,

it should not be by

who best interprets the data or, you know, how do the numbers look better. It should be about what's the the end quality. Well, like, how are my customers liking this?

Yeah. When when you're digging that deep into the stack, it can definitely be easy to forget about the fact that the entire purpose for doing all of these things is

to achieve the specific business objectives for the systems that are in play. Because if all of your tests pass, but none of them actually pertain to the end use case, then it doesn't really do you any good. So I would imagine that having a system where you can categorize the various tests by

the

end objectives that they facilitate for the people who are actually using the system would probably be 1 beneficial way of seeing at a, you know, at a high level. Oh, all these these are the tests that passed for the happy path. The ones that failed are the ones that cover these, you know, various edge cases. So most people can use the system, but if they're trying to do this particular thing, then they might have some trouble. So I I realized that that's probably difficult and that really

being able to manage that information

is, you know, another

avenue of complexity.

Yeah. It gets really hard. Right? Because if you go down the developer side, right, you wanna track, like, development functions. If you go if you abstract out to to the product that you're putting on to the market, then you wanna track marketing requirements. But then you realize that your marketing, you know, requirements or your or your product goals are too abstract. And so then you're like, well, how do I well, this is part of that, but that's also part of that same goal. And this is also part of like, how do you, like, put it all together? And and that's it's it's really important

before

running a test on a product or at the very minimum before, like, even deciding what your test organization

should be doing to, like, sit down and have

a conversation about what you're gonna be measuring because as we all know, you get what you measure. Yep. And, that also brings me to my next question. You know, as you are tracking all of this information and categorizing it in the various ways,

how are some of the ways that you can distill that information so that it can be presented to other areas of the business who might not have a thorough understanding of all the minutiae of what the various tests are certifying?

Right. So,

most of the organizations that I worked in

use some form of project manager

to communicate

up to the business,

where things are at. And that's interesting because that person communicates what he the information gets filtered because you can't possibly,

yeah, go through all the like you were just saying, through through all of the low level stuff. I think in the end, the the better solution

which I've kinda gotten to that most recently

is to just have all of the data. Have it always live in some application,

may whether that's a website or not. And then,

you gotta have different levels

of abstraction. You're gonna have to design that website or that application so that you so that works for you. And that's where the trick is, right?

Figuring out the data model that fits

your your product or your organization

such that you can collect your test results and be like, well, I had a, you know, a 1000 tests. So many passed, so many failed. These are defects. And then find out some way of abstracting that up to, well, product go

x actually passed,

you know, these things and failed these many things and, you know, these are the defects. And then abstract that up to, yeah, we're go or no. Right? And,

it's it's a challenge. There really is no silver bullet for that. And and especially depends on how much the business,

wants to get involved too. Because, like, some folks really, really wanna look into, like, the the deep end. And and,

so a lot of that a lot of that I've accomplished through some form of

hierarchical views.

There's

there's some interesting,

charts if you look at d 3 and and their gallery.

I I like their sunburst

because you can it it helps you do some good drill down where you can, like, kinda abstract things out and and to show that hierarchical data. Some of the other type core type diagrams are actually pretty good for that as well where you can just show all the individual type

functional

stuff and, like, group them together

and then, like, relate that to the other,

you know, say, to goals and things like that and see how how everything is going. Back to the you have to you have to decide what you're measuring. You also have to decide

how you're gonna look at that data ahead of time because otherwise

things are getting pretty complicated, especially

if you change it in the middle of your product cycle.

So could that information also be used to provide a compelling business case for the resources required to test properly? I mean, I feel like we've all been in the position of, you know,

working in a place where

you have to fight for every hour that you're given to build things.

And a lot of the times, like you said previously saying, you know, give me 2 more weeks to write a really good test harness

just isn't gonna fly.

Right. And it can. It all depends

on how well you can tie into

the other side. Like, so you shift your product, and then you have support. Then you have

warranty costs. Then you have

customer complaints and things like that. In the organizations I've worked in, there's, like, field replaceable units. So like and and customer replaceable units and engineers that go out to do all those replacements for you

and, you know, training costs or like so for like a good example when you when you go down training, if you if you make a product,

and while it might be good for the business, if you if you make a product and then you also have to make like a really extensive training course and at the end of it, your customers still don't know how to use it. You know what I mean? You can look at and a lot of that data

actually exists somewhere in some in some database or in someone's spreadsheet. And you'd be surprised that how much people

still use spreadsheets these days,

especially having,

things like Python for easily, like, keeping track of all this information. But,

if you if you actually gain access to that other side of your business, you can then start incorporating all that and say, well, say, hey, this bug over here that's involved with this system is very similar to a bug that we saw on another another version of our product that was in that same system and that generated

number number of

customer complaints, to what cost, of number number of field replaceable units, yada yada yada. So if it's

having the data is the important thing. And a lot of times,

you don't get access to that. If you if you have a big enough organization, a lot of times, it's really hard to get access to it. You wind up having to, rely on just, like,

making internal contacts to just, like, run around and, you know, make an effort to go and find it. Right?

And

another issue that frequently arises with

test infrastructure

is that it can

very often become unwieldy

and difficult to modify just like an improperly structured application. But because of the fact that a lot of times tests end up becoming a secondary concern

and don't become

So you

So you you really have to plan for it. I mean, I really I hate to I hate to say that, but you really just have to plan for it. And a lot of times you're just not given that time to just sit back and say, well, hey, this product is gonna involve these things. But,

you you can usually schedule sometime

in maybe say once a quarter or something like that where you step back and look at your infrastructure itself and say, well, we're mostly running on these things. What do we anticipate

in our portfolio coming down the line?

Oh, well, this system is gonna have

200 gig networking. So we'll let's, like, adjust for that and see what we need,

to add.

But, so you need to be involved in future planning,

and you need to make a conscious effort to go back and say and revise what's going on.

And it's very easy. It's happened

several times

in my

experience. You just kinda put your head down and just go with what you can

and then you wind up in these weird situations where you just don't have the infrastructure to verify things where it's taking too long because you you have to,

tear down what you have because you just don't have enough of what you need. There are with in the near future, some of that

may might be alleviated a little bit, at least in the industry I've worked in due to, like,

software defined networking,

containerization,

and virtual machining, and things like that. I mean, containerization and virtual machining has already like, we've already been making good use of that to help alleviate some of the craziness.

But when you're talking about

extended hardware setups, like with the with the server, I remember

back back in the day when when I used to do some integration test some integration testing,

you know, we supported a whole bunch of network cards and storage adapters. And all those storage adapters supported,

all kinds of different enclosures that you can plug into that. And some of them were 4 port adapters.

So that's 4 enclosures per adapter. And then sometimes the servers that we were on supported an external

enclosure of PCI slots. And so then you'd wind up with like these insane configurations,

where you're sitting with like 27,

adapters plugged into your system

trying to, like, do different things, trying to make sure your system is still up because it's a supported configuration,

trying to get into the operating system, and trying to go through all that. And, like, setting that up, a lot of times, that's another thing. People count,

execution time, but they forget about setup and teardown.

And,

you know, when you when you go off and you have a conversation, oh, but it's just a 6 hour test. You know? Why is it gonna take you to 2 days to do it? Well, I have to install all this stuff, hope it boots, and then, like,

properly configure,

everything to the specification of the test. The test will take care of, like, you know, a 100 functions,

but, you know, it it takes a while to just get it going. And

in order to actually wind up being able to execute all of that, you you really just have to plan for it.

You have to you have to

know ahead of time the server the the product that you're gonna be testing, what its capabilities are going to be, then go back and do some planning to update your infrastructure if needed.

And then plan for

when when requesting your time frame, your execution time frame, to be something big enough to handle

all of the

setup and teardown that you're gonna have to do if you have to iterate on things quite a bit.

And this applies to

other areas of engineering as well. But given the low level nature of the this kind of work, I imagine that maintaining proper visibility

of what's being done can be pretty difficult. So how do you make sure that management

and

the other areas of the business can properly see and appreciate your efforts and the impact that it has on the overall,

success of the business?

That is very, very hard.

That's all I have to say.

You know,

in the grand scheme of things, the best thing is always to be able to tie what you're doing to the customer. Right? Because then you can bring back and say, well, hey, you know, a lot of times you find yourself usually, when you're doing these things really complicated, you find yourself in in situations where there's gonna be some special bid from a customer that wants some specific configuration or it's it's an important enough

and in those cases, like, if you can go back and say, well, hey. You know, I I I And and in those cases, like, if you can go back and say, well, hey. You know, I I I got us these guys, and that equaled so much revenue. Or, hey, you know, we found

this bug,

and given the information I have from this other customer, you know, we would have lost them. And, you know, so that's as long as you can tie it straight to

to customers and revenue and customer satisfaction,

it's it's a lot easier to go back and and have

upper level management able to to appreciate what you're doing. And then when you're talking about the lower levels of management, a lot of that is is counting. And what I

mean by that is, when you go into unit testing, you talk a lot about, you know, you get a new build, you run all these tests with it. Right? So

a lot

of larger,

cycles or time frames in which you're you're verifying

a certain level of complexity, a certain level of function,

for your product. Maybe not, you know, end function. Maybe, you know, we're we're close enough to like, you know, turn on the big stuff. Right? You wind up going through a lot of iterations,

of builds.

And so a lot of times,

just simply being able to show, hey, you know, I was able to to run through, you know, 500 builds in a time frame of, like, a month. And that and that meant verifying this and that and that this thing was good for every single 1 of those builds. And the output of that was, you know, that

my immediate test org did a bit of a bit more work but the expanded test org never got those builds because of the work that I did, you know. And that equals

so much money or so much time

saved in, you know, wasted testing or or wasted

just putting the firmware on on your product or just swapping over to the new version of your application, that's a good way of looking at things as well. But you always gotta tie it to something and, to something related to customers

or or time. It's a skill, I have to say. There are people that are excellent at that and there are people that just don't even think about it. But, it's something you develop over time.

So for anybody who's interested in

moving their career in the direction of test engineering, what what suggestions or recommendations do you have?

Oh, that's interesting.

You know, it's I thought I don't know if there's, like, even,

like, a path or even classes. And

I'm assuming that by now there should be, like, some test engineering courses in in universities and stuff that go into to more detail.

I mean, there's there's a lot of literature

out there on a lot of different methodologies,

especially if you're in software. You can you can read up on that,

and that'll give you a good idea

of what what types of things

to think about. You know, there's different there's so many different aspects.

Everything from

from build verifications

to

to unit testing to continuous delivery to functional verification and integration

and

behavior

driven

design and how that impacts testing and test driven development, which is a big thing these days,

are also

the, things you can read up on.

As far as,

you know, it's just I can't say that there is

1 thing or like a a certain

all all the stuff I just built up over time in in experience and just like getting into it. I have found that it's very interesting

in the sense of you don't only have to know how to, you know, decide how to test. You also have to figure out how your product works. You need to be a tech you need to be a developer. You need to understand

what, you know, what the implications of certain code is,

so that you can properly design your tests. Yeah. I I I don't really it's I don't really have anything. It's very interesting, you know. I was looking at the more I think about it. I was like, there's there's gotta be like a like a an overall

you know, there's maybe there's a couple books specifically

on just test engineering and and and approaches to that, but I I don't really know that. Yeah.

Well, like I was saying, before our interview, I'm looking into maybe possibly putting something together myself, so we'll see. There you go.

So is there anything that we didn't ask about that you think we should have or any other topics that you wanna bring up? No. I think we're pretty good. I'd say we we covered quite a few things here. So hopefully, it's been, useful for for everybody.

I know that I definitely learned a lot more about the kinds of testing that go on

in various organizations.

I work as an infrastructure engineer, you know, as a DevOps engineer. So I've been working on figuring out the best ways to apply tests to my infrastructure. So even just that, it gets complex enough when you have multiple different types of services interacting. And so Yeah. Fun stuff. So for anybody who wants to follow you or keep in touch with you and, see what you're up to would be the best way for them to do that?

Twitter is probably the best way. I am at,

try accept pass.

And with that, we'll move it on into the picks.

My first pick today is the Samsung Galaxy Tab S2. My old tablet

stopped working

a couple of weeks ago, so I had to get a replacement, ended up getting the Galaxy Tab S2.

And I am

very happy with it.

For a while, I kind of resisted going with the Samsung devices just because of the

TouchWiz UI that they lay on top of it. I always wanted to go stock Android, but

the UI version that they have on this is actually very well done,

very unobtrusive,

adds a whole lot whole lot of niceties that just make the overall experience with the tablet very nice. So definitely recommend it for somebody who's looking for a tablet to use in their day to day. And my next pick is the Anker

Soundcore speaker, which is a portable Bluetooth speaker that's got 24 hour battery life.

Sounds really good. It has the volume can go up pretty loud, and it still sounds nice and clear. So I've been using that for listening to music and podcasts

and to hang that up to my new tablet.

And it's pretty inexpensive too. So definitely worth taking a look if you anybody's looking for a Bluetooth speaker to be able to use around the house or when they're out and about. And with that, I'll pass it on to you,

Chris. Thanks, Tobias. My first pick is,

a YouTube channel that I picked before, but I'm picking a particular episode. It's called this episode was written by an AI.

Essentially, the the folks from the idea channel, the PBS idea channel, which I've gushed about before on here, took their entire corpus of scripts

and fed them into an AI as training material into the torch system with, written in Lua. And

what got spat out, they they used as a script for a whole new episode.

And

the host,

Mike, I can't think of his last name. He read the script and it's generated by the AI and it's hilarious. I mean, it is it is really funny. And then also, if you've ever watched the show, it's always interspersed with all these really kind of interesting visuals

and, internet memes and things like that. And they kind of stayed with that theme and memes to what the AI had written. It's just, it's funny and fun and worth watching. And my last pick is something that I don't usually do. I'm going to pick 1 of my own blog posts because

I feel like I really had something to say this time. It's called the 3 R's,

And it's

about 3 aspects I see to being really key in

making and keeping your software organization strong. So I'd be really interested in people's

feedback or comments on that. Thanks. And, Chris, what kind of picks do you have for us?

So

I got a 1 Python module that I I I talked about earlier. It doesn't get a lot of love, but I I I really like it, which is cherry pie. Definitely should take a look at that.

It's,

maps HTTP request to functions and does a whole lot more than that.

It's something like Flask, but different.

There is a

my other pick is, there's a service,

that was that's called etcd,

that's written for

it was written by the core OS guys, which are involved in,

is core OS is like a,

a Linux OS for the sole purpose of running Docker containers.

And etcd is a distributed key value store.

I haven't actually had a chance to

get into the nitty gritty yet, but,

it

it's

it it provides

a a a restful interface for you to just,

put and get, information

out, and it's distributed. So you'll be so you're automatically backed up over whatever systems are running it, which is they use it to manage their core OS deployments.

I wanna use it to to manage

resources and task,

for which to

decide which tasks to queue on them. And I got 1 more thing which is a book called Thinking Fast and Slow by Daniel Kahneman. If you haven't,

read that yet, it's, very neat.

It's a study on how your your mind, how you process, and how you think,

and how you process information.

And the the differentiation

between

what he calls system 1, which is, like, your your quick response,

versus, you know, system 2 type thinking, which is your,

when you actually think through something

and and your,

how and how that that plays into your everyday life and and, you know, what what what system is in charge when. And that, like, after you read that book, you're like, oh, wait. I was, like, on autopilot. So that was probably system 1. You know? Like, stuff like that. It's pretty cool. Yeah. I'll second that pick. I have actually,

just recently finished listening to the audiobook version of that, and there's a lot of really interesting stuff in there. It really talks a lot about how the different systems

play together to create various cognitive biases and how that can affect your decision making process in various, situations.

Yep. Yep. I got I got 1 more thing. I got 1 more thing. I just got back from traveling around, for a little bit. I was in Spain. I have to say, if you have some time, you should book a flight to Spain. Spain is very interesting.

You you can go to several different cities and you'll get a very different experience

because Spain is like

5 ish different regions,

and they all have their own little subculture and all of that. So, like, I've been to Barcelona and Madrid. And while I was in Madrid, I went to, Segovia and and a couple other towns around it, and it was great. Everybody should check it out. Well, we really appreciate you taking the time out of your day to join us and talk to us about your work and how Python fits into it and the various types of testing and complex systems that we should all be thinking about. So thank you very much. And I hope you enjoy the rest of your day. Thank you. Thanks, guys. Thanks. Bye

bye.

The Python Podcast.init

Summary

Brief Introduction

Interview with Cris Medina

Keep In Touch

Picks

Links

The Python Podcast.__init__

Summary

Brief Introduction

Interview with Cris Medina

Keep In Touch

Picks

Links

The Python Podcast.init