The Python Podcast.__init__

Share on social media:

[00:00:14] Unknown:

Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. When you're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it. So check out our friends over at Linode.   With their managed Kubernetes   platform, it's easy to get started with the next generation of deployment and scaling powered by the battle tested Linode platform, including simple pricing, node balancers, 40 gigabit networking, and dedicated CPU and GPU instances.   And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover.  

Go to python podcast.com/linode   today to get a $100 credit to try out their new database service. And don't forget to thank them for their continued support of this show. Your host as usual is Tobias Macy. And today, I'm interviewing Jonathan Wright about the role of automation in your testing and QA strategies. So, Jonathan, can you start by introducing  

[00:01:06] Unknown:

yourself? Well, it's a pleasure to be here. Yes. I'm Jonathan Wright based out in Oxford in the UK.   I've been in the automation landscape for the last 4 decades. Sometimes referred to as the automation cyborg sent back in time to   save the world from bad software.   So I'm looking forward to talking to the Python community as well, especially around automation,   machine learning, and everything that you need to know about, you know, how to get better quality and testing.  

[00:01:34] Unknown:

And do you remember how you first got introduced to Python?  

[00:01:37] Unknown:

Oof. Well, many, many years ago, I must admit, it came in when   people started   tinkering, I suppose, with data science as kind of a discipline.   And, of course, Python, you know, I've been looking at Haskell,   which, you know, is a maths based programming language, which is what I've been playing with at the moment because, you know, you find that you're switching between them, but the majority of my developers write everything in Python. So but I don't know if you've had any experience with category theory. It's a bit of a pain  

[00:02:07] Unknown:

on the brain, should we say? Yeah. It's definitely interesting digging into   more elaborate type systems   and formal functional programming and   especially when you start getting into more, even more complex and esoteric languages like Idris and starting to think about, okay, how can I make sure that everything that I write is provably correct?  

[00:02:27] Unknown:

Absolutely. I kind of joke with 1 of my developers about the fact that obviously Klingons loosely based on maths as well. So, you know, should we, you know, start moving away from English language and and start writing and cling on to avoid ambiguity, but, it's maybe not the fastest growing growing language, whereas obviously   Python, everyone, especially as the data science community just keeps on growing. And also the automation community, you know, there's just so many people using Python today. You know, I think it's only gonna get bigger. Absolutely. Or Esperanto, which was constructed as a language to remove ambiguity, and nobody ever really caught on with it,   at least not wide enough for it to be notable. Yeah. My friend, Tom Gillb, wrote a language called Planguage   back   in 1985.  

It was part of something which he he published his first book on EVO called EVO, which was evolusional   development, which is long before the Agile manifesto.   And he created language with this kind of viewpoint of how do I get rid of that, bigger scene. It was really interesting. I'll make sure in the show notes I I share because, we have something called Guild Fest where everyone meets together   every year to think about what's coming next down the software train. And he shares all of his documentation in a big drop box folder and all of the the his notes and what he's learned over the years. So, you know, that hopefully, that'll be interesting. But, you know, my friend, Paul Gerard, he he had a project recently in LISP and, you know, that he was asked to do LISP in DevOps and also then in the cloud. And it was just kind of those, you know, obviously, predating COBOL as a language, but it's coming back into fashion. It's amazing how these things happen. Absolutely.  

[00:04:08] Unknown:

And   the main focus of our conversation today is around kind of QA and validation and testing   energy on.  

[00:04:24] Unknown:

Yeah, definitely. So I must admit I came came out as a, you know, a computer systems, high performance computing grad, and I was doing things like OCCM, which was distributed parallel processing languages,   and then got a gig working my first job straight out of university at Siemens.   Siemens was a Siemens Engineering, and they were such a engineering based organization that   they actually got me involved in doing   automation,   which was very interesting. When I took the job and they said to me testing, I generally thought that it was gonna be picking phones up and putting them down and making sure they worked.   And then I found myself with an abacus, which was a binary based system to program PABXs.  

And then I started my automation career kind of   the early kind of nineties when, you know, that these systems were just getting built. And then we were doing things like voice over IP testing   of silent suppression stuff. It was really quite interesting. And, of course, a lot of the other people who I was working with all graduates,   in computer kind of studies and programmers and developers and as well as testers, of course.  

[00:05:29] Unknown:

In terms of the   overall concept of kind of QA and testing and automation, there are definitely a lot of different rabbit holes that you can go down. And I'm wondering if you can   maybe share kind of the high level view of what are the commonalities regardless of what specific area you're focused on,   And what are maybe sort of the main categories of how companies and software teams   think about and work towards addressing the testing and validation and QA aspect of their work?  

[00:06:00] Unknown:

Yeah. It's software is infinitely complex, as we know. And, you know, we do have this tendency of trying apply   manufacturing processes like lean and Kanban to something which is software, which is,   you know, you can't apply those same kind of kind of methodologies, should we say. And so testing phases over the time, you know, we could joke about the fact to say, well, what came first, You know, the test or the code. You know? And, of course, with modern day principles   that, you know, such as TDD, test driven development, BDD, behavior driven development, and ATDD,   acceptance test driven development.  

There's that kind of movement towards, you know, building the test first and then creating the code to make it pass. I must admit this time, probably 3 or 4 years ago, I sat with Dan North who, you know, was kind of 1 of the grand grandfather, should we say, of of behavior driven development, and he kind of   had seen where kind of these kind of gherkin based executable specifications had gone the direction of, you know, things like given, when, then, and as a way of trying to define incredibly complex systems,   which, again, in natural language, which had that big receipt in there. And, of course, you know, from a DSL perspective, it was really focused on trying to make it as human readable as possible for the business with this kind of, well, let's get the 3 amigos involved and, you know, developers and testers, which should all be working in harmony.  

Whereas we really know that's not what the real world's like. And and as the pace of software development increases and we go from   DevOps to kind of this continuous delivery kind of cycle of, you know, pushing and going straight to production is,   you know, this ability to do testing and the differentiation between checking and testing, which I guess is kind of when you start looking at things like integration testing and unit testing, component testing and static code analysis.   We really focus on, you know, building a certain amount of confidence or how we would want to test the system. Whereas then when it goes into the real world, a bit like the   house of dragon bringing down, you know, h s HBO   streaming services,   this is much more complex. There's chaos engineering,   site reliability   principles that maybe aren't always, you know, looked at. I noticed the other day, Amazon have now got a fault injection service, which they've got to purposely try and bring down   your Kafka or whatever else you're doing so that you can actually try, you know, put those gremlins. And I did a podcast with a a guy called Colton   who actually did wrote about was was responsible for doing chaos engineering at Amazon.  

It was a fascinating he runs a company called Gremlin With Gremlin in the software kind of thing is how do you find those bugs and how do you find those pieces of malicious software or software that's not optimized that could potentially bring down those pods or however you've created that infrastructure as code. And, you know, I think there's so many layers now that, you know, I genuinely can't even think of a place to start or a place to finish, but I think the only 1 rule that probably   rules to rule them all is that wherever there is a line of code, there should be an associated test, whether it is infrastructure as code, configuration as code.   You know, you need to build that in early, and you need to feel confident that it's got that level of quality built in from day 1. And it's not an afterthought that once we've got an MVP,   you know, we'll come back and get it over, you know, a code coverage over a certain percentage.  

I bought a whole stack of students for a full day of code coverage for testers, and it was generally the most boring subject I could ever think of. But, you know, it was that subjective saying of, well, what is it meaningless if it's less than a 100%.   You know, again, testing is incredibly subjective. There's so many different of camps for it. And, of course, now there's this shift towards software developers and tests to get people involved earlier. So   lots of things to unravel.  

[00:09:51] Unknown:

Yeah. As you said, every layer of the stack has its own, you know, accompanying complexity. And as with everything in software, you can solve it with another layer of indirection. You know, you just need to add that level of abstraction. So I can say, I only need to test from here up or from here down or this layer in the middle. You know? And then, you know, that brings you into integration tests where now you have to test all the way through and figuring out what are the happy paths and, you know, when is good enough good enough, you know, to to your point of code coverage or kind of chaos engineering.   Another interesting aspect of the overall question of kind of testing and quality assurance is who is responsible for it, where early in the, you know, life cycle of software engineering as a discipline,   it was very common for you to have kind of the waterfall approach of, you know, design, build, test, release. And the testing part was usually its own department where   the software engineers would write the code, and then they would hand it off to the QA department and say, good luck. Go see if it breaks.  

And then if it broke, well, then you have another long loop and, you know, with the adoption of things like DevOps practices and the whole idea of shift left,   testing   has become more the responsibility of the developers, and, you know, maybe it goes into the design stage too. You know, it starts to permeate all the aspects. And I'm wondering   how you have seen the distribution of responsibility   affect the overall   concepts of kind of quality assurance? Like, are there things that we've lost in that process?  

[00:11:14] Unknown:

Doctor. Jeremy Begbie Yeah, I think you are right. It's kind of the shift from the traditional V model and requirements engineering, you know, which is why, you know, I referred to Tom Gill originally, but, you know, his first book was requirements engineering with a guy called Eric Simmons.   And it was all these dimensions of how you could look at, you know, documentation inspection. My my friend, Doris Graham, who is the grandmother of of automation, she wrote the first allegedly wrote the first slide of code in 1965   for Bell Labs,   and, you know, she published a book called software inspection.  

And, you know, part of it is there's some disciplines we've lost from that kind of I guess you saw the word there of engineering at the end of the term. You know, quality engineering, of course, being an emerging kind of area over kind of quality assurance, and then you raised the infinite prompt challenge, which was, you know, quality is everyone's responsibility.   But by saying that statement, does that mean that no one's responsible for quality?   And this is what we've been trying to do, especially at Keysight. You know, we've been talking about things like quality stream management, very similar to value stream management of tiers of well, you know, a team can be delivering a widget or or some kind of integration point or node, should we call it in the simplest of terms. So who's responsible for the quality? Is it code or is it the team? You know, does that change as the teams adopt new methodologies and decide that they wanna do Spotify models and squads because it sounds fantastic,   or, you know, they don't have SRE engineers sat there. They don't have release train engineers for scale, agile, or less, or DAD methodologies.  

You know? They kind of can choose   where the book stops as far as well as the quality. You know? A c suite. How do we realize that we should spend more money or spend more time before releasing a product out to market?   So, you know, understanding that, I think, is infinitely impossible from a quality perspective.   So that quality life cycle management from the idealization   of coming up with a concept that sounds like a great idea   to the proving of the hypothesis, which could be minutes, hours, years,   you know, is a time factor based on lots of different methodologies. You know? And I did a fantastic podcast with a guy called Niall Lynch who unfortunately passed away during COVID. But, you know, he created something called time to quality, which was, you know, the time for   both having a perceived understanding that the software is up to a certain point to then understand the technical debt and the challenges with the pattern, we think of an app ops kind of approach. You know? Well, we're gonna use this particular distributed pattern, you know, oh, there's some kind of issues.  

By the time we've got something stable and the quality is up to a level which our end customers wanting, if it's a commodity platform versus   an investment banking versus a retail versus a, you know, kind of a disruptor, which is just trying to get their tech in   space very quick. So, you know, part of it is these are all different paces,   and this shift left left movement has really moved into shift right. And this is kind of what I've been talking about probably the last 5, 10 years is how do we prove   that on the right hand side and learn from the right hand side to feedback in? And I did a TED talk on cognitive learning, and that was that feedback cycle from, well, what's happening in the real world, and how do we measure and learn from that? So we see, you know, how do we know   Salesforce today is at a certain level of quality, and the net release doesn't break that and then potentially break hundreds of organizations   that use Salesforce as their critical kind of platform.  

You know, we have no way nothing stopping anybody releasing software. And so, therefore, kind of our, you know, CEO did this big pitch around, you know, this quality certification   for hardware. Why is there not that in software? Right? Anyone's allowed to release software and or break software. You know, how do we potentially get to that point where this kind of equivalent of a CE kind of tag to say, you know, if you, you know, do business with this, let's say, DOD military contractor who's writing software, it needs to be a certain level equally what's, for instance, for the DOD modernization   bill, they're actually talking about a bill of software quant in essence, which is then saying it's made with Log 4J. I'm building it on Neo 4J for my graph database.  

I'm doing, you know, this particular type of technology which I'm I'm utilizing. And if there's a vulnerability on that, it understands which part of the software   needs patching or even bringing down. We, of course, heard, you know, Apple announcing the vulnerability from a third party,   you know, analyst that found this kind of root access to all devices again, which we've had before. You know, it's that same kind of cycle of, well, how do we deal with, you know, these situations which could be security related? It could be   systemic failure of systems breaking because of the reliance on, you know,   their 2 factor authentication servers or maybe their CloudFlare   kind of built up DNS or the structure they've used on a particular cloud vendor and this multi cloud   kind of landscape. Is this so much complexity there? And   the bug doesn't really stop with anyone. You partly from but it's usually the operational team who are hitting it first end. And then, again, they're dealing with a different incident management and a service management platform to the developers are. The loopbacks feedbacks of, okay. Well, here's the digital twin of the system working in production.  

Now reproduce that within a kind of a TiVo kind of way of in development, which typically are, you know, devs are working in container in k 8 kind of instances on their machine. Maybe they don't have access to that production data, those production scenarios,   those real kind of environment. So learning a lot more about   what we do in the real world   and the real world's configuration and how it actually responds in the wild, it helps us to inform on how will we build better software. And I think that's definitely some of the big trends that I've seen is is a lot more focused as you're probably aware of observability,   you know, and bringing in things like open telemetry   with kind of a open tracing kind of subs kind of replacement, and then looking at how do we take this information to give us better pinpoint failure analysis, root cause analysis   much quicker with, you know, more accuracy. And, you know, you see it for these vendors that are doing, you know, code debugging in production. It's getting to that point now where I think we need to be be honest with ourselves. And if we're building software in a house of cards,   how do we, you know, ensure that those poor IT directors or, you know, software   development   directors   managed to live more than 5 years of their job without some catastrophic   failure   or, you know, having to hide all those skeletons in the closet. You know? Can we get a grade on, you know, whether or not we're using a grade a website or a grade, you know, grade c   website and, you know, there's a good chance that it's gonna fail or if it's built to last and it's robust and secure and reliable.  



[00:18:15] Unknown:

I particularly like the house of cards metaphor because the way you keep it together is with duct tape clearly, and that's what we all end up doing.  

[00:18:22] Unknown:

It is, you know, things like ops dev, you know, building for ops instead of building for dev is, is kind of that kind of landscape of, you know, you'd look at this beautiful pattern and then you look at the 40 year old organization that you're trying to deploy into, and they just don't have the operational skills to be able to deal with these,   you know, complex   architectures. And so and they're also mixing it. It's like you can look at a company and kind of guess what architecture it's built its foundational systems on by the year it was born. And you kinda see the refreshes coming through that logically do, but, you know, some of those organizers, you're just not prepared for the next gen kind of architectures. And then, you know, don't even start mentioning about, you know, building on AI based systems. You suddenly get   a massive challenge. Right? That you've got, you know, this kind of, well, you know, unpredictable   kind of nature of, you know, how is your algo, you know, doing in the real world, and how is it doing it today versus how it does in 6 weeks time, and whether or not it's giving people insurance,   payouts for the things that, you know, you don't want, it's even more difficult for the next gen platforms that people are building today.  



[00:19:30] Unknown:

Another interesting thing that I'd like to dig into before we get too far into the automation piece is the   kind of Venn diagram of the terms quality and testing,   because a lot of software teams will focus very heavily on testing and say, okay. I wrote this piece of functionality. I'm going to write some tests to make sure it does the thing that it's supposed to do and doesn't break under the, you know, circumstances that I can predict. Or, you know, maybe they're even using property based testing so that it can throw in some inputs and scenarios that you can't necessarily think about the top of your head. But in a lot of the description that you were just giving, you focused on this term quality. And I'm wondering if you can maybe give some clarity on the ways that those 2 terms are being used and some of the ways that the focus and the terminology   influences the way that the approach to software kind of build and delivery is done.  



[00:20:21] Unknown:

Yeah, I think it's a great question. And, you know, I'd like to kind of think that quality and testing exists, but never shall the 2 meet. You know, every time I speak to someone, you know, quality means something completely different to testing.   And, you know, a quality assurance team,   responsibilities   are completely different to a traditional testing team.   And, you know, just like we do with every mash up, whether it be product management and engineering to create product engineering as Google did,   We're similar seeing this with quality engineering is how do we bring quality in and engineering into the same teams, have cute quality engineers in there. In the same way, we have test engineers as we have data engineers.  

We're really trying to bake that engineering back into it, but the responsibilities are completely different. You know, quality is very holistic. It's very much about the overall process. You know, in a way you could say, I'm gonna do a audit of your company's processes,   whether it's your CICD process or how you do your static code analysis or your secure coding workshops. You're looking at holistically everything, whereas testing, you might be saying, I'm testing this 1 component. You could be testing it in functional or nonfunctional testing,   you know, API. You could be doing stubs and shims. You could be using service virtualization. You could be using network virtualization.  

You could be using packs and and contract testing. You know, there's lots of different viewpoints of it. Of course, that's why we always find system is integration usually fails when we all start coming together, and we've all used different practices and and techniques   to that point, and we've kind of said it works in our environment and then move it across. Now I know that's the aged old kind of problem of trying to say, well, know, we throw it over. I've been to a different environment, and it doesn't build. But, of course, we know we've learned a lot more than from continuous integration and testing and and DevOps, which had that philosophy   of if it hurts, do more of it. And now we've got to that point where we're terraforming, we're building things up, we're blowing them away,   and we can execute all the tests and then delete them. We can spin up entire performance environments with, you know, terabytes with the databases with, you know, mass subsets   of the scated datasets,   synthetically generated if they need to be. That's realistic enough to be able to run those high volumes on, you know, private, you know, cloud tenanted environments   that have that level of realism. So we've definitely matured,   and we've got this whole kind of lots of different phases that we can do. I guess the question is where as an engineer should I be spending most of my time. Right? You know, again, there's lots of different ways of approaching this from an automation perspective.  

You know, automations, that seems to be the easiest word to throw around. You know, I did a forest reply to a new quadrant we're doing at the moment called automation fabric. And it sounded really exciting when I, you know, opened up the term thinking, oh, it's gonna be like data fabric. I'm gonna be looking at, you know, Azure Databricks and how they're all built with just a standard recipe, which is really sensible.   But the actual fact, it was talking about the consolidation   of things like   hyper automation and RPA and digital process automation, which is based on people driving natural patterns,   to traditional   operations orchestration tools in your IT operations landscape,   to test automation tools where, you know, we're all starting to do automation, whether it be business process automation, whether it be RP traditional RPA.  

It's automation.   You know? And so as you start automating, the more you can automate automate the planet, you know, the easier your repetitive tasks are are kind of reduced. So, you know,   embracing automation, I assess, is 1 of those steps towards testing.   But then I think you've also got this diminishing returns, which is, you know, I go to customers and they'd say, oh, we've got 10, 000 automation scripts. And I say, okay. Why haven't you got 20, 000 scripts? And they kinda go, well, you know, 20, 000 is just too much. And you kinda go, well, what's 10, 000? What's the coverage of those 10, 000 scripts? You know, and I think that's the problem we're starting to build estates   of more and more automation, but then we're having less and less confidence in the automation that's running. So you've got these huge glorious regression packs, which require people to fix them and maintain them and refactor them so many times that you're not sure whether or not it actually is test passing against the requirement   because the requirements traceability is long gone because, you know, we just don't have that link anymore, and people are changing,   you know, sprint from sprint a requirement. And, of course, that's not reflected   to the user behavior in those tests because they're kind of inherited from the lap maybe the last sprint or the sprint behind. So we're starting to see this automation creep where it's just exploding,   and then it's kind of I'm literally overlaying that for some customers. I did it with a large gym and also maybe a company based out of Munich, and I took all of their automation regression, all of their, you know, functional and nonfunctional   stuff and security and and crowd testing and exploratory testing mapped it over the actual functional   functionality of the application estates.  

And you realize actually the coverage is incredibly low that you mentioned the word happy path.   You know, I tried to ban all happy path, but what I was trying to mean is not just happy path for e 2 e and contract testing and, you know, people   creating payloads for APIs that they know work and not have every single type of, you know, account and, you know, country or whatever else you need in there is actually it's as important to have test data coverage   as it is to have test coverage like you mentioned. You know, it's that kind of, yes, I'm going through a happy path, but I'm trying it with all permutations of different types of accounts with different statuses.   But, actually, now I'm I'm moving away from those happy path to really explore some of those edge cases.  

And that's where things like model based testing   really help and and really kind of allow you to understand, well, actually, I've only got 20% coverage of my app with these 10, 000 scripts that I'm running. So, you know, definitely moving towards more model based testing and digital twin testing is is another way of getting more confidence around how your app is performing.  

[00:26:31] Unknown:

In this   conversation about automation, that's another term where it can be very overloaded or misinterpreted.   And I'm wondering if you can talk to some of the ways that that automation manifests   and how it might be   disjoint from what a lot of people think about when they say automation where they're saying, it's in my CICD pipeline. It's automated.   What are some of the other aspects of automation or the   capabilities that are being sought in this process of automating that   are, you know, not already covered by the ways that your typical software development team is approaching their overall life cycle?  

[00:27:08] Unknown:

Yeah, definitely. So, you know, I do blame occasionally Lisa Crispin and everyone else who kind of always throws the triangle of death   of UI being at the top, very brickled automation, and then API, and then component. You know, we have this kind of viewpoint that,   you know, that should be the proportions, and it's like everything is that actually it depends. Right? It depends on how you guys work, the quality of the software engineer to how much testing needs to be done. But, you know, I guess that word resilience kind of comes in there in saying, well, how do you do more than just your standard tests? You know, you could use diff blue to auto generate your components test. Right? That's gonna prove something. Well, it should prove probably only your Java libraries. Some people are using Java. Right? But, you know, part of it   is when you kind of go into a where are the, you know, where are the other phases where we can start leveraging,   you know, doing better testing?  

You know, I mentioned things like service virtualization and network virtualization and this kind of concept around digital twins.   You know, at Keysight, you know, as we're probably more famously known as a hardware company. Right? We're a hardware testing company,   which started off with Hewlett Packard back in the garage in 1938.   And Dave and Bill would both, you know, sat there   doing build and test. Everything was build and test, iteration 1, iteration 2, and they were building these machines   for Disney, and they'd get feedback from the customer saying, you know, they wanted this additional feature or whatever else was needed, and then they would add it and they do mark 1, mark 2, mark 3. And then, of course, Keysight kinda came into this area of, well, yeah, we can, you know, use our test equipment to   emulate this, right, or simulate this kind of you know, in this case, I'm gonna give the example of a a 6 g wave, right, which 1 of our labs did the other day. Now, of course, we know 6 g is not here, and we know it's not gonna be here probably for another 4 years. The definition is probably at least another year away, but we can test a 6 g wave because we can deal with that frequency. We can see   we can measure it on a device. And like you kind of pointed out earlier on, you know, we're quite used to the hardware abstraction layer, which is well, we're just gonna, you know, virtualize it to enter degree   and not care about, you know, things. We'll put it into a container, and we'll believe infinitely scalable because we're gonna do some kind of heat based load balancing profile in production.  

You know, that's great, but, you know, you're just adding more complexity to it to it without understanding that and expecting the, you know, the fact that you're building into a cloud that you add that ability to cover from any kind of systemic   failure or issue that you've got by spinning up more pods or whatever it may be. And it's kind of got us quite lazy in the sense of, you know, we've kind of not really relied on writing code that's   not gonna give us huge amounts of memory leaks in production every 15 minutes and garbage collection running on the compute nodes. You know? We kinda getting into this kind of, you know, ability that we can understand what's going wrong   and how we can actually prevent it, but there are a whole separate set of new tools. Right? These tools aren't the same as what, you know, were there before. So I'd give it a bit of an example with the 6 g. Right? We've got a bit of a demo that I did the other day where we've got a mobile device.  

You know, it's launching something. I don't know. Let's call it speed test. We've got the spectral analysis   tool, which is showing that it's pulling the debt the the network,   coming from the beacon, wherever that might be. We're seeing the battery drain of the instrumentation from the actual device. We're seeing how the different GPUs are running and the cores and the CPUs are performing on that device. And then we've got, you know, other different instrumentation   that we can look at and create that and then emulate it. So, you know, I was talking to our guy earlier on today about, well, I wanna put more I wanna lock more memory up in the device. Okay. Well, we'll increase the device.   And it sounds like, well, yeah, it's just it's a mobile device. Right? It's, you know, who cares kind of thing. But, you know, we've I mean, the same conversation with people like Meta and and   Microsoft at the moment around, you know, mixed reality for the Google Glasses,   v 2, and and the Apple Glasses and, you know, that kind of a state that we're gonna start seeing if the app again, of of apps being moving across for that. And then you've got things like, of course, Meta has already got things like the Oculus for doing VR and that environment. And we're doing high speed,   a 120 frames per second at 4 k per eye. We're doing full computer vision and OCR to see if there's any lagging in there. We're pulling all the metrics off the actual device. Again, we're understanding if it's been affected by interference or anything else, you know, being over a wire. You know, we're emulating the 5 g versus a 2 and a half g network connectivity, connectivity, the last mile, the packet loss, you know, the jitter on the line where the low to late let's call it CDM, but the edge matter is for the   location in. And now you need a Cisco's   you know, every single Cisco qualification to even understand   the landscape of how, you know, that information's getting there. How can we understand that or even build for these next generation, you know, platforms? And, you know, my guys who have, you know, the r and d guys are all looking at things like the quantum computing kind of use cases as well. You're thinking,   you know, it takes me I'd have to read a book, probably what the teaching quantum physics to your, your dog bug to actually either understand   how that works in the first place, but we know there's use cases for it, so we're developing them. And I think that's the problem is we kind of we abstract things to the point where we're like, well, it doesn't really matter. You know, my first day, we launched the autonomous radar   emulator, which allows you to emulate,   you know, animals flying across the road or, you know, a hundred people, you know, and be able to use, you know, car to x, infrastructure to   x techniques to be able to test, you know, computer vision to say whether it should break the IDI systems in the car works. You know, how do you even start testing this stuff? And it's all done at the component level. Right? And that's the problem is when it comes together and it becomes a Tesla,   you know, testing that digital twin is incredibly complex,   you know, and it's out in the wild. And if it's in the wild, then, you know, potentially, you're gonna end up on social, you know,   social media at some point in time. Like, the HBO example is, you know, we know they're doing their best job, and we could probably bet if we are betting people that they don't actually own their streaming   service like Netflix doesn't. They're probably using Amazon, you know, streaming capabilities. So why are they to blame for, you know, people not being able to access the system? The same as, you know, Apple. We always see Apple in the news, but Apple don't have their own cloud system. They use other people's cloud stuff, and it's that kind of thing where we're building on other people's, you know, systems, and we're expecting those 5 nines to protect us from, you know, catastrophic   failure and embarrassment, which could potentially wipe out a software product. We've seen it in the past, and I think we're gonna see this even more when we start going to the   metaverse and and beyond. And, of course, that's just starting to get into crazy land. Right?  



[00:34:09] Unknown:

Absolutely. And the other interesting aspect of   using automation to verify the quality of your software is that it's just more software. So it gets into the problem of who watches the watchers. Like, how many layers do you need before you can say, okay. This has gone far enough. I'm just gonna throw up my hands and say, I don't know for sure. I'm pretty sure it's gonna work. Let's just push the button and see what happens kind of a thing. I'm wondering if you can maybe talk to   some of the   ways that teams who are looking into investing in automation   to help with their quality assurance process   need to be thinking about what are the capabilities that they need to look at, what are the ways that they need to do their own due diligence to understand, is this tool actually going to tell me what I think it's telling me, you know, how do I know that it's doing   the things that I tell it to do properly and that it's giving me the answers that I need to, you know, build that confidence? Some of those questions of kind of the meta problem of quality.  



[00:35:06] Unknown:

As a slightly, which won't be a, a use case which people are in there, but, you know, exactly the same thing. You know, I was talking to the guys at MET the other day, and they were saying, how do we know that, you know, the inputs that you're doing for the headset are a valid movement based on my body structure. Right? So we're connecting connects to look at, you know, body movement. We're making sure that the numbers that we put in could physically happen on a on a human. Right? That they could move their head and the gyro could go at that kind of speed. You know, you're in a confined amount of configuration   parameters, and I think that's where you've gotta start if you've gotta kind of understand,   you know, where to put all your eggs, you know, not in 1 basket of, you know, just doing API testing and, you know, use postman and just get something in there and then start mocking some stuff out.  

And you've got baby steps towards, you know, okay. I'm feeling more confident now. I'm gonna scale up and maybe   start injecting,   large amounts of payloads into my API tests. And, you know, you keep on maturing your automation techniques.   But as you're going through that, you're investing more and more time. And as you said, there's this kind of water effect of, you know, well, how do I know that that's good? How do I know that that was payloads are just completely handcrafted?   And that actually once it goes into the real world that it's just gonna not just explode?   You know, how am I going to be able to simulate, you know, failure to to see how it recovers?  

You know? And I think, you know, little and often is kind of the way to do it. It's not to invest too much of any stack. You know? It makes it you know, even I feel that it's a hard way thing. I used to when I started in the nineties, I used to talk about manumation, you know, doing manual effort to create automation artifacts, and that was in the glory days of record replay. Right? And I think this is the problem is if if it's too good to be true, it probably is too good to be true. And, you know, use the lot of hype, especially with any vendor or any tool or testing product that says, oh, it's just gonna do everything for you. And we know that's not the case. Right? And people sometimes need to be able to look under their hood and see what it is actually doing. It's what assertions it's doing, you know, and be also familiar with it in the language they trust. Right? So if you run it right in in Python, of course, you wanna be able to probably execute your tests within Python, which, you know, Selenium, of course, is a great place to start, you know, in the sense of it's a w 3 and c standard, you know, for web based testing.   So, you know,   use open source as much as you can until you need to scale. And then when you need to scale, you need to start thinking about investing in tools that are enterprise grade. So   that would be my kind of, you know, tip.  



[00:37:39] Unknown:

To the point of trusting the   results that you get from the tool that you're using, there is also what you were saying about understanding   what are its capabilities and its limitations so that you know, okay. This is the tool. This is what it can do. So these are the areas where I'm going to apply it. But, you know, this piece is actually outside of its range of capabilities, so I need to think about a different approach. And I'm wondering   what are some of the   useful   kind of practices or strategies for teams to understand?   What are the maybe core workflows that they need to target in some of the ways that they can start onboarding automation into their quality assurance process   and how to kind of manage that overall life cycle because it's very easy to say, okay. I'll go buy the tool or I'll use the open source project.  

I'll do this thing, and, okay, great. Everything is doing what I want to, and then it just becomes another piece of tech debt because you don't maintain it. And just how to think about the overall investment process of saying, okay, these are the core, you know, workflows. I need to make sure that these work all the time. I can be confident that the tool is going to give me the outputs that I need to have that confidence.   And how do you know sort of when you've been taking it too far where it becomes unmaintainable,   and, you know,   like figuring out what those boundaries are and, you know, how to   understand the overall costs and trade offs of that process?  



[00:38:58] Unknown:

Again, it's an infinite dilemma, which I think everyone's got is, you know, even with just the idea of starting small, but thinking big in the sense of, okay, I know that some of this is throw away, and it's just gonna get me going in the short term. But I need a long term strategy of, well, how do I keep this maintainable and scalable?   I think with everything, it's the same in the sense of even with agile. It's, you know, it works well in pockets of the organization and DevOps principles, but then may not work as well in other places.   Is getting an enterprise level automation strategy might be near impossible. So, you know, there is teams gonna be going off and doing their own thing, and there's nothing wrong with that. But, you know, kind of where I was leading into with the w 3 c or using standardized frameworks as the baseline is an industry standards   kind of allows you to at least ensure what you're investing in makes sense for the long term, but also using blueprints that make things as reusable and lightweight as possible. And, of course, you know, we've talked about things like BDD and Gherkin   and spec flow or maybe all, you know, some of the type of applications do that. And, again, you're investing. You often get investing in that maintain maintenance in keeping those tests relevant.  

That's where I think, you know, sometimes moving to things that are going to be more intuitive and also   self healing and being able to actually maintain itself.   You know? Of course, you know, I said that as a disclaimer earlier on, you know, of course, it's not magic. It's gonna, you know, apply using AI for for testing. So I book which I've just done is is artificial intelligence and software testing, and that's set on the ISO committee for,   on on artificial intelligence for software testing blueprints.   There is proven methodologies that we've used over the last 4 decades   to self heal things like your domain object model, which is usually quite not very robust,   you know, automatically generate API tests from swagger specs or, you know, built using auto generation capabilities   to make it a lot easier to maintain. And, of course, you know, we're very visual human beings. So, yes, the tendency is to script it, but, you know, in actual fact, you know, you want to also be able to visualize that and collaborate on that with other people. So you wanna be able to understand, well, what's in the estate? You know, what bits have been automated, which bits haven't been? So this kind of approach to model based testing really helps because   modeling again is that natural language we kinda started this conversation off with was, you know, if you're able to kind of express it, and we have something called Sentor, which is a English based language from an NLP perspective,   you can write simple statements a bit like not as powerful as Python, of course, but, you know, to that kind of here's some standard functionalities which call things like Google computer vision. So if you're not familiar with OpenCV or, you know, some type of OCR engine, you can just call that feature and then use a tool that it enables you to train it or optimize it. So you've got those capabilities instead of having to build them yourself. Now we know we could, you know, we could grab test or file or something else and, you know, start building something ourselves, but why build it when you can just consume those services, whether it be computer vision or some kind of, you know, data test data management, TDM kind of engine that can generate you synthetic data. You're much easier to utilize   these, and this is why I said the kind of the off the shelf kind of viewpoint is that once you get to a certain point, a bit like postman, you need to pay for a subscription so you can all collaborate together and start doing more scale and run more tests and, you know, create more insertions and create more marks or do what it is, what you need to give you more confidence. And I think that's the big thing to be aware of is   you need to, at some point, spend some money   and dip your toe into some of those additional functions that are gonna really accelerate your   time your quality engineering efforts without having to, you know,   spend huge amounts of time maintaining your scripts as you would have to do maybe potentially a code base. And, also, you know, who else in your team can be involved with that, whether that be a a traditional software developer in test or it could be a technical test or even a nontechnical tester. And, of course, that's the dream is that everyone can collaborate and build these. The reality is there's always a certain level of code that needs to be written,   and things like Sentelk definitely drop the barrier of entry because you don't it's kind of language agnostic in a way. You know, everything's very basic.  

And that's why in the early days, people were using things like TSL, test script language, and then things like visual basic, which was very unfortunate   to actually write. There were simple languages, but you can build everything on Python. You can build everything   with Sandstorm as well, which calls, you know, can be bit wrapped in Python. So, you know, I'd recommend balance it and work out when not to spend loads of times building your own framework and when to, you know, spend time actually thinking more of the the logic and the, you know, the DSL that you needed to define.  

[00:43:49] Unknown:

In this overall   space of automation,   particularly as you start getting into more   AI and ML powered systems,   what are some of the limitations that people need to be aware of? What is the kind of upper bound of what people can expect from automation? Because when you start throwing around the terms AI and ML, it just starts to sound like magic, and you say, oh, I just pay for that, and it'll do everything I want. And, you know, I don't have to worry about it. So, like, what are some of the kind of natural   limits that people can expect as they start to dig more into some of these, you know, advanced systems where they are using more sophisticated processes?  



[00:44:25] Unknown:

Yeah. Well, you've just gotta think about it in the same way that you would apply ML with your own projects. Right? They've gotta be explainable AI. Right? You know, you need to be understand the use case because they're very narrow as we know within the landscape.   And something very simple like self healing, a domain object model reference of an x path or a regular expression   has been around for years.   So, you know, what's revolutional about, you know, what's coming out at the moment? Of course, computer vision helps. You know, computer vision, you know, 1 of my guys called this morning, did a bit of a demo with meta, you know, where we're identifying things in that environment. So, yeah, there's a door over there or something's happening over here. Yeah. That's pretty simple. You know, my friend got created a company called game driver dot io, which, you know, does all that with Unity. Right? Is it knows the Unity libraries? It knows what it's building. So when it sees a person stood in a game, it understands how it should look   versus the reference libraries. Right? Is you know, there's a lot of stuff that sounds like, well, brilliant. It can do this kind of stuff, but what are those   AI use cases that actually,   you know, are gonna help me? You know, I tried to do a demo yesterday with someone that is showing them our bug hunting algorithm, and, you know, it's like, well, what it does is it understands where within a system it's potentially it finds an issue. And then it you know, there was always this rule that wherever there's a defect, there's typically another defect behind it. So, you know, part of it is if it looks at it and yes. Let's think it's just react or fluster or whatever else it is. You know, if it sees that, you know, it's in Salesforce and it's dropping down a box, which doesn't allow you to change the location to New Zealand,   you know, there's a good likelihood   in another area which you can edit the you know, your address details is also gonna fail. Right? So it starts identifying   the meta domain language behind that to say, well, yeah, I understand that potentially anywhere else, which has got a sign up or a registration   might also feature this, which might also reference it. You could look where the databases,   you know, are also being accessed from a select statement. Right? You know, that's really quite basic, but it's doing bug hunting. So it is going around trying to focus based on weights where it should spend most of its time generating more tests   to be more rigorous, I. E. Go through every single 1 of those permutations of languages.  

You know, the same thing goes for our exploratory testing stuff. It is very rudimental   in the sense of it looks at users computer vision to recognize what's on the screen, and it understands   within that 1 page,   you know, there's a menu section. If I go to that menu and return back to that menu, of course, with any deep learning system that's been reward based, it's gonna go, okay. I'm here. So it creates the model. It knows where it is. It's back at the menu.   Now it knows the steps it needs to do to recreate to that point of time before. So then you literally just say, I wanna be at this stage where I'm just about to process an invoice.   Get me to that state, and the system will be able to get to there because it understands   what it's looking for and how it's done it before in the past. So there's a lot of, you know, simple applications. You know, I've tomorrow, I'm doing,   with my friend, doctor Tarek King, who's from test dotai,   who's done published a lot of machine stuff around   AI. And the last 2 books that I've done is is featured in. You know, he talks about these standard use cases and, you know, we've all, you know, talked about things like, oh, well, it log in turns into log on and how do I understand? Well, yeah, we've seen fuzzing from, you know, AWS for years where it can kind of understand as an email   and password box. It then gets the understanding that it needs to register first, needs to activate an email, and then come back in and actually do it or get a 2 factor authentication code to progress or have certain site permissions.  

So part of it is these systems are getting better, and we are learning and is becoming smarter.   Is it gonna replace a human anytime soon? Absolutely not.   But does it save you time? Absolutely.   Can it auto discover things and, you know, understand   structures of, an application quite easily? Yes.   Is there much work to do to make it even bigger and better? Absolutely. And that's where we're trying to which is why we're investing so much in   AI driven testing at the moment is to try and define the more use cases that we can use.  

[00:48:42] Unknown:

In terms of your experience of working in this space   and watching the ways that software teams and QA teams are   trying to get their hands around this infinite complexity of software systems and validation and ensuring that the projects that you're working on are doing the things that you are hoping that they're doing and that they're supposed to be doing. What are some of the most interesting or innovative or unexpected ways that you have seen automation applied to that question of quality assurance?  

[00:49:11] Unknown:

Yeah. Probably lots of crazy things that I've seen people use automation for. And I think this blurs the boundaries between, you know, automation and RPA, which their similarities are very similar. You know, I remember   myself using an automation tool to board loans for, Lehman Brothers just before it sell, you know, and actually using the automation engine to build to go from a mainframe system to take that information and do some basic well, it says flat in this system, and it means apartment in this system, you know, and then be very flexible and have this, I'm gonna use the word cognitive recognition, but not nowhere as advanced where it can at least   understand 2 different systems.  

And that could be, you know, your SAP to your SAP HANA migrations. You know, it it's getting to that point of when do we start using   robotic process automation   and traditional automation and operations orchestration   together. I mean, you know, I've seen a lot of people   using things like process mining, using process mining tools to understand what users are doing to then be able to try and replicate it, you know, whether, you know, it's we've I've seen it done for Brexit where they use   how people are behaving at the moment, and then they model what it would look like with a no deal situation   versus a deal situation, and then they build the system based on   what those models look like. And I think we're starting to see lots of it ways of maybe modeling things before they exist. And,   yeah, I threw out about 10, 15 years ago at Gartner this idea of life cycle virtualization,   which was, could you virtualize everything, even the system without even having any code? You know, could you model what that would look like before you decide to build it and then work out whether or not that's a good idea? And I think we're getting to that point now where we can understand if you do this, it's gonna break this part of the system. Or if you do this, there's gonna be a higher risk level in the kind of the risk based testing kind of methodologies,   looking at impact analysis, looking at how you can pull all this information   together and all this observability together   to make more better decisions and and more informed decisions of how long is it gonna take to test. You know, this idea of everything can fit into 2 weeks and, you know, within those 2 weeks, something of value can get delivered. You know, is there always gonna be the same amount of automation work done? Is there always gonna be the same amount of testing work? Well, it depends. It depends on this. You know, I always mispronounced psychometric complexity, but, you know, it's the complexity of the code base, how, you know, how much tech debt as you mentioned is in there and how old the code base is, you know, how many lines of code it is. You know, I think we take everything as a standard blueprint of the best scenario, and then we go at it with best intentions.  

And then we find that we're in a bit of a position where people are making decisions maybe blindly   whether or not something's good or it's not good or it's ready to go. I think we now need to get to this point where we're starting to build in the quality into the systems with the idea that it will fail. And the question is whether or not it fails gracefully.   And, you know, I think this is the new reality is that testing   is moving further down the life cycle again because we just don't have enough time for anything. So test everywhere,   test often, and deploy   even more often, and then learn from what's going on. I I still think we're missing those   tools which, you know, those ops tools that ops have had for such a long time, like network traffic and gateway utilization   and everything else. We get people don't have access to those same large scale operational tools that give you all this great insight about,   you know, how the system's actually working in the real world. We're kind of still   very much, you know, not lowering the boundaries of dev and ops. We're building, you know, infrastructure teams and, you know, everything else into a single team and then generating more silos. And, you know, I think, you know, we need to   at least think about the patterns that we're gonna build in the future. And I am confident that they're going to change and people will start doing things,   you know, a lot of more center of enablement, not the center of excellence days of building teams that are automation   skilled is unequally not expecting everyone to be a jack of all trade, master of none, is I think we're gonna start seeing some really smart models of how we do distributed development   at scale   using augmented intelligence   from lots of people in lots of different areas and not just in   you have a developer, 2 testers, and a project team. You know? Things have got to be more flexible and adaptive. I think we're moving into this new world where we have to do that, and we have to rely as a service on other people, you know, whether it be,   and we have to rely as a service on other people, you know, whether it be computer vision, retail as a service for   algorithms of looking at people's t shirts and, you know, adaptive marketing or whatever else it may be, it's starting to be consumer driven in these incredibly   complex ecosystems. And we're also gonna need to start offering people   visibility into those environments. So if you build with   HBO's   gateway, you need access to all their debugging information. So because it's an endpoint which that you rely within your organization, you can't just consume something and blindly have no idea how it's performing, how reliable it is, how resilient it is, even the access to the code base. You're never gonna get that. Whereas now I think we're gonna start seeing that bit of, you know, I'm picking   this service over this service because of its track history, not just down detector and guessing that they're building on something else. Right? So, you know, I think we're gonna start seeing really smart solutions in the future, and we're gonna be the ones responsible   for building resilience in to get us to that grade where we're getting an a grade or a b grade or a c grade of must try harder kind of thing or must spend more time building more resilience in, and here's where you're you're failing. It's security or it's, you know, it's, you know, process or whatever it may be, your pipeline, you know, your pipeline's not very good. Whatever it is, I think we're gonna start having to look at those dimensions of quality and be scored by them. Yeah. I think that your comment about the visibility into the systems that you're relying on is very key and definitely 1 of the  

[00:55:15] Unknown:

prime motivations for why open source has become as ubiquitous as it is is because everybody needs to be able to look under the covers and understand what's actually happening.   And I think that to your point, for some of these commercial products,   there is starting to be more of a kind of 2 way street of understanding. Okay.   I'm trusting you to do this thing. I need you to give me a feed of information back so that I can make sure that my assumptions are actually correct or, you know, when those assumptions aren't holding true, I can figure out why kind of a thing.  

[00:55:45] Unknown:

Yeah. And I I think, you know, we see it with bleeding edge technology, like smart city as an experiment. Right? Is this country sharing their large datasets of how they badly adopted,   you know, a smart city system like Dubai.   And then they use that historical information whether it be 4, whether it be information they've observed, and they allow people to run their own scenarios against   what those failed systems are. You know, maybe that's where we're going next is you're going actually to Netflix and saying,   yes. There's all this great chaos engineering and, you know, you're delivering 26 times a day into production or 26 times a second into production.  

But, you know, here's the blueprint of how it actually works from a design methodology,   and we'll share that so we can run our own models of well, we're not just blindly going to use Spotify because we also like listening to their music and it sound squad sounds cool. We'll actually say this fits this methodology and these   architectural blueprints   fit my kind of organization and my risk profile   and what I'm expecting from my customer.   And, you know, these systems that people are building for, you know, critical sits infrastructure systems   are gonna have a different risk profile to people who are wanting to bring out   gaming platforms. You know? I I think this is what we need to start doing is we need to start maybe sharing from an industry perspective like we have been with open source. What else can we share? What kind of historical data can we do? And that's why when they said automation fabric, I was hoping that that's what was gonna be. We start seeing people being able to share a lot more information and learnings about what they do. And, you know, I should have a last access to Atlassian's platforms   to build and extend and and adopt how I do it, but I also wanna know what everybody else is doing and how there have been successes   or learning from failures. Right? And that's the 1, 000, you know, dollar question is, you know, celebrating those failures and you know, but learning from them. You know, they shouldn't just be somebody   tweeting that they're not happy because something a weird edge cases happen.  

It should actually be more of, okay, what can we learn from this? And now we've done our investigation.   What can we share? What can we share with the creators of Kafka?   What can we share with the community   to kind of learn from these blueprints? And in the app ops kind of mentality of how do we build stuff that's gonna be easier and not try and reinvent the wheel every time we do it. Because, you know, we got this kind of this self desire to want to learn new things and try and build new things, you know, has all the code that ever needs to be written and be written, you know, these kind of   big questions that I'm not saying are right or not right. You know? I think it's more of,   you know, if we're gonna evolve, how do we evolve software development and take it to the next level, and what does that look like in the same way way people start talking about web 3 and decentralized Internet. You know? Okay. Well, what does that mean for us? What does our industry need to do? How do we work together and build more things like open telemetry and maybe even open testing? So we're all, you know, sending flags to each other saying, I'm gonna send you a you know, even in production   a a payload, which I want you to accept, but actually it's a dummy response.  

Instead of my favorite thing with eBay and Amazon is when people post listings that say do not buy because there's no sandbox environment, and they've literally all hit in the API   in the way that I'm sure open banking is as well. So, you know, there's so much more we need to mature. And I think engineering, which is where we started, is adding engineering back into things   and being able to push back and say, no. This isn't   t shirt sized x x x l. This is just an infinite   probability engine which we need to build, and it's gonna take 442   years, and we need to go to the crowd to bring in 100 of engineers   like Kaggle would do for a Python equivalent   of getting people to start collaborating on a incredibly difficult problem, be rewarded maybe even financially   to, you know, solve problems that, you know, you can't expect just your 1 ML,   you know, data scientist guy to solve. You know, you've got to start looking at the world as a kind of   a smart kind of low, scalable  

[00:59:53] Unknown:

solution like Amazon Turk, but scaled up to the next level. In terms of your experience of working in this space and working with teams who are trying to   understand and address these challenges, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?  

[01:00:10] Unknown:

Well, personally, there's   too many. I think we all have probably too much access to too much systems. Right? You know, it's kind of 1 of those things that maybe we don't need access to certain things or certain systems, but we also don't know how those systems are always connected. You know, we joke about, you know, HBO going down, but, you know, we don't know if that was just someone   running some, you know, stress or load or spike testing,   and they pointed it to the wrong environment, and they hit production. Right? There was, you know, some debugging   mode that was still turned on that, you know, caused some, you know, pointing to the wrong structure of a test   string within their schema. We just don't know. Right? And I think that's the problem is   we don't know what we don't know, and we just assume   that   this is, you know, gonna work. And, you know, I've had times before where people said to me, it's fine. You can hit this node as hard as you want. I've hit it. And then then come back and said, oh, by the way, that shared a cluster with some production systems, but we didn't realize you were gonna hit the network so hard and bring the network down,   you know, and does attack the entire thing. You know? This is the thing is with great power could be great responsibility   and, you know, the guy who, you know, brought down PlayStation for the same price of some compute time is, you know, his monthly subscription because you got this power at your fingertips.  

You know, I've made those mistakes.   We're all human. I think we gotta celebrate those, bring those to light as well, not cover them up. Now the days of, you know, hacking it, you know, going in and just ninjaing stuff in production,   you know, those days are over. You know? Now we need to be confident enough to be able to not just rely on whoever's we used to have a team which we used to refer to whoever's on call as Superman. So that Superman or Superwoman would have to, you know, jump in at the last minute, read somebody else's code base, get a a fix in, you know, as soon as possible is, you know, we can't   expect that from everyone. You know, we can't be based on a superhero culture as much as we are at the moment is, you know, what happens if those 3 people which you know in your organization are holding it all up disappear?  

You know, we can't the organization not to continue. And so therefore, we have to do which we're programmed not to do is to build   our single point of failure out of everything we're doing and allow people to have that visibility,   clear documentation   that makes us interchangeable. And I have seen organizations,   very few,   usually Danish or something like that, who literally, they turn around and, you know, I come into an organization. They go,   we've wiped out 4 of our dev guys. They're just, you know, all off ill.   We're losing them this week,   and there'd be no drama. It'd be like, oh, it's fine. You know, we've got a flex team. We've got people who've already been, you know, code reviewing them. You know, they'll pick up the workflows. They'll pick the tasks up. They'll continue. We'll have the same velocity, which we already estimated because we had already factored this in based on learning over 3 years how our team works   that they literally had this kind of Microsoft did it for a small period of time while I was there, where it's literally, you know, the to the point where people aren't stressed and worried about, well, what happens if, you know, I can't get this out or, you know, the cutting corners, you know, that pressure disappears   and, you know, so does that kind of single point of failure of superstars. And, you know, let's move into that kind of well, we're here and we're doing, you know, some you know, think of it as a more for mister robot. You know, you're just kind of coming in to autonomous and anonymous and just doing a bit of work with hundreds of other people who are also contributing to that. And if you're not there, it's fine. They're still gonna get the the end result. And, you know, that's how we need to start working in kind of swarms of people that, you know, if the hive is a problem and the queen ends up having to be replaced, then, you know, there is no dramas, there's no stress, there's no   politics or, you know, who's got access to all these secrets. You know, literally, it's it's no stress development. And then maybe we've just invented something completely new, which is gonna be great for neurodiversity.  

Everywhere is people are just gonna be sat there going,   I'm not I don't suffer from these problems anymore because actually my environment,   which I'm working in is just like a delight every time I come into work.   Just think of that.  

[01:04:29] Unknown:

And for people who are interested   in   understanding more about the capabilities of automation, how to think about applying automation to their software quality problems, what are some of the resources that you recommend for people who wanna dig deeper?  

[01:04:44] Unknown:

Loads out there, you know, loads of great podcasts as well. You know, I do the qalead.com.   You know, I'd recommend   test guild with Joe Conantonio who do a performance and automation   guild. You know, I'd set as the president of the Vivint community, which is the largest independent software community, about 80, 000 users in over a 100 countries. And, you know, again, you can go on there and there's loads of resources for free, loads of people to talk to and have those peer conversations in a community area. It's a non for profit that's run. You know, always just reach out to, you know, to peers. Right? I've got the easiest connection. It's linkedin.com   /n/automation.  

So, you know, you know, I can always point and share you resources,   of courses that you can do that are all free and can get you started on your kind of automation awesomeness   journey. You know, the community's there and and want to support this. So many good events that are running up. You know, I was chatting the other day that they're running up automation star event. You know, I'm in Star West in LA next month, and, you know, there's so many good events that are out in the calendar go to test events. You know? There was always a stigma of, you know, should developers and testers all be, you know, going to these kind of events, but, you know, we're doing KubeCon   a week after we're doing Star West. You know, part of it is I think the development community and the test community   are sharing events a lot more now. And, you know, I when I was in Australia,   there's a an event called fusion, and it had project managers,   developers, and testers all in the same streams and stuff. And I think that's what we've gotta do is we've gotta start getting those events where we can all share knowledge together in the software kind of challenges as a whole, really.  



[01:06:25] Unknown:

Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add preferred contact information to the show notes. And with that, I'll move us into the picks.   This week, I'm going to choose The Sandman,  

[01:06:36] Unknown:

particularly because I just started watching the Netflix series that just came out, but also the graphic novels, which are the source material by Neil Gaiman, who's 1 of my favorite authors. So definitely recommend looking at everything he's done. So with that, I'll pass it to you, Jonathan. Do you have any picks this week? You know, of course, I could have easily gone for the game of quality and game with the new house of dragon and which we've already mentioned, which was very good. I am very excited about it, and I will get around to going through the books at the same pace, I think, as I did last time. I'll reread them as the case may be. I'm gonna kind of throw a bit of a an odd 1 out for everyone and go with mystic quest. I know it's already had so many seasons. It's quite, you know, the last 1 they did, but during the COVID lockdown,   And it's also on Apple TV, which is, obviously a bit of a pay. But, you know, if you haven't seen Mystic Quest and maybe not seen, you know, It's Always Sunshining in in Philadelphia,   It's worth checking out. It's about a game development company, and it talks a lot about there's testers in there, some great series is where the 2 testing   ladies through always throwing kind of bugs into the problem. You've got people like Snoop Dogg doing stuff. There's some in the metaverse.  

It's, you know, fairly up to date, but it gives you an idea of of software development and how that, you know, it becomes an interesting storyline as well. And, of course, you know, there's lots more. I'm not gonna give you any spoilers. So you haven't checked it out. Mystic Quest is, you know, my pick at the 3rd time.  

[01:07:59] Unknown:

Alright. Well, thank you very much for taking the time today to join me and share your experiences   and insights on   the ways that automation can help alleviate some of the pain of quality assurance and remove some of the load on people doing the work. So I appreciate all the time and energy that you're putting into that, and I hope you enjoy the rest of your day. Thanks so much, and make sure you subscribe.   Thank you for listening. Don't forget to check out our other shows, the Data Engineering Podcast, which covers the latest on modern data   to subscribe to the show, sign up for the mailing list, and read the show notes. And if you learned something or tried out a project from the show, then tell us about it. Email hosts at pythonpodcast.com   with your story. And to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.