Industrial Automation with Jonas Neubert

Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great.

I would like to thank everyone who supports us on Patreon. Your contributions help to make the show sustainable.

When you're ready to launch your next project, you'll need somewhere to deploy it, so you should check out linode at ww

w.podcastinit.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your app or experimenting with something that you hear about on the show.

You can visit the site at www.podcastinit.com

to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. To help other people find the show, please leave a review on Itunes or Google Play Music. Tell your friends and coworkers and share it on social media.

Your host as usual is Tobias Macy. And today, I'm interviewing Jonas about using Python for industrial automation as a follow on from his talk at PyCon where he gave a wonderful presentation about using some Python libraries for being able to, control his mini factory that he brought on stage. So if you haven't seen the talk, I definitely recommend giving it a listen and a watch. And, we've got some good questions to follow on from that. So with that, I will ask you to introduce yourself.

Yeah. Hi.

So my name is Jonas, and I I consider myself, like a multidisciplinary

engineer in a way. I was, I was trained in undergraduate in grad school as a mechanical

engineer. Always been coding a little bit here and there, also picked up some electrical engineering along the way when I was working on robotics. And more recently, after moving to the San Francisco Bay Area, like everyone who does that, I steadily

became a software engineer. So what I'm really spending my days with these days is somewhere at the intersection of software and factories. So industrial automation, obviously, is, I think the title of our of the podcast here. So so that's what I do. And right now, I'm working for a company called Tempo Automation,

where we have a circuit board factory and and try to apply

really clever automation and software tricks to to get you prototype and small run circuit board super fast within 3 days. If everything goes goes well. You have a not too complicated job for us. And do you remember how you first got introduced to Python?

I I sure do remember. I it was a quite traumatic

experience, actually.

I used to like I said, I always used to be coding a little bit and, you know, since since high school, I think, and I always use PHP. And at at some point, you know, I I really didn't know that much about software engineering or anything, but I ended up making this app for this was when when Palm

still made smartphones like the Palm Pre, and we made a World Cup, like life

score app, and that got pretty popular. And this is the first time I ran into the situation that, you know, my my shared web host definitely didn't take the load anymore. So I made the snap decision to switch to Google App Engine, and back then, they had Java and Python to choose from as languages. So that's that was my very rushed introduction to Python the first time around.

I think,

after after that, I let it rest for a little bit, and then I started using Python in earnest just, 4 4 or 5 years ago when I started a job that just, happened to have Python as the primary language in. But since then, I've been using it, I think, every day.

And how did you first get involved in factory automation?

I've always had a had this interest

in how things get made in complicated

machinery. So I remember when I when I was in high school, I always I went to school in Germany, so there isn't really a notion of middle school, high school like this. But during these years, I also went to schools that were really into humanities

and languages.

So

engineering was never really part of my curriculum that much, you know, besides the usual maths and physics classes and things like that. But I I had put it in my in my head to to use that that industry internship that we had to do in, I think, grade 11,

to do that in some something that has to do with a factory. And and there was a a speaker who came to our school just to give some some talk, and he worked at a semiconductor factory that wasn't too far away. And I just went up to him and said like, hey. I'm I'm interested in factories. I don't really know anything about it, and I don't really know what I would be doing there. But sounds like you're an engineer at a semiconductor factory. How about, you know, giving me an internship? And and that's what he did. So I ended up spending, like, 3 weeks,

at this company called Infineon. They they're or they're long gone. They they used to be they used to make DRAM, like they were a big 1 of the biggest DRAM producers in the world, and they made these 300 millimeter silicone wafers. So it's, like, 12 inches, like just disks that basically came out of their factories and, you know, ultimately get sold in in little

black plastic enclosures that sit on a circuit board that used to be slotted into your main board on your computer. That's how it worked back then. Right now, it's all a little smaller. And I, yeah, I spent 3 weeks there, and they were adding this transport system in the factory so that instead of operators carrying around these really heavy

pods full of, silicone wafers, they would be running under, like, a rail system under the ceiling and and trying to get all the IT in. And

so complete complete random

chance event that I got to see this, but I basically,

from then on, I did I did a whole bunch of other things, did a little bit of building small robots in school, and, you know, couple of coding small jobs here there, but I always came back to factories for other internships. And then now, you know, doing it

professionally,

for, you know, for for over 5 years now, doing anything from steel factories where there's a 20 ton coils of steel flying over your head on cranes to my my previous job that I worked in for a couple of years was at a at a biotech company that had a a lab where, you know, the factory is basically working with with blood and DNA.

Very small, but same principles. And it's just really fun because these it it's almost it's like software, but you can watch it do things. Things are moving. So that that's what really got me hooked, you know, over a decade ago at that internship, and that's what still

excites me, like, really every day. Yeah. It's definitely interesting being able to see the things that you're working on manifest in the realm and a dimension that you can actually

tangibly experience as opposed to the very abstract notions that we all have to carry around in our head when we're actually building some of these more digital systems or web systems where the only incarnation of the work that you're doing is when it actually gets displayed on a screen, whereas all of the actual sort of logical bits are hidden from view. And so being able to actually see those implemented in mechanical actions must be quite interesting.

Yes. It it definitely is. And, you know, there's nothing wrong with writing software. Like, software changes lives, like, sure changes my life. It just seems to me like,

especially in software engineering,

everyone is so focused on on a few

domains of application. You know? Everyone's making like, these days, a lot of people are making apps or data science is is a big thing and and

nothing wrong with doing that. But considering,

like, the sheer size of the domain of making physical goods, like, you know, you look around you, all these things are made in a factory somewhere.

Very few

people actually have

an interest or a knowledge in working on these things, which is surprising, but it's also Obviously, it's great for me, right? Because I there's plenty of work waiting to be done in that field. And

if it's not so overcrowded, works for me.

Given that so many people who do work in software are dealing with these more ephemeral implementations

of that software,

what are some of the technical challenges that are unique

to a factory in physical computing environment where you're actually interfacing directly with the hardware and the actions that it needs to take in three-dimensional space?

Yeah. So there aren't that many unique challenges that, you know, once you've taken care of this kind of, like, instrument driver layer where you do whatever it takes to speak to the instrument, you a factory turns into this into a system that's actually quite similar to many software only systems. You know? Like, if you think about service oriented architectures that many people are using where, you know, different

servers offer, like, interface to do some sort of compute task. Manufactory is really nothing else except that it's not a compute task, but a a task to move a crate along a conveyor belt or to, I don't know, run a run a robot to do a to weld, 2 pieces of metal together.

So in a way, it's very similar. On the other hand, there are, of course, a couple of things that are specific to working with the real world that just don't matter in a purely software realm. 1 being that developers these days are very used to just, you know, spinning up a second and a third and a hundredth,

copy of a of a server

to, you know, to spread the load or something like that. Obviously, that doesn't work. If you have 1 robot, it's a singleton device that isn't easily replicated unless you have to spare change to buy a second robot. Some other things are, that that I've that I've struggled with or that I've seen others struggle with who come from a pure software background is that the time is, you know, is running at the pace it's running at. So physical physical actions take time to complete. So you're act you can't just specify that your services

respond within 10 seconds, write it in the spec, and just deploy enough compute power to make it happen. Some things take a minute. Some things take an hour to actually just perform because because physics. So so this is, you know, these are the types of things you run into, but none of them, you know, once you wrap your head your head around the application and and understand what you're trying to do, none of those are rocket science really. It's just it adds a couple of interesting constraints

to how you can write your code. And some things that matter elsewhere, like super fast compute performance, might not matter for you because you're working with a slow robot and there's no need to be fast. And I imagine that given the fact that you are dealing with these physical systems

that some of the concepts that you might be less likely to just sort of be cavalier about how you're implementing your code or deploying it because of the fact that if you do have some critical failure in the control system that you're sending to this particular piece of hardware, it might actually result in a failure of the physical device, which could end up costing, you know, several 100 or 1, 000 of dollars to replace. Whereas if you have a bug in your web server and you deploy it and it fails, well, then you just, you know, destroy that instance and rebuild a new 1. So, what are some of the testing methodologies that you have to be a little bit more careful about and, some of the ways that you do ensure proper reliability and safety in the code that you're deploying to these instruments?

This is a topic you could spend, you know, months talking about, and, you know, there are people who actually talk talk about it for months. But let's try to let let me see what other, like, the the quick sound bites to to break out of that. 1 thing I I covered in my talk is that you you have to be careful which type of technology you, like, what you apply it to. Generally, as a rule of thumb, if something is critical to the safety of a machine or even the safety of humans, you, using Python for it seems seems not right because Python, you know, doesn't have real time guarantees

and Python, you know, Python code runs usually on operating systems with other tests running in parallel. You know, it might stall, it might crash. So safety critical things, if you find yourself in a situation where you use Python to implement a safety critical feature, take a step back, think about it, maybe use something that's more appropriate for that.

In in my talk at PyCon, if you watch the video, you'll see that I have 1 of those programmable logic controllers, PLCs, which are very frequently found. Pretty much every factory will have 1 or many of those. And and

those, you know, you program them in this ancient language, and it kind of seems funny if you think about it through software developer eyes, but they are designed

to be, you know, fault tolerant. You if you have a safety system like a light curtain hooked up to them,

you'd probably be using extra expensive gear where all the wiring is duplicate or triplicate to be fault tolerant and things like that. So safety of humans and machines is a very serious topic and that you really wanna make sure you design you design that correctly. So that that's 1 aspect of that. The other 1 is testing, big scope of how do you test things

when when the, you

know, like a side effect of the execution of your code is is a robot, like, flinging around a half ton car, say, or something like that.

And there are a lot of standards and best practices that people in industrial automation would use. So there's like this entire field and many books written on acceptance testing. So very much analogous to how software developers would be used to having unit tests and integration tests and maybe a continuous integration type system that's running their tests for them before doing a release.

There there are analogous methods

in in building industrial sys like industrial

automation systems and factories where you usually start with a functional requirements document where you just enumerate in in painstaking detail what's supposed to happen like this. If if operator presses the green button,

the conveyor belt starts moving at, I don't know, 10 meters a second, something like that, and reaches that speed after a speed up phase, you know, within, I don't know, within 10 seconds, something like that. If, you know, if, if this sensor detects

that a light broken

a light curtain is broken, the robot arm stops within, you know, 10 milliseconds. And if it if it happens to be holding a sharp object, you know, we we test that the sharp object doesn't penetrate deeper than 10 millimeters into

a into a test object that's as soft as a human would be. Things like that are actually, like, in these requirements documents. And then, the way it usually works is that the,

the person

or the group of people building a system like this are different from the ones operating it. So if you operate a factory,

you usually don't have software developers and integration engineers on hand. You ask a system integrator to go out and get all the components, put them together, do the programming, put all the wires together, and then they ship that system to your factory. So there there's a pretty I think it's a pretty common commonly used protocol of having a a factory acceptance test first where while it's still in the place where, you know, at the system integrator, they test all the functional aspects that make sense to test when you have the people around who built the system,

and, you know, that that's basically, like, can the robot arm actually reach all the places where it needs to pick things up and drop them off,

and and, you know, do, like, do all the switches work, do do all the things that are meant to move move, and all the things that are meant to be fixed, already fixed. Once they're happy with that and, you know, iron out all the the little bugs, in the system, they ship it to the factory that's trying to actually use the system to for their for whatever their process is and then you do a site acceptance test, which which is all the things that make sense to do when you have the the end users around. So this is more like the integration test in in a software sense, you know, where people go around, press buttons, and and and actually try to to build whatever widget it is they're trying to build with that machine you gave them. And and, you know, that's just a very high level. I'm actually not really an expert on this. Actually, there's, like, there's another there's a whole other dimension if you are in a regulated environment. So if you're doing laboratory automation and you do, like, diagnostic tests, then you have to deal with reg regulatory agencies who come in and do an audit, which is a whole other area, and there are entire job titles dedicated to just dealing with all this. So so I'm by far from expert in this, but that's roughly what you do. And it it all maps kind of 1 to 1 to software development concepts like unit testing for the small functional tests, integration tests, for actually executing protocols that people would do, and so on. Yeah. From a high level, it definitely seems like you'd be taking sort of similar approaches of the idea of the testing pyramid where you have all of your fast, easy tests that aren't dependent on actually interfacing with the hardware to just exercise the software itself. And then when you have to actually ship it and make sure that everything's working together, like you said, you have the integration testing as in the form of the factory accepts

factory acceptance.

Yeah. That's that's right. And in a way, even with the little mini factory that I brought to to Python, which was just a conveyor belt, a barcode scanner, and a and a PLC to control it all, I actually had all of those. Right? So when I first arrived in Portland,

I opened up the suitcase that, you know, the the demo was traveling in, and I had a little almost like a unit test script that just, like, actuated every actuator once. Like, each of the 2 little paddles that were there for sorting things got just, like, flipped once. The conveyor went, like, on, went through, like, 2 type like, 2 speed settings,

and the and the barcode was trying to be scanned. And that was, like, you know, my unit test suite. And I carried

actually did the functional test of actually running the demo that I was going to run on stage. And, you know, that's, you know, toy little toy example, but that's in the end, that's what you have to do to confirm that it's actually working. And I imagine too that given the fact that you do have the physical interface and the the mechanisms

that are operating within the, you know, robot or the piece of hardware,

that that would also introduce an entirely new dimension to debugging issues because of the fact that, well, the software itself may be executing perfectly. There might be some,

mechanical failure that's happening that causes the actual action not to be formed exactly as it's intended to be. So while you may initially suspect that there's a glitch in the software that you deployed, it turns out that there's actually, you know, something with the physical interface that you have to find an address. So as with testing in general, you get into a combinatorial

matrix that it's just sort of impossible to really find your way out of, and you just have to do best effort. That that's true. I mean, the the concept of mocking takes a whole new dimension when the mock you're writing is for a physical machine and not, like, let's say a web server. And yes,

simulations

of machines,

like mocking out simple things like barcode scanners,

is definitely,

something you do as you, you know, if you're on the software end of writing an an industrial automation system, you will you will use these mocks. However, if you're, like, on the on the controls end and you're in charge of, you know, wiring up all the sensors, then your debugging toolkit will probably include include

multimeters and protocol analyzers. And the coolest ones in my mind are the bugs that are completely unclear where in the stack they are, where, you know, you you have a barcode reader and every 100 scan,

1 of the characters is missing. And, you know, it could be anything. You could have a bug in your code. You could have the barcode reader misconfigure or maybe it's just broken, and months later, you find out the reason is that you ran the cable too close to a to a motor, which introduced noise into the transmission. And because the vendor used a protocol that doesn't have checksums, the serial transmission got scrambled. You know, that's an actual story, and I think those are the coolest bugs to debug. But, obviously, if if all you're trying to do is earn money with the factory, those are probably the worst ones.

Yeah. I was just gonna say, it sounded like there was some actual, experienced pain behind that story there. Yeah. I mean, I I shouldn't take too much credit for that. I think most of the pain was experienced by my colleagues,

on on that 1 who were doing, like, the control side of things. But yeah, I mean, true story that that's something that definitely happened.

And so when most people think factories, they're not necessarily going to jump immediately

to Python as the language that controls them. So I'm wondering what are some of the languages that are most frequently used for command and control of these industrial systems, and how does Python play a part?

Yeah. So,

really, the the short and honest answer is that Python plays no part unless you make it play a part. Python is really a tiny little niche in this area. There are many languages involved,

and it really depends

entirely even what industry you're in. Many many pieces of equipment just come with vendor specific

languages. Oftentimes, this is like, you know, just text files, like lists lists of instructions.

If if you've ever worked with 3 printers,

you might be familiar with G code, which is just like this long list of instructions for where to move the print head and and how fast to deposit material. That's that's actually a language that's been around forever for computer controlled machining, like CNC machining,

anything where you have a gantry that moves in x y z and then machines would use that language. There are other pieces of equipment,

and I find that a bit of a disturbing trend where there's just the assumption that you don't program them in a in a software development sense. You you use, like, GUI tools,

and they give you, you know, usually some Windows software and you click something together.

Sometimes it's like lab view graphical programming style. Sometimes it's, you know, much more primitive things.

Then then when you get to actual real programming languages, like the way you and I would think about it, there are some there are some traditional ones. 1 thing you have to keep in mind is that factories have a much longer lifetime than your typical web, you know, web, application.

You know, they they might just sit there for 20 years without, you know, any significant updates. So a lot of these languages say that that get used to are pretty old. So 1 of them, like the 1 that there's a standard called,

let me look, let me look this up here in my notes, IEC, 61131,

which is a family of, I think, 5 languages that you use to program these real time control systems.

The, programmable logic controller that I mentioned earlier, it looks a little bit like Pascal.

That's the 1 I use that's structured text. There's a thing called ladder logic, and I really encourage everyone to look this up. This is like a a graphical language where you have, like, 2 lines right and left that represent basically wires that have a voltage on it, and you connect them with, like, switches and and elements like that, and you program that way. It's kind of a cool concept, but really hard to explain.

You know, you don't see how I'm waving my hands in the air here. So there are some of those languages. And then as we get, like, as we get to more modern things, it's it's mostly a Microsoft stack that you would find. If if you do find, like, a a modern software stack in a factory, maybe because, you know, it's a more recent factory that that was built within the past decades or or because you're in an industry that just is more advanced, like semiconductor being 1. Most of the software is is like a Microsoft stack, and that's actually that makes it all the way through to the industry standards. Like, there's, there's an a standard party called OPC,

which is pretty widespread,

in industrial automation, which is a standard for how these, you know, real time systems share data and make it so that the data that they get from sensors can show up on the screen somewhere to be monitored, for example. And the original version, which I think came out probably 20 years ago, was definitely, I think, 10 years ago, that was the the standard specified that it had to be,

a COM object. And I'm not super into Windows development, so I don't even really know what all that means, but it it was called OPC OLE, I think. And only more recently have they caught on to the idea of maybe being, you know, operating system agnostic and not specifying, you know, specific,

basically, vendor technologies for this. But, anyway,

if you if you're going out there looking for,

software engineering and factories jobs, more likely than not, c sharp will be on the requirements list and generally, like, dot net type skills. That that's really the common 1. And

and Python, I randomly got into the Python

and automation environment jobs, but that was really a weird case because the company had started without a factory or in this case a diagnostic lab and then became

a wrote a lot of software in Python and then started also operating this lab, so they just stuck with Python. So but if you have situations like that, you get to use Python

in industry. Otherwise, you might have to do a little bit of work to to get it in there. And then obviously on the periphery,

where, you know, data that's already come out of these systems need to be analyzed, just like everywhere else, people use whatever fits best, and I think that's where Python is making the most inroads data analysis

on, you know, sensor data and things like that. I think that's where you find it where you're most likely to find Python. Yeah. And I imagine too that in terms of being able to glue together some of these different

proprietary systems with their own particular protocols and data formats.

And, you know, as you're mentioning, some of the systems that ship with only a GUI tool being able to reverse engineer some of the ways that it's trying to communicate and actually build Python wrappers for that to be able to glue together the entire factory seems like it would be 1 of the areas that Python could very effectively be leveraged within an industrial environment.

Definitely. And and, actually, this whole converter thing, that that's a nice that's a cool little cottage industry. There's, like, 100, probably thousands of little companies out there that sell you little boxes that translate from 1 protocol to another and, you know, like serial to Ethernet,

old OPC standard to, I don't know, some Ethernet based standard, stuff like that, and and Python can do that job. I think,

the job where

where Python makes even more sense is at the interface of these factory control systems

and other software systems that have previously not even been connected to the factory, such as, you know, inventory management systems

and and things like that. So to to to bridge the gap

between things that are already written in in some kind of modern software and the more traditional controls type thing, I think that's that's really the sweet spot where where Python as the glue makes makes a lot of sense. And in your Python presentation, you commented on the fact that security in industrial automation systems is

generally fairly lacking,

which particularly given the fact that a lot of these facilities are, as you mentioned, you know, potentially decades old in working with

hardware and software that hasn't necessarily been updated since it was first installed. So given that, what are some of the most common issues that you have seen, and what are some of the other factors that play a role in introducing those different security vulnerabilities?

So to be fair to all my colleagues out there in industrial automation, I didn't think I I think I didn't say,

in the in the presentation that security is generally a problem. I think what I said is that if you're focusing on the security aspect of automation, you will have fun.

You said that you'll be busy, so I was reading a bit into that. That's right.

Yeah. So the the main problem is that there there has been a recent trend to connect these systems that, you know, either were built before there was the Internet as we know it or that just were built without an intention of connecting them to the Internet, and now we connect them, you know, to the cloud and and things like that. And I think that's really where the security problems arise. So actually, this is a kind of a,

a toy example here, but 1 thing that happened to me this week is I was playing around with a barcode

reader, like 1 of the gun type ones that, you know, has, like, a trigger, like a gun, and you just plug it into your computer, and it actually acts like a keyboard. So you have something with a barcode on it. Let's I don't know your your airline boarding pass. You, you know, click the trigger on that gun. The little light comes on. It it goes, like, beep and reads your barcode. And what actually happens is that it like a fake keyboard types in these characters. And, you know, systems like that are everywhere, distribution centers,

at the at the airport, obviously, your grocery store, everywhere. And they were obviously built with this 1 very specific application in mind, like this 1 type of barcode, and they always type in what what they see. So this week, I was using 1 of those on a barcode that happens to be floating around in our factory, and I was having the Chrome developer console open while I had to type in a barcode into a text box. And suddenly I had a breakpoint set in my JavaScript.

And what had happened is that these barcodes, they don't only include

readable, like, printable characters. They also sometimes include, like, lower ASCII, like, control characters and things. And this specific 1 ended up typing f8, which apparently, if you have React on your website, triggers a breakpoint in React somehow.

You know, and this is this is, like, this, I think, is a great example of how you suddenly get an attack surface on systems like this. Right? Now you can start thinking, oh, well, if I can put, like, an f 8 key into

a a barcode, can I put the Windows key in it? And if I can put the Windows key, can I start any program on the user's computer without them having really any chance to stop it? You know, how far can I take this? And that's just the barcode.

Now you have systems. You know, a barcode has, like, maybe a 100 characters in it if you have a a big 1, like a a 2 d barcode. Now you have systems in your factory that were built with the same kind of mindset

of just making something work that communicate,

you know, that maybe used to communicate over a serial protocol and they send each other thousands of characters every minute,

and someone had the genius idea of maybe we should run that content over Ethernet and didn't realize that that Ethernet

is also connected to the office network, and before you know it, you have, like,

these crazy hacks that hit the news, like Stuxnet, when, you know, some government entity attacked the Iranian,

uranium

enrichment,

facilities, like this was, I think, 3 or 4 years ago, and, you know, they defund a way to deploy

some of the software inside that facility. And once they were in, they, you know, I mean, I don't know the details, but I assume they were listening on a network on what's going on there and then just started talking too. And because these systems weren't designed to authenticate who's speaking or use encryption, that might have been quite easy. And, I was actually just reading this week,

this report. It's by a security company called ESET

about a a malware called Industroyer,

which I think is an awesome word. And apparently,

this Industroyer

might be responsible

for for a big power outage in Ukraine just half a year ago. And there was this description of how it works, and it it barely qualifies as a hack. It's basically software that happens to implement these these standards.

You know, I I just wrote this where I said I wrote this down so much as before before the we started recording it. Like, IEC

61850,

apparently, is a standard for how electrical substations in the electrical grid communicate. And, you know, Industrior basically just implements that standard and knows what commands to send to switch off a couple switches or something like that. And that's that's the security aspect of industrial automation. And there are, you know, very promising

trends. If you go to into, to trade shows, there are companies there that, you know, sell you products or libraries that do very clever you know, that that just do best practices, encrypt your communication,

check authenticity, stuff like that. But there's a lot of old technology out there, and there's I think, we just haven't seen enough hacks that there's a strong security

mindset among the people who built these systems and use them. So I think we'll we'll have another couple of years or maybe a couple of decades of pretty exciting stories about hacks in that space coming up or, you know, for better or worse. Yeah. And also given that a lot of these systems that do have these vulnerabilities

in them, there's a lot more of a capital expenditure needed to be able to actually

upgrade or replace them versus, you know, adding an encryption library to your software that's running your web server.

That's true. I mean, it it depends. Right? Some things, it's as easy as

not using a default password or

maybe,

you know, not plugging it into the into the company wide network after all because, or maybe having just like having someone install a firewall. You know? There's, like, some simple

practices that I'm sure, you know, some listeners are laughing at me now. It's like, well, of course, everyone's doing that. I have a factory and I have a firewall. Like, what else would you be doing? Well, I've seen a factory, you know, that doesn't do that just because it's, you know, a small operation and they they don't know. So I think it's the same as everywhere. Right? We had, like, all these credit card hacks and now we slowly get, like even in the US, we get chip and PIN. And, you know, in Europe, they do, like I don't know if they still do verified by by Visa or get, like, transaction

numbers. Once it goes wrong a couple of times, people, you know, catches on and people start acting on it. And I think,

we're just at this point where the field of industrial automation learns that lesson.

And so for doing development

that's targeted at a,

factory or a, you know, industrial facility,

how does the overall

life cycle of that software differ from when you are doing, you know, web app development where you have the sort of QA environment and then the production environment? And

what are the life cycles like

on production rollouts where a lot of companies that are doing purely software based systems are moving more towards continuous delivery where they'll have multiple deploys a day. Is that something that you see at all in these manufacturing facilities, or is it more the typical waterfall approach where you have a, you know, large set of changes that you will batch up and then deploy all at once to a system at, you know, set intervals?

I I think there's a broad spectrum depending on the application, and there's also a bit of a change happening, you know, currently just as technology gets more modern. 1 thing is if if you have

a steel factory that's making steel or, you know, something that's just not super like, where the process isn't changing, that's not very IT intensive. The model there really is you write it once, goes into the factory, and then it runs until the factory gets decommissioned.

So that's the life cycle there. Unlike more,

there are many applications, however, where that, get hooked up, let's say, to the cloud now and where

maybe sensor data gets gets streamed at very high measurement

frequencies

so that there are machine learning systems that try to predict,

when maintenance is required or predict failures. Like, that's a very common thing. It's condition monitoring. That that's really taking off now. And when you hear industrial Internet of Things, in many cases, what they actually are talking about is this base use case of let's get all the data we can get out of our factory and analyze it. If that's the area you're in, I would assume that if you're, you know, if you're writing a software that's closer to the sensors, then you're maybe on, like, a monthly release cycle. And the further away you get from the machinery, I think it's you get more to the continuous delivery type mindset. And

so, it really depends on the industry you're in,

and

what it like, where exactly the software you're writing is in the

stack. 1 of the things too that I think would be fairly challenging is the idea of having a QA environment that's a close replica of production because of the amount of capital expenditure that's necessary to build the factory in the first place. You're not gonna build a second 1 just to test all of your software. So that's where I think we go back to our earlier conversation about testing and being able to roll out these changes in a, in a staged

way where you have all of your unit tests to make sure the software functions and then doing the factory acceptance testing. So I imagine that would also affect the overall release cycles that you can work with when you're actually directly interfacing with the machinery.

Definitely. And 1 thing I should also say to qualify

many of the things I'm saying is that this role that I'm in, where I'm a software engineer that's working at the company that's operating the factory, that's extremely rare. In most cases, at least the software that's close to the control system and, you know, running the machines, that's written by the machine vendor and

they obviously have different different

ability to test things. Because if you are, you know, if you are, like, a robot arm vendor, say,

you sell, you know, thousands of these robot arms every year, obviously,

you have, like, a bank of tens or 50 of them for for your testing. Similarly,

and then and then if you get to system integrators people who actually write software for for a factory they own, like, which is which is the the role that I'm in, then you have the inverse. You have much more control and you have the freedom to deploy more software, but you have much less ability to test with the safety net. I I've been in an environment where we got pretty close to having a replica of the factory, but I'm I'm I've also been in an environment where every time you deploy software, you cause downtime

and then you have to do testing with the end users, like all the you know, you did all the factory access and test and and site acceptance test and user acceptance test for all the same thing. You tried to do that in, you know, maybe a 2 hour overnight time slot, and the next morning, the factory had to be working again. That's also a possible environment. And I prefer the latter because even though it's more stress, it obviously gives you a lot more freedom to do to innovate

and

to also to iterate and and and actually make, tease out that the last bit of performance or of your factory or build cool new features that others can't replicate because they would have to call up some machine vendor or systems integrator to build it. So there's a trade off there.

And aside from manufacturing facilities, what are some of the other types of industrial environments

that might require the same kinds of systems integration or hardware automation that you're aware of?

Yeah.

There are a lot. So when when I say industrial automation, I sort of implicitly always include laboratory automation because that's that's the setting that that I've spent many years in. And while there are some differences,

in the end, you deal with robots and machines that have some sort of API or

is is a huge area that's also, you know, distribution centers and,

and things like that. Everyone loves ordering from Amazon now, and these warehouses are, you know, in effect, gigantic robots

that just ferry things around. You know, things like airports, the luggage handling system,

again, it's just automation all the way. Like, it's it's basically a factory. All the infrastructure around us, I just mentioned the power grid in the context of the security question. Those are the things that come to mind readily, but you just look around you and the the built environment and and all the things that you're using every day and you think about where they come from, it's probably a place that that's using industrial automation technology.

Yeah. And also as things like self driving cars or smart cities

become more prevalent, I imagine that'll also introduce new arenas where this kind of skill set would be able to be applied.

Definitely. And and cars

are actually quite similar. I,

I I 1 of the talks I went to at at PyCon was was actually about hacking cars. And just like a factory, cars tend to have many processors distributed throughout an an onboard network, and, of course, there are motors and

get

get the windows up and down and all all these things. So in a way, cars

are very much equivalent to factories, but, obviously, then there there are differences. Like, the communication protocols are different

and, you know, you use different products. But I think very similar principles apply. And what are some of the most interesting or challenging projects that you've worked on?

Oh, that's a broad question.

Yeah. I don't know if I have an answer already for that, but there there are, I mean, challenging

in in this in this type of work, probably like most other places, there are 2 types of it. There's the project that's challenging because it's incredibly deep and you just have to you have to be 1 of they call it don't they call it like a t shaped person who has, like, an enormous amount of expertise in 1 area and and and can cover that. And, well, that's not really me, so I probably don't have an example for that. And then there's there are the projects that are that are challenging because just of the sheer complexity of all the moving pieces. And in this case, moving is, like, literal. Right? Moving pieces in the factory. And those those are abundant in in automation. I think the most challenging 1 was definitely the this diagnostic lab that I worked at. Now that I've mentioned it like 10 times in in this talk, I should probably mention what company it was. So I I worked at this company called Counsyl who offers genetic tests. And when you say genetic test, the people who are somewhat knowledgeable are aware that there's a sequencing machine at the end that performs the sequencing and turns your DNA into a series of characters

that can then be analyzed. It's like the helicopter

height high level view of what's going on. What most people don't know is that there is, like, a sequence of, you know, a 100 steps before that where your blood or your saliva that, you know, your your doctor probably collected from you gets broken down so that there there isn't even any DNA to be analyzed. It gets broken down and kind of sorted and filtered and and all these things. And each 1 of those, you know, when I say filtered,

this might involve 10 high level process steps, each of which can be broken down in another 50 little ones. So we built this we built this facility where the ultimate goal was that really, like, your sample tube of blood or saliva gets insert like, it's inserted into a machine. Somebody clicks like, hits go, and then it just goes through all these steps. And and doing that with, you know, hundreds of thousands of samples

every day. And we in terms of the software, we ended up so I was leading the software part of the automation team at that company,

and we ended up with literally, like, hundreds of individual,

virtual machines running our servers, each 1 being in charge of either talking to an instrument or doing some kind of work cell level scheduling where you have like a handful of machines that work together, all the way up to scheduling basically what's going on in the building, making sure that everyone's samples in the right place at the right time. And and that was 1 of those projects where that was just extremely challenging because of the sheer complexity of all the things that are that are happening there. And for every instrument, you have to, you know, go from the detail level of how are the bytes aligned in the communication protocol all the way to when does it need to be switched on so that I don't heat the the sample for too long, things like that. And that was challenging. And that's also, like, for me, that's the kind of project I just love working on with which is why I love working on factories. So so those that was very cool. For my day to day, I work on doing cloud automation, which, again, it's interesting because you're tying together all these various bits and pieces to be able to get 1 continuous flow that ends with the result that you're looking for. But I imagine that having the extra layer of complexity of actually having all of these physical moving parts

and all of the different domain knowledge that's necessary to be able to properly drive the different

mechanisms from a electronics perspective as well as from, you know, particularly in the case that you were saying about doing the genetic testing of understanding

the material limitations

of the substance that you're processing to make sure that it's within its tolerances during this entire sequence of

events. Sure. And it's 1 of those things that you just can't work on this as an individual. You know, there are some some problems in the world a person can solve, and then there are others that are just too big, so, you need a team for it.

And that, you know, I think both your line of work and mine definitely fall into that latter category where you you have a team. So I never had to understand genetics to to help automate the genetics lab. And you stand on the shoulders of giants. Right? There are other people who who are that other type of person I talked about, who are, like, you know, the t shape, who who, go really deep on 1 thing,

and there's just fascinating work going on at the super detailed level. I talked with, with a gentleman who works at a company called Cognex who do vision type systems and and barcode readers, and he has been working on this project where when barcodes

get read, you want to be as fast as possible because that means that if you're in a distribution center, you can move things past the scanner much faster. So he was telling me how the software innovation in that field is getting to the point where you need less than 1 pixel per bar in the barcode to still be able to read the barcode. So, you know, just think about, like, the cool algorithms work that goes into that or people who work like

actually, same person who told me about this, like people who do barcode reading again, but oftentimes barcodes get damaged, and if you operate a factory and you have like a level of 10% of damaged barcodes, you want to know why. So someone out there has gone out and written an algorithm that figures out what's wrong with the barcode. Has it been torn or folded over or is something stuck on top of it? So you, you know, there's like like anywhere else, like, you know, in some cloud computing, same as in industrial automation, there there's just an amazing number of little projects done by by really clever people, and only because you put it all together,

does anything in this world work. And so in your day to day, what are some of the most useful Python packages that you leverage?

Yeah. So I knew that this question was coming. So I was,

like, racking my brain on what a good answer would be. And surprisingly, I couldn't really come up with any that are specific

to industrial automation. The 1 I could say that I've been using a lot is PySerial, which is the, you know, a very old, very powerful Python package that you can use to talk to the serial port. But then again, that's, you know, that's just 1 example. So the other the other packages that I I ended up putting on my list by going through, you know, requirements files that I've been touching recently are actually just ones that are related to to testing. I think those are like, I I have a list here with, like, things like factory boy for Django to to do, basically deal with

fixtures in a better way, the parameterized

package to to write tests in a more efficient manner, freeze gun.

Time often like, I mentioned time earlier is, like, 1 of the things that you have to be more aware of when you do work with physical systems than when not. There's a package called freeze gun that you can use to basically freeze time in your tests. That's 1 that I've I've I've used

quite often. So these are these are the things that that end up being on my list. For the actual

for actually talking to instruments, quite often the standard library is good enough. You know, if you're just, like, putting, like, putting a bunch of bytes together to send out just the right byte string that an instrument needs, well, then you've got struct. If you need a little server like like what I demoed in my PyCon talk, just like a an HTTP server that you can use to receive commands from over the network and turn it into something to make an instrument do something. Well, then there's XML RPC. That's that's there in the standard library.

So, yeah, long story short, I didn't really have come up with a good list of packages here.

Well, it's actually kind of a good sign when you're able to do most of the work that you need to do just with what already ships with the language and not necessarily having to reach out into sort of exotic libraries and packages that are necessarily domain specific. Yeah. And that that was actually 1 of the points of my my Python talk. I mean, I used PySerial. Okay.

But beyond that, most of the things I showed were either pure standard library Python

or, like, pretty small packages that

do a bit a little bit of convenience and, you know, add a little layer of abstraction

to to make the commands shorter. But 1 of the points was to say, hey. Look. This is easy, and you can do it. In fact, someone told me after the talk that this was this was a talk that really gave them the sense that they could sit down,

like, right now or right after they get home from PyCon and just start doing this because

it's very approachable and accessible. There's no crazy

science to master, to get started, you know, making a little robot move from 1 end of the table to the other. It's it's actually very easy. That's kind of was kind of the point of my talk, actually. And for somebody who does want to get involved in industrial automation or working with manufacturing facilities, what are some of the kinds of experience or background that they should have, and what are some of the resources that you recommend people take a look at if they do want to get involved in that line of work? No. That that is a very tricky question actually because

it's definitely not the field where where there's an abundance of blog posts out there. Like, how to get started controlling conveyor belts is I bet you there's, like, 0 results if you, like, look for that. So it

depending on where, like, where you come from, there might be different approaches. So if you're already in an area like machine learning, I think the right approach would be to just start moving gradually moving to applications where the data source is industrial systems. And with this whole industrial Internet of things trend going on, that should be something that's relatively easy to find both in terms of, like, if you're looking for a job, that's a, you know, industrial Internet of things could be your keyword or if you're looking for tutorials

or white papers and things like that, that would work. If you're really trying to get into

that that type of software that I was talking about a little bit here, where you control individual instruments, the prerequisite really is to know a little bit about how a factory works. And even though everything

that we have is made in factories,

people know surprisingly little about, like, what even what the pieces are that that make up a factory.

So you would really start there. And some things I could recommend is try to get on factory tours. I think that's that's the biggest 1. In some in some cities,

there are factory tour meetups

where, you know, groups of people just go out and somehow get themselves invited for tours of factories. Sometimes when you're the customer of a product or you know someone who works in a factory,

you can you can get a tour relatively easily. I know that Tesla the Tesla factory in Fremont here in the Bay Area, they offer that to employees and their friends. I think they have a regular tour schedule going on, and I have so far not been able to get in. Well, if there if anyone who's listening wants to invite me, just, my contact details on the show notes, anyway.

Then, if that doesn't work, there's just YouTube and I guess, television. Also, if there's still television watching people out there, there's, like, these shows, like, how it's made or, like, just a lot of just search for factory tour on YouTube or, you know, if you're into cars, maybe. I know that, BMW has a has a set of really cool factory tours out there.

Also, just machine vendors trying to sell their machines. Any machine you can think of, just type into YouTube, like, machine name factory or machine name demonstration,

machines make paper clips, watching machines make screws, watching machines make, I don't know, you know, like packaging salami. Like, just whatever. Like, pick a thing that is in, like, your field of peripheral vision right now. Type in object name, how it's made so you to get an idea of how, like, what are the machines that you would encounter. And then if you're getting a little more serious about it, once you have a handle of just like rough ideas of the nature of machines,

you can

you could guys start going to trade shows. Generally, vendors and manufacturers

are the biggest source of educational material in this area. There aren't really any books about many many of those topics. Like, I know that there's 1 1 big book about barcodes, for

papers and papers and documentation

and maybe on-site demo if you work for someone who could pass off as a potential customer and just have them teach you all about it. That's really the way I would go. A couple of more scrappy ways, you can go on Ebay. There's an entire industry of people who sell secondhand factory equipment. Just go to the right section there. Just start exploring. If you see a thing that you don't know what it does, Google it,

and find the manual and just start reading. And that that that's those are the things I could come up with when I was preparing, you know, for because someone asked this question on Twitter, I think, and and I saw that earlier today. And I was trying to think, oh, what am I gonna tell them? Also, I oh, 1 more thing, catalogs. Just go go to the catalogs of manufacturers or there's companies that sell them. Macmaster.com

is a website that sells anything from screws to programmable logic controllers. Just take a look. Like, even finding out what the options are, is really interesting. Like, for my PyCon talk, this was the first time I was buying a conveyor belt, finding out like what are the things I can specify when I purchase a conveyor belt. That was extremely educational actually. So yeah. But but,

really, the big takeaway is, you have to lean on the people who build this equipment or sell it for information because there aren't any

conveyor belts for dummies books out there by O'Reilly as far as I know.

Looking forward, what are some of the innovations in industrial automation that you're most excited about? Well, I'm excited about everything,

every application

where the factory gets connected to complicated software systems. So every time someone deploys some machine learning,

algorithm

to a factory or some advanced scheduling algorithm. Those are the things where

where, you know, the the interface of modern software development,

the things that we know of from, you know, the Googles and the Facebooks and, I don't know, the the Yelps and Instagrams out there, like, the tech they're building where that sort of that that software

that level of software get

caliber gets combined with with industry.

So I mentioned this earlier, there's a big trend to connect sensors to advanced algorithms and just in general, like, the the cloud to have the data there and then do useful things with it. The next step, of course, is to go the other way and send, like, control what's happening in your factory based on information that lives outside. And if you take that to the extreme, so the simple version of that is you have a car production line, and the cars that are coming off all come in different colors and with different

interior options. Right? So that's obviously

software on the outside that's running, like, the the orders database and the inventory database controlling what each robot is doing as as it assembles the car, making sure that the right type of seat is right next to the production line when the car that needs that seat gets assembled. So that's it's like the base level that's been going on for many years. What's happening now is that this mass customization

starts becoming feasible where every item

that's going through a factory is completely different. So you see that, like, 3 d printing is a great example. There are people who,

make customized

insoles or even entire shoes. Once you have a factory where every shoe that's coming off has a name tag on it because it only fits that person, that's really cool, and we're actually getting there. Likewise, you know, other applications are prototypes.

So the company I work for, we make these circuit boards, and the problem of making thousands of circuit boards very quickly in a highly automated fashion. That's been solved for a while. The problem of making 5 circuit boards of a type in a very automated fashion. That's something that only getting to now because that requires hooking up the software

that, you know, ingests the order and actually figures out how the machine to be set up, hooking it up to the machine. So I think that that weird thing where the interface of software systems and programming

on already automated machines to be even more automated. That's something that that I think is really taking off now. And and then, actually, 1 thing I should definitely mention is just life sciences in general. If you've been if you know anything about biotech or gen genetics in particular, maybe you've heard about CRISPR, like, that field is just taking off, I think.

People are, like, synthesizing

genomes. You design a genome on your computer, and and you you send it to a factory that just synthesizes

that DNA that actually puts, like takes in a string you send and put the DNA together, send it back to you, using that for a variety of applications. I think that's just fascinating,

and there was also a PyCon talk about this.

Let me see.

I think Riley Doyle was the speaker, and he was talking reprogramming the human genome. He mentioned a couple of those examples where where Python is actually used to power applications like that. There there used to be a, a company called Transcriptic

where that literally had a web API that lets you control their lab robots to do biology

experiments for them. I think they are not doing this in a fashion that's open to the public anymore, but I think that sort of thing is taking off. I know that Autodesk,

the company that's doing, computer aided design software, is starting to get into life science software. So you much like an architect would plan a skyscraper

using Autodesk CAD software, there's now software to construct genomes and then, you know, again, send it up to an automated factory. These are factories that could never be not automated because no human can actually perform that work, and

and then they they have to ultra customize everything that's coming off the line as different type setup. That that's something that I find really exciting, and I think we all see a lot of those applications

where everything is different when it comes up. Like, every every item coming off the production line is different from the 1 before. We'll see a lot of those, hopefully, over the next 5 to 10 years. Maybe it takes a little longer. Alright. Yeah. It's definitely a very interesting and wide open area. It'll be interesting to see how these things evolve

as technology increases

and the capabilities of these facilities increase

accordingly. And so are there any other topics or questions that you think we should cover before we start to close out the show? I think the 1 thing I should say is,

so we are hiring at Tempo Automation. I I already mentioned what we do. So if you're at all interested in electronics,

circuit boards, have any background in that area, or just really like using Python for for cool applications,

you should, check out our website or just contact me. And so we're looking for, yeah, for people familiar with the Python stack and really interested in working with factories. Alright. Well, for anybody who wants to get in touch and follow the work that you're up to, I'll have you add your preferred contact information to the show notes. And with that, I will move us to the picks. And so this week, I'm gonna choose the band OPETH,

which is a progressive metal band out of I can't remember exactly which country, but I believe they're from Scandinavia

somewhere. And, yeah, just a really interesting band, a lot of different musical styles across their different albums. I've enjoyed the enjoyed them for a while. And, if you're into

sort of progressive rock or progressive metal, I definitely recommend giving them a listen. And so with that, I will pass it to you. Do you have any picks for us today, Jonas? I do. And I thought that because

PyCon is still not too long in the past, I would just have a small selection of of PyCon talks that I enjoyed. And I also didn't really get to attend that many talks, so most of these I watched on YouTube after the conference. So there were some that were in in the same vein as, you know, my talk about factory automation or someone's using Python for something else. I found a really great talk,

was by by Eric Evenchyk, I think is his name, hacking cars with Python. So we already kind of alluded to that during talk during talking here that you can use Python to to do to work with cars much like you can use it for factories. There's another talk about,

a bike speedometer with micropython

by by Tim Head, I think is his name, and that was that's something I just have to do more reading about, about micropython,

which is using Python to program microcontrollers.

It's a really cool thing because programming microcontrollers,

like embedded development, is hard. And if you can use a language like Python that's pretty, you know, approachable

for it, that's great. So that talk

was was good fun. Katherine

Scott talked about Python from space. I should mention here that I'm kind of like an Enpassant colleague of hers. We work at the same company, but not at the same time. But she was talking about,

working with data from from the satellite. So her company has, is is called Planet, and they have hundreds of satellites up in orbit, and they get pictures of pretty much everywhere in the world. I think they're aiming for daily pictures of every spot on Earth, and she was showing off, like, in this nonstop live demo style talk what you can do with that. That was really great. And let's see what what else. There were a couple of talks about Unicode. I just would encourage everyone to to look into Unicode. There was a talk about there's a talk by by this fellow, Lucas

Langer, who has a, you know, like, a non ASCII character in his name. He gives

very funny talks about Unicode because, you you know, he's kinda like affected.

Another talk was about comparing and sorting Unicode, which is surprisingly non obvious and kind of mind bending. And then I I probably won't go into details on those, but there was a series of talks about Python and science, and I just I was really fascinated by that. I think, both of the keynotes by Jake Vander Plas and and,

I think her name is Katie Hoff, were were scientists,

and and just talk about

how how software skills are super applicable to science and kind of encourage a community to maybe help out with these open source science projects

because you don't need to be a scientist to write software for science. I thought that was,

great. And then I I guess I'm not I I'm I'm always super behind on my podcast, so I completely forgot that we can have nontechnical,

like non Python things as PIX. So I,

I definitely wanna mention a newsletter. There's a newsletter called The Prepared by, Spencer Wright. And every week, he sends out links related

to manufacturing,

logistics, sometimes transport, and cities,

which as you found out by now, I'm super interested in. And if you're interested in that, you should totally subscribe to that newsletter because I I usually click every single link in there. So I I

really strongly recommend that newsletter. And finally, something completely, like, out there nontechnical.

I recently did an Amtrak ride from Chicago to the Bay Area,

And, you know, I'm not from this country, so I always laugh about, like, the rail infrastructure,

the passenger rail infrastructure in the US, but that was just mind blowing, seeing all these all these landscapes

and just seeing seeing parts of the country you would never see. So, next year,

PyCon is in,

now I forgot. Do you remember what the city is called? At Cleveland. They have an Amtrak station. So maybe when you plan your trip, consider

taking the train there because

I think if you haven't done this before, especially

the the the trains that go, you know, across the continental divide, that's something I think everyone should have seen this. So do it. Well, I really appreciate you taking the time out of your day to share your interest and passion for factory automation and industrial automation. It's definitely a really interesting area and 1 that most people don't necessarily get to get a, you know, inside view of. So I appreciate you taking the time to put the talk together at PyCon and, take the time tonight to, you know, dig deeper into that. So I hope you enjoy the rest of your evening. Yeah. Thanks. And and thank you for having me.

The Python Podcast.init

Summary

Preface

Interview

Keep In Touch

Picks

Links

The Python Podcast.__init__