Summary
The importance of testing your software is widely talked about and well understood. What is not as often discussed is the different types of testing, and how end-to-end tests can benefit your team to ensure proper functioning of your application when it gets released to production. This week Luciano Renzi shares the work that he has done on Golem, a framework for building and executing an automation suite to exercise the entire system from the perspective of the user. He discusses his reasons for creating the project, how he things about testing, and where he plans on taking Golem in the future. Give it a listen and then take it for a test drive.
Preface
- Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
- I would like to thank everyone who supports us on Patreon. Your contributions help to make the show sustainable.
- When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at podastinit.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your awesome app. And now you can deliver your work to your users even faster with the newly upgraded 200 GBit network in all of their datacenters.
- If you’re tired of cobbling together your deployment pipeline then it’s time to try out GoCD, the open source continuous delivery platform built by the people at ThoughtWorks who wrote the book about it. With GoCD you get complete visibility into the life-cycle of your software from one location. To download it now go to podcatinit.com/gocd. Professional support and enterprise plugins are available for added piece of mind.
- Visit the site to subscribe to the show, sign up for the newsletter, and read the show notes. And if you have any questions, comments, or suggestions I would love to hear them. You can reach me on Twitter at @Podcast__init__ or email hosts@podcastinit.com)
- To help other people find the show please leave a review on iTunes, or Google Play Music, tell your friends and co-workers, and share it on social media.
- Your host as usual is Tobias Macey and today I’m interviewing Luciano Renzi about Golem, a framework and automation tool for end-to-end testing in Python
Interview
- Introductions
- How did you get introduced to Python?
- What is golem and what motivated you to create it?
- What was your inspiration for the name?
- Why did you choose to use Python for Golem and if you were to start over today would you make the same choice?
- For someone who is unfamiliar with the concept, can you describe what end-to-end testing is and the reasons for making it part of their development process?
- What is the main goal of Golem
- What does the internal architecture and implementation of Golem look like and how has that evolved from when you first started the project?
- How does Golem compare to other Python libraries for automated browser testing and what was lacking in the existing solutions when you created it?
- What are the differences between golem and robot framework?
- What about projects written in other languages such as protractor?
- One of the intriguing features of Golem is the web interface for constructing tests. What are the benefits of codeless automation & record-playback functionality?
- What are some of the most challenging aspects of building and maintaining Golem?
- It seems that every browser automation library is ultimately a wrapper around Selenium. Why is a wrapper necessary and why haven’t any strong alternatives been created?
- What are the advantages of making Golem a framework for test automation, rather than a library?
- What are some of the most interesting or unexpected uses for Golem that you have seen?
- What do you have planned for the future of Golem?
- What is the current state of end to end automation and how do you see it evolving in the future?
- How do you think machine learning and AI will be used in test automation?
Keep In Touch
- luciano-renzi on GitHub
- @lucianorenzi_ on Twitter
Picks
- Tobias
Links
- Golem
- Elementum
- Pascal
- Watir
- JUnit
- Selenium
- Page Object Pattern
- Selenium Grid
- Sauce Labs
- py.test
- Robot Framework
- Mechanize
- Acceptance Tests
- Protractor
- Webdriver.io
- Appium
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
[00:00:14]
Unknown:
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. I would like to thank everyone who supports the show on Patreon. Your contributions help to make the show sustainable. When you're ready to launch your next project, you'll need somewhere to deploy it, so you should check out linode@podcastinit.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your app. And And now you can deliver your work to your users even faster with the newly upgraded 200 gigabit network in all of their data centers. If you're tired of cobbling together your deployment pipeline, then it's time to try out GoCD, the open source continuous delivery platform built by the people at Thoughtworks who wrote the book about it. With GoCD, you get complete visibility into the life cycle of your software from 1 location. To download it now, go to podcastinit.com/gocd.
Professional support and enterprise plug ins are available for added peace of mind. You can visit the site at podcastinnit.com to subscribe to the show, sign up for the newsletter, and read the show notes. And if you have any questions, comments, or suggestions, I would love to hear them. You can reach me on Twitter at podcast in it or email me at host at podcast in it.com. To help other people find the show, please leave a review on iTunes or Google Play Music. Tell your friends and coworkers and share it on social media. Your host as usual is Tobias Macy. And today, I'm interviewing Luciano Puccio about Golem, a framework and automation tool for end to end testing in Python.
So, Luciano, could you please introduce yourself?
[00:01:34] Unknown:
Oh, yeah. Sure. Hello. My name is Luciano Puccio. I am a software engineer in test. I live in Buenos Aires, Argentina. I'm currently working in Elementum, which is a start up. 1 of the leaders and innovators in the supply chain space. I have been working in software development for around 10 years. And I'm also, of course, the creator of Golem, which is a test automation tool built in Python. And do you remember how you first got introduced to Python? It was quite a low, long road until I discovered Python. I remember that when I was at the university, they used Pascal to teach algorithms.
I think that's that was a terrible choice at the time. It was a long time ago, and and I really hope that they are using something different. I actually heard that they are using Python now to to cheat to teach algorithms and to introduce people to program. After that, I got a job where they were using c sharp. I didn't know anything about c sharp, so I had to learn it from 0. They were using an automation tool that was embedded into Visual Studio. It's called Codely. So I I I had to learn c sharp. Then, the time I I learned by myself, Watir, which is an automation tool written in Ruby.
So, I really liked Ruby at the time, and the Watir API seemed a little bit nicer than the Selenium. At the time Selenium was Selenium RC. The Selenium 2 web driver was not out yet. So I was like, I I preferred, Wateer instead of Selenium, which is now quite the opposite. Let me see. Afterwards, I had to learn Java for another job. So I became proficient enough in Java to do my job there. But at the same time, I got, curious with web development. And by pure luck, I chose, Django as the framework that I used to learn some web development.
So what I did was that, I was kind of following the the tutorial of Django, which is quite nice, and I was learning Python on the go just enough to to know how to implement what I wanted to do with Django. And now I think that in retrospective, that's not the best way to go. You should first start with the language and grab the basics. But then, you should try to apply this knowledge to a real project. And when you are speaking about Python, it is a nice option to to implement the knowledge that you are getting with web development, and, Chango is a great framework.
Although, for beginners, I I I will choose a lighter version, like, a a lighter option, like Flask, for example. Another option to start with Python is to try to, automate tests, of course. After all these road, I kind of started playing with the Python bindings. And at the time, I was comparing the Python bindings of Selenium to the bindings of Java. So I think I started to fall in love with Python because 1 of the main things I like about the Python language is that it kind of forces you to use good practices from the start, like, just because you have to write the indented code, but but it goes a long more way than that.
And I see that every Python developer I have met is is very concerned about good practices. There are a lot of coding standards that are really, really implemented throughout, all the Python libraries that or or or all the Python code that you come along. So I think it it adds a lot a lot of value, and I often find that after a a long time coding in Python, then maybe you go back to another language and and you find yourself trying to, implement these good practices, like code indentation, although it's not necessary. But to me, that's that's a great asset that the language has that the language adds to the programming practice.
The other thing is that you can achieve a lot of things with the standard library of Python. Maybe, programmers from another language will be surprised of how many things you can achieve without installing anything else. And if there's something that you cannot achieve with the with the standard library, there is always an external library that is objectively the best, needs well defined well implemented and with good documentation. So I haven't had, really any situation where I couldn't use Python, and I think that's that's a very important thing.
To give you an example, a few a few days ago, I needed a script to gather some metrics from Shida and Sonar, and I chose Python to do this. Then there was, already built Jira and Sonar, libraries, you know, to interact with the with the APIs of these, applications. And, you know, be, writing this this script was a a risk to me. It was quite quite easily, and I could implement it, really fast.
[00:07:32] Unknown:
And it sounds like you've had a fair bit of experience working with Selenium and other tools for doing web automation. So I'm wondering if you can briefly describe what Gollum is and what was the motivation to create it when it sounds like you had already gotten familiar with some of the patterns and tooling that existed in Python and elsewhere for being able to do similar work?
[00:07:51] Unknown:
Yeah. Yeah. For sure. I started Golium, but, before before that, I had a quite a quite a lot of experience with different, with the different, tools available. So I think that it all began a few years ago when I I was working for a software company that just a few QAs were, doing front end automation with a framework, they built themselves in Java, and it was using ShayUnit. But there were also more than 50 testers that did only manual tests and did not have any experience with test automation or computer programming or Selenium or Java.
So 1 of the tasks I had in this company was to train all these manual testers to start using the automation tool. And to me it was a daunting task because mainly because you are teaching Java to someone who might have absolutely 0 background in computer programming. And after all these training sessions, although I have a few successes that I I achieved that some of the testers could start, participating in the automation efforts. I think it in general, it was a failure. And I thought for I thought for a long time about this problem, and my conclusion was that this whole business of trying to introduce manual testers to test automation could could be done in really differently.
So I thought there has to be a better way. I mean, it should not be so difficult. So I decided that I will try to build my own framework from scratch so I could implement everything I have learned so far and without any constraints or restrictions in any way. And, of course, I was doing it, mostly for for myself. So Golem has, some super useful features like, the web module that helps you build and run tests and suites, and you can also write the test directly in Python code if you want. It has a complete web reports module that can display the results of a running suite in real time, a huge list of predefined actions that cover a large array of use cases.
With Goletm, you can implement page object pattern, of course, and you can write your own custom actions as, normal Python functions. So Golang can run test in parallel. This is very important to me. It can run locally or remotely with a Selenium grid or in a remote in a remote device provider like Sauce Labs. It can be executed from a command line or from any, continuous integration tool like Jenkins. It has an interactive console. So, you can stop a test at any event time and fiddle with the application in real time. So it helps you sometimes understand why a test might not work, which is, some it's not possible in other languages, especially languages that are compiled.
Then, with Golem, you can implement data driven and keyword driven tests. It is possible to implement a certain test driven development as well. And, of course, there are a lot of improvements that are still on queue to be added. That's why I'm listening carefully to the feedback I'm getting from the current users. 1 of the main aims of Golem is to have a complete test automation tool that's easy to learn even if you don't have much experience with Selenium or computer programming. But at the same time, it should create robust tests and foster collaboration between the team members.
[00:12:05] Unknown:
So I was just wondering if you can give a bit of the backstory for why you chose the name Gollum for the framework.
[00:12:12] Unknown:
So, I frankly don't recall what I what I called it golem. Although in retrospective, I think I think it's quite an original name and, now what I see, it's it's, it's precious to have something that that is original and not not already taken. And since, I don't know, most most automation frameworks are are named, robot something, I think it's original, but at the same time it's kindly of align with the the rest of the tools. So a golem is, according to mythology, an automaton moss monster built with clay that you could give instructions to by sticking a a paper note in its mouth. So the monster will carry these instructions.
So in a sense, you could say that a golem is a mythological robot, you know, kind of. So it's appropriate, I think.
[00:13:10] Unknown:
Yeah. It definitely is very fitting in terms of the naming as it relates to the purpose of the framework because, as you said, a golem is an automaton that does your bidding, and you're using the test framework as a means doing your bidding to be able to repeatedly execute the acceptance and end to end tests for your project. So as you said, it's a very well chosen and original name.
[00:13:31] Unknown:
Yeah. Absolutely.
[00:13:33] Unknown:
Why did you choose to use Python as the language for implementing Gollum? And if you were to start over today, do you think you would make the same choice?
[00:13:42] Unknown:
So I chose Python deliberately, not not just because I'm in love with it, which I am. Because 1 of the goals of Golem is to help newcomers get started in test automation. And I think that by far, Python is the most appropriate programming language for this purpose and for many reasons. It's eloquent and readable. It's, closer to natural language or to cell the code at least. It doesn't need compiling. You don't have to learn any other tool like Maven in the case of Java. You you are not tied to a specific IDE either. It comes, preinstalled in Linux and macOS.
Although macOS comes with, Python 2.7, it can be easily upgraded. Also, it's important to know that Python, according to recent statistics, is the 2nd most popular language in GitHub, and it is growing fast. The first 1 is JavaScript, of course, but I think that if there were more programming languages available in the browser, the statistics will look different. And I have a dream that 1 day the browsers will come with a Python interpreter built in. And finally, if if I were to start Golem again, I I would choose Python, without a doubt.
[00:15:10] Unknown:
For somebody who is unfamiliar with the concept or the technicalities of end to end testing, I'm just wondering if you can take some time to describe what it is and why somebody should consider making it part of their development process if it isn't already.
[00:15:27] Unknown:
Yeah. Sure. So, first of all, I see that there there is a misconception about what end to end test means. I think it has become a buzzwords, a buzzword and it would be wouldn't be the first time. So I'm looking at the cloud for example. So first of all, imagine that you have tests that assert functions, methods, and classes. You can call this unit test, of course. Then you have tests that assert the interaction between different classes and modules, probably between different repositories. And you can call this integration tests. Above this, you have tests that use real data and real databases in a production like environment.
And the system under test is actually connected to any external systems or services. You can call this end to end tests. You can say that graphical user interface tests are end to end tests, but not always. And also, if you are testing from the graphical user interface, but the back end data is mocked, then you are not actually doing end to end tests. So just a clarification that end to end doesn't actually mean graphical user interface. So we call them you can create acceptance tests for the graphical user interface, which can at the same time be end to end tests.
I have planned for the future of Golem to test APIs as well and security and performance too. It's an idea that I'm fiddling with to have different kind of tests but in the same tool. Regarding end to end tests and acceptance tests, it is becoming mandatory in development instead of just a nice thing to have, especially in Azure development. Why is this? It is because now you have web and mobile applications that should work in many browsers, in many devices, and operating systems. And to test this manually, it's just not possible. We have to keep in mind that unit and integration tests have a higher priority over end to end test and acceptance test because the latter are much more expensive to write and maintain.
However, if even if you have more than 90% of unit test coverage, you will still have 0 certainty that after you build and you deploy the application, it will work as expected or let alone know if it works at all. So I think it is still required to have graphical user interface tests. That's a very famous pyramid, of testing priorities. In this pyramid, you have unit test at the base. Above this, there are integration test, then end to end test, and finally at the very top, graphical user interface test. So I I think this is correct, and it should be always kept in kept in mind.
[00:18:45] Unknown:
And you mentioned briefly when you were discussing how you first created Gollum, but I'm wondering if you can just talk a bit more about what your main goals and priorities were when you were working on implementing it.
[00:18:59] Unknown:
Yeah. My my main motivation when I started writing Golem was to provide the best tool and with the bet with the least amount of learning curve as possible. It was targeted mainly to manual testers that don't have much experience with test automation or computer programming. The reason to devote so much time perfecting the tool is because in software development projects, quality is often the first thing that is neglected when the clock starts ticking. So when you succeed in convincing the program manager or the tech lead or who it may concern to give you some priority and some resources to quality, you achieve the intention of testing. The next the next step is to take advantage of this intention to actually improve the quality of the project.
In this step, it is crucial that the tool you use is polished enough so it does not become an an obstacle in itself. Because as soon as this happened, the intention of testing might be lost, probably forever. So this can be because the tool that you choose is hard to learn, or it takes a week to have it installed and working, or because the documentation is incomplete, or debugging the test proves to be impossible. So the core the core idea of Golem is that it proposes from the beginning a structure and a set of useful features, and it comes ready to use.
But at the same time, it is highly extendable, so it has sensible defaults. The idea is that if you know how to extend golem, you can do it. But if you don't, then golem by default should work in 99% of the use cases.
[00:20:57] Unknown:
Yeah. I can definitely agree with you on the topic that if you don't get started early with end to end testing in a project, then it is all too easy to just forego it entirely because after you get to a certain point in the life cycle of the project and you're getting ready to launch it, and that's the time that you start thinking about those tests. Then by the time you get there, then there's just so much surface area to cover that it's hard to know where to start. And so it's easy to just say, okay. Well, it's going to take too much additional investment to make it worthwhile, so we'll just skip it entirely. I've been on some projects where that was the case where we get to the point where it wasn't until there were specific failures that we needed to be able to guard against in the user interface of the application that we actually took the time and effort to implement those end to end tests, whereas if they had been in place earlier, it would have been a much lower maintenance burden to continue having a decent amount of coverage of that experience.
[00:21:57] Unknown:
Yeah. Yeah. For sure. There is a current trend to shift left to try to move the the testing process, not only at the same time of the development process, but even, before. Before the development starts, you should already start, thinking about testing.
[00:22:15] Unknown:
When I was looking through the documentation and the examples for Gollum, it looks like the design of a code and the implementation is either using pytest or heavily influenced by it. So I'm just wondering if you can talk about what the internal architecture and the implementation of Golem looks like and how that has evolved over the time that you've been building and maintaining it.
[00:22:40] Unknown:
Yeah. So I think that, Golem tries tries to take, to grab the best of current testing frameworks like anything from the CheUnit family, and also Pytest, which is a great tool in itself in Python. So So yeah, a main focus of Golem is to have the ability to run the test in parallel, which is something that is it's truly important when you start to expand your test suite. You will find out that if you run your test in serial mode, it will take it can take hours. So that's that's absolutely the opposite of what you want if you want to achieve continuous testing, for example.
So if I have to to speak about the the architecture of of Golem, in itself, I think that, there are some things that can be improved because column was something like, it it it has been evolving along my my knowledge of test automation and Python. So there are for sure things that that that can be improved. So it it it's actually quite something quite common that, when you go back to a code that you wrote, like, 2 years ago and and you say, like, what what the heck was I thinking about. Right? So but but, yeah, it's it's a it's an ongoing work for sure.
[00:24:09] Unknown:
There are a number of projects both in Python and other languages that serve as wrappers for a Selenium to make it easier to write tests that are more idiomatic to the language that you're using. I'm curious how Gollum compares to the other Python libraries that are available for doing automated browser testing, whether that's with or without Selenium. And what did you find lacking in the existing solutions when you were creating it?
[00:24:38] Unknown:
Yeah. So if if you are talking just for the Python language, the other most popular library is of course Robot Framework, which has a large community and has been going on for for quite a few years. In Python itself, there are there are many other options. What I think is that automation frameworks in general, when the mostly when they are open source, like, you find that you have to start, compiling different pieces together. So I think that 1 of the goals of Golemy is that you don't have to if if you if you don't know how to do that. If you don't if you don't have the the expertise to actually assert and research different tools or and how can they be compiled together.
So Golium is kind of a complete tool to to to achieve the goal, that you that you want to achieve. Of course, there are there are a lot of of tools that are, appearing, if I am correct, when I say that Python is is the best, option for for for the goal that Golem has, then it's probable that I I can grow Golem to be this go to solution for test automation. That that's kind of the goal.
[00:26:03] Unknown:
You called out the robot framework as the main competitor in terms of scope and capabilities to Gollum. So can you do a bit of a compare and contrast between the work that you've done with Gollum and what's available in Robot Framework?
[00:26:19] Unknown:
Yeah. There are some key differences. Robot Framework, is a tool more geared towards, acceptance test driven development, and it's also very heavy towards keyword driven testing. But the main difference I think does is that, in Robot Framework, all the tests have to be written in test files in a tabular format, then you can extend them by writing libraries in Python or Java. The main difference I see is that Golem, first of all, uses Python directly to write tests, and this test can be parsed, displayed, and edited in the test builder which is a web application. This test builder web module has the main goal of easing in newcomers into the tool and the practice of test automation.
Another goal of this web module is to use it as a collaboration tool because you can have it installed in your local machine, but you can also deploy it to a local network or to the Internet itself and allow allow team members to access it using their own credentials in order to add or edit the tests without having to install anything on their own machines. This can be really useful when dealing with, nontechnical testers or any other member of the team who normally would not engage in testing activities. And I was there to say that the learning curve of robot framework is a little bit steeper than the 1 of Gollum.
[00:28:02] Unknown:
And what about projects that are written in other languages such as protractor, which is 1 of the popular options for JavaScript?
[00:28:11] Unknown:
Yeah. Protractor is quite popular in JavaScript as well as WebDriver IO. I think that in the case of the use of JavaScript, we have to say that it is a very popular language these days and it should not be underestimated. However, I find that the asynchronous nature of JavaScript can sometimes complicate things. It often leads to code that is hard to read and maintain and it can be difficult to debug. This is because without a good grasp of how JavaScript works internally, you can easily lose control of the flow of the tests. Mainly, for me, from the perspective of someone who is just starting to use Selenium, and it's and it's new to computer programming, this can be an intimidating experience.
In general, for test automation, there is no actual need to have a synchronous code. If you need your test scripts to be asynchronous, that's probably a red flag. So from what I have seen with JavaScript, it's a constant battle to sync circumvent a feature of the language, the asynchronous behavior that 99% of the time is not actually needed. So I I know you can use the new async away syntax in JavaScript to avoid the 4 the famous callback held, But it looks like, to me, it looks like forcing the language to do something it's not actually the same to do. Need can add a lot of boilerplate, and it looks just ugly, you know, that but that's just my opinion of course.
As as of Protractor itself, it is a framework built on top of Selenium with some helper functions for Angular apps. The thing is that I see no real need to use a specialized tool to test Angular apps because you can just as well use a pure Selenium binding, and you can save yourself from the overhead and the extra moving parts that Protractor adds. For the case of JavaScript, the pure, Selenium binding is called WebDriver JS. To me, it's a bad idea to use a framework specialized for a particular development framework. Why is that? It's because when you're dealing with web applications, you deal with HTML, and you you don't actually interact with the code of the app itself. So it there is a separation, and the framework that was used to develop the application is, irrelevant when trying to automate it. The test automation framework should be as agnostic as possible because eventually, you might need the same framework to automate another application or a module of the application that is built with a different development framework or stack, or you might need to test mobile or desktop applications.
And because you selected a framework that was specialized for a development framework, you will need probably a lot of refactoring and tweaking, or maybe you you will need to throw the tests away and start again from 0. So I will always try to build the framework as as not as agnostic as possible. And it is also nice to consider that in web development, you cannot predict which framework will be popular in a year or 2. It's a it's a very dynamic development ecosystem. If if you compare Golem to Protractor, besides all the extra features Golem has, because I consider Golem like a complete tool more than just a framework. You can, of course, use Golem to test 1 page applications or any other kind of dynamic applications as well.
[00:32:17] Unknown:
You mentioned earlier the web UI for Gollum for being able to generate tests from that interface rather than having to necessarily write the code for it. And I'm curious what the record and playback functionality looks like and, I guess, what the gradations are between that in the more form driven test generation and then compared to the actual written tests.
[00:32:42] Unknown:
Yeah. So my thought process when I designed the web interface was that back then when I was trying junior testers in test automation and in Java, it was difficult to convey that, in order to write this automated test, you you only needed a small subset of the capabilities of the language. You have to start from 0. You have to teach what is a class, what is an object, a variable function, etcetera, and it takes a lot of effort. So I thought, what if you strip everything that's just noise away, and you only end up with a bare minimum of things required to learn?
And the result is a subset of the programming language capabilities that is simple enough to put behind a graphical user interface. And you can add, some cool features like autocomplete, some forms, and inputs, etcetera. This, once you once you get the handle of these simple capabilities, you can discover that behind all of this, it's just code. And you can dig deeper if you want at your own pace. In the end, you can choose to use, the graphical user interface or write code directly or both. Maybe, use the web interface for, quick edits of the tests or for collaboration among the team, or you can just, write, the code directly in your in your preferred IDE or test editor.
That's the 2 options that you have. In the case of codeless automation, that's, a topic that is popping up frequently these days. I think that it is mostly a marketing gimmick, and there is no actual codeless automation. Such tools that, promise codeless automation might impose too much restrictions for the tests to actually be codeless. And the tests end up being pretty much useless for anything that strays away from the automation version of the hello world example, like, navigating to a URL and clicking some buttons. It is practically impossible to find an application that's just so simple that you can use a tool like this. For the concept of record and playback, it hasn't died yet, unfortunately.
But to me, record and playback does not work in the long run. I have seen entire suites of tests written with with record and playback that had to be thrown away completely because, eventually they become obsolete. So back to Golem, I say that Golem would never have record and playback features. And in a sense, you could say that it has some concepts of codeless automation because you can use the test the test builder to write a huge array of different test scenarios and actions without touching a single line of code. But can you cover all scenarios just by using the test builder? I think you cannot.
That's why you you can also write an edit test using an IDE, by writing, code directly. So it's important that people should not forget that test automation is computer programming, and the test automation tool is a software development project in itself. And I would recommend, any team to have at least 1 person that is expert expert in this field to be around and to to provide, good counsel on on this kind of things.
[00:36:42] Unknown:
And what have been some of the most challenging aspects of building and maintaining Gollum?
[00:36:47] Unknown:
Yeah. First of all, Gollum is still in beta. Although it can already be used to test web applications, and there are some people already using it, providing feedback, which I really value. And also it has it has revealed, good price so far, but it is important to remember that it is still in beta. But it means, yeah, it means there are still some pieces that are falling into place. For example, it still does not support test for mobile applications, which is my priority number 1. And the API test support is still very poor. However, I think this is a thrilling moment to get involved in the project because the discussion is still open.
Now is the time to participate. I'm proposing improvements to the tool so we can achieve this huge goal that I that I am proposing, and become a a a great tool for testers and made by testers. 1 of the challenges of developing a tool like Golem is keeping it unbiased about what features should be part of the of the framework and what parts should be implemented by the users themselves. A framework that satisfies every possible scenario, could not exist. But what we can aim is to have a good enough tool that covers the most common scenarios and lets you easily extend it to better suit your own project.
[00:38:24] Unknown:
When talking about browser automation, it seems that virtually every project that I've come across that does that is actually a wrapper around the Selenium project. And I'm just wondering, from your perspective, why is a wrapper for Selenium necessary in the first place? And why do you think there haven't been any strong alternatives to Selenium that have come along and served to sort of broaden the range of options for that kind of application?
[00:38:54] Unknown:
Yeah. It's important to know that WebDriver is becoming a w 3 c standard probably this year or the next. And it is currently already implemented widely across browser vendors. So it it it makes no sense to think of a different tool for for driving a browser. If you are talking about visual testing, then there there are, other tools for sure. But for driving the browser or a mobile or desktop application, Selenium driver and Appium are the de facto choices. Selenium, has a huge, enormous community, and it is very active and thriving. For the wrappers, Selenium is just an implementation of the WebDriver API.
So in a sense, it's pretty low level and it should be. Recently, Simon Stewart, which is 1 1 of the creators of Selenium and the creator of the web driver itself, said that he encourages quality engineers to build automation frameworks around selenium, writing our own wrappers to extend it. So I think that my job as the creator of golem is to create a wrapper of selenium that can satisfy as many projects and use cases as possible while still being eloquent and easy to learn. So it's it's quite a challenge, I think.
[00:40:26] Unknown:
And you've talked a bit about the idea of frameworks versus libraries versus the full application capabilities of some of the different projects. And I'm curious what your thoughts are on the advantages of making Gollum a complete framework as opposed to just a library implementation that will be used and driven by a different application?
[00:40:50] Unknown:
So, I have modeled Gollum after chango. In chango, what you first do is you use the chango admin to create a project, which is a directory with some files inside. Then inside this project, you create applications. In Django, an application is just a modular part of the larger application. In this sense, Golem has a a similar workflow because you create a test directory with the golem admin. Then inside this test directory, you define some global configurations and settings and different kind of, stuff like the the the WebDriver executables, etcetera.
And then inside this this test directory, you add projects. So in the end, you have this test directory that you can easily put into a git repo, for example. And there's a clear separation between framework code, like the the golem code itself and your test code. So this differentiation is is very important because even though there are many ways to extend it, golem itself should be a package that you can install with the Python package manager, I need to have a consistent behavior by default, everywhere.
[00:42:13] Unknown:
I know that the primary use case for Gollum is for being able to build and run tests in an automated fashion, But I'm curious if there are any particularly interesting or unexpected uses of Gollum that you have seen, whether in terms of interesting ways of building and running the tests or using it purely as a way of automating browser interactions, but not necessarily with the end goal of producing test output.
[00:42:42] Unknown:
Yeah. 1 1 of the most interesting things, that I have seen in Golem is that since it is an automation tool and at the same time it's a it's a web application, it it can test itself. So which is, really meta if you think about it. So right now there is a regression suite that I'm working on expanding, but, it actually tests the golden web module by itself. So you can say that it's an automation tool that kind of tests itself because it has also a graphical user interface. So it's kind of, recursive.
[00:43:26] Unknown:
Yeah. That's definitely amusing to think about and from the pure automation aspects too. There are some applications that I've been faced with where I would like for there to be an API available so that I can just issue an API call and either create data on the platform or retrieve information from it, but the only interface that's available to me is through the web UI. So looking at tools like Gollum, I've been considering using that as my way of being able to have a Python interface to this particular web application to be able to perform these actions in an automated fashion as opposed to having to go through and click and type my way through interactively?
[00:44:07] Unknown:
Yeah. Yeah. For sure. For sure. The the the applications of of a tool like this are are quite quite a lot. It actually right now, the the support for API test is is quite poor, but you can use it it has embedded the request requests library of Python. So you can make, like, HTTP get or or post requests. But in the future, it it should be, like, more robust to to create, requests with different, headers or or params and then kind of asserting different values from the response. So that that's that's another another interesting application is to kind of blur the differentiation between graphical user interface and API and other kind of tests. And you can maybe have a test that does, maybe a call to an API or a call to a database, and then access something from the graphical user interface. It's it's kind of blurring the line between the different kind of tests.
[00:45:12] Unknown:
What are some of the things that you have in mind for the future of Gollum whether in terms of enhancements or new features?
[00:45:20] Unknown:
So quite a few. In the future, I think Golem should be able to automate test for mobile and desktop applications, and this can be achieved using Appium, which is a sister project of Selenium. And it also it it should also seamlessly test APIs. The idea behind this is that to have a single tool to test different kind of things. To me, this this will reduce the cognitive load of everyone in the team as opposed to the trend that I'm seeing that there are a huge amount of different tools that have each 1 has a a single purpose. So column should be like a a unifying tool for different kind of tests.
Other things it should be able to test are load, performance, and security as well. It could also have potential to run unit and integration tests as well. This is because, it it it will it will use the the same report module, which I'm really proud of. But surely, there is still a a long way to to go, but, I think we are on track. Another interesting thing is it should have integration with issue trackers like Cheetah. That that's possible but, quite challenging because, you're talking about a tool or a framework that knows when it had when it has found the bug, and it can check if the bug is already reported. And then if it's not reported, raise a new a new bug in the issue tracker.
If it's already reported, it should kind of, leave a comment or an update. So that is possible but quite challenging, and it will be something else awesome to have.
[00:47:12] Unknown:
As we close out, I'm curious if you can talk a bit more about the current state of end to end automation capabilities and where you see the evolution of the space moving to in the future, particularly with the advances in the fields of machine learning and artificial intelligence?
[00:47:32] Unknown:
So I see that there's a strong push for end to end automation, but I see that we are lacking in is, enough people with the required skills to do so. I think that the the theory of this automation is is well spread out yet, but, also the tools need some polishing to to achieve, the the best state possible. That's 1 of the issues Golem is trying to solve, like kind of a a bridge from where we are now to where we are headed. There's also a trend to integrate test automation with DevOps and to have a a working continuous integration pipeline and achieve continuous delivery and continuous testing.
And there is a trend to fade the lines that separate the rows inside the development team. So there should not be a strict separation between developers and testers. Everyone in the team should be involved in the quality as quality is something that that's built in and everyone's responsible. And we should forget about the idea that quality is something that you can, verify at the end or you can patch at the end by some manual testing. But it has to be incorporated into the workflow from the beginning. And it is, of course, a collaborative effort, And that's the idea of what I mentioned earlier of shift left.
To achieve this state, the I think the tools chosen are 1 of the most important factors because in order to enforce the adoption of these testing practices, for all team members. It is primordial to have a tool that's easy to use by everyone, but is still capable of generating robust tests, and the tool itself should not be an obstacle in order to achieve the quality goals that the project has.
[00:49:38] Unknown:
And are there any other topics that you think we should cover before we start to close out the show?
[00:49:45] Unknown:
Yeah. There is a a futuristic topic you could say that is the the use of AI, machine learning for testing. I see that it's becoming quite popular, and there there are there are sure some applications in the testing space of AI and machine learning, and there are people already working on on it. I think that 1 of the things that can be improved by AI is the writing of the tests themselves or improving their record and playback support. Other viable options are auto generated tests or the use of predictive analytics. However, I think that there will always be room for a real quality analyst inside the development team because it is usually the 1 with the strong quality point of view, and and it is truly required to have.
But from the quality analysis to the quality control, there are some steps of separation. And, of course, the the the latter can be, improved by AI, for sure, in in the future. For this, I hope that Golem can be used as a stepping stone for innovation in test automation. But what I'm sure is that the the software quality roles within software development are evolving and will have huge changes in the near future. And this, promises to be a thrilling, experience, and I am really glad to be part of it.
[00:51:20] Unknown:
Alright. Is there anything else that we should talk about before we close out the show?
[00:51:26] Unknown:
Not that I can think of. I invite everyone to to test Golem to see how it works. And, of course, the feedback is is really welcomed but because at at this stage it's kind of green. It's still not out to out of the oven. So so yeah. So to to test it and to let me know what what what you think.
[00:51:49] Unknown:
And for anybody who wants to get in touch with you to follow the work that you're up to whether on Gollum or other things, I'll have you add your preferred contact information to the show notes. And so with that, I'll move us to the picks. And my pick today is 1 that I think has been mentioned before on the show, but I just listened to the audiobook of Weapons of Math Destruction, just talking about the ways that big data and machine learning can be used and are being used to potentially discriminate from groups of people largely just because of the underlying data and discussing some of the biases that exist in the different datasets because of the collection methods or the ways that they're being analyzed.
So it's just a interesting and well considered and well written book talking about some of the issues that are facing us in terms of how these different algorithms are being given control over more and more different parts of our lives. So it's definitely 1 that I recommend checking out. And, Luciano, do you have any PIXPRESS this week?
[00:53:04] Unknown:
Not actually.
[00:53:05] Unknown:
Well, I appreciate you taking the time out of your day to join me and talk to me about the work that you've done on Gollum. It is definitely an interesting project and 1 that I'll be making sure to introduce to my colleagues at work so that we can see if we can improve the overall coverage of our acceptance tests. And I hope you enjoy the rest of your evening.
[00:53:30] Unknown:
Okay. Okay. Thanks a lot for having me as a guest, and it has been a very good experience.
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. I would like to thank everyone who supports the show on Patreon. Your contributions help to make the show sustainable. When you're ready to launch your next project, you'll need somewhere to deploy it, so you should check out linode@podcastinit.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your app. And And now you can deliver your work to your users even faster with the newly upgraded 200 gigabit network in all of their data centers. If you're tired of cobbling together your deployment pipeline, then it's time to try out GoCD, the open source continuous delivery platform built by the people at Thoughtworks who wrote the book about it. With GoCD, you get complete visibility into the life cycle of your software from 1 location. To download it now, go to podcastinit.com/gocd.
Professional support and enterprise plug ins are available for added peace of mind. You can visit the site at podcastinnit.com to subscribe to the show, sign up for the newsletter, and read the show notes. And if you have any questions, comments, or suggestions, I would love to hear them. You can reach me on Twitter at podcast in it or email me at host at podcast in it.com. To help other people find the show, please leave a review on iTunes or Google Play Music. Tell your friends and coworkers and share it on social media. Your host as usual is Tobias Macy. And today, I'm interviewing Luciano Puccio about Golem, a framework and automation tool for end to end testing in Python.
So, Luciano, could you please introduce yourself?
[00:01:34] Unknown:
Oh, yeah. Sure. Hello. My name is Luciano Puccio. I am a software engineer in test. I live in Buenos Aires, Argentina. I'm currently working in Elementum, which is a start up. 1 of the leaders and innovators in the supply chain space. I have been working in software development for around 10 years. And I'm also, of course, the creator of Golem, which is a test automation tool built in Python. And do you remember how you first got introduced to Python? It was quite a low, long road until I discovered Python. I remember that when I was at the university, they used Pascal to teach algorithms.
I think that's that was a terrible choice at the time. It was a long time ago, and and I really hope that they are using something different. I actually heard that they are using Python now to to cheat to teach algorithms and to introduce people to program. After that, I got a job where they were using c sharp. I didn't know anything about c sharp, so I had to learn it from 0. They were using an automation tool that was embedded into Visual Studio. It's called Codely. So I I I had to learn c sharp. Then, the time I I learned by myself, Watir, which is an automation tool written in Ruby.
So, I really liked Ruby at the time, and the Watir API seemed a little bit nicer than the Selenium. At the time Selenium was Selenium RC. The Selenium 2 web driver was not out yet. So I was like, I I preferred, Wateer instead of Selenium, which is now quite the opposite. Let me see. Afterwards, I had to learn Java for another job. So I became proficient enough in Java to do my job there. But at the same time, I got, curious with web development. And by pure luck, I chose, Django as the framework that I used to learn some web development.
So what I did was that, I was kind of following the the tutorial of Django, which is quite nice, and I was learning Python on the go just enough to to know how to implement what I wanted to do with Django. And now I think that in retrospective, that's not the best way to go. You should first start with the language and grab the basics. But then, you should try to apply this knowledge to a real project. And when you are speaking about Python, it is a nice option to to implement the knowledge that you are getting with web development, and, Chango is a great framework.
Although, for beginners, I I I will choose a lighter version, like, a a lighter option, like Flask, for example. Another option to start with Python is to try to, automate tests, of course. After all these road, I kind of started playing with the Python bindings. And at the time, I was comparing the Python bindings of Selenium to the bindings of Java. So I think I started to fall in love with Python because 1 of the main things I like about the Python language is that it kind of forces you to use good practices from the start, like, just because you have to write the indented code, but but it goes a long more way than that.
And I see that every Python developer I have met is is very concerned about good practices. There are a lot of coding standards that are really, really implemented throughout, all the Python libraries that or or or all the Python code that you come along. So I think it it adds a lot a lot of value, and I often find that after a a long time coding in Python, then maybe you go back to another language and and you find yourself trying to, implement these good practices, like code indentation, although it's not necessary. But to me, that's that's a great asset that the language has that the language adds to the programming practice.
The other thing is that you can achieve a lot of things with the standard library of Python. Maybe, programmers from another language will be surprised of how many things you can achieve without installing anything else. And if there's something that you cannot achieve with the with the standard library, there is always an external library that is objectively the best, needs well defined well implemented and with good documentation. So I haven't had, really any situation where I couldn't use Python, and I think that's that's a very important thing.
To give you an example, a few a few days ago, I needed a script to gather some metrics from Shida and Sonar, and I chose Python to do this. Then there was, already built Jira and Sonar, libraries, you know, to interact with the with the APIs of these, applications. And, you know, be, writing this this script was a a risk to me. It was quite quite easily, and I could implement it, really fast.
[00:07:32] Unknown:
And it sounds like you've had a fair bit of experience working with Selenium and other tools for doing web automation. So I'm wondering if you can briefly describe what Gollum is and what was the motivation to create it when it sounds like you had already gotten familiar with some of the patterns and tooling that existed in Python and elsewhere for being able to do similar work?
[00:07:51] Unknown:
Yeah. Yeah. For sure. I started Golium, but, before before that, I had a quite a quite a lot of experience with different, with the different, tools available. So I think that it all began a few years ago when I I was working for a software company that just a few QAs were, doing front end automation with a framework, they built themselves in Java, and it was using ShayUnit. But there were also more than 50 testers that did only manual tests and did not have any experience with test automation or computer programming or Selenium or Java.
So 1 of the tasks I had in this company was to train all these manual testers to start using the automation tool. And to me it was a daunting task because mainly because you are teaching Java to someone who might have absolutely 0 background in computer programming. And after all these training sessions, although I have a few successes that I I achieved that some of the testers could start, participating in the automation efforts. I think it in general, it was a failure. And I thought for I thought for a long time about this problem, and my conclusion was that this whole business of trying to introduce manual testers to test automation could could be done in really differently.
So I thought there has to be a better way. I mean, it should not be so difficult. So I decided that I will try to build my own framework from scratch so I could implement everything I have learned so far and without any constraints or restrictions in any way. And, of course, I was doing it, mostly for for myself. So Golem has, some super useful features like, the web module that helps you build and run tests and suites, and you can also write the test directly in Python code if you want. It has a complete web reports module that can display the results of a running suite in real time, a huge list of predefined actions that cover a large array of use cases.
With Goletm, you can implement page object pattern, of course, and you can write your own custom actions as, normal Python functions. So Golang can run test in parallel. This is very important to me. It can run locally or remotely with a Selenium grid or in a remote in a remote device provider like Sauce Labs. It can be executed from a command line or from any, continuous integration tool like Jenkins. It has an interactive console. So, you can stop a test at any event time and fiddle with the application in real time. So it helps you sometimes understand why a test might not work, which is, some it's not possible in other languages, especially languages that are compiled.
Then, with Golem, you can implement data driven and keyword driven tests. It is possible to implement a certain test driven development as well. And, of course, there are a lot of improvements that are still on queue to be added. That's why I'm listening carefully to the feedback I'm getting from the current users. 1 of the main aims of Golem is to have a complete test automation tool that's easy to learn even if you don't have much experience with Selenium or computer programming. But at the same time, it should create robust tests and foster collaboration between the team members.
[00:12:05] Unknown:
So I was just wondering if you can give a bit of the backstory for why you chose the name Gollum for the framework.
[00:12:12] Unknown:
So, I frankly don't recall what I what I called it golem. Although in retrospective, I think I think it's quite an original name and, now what I see, it's it's, it's precious to have something that that is original and not not already taken. And since, I don't know, most most automation frameworks are are named, robot something, I think it's original, but at the same time it's kindly of align with the the rest of the tools. So a golem is, according to mythology, an automaton moss monster built with clay that you could give instructions to by sticking a a paper note in its mouth. So the monster will carry these instructions.
So in a sense, you could say that a golem is a mythological robot, you know, kind of. So it's appropriate, I think.
[00:13:10] Unknown:
Yeah. It definitely is very fitting in terms of the naming as it relates to the purpose of the framework because, as you said, a golem is an automaton that does your bidding, and you're using the test framework as a means doing your bidding to be able to repeatedly execute the acceptance and end to end tests for your project. So as you said, it's a very well chosen and original name.
[00:13:31] Unknown:
Yeah. Absolutely.
[00:13:33] Unknown:
Why did you choose to use Python as the language for implementing Gollum? And if you were to start over today, do you think you would make the same choice?
[00:13:42] Unknown:
So I chose Python deliberately, not not just because I'm in love with it, which I am. Because 1 of the goals of Golem is to help newcomers get started in test automation. And I think that by far, Python is the most appropriate programming language for this purpose and for many reasons. It's eloquent and readable. It's, closer to natural language or to cell the code at least. It doesn't need compiling. You don't have to learn any other tool like Maven in the case of Java. You you are not tied to a specific IDE either. It comes, preinstalled in Linux and macOS.
Although macOS comes with, Python 2.7, it can be easily upgraded. Also, it's important to know that Python, according to recent statistics, is the 2nd most popular language in GitHub, and it is growing fast. The first 1 is JavaScript, of course, but I think that if there were more programming languages available in the browser, the statistics will look different. And I have a dream that 1 day the browsers will come with a Python interpreter built in. And finally, if if I were to start Golem again, I I would choose Python, without a doubt.
[00:15:10] Unknown:
For somebody who is unfamiliar with the concept or the technicalities of end to end testing, I'm just wondering if you can take some time to describe what it is and why somebody should consider making it part of their development process if it isn't already.
[00:15:27] Unknown:
Yeah. Sure. So, first of all, I see that there there is a misconception about what end to end test means. I think it has become a buzzwords, a buzzword and it would be wouldn't be the first time. So I'm looking at the cloud for example. So first of all, imagine that you have tests that assert functions, methods, and classes. You can call this unit test, of course. Then you have tests that assert the interaction between different classes and modules, probably between different repositories. And you can call this integration tests. Above this, you have tests that use real data and real databases in a production like environment.
And the system under test is actually connected to any external systems or services. You can call this end to end tests. You can say that graphical user interface tests are end to end tests, but not always. And also, if you are testing from the graphical user interface, but the back end data is mocked, then you are not actually doing end to end tests. So just a clarification that end to end doesn't actually mean graphical user interface. So we call them you can create acceptance tests for the graphical user interface, which can at the same time be end to end tests.
I have planned for the future of Golem to test APIs as well and security and performance too. It's an idea that I'm fiddling with to have different kind of tests but in the same tool. Regarding end to end tests and acceptance tests, it is becoming mandatory in development instead of just a nice thing to have, especially in Azure development. Why is this? It is because now you have web and mobile applications that should work in many browsers, in many devices, and operating systems. And to test this manually, it's just not possible. We have to keep in mind that unit and integration tests have a higher priority over end to end test and acceptance test because the latter are much more expensive to write and maintain.
However, if even if you have more than 90% of unit test coverage, you will still have 0 certainty that after you build and you deploy the application, it will work as expected or let alone know if it works at all. So I think it is still required to have graphical user interface tests. That's a very famous pyramid, of testing priorities. In this pyramid, you have unit test at the base. Above this, there are integration test, then end to end test, and finally at the very top, graphical user interface test. So I I think this is correct, and it should be always kept in kept in mind.
[00:18:45] Unknown:
And you mentioned briefly when you were discussing how you first created Gollum, but I'm wondering if you can just talk a bit more about what your main goals and priorities were when you were working on implementing it.
[00:18:59] Unknown:
Yeah. My my main motivation when I started writing Golem was to provide the best tool and with the bet with the least amount of learning curve as possible. It was targeted mainly to manual testers that don't have much experience with test automation or computer programming. The reason to devote so much time perfecting the tool is because in software development projects, quality is often the first thing that is neglected when the clock starts ticking. So when you succeed in convincing the program manager or the tech lead or who it may concern to give you some priority and some resources to quality, you achieve the intention of testing. The next the next step is to take advantage of this intention to actually improve the quality of the project.
In this step, it is crucial that the tool you use is polished enough so it does not become an an obstacle in itself. Because as soon as this happened, the intention of testing might be lost, probably forever. So this can be because the tool that you choose is hard to learn, or it takes a week to have it installed and working, or because the documentation is incomplete, or debugging the test proves to be impossible. So the core the core idea of Golem is that it proposes from the beginning a structure and a set of useful features, and it comes ready to use.
But at the same time, it is highly extendable, so it has sensible defaults. The idea is that if you know how to extend golem, you can do it. But if you don't, then golem by default should work in 99% of the use cases.
[00:20:57] Unknown:
Yeah. I can definitely agree with you on the topic that if you don't get started early with end to end testing in a project, then it is all too easy to just forego it entirely because after you get to a certain point in the life cycle of the project and you're getting ready to launch it, and that's the time that you start thinking about those tests. Then by the time you get there, then there's just so much surface area to cover that it's hard to know where to start. And so it's easy to just say, okay. Well, it's going to take too much additional investment to make it worthwhile, so we'll just skip it entirely. I've been on some projects where that was the case where we get to the point where it wasn't until there were specific failures that we needed to be able to guard against in the user interface of the application that we actually took the time and effort to implement those end to end tests, whereas if they had been in place earlier, it would have been a much lower maintenance burden to continue having a decent amount of coverage of that experience.
[00:21:57] Unknown:
Yeah. Yeah. For sure. There is a current trend to shift left to try to move the the testing process, not only at the same time of the development process, but even, before. Before the development starts, you should already start, thinking about testing.
[00:22:15] Unknown:
When I was looking through the documentation and the examples for Gollum, it looks like the design of a code and the implementation is either using pytest or heavily influenced by it. So I'm just wondering if you can talk about what the internal architecture and the implementation of Golem looks like and how that has evolved over the time that you've been building and maintaining it.
[00:22:40] Unknown:
Yeah. So I think that, Golem tries tries to take, to grab the best of current testing frameworks like anything from the CheUnit family, and also Pytest, which is a great tool in itself in Python. So So yeah, a main focus of Golem is to have the ability to run the test in parallel, which is something that is it's truly important when you start to expand your test suite. You will find out that if you run your test in serial mode, it will take it can take hours. So that's that's absolutely the opposite of what you want if you want to achieve continuous testing, for example.
So if I have to to speak about the the architecture of of Golem, in itself, I think that, there are some things that can be improved because column was something like, it it it has been evolving along my my knowledge of test automation and Python. So there are for sure things that that that can be improved. So it it it's actually quite something quite common that, when you go back to a code that you wrote, like, 2 years ago and and you say, like, what what the heck was I thinking about. Right? So but but, yeah, it's it's a it's an ongoing work for sure.
[00:24:09] Unknown:
There are a number of projects both in Python and other languages that serve as wrappers for a Selenium to make it easier to write tests that are more idiomatic to the language that you're using. I'm curious how Gollum compares to the other Python libraries that are available for doing automated browser testing, whether that's with or without Selenium. And what did you find lacking in the existing solutions when you were creating it?
[00:24:38] Unknown:
Yeah. So if if you are talking just for the Python language, the other most popular library is of course Robot Framework, which has a large community and has been going on for for quite a few years. In Python itself, there are there are many other options. What I think is that automation frameworks in general, when the mostly when they are open source, like, you find that you have to start, compiling different pieces together. So I think that 1 of the goals of Golemy is that you don't have to if if you if you don't know how to do that. If you don't if you don't have the the expertise to actually assert and research different tools or and how can they be compiled together.
So Golium is kind of a complete tool to to to achieve the goal, that you that you want to achieve. Of course, there are there are a lot of of tools that are, appearing, if I am correct, when I say that Python is is the best, option for for for the goal that Golem has, then it's probable that I I can grow Golem to be this go to solution for test automation. That that's kind of the goal.
[00:26:03] Unknown:
You called out the robot framework as the main competitor in terms of scope and capabilities to Gollum. So can you do a bit of a compare and contrast between the work that you've done with Gollum and what's available in Robot Framework?
[00:26:19] Unknown:
Yeah. There are some key differences. Robot Framework, is a tool more geared towards, acceptance test driven development, and it's also very heavy towards keyword driven testing. But the main difference I think does is that, in Robot Framework, all the tests have to be written in test files in a tabular format, then you can extend them by writing libraries in Python or Java. The main difference I see is that Golem, first of all, uses Python directly to write tests, and this test can be parsed, displayed, and edited in the test builder which is a web application. This test builder web module has the main goal of easing in newcomers into the tool and the practice of test automation.
Another goal of this web module is to use it as a collaboration tool because you can have it installed in your local machine, but you can also deploy it to a local network or to the Internet itself and allow allow team members to access it using their own credentials in order to add or edit the tests without having to install anything on their own machines. This can be really useful when dealing with, nontechnical testers or any other member of the team who normally would not engage in testing activities. And I was there to say that the learning curve of robot framework is a little bit steeper than the 1 of Gollum.
[00:28:02] Unknown:
And what about projects that are written in other languages such as protractor, which is 1 of the popular options for JavaScript?
[00:28:11] Unknown:
Yeah. Protractor is quite popular in JavaScript as well as WebDriver IO. I think that in the case of the use of JavaScript, we have to say that it is a very popular language these days and it should not be underestimated. However, I find that the asynchronous nature of JavaScript can sometimes complicate things. It often leads to code that is hard to read and maintain and it can be difficult to debug. This is because without a good grasp of how JavaScript works internally, you can easily lose control of the flow of the tests. Mainly, for me, from the perspective of someone who is just starting to use Selenium, and it's and it's new to computer programming, this can be an intimidating experience.
In general, for test automation, there is no actual need to have a synchronous code. If you need your test scripts to be asynchronous, that's probably a red flag. So from what I have seen with JavaScript, it's a constant battle to sync circumvent a feature of the language, the asynchronous behavior that 99% of the time is not actually needed. So I I know you can use the new async away syntax in JavaScript to avoid the 4 the famous callback held, But it looks like, to me, it looks like forcing the language to do something it's not actually the same to do. Need can add a lot of boilerplate, and it looks just ugly, you know, that but that's just my opinion of course.
As as of Protractor itself, it is a framework built on top of Selenium with some helper functions for Angular apps. The thing is that I see no real need to use a specialized tool to test Angular apps because you can just as well use a pure Selenium binding, and you can save yourself from the overhead and the extra moving parts that Protractor adds. For the case of JavaScript, the pure, Selenium binding is called WebDriver JS. To me, it's a bad idea to use a framework specialized for a particular development framework. Why is that? It's because when you're dealing with web applications, you deal with HTML, and you you don't actually interact with the code of the app itself. So it there is a separation, and the framework that was used to develop the application is, irrelevant when trying to automate it. The test automation framework should be as agnostic as possible because eventually, you might need the same framework to automate another application or a module of the application that is built with a different development framework or stack, or you might need to test mobile or desktop applications.
And because you selected a framework that was specialized for a development framework, you will need probably a lot of refactoring and tweaking, or maybe you you will need to throw the tests away and start again from 0. So I will always try to build the framework as as not as agnostic as possible. And it is also nice to consider that in web development, you cannot predict which framework will be popular in a year or 2. It's a it's a very dynamic development ecosystem. If if you compare Golem to Protractor, besides all the extra features Golem has, because I consider Golem like a complete tool more than just a framework. You can, of course, use Golem to test 1 page applications or any other kind of dynamic applications as well.
[00:32:17] Unknown:
You mentioned earlier the web UI for Gollum for being able to generate tests from that interface rather than having to necessarily write the code for it. And I'm curious what the record and playback functionality looks like and, I guess, what the gradations are between that in the more form driven test generation and then compared to the actual written tests.
[00:32:42] Unknown:
Yeah. So my thought process when I designed the web interface was that back then when I was trying junior testers in test automation and in Java, it was difficult to convey that, in order to write this automated test, you you only needed a small subset of the capabilities of the language. You have to start from 0. You have to teach what is a class, what is an object, a variable function, etcetera, and it takes a lot of effort. So I thought, what if you strip everything that's just noise away, and you only end up with a bare minimum of things required to learn?
And the result is a subset of the programming language capabilities that is simple enough to put behind a graphical user interface. And you can add, some cool features like autocomplete, some forms, and inputs, etcetera. This, once you once you get the handle of these simple capabilities, you can discover that behind all of this, it's just code. And you can dig deeper if you want at your own pace. In the end, you can choose to use, the graphical user interface or write code directly or both. Maybe, use the web interface for, quick edits of the tests or for collaboration among the team, or you can just, write, the code directly in your in your preferred IDE or test editor.
That's the 2 options that you have. In the case of codeless automation, that's, a topic that is popping up frequently these days. I think that it is mostly a marketing gimmick, and there is no actual codeless automation. Such tools that, promise codeless automation might impose too much restrictions for the tests to actually be codeless. And the tests end up being pretty much useless for anything that strays away from the automation version of the hello world example, like, navigating to a URL and clicking some buttons. It is practically impossible to find an application that's just so simple that you can use a tool like this. For the concept of record and playback, it hasn't died yet, unfortunately.
But to me, record and playback does not work in the long run. I have seen entire suites of tests written with with record and playback that had to be thrown away completely because, eventually they become obsolete. So back to Golem, I say that Golem would never have record and playback features. And in a sense, you could say that it has some concepts of codeless automation because you can use the test the test builder to write a huge array of different test scenarios and actions without touching a single line of code. But can you cover all scenarios just by using the test builder? I think you cannot.
That's why you you can also write an edit test using an IDE, by writing, code directly. So it's important that people should not forget that test automation is computer programming, and the test automation tool is a software development project in itself. And I would recommend, any team to have at least 1 person that is expert expert in this field to be around and to to provide, good counsel on on this kind of things.
[00:36:42] Unknown:
And what have been some of the most challenging aspects of building and maintaining Gollum?
[00:36:47] Unknown:
Yeah. First of all, Gollum is still in beta. Although it can already be used to test web applications, and there are some people already using it, providing feedback, which I really value. And also it has it has revealed, good price so far, but it is important to remember that it is still in beta. But it means, yeah, it means there are still some pieces that are falling into place. For example, it still does not support test for mobile applications, which is my priority number 1. And the API test support is still very poor. However, I think this is a thrilling moment to get involved in the project because the discussion is still open.
Now is the time to participate. I'm proposing improvements to the tool so we can achieve this huge goal that I that I am proposing, and become a a a great tool for testers and made by testers. 1 of the challenges of developing a tool like Golem is keeping it unbiased about what features should be part of the of the framework and what parts should be implemented by the users themselves. A framework that satisfies every possible scenario, could not exist. But what we can aim is to have a good enough tool that covers the most common scenarios and lets you easily extend it to better suit your own project.
[00:38:24] Unknown:
When talking about browser automation, it seems that virtually every project that I've come across that does that is actually a wrapper around the Selenium project. And I'm just wondering, from your perspective, why is a wrapper for Selenium necessary in the first place? And why do you think there haven't been any strong alternatives to Selenium that have come along and served to sort of broaden the range of options for that kind of application?
[00:38:54] Unknown:
Yeah. It's important to know that WebDriver is becoming a w 3 c standard probably this year or the next. And it is currently already implemented widely across browser vendors. So it it it makes no sense to think of a different tool for for driving a browser. If you are talking about visual testing, then there there are, other tools for sure. But for driving the browser or a mobile or desktop application, Selenium driver and Appium are the de facto choices. Selenium, has a huge, enormous community, and it is very active and thriving. For the wrappers, Selenium is just an implementation of the WebDriver API.
So in a sense, it's pretty low level and it should be. Recently, Simon Stewart, which is 1 1 of the creators of Selenium and the creator of the web driver itself, said that he encourages quality engineers to build automation frameworks around selenium, writing our own wrappers to extend it. So I think that my job as the creator of golem is to create a wrapper of selenium that can satisfy as many projects and use cases as possible while still being eloquent and easy to learn. So it's it's quite a challenge, I think.
[00:40:26] Unknown:
And you've talked a bit about the idea of frameworks versus libraries versus the full application capabilities of some of the different projects. And I'm curious what your thoughts are on the advantages of making Gollum a complete framework as opposed to just a library implementation that will be used and driven by a different application?
[00:40:50] Unknown:
So, I have modeled Gollum after chango. In chango, what you first do is you use the chango admin to create a project, which is a directory with some files inside. Then inside this project, you create applications. In Django, an application is just a modular part of the larger application. In this sense, Golem has a a similar workflow because you create a test directory with the golem admin. Then inside this test directory, you define some global configurations and settings and different kind of, stuff like the the the WebDriver executables, etcetera.
And then inside this this test directory, you add projects. So in the end, you have this test directory that you can easily put into a git repo, for example. And there's a clear separation between framework code, like the the golem code itself and your test code. So this differentiation is is very important because even though there are many ways to extend it, golem itself should be a package that you can install with the Python package manager, I need to have a consistent behavior by default, everywhere.
[00:42:13] Unknown:
I know that the primary use case for Gollum is for being able to build and run tests in an automated fashion, But I'm curious if there are any particularly interesting or unexpected uses of Gollum that you have seen, whether in terms of interesting ways of building and running the tests or using it purely as a way of automating browser interactions, but not necessarily with the end goal of producing test output.
[00:42:42] Unknown:
Yeah. 1 1 of the most interesting things, that I have seen in Golem is that since it is an automation tool and at the same time it's a it's a web application, it it can test itself. So which is, really meta if you think about it. So right now there is a regression suite that I'm working on expanding, but, it actually tests the golden web module by itself. So you can say that it's an automation tool that kind of tests itself because it has also a graphical user interface. So it's kind of, recursive.
[00:43:26] Unknown:
Yeah. That's definitely amusing to think about and from the pure automation aspects too. There are some applications that I've been faced with where I would like for there to be an API available so that I can just issue an API call and either create data on the platform or retrieve information from it, but the only interface that's available to me is through the web UI. So looking at tools like Gollum, I've been considering using that as my way of being able to have a Python interface to this particular web application to be able to perform these actions in an automated fashion as opposed to having to go through and click and type my way through interactively?
[00:44:07] Unknown:
Yeah. Yeah. For sure. For sure. The the the applications of of a tool like this are are quite quite a lot. It actually right now, the the support for API test is is quite poor, but you can use it it has embedded the request requests library of Python. So you can make, like, HTTP get or or post requests. But in the future, it it should be, like, more robust to to create, requests with different, headers or or params and then kind of asserting different values from the response. So that that's that's another another interesting application is to kind of blur the differentiation between graphical user interface and API and other kind of tests. And you can maybe have a test that does, maybe a call to an API or a call to a database, and then access something from the graphical user interface. It's it's kind of blurring the line between the different kind of tests.
[00:45:12] Unknown:
What are some of the things that you have in mind for the future of Gollum whether in terms of enhancements or new features?
[00:45:20] Unknown:
So quite a few. In the future, I think Golem should be able to automate test for mobile and desktop applications, and this can be achieved using Appium, which is a sister project of Selenium. And it also it it should also seamlessly test APIs. The idea behind this is that to have a single tool to test different kind of things. To me, this this will reduce the cognitive load of everyone in the team as opposed to the trend that I'm seeing that there are a huge amount of different tools that have each 1 has a a single purpose. So column should be like a a unifying tool for different kind of tests.
Other things it should be able to test are load, performance, and security as well. It could also have potential to run unit and integration tests as well. This is because, it it it will it will use the the same report module, which I'm really proud of. But surely, there is still a a long way to to go, but, I think we are on track. Another interesting thing is it should have integration with issue trackers like Cheetah. That that's possible but, quite challenging because, you're talking about a tool or a framework that knows when it had when it has found the bug, and it can check if the bug is already reported. And then if it's not reported, raise a new a new bug in the issue tracker.
If it's already reported, it should kind of, leave a comment or an update. So that is possible but quite challenging, and it will be something else awesome to have.
[00:47:12] Unknown:
As we close out, I'm curious if you can talk a bit more about the current state of end to end automation capabilities and where you see the evolution of the space moving to in the future, particularly with the advances in the fields of machine learning and artificial intelligence?
[00:47:32] Unknown:
So I see that there's a strong push for end to end automation, but I see that we are lacking in is, enough people with the required skills to do so. I think that the the theory of this automation is is well spread out yet, but, also the tools need some polishing to to achieve, the the best state possible. That's 1 of the issues Golem is trying to solve, like kind of a a bridge from where we are now to where we are headed. There's also a trend to integrate test automation with DevOps and to have a a working continuous integration pipeline and achieve continuous delivery and continuous testing.
And there is a trend to fade the lines that separate the rows inside the development team. So there should not be a strict separation between developers and testers. Everyone in the team should be involved in the quality as quality is something that that's built in and everyone's responsible. And we should forget about the idea that quality is something that you can, verify at the end or you can patch at the end by some manual testing. But it has to be incorporated into the workflow from the beginning. And it is, of course, a collaborative effort, And that's the idea of what I mentioned earlier of shift left.
To achieve this state, the I think the tools chosen are 1 of the most important factors because in order to enforce the adoption of these testing practices, for all team members. It is primordial to have a tool that's easy to use by everyone, but is still capable of generating robust tests, and the tool itself should not be an obstacle in order to achieve the quality goals that the project has.
[00:49:38] Unknown:
And are there any other topics that you think we should cover before we start to close out the show?
[00:49:45] Unknown:
Yeah. There is a a futuristic topic you could say that is the the use of AI, machine learning for testing. I see that it's becoming quite popular, and there there are there are sure some applications in the testing space of AI and machine learning, and there are people already working on on it. I think that 1 of the things that can be improved by AI is the writing of the tests themselves or improving their record and playback support. Other viable options are auto generated tests or the use of predictive analytics. However, I think that there will always be room for a real quality analyst inside the development team because it is usually the 1 with the strong quality point of view, and and it is truly required to have.
But from the quality analysis to the quality control, there are some steps of separation. And, of course, the the the latter can be, improved by AI, for sure, in in the future. For this, I hope that Golem can be used as a stepping stone for innovation in test automation. But what I'm sure is that the the software quality roles within software development are evolving and will have huge changes in the near future. And this, promises to be a thrilling, experience, and I am really glad to be part of it.
[00:51:20] Unknown:
Alright. Is there anything else that we should talk about before we close out the show?
[00:51:26] Unknown:
Not that I can think of. I invite everyone to to test Golem to see how it works. And, of course, the feedback is is really welcomed but because at at this stage it's kind of green. It's still not out to out of the oven. So so yeah. So to to test it and to let me know what what what you think.
[00:51:49] Unknown:
And for anybody who wants to get in touch with you to follow the work that you're up to whether on Gollum or other things, I'll have you add your preferred contact information to the show notes. And so with that, I'll move us to the picks. And my pick today is 1 that I think has been mentioned before on the show, but I just listened to the audiobook of Weapons of Math Destruction, just talking about the ways that big data and machine learning can be used and are being used to potentially discriminate from groups of people largely just because of the underlying data and discussing some of the biases that exist in the different datasets because of the collection methods or the ways that they're being analyzed.
So it's just a interesting and well considered and well written book talking about some of the issues that are facing us in terms of how these different algorithms are being given control over more and more different parts of our lives. So it's definitely 1 that I recommend checking out. And, Luciano, do you have any PIXPRESS this week?
[00:53:04] Unknown:
Not actually.
[00:53:05] Unknown:
Well, I appreciate you taking the time out of your day to join me and talk to me about the work that you've done on Gollum. It is definitely an interesting project and 1 that I'll be making sure to introduce to my colleagues at work so that we can see if we can improve the overall coverage of our acceptance tests. And I hope you enjoy the rest of your evening.
[00:53:30] Unknown:
Okay. Okay. Thanks a lot for having me as a guest, and it has been a very good experience.
Introduction to Luciano Puccio and Golem
Luciano's Journey to Python
Motivation Behind Creating Golem
Origin of the Name 'Golem'
Choosing Python for Golem
Understanding End-to-End Testing
Goals and Priorities in Developing Golem
Challenges of Late Testing Implementation
Golem's Architecture and Influence from Pytest
Comparison with Other Automation Tools
Comparing Golem with Robot Framework
JavaScript Automation Tools: Protractor and WebDriver IO
Golem's Web UI and Test Generation
Challenges in Building and Maintaining Golem
Selenium and the Need for Wrappers
Framework vs. Library: Golem's Approach
Unexpected Uses of Golem
Future Enhancements for Golem
Future of End-to-End Automation
AI and Machine Learning in Testing
Closing Remarks
