Visit our site to listen to past episodes, support the show, join our community, and sign up for our mailing list.
Summary
Any programmer who has dealt with a website for any length of time knows that writing JavaScript isn’t always the most enjoyable. Wouldn’t you rather write that code in Python and just have it work on your website? In this episode we learn about Transcrypt with its creator Jacques de Hooge. Transcrypt is a Python to JavaScript transpiler that embraces the JavaScript ecosystem while letting you use the familiar syntax of Python for writing your logic, rather than trying to shoehorn a Python runtime into your browser.
Brief Introduction
- Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
- Subscribe on iTunes, Stitcher, TuneIn or RSS
- Follow us on Twitter or Google+
- Give us feedback! Leave a review on iTunes, Tweet to us, send us an email or leave us a message on Google+
- Join our community! Visit discourse.pythonpodcast.com for your opportunity to find out about upcoming guests, suggest questions, and propose show ideas.
- I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. For details on how to support the show you can visit our site at pythonpodcast.com
- Linode is sponsoring us this week. Check them out at linode.com/podcastinit and get a $20 credit to try out their fast and reliable Linux virtual servers for your next project
- I would also like to thank Hired, a job marketplace for developers and designers, for sponsoring this episode of Podcast.__init__. Use the link hired.com/podcastinit to double your signing bonus.
- ODSC East in Boston is happening on May 21st – 22nd. Use the discount code EP for 20% off when you register
- Your host today is Tobias Macey
- Today I am interviewing Jacques de Hooge about his work on the Transcrypt Project
Interview with Jacques de Hooge
- Introductions
- How did you get introduced to Python? – Tobias
- What is Transcrypt and what inspired you to create it? – Tobias
- As you mention in the documentation, there are a number of projects that attempt to shoehorn Python into the browser. What makes Transcrypt different? – Tobias
- I like that you decided to embrace the web environment by calling into JavaScript libraries. What are some of the challenges that you encountered while creating that functionality? – Tobias
- How is the transpilation performed and what are some of the methods that you used to get the build size as small as it is? – Tobias
- Given the nature of JavaScripts prototypical inheritance and differences in class semantics, I imagine that adding support for multiple inheritance and reflecting the structure of Python classes must have been challenging. Can you describe that process and how you arrived at your current solution? – Tobias
- Which aspects of the language were most difficult to translate to JavaScript? – Tobias
- Is Transcrypt complete and stable enough to be used in production? – Tobias
Keep in Touch
Picks
- Tobias
- Jacques
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
[00:00:14]
Unknown:
Hello, and welcome to podcast.net, the podcast about Python and the people who make it great. You can subscribe to our show on Itunes, Stitcher, TuneIn Radio, or add our RSS feed to your pod catcher of choice. You can also follow us on Twitter or Google plus, and please give us feedback. You can leave a review on Itunes to help other people find the show, send us a tweet or an email, leave us a message on Google Plus or our show notes, and you can also join our discourse forum at discourse.pythonpodcast.com for your opportunity to find out about upcoming guests, suggest questions, and propose show ideas.
I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. For details on how to support the show, you can visit our site at pythonpodcast.com. Linode is sponsoring us this week. Check them out at linode.com/podcastonit and get a $20 credit to try out their fast and reliable Linux virtual servers for your next project. I would also like to thank Hired, the job marketplace for developers and designers for sponsoring this episode of podcast.init. Use the link hired.com/podcastinit to double your signing bonus.
Coming up in Boston on May 21st 22nd is the open data science conference. So if you are in the area and you're interested in learning about how Python can be used in the areas of data science, then you should check it out. You can use the discount code e p for 20% off your ticket. That's odsc.com. Today, we are interviewing Jacques de Hoog about his work on the transcript project. Jacques, could you introduce yourself, please?
[00:01:44] Unknown:
Yes. Okay. I will. Well, I'm a software developer ever since I was 15. My father worked with computers, shortly after the 2nd world war, and I've been programming all my life. That's part of my job. I make technical computer programs, so, real time control systems and also some cardiology software for for medical imaging. And I give lectures at the Rotterdam Academy, part time, about 30% of my time. And the rest of my time, I spent making, open source software.
[00:02:20] Unknown:
And how did you get introduced to Python?
[00:02:23] Unknown:
Oh, it's a funny story. I was a quality manager with Fugro, that's a worldwide engineering firm. And there was this guy, William Chester, from Great Britain. He was a very intelligent guy, and he I I was making a a user interface of c plus plus with scripting possibilities. And he came to me and he said, Jacques, you're implementing Python in c plus plus. And I said, Python, what's that? I said, well, take a look. And I I took a look, and it turned out to be a runtime type language, scripting language, and I said, I was quality manager. So I said, no way that we will will use this language in this company because it will become a mess if we don't have strict typing. So I was very much opposed to introducing, Python, but still, I tried it a little bit and a little bit more and a little bit more. And, well, it didn't took long before I was amazed and very much convinced of its power because, it's true that sometime I miss a strict typing, but it's so expressive and so brief, and the data structures are so powerful that, well, it has saved me a lot of time. So that was my introduce introduction to Python.
[00:03:28] Unknown:
And, you mentioned that you like to do some open source work, which brings us to the transcript project. And I'm wondering if you can describe, what transcript is and what inspired you to create it.
[00:03:41] Unknown:
Yes. Well, it is a a Python to JavaScript compiler. It's not new. There are several of them. And it it allows you to run a large syntactically and semantically faithful subset of Python 3.5 in your browser. Well, there are more products that do that, but I think the special thing is that while it has implemented a large part of Python 3.5, the code that is produced, the JavaScript code is very lean and very readable and very debuggable. So that is what it is actually. And I've programmed and also using Django, and I always longed to be able to use Python on the client side as well. And I had an upcoming project where a lot of client programming had to be done, so I looked into what was there already. And, well, there are many many transpilers, Python to JavaScript compilers.
There are, for instance, PyPy GS, which is really generating c code and is running on top of axm.gs, which is, well, very powerful but also a bulky download, 1 side of the spectrum, and then you've got, you've got, RapidScript on the other side of the spectrum, which is a thin layer around JavaScript and misses some features that I wanted in. Some people will disagree. I I like multiple inheritance, for instance. I know that Java programmers don't care. James Gosling doesn't care. I do care about multiple inheritance. I'm I'm from the c plus plus world. And, well, I started out using a RapidScript, and I liked it, but when I was using it, I missed some things, especially the multiple inheritance, bound function pointer assignments, things like that.
And well, I decided that there was a room for another product and, all my life, I liked making interpreters, compilers, stuff like that. I have made a modular 2 interpreter in the old days. So I thought this may be fun, and I came across the AST module of Python. And what I wanted to do is to start out from the Python end of things. I mean, if I take, for instance, RapidScript, it starts out from the JavaScript side, using Node. Js and things like that. I wanted this to be a Python thing. I wanted to this to be distributed via PyPI. I wanted a pure Python syntax, so using the Python parser, stuff like that. So I set out and it was a kind well, it wasn't very serious. I put on the website, don't use this because it may go away any minute. But when I was programming it, actually, when I made a start, it turned out to be easier than I thought and, well, there were some hard parts, but I started to like it and it became more and more serious and I was amazed what you could implement in a still generating very compact source code. So I decided that was serious after all and I I reserved an URL a URL for its transcript.org.
And, well, it became a real project in that way.
[00:06:45] Unknown:
And, where did you get the name from?
[00:06:48] Unknown:
Well, to transcribe, you know, that's what monks did. They couldn't make a photocopy, so they they took, 1 of those big codexes and they wrote it down down by hand. I've tried to program multiple inheritance directly in JavaScript, but I don't like it. So how I see it is that the transcription is made, as a kind of compilation from transcript to JavaScript. So that's where the name comes from.
[00:07:12] Unknown:
So I like that you decided to embrace the web environment by calling it JavaScript libraries. What are some of the challenges that you encountered while creating that functionality?
[00:07:21] Unknown:
Python is famous for its libraries. Yeah. It's it's better easy to include it, thing. But I realized that, for the web, you cannot simply copy or reimplement the the desktop libraries of Python. The demands made are very different. For the web, good looks are important. Look to all this, jQuery, gooey stuff, and many things have been made. There's a very large body of high quality libraries for JavaScript, actually. I I don't know exactly how people succeed in writing such good libraries in such a language, but, okay, they're there. It's an illusion that you can bypass that and and re redo that in Python. So I felt from the beginning, if this is to be taken serious, it has to to gel really well with things like jQuery and, well, all the all the other stuff that's there, the beautiful stuff that's there for the web. So, well, what I encountered there were a few problems. 1 of the problems that the the JavaScript community doesn't doesn't particularly care about, it seems, are namespace pollution and name clashes. Well, there are there are solutions emerging now, but, I mean, there's still no decent module system, for instance. And, I I hate that. I hate it when I make a program and some new library comes around and have to change my variable names, for instance.
So, I try to come up with an encapsulation mechanism so that you can use a JavaScript library that normally has its stuff in global name space. You can encapsulate it so that if viewed from Python, it is in the module name space. That was that was 1 of the problems. Another problem was in the data structures, there are broadly 2 ways to solve the problem of, having Python data structures alongside JavaScript data structures. The the most dogmatic solution is that you you make separate data structures in Python. I mean, that means that a Python dictionary is something quite different from a JavaScript object, and a Python list will be something totally different from a JavaScript array.
But if you do that, you will have to convert and you'll have to write a lot of code, which all has to be downloaded to the client to run it. On the other hand, you have the people who just say, well, an array is almost a list, so we add some methods. And then, we we add some Python methods, and then an array will be a list. But that's not without problems because, for instance, an array has a sort method sort method in JavaScript, and a list has also a sort method in Python, but those 2 are different. The parameters are interpreted different. So what I did is, come up with a concept of pragmas, which are compiler directives, and 1 of the pragmas is the aliens pragma.
And when I say sort in Python, this is translated to a py_sort in JavaScript. So I can enrich the native Java JavaScript types with extra methods without causing any name clashes and without causing any code bloat or delay. That was an important, well, discovery that you could do that. And then there was the new statement. In in in Python, you don't have to say new, you just call the constructor and and things are instantiated. Now what you can do is encapsulate every JavaScript new in every library, and then you don't have to call new anymore. It is possible in transcript, you can do that. But since there are so many libraries and are growing and are changing, I don't think that's generally a good idea. So I had to have some kind of new statements for the for the JavaScript stuff, and I wanted to be syntactically, confirmed the Python syntax, not the JavaScript syntax, because you you cannot use a new keyword in Python, before an, a class definition or something like that. So I made it a function, simple and what worked well. So there are 2 ways to instantiate objects in transcript. You can call that new function, or you can encapsulate the library so that you don't have to call it. You just call the constructor.
And for heavy used libraries, the encapsulation is the way to go, but for occasionally used and very large libraries, I I now use fabric JS, which is changing all the time and also, of course, jQuery is changing all the time. I don't think you must encapsulate or change anything. You just use it as is. And then there was still 1 problem, that's the problem of the ambiguous keys. When I use a dictionary in Python, I can have numerical keys, I can have string literals as keys. If I call a function, it's evaluated, and what it returns is used as a key. But in JavaScript, things are much more informal.
If you put quotes around something, it's a key, but if you leave out a quote, it's the same key. And if it's an integer, well, it may be interpreted as an integer, but then also can be interpreted as a string. So it's all very vague in my view as a Python programmer. So how to solve that? Well, I thought there are 2 groups of people who will use transcript. 1 group, hopefully, are the Python programmers programming free web, and maybe there are some JavaScript programmers as well that like the structure of this language. So I made another pragma. There are several of them so that you can locally switch between the JavaScript interpretation of keys or the Python interpretation of keys. So you can use these strict Python interpretations. You have to have quotes around the key, and a number is a number and never a string. But you can also use the loose, somewhat informal JavaScript way of working.
I don't care whether you use quotes, and if it looks like an integer, well, still maybe a string. So both possibilities are available, and those were the main things I think well, you have you have this dollar sign thing. I I wanted pure Python syntax and a dollar sign cannot be an identifier, But the aliases, the the way the the the mechanism that you can replace 1 identifier by another also solves that in a very neat way. So there were no real big problems in that respect.
[00:13:17] Unknown:
You also incorporated the DOM API into transcript. Were there any additional challenges posed by that? I can imagine that that would simplify actually writing the transcript code as opposed to having to always translate into Python and then figure out how that maps to DOM elements. But having it natively supported in the Python library, I imagine, would greatly simplify the developer flow. Flow. But, was it difficult to bring that into Python and, maybe some background on how you did that?
[00:13:47] Unknown:
Well, the fun of it is that I didn't have to support it at all since every Python function call is just translated into a JavaScript function call and the the extra mechanisms, for instance, for, varargs and things like the star and the keywordargs, they they are optional. So if you do nothing special, your Python function call just translates into a JavaScript function call, which means that you can all use all those dumb functions without any special measures and also dumb objects. Well, a JavaScript object is just a special case of a transcript object. So you can use those JavaScript objects, DOM objects, library objects, and their methods without any special measures. So I didn't have to do much there.
[00:14:36] Unknown:
It's always a good sign of proper design when things like that just happened to shake out naturally based on how the library was written.
[00:14:44] Unknown:
I agree. I agree. But a design I think a design is good when it starts to shrink. Yes. At first, it gets bigger and bigger and more complicated, and then things start to fit together and it shrinks and it shrinks and it shrinks. And I agree. That that means you're on the right way. Yep. Transcript is simple. You you anyone can understand the core, and the core of it is maybe, well, maybe maybe it's 6 k or something like that. And, well, I gave that part a lot of thought because that really matters. For instance, function calling. Halfway, I got a criticism that was, I didn't like it, but the guy was right. He said to me, well, you have made this function pointer assignment, and you did that by making each function a property, but that gives a lot of overhead. Well, at first, I thought, well, it's a little bit of overhead, who cares? But in an inner loop, it can be important. So I thought about that, and then I read something about memoizing, core memoizing, and I decided to make optionally, make functions not a member of a prototype, but directly of the object even if they are inherited. It means that you call a function 1 time, then the property is replaced by the address of the function that has been looked up along the prototype chain. Next time you call it, it will be called directly. So, actually, it's faster than JavaScript itself in that respect. So this type of criticism, well, I like very much that people, think about these things and give feedback about it because it really makes it better.
[00:16:08] Unknown:
Yeah. Definitely. And you mentioned briefly that, at least in that particular case, the performance of the compiled transcript is actually better than native JavaScript code. I'm wondering in the more general sense how the performance compares between, what you compile from transcript and what an average web developer would write in vanilla JavaScript?
[00:16:29] Unknown:
Well, of course, if you were writing in native JavaScript yourself and you really optimize by hand, you can't be beaten. The same holds for Assembler, by the way. And you can say a program is c, but since anything you can do in c, you can also do in Assembler, you can always win. But in practice, of course, you can when you program in JavaScript, you can also search the prototype chain for once and then replace that method by a direct call, but people don't do that. And, well, with regard to the performance, I think this in the same order of magnitude, it should be it should be roughly equal.
There are some things it depends on which switches you use. There was the point of operator overloading. If JavaScript doesn't have operator overloading. And at first, I thought, well, I don't care. But since I do a lot of numerical analysis and a lot of matrix algebra and differential equations, I wanted it in. But if you do that, every plus, if you add 2 integers, there will be a call to an as method, and that's unacceptable. So, again, the same solution. I noticed in my own code that there there are some parts where a lot of computations get done with matrices and vectors and other parts are just simply a GUI code or or, well, just using integers. So you can switch operator overloading on locally and you can switch it off again. Same holds for, having keyword arguments. The keyword argument mechanism is very handy, but it's also very expensive in JavaScript. So also you can switch that on or off. So I think by that type kind of measures, I expect I haven't done a benchmark and I probably won't do that myself because it will be biased, but I invite people to try.
If I look at what type of code is generated, the the performance should be about the same. And if you don't constantly use things like a variable number of arguments and things like that or keyword arguments, I think the the Python code will be about 20%. Sorry, the JavaScript code will be about 20% larger, and that is the case because JavaScript version 5 that I used doesn't have things like list comprehensions. So, of course, it will be longer because you have to translate it to a for loop, but it will be about 20% longer. So I don't expect there to be any performance problems, really. And, by the way, I think you should start optimize for performance too early. It's more important that you understand your program and you can roughly optimize the the logic of it and people optimizing so so, optimizing on in minor spots.
There's another which allows you to embed native JavaScript into your transcript. So if there's a point where you say, well, I'm in an inner loop and it's not fast enough, I doubt whether that's the case. But if it is, you can just insert native JavaScript at the point in your code.
[00:19:14] Unknown:
And you were mentioning being able to turn on different features on and off like the keyword arguments and operator overloading? And is that all just handled in with pragma, statements in the code?
[00:19:25] Unknown:
Mostly it's handled by pragma statements. There are there are generally 2 ways. You can use a command line parameter, which in many cases I'd advise, whereas it's very easy to switch all the Python facilities on, but then you will have bloated code because, well, there are some things that Python can do that JavaScript just cannot yet do, operator overload, for instance. So to augment the the compiler switches, there's a a growing set of that will enable you to switch things off and on locally. They're documented in the documentation. It's explained what they're for.
Another problem that they encounter, there's a static checker on board and also minifier, Google, what's it called? Google Closure Compiler. And there's a there's a static checker, and the static checker doesn't know about Java JavaScript Globals. So whenever you use a global, I think you shouldn't, but whenever you do, you do complain that you use something that's not defined because it cannot see a definition. So another pragma is about telling the the checker, okay, I use console, I use window, names like that. Don't bother because they are defined. They're not defined in my program, but don't complain about that. So I use that mechanism a lot.
[00:20:31] Unknown:
And so how is the transpilation performed, and what are some of the methods that you use to keep the build size as small as it is?
[00:20:38] Unknown:
The parsing is done by the AST module. That will be clear because, well, I cannot cannot beat the native parser. It's the standard. When I was developing, I soon found out that if it it was growing and growing and growing the code, and it's very simple to throw something over when you implement a new feature. So what I added in a very early stage was automated testing, back to back testing with CPython. That is an important part of development process because I can add things and just run the auto test. It it's included, and I hope people that write libraries for it will also auto test them properly.
So the testing problem was something I had to tackle early, to be able to develop rapidly. Another thing is the way code is actually generated. The AST module has methods for that. It's a it's the visitor pattern. You have a syntax tree with nodes and you can visit it. At first, I visited all the nodes as they were, but then there came some, well, some trouble in paradise because, for instance, tuple assignment is not as simple as it may seem. There are some transpilers that have tuple assignment, but it doesn't work properly. It only works for if you assign a flat tuple to another tuple. But in CPython, this may be hierarchical and nested.
And you can use tuple assignments just to assign a, a comma b is, 1 comma 2, but you can also do this in a for loop using enumerate, for instance. And there are other places. So I found myself repeatedly writing the same code generation, and that was when I said to myself, I have I I must not do that. If I have an assignment in the header before loop, it's just an assignment. So what I do is I generate an assignment node on the fly. I alter my syntax tree, and I use exactly the same code generation, which lowers the the possibility of errors. So I use the syntax tree, but not as is I modify it. And, well, that's that was also a rather important, well, discovery.
And there were some things where I had to try to make a wise choice. 1 such point was truthiness. A list in Python, an empty list is is evaluated as false. But in JavaScript, an empty array is evaluated as true. Now it's very seductive to put all those bells and whistles in and convert this stuff. But, okay, what would you gain? If you don't convert automatically, you have to check for the length of an array rather than say, if array you have to say if array dot length or a len array Python. That's a minor price to pay compared to having to check anything for matching type and perform either auto conversion at runtime. So that was a point where I said, okay, truthiness, whether something is true or not, JavaScript is fundamentally different from Python, and I will not add the mechanisms to do the automatic conversion because it will make it slow and bloated. And I don't mean it to be perfect complete Python. It is it it goes quite far, but I mean it to be usable fast enough. And so that was well, I didn't like to have to make the choice, but I I had to.
There's the, there's a module called asteroid, which adds on extra functionality on top of the AST. I don't know if any of the added capabilities would be useful to you or not. I've looked into it. Yes. Right. There are quite some transpirters that use it. I forgot which ones, but it is used. No. I didn't feel the need for it. I didn't feel need for it. The AST module had enough functionality. If you look at the code generator, it's it's all in the distribution. It's the let me see. It's called compiler dot py. And you see that it's very simple and straightforward. It wasn't difficult. AST module is is powerful enough. I know there are transpilers that try to abstract away the version dependency from a certain Python version by making a kind of intermediate tree, but I didn't want that. I wanted to be very close to especially certain Python versions. So that's that's a point to consider.
You it it it is not a Python transpire. It's a Python 3.5 transpire. And as soon as 3.6 get out, you will have a 3.6 transpiler. Why? Well, because if you don't do that, you cannot use the new features. You you have to to zoom out a little bit and get a little bit vague. But, no, I use nothing on top of ASD. I took a look at quite some things there, and, I just didn't feel the need. It's it's powerful enough as is.
[00:25:03] Unknown:
Do you anticipate any problems or, excess work that comes from having to match so closely the version of transcripts to the version of Python that it's running on?
[00:25:14] Unknown:
Well, yes and no. I hope that the revolution that took place from Python 2.7 to 3.0 doesn't get repeated. I mean, I I find it very courageous of, Guido that he took that step and, well, I think the whole developer community, well, was in a shock when it hit. But I I don't expect any revolutionary changes in Python itself. So, transcript will be growing and will be in an evolutionary process. And I will not try to keep up within a week within the newest Python version. But since, well, it's a moving target, Python itself, if transcript wants to remain, well, a cutting edge product, it will have to be adapted anyhow. So I'll try to keep it in place. I don't see it as an extra burden.
The the alternative was would be, well, okay, I make something that roughly looks like Python and I keep it the way it is right now and I freeze it. Well, that's not what I want. I expect Python 3.6 to be an evolutionary step, and in some respects, transcript will take that same step and in others, it won't. It it will never be a complete Python implementation because that would be slow and bulky.
[00:26:24] Unknown:
So given the nature of JavaScript's prototypical inheritance and differences in class semantics, I imagine that adding support for a multiple inheritance in reflecting the structure of Python classes must have been pretty challenging. I wonder if you can describe the process that you undertook and how you arrived at your current solution.
[00:26:41] Unknown:
Yeah. Well, it was funny. When I started out, I had the automatic misconception that transcript inheritance should somehow have something to do with JavaScript inheritance When there's an inheritance mechanism there, so my prejudgment was if I use the JavaScript addition to add, which which you should because of performance, then I should also use the JavaScript inheritance to make transcript or Python inheritance. Well, that's not true. You don't often inherit. You don't often encounter and interpret a clause definition. So inheritance, in fact, doesn't have to be very time optimized. It has to be space optimized, but not time optimized because, well, how many classes will you have in application? 100? Well, then that means that you read 100 times to a class definition. So what I did is, at first, I started out using JavaScript inheritance. It turned out to be hard and clunky and and artificial. So then I said, well, okay. I won't use it at all. I will just try to implement Python inheritance in JavaScript. So I just copied the methods, the properties, things like that, and you you really need to have a lot of freedom freedom there because implementing properties, well, was difficult enough in itself without bothering about JavaScript's, prototypical inheritance.
So I just bypassed it, and, well, 1 good idea was at first at first, I thought a class should be an object. But since you can call a class to instantiate something, I discovered that a class must not be an object, but a function. That was 1 step. And another step was indeed to say, well, what do I need in Python for multiple inheritance? I just write it out in JavaScript, and then it turns out that it's still very compact. If you look at at the core module, it's it's called, underscore underscore core underscore underscore dot mod dot gs, and you see it's only a few k of code for everything, including multiple inheritance. So it took quite some thinking, but in the end, it turned out to be very simple. The the the biggest step was to let go of JavaScript inheritance.
You cannot you cannot use that. It just gets in your way.
[00:28:43] Unknown:
So what aspects of the Python language were most difficult to translate to JavaScript?
[00:28:48] Unknown:
Well, 2 things, I think. Yes. Well, 1 thing I couldn't translate, that's what that that was the truthiness, automatic conversion. I I think it would be too slow, so I let go of that. The other 2 things that were hard, 1 was the keyword arguments. I really have to think think very deeply, and still it is, well, it takes a lot of JavaScript code to implement that, so we had to make it switchable. You switch it on only for a function that you expect to call with keywords. That was 1 point. And the other point that was hard, to implement was was the operator offloading.
It's not hard to implement, but it's hard to do it properly without hampering your performance. Oh, yes. And, of course, 1 thing I I almost forgot because it was very early in the process, I wanted to be able to assign a method to a variable. So it should be a bound method, which means that you have to implement somehow the descriptor mechanism. Well, that really was very hard, and I think I had some sleepless nights over there. But, yeah, it worked out.
[00:29:48] Unknown:
You mentioned that you don't anticipate much difficulty in keeping up with different versions of Python. But, what about when the newest JavaScript standard, the ES 6, is more generally used and widely supported in browsers? Do you anticipate a lot of work necessary for updating going forward, I'm wondering if you anticipate, the large body of work going forward. I'm wondering if you anticipate, a large body of work necessary to maintain compatibility there.
[00:30:21] Unknown:
Yeah. Well, 1 in programming is always a lot of work. It's, it seems simple if you think about it, but if you have to implement it, it's still a lot of work. I expect it to be dual. For for 1 thing, I will not switch too early to JavaScript version 6 because indeed, as you say, it has to be available on many browsers. If you want to make websites using that, you have to be sure that your clients can understand can understand it. But I see some developments with the JavaScript, comprehensions, for instance. And, well, some other stuff that makes it attractive to make the switch, but it will be an option.
To be a command line switch or not a pragma. I don't think it will be a lot of work because, well, once JavaScript will support, comprehensions, array comprehensions, well, compiling list comprehensions will be simpler. And what you see is that JavaScript is in a in a process of keeping up with some good Python ideas, modules, things like that. And, well, my father used to say that, a camel is a horse designed by committee. I think that's happening with with JavaScript. In the end, everything will be in as is the case with c plus plus, but it will won't have the simplicity. But I I have the idea that the newer features of JavaScript are converging with Python, but what I'm afraid will never converge is the prototypical inheritance. And, well, it's totally subjective. I know that there are people who dream in JavaScript, but I can't get my head around it. I understand how it works, but it doesn't fit my way of thinking. So it it will, JavaScript and Python will converge, which will make it not too difficult to trans to transpire, to compile for newer JavaScript versions. I expect that not to be very much work. It will be an option for a long time and, well, I don't expect JavaScript to be at 1 point so so good that the Python community will switch to it because, well, the the power of Python is that it has been designed, I think, and it's guided by 1 man. And, well, that keeps it kind of unified.
And the transcript is a, a I hope it will be a community thing. I hope it will be used by many people. Currently, some of my students are writing libraries for it. NumPy is high on the wish list, and it should use the graphic card for computations. Another thing is not, but I'm afraid that will be hard to do. But what I plan to do, I will not develop everything myself, but I plan to keep my hand on the wheel as well because I noticed that people make demands for all kind of features, and it's easy to make it grow in every direction, but it should stay a bit focused. And so I won't have all to do all that work myself, I think. I hope. I hope it catches on. And but but I will try to control the direction it it moves in because I want it to stay lean on me.
[00:33:22] Unknown:
So is transcript complete and stable enough to be used in production?
[00:33:27] Unknown:
Yes and no. I use it for production, and I think I'm a lot happier with it than programming JavaScript. It's quite stable, but to be honest, it's an alpha version, and it will remain an alpha version until it's, well, really stable. So I hope people will give feedback as some people do. I I hope to I cannot test everything myself. I mean, the auto test is there, but people come up with users that I haven't thought of of. So even as it currently is, I prefer it for my own commercial projects. I prefer it above JavaScript. It is quite well tested, quite stable, but there there will be some corpses in the cupboard as well. There will be some things that are really wrong and are just never bumped into them. I hope people will report. I expect it to be in alpha for another few months, and, well, hopefully, the crooks will be out there. There are no showstoppers in right now as far as I know. I didn't write it well, I didn't write it in such a way that I first made it very clumsy and and thoughtless, and then I improve it or something like that. I like to try to do it right the first way, but, of course, it's work of a human being, so there will be there will be problems. I I haven't run into any right now. I I don't know of any bugs, but if people start testing, they will come up with things. I'm I'm convinced.
[00:34:44] Unknown:
So you were mentioning how you've had people requesting porting things like NumPy and Matplotlib into transcript, JavaScript environment. So I'm just curious if you have any thoughts on that matter.
[00:35:00] Unknown:
Yes. I have. I won't go the asm.gs way, but for NumPy, as I said, 1 of my students is currently, studying at the Delft University. He's looking into that with a friend. What I would like to do for NumPy is use the graphic card. So, you know, when you use your your g p GPU, you use your card for computations, it can be about 30, 40 times faster than even when you program it directly in c plus plus. So c plus plus, well, no. I I don't see that happening anytime soon. Because if that would be possible, if it would be possible to have efficient coding c plus plus running in the browser, transcript would be not necessary at all. We would just run c Python. But I I don't see that happening anytime soon, but I think that using the graphic card is 1 way to go using asm.js is another way to have efficiency and maybe even run some c code where necessary. But, well, I accept the fact that the browser world is dominated by JavaScript. So even suppose you use asm.gs, you can probably make things that are very efficient, but they are also very unreadable. And people wanna debug and they wanna understand their code. I don't think there will be much c code. I think there will be fast code probably on the graphic card, but not much no no no c code on top of asm.jsi.
I don't expect that to have much advantages.
[00:36:13] Unknown:
So are there any questions that I didn't ask that you think I should have or anything else that you wanna bring up before we move on?
[00:36:18] Unknown:
Well, I think your questions were quite good and, we had a good view on, what transcript is, but, I have a question. Sure. I think, technically, this is rather good than anyone just for itself. I won't run and I have re had a request for benchmarks, but I won't do that because it's so easy to to skew them, to use a piece of code, transcript, is is very good at. I also don't want to say this is the 1 compiler to end it all and to replace other compilers. But what I would like is for people to take a look at it, judge it on its merits, and give feedback because there are a lot of, people starring on GitHub and there are a lot of downloads on PyPI, but I I get too little feedback. I I need, well, input, to be able to test it and what it needs most, and I have no idea how to do it. I hope this interview helps and, well, thank you for that. But what it needs is mindshare. If people start to develop serious things in it and they start to invest money in making applications with this, it must not go away. Well, what's important just in that respect, I don't see it as my property. I see it as my baby. That's true. But, well, my children are also not my property and, transcript isn't isn't. I will keep a hand on the wheel, but this is open source. It won't go away. It can't go away. You can anyone can fork it. If you look at the source, it's simple. So somehow, I don't know how to make that happen, but somehow I hope it gains mindshare and people can interest in it. I've been a developer for ever since I was 15. I'm 57 now, so you can know how long that is. I still service software that I made 30 years ago, still used, and it's still still okay. So what I would like to say is this is really serious. It's not going away. It's not a hobby project. I will try to to involve more people, and it's open source. Well, I hope people will use it. Yeah. I think that's very reassuring hearing you say how it's not a hobby project and it's something that you consider a serious piece of work because I know that a number of the other Python to JavaScript transpilation libraries are seen more of from an experimental perspective and not really intended necessarily for production everyday use. So I definitely think that that's another point where transcript kinda stands out from the crowd. It is for production work. I I program for a living and I I programming medical imaging. That that was my problem. And I wanted to run stuff in the browser and I couldn't in a way that was acceptable for me. So it's for me, it's very serious. It's part of a multi year pro project that many people are working on. And, well, I just thought this is usable. More people can use it. But it indeed, it is very serious.
[00:38:48] Unknown:
Yes. So for anybody who wants to follow you and get in touch with you and, keep up to date with the things you're working on, what would be the best way for them to do that? There are some, some ways to do that. There is the transcript site,
[00:39:01] Unknown:
www.transcript.org. There's a forum where it's still very quiet because who wants to be the first? There's a forum and you can reach me there. There on PyPI, there's my email address, you can reach me there. And, well, I like to have feedback, so feel free to to send an email or to answer the forum. I will react. I will will not I will not blindly implement any feature that's re requested because then it would be bloated. But approach me by email, for instance, or on the forum, and, I will certainly react and, well, that that's the way to to make contact about what your wishes are. And I invite you to write libraries for this because I will not do that myself. I will try to make the core as good as possible, but I hope people will start writing libraries. Okay. So with that, I will move us on to the picks.
[00:39:49] Unknown:
And for my first pick today, I'm gonna choose the cookie cutter library. I've, just started doing some more work in SaltStack for my job, and I ended up writing a cookie cutter template for being able to easily generate the directory layout and, some, initial files for SaltStack formulas. So it definitely is saving a lot of time in that respect and, making sure that anybody who
[00:40:17] Unknown:
is leveraging that in inside my team is, starting from the same point. So definitely very useful for that. And I'm just gonna leave it at that today. So, Jacques, what do you have for your picks? I have something, and it's meant especially for the Dutch audience, but also for a broader audience. I've been programmed all my life, and I learn something every day from our students. And I think, well, there's a lot more to programming than is taught in educational institutions. So I'd like to invite young people to dive into this and will not be shied away by all the the jive talk around it and and all the design methods and things like that. Programming in itself, I've done it all my life, it is fun, it is rewarding, you can earn a living with with it, and, well, it's I know it isn't a popular thing to say, among among managers, but it's really an art. I like my code to be beautiful and I programming's fun.
[00:41:12] Unknown:
So I just wanna invite people to to be a programmer and to to take, to take pride in your job and try to do it properly. Well, Zach, I definitely appreciate you taking the time to, take the time out of your day to tell me and my audience more about transcript. I'm sure that there are a number of people who are very interested to test it out. And hopefully hopefully, you can get some feedback from that. So, thank you again, and I hope you enjoy the rest of your evening. Oh, thank you very much for giving me the opportunity to expose this to the public, and to enjoy your day. Bye. Alright. Thank you. Bye.
Hello, and welcome to podcast.net, the podcast about Python and the people who make it great. You can subscribe to our show on Itunes, Stitcher, TuneIn Radio, or add our RSS feed to your pod catcher of choice. You can also follow us on Twitter or Google plus, and please give us feedback. You can leave a review on Itunes to help other people find the show, send us a tweet or an email, leave us a message on Google Plus or our show notes, and you can also join our discourse forum at discourse.pythonpodcast.com for your opportunity to find out about upcoming guests, suggest questions, and propose show ideas.
I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. For details on how to support the show, you can visit our site at pythonpodcast.com. Linode is sponsoring us this week. Check them out at linode.com/podcastonit and get a $20 credit to try out their fast and reliable Linux virtual servers for your next project. I would also like to thank Hired, the job marketplace for developers and designers for sponsoring this episode of podcast.init. Use the link hired.com/podcastinit to double your signing bonus.
Coming up in Boston on May 21st 22nd is the open data science conference. So if you are in the area and you're interested in learning about how Python can be used in the areas of data science, then you should check it out. You can use the discount code e p for 20% off your ticket. That's odsc.com. Today, we are interviewing Jacques de Hoog about his work on the transcript project. Jacques, could you introduce yourself, please?
[00:01:44] Unknown:
Yes. Okay. I will. Well, I'm a software developer ever since I was 15. My father worked with computers, shortly after the 2nd world war, and I've been programming all my life. That's part of my job. I make technical computer programs, so, real time control systems and also some cardiology software for for medical imaging. And I give lectures at the Rotterdam Academy, part time, about 30% of my time. And the rest of my time, I spent making, open source software.
[00:02:20] Unknown:
And how did you get introduced to Python?
[00:02:23] Unknown:
Oh, it's a funny story. I was a quality manager with Fugro, that's a worldwide engineering firm. And there was this guy, William Chester, from Great Britain. He was a very intelligent guy, and he I I was making a a user interface of c plus plus with scripting possibilities. And he came to me and he said, Jacques, you're implementing Python in c plus plus. And I said, Python, what's that? I said, well, take a look. And I I took a look, and it turned out to be a runtime type language, scripting language, and I said, I was quality manager. So I said, no way that we will will use this language in this company because it will become a mess if we don't have strict typing. So I was very much opposed to introducing, Python, but still, I tried it a little bit and a little bit more and a little bit more. And, well, it didn't took long before I was amazed and very much convinced of its power because, it's true that sometime I miss a strict typing, but it's so expressive and so brief, and the data structures are so powerful that, well, it has saved me a lot of time. So that was my introduce introduction to Python.
[00:03:28] Unknown:
And, you mentioned that you like to do some open source work, which brings us to the transcript project. And I'm wondering if you can describe, what transcript is and what inspired you to create it.
[00:03:41] Unknown:
Yes. Well, it is a a Python to JavaScript compiler. It's not new. There are several of them. And it it allows you to run a large syntactically and semantically faithful subset of Python 3.5 in your browser. Well, there are more products that do that, but I think the special thing is that while it has implemented a large part of Python 3.5, the code that is produced, the JavaScript code is very lean and very readable and very debuggable. So that is what it is actually. And I've programmed and also using Django, and I always longed to be able to use Python on the client side as well. And I had an upcoming project where a lot of client programming had to be done, so I looked into what was there already. And, well, there are many many transpilers, Python to JavaScript compilers.
There are, for instance, PyPy GS, which is really generating c code and is running on top of axm.gs, which is, well, very powerful but also a bulky download, 1 side of the spectrum, and then you've got, you've got, RapidScript on the other side of the spectrum, which is a thin layer around JavaScript and misses some features that I wanted in. Some people will disagree. I I like multiple inheritance, for instance. I know that Java programmers don't care. James Gosling doesn't care. I do care about multiple inheritance. I'm I'm from the c plus plus world. And, well, I started out using a RapidScript, and I liked it, but when I was using it, I missed some things, especially the multiple inheritance, bound function pointer assignments, things like that.
And well, I decided that there was a room for another product and, all my life, I liked making interpreters, compilers, stuff like that. I have made a modular 2 interpreter in the old days. So I thought this may be fun, and I came across the AST module of Python. And what I wanted to do is to start out from the Python end of things. I mean, if I take, for instance, RapidScript, it starts out from the JavaScript side, using Node. Js and things like that. I wanted this to be a Python thing. I wanted to this to be distributed via PyPI. I wanted a pure Python syntax, so using the Python parser, stuff like that. So I set out and it was a kind well, it wasn't very serious. I put on the website, don't use this because it may go away any minute. But when I was programming it, actually, when I made a start, it turned out to be easier than I thought and, well, there were some hard parts, but I started to like it and it became more and more serious and I was amazed what you could implement in a still generating very compact source code. So I decided that was serious after all and I I reserved an URL a URL for its transcript.org.
And, well, it became a real project in that way.
[00:06:45] Unknown:
And, where did you get the name from?
[00:06:48] Unknown:
Well, to transcribe, you know, that's what monks did. They couldn't make a photocopy, so they they took, 1 of those big codexes and they wrote it down down by hand. I've tried to program multiple inheritance directly in JavaScript, but I don't like it. So how I see it is that the transcription is made, as a kind of compilation from transcript to JavaScript. So that's where the name comes from.
[00:07:12] Unknown:
So I like that you decided to embrace the web environment by calling it JavaScript libraries. What are some of the challenges that you encountered while creating that functionality?
[00:07:21] Unknown:
Python is famous for its libraries. Yeah. It's it's better easy to include it, thing. But I realized that, for the web, you cannot simply copy or reimplement the the desktop libraries of Python. The demands made are very different. For the web, good looks are important. Look to all this, jQuery, gooey stuff, and many things have been made. There's a very large body of high quality libraries for JavaScript, actually. I I don't know exactly how people succeed in writing such good libraries in such a language, but, okay, they're there. It's an illusion that you can bypass that and and re redo that in Python. So I felt from the beginning, if this is to be taken serious, it has to to gel really well with things like jQuery and, well, all the all the other stuff that's there, the beautiful stuff that's there for the web. So, well, what I encountered there were a few problems. 1 of the problems that the the JavaScript community doesn't doesn't particularly care about, it seems, are namespace pollution and name clashes. Well, there are there are solutions emerging now, but, I mean, there's still no decent module system, for instance. And, I I hate that. I hate it when I make a program and some new library comes around and have to change my variable names, for instance.
So, I try to come up with an encapsulation mechanism so that you can use a JavaScript library that normally has its stuff in global name space. You can encapsulate it so that if viewed from Python, it is in the module name space. That was that was 1 of the problems. Another problem was in the data structures, there are broadly 2 ways to solve the problem of, having Python data structures alongside JavaScript data structures. The the most dogmatic solution is that you you make separate data structures in Python. I mean, that means that a Python dictionary is something quite different from a JavaScript object, and a Python list will be something totally different from a JavaScript array.
But if you do that, you will have to convert and you'll have to write a lot of code, which all has to be downloaded to the client to run it. On the other hand, you have the people who just say, well, an array is almost a list, so we add some methods. And then, we we add some Python methods, and then an array will be a list. But that's not without problems because, for instance, an array has a sort method sort method in JavaScript, and a list has also a sort method in Python, but those 2 are different. The parameters are interpreted different. So what I did is, come up with a concept of pragmas, which are compiler directives, and 1 of the pragmas is the aliens pragma.
And when I say sort in Python, this is translated to a py_sort in JavaScript. So I can enrich the native Java JavaScript types with extra methods without causing any name clashes and without causing any code bloat or delay. That was an important, well, discovery that you could do that. And then there was the new statement. In in in Python, you don't have to say new, you just call the constructor and and things are instantiated. Now what you can do is encapsulate every JavaScript new in every library, and then you don't have to call new anymore. It is possible in transcript, you can do that. But since there are so many libraries and are growing and are changing, I don't think that's generally a good idea. So I had to have some kind of new statements for the for the JavaScript stuff, and I wanted to be syntactically, confirmed the Python syntax, not the JavaScript syntax, because you you cannot use a new keyword in Python, before an, a class definition or something like that. So I made it a function, simple and what worked well. So there are 2 ways to instantiate objects in transcript. You can call that new function, or you can encapsulate the library so that you don't have to call it. You just call the constructor.
And for heavy used libraries, the encapsulation is the way to go, but for occasionally used and very large libraries, I I now use fabric JS, which is changing all the time and also, of course, jQuery is changing all the time. I don't think you must encapsulate or change anything. You just use it as is. And then there was still 1 problem, that's the problem of the ambiguous keys. When I use a dictionary in Python, I can have numerical keys, I can have string literals as keys. If I call a function, it's evaluated, and what it returns is used as a key. But in JavaScript, things are much more informal.
If you put quotes around something, it's a key, but if you leave out a quote, it's the same key. And if it's an integer, well, it may be interpreted as an integer, but then also can be interpreted as a string. So it's all very vague in my view as a Python programmer. So how to solve that? Well, I thought there are 2 groups of people who will use transcript. 1 group, hopefully, are the Python programmers programming free web, and maybe there are some JavaScript programmers as well that like the structure of this language. So I made another pragma. There are several of them so that you can locally switch between the JavaScript interpretation of keys or the Python interpretation of keys. So you can use these strict Python interpretations. You have to have quotes around the key, and a number is a number and never a string. But you can also use the loose, somewhat informal JavaScript way of working.
I don't care whether you use quotes, and if it looks like an integer, well, still maybe a string. So both possibilities are available, and those were the main things I think well, you have you have this dollar sign thing. I I wanted pure Python syntax and a dollar sign cannot be an identifier, But the aliases, the the way the the the mechanism that you can replace 1 identifier by another also solves that in a very neat way. So there were no real big problems in that respect.
[00:13:17] Unknown:
You also incorporated the DOM API into transcript. Were there any additional challenges posed by that? I can imagine that that would simplify actually writing the transcript code as opposed to having to always translate into Python and then figure out how that maps to DOM elements. But having it natively supported in the Python library, I imagine, would greatly simplify the developer flow. Flow. But, was it difficult to bring that into Python and, maybe some background on how you did that?
[00:13:47] Unknown:
Well, the fun of it is that I didn't have to support it at all since every Python function call is just translated into a JavaScript function call and the the extra mechanisms, for instance, for, varargs and things like the star and the keywordargs, they they are optional. So if you do nothing special, your Python function call just translates into a JavaScript function call, which means that you can all use all those dumb functions without any special measures and also dumb objects. Well, a JavaScript object is just a special case of a transcript object. So you can use those JavaScript objects, DOM objects, library objects, and their methods without any special measures. So I didn't have to do much there.
[00:14:36] Unknown:
It's always a good sign of proper design when things like that just happened to shake out naturally based on how the library was written.
[00:14:44] Unknown:
I agree. I agree. But a design I think a design is good when it starts to shrink. Yes. At first, it gets bigger and bigger and more complicated, and then things start to fit together and it shrinks and it shrinks and it shrinks. And I agree. That that means you're on the right way. Yep. Transcript is simple. You you anyone can understand the core, and the core of it is maybe, well, maybe maybe it's 6 k or something like that. And, well, I gave that part a lot of thought because that really matters. For instance, function calling. Halfway, I got a criticism that was, I didn't like it, but the guy was right. He said to me, well, you have made this function pointer assignment, and you did that by making each function a property, but that gives a lot of overhead. Well, at first, I thought, well, it's a little bit of overhead, who cares? But in an inner loop, it can be important. So I thought about that, and then I read something about memoizing, core memoizing, and I decided to make optionally, make functions not a member of a prototype, but directly of the object even if they are inherited. It means that you call a function 1 time, then the property is replaced by the address of the function that has been looked up along the prototype chain. Next time you call it, it will be called directly. So, actually, it's faster than JavaScript itself in that respect. So this type of criticism, well, I like very much that people, think about these things and give feedback about it because it really makes it better.
[00:16:08] Unknown:
Yeah. Definitely. And you mentioned briefly that, at least in that particular case, the performance of the compiled transcript is actually better than native JavaScript code. I'm wondering in the more general sense how the performance compares between, what you compile from transcript and what an average web developer would write in vanilla JavaScript?
[00:16:29] Unknown:
Well, of course, if you were writing in native JavaScript yourself and you really optimize by hand, you can't be beaten. The same holds for Assembler, by the way. And you can say a program is c, but since anything you can do in c, you can also do in Assembler, you can always win. But in practice, of course, you can when you program in JavaScript, you can also search the prototype chain for once and then replace that method by a direct call, but people don't do that. And, well, with regard to the performance, I think this in the same order of magnitude, it should be it should be roughly equal.
There are some things it depends on which switches you use. There was the point of operator overloading. If JavaScript doesn't have operator overloading. And at first, I thought, well, I don't care. But since I do a lot of numerical analysis and a lot of matrix algebra and differential equations, I wanted it in. But if you do that, every plus, if you add 2 integers, there will be a call to an as method, and that's unacceptable. So, again, the same solution. I noticed in my own code that there there are some parts where a lot of computations get done with matrices and vectors and other parts are just simply a GUI code or or, well, just using integers. So you can switch operator overloading on locally and you can switch it off again. Same holds for, having keyword arguments. The keyword argument mechanism is very handy, but it's also very expensive in JavaScript. So also you can switch that on or off. So I think by that type kind of measures, I expect I haven't done a benchmark and I probably won't do that myself because it will be biased, but I invite people to try.
If I look at what type of code is generated, the the performance should be about the same. And if you don't constantly use things like a variable number of arguments and things like that or keyword arguments, I think the the Python code will be about 20%. Sorry, the JavaScript code will be about 20% larger, and that is the case because JavaScript version 5 that I used doesn't have things like list comprehensions. So, of course, it will be longer because you have to translate it to a for loop, but it will be about 20% longer. So I don't expect there to be any performance problems, really. And, by the way, I think you should start optimize for performance too early. It's more important that you understand your program and you can roughly optimize the the logic of it and people optimizing so so, optimizing on in minor spots.
There's another which allows you to embed native JavaScript into your transcript. So if there's a point where you say, well, I'm in an inner loop and it's not fast enough, I doubt whether that's the case. But if it is, you can just insert native JavaScript at the point in your code.
[00:19:14] Unknown:
And you were mentioning being able to turn on different features on and off like the keyword arguments and operator overloading? And is that all just handled in with pragma, statements in the code?
[00:19:25] Unknown:
Mostly it's handled by pragma statements. There are there are generally 2 ways. You can use a command line parameter, which in many cases I'd advise, whereas it's very easy to switch all the Python facilities on, but then you will have bloated code because, well, there are some things that Python can do that JavaScript just cannot yet do, operator overload, for instance. So to augment the the compiler switches, there's a a growing set of that will enable you to switch things off and on locally. They're documented in the documentation. It's explained what they're for.
Another problem that they encounter, there's a static checker on board and also minifier, Google, what's it called? Google Closure Compiler. And there's a there's a static checker, and the static checker doesn't know about Java JavaScript Globals. So whenever you use a global, I think you shouldn't, but whenever you do, you do complain that you use something that's not defined because it cannot see a definition. So another pragma is about telling the the checker, okay, I use console, I use window, names like that. Don't bother because they are defined. They're not defined in my program, but don't complain about that. So I use that mechanism a lot.
[00:20:31] Unknown:
And so how is the transpilation performed, and what are some of the methods that you use to keep the build size as small as it is?
[00:20:38] Unknown:
The parsing is done by the AST module. That will be clear because, well, I cannot cannot beat the native parser. It's the standard. When I was developing, I soon found out that if it it was growing and growing and growing the code, and it's very simple to throw something over when you implement a new feature. So what I added in a very early stage was automated testing, back to back testing with CPython. That is an important part of development process because I can add things and just run the auto test. It it's included, and I hope people that write libraries for it will also auto test them properly.
So the testing problem was something I had to tackle early, to be able to develop rapidly. Another thing is the way code is actually generated. The AST module has methods for that. It's a it's the visitor pattern. You have a syntax tree with nodes and you can visit it. At first, I visited all the nodes as they were, but then there came some, well, some trouble in paradise because, for instance, tuple assignment is not as simple as it may seem. There are some transpilers that have tuple assignment, but it doesn't work properly. It only works for if you assign a flat tuple to another tuple. But in CPython, this may be hierarchical and nested.
And you can use tuple assignments just to assign a, a comma b is, 1 comma 2, but you can also do this in a for loop using enumerate, for instance. And there are other places. So I found myself repeatedly writing the same code generation, and that was when I said to myself, I have I I must not do that. If I have an assignment in the header before loop, it's just an assignment. So what I do is I generate an assignment node on the fly. I alter my syntax tree, and I use exactly the same code generation, which lowers the the possibility of errors. So I use the syntax tree, but not as is I modify it. And, well, that's that was also a rather important, well, discovery.
And there were some things where I had to try to make a wise choice. 1 such point was truthiness. A list in Python, an empty list is is evaluated as false. But in JavaScript, an empty array is evaluated as true. Now it's very seductive to put all those bells and whistles in and convert this stuff. But, okay, what would you gain? If you don't convert automatically, you have to check for the length of an array rather than say, if array you have to say if array dot length or a len array Python. That's a minor price to pay compared to having to check anything for matching type and perform either auto conversion at runtime. So that was a point where I said, okay, truthiness, whether something is true or not, JavaScript is fundamentally different from Python, and I will not add the mechanisms to do the automatic conversion because it will make it slow and bloated. And I don't mean it to be perfect complete Python. It is it it goes quite far, but I mean it to be usable fast enough. And so that was well, I didn't like to have to make the choice, but I I had to.
There's the, there's a module called asteroid, which adds on extra functionality on top of the AST. I don't know if any of the added capabilities would be useful to you or not. I've looked into it. Yes. Right. There are quite some transpirters that use it. I forgot which ones, but it is used. No. I didn't feel the need for it. I didn't feel need for it. The AST module had enough functionality. If you look at the code generator, it's it's all in the distribution. It's the let me see. It's called compiler dot py. And you see that it's very simple and straightforward. It wasn't difficult. AST module is is powerful enough. I know there are transpilers that try to abstract away the version dependency from a certain Python version by making a kind of intermediate tree, but I didn't want that. I wanted to be very close to especially certain Python versions. So that's that's a point to consider.
You it it it is not a Python transpire. It's a Python 3.5 transpire. And as soon as 3.6 get out, you will have a 3.6 transpiler. Why? Well, because if you don't do that, you cannot use the new features. You you have to to zoom out a little bit and get a little bit vague. But, no, I use nothing on top of ASD. I took a look at quite some things there, and, I just didn't feel the need. It's it's powerful enough as is.
[00:25:03] Unknown:
Do you anticipate any problems or, excess work that comes from having to match so closely the version of transcripts to the version of Python that it's running on?
[00:25:14] Unknown:
Well, yes and no. I hope that the revolution that took place from Python 2.7 to 3.0 doesn't get repeated. I mean, I I find it very courageous of, Guido that he took that step and, well, I think the whole developer community, well, was in a shock when it hit. But I I don't expect any revolutionary changes in Python itself. So, transcript will be growing and will be in an evolutionary process. And I will not try to keep up within a week within the newest Python version. But since, well, it's a moving target, Python itself, if transcript wants to remain, well, a cutting edge product, it will have to be adapted anyhow. So I'll try to keep it in place. I don't see it as an extra burden.
The the alternative was would be, well, okay, I make something that roughly looks like Python and I keep it the way it is right now and I freeze it. Well, that's not what I want. I expect Python 3.6 to be an evolutionary step, and in some respects, transcript will take that same step and in others, it won't. It it will never be a complete Python implementation because that would be slow and bulky.
[00:26:24] Unknown:
So given the nature of JavaScript's prototypical inheritance and differences in class semantics, I imagine that adding support for a multiple inheritance in reflecting the structure of Python classes must have been pretty challenging. I wonder if you can describe the process that you undertook and how you arrived at your current solution.
[00:26:41] Unknown:
Yeah. Well, it was funny. When I started out, I had the automatic misconception that transcript inheritance should somehow have something to do with JavaScript inheritance When there's an inheritance mechanism there, so my prejudgment was if I use the JavaScript addition to add, which which you should because of performance, then I should also use the JavaScript inheritance to make transcript or Python inheritance. Well, that's not true. You don't often inherit. You don't often encounter and interpret a clause definition. So inheritance, in fact, doesn't have to be very time optimized. It has to be space optimized, but not time optimized because, well, how many classes will you have in application? 100? Well, then that means that you read 100 times to a class definition. So what I did is, at first, I started out using JavaScript inheritance. It turned out to be hard and clunky and and artificial. So then I said, well, okay. I won't use it at all. I will just try to implement Python inheritance in JavaScript. So I just copied the methods, the properties, things like that, and you you really need to have a lot of freedom freedom there because implementing properties, well, was difficult enough in itself without bothering about JavaScript's, prototypical inheritance.
So I just bypassed it, and, well, 1 good idea was at first at first, I thought a class should be an object. But since you can call a class to instantiate something, I discovered that a class must not be an object, but a function. That was 1 step. And another step was indeed to say, well, what do I need in Python for multiple inheritance? I just write it out in JavaScript, and then it turns out that it's still very compact. If you look at at the core module, it's it's called, underscore underscore core underscore underscore dot mod dot gs, and you see it's only a few k of code for everything, including multiple inheritance. So it took quite some thinking, but in the end, it turned out to be very simple. The the the biggest step was to let go of JavaScript inheritance.
You cannot you cannot use that. It just gets in your way.
[00:28:43] Unknown:
So what aspects of the Python language were most difficult to translate to JavaScript?
[00:28:48] Unknown:
Well, 2 things, I think. Yes. Well, 1 thing I couldn't translate, that's what that that was the truthiness, automatic conversion. I I think it would be too slow, so I let go of that. The other 2 things that were hard, 1 was the keyword arguments. I really have to think think very deeply, and still it is, well, it takes a lot of JavaScript code to implement that, so we had to make it switchable. You switch it on only for a function that you expect to call with keywords. That was 1 point. And the other point that was hard, to implement was was the operator offloading.
It's not hard to implement, but it's hard to do it properly without hampering your performance. Oh, yes. And, of course, 1 thing I I almost forgot because it was very early in the process, I wanted to be able to assign a method to a variable. So it should be a bound method, which means that you have to implement somehow the descriptor mechanism. Well, that really was very hard, and I think I had some sleepless nights over there. But, yeah, it worked out.
[00:29:48] Unknown:
You mentioned that you don't anticipate much difficulty in keeping up with different versions of Python. But, what about when the newest JavaScript standard, the ES 6, is more generally used and widely supported in browsers? Do you anticipate a lot of work necessary for updating going forward, I'm wondering if you anticipate, the large body of work going forward. I'm wondering if you anticipate, a large body of work necessary to maintain compatibility there.
[00:30:21] Unknown:
Yeah. Well, 1 in programming is always a lot of work. It's, it seems simple if you think about it, but if you have to implement it, it's still a lot of work. I expect it to be dual. For for 1 thing, I will not switch too early to JavaScript version 6 because indeed, as you say, it has to be available on many browsers. If you want to make websites using that, you have to be sure that your clients can understand can understand it. But I see some developments with the JavaScript, comprehensions, for instance. And, well, some other stuff that makes it attractive to make the switch, but it will be an option.
To be a command line switch or not a pragma. I don't think it will be a lot of work because, well, once JavaScript will support, comprehensions, array comprehensions, well, compiling list comprehensions will be simpler. And what you see is that JavaScript is in a in a process of keeping up with some good Python ideas, modules, things like that. And, well, my father used to say that, a camel is a horse designed by committee. I think that's happening with with JavaScript. In the end, everything will be in as is the case with c plus plus, but it will won't have the simplicity. But I I have the idea that the newer features of JavaScript are converging with Python, but what I'm afraid will never converge is the prototypical inheritance. And, well, it's totally subjective. I know that there are people who dream in JavaScript, but I can't get my head around it. I understand how it works, but it doesn't fit my way of thinking. So it it will, JavaScript and Python will converge, which will make it not too difficult to trans to transpire, to compile for newer JavaScript versions. I expect that not to be very much work. It will be an option for a long time and, well, I don't expect JavaScript to be at 1 point so so good that the Python community will switch to it because, well, the the power of Python is that it has been designed, I think, and it's guided by 1 man. And, well, that keeps it kind of unified.
And the transcript is a, a I hope it will be a community thing. I hope it will be used by many people. Currently, some of my students are writing libraries for it. NumPy is high on the wish list, and it should use the graphic card for computations. Another thing is not, but I'm afraid that will be hard to do. But what I plan to do, I will not develop everything myself, but I plan to keep my hand on the wheel as well because I noticed that people make demands for all kind of features, and it's easy to make it grow in every direction, but it should stay a bit focused. And so I won't have all to do all that work myself, I think. I hope. I hope it catches on. And but but I will try to control the direction it it moves in because I want it to stay lean on me.
[00:33:22] Unknown:
So is transcript complete and stable enough to be used in production?
[00:33:27] Unknown:
Yes and no. I use it for production, and I think I'm a lot happier with it than programming JavaScript. It's quite stable, but to be honest, it's an alpha version, and it will remain an alpha version until it's, well, really stable. So I hope people will give feedback as some people do. I I hope to I cannot test everything myself. I mean, the auto test is there, but people come up with users that I haven't thought of of. So even as it currently is, I prefer it for my own commercial projects. I prefer it above JavaScript. It is quite well tested, quite stable, but there there will be some corpses in the cupboard as well. There will be some things that are really wrong and are just never bumped into them. I hope people will report. I expect it to be in alpha for another few months, and, well, hopefully, the crooks will be out there. There are no showstoppers in right now as far as I know. I didn't write it well, I didn't write it in such a way that I first made it very clumsy and and thoughtless, and then I improve it or something like that. I like to try to do it right the first way, but, of course, it's work of a human being, so there will be there will be problems. I I haven't run into any right now. I I don't know of any bugs, but if people start testing, they will come up with things. I'm I'm convinced.
[00:34:44] Unknown:
So you were mentioning how you've had people requesting porting things like NumPy and Matplotlib into transcript, JavaScript environment. So I'm just curious if you have any thoughts on that matter.
[00:35:00] Unknown:
Yes. I have. I won't go the asm.gs way, but for NumPy, as I said, 1 of my students is currently, studying at the Delft University. He's looking into that with a friend. What I would like to do for NumPy is use the graphic card. So, you know, when you use your your g p GPU, you use your card for computations, it can be about 30, 40 times faster than even when you program it directly in c plus plus. So c plus plus, well, no. I I don't see that happening anytime soon. Because if that would be possible, if it would be possible to have efficient coding c plus plus running in the browser, transcript would be not necessary at all. We would just run c Python. But I I don't see that happening anytime soon, but I think that using the graphic card is 1 way to go using asm.js is another way to have efficiency and maybe even run some c code where necessary. But, well, I accept the fact that the browser world is dominated by JavaScript. So even suppose you use asm.gs, you can probably make things that are very efficient, but they are also very unreadable. And people wanna debug and they wanna understand their code. I don't think there will be much c code. I think there will be fast code probably on the graphic card, but not much no no no c code on top of asm.jsi.
I don't expect that to have much advantages.
[00:36:13] Unknown:
So are there any questions that I didn't ask that you think I should have or anything else that you wanna bring up before we move on?
[00:36:18] Unknown:
Well, I think your questions were quite good and, we had a good view on, what transcript is, but, I have a question. Sure. I think, technically, this is rather good than anyone just for itself. I won't run and I have re had a request for benchmarks, but I won't do that because it's so easy to to skew them, to use a piece of code, transcript, is is very good at. I also don't want to say this is the 1 compiler to end it all and to replace other compilers. But what I would like is for people to take a look at it, judge it on its merits, and give feedback because there are a lot of, people starring on GitHub and there are a lot of downloads on PyPI, but I I get too little feedback. I I need, well, input, to be able to test it and what it needs most, and I have no idea how to do it. I hope this interview helps and, well, thank you for that. But what it needs is mindshare. If people start to develop serious things in it and they start to invest money in making applications with this, it must not go away. Well, what's important just in that respect, I don't see it as my property. I see it as my baby. That's true. But, well, my children are also not my property and, transcript isn't isn't. I will keep a hand on the wheel, but this is open source. It won't go away. It can't go away. You can anyone can fork it. If you look at the source, it's simple. So somehow, I don't know how to make that happen, but somehow I hope it gains mindshare and people can interest in it. I've been a developer for ever since I was 15. I'm 57 now, so you can know how long that is. I still service software that I made 30 years ago, still used, and it's still still okay. So what I would like to say is this is really serious. It's not going away. It's not a hobby project. I will try to to involve more people, and it's open source. Well, I hope people will use it. Yeah. I think that's very reassuring hearing you say how it's not a hobby project and it's something that you consider a serious piece of work because I know that a number of the other Python to JavaScript transpilation libraries are seen more of from an experimental perspective and not really intended necessarily for production everyday use. So I definitely think that that's another point where transcript kinda stands out from the crowd. It is for production work. I I program for a living and I I programming medical imaging. That that was my problem. And I wanted to run stuff in the browser and I couldn't in a way that was acceptable for me. So it's for me, it's very serious. It's part of a multi year pro project that many people are working on. And, well, I just thought this is usable. More people can use it. But it indeed, it is very serious.
[00:38:48] Unknown:
Yes. So for anybody who wants to follow you and get in touch with you and, keep up to date with the things you're working on, what would be the best way for them to do that? There are some, some ways to do that. There is the transcript site,
[00:39:01] Unknown:
www.transcript.org. There's a forum where it's still very quiet because who wants to be the first? There's a forum and you can reach me there. There on PyPI, there's my email address, you can reach me there. And, well, I like to have feedback, so feel free to to send an email or to answer the forum. I will react. I will will not I will not blindly implement any feature that's re requested because then it would be bloated. But approach me by email, for instance, or on the forum, and, I will certainly react and, well, that that's the way to to make contact about what your wishes are. And I invite you to write libraries for this because I will not do that myself. I will try to make the core as good as possible, but I hope people will start writing libraries. Okay. So with that, I will move us on to the picks.
[00:39:49] Unknown:
And for my first pick today, I'm gonna choose the cookie cutter library. I've, just started doing some more work in SaltStack for my job, and I ended up writing a cookie cutter template for being able to easily generate the directory layout and, some, initial files for SaltStack formulas. So it definitely is saving a lot of time in that respect and, making sure that anybody who
[00:40:17] Unknown:
is leveraging that in inside my team is, starting from the same point. So definitely very useful for that. And I'm just gonna leave it at that today. So, Jacques, what do you have for your picks? I have something, and it's meant especially for the Dutch audience, but also for a broader audience. I've been programmed all my life, and I learn something every day from our students. And I think, well, there's a lot more to programming than is taught in educational institutions. So I'd like to invite young people to dive into this and will not be shied away by all the the jive talk around it and and all the design methods and things like that. Programming in itself, I've done it all my life, it is fun, it is rewarding, you can earn a living with with it, and, well, it's I know it isn't a popular thing to say, among among managers, but it's really an art. I like my code to be beautiful and I programming's fun.
[00:41:12] Unknown:
So I just wanna invite people to to be a programmer and to to take, to take pride in your job and try to do it properly. Well, Zach, I definitely appreciate you taking the time to, take the time out of your day to tell me and my audience more about transcript. I'm sure that there are a number of people who are very interested to test it out. And hopefully hopefully, you can get some feedback from that. So, thank you again, and I hope you enjoy the rest of your evening. Oh, thank you very much for giving me the opportunity to expose this to the public, and to enjoy your day. Bye. Alright. Thank you. Bye.
Introduction and Announcements
Interview with Jacques de Hoog
Jacques' Introduction to Python
The Transcript Project: Overview and Inspiration
Challenges and Solutions in Transcript Development
Performance and Optimization
Future of Transcript and Python Compatibility
JavaScript Evolution and Transcript
Transcript's Stability and Production Readiness
Community Involvement and Future Development
Picks and Closing Remarks
