Summary
One of the biggest issues facing us is the availability of sustainable energy sources. As individuals and energy consumers it is often difficult to understand how we can make informed choices about energy use to reduce our impact on the environment. Electricity Map is a project that provides up to date and historical information about the balance of how the energy we are using is being produced. In this episode Olivier Corradi discusses his motivation for creating Electricity Map, how it is built, and his goals for the project and his other work at Tomorrow Co.
Preface
- Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
- When you’re ready to launch your next app you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 200Gbit network, all controlled by a brand new API you’ve got everything you need to scale up. Go to podcastinit.com/linode to get a $20 credit and launch a new server in under a minute.
- Finding a bug in production is never a fun experience, especially when your users find it first. Airbrake error monitoring ensures that you will always be the first to know so you can deploy a fix before anyone is impacted. With open source agents for Python 2 and 3 it’s easy to get started, and the automatic aggregations, contextual information, and deployment tracking ensure that you don’t waste time pinpointing what went wrong. Go to podcastinit.com/airbrake today to sign up and get your first 30 days free, and 50% off 3 months of the Startup plan.
- To get worry-free releases download GoCD, the open source continous delivery server built by Thoughworks. You can use their pipeline modeling and value stream map to build, control and monitor every step from commit to deployment in one place. And with their new Kubernetes integration it’s even easier to deploy and scale your build agents. Go to podcastinit.com/gocd to learn more about their professional support services and enterprise add-ons.
- Visit the site to subscribe to the show, sign up for the newsletter, and read the show notes. And if you have any questions, comments, or suggestions I would love to hear them. You can reach me on Twitter at @Podcast__init__ or email hosts@podcastinit.com)
- Your host as usual is Tobias Macey and today I’m interviewing Olivier Corradi about Electricity Map and using Python to analyze data of global power generation
Interview
- Introductions
- How did you get introduced to Python?
- What was your motivation for creating Electricity Map?
- How can an average person use or benefit from the information that is available in the map?
- What sources are you using to gather the information about how electricity is generated and distributed in various geographic regions?
- Is there any standard format in which this data is produced?
- What are the biggest difficulties associated with collecting and consuming this data?
- How much confidence do you have in the accuracy of the data sources?
- Is there any penalty for misrepresenting the fuel consumption or waste generation for a given plant?
- Can you describe the architecture of the system and how it has evolved?
- What are some of the most interesting uses of the data in your database and API that you are aware of?
- How do you measure the impact or effectiveness of the information that you provide through the different interfaces to the data that you have aggregated?
- How have you built a community around the project?
- How has the community helped in building and growing Electricity Map?
- What are some of the most unexpected things that you have learned in the process of building Electricity Map?
- What are your plans for the future of Electricity Map?
Keep In Touch
Picks
- Tobias
- Olivier
Links
- Electricity Map
- Machine Learning
- Youtube
- Climate Change
- Fossil Fuels
- Carbon Intensity
- Greenhouse Gas Equivalencies Calculations
- Open Data
- Electricity Map Project Source
- Lignite
- Marginal Carbon Intensity
- Electricity Map Forecast API
- IPCC (Intergovernmental Panel on Climate Change
- Redis
- D3.js
- Spark
- Tensorflow
- Spatiotemporal Data
- MongoDB
- Matrix Inversion
- PyGRIB
- Tomorrow Co.
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
[00:00:14]
Unknown:
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. When you're ready to launch your next app, you'll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 200 gigabit network, all controlled by a brand new API, you've got everything you need to scale. Go to podcastinit.com/linode to get a $20 credit and launch a new server in under a minute. Finding a bug in production is never a fun experience, especially when your users find it first. Air brake error monitoring ensures that you'll always be the first to know so you can deploy a fix before anyone is impacted.
With open source agents for Python 23, it's easy to get started, and the automatic aggregations, contextual information, and deployment tracking ensure that you don't waste time pinpointing what went wrong. Go to podcastinnit.com/airbreak today to sign up and get your 1st 30 days free and 50% off 3 months of the start up plan. To get worry free releases, download Go CD, the open source continuous delivery server built by Thoughtworks. You can use their pipeline modeling and value stream app to build, control, and monitor every step from commit to deployment in 1 place. And with our new Kubernetes integration, it's even easier to deploy and scale your build agents.
Go to podcastinit.com/gocd to learn more about their professional support services and enterprise add ons. And visit the site at podcastinit.com to subscribe to the show, sign up for the newsletter, and read the show notes. Your host as usual is Tobias Macy. And today I'm interviewing Olivier Caradi about electricity map and using Python to analyze data of global power generation.
[00:01:49] Unknown:
And Olivier, could you start by introducing yourself? Yes, of course. Thank you for inviting me here. So I'm Olivier Corady. I'm a French and Danish data scientist and machine learning expert. And I'm basically born in data before it was actually called data science, but I've always enjoyed since little
[00:02:10] Unknown:
extract knowledge based from it. And do you remember how you first got introduced to Python?
[00:02:15] Unknown:
It's, it's tough to remember exactly, but I think it was, when it was in high school and trying to build this home project that would download automatically videos from YouTube because I had very, very bad connection. And on the train, I wanted to be able to look at those videos. And so what I did is, like, a cron job that would use Python to download those videos and store them locally, transfer them to my phone, and then I could be able to see them
[00:02:42] Unknown:
on my phone. And as I mentioned, we're talking about the work that you've done on the electricity map website and API and database. And I'm wondering if you can talk about your motivation for creating that project in the first place.
[00:02:56] Unknown:
Yes, of course. So so approximately 2 years ago, it was April, I was wondering a little bit about the situation of of climate change and what we could do about it. And I stumbled upon the fact that all of our energy basically comes from fossil fuels, which is very bad. It's 80% of it. And then realized if we were if if we are to live in a fully electric world where we don't use fossil fuels it really matters how electricity is produced And as still a lot of electricity is produced by fossil fuels, coal, and gas, it's 2 thirds approximately. I figured, well, we really need to get this knowledge out to people, make something very intuitive that shows in real time what is the wind turbines producing. Are we able to turn off the coal in the gas power plants?
And how do we actually make an energy system or an electricity system that produces electricity even when you don't have the wind blowing and the sun shining? And this is really the motivation behind the project.
[00:03:55] Unknown:
And for anybody who hasn't visited the site, if you go to it, you see a map of the world with various different political regions colored according to the overall, carbon production been happening at a given point in time based on the electricity plants and how they're generating their power. And I'm wondering if you can describe how an average person who is visiting the site or using the APIs that power it can benefit from the information that you are making available through those means?
[00:04:29] Unknown:
Yes. In a in a in a very basic way, what we try to showcase is the so called carbon intensity, and it has a weird unit. It's called grams of CO2 equivalent per kilowatt hour. But what it basically means is that it tells you how much greenhouse gas or CO2 is emitted for every unit of energy that you will consume. Let's say that for example, you put your electric vehicle to charge, We want to give you a signal that tells you this is a good time or this is a bad time. And so the map basically is colored based on this metric that tells you is now a good time to consume or is now a bad time to consume. And so it takes a lot of things into considerations in order to calculate this. It looks at how electricity is produced locally, but it also looks at how it is imported, where it's imported from, and how that country that is currently exporting power to you is generating its electricity. An example I often take is Denmark which has a lot of wind power. On a day where it doesn't have a lot of wind power it will import power from Germany or from Sweden or from Norway And there's a significant difference if you import from Norway that has a lot of hydro electricity, which is low carbon, compared to if you import from Germany on the on a day when there's no winds, and no sun where it will be mostly cold generated. So we really try to tell something very intuitive. We tried to boil down a very complicated number to simply a color on the map, and I think this is why it caught people's interest. It's that it's succeeded in boiling down something quite complex to simply a color.
[00:06:08] Unknown:
And the color for the region on the map factors in those transfers. So for instance, as you mentioned, if you're transferring energy in from a grid that's producing energy via a lot of fossil fuels, it will actually influence the way that the visual representation is displayed for the region that you happen to be in regardless of how your local power generation is happening. Is that accurate?
[00:06:33] Unknown:
Yeah, that is correct. And this is also why the we've we've drawn the interconnectors or the exchanges of electricity between the countries as small arrows that give you a direction of the flow. And the arrows are also colored by actually the carbon content or the carbon intensity of the electricity that transits. So it makes it very simple to understand if you have a country that is darker because it's it's less it emits a lot of carbon for every energy that's consumed. You can actually immediately see if it imports power that is greening it or making it worse based on just looking at the color of the of the small arrows on the map.
[00:07:12] Unknown:
And in order to be able to provide all this information, you need to gather it from various sources. And I'm curious what what those sources are that you're using to be able to gather the information and aggregate it for how the electricity is being generated and distributed in this near real time fashion?
[00:07:33] Unknown:
Yeah. So when we started out the project, I had only 3 data sources. So the data sources I had at the time were the so called transmission system operators in France, Germany, and Denmark and it's a nationally, owned entity publicly owned entity, that is responsible for controlling the electricity, balancing in the system, basically making sure that there's no blackouts. And so because they're publicly owned, they have an incentive in publishing open data about how electricity is generated in real time in order to both inform the citizens but also in order to make sure that the best investments are taking in the grid. So they publish a lot of open data and we've been connecting with those real time open data feeds, integrating it into the Python back end, and then visualizing it on the map itself. And an interesting thought that happened after this is how can we make this scalable if we wanted to get all countries in the world to be painted with the proper color and this is the reason why we made the project open source. So if I can say a few things about that is that we have an open source repository where we have a list of Python scripts that we call parsers, which will connect to each of those entities that are publishing open data.
And this script basically returns in a standardized format that we defined ourselves. What is the production mix? So what is what units are producing what in a given area, and what are the exchanges with the neighbors? And so anyone that can produce such a script that knows about the location of data can write such a script. We're taking it to our back end and then it automatically colors the color on the map and this is the reason why we've been able to scale up and have so many countries is through all of the contributors that come and saw the project said I actually know where this data and I know a little bit of Python so I can write a script, contributed to it.
And now I think we started with 3 countries and now I think we have over 100 areas, covered. And I say areas because there's actually some countries that we've broken down by state. For example, India is broken down by state. Australia is also broken down by states, and we're in the process of doing the same thing with the US.
[00:09:55] Unknown:
And I imagine that because of the number of different sources that you're aggregating from that there's going to be a disparity in the robustness and the format and the accuracy of the data that's being provided. So I'm wondering how you've managed to unify that information for being able to display it in a consistent fashion.
[00:10:18] Unknown:
Yes. So there's there's a few issues with data quality. There's first of all, can you trust the data? Is it accurate enough? Because sometimes we have, for example, data providers reporting twice as much wind produced as it actually is produced. We have issues where there's some parts of the generation mix, which is missing. For example, countries would say right now there's no coal being produced, even though there is some, we just don't know exactly how much. And this would, for example, turn the map into a greener color than it's really supposed to be because we're having missing data. So for all of those things, we have over the months created a list of checks.
So we know that this country is supposed to produce at least this amount of that particular generation. And so we constructed sort of a list of different checks that we go through. So we know is the data valid or isn't it valid. So this is the first thing we do. The second thing is that we have sometimes data that is delayed. So for example, data provider will sometimes provide data about the current system, but it will only provide it, 5 hours from now. So the because the full system is completely interconnected in the sense that if you have just 1 data item that changes, for example, in Spain, it will affect how the electricity consumed in Denmark would look like even by a small amount. We have to make sure that all the data we have on the map is somehow representative of the same point in time. In order to do that, the the simple, solution we came up with is just saying if a data item is older than 2 hours, then we simply discard it, and else we just take the last 1 that is within those 2 hours. So if you look at the electricity map, you know that all the data that is that you're looking at is fresh by at least 2 hours, else, we remove it. And I noticed what as I was exploring the map that there are cases where the source of the energy production Yes. So so, philosophically, what we've decided to do when we're dealing with, the different data providers is that we really wanted to stay as close as possible to the data source. And we have a specific breakdown of how generation units are classified. We have for so if you take coal, for example, there's both lignite mines and different types of coal power plants. We've just decided to have 1 single entity to represent all of those for simplicity because we thought it reduces the complexity and simplifies the message we want to carry.
However, the data source providers, they don't necessarily give us directly the the call unit. They will give us the breakdown. So we have to have some rules into how we aggregate the different units that the data provider will give us. And sometimes, those rules are not, very easy to understand. Sometimes, you have units that will both produce biomass using biomass, using gas, and using coal. So how do you classify those? There's all sorts of edge cases. And we've decided to make the least amount of assumptions as possible. And so every time we have a unit where we're not completely sure about it, we put it in unknown, and this has a dual effect. The second effect is also that it actually motivates the data providers to open up more data and more accurate data as they come to the map and actually see unknown. Or if visitors come to the map and see unknown, and then they will start talking to data provider and say, why don't we have better data here? And this pushes up different data providers to open up more precise data.
[00:14:03] Unknown:
And beyond the, visual display of the map, you also have APIs and the actual actual database available for use. And do you maintain the, higher granularity of the data
[00:14:17] Unknown:
in those sources for people to be able to provide their own analyses as well? Yes. So so another service we're trying to do, which is a commercial service on the other hand, so the map is completely free. Everyone can access it. Part of it is open source. But we have a commercial service behind the scenes where we enable people to look at highly granular data that we compute based on everything that we have. So there's all sorts of very, interesting AI that we can do on on this data that we can talk about, if you wish.
And because it starts being technical, but the power industry is very interested in understanding exactly what happens if you add a little bit of extra wind power in a certain country. Will that decarbonize it effectively or you know won't it change everything because anyway there's so many wind turbines already that it doesn't really affect the system. And those are the kind of analysis that we would do and that we would provide through different APIs. And so the 1 I just mentioned is is called the marginal carbon intensity. Something else we also do with the APIs is that we do predictions. So we do predictions 24 hours in advance of how the whole electricity map would look like and this is useful if you, for example, want to plan ahead and you want to use electricity at the right time. So let's say that you have an electric vehicle, you come home, you plug it in, in the evening and then it has the whole night to charge. Well, the color on the map will change during the night depending on lots of various factors. For example, the wind can start blowing from 1 hour to the other. And so what you really want is you want your electric vehicle to automatically fetch the forecast and then say, I know how much I need to be charged by tomorrow. But if I just waited 1 extra hour I know I will be fully charged by tomorrow morning but I can still optimize and reduce the carbon, that it will be emitted in in the atmosphere due to my electricity consumption. And those are also projects that we're doing. We're doing this with electric vehicles and we're doing it also with heating systems.
And so this is all part of the API or the forecast API that we're providing. We really imagine a world where any IOT device that is consuming electricity and is connected to the Internet would respond to this forecast
[00:16:28] Unknown:
in order to, you know, consume at the right time. And another factor that can complicate the balance of power generation in an electrical grid is things like residential wind and solar, where if an individual is producing more than they're able to consume in some areas, they're able to feed that back into the grid and sell it back to their provider. Is something like that factored into the data that gets reported to electricity map?
[00:16:57] Unknown:
So that depends. And that's a that's a very good question. So so there's 2 types of decentralized generation you can have. You can have something that's called behind the meter. And so this basically means that if you have a solar panel on top of your house, instead of consuming grid electricity, you can just cut the cord so to speak for a couple of hours and then use your own electricity that you might for example have stored in a battery. This is behind the meter behind the electricity meter that your electricity distributor might have access to, and so we have no way of really accessing that data. But for now, it's such a small portion of of people that have such system installed that it doesn't really change the big picture, but this is a blind spot for us. The second thing is the thing you mentioned where you have so much surplus of local generation that you're actually giving back to the system, and so the meter starts running backwards you might say, and this data depending on the different data providers we have an idea of how much is getting fed back to the grid But again, it's so small amounts compared to the very, very big industrial scale large production that we have that it basically doesn't change anything for now.
[00:18:07] Unknown:
And going back to the idea of the accuracy of the data and how it's represented, Are there any penalties for any power plants or anything like that that misrepresents the types of fuel that they're consuming or the amount of waste that's being generated by any given plant?
[00:18:24] Unknown:
So so there's, we have to distinguish different areas of focus here because if you look at, for example, Europe, there's a European directive that stipulates that data should be opened up, and that this data should be accurate up to certain specifications. However, there's no to my knowledge there's no financial penalty that is that is happening there if you don't do what is supposed to be done. We have countries like, for example, the Netherlands which are not supplying this data at all and there's a lot of regulators that have, you know, told them they have to do it and so on but it's still not happening. So the only thing we can do is actually put them pressure but there's no financial penalties happening if they don't report things correctly.
However, you have some I know I have knowledge of some countries also that are being watched very closely by regulators and the regulators are assessing those countries through this open data. And so, if the open data is not accurate enough then the regulators come to their job and there they might get some penalties directly. But the end game is really this. There's no, there's no real painful penalty for not publishing the data. So it poses the problem of can we actually trust the data that comes out of it and because it's publicly owned entities that are publishing data, in principle it's like you have you're trusting public entity like the states. So in principle, the level of trust should be quite high, but we we notice some discrepancy sometimes, and we try to factor them and be transparent about them.
[00:19:58] Unknown:
And in terms of the overall impact of the electricity generation on the carbon output of global society, do you have any sense of the percentage that is attributable to electric consumption and electric generation as opposed to other sources such as vehicle exhausts or, any other forms of manufacturing?
[00:20:23] Unknown:
Yes. If I there's 1 number I have in mind is that coal power plants in the world, I think, are responsible for between 15 20% of global greenhouse gas emissions, which is huge. And coal is mostly used for electricity in the world. So that gives you sort of an order of magnitude where transportation, for example, I think is only a couple of percent of global of greenhouse gas emissions. Those numbers should be taken with a grain of salt. I can just double confirm this is just by memory. I can attach a couple of links, afterwards just to be sure. Okay.
[00:20:57] Unknown:
And 1 other thing that's worth calling out is that just now you used the broader term of greenhouse gas and earlier you mentioned that the units of measure that you're using for displaying in the electricity map is the grams of c 02 equivalent per kilowatt hour because, as some people may know, it's not just carbon dioxide that has an impact on global climate change. It's other things such as methane or other emissions. And so I'm wondering if you can just briefly speak to the calculations that you perform to be able to unify the emissions output into that single metric?
[00:21:37] Unknown:
Yes. Of course. So if you take methane, for example, methan methane is 10 to 20 times more potent in terms of how much warming it will cause over the next 100 years compared to co2 for an equal weight. Hence, what is usually done in the industry when you do carbon accounting you want to assess the climate impact is that you say, I take my amount of methane, I look at how much it's going to warm the atmosphere in the next 100 years, and I convert it into an equivalent mass of co2. And so by doing this, you can take any greenhouse gas and say, converting all the effects it will have on the warming and converting that to an equivalent mass of co2, and that gives you a unit called co2 equivalent. So now we we don't do ourselves the the c o 2 calculations, in the sense that we don't look at each power plant, how much methane it would leak, and how much green, how much CO2 would emit because that would make us highly vulnerable to trust issues.
Now the the the UN has funded something called the IPCC, which is a globally renowned institute that looks at meta analysis. So it looks at many, many different scientific studies, aggregates them, and then is able to say, okay. Typically, a coal power plant in the world emits that much greenhouse gas, so c 2 equivalent per unit of electricity produced. And we use those numbers, and we then factor in our analysis. Then we trace the origin of electricity, and then we can color the map based on this cu2 number. We have a whole section I can put a link to that on our website that explains a little bit the climate perspective, how c o 2 equivalent is calculated, how do IPCC numbers come from, and how this all relates together.
I'll put a link in the in the description afterwards if you're keen to have that.
[00:23:28] Unknown:
And going deeper now in into the technical architecture of how this all operates, I'm wondering if you can give an overview of the way that the data flows from the gathering point from of these different open data providers through to the, analysis and representation in the front end?
[00:23:50] Unknown:
Yes. Definitely. So so what we have is we have 2 components. We have 1 that we call defeater. It basically is a Python script that runs every every minute. It takes all of the different parsers that we have, so all the scripts that will connect to open data to to get the information, and it runs this every minute, gathers all the data, and puts it in a database. And then we have another system that also runs, every minute as soon as the previous system is finished and you have collected all the data put in the database, we have this other system that is responsible for querying the last 2 hours of data that we have access to, computing all the import exports in order to figure out where does electricity come from in a given zone, then it applies all the c o 2 equivalent, what we call the carbon intensity factors, of each of the fuel sources, and then it it comes up with a list of countries with their associated carbon intensities.
And all of this is basically put in a in a Redis cache so that's such that it's ready to be consumed by the front end such that we don't have to re query all of the database every time a visitor comes to the website. And so the front end we can speak a little bit about how the front end is is designed also if if you'd like, but it basically just calls an API that's just just serving the the cache. And in that way, we have a very efficient performance system that is always up to date. And for
[00:25:19] Unknown:
the, web architecture of the application, are you using a framework such as Flask or Django?
[00:25:25] Unknown:
Not even. We, we basically have on the so the front end is served by a note back end, that just exposes an API with some JSON that comes from the Redis cache, which is then populated by this Python script that runs every minute. And on the front end, we're using, a lot of d 3 in order to on the front end side, manipulate all the different objects, but we don't use any Django framework or something else. It's exclusively JavaScript based As soon as you go out of the of the cache and you're running into the just the front end,
[00:25:59] Unknown:
everything is JavaScript. Yeah. That makes sense for simplifying the delivery of the application so that you can, you know, bundle it all up and use Python where it is powerful in terms of being able to consume and process and analyze the data. And on the website, I noticed that you have an architecture diagram where it looks like you're also using TensorFlow and Spark for being able to do some of the heavier processing. I'm assuming that that is part of what's powering the paid API that you mentioned of the forecasting.
[00:26:31] Unknown:
Yes. That's correct. So we have to we have to basically be able to stream all this data in real time and make sense of it in order to run our forecasting models on top of it. And because the forecasting models are basically spatiotemporal so they will take into account data in the past and if you want to forecast a specific country you need to take into account how its neighbories countries are performing, how the imports exports are happening, and so this creates a special temporal dependency, a sort of special temporal graph, if you will, in the past that we have to process in real time. And this is why we use stuff like TensorFlow and Spark because it can start being heavy in terms of the amount of data that we're processing and the amount of calculations we're doing. And how has the overall system architecture evolved over the course of time that you've been working on the project? It's actually pretty funny because in the beginning, it just had to work in this fastest way possible with the simplest setup. So we simply had a a MongoDB database that would store all of this data, and every time a visitor would come to the website, it would query that database.
And so it was a pretty heavy calculation that would be done. And even the the whole flow tracing, figuring out where electricity comes from based on all the imports and the export, all of this was coded in the front end directly. So this algorithm requires a small matrix inversion in order to figure out all the couple flows, and it would have numerical libraries that would actually be in the front end, written in JavaScript just because it was faster to, you know, debug it and iterate. And as time moved on and when this project got a bit of traction, at some point, we were on, on the front page of Hacker News for 2 days. And you can imagine that if you have those 10, 000 visitors per hour, the MongoDB back end was just, you know, blowing up. And this is where we transformed the architecture and rewrote it during a couple of nights where it just had this caching system, and we also put the matrix inversion system in the back end such that we have a, you know, faster computations on everything. Something that also evolved quite significantly that is that on on the website, you can also see the current weather patterns. So we're showcasing how much solar energy is currently hitting the ground, over the world, and we're also showing how much the wind is blowing all over the world through a small toggle that you can activate and deactivate. And this is actually quite a lot of data to be processing on the on the server. We're storing historical weather data for 2 years and we're talking terabytes of data. And so we've been making quite a lot of changes also in in the way we handle those weather forecasts and we do everything in python through something that's called pygrib, which is a small python library that's very handy to manipulate meteorological data. So, yeah, the infrastructure has been involved quite heavily, and then we've built all the analytics back end on top of that, which is also another story.
[00:29:21] Unknown:
And what are some of the most interesting or unexpected uses of the information that you're providing through the map or for through your API that you're aware of? So there's a few interesting uses
[00:29:33] Unknown:
of the of the map itself. I was quite happy to see that it's being used in universities and in primary school and higher in in secondary school also, by teachers in order to show students and get them a bit of intuition about how the whole electricity system work. And I think this is really something that I'm very very happy about because this was the whole point of the project is to get more people to play around with this data and and be educated by it. Now 1 of the unforeseen uses of the electricity map is for lobbying because I didn't realize how much nuclear lobbyists are interested in this kind of map. Because we're focusing on on c o 2 emissions or greenhouse gas emissions.
What you end up with is having nuclear powered countries. Nuclear is low carbon and so you end up having France, for example, being green on the map because it's mostly powered by nuclear. And so the nuclear lobbying are they are very, very interested in showcasing the map everywhere. And so this map has been used in a lot of very very aggressive debates on should we actually decarbonize the system using wind and solar, should we do it with nuclear instead. It's been very challenging for us to remain completely neutral on this issue because every time we'd showcase a particularly interesting situation where, for example, Germany was producing a lot of wind power and we said look Germany is green because it's windy right now, how cold is that? Then you would have the whole nuclear lobby just jumping on us and saying, well, but, if they just did like France, which would have low nuclear power low carbon nuclear power all the time, it would be much better. And so an unexpected use for us for the data is is really how those 2 sides have been fighting against each other's and using the map as sort of arguments
[00:31:20] Unknown:
data driven arguments. Yeah. It's a good example of the fact that data is neutral, and it can be used to argue whatever point you want depending on how you decide to interpret it.
[00:31:30] Unknown:
Exactly. And I think this is also the philosophy behind the company that we created based on this electricity map because we believe that if you want to find scalable solutions to climate change what you need is you need the whole society to have a more objective data driven view of what are the things that I do in my daily life that are impacting the climate. What is something that actually matters and what is something that is completely useless because it's only of magnitude less that something else that we do. And this is where data comes to the rescue. And I I strongly believe that data is the way forward to make better decisions. And so we we really want to do more than just electricity if we could,
[00:32:12] Unknown:
later on. And have you found any ways to measure the impact or effectiveness of the information that you're providing through the electricity map and the APIs?
[00:32:23] Unknown:
So so I think there's 2 types of impact. We can look at it in terms of audience. How many people actually look at this, find it interesting, share it on social media, and so on. And the kind of the the metrics I have in mind is that now even 2 years after, we still have approximately 3, 000 unique users coming to the map every day, to look at it, tweet about it, and so on. And we've never done any marketing, so it shows that there is some kind of interest, and there is some kind of impact in the sense that teachers are using it in university and so on. If you think about quantitatively measuring the, you know, amount of greenhouse gas we'll be able to reduce using such a tool, it starts being a bit more complicated. But for example, with the projects we're doing with electric vehicles, we measured that if an electric vehicle is using our system, our API in order to charge at the right time during the night, you can save approximately 20% greenhouse gas emissions every night based on a fully electric vehicle charge. So it starts to be coming interesting if you multiply those initiatives to, you know, the heating in your home if it's electric and you start going to industrial scale like for example supermarkets they have a lot of cooling electricity they use. They could also use it at the right time. So I I think we're going to have bigger and easily quantifiable impact in the future but right now it's still a bit early to say. And at a broader scale,
[00:33:52] Unknown:
the ways that this information can be used is potentially in terms of building a newer or more intelligent grid for determining how the electricity is distributed within the grid based on how it's being produced at any given point in time. So I don't know if you've put any thought into how that can take place or some or working with any of the power companies for being able to incorporate the information that you're providing? Yeah. So we work a lot with the different,
[00:34:22] Unknown:
energy and electricity distribution companies, in in various countries in order to do exactly that. So there's sort of 2 timescales that are relevant here. You can look at the short timescale where we're talking hours to days. You know, the electricity system is fixed. There's not so much stuff that you can change. So the only thing you can do as a consumer or as a company is that you can decide when to use electricity at the right time. This is really the only thing you can do because you cannot say you cannot control electrons. You cannot say, please go to that company instead of the other 1. So that you cannot do. And then if you look at timescales that are a bit longer, for example, months to years then it starts becoming interesting because thanks to the data that we have we can start saying, you know, should you invest in putting a wind turbine in this country or should you invest in augmenting the exchange capacity between 2 countries such that the flow of electricity can be bigger between the both of them because it will decarbonize much more efficiently the 1 that's importing, for example. And those are the kind of arbitrages we are able to do things to all the data, and we're working very closely to with different electricity, companies in order to do exactly that. But it raises a more fundamental question, which is how do you actually measure where electricity comes from, and how do you measure the benefits of, you know, taking different decisions? And there's there's many different answers to that, unfortunately. We're looking at something very physical. We're saying, if you're connected to a system, then you inherit all the electrons that come to you, and you cannot choose exactly where they come from. You cannot choose to, for example, consume directly from the wind turbine that's on the other side of the country. But, however, you see more and more of those 100% renewable contracts that are happening in electricity suppliers or you see mobility providers like electric vehicles that will tell you well this is 100% green electricity because we signed a contract with someone that enables us to tell that we're 100% green. Unfortunately, the problem with those things is that that it's it's not physical. So you even have a discrepancy in how to actually agree on where electricity comes from And this is also a place where we're trying to do some lobbying, trying to educate people on what are the different ways of accounting and what is the way that is the most efficient to decarbonize the whole system the fastest possible. And we definitely believe it's the it's taking a physical approach that will make the most sense in the longer term. And
[00:36:46] Unknown:
going back to the community reaction to the project that you've built, you mentioned earlier that a lot of the data sources that you've been able to add over the past few months and the couple of years that you've been working on it is because of community contributions of people different data sources and wrote the parsers for consuming it and providing it to the project as a whole. I'm wondering if you can talk to how you built that community around the project and how that community has helped in building and growing the electricity map project. So the way we engineered it,
[00:37:24] Unknown:
so to speak, in order to foster a community around the project is that we started being open about everything that we did, and we wanted to make it very very visible to any visitor that came to the map that if he had knowledge of any data that could help us that he should know exactly where to, you know, communicate that knowledge. And so 1 of the concrete things we did is that on the electricity map, you'll see on the on the bottom left side a big slack button, and so if you click on it you immediately get invited to our slack channel where you can join, you can talk to us, and you can immediately let let us know if you know anything, if you have ideas on how to improve the map. And through this Slack channel, we've really been able to foster a whole community around the project where, you know, we post links, we discuss another topics, mostly people interested in data science and climate change in general. The second thing that we also did is that we made it very explicit that if you we wrote basically on the website directly that if you knew anything about data or if you have any ideas on how to improve the system, you should click here and that link would directly take to GitHub issues.
And I've actually been very surprised at how many non technical people have ended up coming on GitHub and opening issues, which I find a bit fascinating because GitHub is not the most user friendly platform wherein you're not a developer. So we have roughly, you know, half of the contributors that are non technical people that don't code, that have been submitting pull requests where they would just, you know, change some numbers in a JSON file because just by pattern matching they knew which numbers to change and then they would pop up on the electricity map, carbon intensity factors for example, and the rest are people that would be coding the parsers that maybe the other half, the non technical people, would have identified, the data sources they would identify, then the the coders would take them and then and then implement them. And through the course of time, I think we've had 50 different contributors doing pull requests on the on the website. And we have, I think, 2 to 300 registered Slack users. And in our day to day operations, we have like a small core group of 4 to 5 contributors that are very active in helping us add new countries, find them correct data that's incorrect, and so on. And for my sake, I think it's been almost a year since I actually coded, myself a parser that would get some data and put it in the system, I've been mostly focused on on making the UI and the UX of the front end robust, fast, and usable by anyone.
For example, building a mobile app has been something that's been taking a lot of time and the community has been very helpful also in order to give us feedback, early testing, on everything that we did.
[00:40:13] Unknown:
And in the process of building any substantial project, there are usually a number of things that you end up having to tackle that you didn't foresee at the outset. So I'm wondering if you can speak to some of the pieces of information or challenges that you have come across that were most unexpected that you ended up learning in the process of building the
[00:40:39] Unknown:
angle you angle you take. How easy or complicated it would be to actually foster a community. And that's something we really didn't foresee at all. How much time you need to spend in order to, you know, communicate all the knowledge that you have to everyone such that they can help. So at some point you start being dedicated to help people and let them be empowered by by what you're doing because that's the only way that it really gets, you know, involved in the project. So this is this is this is 1 thing that was completely unexpected. The second thing that's completely expected is really, the challenge in the amount of details you need to go through in order to present something that is trustworthy in terms of carbon intensity. So basically, there's so many level of details you can start drilling through if you want to accurately show data. 1 of the examples I can take is the real time data that we get is not it's accurate up to a certain level. So the data providers, they're measuring and monitoring the system to a certain amount, but then they will get extra information as time goes on. For example, 2 weeks later, they will get extra information about some units that didn't report accurately and so they will be able to correct that data. And 1 month after, they have even more accurate data.
And talking to some of those data providers, we realized that the the final data, the 1 they use for billing purposes, it can even be 3 years old. So it means that 3 years after, you can still have corrections on the data you have. Granted you're you're correcting very very small numbers and the real time data is is sufficient for our purposes. But as so depending on who you speak to exactly and the level of details they want to or the level of precision they want on on both the electricity side or the carbon side, you end up going in discussions that are very very detail oriented about exactly how wind turbine, you know, produces, how a coal power plant, what kind of fuel it's using, what kind of technologies it's internally using depending on because that will impact everything. So this is something I did not foresee the complexity the potential complexity and we sort of had to draw the line at some points and say okay this is good enough for what we're doing and we're not gonna spend more time making it more precise.
[00:42:57] Unknown:
And looking forward, what are some of the plans that you have for the future of the electricity map project?
[00:43:04] Unknown:
So our our vision is really to be able to cover the map, the world fully. So think of it about Google Maps of electricity. We want to be able to be as close to real time as possible and have the highest spatial granularity as possible. In principle, we should be able to tell you exactly in your city how clean is the electricity that you're consuming. So this is really the the vision that we have and the end goal for for electricity map. The thing we have in store we we are really trying to focus on getting a bigger impact, making sure that our APIs are used everywhere. That basically the whole world of electricity consuming devices starts consuming at the right time. We are experimenting also with more regulatory projects, figuring out how can we actually certify electricity better. How can we make sure that things are communicated to an even broader audience. So something I can share that we're working on is actually an application that we would be able to that would be on an app store, would be available to anyone, and the objective is really that, you know, my mother or my grandmother should be able to look at this thing and understand in a very intuitive way. Okay. The, you know the wind is blowing outside so currently wind turbines are producing electricity so this is a good time to turn on my television or put my dishwasher to use for example. And getting to that level we're not we we haven't gotten there yet with electricity map it's still very much expert view. It's kind of a geeky, nerdy way of looking at things. We need to touch the broader public and this is something that is also going to be objective in the next years. All right. And are there any other aspects
[00:44:39] Unknown:
of this topic that you think we should discuss before we start to close out the show?
[00:44:44] Unknown:
No. Not specifically. I think we've been
[00:44:47] Unknown:
covering quite a few areas of interest. I think that's good on my side. Alright. Well, for anybody who wants to get in touch with you or follow the work that you're up to, I'll have you add your preferred contact information to the show notes. And with that, I'll move us into the picks. And this week, I'm going to choose rollerblading. My family and I recently picked up rollerblades so that we could all do it together. And, there's a local roller rink where we can go and practice and there are some, local parks where we plan on going and, going rollerblading as a family this summer. So it's a fun way to get outside and get some exercise and try something new. So for anybody who's interested in something like that, it's worth a shot. So with that, I'll pass it to you, Olivier. Do you have any picks this week? So so I have, I have actually 2,
[00:45:37] Unknown:
but but the first 1 is a documentary that I watched recently, which is a documentary about AlphaGo, the DeepMind program that that did a big, leap forward by being the 1st artificial intelligence to beat the world champion at Go. And I found that documentary quite fascinating about the amount of resources that were deployed and, into tackling this problem and also seeing exactly how far we've come with the power of data and making decisions based on that. And I was wondering when I watched it, you know, if we put exactly the same resources to try to tackle climate change, what would happen exactly? Because we have so much knowledge that we basically know more or less what to do. It's just a matter of deploying everything and actually doing it. And I think, I think we need more resources to, to be dedicated to this. But AlphaGo was a very fascinating documentary, so I'd highly recommend it. And and the the second pick that I had is is mostly if anyone is interested, I think, in getting, like, a pragmatic overview over, climate change and what we as consumers can do, we, we tried to put together a very short guide that tells about all of those things, and it's available on our website. So so I think if anyone is looking for a bit of reading for the weekends, I'd highly recommend that if you're interested in climate change and want to have an impact. So so it's available on our website on tmrow.com.
You just look for climate change. Alright.
[00:47:02] Unknown:
Well, thank you very much for taking the time to join me today and talk about the work you're doing with electricity map. It's fascinating project and 1 that I was happy to find. And I hope you continue to find new sources of data and new ways to look at it and make it usable by the public. So I appreciate that, and I hope you enjoy the rest of your day. Thank you. And likewise, thank you for inviting me to be us.
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. When you're ready to launch your next app, you'll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 200 gigabit network, all controlled by a brand new API, you've got everything you need to scale. Go to podcastinit.com/linode to get a $20 credit and launch a new server in under a minute. Finding a bug in production is never a fun experience, especially when your users find it first. Air brake error monitoring ensures that you'll always be the first to know so you can deploy a fix before anyone is impacted.
With open source agents for Python 23, it's easy to get started, and the automatic aggregations, contextual information, and deployment tracking ensure that you don't waste time pinpointing what went wrong. Go to podcastinnit.com/airbreak today to sign up and get your 1st 30 days free and 50% off 3 months of the start up plan. To get worry free releases, download Go CD, the open source continuous delivery server built by Thoughtworks. You can use their pipeline modeling and value stream app to build, control, and monitor every step from commit to deployment in 1 place. And with our new Kubernetes integration, it's even easier to deploy and scale your build agents.
Go to podcastinit.com/gocd to learn more about their professional support services and enterprise add ons. And visit the site at podcastinit.com to subscribe to the show, sign up for the newsletter, and read the show notes. Your host as usual is Tobias Macy. And today I'm interviewing Olivier Caradi about electricity map and using Python to analyze data of global power generation.
[00:01:49] Unknown:
And Olivier, could you start by introducing yourself? Yes, of course. Thank you for inviting me here. So I'm Olivier Corady. I'm a French and Danish data scientist and machine learning expert. And I'm basically born in data before it was actually called data science, but I've always enjoyed since little
[00:02:10] Unknown:
extract knowledge based from it. And do you remember how you first got introduced to Python?
[00:02:15] Unknown:
It's, it's tough to remember exactly, but I think it was, when it was in high school and trying to build this home project that would download automatically videos from YouTube because I had very, very bad connection. And on the train, I wanted to be able to look at those videos. And so what I did is, like, a cron job that would use Python to download those videos and store them locally, transfer them to my phone, and then I could be able to see them
[00:02:42] Unknown:
on my phone. And as I mentioned, we're talking about the work that you've done on the electricity map website and API and database. And I'm wondering if you can talk about your motivation for creating that project in the first place.
[00:02:56] Unknown:
Yes, of course. So so approximately 2 years ago, it was April, I was wondering a little bit about the situation of of climate change and what we could do about it. And I stumbled upon the fact that all of our energy basically comes from fossil fuels, which is very bad. It's 80% of it. And then realized if we were if if we are to live in a fully electric world where we don't use fossil fuels it really matters how electricity is produced And as still a lot of electricity is produced by fossil fuels, coal, and gas, it's 2 thirds approximately. I figured, well, we really need to get this knowledge out to people, make something very intuitive that shows in real time what is the wind turbines producing. Are we able to turn off the coal in the gas power plants?
And how do we actually make an energy system or an electricity system that produces electricity even when you don't have the wind blowing and the sun shining? And this is really the motivation behind the project.
[00:03:55] Unknown:
And for anybody who hasn't visited the site, if you go to it, you see a map of the world with various different political regions colored according to the overall, carbon production been happening at a given point in time based on the electricity plants and how they're generating their power. And I'm wondering if you can describe how an average person who is visiting the site or using the APIs that power it can benefit from the information that you are making available through those means?
[00:04:29] Unknown:
Yes. In a in a in a very basic way, what we try to showcase is the so called carbon intensity, and it has a weird unit. It's called grams of CO2 equivalent per kilowatt hour. But what it basically means is that it tells you how much greenhouse gas or CO2 is emitted for every unit of energy that you will consume. Let's say that for example, you put your electric vehicle to charge, We want to give you a signal that tells you this is a good time or this is a bad time. And so the map basically is colored based on this metric that tells you is now a good time to consume or is now a bad time to consume. And so it takes a lot of things into considerations in order to calculate this. It looks at how electricity is produced locally, but it also looks at how it is imported, where it's imported from, and how that country that is currently exporting power to you is generating its electricity. An example I often take is Denmark which has a lot of wind power. On a day where it doesn't have a lot of wind power it will import power from Germany or from Sweden or from Norway And there's a significant difference if you import from Norway that has a lot of hydro electricity, which is low carbon, compared to if you import from Germany on the on a day when there's no winds, and no sun where it will be mostly cold generated. So we really try to tell something very intuitive. We tried to boil down a very complicated number to simply a color on the map, and I think this is why it caught people's interest. It's that it's succeeded in boiling down something quite complex to simply a color.
[00:06:08] Unknown:
And the color for the region on the map factors in those transfers. So for instance, as you mentioned, if you're transferring energy in from a grid that's producing energy via a lot of fossil fuels, it will actually influence the way that the visual representation is displayed for the region that you happen to be in regardless of how your local power generation is happening. Is that accurate?
[00:06:33] Unknown:
Yeah, that is correct. And this is also why the we've we've drawn the interconnectors or the exchanges of electricity between the countries as small arrows that give you a direction of the flow. And the arrows are also colored by actually the carbon content or the carbon intensity of the electricity that transits. So it makes it very simple to understand if you have a country that is darker because it's it's less it emits a lot of carbon for every energy that's consumed. You can actually immediately see if it imports power that is greening it or making it worse based on just looking at the color of the of the small arrows on the map.
[00:07:12] Unknown:
And in order to be able to provide all this information, you need to gather it from various sources. And I'm curious what what those sources are that you're using to be able to gather the information and aggregate it for how the electricity is being generated and distributed in this near real time fashion?
[00:07:33] Unknown:
Yeah. So when we started out the project, I had only 3 data sources. So the data sources I had at the time were the so called transmission system operators in France, Germany, and Denmark and it's a nationally, owned entity publicly owned entity, that is responsible for controlling the electricity, balancing in the system, basically making sure that there's no blackouts. And so because they're publicly owned, they have an incentive in publishing open data about how electricity is generated in real time in order to both inform the citizens but also in order to make sure that the best investments are taking in the grid. So they publish a lot of open data and we've been connecting with those real time open data feeds, integrating it into the Python back end, and then visualizing it on the map itself. And an interesting thought that happened after this is how can we make this scalable if we wanted to get all countries in the world to be painted with the proper color and this is the reason why we made the project open source. So if I can say a few things about that is that we have an open source repository where we have a list of Python scripts that we call parsers, which will connect to each of those entities that are publishing open data.
And this script basically returns in a standardized format that we defined ourselves. What is the production mix? So what is what units are producing what in a given area, and what are the exchanges with the neighbors? And so anyone that can produce such a script that knows about the location of data can write such a script. We're taking it to our back end and then it automatically colors the color on the map and this is the reason why we've been able to scale up and have so many countries is through all of the contributors that come and saw the project said I actually know where this data and I know a little bit of Python so I can write a script, contributed to it.
And now I think we started with 3 countries and now I think we have over 100 areas, covered. And I say areas because there's actually some countries that we've broken down by state. For example, India is broken down by state. Australia is also broken down by states, and we're in the process of doing the same thing with the US.
[00:09:55] Unknown:
And I imagine that because of the number of different sources that you're aggregating from that there's going to be a disparity in the robustness and the format and the accuracy of the data that's being provided. So I'm wondering how you've managed to unify that information for being able to display it in a consistent fashion.
[00:10:18] Unknown:
Yes. So there's there's a few issues with data quality. There's first of all, can you trust the data? Is it accurate enough? Because sometimes we have, for example, data providers reporting twice as much wind produced as it actually is produced. We have issues where there's some parts of the generation mix, which is missing. For example, countries would say right now there's no coal being produced, even though there is some, we just don't know exactly how much. And this would, for example, turn the map into a greener color than it's really supposed to be because we're having missing data. So for all of those things, we have over the months created a list of checks.
So we know that this country is supposed to produce at least this amount of that particular generation. And so we constructed sort of a list of different checks that we go through. So we know is the data valid or isn't it valid. So this is the first thing we do. The second thing is that we have sometimes data that is delayed. So for example, data provider will sometimes provide data about the current system, but it will only provide it, 5 hours from now. So the because the full system is completely interconnected in the sense that if you have just 1 data item that changes, for example, in Spain, it will affect how the electricity consumed in Denmark would look like even by a small amount. We have to make sure that all the data we have on the map is somehow representative of the same point in time. In order to do that, the the simple, solution we came up with is just saying if a data item is older than 2 hours, then we simply discard it, and else we just take the last 1 that is within those 2 hours. So if you look at the electricity map, you know that all the data that is that you're looking at is fresh by at least 2 hours, else, we remove it. And I noticed what as I was exploring the map that there are cases where the source of the energy production Yes. So so, philosophically, what we've decided to do when we're dealing with, the different data providers is that we really wanted to stay as close as possible to the data source. And we have a specific breakdown of how generation units are classified. We have for so if you take coal, for example, there's both lignite mines and different types of coal power plants. We've just decided to have 1 single entity to represent all of those for simplicity because we thought it reduces the complexity and simplifies the message we want to carry.
However, the data source providers, they don't necessarily give us directly the the call unit. They will give us the breakdown. So we have to have some rules into how we aggregate the different units that the data provider will give us. And sometimes, those rules are not, very easy to understand. Sometimes, you have units that will both produce biomass using biomass, using gas, and using coal. So how do you classify those? There's all sorts of edge cases. And we've decided to make the least amount of assumptions as possible. And so every time we have a unit where we're not completely sure about it, we put it in unknown, and this has a dual effect. The second effect is also that it actually motivates the data providers to open up more data and more accurate data as they come to the map and actually see unknown. Or if visitors come to the map and see unknown, and then they will start talking to data provider and say, why don't we have better data here? And this pushes up different data providers to open up more precise data.
[00:14:03] Unknown:
And beyond the, visual display of the map, you also have APIs and the actual actual database available for use. And do you maintain the, higher granularity of the data
[00:14:17] Unknown:
in those sources for people to be able to provide their own analyses as well? Yes. So so another service we're trying to do, which is a commercial service on the other hand, so the map is completely free. Everyone can access it. Part of it is open source. But we have a commercial service behind the scenes where we enable people to look at highly granular data that we compute based on everything that we have. So there's all sorts of very, interesting AI that we can do on on this data that we can talk about, if you wish.
And because it starts being technical, but the power industry is very interested in understanding exactly what happens if you add a little bit of extra wind power in a certain country. Will that decarbonize it effectively or you know won't it change everything because anyway there's so many wind turbines already that it doesn't really affect the system. And those are the kind of analysis that we would do and that we would provide through different APIs. And so the 1 I just mentioned is is called the marginal carbon intensity. Something else we also do with the APIs is that we do predictions. So we do predictions 24 hours in advance of how the whole electricity map would look like and this is useful if you, for example, want to plan ahead and you want to use electricity at the right time. So let's say that you have an electric vehicle, you come home, you plug it in, in the evening and then it has the whole night to charge. Well, the color on the map will change during the night depending on lots of various factors. For example, the wind can start blowing from 1 hour to the other. And so what you really want is you want your electric vehicle to automatically fetch the forecast and then say, I know how much I need to be charged by tomorrow. But if I just waited 1 extra hour I know I will be fully charged by tomorrow morning but I can still optimize and reduce the carbon, that it will be emitted in in the atmosphere due to my electricity consumption. And those are also projects that we're doing. We're doing this with electric vehicles and we're doing it also with heating systems.
And so this is all part of the API or the forecast API that we're providing. We really imagine a world where any IOT device that is consuming electricity and is connected to the Internet would respond to this forecast
[00:16:28] Unknown:
in order to, you know, consume at the right time. And another factor that can complicate the balance of power generation in an electrical grid is things like residential wind and solar, where if an individual is producing more than they're able to consume in some areas, they're able to feed that back into the grid and sell it back to their provider. Is something like that factored into the data that gets reported to electricity map?
[00:16:57] Unknown:
So that depends. And that's a that's a very good question. So so there's 2 types of decentralized generation you can have. You can have something that's called behind the meter. And so this basically means that if you have a solar panel on top of your house, instead of consuming grid electricity, you can just cut the cord so to speak for a couple of hours and then use your own electricity that you might for example have stored in a battery. This is behind the meter behind the electricity meter that your electricity distributor might have access to, and so we have no way of really accessing that data. But for now, it's such a small portion of of people that have such system installed that it doesn't really change the big picture, but this is a blind spot for us. The second thing is the thing you mentioned where you have so much surplus of local generation that you're actually giving back to the system, and so the meter starts running backwards you might say, and this data depending on the different data providers we have an idea of how much is getting fed back to the grid But again, it's so small amounts compared to the very, very big industrial scale large production that we have that it basically doesn't change anything for now.
[00:18:07] Unknown:
And going back to the idea of the accuracy of the data and how it's represented, Are there any penalties for any power plants or anything like that that misrepresents the types of fuel that they're consuming or the amount of waste that's being generated by any given plant?
[00:18:24] Unknown:
So so there's, we have to distinguish different areas of focus here because if you look at, for example, Europe, there's a European directive that stipulates that data should be opened up, and that this data should be accurate up to certain specifications. However, there's no to my knowledge there's no financial penalty that is that is happening there if you don't do what is supposed to be done. We have countries like, for example, the Netherlands which are not supplying this data at all and there's a lot of regulators that have, you know, told them they have to do it and so on but it's still not happening. So the only thing we can do is actually put them pressure but there's no financial penalties happening if they don't report things correctly.
However, you have some I know I have knowledge of some countries also that are being watched very closely by regulators and the regulators are assessing those countries through this open data. And so, if the open data is not accurate enough then the regulators come to their job and there they might get some penalties directly. But the end game is really this. There's no, there's no real painful penalty for not publishing the data. So it poses the problem of can we actually trust the data that comes out of it and because it's publicly owned entities that are publishing data, in principle it's like you have you're trusting public entity like the states. So in principle, the level of trust should be quite high, but we we notice some discrepancy sometimes, and we try to factor them and be transparent about them.
[00:19:58] Unknown:
And in terms of the overall impact of the electricity generation on the carbon output of global society, do you have any sense of the percentage that is attributable to electric consumption and electric generation as opposed to other sources such as vehicle exhausts or, any other forms of manufacturing?
[00:20:23] Unknown:
Yes. If I there's 1 number I have in mind is that coal power plants in the world, I think, are responsible for between 15 20% of global greenhouse gas emissions, which is huge. And coal is mostly used for electricity in the world. So that gives you sort of an order of magnitude where transportation, for example, I think is only a couple of percent of global of greenhouse gas emissions. Those numbers should be taken with a grain of salt. I can just double confirm this is just by memory. I can attach a couple of links, afterwards just to be sure. Okay.
[00:20:57] Unknown:
And 1 other thing that's worth calling out is that just now you used the broader term of greenhouse gas and earlier you mentioned that the units of measure that you're using for displaying in the electricity map is the grams of c 02 equivalent per kilowatt hour because, as some people may know, it's not just carbon dioxide that has an impact on global climate change. It's other things such as methane or other emissions. And so I'm wondering if you can just briefly speak to the calculations that you perform to be able to unify the emissions output into that single metric?
[00:21:37] Unknown:
Yes. Of course. So if you take methane, for example, methan methane is 10 to 20 times more potent in terms of how much warming it will cause over the next 100 years compared to co2 for an equal weight. Hence, what is usually done in the industry when you do carbon accounting you want to assess the climate impact is that you say, I take my amount of methane, I look at how much it's going to warm the atmosphere in the next 100 years, and I convert it into an equivalent mass of co2. And so by doing this, you can take any greenhouse gas and say, converting all the effects it will have on the warming and converting that to an equivalent mass of co2, and that gives you a unit called co2 equivalent. So now we we don't do ourselves the the c o 2 calculations, in the sense that we don't look at each power plant, how much methane it would leak, and how much green, how much CO2 would emit because that would make us highly vulnerable to trust issues.
Now the the the UN has funded something called the IPCC, which is a globally renowned institute that looks at meta analysis. So it looks at many, many different scientific studies, aggregates them, and then is able to say, okay. Typically, a coal power plant in the world emits that much greenhouse gas, so c 2 equivalent per unit of electricity produced. And we use those numbers, and we then factor in our analysis. Then we trace the origin of electricity, and then we can color the map based on this cu2 number. We have a whole section I can put a link to that on our website that explains a little bit the climate perspective, how c o 2 equivalent is calculated, how do IPCC numbers come from, and how this all relates together.
I'll put a link in the in the description afterwards if you're keen to have that.
[00:23:28] Unknown:
And going deeper now in into the technical architecture of how this all operates, I'm wondering if you can give an overview of the way that the data flows from the gathering point from of these different open data providers through to the, analysis and representation in the front end?
[00:23:50] Unknown:
Yes. Definitely. So so what we have is we have 2 components. We have 1 that we call defeater. It basically is a Python script that runs every every minute. It takes all of the different parsers that we have, so all the scripts that will connect to open data to to get the information, and it runs this every minute, gathers all the data, and puts it in a database. And then we have another system that also runs, every minute as soon as the previous system is finished and you have collected all the data put in the database, we have this other system that is responsible for querying the last 2 hours of data that we have access to, computing all the import exports in order to figure out where does electricity come from in a given zone, then it applies all the c o 2 equivalent, what we call the carbon intensity factors, of each of the fuel sources, and then it it comes up with a list of countries with their associated carbon intensities.
And all of this is basically put in a in a Redis cache so that's such that it's ready to be consumed by the front end such that we don't have to re query all of the database every time a visitor comes to the website. And so the front end we can speak a little bit about how the front end is is designed also if if you'd like, but it basically just calls an API that's just just serving the the cache. And in that way, we have a very efficient performance system that is always up to date. And for
[00:25:19] Unknown:
the, web architecture of the application, are you using a framework such as Flask or Django?
[00:25:25] Unknown:
Not even. We, we basically have on the so the front end is served by a note back end, that just exposes an API with some JSON that comes from the Redis cache, which is then populated by this Python script that runs every minute. And on the front end, we're using, a lot of d 3 in order to on the front end side, manipulate all the different objects, but we don't use any Django framework or something else. It's exclusively JavaScript based As soon as you go out of the of the cache and you're running into the just the front end,
[00:25:59] Unknown:
everything is JavaScript. Yeah. That makes sense for simplifying the delivery of the application so that you can, you know, bundle it all up and use Python where it is powerful in terms of being able to consume and process and analyze the data. And on the website, I noticed that you have an architecture diagram where it looks like you're also using TensorFlow and Spark for being able to do some of the heavier processing. I'm assuming that that is part of what's powering the paid API that you mentioned of the forecasting.
[00:26:31] Unknown:
Yes. That's correct. So we have to we have to basically be able to stream all this data in real time and make sense of it in order to run our forecasting models on top of it. And because the forecasting models are basically spatiotemporal so they will take into account data in the past and if you want to forecast a specific country you need to take into account how its neighbories countries are performing, how the imports exports are happening, and so this creates a special temporal dependency, a sort of special temporal graph, if you will, in the past that we have to process in real time. And this is why we use stuff like TensorFlow and Spark because it can start being heavy in terms of the amount of data that we're processing and the amount of calculations we're doing. And how has the overall system architecture evolved over the course of time that you've been working on the project? It's actually pretty funny because in the beginning, it just had to work in this fastest way possible with the simplest setup. So we simply had a a MongoDB database that would store all of this data, and every time a visitor would come to the website, it would query that database.
And so it was a pretty heavy calculation that would be done. And even the the whole flow tracing, figuring out where electricity comes from based on all the imports and the export, all of this was coded in the front end directly. So this algorithm requires a small matrix inversion in order to figure out all the couple flows, and it would have numerical libraries that would actually be in the front end, written in JavaScript just because it was faster to, you know, debug it and iterate. And as time moved on and when this project got a bit of traction, at some point, we were on, on the front page of Hacker News for 2 days. And you can imagine that if you have those 10, 000 visitors per hour, the MongoDB back end was just, you know, blowing up. And this is where we transformed the architecture and rewrote it during a couple of nights where it just had this caching system, and we also put the matrix inversion system in the back end such that we have a, you know, faster computations on everything. Something that also evolved quite significantly that is that on on the website, you can also see the current weather patterns. So we're showcasing how much solar energy is currently hitting the ground, over the world, and we're also showing how much the wind is blowing all over the world through a small toggle that you can activate and deactivate. And this is actually quite a lot of data to be processing on the on the server. We're storing historical weather data for 2 years and we're talking terabytes of data. And so we've been making quite a lot of changes also in in the way we handle those weather forecasts and we do everything in python through something that's called pygrib, which is a small python library that's very handy to manipulate meteorological data. So, yeah, the infrastructure has been involved quite heavily, and then we've built all the analytics back end on top of that, which is also another story.
[00:29:21] Unknown:
And what are some of the most interesting or unexpected uses of the information that you're providing through the map or for through your API that you're aware of? So there's a few interesting uses
[00:29:33] Unknown:
of the of the map itself. I was quite happy to see that it's being used in universities and in primary school and higher in in secondary school also, by teachers in order to show students and get them a bit of intuition about how the whole electricity system work. And I think this is really something that I'm very very happy about because this was the whole point of the project is to get more people to play around with this data and and be educated by it. Now 1 of the unforeseen uses of the electricity map is for lobbying because I didn't realize how much nuclear lobbyists are interested in this kind of map. Because we're focusing on on c o 2 emissions or greenhouse gas emissions.
What you end up with is having nuclear powered countries. Nuclear is low carbon and so you end up having France, for example, being green on the map because it's mostly powered by nuclear. And so the nuclear lobbying are they are very, very interested in showcasing the map everywhere. And so this map has been used in a lot of very very aggressive debates on should we actually decarbonize the system using wind and solar, should we do it with nuclear instead. It's been very challenging for us to remain completely neutral on this issue because every time we'd showcase a particularly interesting situation where, for example, Germany was producing a lot of wind power and we said look Germany is green because it's windy right now, how cold is that? Then you would have the whole nuclear lobby just jumping on us and saying, well, but, if they just did like France, which would have low nuclear power low carbon nuclear power all the time, it would be much better. And so an unexpected use for us for the data is is really how those 2 sides have been fighting against each other's and using the map as sort of arguments
[00:31:20] Unknown:
data driven arguments. Yeah. It's a good example of the fact that data is neutral, and it can be used to argue whatever point you want depending on how you decide to interpret it.
[00:31:30] Unknown:
Exactly. And I think this is also the philosophy behind the company that we created based on this electricity map because we believe that if you want to find scalable solutions to climate change what you need is you need the whole society to have a more objective data driven view of what are the things that I do in my daily life that are impacting the climate. What is something that actually matters and what is something that is completely useless because it's only of magnitude less that something else that we do. And this is where data comes to the rescue. And I I strongly believe that data is the way forward to make better decisions. And so we we really want to do more than just electricity if we could,
[00:32:12] Unknown:
later on. And have you found any ways to measure the impact or effectiveness of the information that you're providing through the electricity map and the APIs?
[00:32:23] Unknown:
So so I think there's 2 types of impact. We can look at it in terms of audience. How many people actually look at this, find it interesting, share it on social media, and so on. And the kind of the the metrics I have in mind is that now even 2 years after, we still have approximately 3, 000 unique users coming to the map every day, to look at it, tweet about it, and so on. And we've never done any marketing, so it shows that there is some kind of interest, and there is some kind of impact in the sense that teachers are using it in university and so on. If you think about quantitatively measuring the, you know, amount of greenhouse gas we'll be able to reduce using such a tool, it starts being a bit more complicated. But for example, with the projects we're doing with electric vehicles, we measured that if an electric vehicle is using our system, our API in order to charge at the right time during the night, you can save approximately 20% greenhouse gas emissions every night based on a fully electric vehicle charge. So it starts to be coming interesting if you multiply those initiatives to, you know, the heating in your home if it's electric and you start going to industrial scale like for example supermarkets they have a lot of cooling electricity they use. They could also use it at the right time. So I I think we're going to have bigger and easily quantifiable impact in the future but right now it's still a bit early to say. And at a broader scale,
[00:33:52] Unknown:
the ways that this information can be used is potentially in terms of building a newer or more intelligent grid for determining how the electricity is distributed within the grid based on how it's being produced at any given point in time. So I don't know if you've put any thought into how that can take place or some or working with any of the power companies for being able to incorporate the information that you're providing? Yeah. So we work a lot with the different,
[00:34:22] Unknown:
energy and electricity distribution companies, in in various countries in order to do exactly that. So there's sort of 2 timescales that are relevant here. You can look at the short timescale where we're talking hours to days. You know, the electricity system is fixed. There's not so much stuff that you can change. So the only thing you can do as a consumer or as a company is that you can decide when to use electricity at the right time. This is really the only thing you can do because you cannot say you cannot control electrons. You cannot say, please go to that company instead of the other 1. So that you cannot do. And then if you look at timescales that are a bit longer, for example, months to years then it starts becoming interesting because thanks to the data that we have we can start saying, you know, should you invest in putting a wind turbine in this country or should you invest in augmenting the exchange capacity between 2 countries such that the flow of electricity can be bigger between the both of them because it will decarbonize much more efficiently the 1 that's importing, for example. And those are the kind of arbitrages we are able to do things to all the data, and we're working very closely to with different electricity, companies in order to do exactly that. But it raises a more fundamental question, which is how do you actually measure where electricity comes from, and how do you measure the benefits of, you know, taking different decisions? And there's there's many different answers to that, unfortunately. We're looking at something very physical. We're saying, if you're connected to a system, then you inherit all the electrons that come to you, and you cannot choose exactly where they come from. You cannot choose to, for example, consume directly from the wind turbine that's on the other side of the country. But, however, you see more and more of those 100% renewable contracts that are happening in electricity suppliers or you see mobility providers like electric vehicles that will tell you well this is 100% green electricity because we signed a contract with someone that enables us to tell that we're 100% green. Unfortunately, the problem with those things is that that it's it's not physical. So you even have a discrepancy in how to actually agree on where electricity comes from And this is also a place where we're trying to do some lobbying, trying to educate people on what are the different ways of accounting and what is the way that is the most efficient to decarbonize the whole system the fastest possible. And we definitely believe it's the it's taking a physical approach that will make the most sense in the longer term. And
[00:36:46] Unknown:
going back to the community reaction to the project that you've built, you mentioned earlier that a lot of the data sources that you've been able to add over the past few months and the couple of years that you've been working on it is because of community contributions of people different data sources and wrote the parsers for consuming it and providing it to the project as a whole. I'm wondering if you can talk to how you built that community around the project and how that community has helped in building and growing the electricity map project. So the way we engineered it,
[00:37:24] Unknown:
so to speak, in order to foster a community around the project is that we started being open about everything that we did, and we wanted to make it very very visible to any visitor that came to the map that if he had knowledge of any data that could help us that he should know exactly where to, you know, communicate that knowledge. And so 1 of the concrete things we did is that on the electricity map, you'll see on the on the bottom left side a big slack button, and so if you click on it you immediately get invited to our slack channel where you can join, you can talk to us, and you can immediately let let us know if you know anything, if you have ideas on how to improve the map. And through this Slack channel, we've really been able to foster a whole community around the project where, you know, we post links, we discuss another topics, mostly people interested in data science and climate change in general. The second thing that we also did is that we made it very explicit that if you we wrote basically on the website directly that if you knew anything about data or if you have any ideas on how to improve the system, you should click here and that link would directly take to GitHub issues.
And I've actually been very surprised at how many non technical people have ended up coming on GitHub and opening issues, which I find a bit fascinating because GitHub is not the most user friendly platform wherein you're not a developer. So we have roughly, you know, half of the contributors that are non technical people that don't code, that have been submitting pull requests where they would just, you know, change some numbers in a JSON file because just by pattern matching they knew which numbers to change and then they would pop up on the electricity map, carbon intensity factors for example, and the rest are people that would be coding the parsers that maybe the other half, the non technical people, would have identified, the data sources they would identify, then the the coders would take them and then and then implement them. And through the course of time, I think we've had 50 different contributors doing pull requests on the on the website. And we have, I think, 2 to 300 registered Slack users. And in our day to day operations, we have like a small core group of 4 to 5 contributors that are very active in helping us add new countries, find them correct data that's incorrect, and so on. And for my sake, I think it's been almost a year since I actually coded, myself a parser that would get some data and put it in the system, I've been mostly focused on on making the UI and the UX of the front end robust, fast, and usable by anyone.
For example, building a mobile app has been something that's been taking a lot of time and the community has been very helpful also in order to give us feedback, early testing, on everything that we did.
[00:40:13] Unknown:
And in the process of building any substantial project, there are usually a number of things that you end up having to tackle that you didn't foresee at the outset. So I'm wondering if you can speak to some of the pieces of information or challenges that you have come across that were most unexpected that you ended up learning in the process of building the
[00:40:39] Unknown:
angle you angle you take. How easy or complicated it would be to actually foster a community. And that's something we really didn't foresee at all. How much time you need to spend in order to, you know, communicate all the knowledge that you have to everyone such that they can help. So at some point you start being dedicated to help people and let them be empowered by by what you're doing because that's the only way that it really gets, you know, involved in the project. So this is this is this is 1 thing that was completely unexpected. The second thing that's completely expected is really, the challenge in the amount of details you need to go through in order to present something that is trustworthy in terms of carbon intensity. So basically, there's so many level of details you can start drilling through if you want to accurately show data. 1 of the examples I can take is the real time data that we get is not it's accurate up to a certain level. So the data providers, they're measuring and monitoring the system to a certain amount, but then they will get extra information as time goes on. For example, 2 weeks later, they will get extra information about some units that didn't report accurately and so they will be able to correct that data. And 1 month after, they have even more accurate data.
And talking to some of those data providers, we realized that the the final data, the 1 they use for billing purposes, it can even be 3 years old. So it means that 3 years after, you can still have corrections on the data you have. Granted you're you're correcting very very small numbers and the real time data is is sufficient for our purposes. But as so depending on who you speak to exactly and the level of details they want to or the level of precision they want on on both the electricity side or the carbon side, you end up going in discussions that are very very detail oriented about exactly how wind turbine, you know, produces, how a coal power plant, what kind of fuel it's using, what kind of technologies it's internally using depending on because that will impact everything. So this is something I did not foresee the complexity the potential complexity and we sort of had to draw the line at some points and say okay this is good enough for what we're doing and we're not gonna spend more time making it more precise.
[00:42:57] Unknown:
And looking forward, what are some of the plans that you have for the future of the electricity map project?
[00:43:04] Unknown:
So our our vision is really to be able to cover the map, the world fully. So think of it about Google Maps of electricity. We want to be able to be as close to real time as possible and have the highest spatial granularity as possible. In principle, we should be able to tell you exactly in your city how clean is the electricity that you're consuming. So this is really the the vision that we have and the end goal for for electricity map. The thing we have in store we we are really trying to focus on getting a bigger impact, making sure that our APIs are used everywhere. That basically the whole world of electricity consuming devices starts consuming at the right time. We are experimenting also with more regulatory projects, figuring out how can we actually certify electricity better. How can we make sure that things are communicated to an even broader audience. So something I can share that we're working on is actually an application that we would be able to that would be on an app store, would be available to anyone, and the objective is really that, you know, my mother or my grandmother should be able to look at this thing and understand in a very intuitive way. Okay. The, you know the wind is blowing outside so currently wind turbines are producing electricity so this is a good time to turn on my television or put my dishwasher to use for example. And getting to that level we're not we we haven't gotten there yet with electricity map it's still very much expert view. It's kind of a geeky, nerdy way of looking at things. We need to touch the broader public and this is something that is also going to be objective in the next years. All right. And are there any other aspects
[00:44:39] Unknown:
of this topic that you think we should discuss before we start to close out the show?
[00:44:44] Unknown:
No. Not specifically. I think we've been
[00:44:47] Unknown:
covering quite a few areas of interest. I think that's good on my side. Alright. Well, for anybody who wants to get in touch with you or follow the work that you're up to, I'll have you add your preferred contact information to the show notes. And with that, I'll move us into the picks. And this week, I'm going to choose rollerblading. My family and I recently picked up rollerblades so that we could all do it together. And, there's a local roller rink where we can go and practice and there are some, local parks where we plan on going and, going rollerblading as a family this summer. So it's a fun way to get outside and get some exercise and try something new. So for anybody who's interested in something like that, it's worth a shot. So with that, I'll pass it to you, Olivier. Do you have any picks this week? So so I have, I have actually 2,
[00:45:37] Unknown:
but but the first 1 is a documentary that I watched recently, which is a documentary about AlphaGo, the DeepMind program that that did a big, leap forward by being the 1st artificial intelligence to beat the world champion at Go. And I found that documentary quite fascinating about the amount of resources that were deployed and, into tackling this problem and also seeing exactly how far we've come with the power of data and making decisions based on that. And I was wondering when I watched it, you know, if we put exactly the same resources to try to tackle climate change, what would happen exactly? Because we have so much knowledge that we basically know more or less what to do. It's just a matter of deploying everything and actually doing it. And I think, I think we need more resources to, to be dedicated to this. But AlphaGo was a very fascinating documentary, so I'd highly recommend it. And and the the second pick that I had is is mostly if anyone is interested, I think, in getting, like, a pragmatic overview over, climate change and what we as consumers can do, we, we tried to put together a very short guide that tells about all of those things, and it's available on our website. So so I think if anyone is looking for a bit of reading for the weekends, I'd highly recommend that if you're interested in climate change and want to have an impact. So so it's available on our website on tmrow.com.
You just look for climate change. Alright.
[00:47:02] Unknown:
Well, thank you very much for taking the time to join me today and talk about the work you're doing with electricity map. It's fascinating project and 1 that I was happy to find. And I hope you continue to find new sources of data and new ways to look at it and make it usable by the public. So I appreciate that, and I hope you enjoy the rest of your day. Thank you. And likewise, thank you for inviting me to be us.
Introduction to Olivier Caradi and Electricity Map
Olivier's Journey with Python
Motivation Behind Electricity Map
Understanding Carbon Intensity
Data Sources and Aggregation
Unifying and Validating Data
Handling Decentralized Generation
Accuracy and Trust in Data
Impact of Electricity Generation on Global Emissions
Technical Architecture of Electricity Map
Community Contributions and Growth
Future Plans for Electricity Map
Picks and Recommendations
