The Python Podcast.__init__

Share on social media:

[00:00:13] Unknown:

Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. When you're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it. So check out our friends over at Linode.   With their managed Kubernetes platform, it's easy to get started with the next generation of deployment and scaling powered by the battle tested Linode platform,   including simple pricing, node balancers, 40 gigabit networking, and dedicated CPU and GPU instances.   And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover.  

Go to python podcast.com/linode   today to get a $100 credit to try out their new database service, and don't forget to thank them for their continued support of this show.   The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it. SelectStar's   data discovery platform solves that out of the box with a fully automated catalog that platform solves that out of the box with a fully automated catalog that includes lineage from where the data originated all the way to which dashboards rely on it and who is viewing them every day.   Just connect it to your DBT,   Snowflake, Tableau, Looker, or whatever you're using, and select star will set everything up in just a few hours.  

Go to python podcast.com/selectstar   today to double the length of your free trial and get a swag package when you convert to a paid plan.   Your host as usual is Tobias Macy. And today, I'm interviewing Nicholas Hoehning about Flex Measures, an open source project designed to manage energy resources dynamically   Nicholas, can you start by introducing yourself?  

[00:01:44] Unknown:

Yes. Hi, Tobias. I'm Nicholas. I'm German   and living in the Netherlands   since 15 years. I've been a programmer for a bit more than 20 years   using Python for a long time. I've been working on energy since 12 years, I think,   from that vintage point.  

[00:02:03] Unknown:

And do you remember how you first got introduced to Python?  

[00:02:06] Unknown:

1st, I was I had been using   very little c and Visual Basic. I've been working on   access   databases   and and word   macros.   At at some point, I did then, go to university.   During my bachelor's, I got to work on a start up. It was a neuroscience   startup   that is still around. It's called Iquant.   The idea was to   transfer neuroscience   models of visual attention   into a web application so that you can, you know, basically upload your   ad and find out   where people might look first.   And we do we use a lot of Python there, and I had to you know, I was sitting at lunch, reading the standard Python tutorial so I could get to work on that startup, still remember.  

[00:02:55] Unknown:

In terms of the Flex Measures project that you are working with, can you describe a bit about what it is and some of the story behind how it came to be and why you decided that this dynamic   energy demand response system is something that you wanted to spend your time and energy on?  

[00:03:11] Unknown:

Yeah. Sure. And I'd love to yeah. It has become my life topic.   I started a PhD   around this topic in 2009   when   it wasn't absolutely clear that we will get the kind of energy transition that we are basically in right now,   driven by, you know, lots of renewables   being the major source at some point.   Flex measures, basically, an energy management system.   It helps you manage your energy. In the last couple of years, energy management has meant that you manage   the data around your energy, metering data.   You put it on NICE dashboards and you do billing around it. You know, that's also significant work to be done. Flex Measures focuses more on   the energy flexibility   and optimizing   when energy is consumed during the day   because that's an a very important piece in the energy transition.  

As any any expert will tell you, we'll need to get to this kind of optimization,   at some point.   Otherwise, we will not get the fit between   available renewable energy and the actual consumption to have a stable energy system. So flex measures helps you find the best times   for your flexible energy assets. Let's say, a battery. That's the best example of a flexible energy asset.   When should that battery be charging or discharging?   The other examples for flexible assets are heating. You can heat before   somewhat sometime before you're really home, for instance.   Heat pumps are a big topic.  

The industry, you have a lot of potential   to run energy intensive processes at a slightly different time. And Flex Measures really helps you   model all your energy data in real time and make that decision.  

[00:04:58] Unknown:

You mentioned that a number of the other energy management systems are focused more on a kind of 1 way path of information where you're just retrieving data from the system and then using that to build some analysis, but that doesn't get acted upon in any sort of automated fashion. I'm wondering if you can just give a bit more of an overview of what the space looks like for these energy management systems and if there are any kind of alternatives to flex measures or if it's something that is relatively unique in the space.  

[00:05:27] Unknown:

There are some projects   that have on the road map to also do this kind of real time optimization,   but you got a lot of inertia in these projects as well in companies. So I see a lot of companies who   had to start basically going to customers who have not even any metering   installed whatsoever   other than having a main meter. And the main meter's data   is not real time as well.   So these companies started with bringing in their Raspberry Pis or whatever it is,   becoming good at that, starting to measure, becoming good at visualizing information, becoming good at doing them some some aspects of billing maybe.   And to do this step into data science optimization,   I see   I see the difficulties in some organizations to to that extra step. So   these people are happy   to get some other project and get some other help with that. I don't know all the projects out there, of course. So Flex Measures is open source, and another open source project that I know about is just called OpenEMS.  

That's a Java project in,   originated in Germany   that's, built a quite interesting foundation around it. But also there, there's a big focus in inertia on getting the metering   and visualization   correctly done.   And there's not so much support yet to   optimize the actual   what actually happens and and influence that, and we have decided to only focus on that.  

[00:06:55] Unknown:

As far as the   primary focus and goals of Flex Measures, I'm wondering if you can just talk to some of   the kind of outcomes that you're aiming for and   some of the,   I guess, target audience for the project.  

[00:07:10] Unknown:

As a big goal, the big hairy goal is that Plaxminder should really help to   speed up the energy transition and, you know, become 1 of the standard pieces of climate tech,   infrastructure tech that you would also always consider when you start a new   modern energy project where   your real time optimization   is a part of the solution.   The main way I think Flex Measures can help is really being a developer friendly tool,   solve a lot of these   things that take you weeks of work, get your data model right, get your data integrations right,   get some monitoring   right,   decide how to implement where your algorithms should be run,   have a queue for that, you know, processing queues.  

There's all these things that developers know   will keep you   busy before you get to the real   use case, before you really help your customer, or you are implementing something very   shortly, very quick hack.   See that happening a lot. Other people asking their cousin's neighbor to have a simple algorithm put somewhere, but this is about, you know,   real time data pipelines who have to be kept alive and be checked on health. Yeah. I think we succeed if developers around the world who I expect in the next 10 years to   be looking around how they should approach the similar similar challenges to always consider flex measures.  

And that can be in a start up somewhere.   That can be in   a larger energy company   where they have already their ICT team, but, haven't solved or started to do this yet.   And and I think   this is a problem that happens in all the the the parts of the energy system. For me,   there's 2 big parts. That's industry and,   what's usually   called smart city at the moment. So anything in the the built environment, those are 2 huge sectors of the energy sector, which in itself is 1 of the biggest   sectors in the world that needs to undergo the most rapid transition you could imagine. The other plans are 15 year plans basically to basically change   how everything works.  

So we really hope flex measures can help get that super big vision done in time.  

[00:09:24] Unknown:

Do you also see flex measures as something that an individual homeowner might use as part of managing their energy systems, particularly as more people invest in solar and small scale wind?  

[00:09:36] Unknown:

Yes. I want to talk about a small project we've been doing. So   besides working with customers or with companies that can scale up innovation.   We've seen that this is still slow. Right? So these companies all have to think differently before they haven't been involved with energy so far. It's a long process. So we have, on the sides, created a living lab with just 1 household   with someone who's also very enthusiastic   about, specifically,   vehicle to grid charging.   They bought the exact car and the exact charger that can do that already, the 1 that is allowed to do that in the Netherlands.   And we've really implemented   the automated optimization   of the charging and discharging in in his house. And for that, we needed to actually do something in the house.  

Flex measures is basically supposed to run-in the cloud.   It's dockerized. It can also run locally.   But it basically makes decisions. It doesn't really show you something.   So I want to pitch a very cool project that's called Home Assistant.   It's also open source. And it's also originated in the Netherlands,   but they have a   company in the US also now. Very interesting   group of people. Basically, in the home automation   sector with their software.   We use Home Assistant together with Flex Measures to optimize the charging, discharging, and and taking into account the the solar panel production to really create a cool project. We also open sourced our Home Assistant plug in.   So that's out there. We were on Dutch TV and a couple of people wanted to use it as well. And now there's this week, the first 2 others who've implemented it in their house. So now we have 3.  

But for us, as a company, that's not the focus, but it's where you can really do something right now and really show something. And there's always these, enthusiasts   that just want to build something in their house like that.  

[00:11:26] Unknown:

As far as the overall question   of energy systems, I mean, that can be   it's a very broad term with a lot of different,   I guess, energy sources and use cases   kind of embedded in it, where 1 element of it is the electrical grid. Another is the usage of fossil fuels for transportation and shipping.   There's home heating. There's heating at, you know, industrial scale for large factories.   And the majority of those,   I guess, ecosystems of energy usage are governed largely by different market forces where there is the, you know, energy producer, which is becoming a bit of a fuzzy line now with more homes and residential and industrial installations having their own energy generation through things like solar and wind. And then there's the energy consumer and the distribution system.   And the pricing of those different resources are governed by, you know, some level of dynamic usage where, you know, there are peak loads, and so that can increase energy prices for certain periods.  

And I'm wondering   what you see as some of the benefits that Flex Measures can provide to   this marketplace ecosystem, particularly given the fact that it's not as clear cut now as it once was?  

[00:12:42] Unknown:

No. You're absolutely right. It's the vast,   sometimes confusing   system. I like the analogy that because somehow it's all connected and it has to happen, in real time and the frequency of the grid,   as far as electricity is concerned is held in some   synchronization.   We've created the largest man made machines with the grids. So you got the US grid, the European grid, and the Asian grid, basically, as the largest 3 machines you could even imagine where your house is a part of it, or the electric assets,   and it happens in real time. Different markets   buy and sell and allocate energy for different horizons.  

And FlexMaster is certainly   not very useful in the range of, let's say, you're buying your energy a couple months ahead and buying these big budgets.   Also, I wouldn't say it's usable in the second range. There's some balancing markets to get this frequency challenge on the grid, right, where you have to react within a second.   But everything in between,   where you have to react basically during the day   or you think in the evening about your plan the next day,   that's where Flex Measures,   is really useful.   And as I said, it's really on the electricity   range   where we want to use it.  

So if you're a consumer,   you can use flex measures to use greener electricity, but it will help the grid stability. But I think mostly what people care about, of course, is to to have their energy bill in check. The spread on market prices has gone up.   Energy bills   are rising a lot. So there's a lot of urgency, I think. Now there's   clarity that this is very important for them to achieve.   And   from a grid perspective,   if you're a network operator,   it also is great if a lot of   decentralized decisions are happening   that help the grid stability.  

We we have to follow what renewables are doing to a big extent more and more.   We cannot influence when we get that solar power or wind power, but we can influence   when we use power. And if a lot of decentralized   decisions are being made to shift their consumptions   towards when renewable energy is really there. We get a more stable grid.   And   the only other solution for network operators would be to, you know, double or triple every cable in size.   So   this is a big societal   cost saving we can achieve with installing   better decision making in in the nodes, which turns how the energy system works on its head. Right? So still now in the last decades, we have optimized a few power plants per country   and just forecasted what consumers will be doing.  

And now that partly turns on its head. It's very interesting.  

[00:15:27] Unknown:

Digging a bit more into that question of renewables   and their broader distribution, particularly with residential installations as part of that mix, I'm curious   what you see as some of the   broader impact on the way that the kind of marketplace economy around electrical grids in particular   is likely to shift as the power generation and power distribution   becomes   bidirectional   on a more routine basis.  

[00:15:56] Unknown:

1 thing we see in the Netherlands, and I think that the Netherlands are a bit ahead of the curve there, is   more dynamic prices.   I think you can choose between   5 suppliers now that will buy   power on the day ahead market, which has 24   different   slots. But then they will just   give you those   24 prices.   So the the the afternoon   before, you will know what your 24 prices will be the next day. And if you have,   you know, some optimized   optimized system like what we did, for instance, in the vehicle to grid project,   that's going to turn out quite well for you then. You will not consume energy when those prices are high. Of course,   what's interesting is that this might create more of a divide between those people that can afford that   that has this cool   infrastructure plus software, and they get their average energy bill.  

Well, they keep it low, let's say.   And who's going to pay a lot? That's probably going to if everything goes like this, it's going to be the poor people.   So this is outside of the technical   realm, also very interesting   political discussion where we still have some time. And I'm also involved in some of these discussions to discuss how we can   achieve   a more just way to do that.   What we really need is not for individual homeowners to   invest in their own house and make it smarter.   I like it. I'm enthusiastic about it when it happens, but what we really need is this scale.   We really need to do these things for everybody on a big scale.  

So for instance, big housing   corporations   already know, given their ESG   strategy,   they have to renovate   at least 4% of the whole portfolio every year. And every year that they don't do that, it's just getting to it, getting to be more pressing, like,   5%, 6%.   And then, yeah, you need to   make a whole neighborhood or a whole big flat   react to flexibility   demands from the grid.   And that can be very exciting that you then, for instance, have 1 giant heat pump for a whole flat. And especially if you get to bigger numbers,   there will be   a better way to react.  

For instance, it's almost impossible   to forecast for a single house when you will be home, when will you cook.   You can have a little bit of a idea. But if you forecast a 100 consumers,   you get a much better data   forecast, which you can really work with. So I think   that's really what I need to see, these these bigger scaled approaches.   And, yeah, I hope Flex Measures can be can be helpful with that.  

[00:18:41] Unknown:

Digging into some of the way that Flex Measures   operates, I'm wondering if you can talk to the sources of data that you're looking at, some of the ways that you manage that integration   and processing,   and the types of interaction that different kind of users or operators might have with Flex Measures and how much of it is just you kind of set it and forget it? Yes. Sure.  

[00:19:05] Unknown:

So my simple picture of flex measures is data in, scheduling out.   Data in means certainly metering data.   You want to know what all your assets   did in the past. It can be very useful to, for instance, for a battery, know the possible state of charge it has now.   It can help to forecast what will happen in the near future.   We also read in prices,   market prices, for instance, the next day,   weather forecasts.   So you can then forecast your solar generation if that's on your roof.   Sometimes the occupants'   plans are also data that you need to take in. So in the vehicle to grid, living lab in this application,   The occupants' data might be that they say they will need the car for a longer trip   tonight at 6 PM. That's important data that influences   the scheduling because the car should be a 100% full by then.  

Or in a factory,   they also have new constraints every day, how much   processing work they need to be doing.   That's,   about the data. And you said earlier that set and forget might be   an interesting concept. I really agree.   For,   most people   who will live in modern energy environments   with with some optimization   happening,   they should not even   care about that during a normal day.   So in the vehicle to grid project that we've been doing, that really was also design goal.   It worked.   And from then on, it just quietly   optimized every day what the battery was doing.   We did, however, in that project, think you might want an override button. You might not want to   meddle with what exactly the optimization is doing, but   it's nice, of course, in an optimized system to have the ability to say, just stop what you're doing or   the other button says,   charge the car now.  

I don't care about the price or whatever.   The idea is that these buttons are almost never used, but they should be there. So I think that end users who live in these   built environments should really not be involved that much,   and it's more the   servicing   around that. So Flex Measures   needs to be hosted, and we need to   check that that data is always flowing   and check with every error that might   come up in in the scheduling.   And you have some companies that you call energy service companies.   So a lot of them, for instance, have these more traditional   energy management systems I talked about earlier.  

And   I think that they would be users of Flex Measures.   If we don't do it for them, they would really   build it into their monitoring pipeline,   have a KPI and reporting pipeline to regularly check if that's happening correctly.   1 thing that's nice to check is, the universal   smart energy framework.   It's called Yusef.   My cofounder was on the design committee, and they talked about the different roles in modern smart energy system.   You mentioned the prosumer already.   I mentioned the energy service companies, and you also have companies they're called aggregators.   They   aggregate the energy flexibility   from   a lot of customers   into their virtual power plant and and just trade that flexibility   on markets.  

I think you could build energy aggregator using FlexNetters,   but that's not what we do. We use it for   optimization behind your meter.  

[00:22:44] Unknown:

As far as the actual   implementation   of Flex Measures, I'm wondering if you can talk to some of the   design considerations that went into it, some of the challenges of being able to   flexibly manage scale of inbound data and the types of requests that you're getting.   And also, I'm interested in maybe some of the capabilities of being able to federate across different installations of flex measures and, you know, route different decision requirements or different data points based on different kind of inbound information?  

[00:23:16] Unknown:

So from the beginning,   a few years ago,   I think, I didn't mention that before. But the first   project where Flex Measures really got its first development   time, that was with a South Korean customer called A1 Engineering.   They wanted to do simulation   first of a of a system with a lot of smart charging,   but then also move to   real time operation in a later phase. So this is   the main design idea   that has been there from the start that you can use   something   in reality in real operation,   but it's also useful   initially in a phase where you are simulating what might actually happen.   And that's still very relevant.  

So we're doing a simulation with a bigger real estate company right now. That's their first step. I think that that main design goal still stands.   Now I haven't used multiple Flex Measures nodes and then federated the decisions yet. So I think   that could be built, like, a lower level of Flex Measures instances, basically, governing,   smaller installations.   And another 1,   maybe trying to aggregate. As I mentioned earlier, the it's a concept of the flexibility aggregator   and aggregate across   those behind the meter installations that could be a higher level of extensions   instance. Yeah. Now we haven't done that yet.  

We consciously   can decide not to, you know, let the volume of data grow too large.   So for instance,   you might run into projects where they have 10 second   interval readings.   But to make the kind of decisions that Flex Measures needs to make, let's say, on an hourly basis or 15 minute basis,   you don't need to read in the finely   structured data. It would be best if that other system gives us already aggregated   data. For instance, in in 5 minute steps, that'll be fine. And then the amount of data isn't growing that much,   but we're already looking into that. So we   our data is stored in, Postgres.  

Of course, that we also made that choice because you have so much support for   growing bigger with your databases and have all these clusters, clustering strategies, and sharding.   And, also, of course, there's interesting   technology built upon Postgres. I think 1 of called Timescale DB that really focuses only on the time series   aspect.   And 1 thing that we considerably   put some effort in is to model   this data because it comes, as you said, from different sources.   There might be a company that   installed the sensors   at the factory or is responsible   at the houses to measure what's happening.  

So weather forecasting   service. So we use, open weather map.   Flex measures can read data from there.   The European transmission system operators have a service   in open transparent API as well. So you got multiple sources.   They all send you data that might be   about different things.   It has   different time zones.   It's crucial to maybe know when they know about that.   So not only the time that they're talking about,   1 point in time with 1 measurement,   has 1   date time, of course, about the time that you're talking about. But we're also interested in the daytime when they said this would happen.   If it's before the time, this is a forecast. If it's after the time, it's a measurement   that can be quite useful when you write optimization algorithms.  

All this led us to write our own,   basically,   time series data model. We called it Timely Beliefs.   It's actually an extension   on pandas. So what it makes possible is that you have a pandas data frame   or basically a class that extends from Pandas data frame, and we call it the beliefs data frame.   We wrote a lot of functionality around that that you can, you know, merge these data frames or change the resolution or whatever. It's a very interesting project.   They took a lot of our time, and now we're profiting from that within Flex Measures to really understand all the time   what that data really means.  



[00:27:32] Unknown:

As far as the kind of architecture of the project, digging a bit more into that, I'm also interested in understanding the   kind of extension points that you've built to be able to allow   different users of Flex Measures to adapt to different data sources or   being able to extend the types of decisions and machine learning capabilities that they want to use or the types of predictions that they want to generate   and maybe a bit more of the kind of data integration and kind of predictive capabilities that you've built into it? Flex Measures is extensible  

[00:28:05] Unknown:

with plugins   done last year, and we are using that ourselves   because often   every project has these differences,   different data sources, or or needs.   Often, it's also the visualization   that you want to adapt to a customer.   That's very important   to the whole architecture that you can add your own   API endpoints even,   your own visualization, and and we have a command line interface.   You can even add your own   command   line interface commands.   For instance, 1 that regularly   gets data from some other open API,   and you want to run that hourly in a Chrome job as a Flex Measures command.   Flex Measures actually is built on Flask,   and Flask has some nice capabilities for that.  

Of course, for APIs, but also the it integrates with click. So command line interfaces are, nice to write.   And with the Flask blueprint   technology, it was   quite a joy to write our 1st plug in   architecture. So what you do in the plug in is mostly   add a blueprint   to flex measures.   Flask has a CLI   command, and we renamed that to flex measures.   So,   yeah, it all ties back into what I said earlier. We want flex measures to be usable by software developers. And then,   of course, APIs are super important, but also the command line interface, I think, is is super important. We wrote a   tutorial how you would use Flex Measures within a few minutes, you know, just to show you   within a few minutes what you might be doing,   which is based on Docker. So you don't need to install all these things. Just pull the Docker image. And then that tutorial is   a few   calls to the command line interface   about modeling that couple assets that you have a battery, for instance, loading in some data   from a CSV file and asking Flex Measures to compute a great   optimized schedule for the next day. That's all command line interface command. That's what I really like.  

Optimization   in the background   happens   I think we use pyomo.   It's also a very powerful   library for for these kind of things.   And we used there was some sklearn   that we used. But recently,   we've picked up sktime,   which is really   a package focusing on time series   forecasting, doing smart things around that, and it's very nicely done, it seems.  

[00:30:37] Unknown:

As you have been   building Flex Measures and   iterating on the problem space and also with the   donation of it to the LF Energy   Foundation.   I'm wondering what are some of the ways that the design and goals and scope of the project have changed or evolved since you first started working on it? I said earlier that,  

[00:30:58] Unknown:

actually,   the overall design goal to make both simulation and optimized real time operation possible from the same code base is still intact.   I liked open source in Flex Measures. It it happened roughly,   well, less than 2 years ago. I we hit that button in GitHub   in a couple months later, donated to the Linux Energy Foundation.   I liked that that pushed us to even more transparency   and even more having a focus on code quality   to make sure that   Blake 8,   MyPy,   and Black always run. It's always looking good.   And documentation   got a lot of boost.  

And, also,   in the meantime, we did once make big changes to the data model. When we worked for industry customers,   we noticed that our initial data model was a bit too simplistic.   I think we modeled   power,   weather data, and prices.   And if you go to industry, you quickly realize they want to measure a lot of other things that all have   some influence on your   outcomes.   For instance, you might be measuring steam flow,   and  

[00:32:11] Unknown:

that wasn't possible. So we took a lot of time to make our data model more generic. Now we can model anything. Another interesting aspect of the kind of open sourcing question, and it's addressed a little bit by your donation of the project to the LF Energy Foundation. But I also know that you are running a business where Flex Measures serves as a kind of central portion of what you're providing   at CITA. And I'm curious if you can talk to some of the ways that you think about   which pieces of your work you want to put into the open source Flex Measures project and what are some of   the other kind of capabilities or layers that you're building on top of that that are part of the kind of commercial and business focused offering?  



[00:32:51] Unknown:

It's a very interesting time to have a commercial open source   start up.   I was recently involved in an accelerator,   on the US side.   I had to attend at night, always at 11 PM.   But   there's really some investors in the US   that are looking for open source companies to invest in   because you got, in the last couple of years, some really interesting examples,   mostly on the database   developer tooling side.   So, obviously, in the energy sector, this is quite new. But it's clear that you can   run a business even if you're   very open.  

LIMX Energy Foundation advised to use permissive licenses. So in this case, we use Apache 2.   You could also use MIT, for instance,   over a Copyleft licenses because the idea is to just, you know, open the doors wide to   growing the community,   letting other people use the tooling you've you write. Even if they don't re even tell you about it, they should have no doubts that they can use it freely   because you want to just grow, grow, grow, grow. That's after the Linx Energy Foundation told us that they use those licenses.   I also heard it from this venture capitalist accelerator program. It's a very interesting   vision on innovation, on open innovation that this might be 1 of the future scenarios that comes true. Right? That the pool of open source   innovation becomes   as big in other sectors has become in databases   and developer   tooling. Nonetheless, well, we have made Flex Measures possible to create plug ins.  

And   oftentimes, the customer says, well, if you're developing something here about our processes, I don't want that to be public. Or, if I pay you to do something, I don't want it to be public. And with the plug ins, that's very possible, of course. Then you have   Flex Measures as an open platform.   And, also, Timely Believes is open,   and we occasionally   open up other projects that we work on. For instance, I mentioned that, Home Assistant plug in in the in the vehicles to grid   project that's open.   I think we have a couple   closed repositories   lying around right now with work for customers.  

It's something that's maybe not ready for the world.   It will always happen.  

[00:35:05] Unknown:

I'm also interested in understanding a bit about given the kind of predictive and machine learning focused capabilities   that are part of the Flex Measures platform   and the fact that you are   building   predictions about and   energy capacity,   what are some of the   challenges that you face in terms of accounting for whether to bias towards false positives or false negatives, and what levels of uncertainty you're willing to accept in some of those predictions and decisions that you  

[00:35:40] Unknown:

make? Yeah. That is a very interesting topic to go into.   It's difficult to answer   without having even more experience. We had   a project trying to find out if   forecasting should be done with something very sophisticated. You know? You can use linear regression to forecast, let's say, the solar production,   RIMO models,   but But you can also use a neural network.   We found that the the neural network isn't   really better   that much better, and you don't know what's going on. And just recently,   I saw a paper come by   that some researchers even confirmed that. So for time series forecasting,   currently, there doesn't seem to be a reason to become very sophisticated.  

What we did do was   we saw that   there is algorithms   which tell you about the uncertainty   that lies in the prediction. So it's mostly,   ensemble   algorithms like random forest.   They have, you know, a lot of forecast and then make a a mean forecast, but they tell you how far all of the others   were distributed.   And   that information can potentially, of course, be useful for exactly your question that you know that some forecasts are   shaky,   and you might decide   for a scheduling algorithm that there's a policy of not acting in those situations.   And that's something I didn't mention about the timely beliefs data model. 1 column in the belief data frame is the uncertainty. So how certain   forecast, for instance, would be. And it reminds me of a setting we have in a scheduling algorithm right now.  

For the battery charging,   1 property you need then is, round trip efficiency.   So   you can discharge   a battery.   Let's say, you discharge 1 kilowatt hour   and then you charge that back.   Let's say the other 1 around. You charge it   first and then you discharge it,   and you've made some arbitrage because your 24 hourly prices had this nice difference   or that some difference. But the battery is not 100% efficient,   so you end up maybe that it's 80% efficient,   could become 90% in some batteries.   So you're losing some of that power in that process.   And   it's very important to model that so that the scheduling algorithm then sometimes does nothing.  

Price difference difference is too low,   then it doesn't make sense to do all that. So sometimes the schedule is to do nothing even if there are some price differences. And I see the   policy of doing nothing if the uncertainty is too high, and that's a similar thing.  

[00:38:20] Unknown:

In your work of building flex measures   and working with different companies to help implement it and   providing it to the community and working with people who are using it for their own purposes. What are some of the most interesting or innovative or unexpected ways that you've seen flex measures applied?  

[00:38:36] Unknown:

There's not that much,   but there was a project where flex measures wasn't even being used for any scheduling,   which is   basically the end goal of using Flex Measures is to have a schedule and   then change the times when you do use energy.   And in that case, it was a project where it was   just so useful to   use Flex Measures to just be a plain energy management system,   getting all this data together with data pipelining being done quite fast,   but then not doing forecasting but in doing something else. There was a different algorithm that was basically some advanced analytics.   And it didn't you have to use electricity. It was about gas consumption in in industry.  

So that is possible, but that's not where I want to take it.  

[00:39:22] Unknown:

In your work of building the project and building a business around it and donating it to the community, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?  

[00:39:33] Unknown:

Something that you have to learn on your own is how long it takes to integrate   data   that constantly flows and that has these   intricacies that I mentioned earlier about time series   that you could get slightly wrong. It could be   you get the time zone slightly wrong or   somebody   sends you data that isn't localized.   You might   be 1 hour off half of the year.   Your whole schedule will not be good.   They also might, send you data that gets the daylight saving time wrong, and then there might be 2 hours in the year where you fail when daylight saving time is switching back and forth.   And, yeah, the sheer amount   of   attention and work that gets into that   is is quite staggering. We had an industry client with a partner who does the censoring,   and they didn't have the API ready for us. So   it was   not what I had expected in 2021 from that kind of player. What we could get from them was emails with Excel spreadsheets,   And we decided that they send us 3 per day. So every 8 hours, we have we wrote a Flexner's plug in that has its own command line interface command to just handle that email   and get the data in.  

Yeah. You can imagine that that prolongs that kind of pilot   considerably.  

[00:40:57] Unknown:

Yeah. It's definitely remarkable   how much   of modern   data engineering and data management still relies on CSVs and FTP.  

[00:41:07] Unknown:

Yes.   Absolutely. And   if you look at the energy sector as a whole that these kind of experiences,   1 dimension   towards the realization that there will not be 1 end to end solution that can be used across the whole country or across a continent that that will just take over. We have to   build a lot of   unique end solutions.   Let's call them apps   that work differently in different energy systems, countries,   cultural contexts,   sectors.   And I think   flex measures can be a basis for all of that, but you always need to   put something on top that really fits   to that situation.  



[00:41:51] Unknown:

For people who are interested in being able   to gain some more insight and potentially act on the different aspects of energy usage, whether it's at the residential or industrial or grid scale? What are the cases where Flex Measures is the wrong choice?  

[00:42:06] Unknown:

I've mentioned that earlier, but there are some people who have to act within seconds.   And that's usually if you're really taking part in this first   reserve mechanism   reacting to balancing problems,   frequency things.   And   we have put a lot of focus on flex measures to be developer friendly and correct about the data so that developers really understand what they're building.   And we haven't made it, you know, a Rust project   that lets you do all that within seconds.  

[00:42:37] Unknown:

As you continue to build and iterate on the project   and continue to iterate on the business that you're building on top of it, what are some of the things you have planned for the near to medium term or any particular problem areas that you're excited to dig into?  

[00:42:51] Unknown:

Yeah. So first, about the FlexVicious project,   I think   what we really could do with that project is to support   the 2 groups of users that I see   in what they are   doing as a as a first steps. So I see the first group as being the 90%   users   who have a use case that   really happens   a lot in the world, and Flex Measures should support out of the box   that the optimization   for that use case can be done within, you know, a few hours.   Data integration plus having that optimization work should be a few hours. So that that's a a little bit of a benchmark.   Let's say, as an example,   somebody has solar on the roof and there's somehow   a battery, a home battery, or an electric car,   and they have a dynamic tariff.  

That might happen a lot in the next years across the world, something like that.   And that should just work. And,   basically, that does work now. What we have to do is documentation,   write an excellent tutorial   so that   everybody could do that.   And there's probably a list of 4 or 5 more of these   situations   that should just work.   And the other group of users are the power users   that really have a unique problem and may come with their unique   idea of solving it.   And then I'm talking mostly about the algorithms.   Right now, you can already   plug in your own scheduling algorithm.  

Or to be honest, at the time of recording, that is a PR I'm   having reviewed.   But that's really important for some people. That's researchers,   certainly.   That's their whole line of work to come up with better algorithms.   Maybe find that neural network algorithm that really does make the difference.   But you see that also in some companies really have   certainly an industry that just have their unique challenge. And they should be using flex measures,   you know, as is and then just change that 1 thing that there's   a common interface   with how   the question to make a new schedule is described.  

But they could, in the back, just switch out the algorithm that is used to make the schedule.   I think these 2 groups of users deserve   each of their own dedicated   tutorials by us to really make it clear, to make it a joy to do so much as our work. I mentioned 2 sectors that we're working on.   The industry   world is very interesting and challenging,   but it's also very diverse.   Right now, those environments,   what excites me the most, and I'm I'm talking to   companies there that really understood the scale   that I mentioned earlier, they have renovate or we modernized the whole portfolio   within a very short amount of years. So scale is their big topic.  

And the third thing I didn't mention is microgrids.   So microgrid is your own little electricity grid   that is connected   maybe to the main grid   or not,   completely run on its own. And that's a situation   that we might see all across the world. So there might be   more   self serving neighborhoods   that are not completely   dependent on the grid in Western countries, but certainly in Africa, that's something that's just the 1 way to go for a lot of regions.   There are some interesting companies there that I hope to do more with in the near future because Flex Measures should be able to   basically   run a small village.  

You might think there's too many assets, but what really matters is the flexible assets. So that's where   that defines how much flex flex measures has to do, how many flexible things are there.   Emobility   is making huge steps in Africa, certainly.   We're   hoping to be involved in a project in Victoria Lake where all the fishers get upgraded to electric boats, for instance,   And they use energy for productive uses,   like cooling the fish they caught,   water purification.   That's all flex it's all something you can shift around in the day. And then flexiators can really help, for instance, in the design phase already to not buy oversized   equipment.  



[00:47:14] Unknown:

I don't think you will become rich in Africa, but that's something I'd really like to be involved in. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And so with that, I'll move us into the picks.   This week, I'm going to choose the show She Hulk from the Disney plus. Just a fun and goofy show. Some interesting storyline there, so I've been enjoying watching that. So I recommend that for people who are looking for something to do in the evening. And with that, I'll pass it you, Nicholas. Do you have any pics this week?  

[00:47:46] Unknown:

Yeah. I'm watching a very nice German series on Netflix,   and it's called KLEO,   k l e o.   It has a bit of kill bill   visuals because it's an ex   agent from the East German   Ministry of Defense who got thrown into jail. And she   comes out of jail right when the wall has been torn down and tries to find out what happened, who did it.   It's not just Kill Bill, but it's also charming and funny.  

[00:48:14] Unknown:

Thank you very much for taking the time today to join me and share the work that you're doing on flex measures. It's definitely a very interesting project in a very important area of effort. So I appreciate all the time and energy that you and the other contributors are putting into that. And I hope you enjoy the rest of your day. Thanks, Tobias. It was a joy to be here.   Thank you for listening. Don't forget to   Visit the site at pythonpodcast.com   to subscribe to the show, sign up for the mailing list, and read the show notes. And if you learned something or tried out a project from the show, then tell us about it. Email host hostpythonpodcast.com   with your story.  

And to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.