A couple of years ago, I published some work on dynamic simulations of combined transmission and distribution systems. I presented part of the work at a conference followed by an extended journal version. While the work was very interesting (😏), I was really surprised by how many people contacted me to get access to the test-system model that we used in the results section. It was a simple synthetic system carefully put together to test the simulation algorithm, but apparently, it was a hot commodity in our research community. This inspired me to write this blog post.
Why test-system models?
Researchers working in electric power systems (or, Smart Grids – a discussion for another time) know very well that to publish your work, you need good test-system models for validation and testing. Simply put, you cannot publish anything without having a Results section showing that your method works on some real/realistic/well-known test model. We belong to those disciplines that find it difficult to test or validate our work with real-world experiments. That’s because very few of us can go up to a system operator or generator/wind farm owner and say “Hey, let’s try this idea on your system. I’m 99% sure nothing will explode!”. You first need to have a solid theoretical background, supported by an abundance of simulation results and probably some small-scale experimental results. Then, you might have a chance of convincing the Industry to take a look. To make things worst, real test-system models are hard to come by. Very few people can have access to the real data of system operators for obvious security reasons (they are basically the blueprints of one of the most important national infrastructures).
Which test-system model?
So, how do “normal” researchers get to publish their work? For decades now, the IEEE test-system models have been the usual go-to for most of us. These models were set up a long time ago by IEEE committees and work groups – usually accompanied by detailed reports providing the data as well as some initial analysis. These systems have had a lot of toothing problems over the years (bugs, missing data, abnormalities, etc.). However, the IEEE brand adds to their authority and makes them the standard for validating and comparing research results. Of course, other models were proposed over the years by academic groups or individuals and due to their widespread use found their way into this reference group.
Most of these reference models are synthetic, loosely based on some existing systems. Some designed only for static problems (power-flow), fewer for reliability, some for voltage stability, others for transients (a small review is available here), and lately for optimization. Over the decades, researchers have “tortured” these models, tweaking and spinning to fit their research, translating from one format to the other – since we cannot decide and stick to a common modelling framework (yet another discussion for a later time), etc.
One can argue that there is no problem with modifying these systems if you can justify the modifications. Well, the problem can be summarized in one word: Reproducibility. In power systems, researchers are not required to publish their test systems along with the paper. So, when a paper states “We used the IEEE test-system XYZ but modified all the generator to use the AVR from paper , all the models to have a mix of loads from [2-5], and simulated with a our software SuperSim”, good luck trying to reproduce the results and comparing against yours!
The future of test-system models
So, how can we make things better? How can we make research more transparent and reproducible?
The solution sounds simple: we need to publish the test-models and data used for the acquiring results in open-access along with the code to reproduce the results. This way, anyone can reproduce our results and test their own techniques against ours. Several platforms make sharing data, code, or both, easy anymore. Anyone can publish their code on Github or Bitbucket and their datasets on platforms like Figshare and Zenodo. Other platforms even allow posting both the data and the source code in a single place, like Code Ocean.
The technological tools to make research in power systems reproducible and transparent are available. Yet, the most challenging task is changing the mentality of people. Starting from the journal editors to demand this transparency in their work, to the grant funders and down to the PhD students. It will take a lot of work, but I believe it’s a worthy cause for advancing science and achieving a truly Smart Grid.
Closing, the test-system model that I mentioned in the beginning of this blog post has been transformed into an open-source system and is free for everyone to use, share and compare. We hope it will provide a small push to the right direction. The write-up can be found in this paper.