Accidents, such as a plane crashes, usually have a complex chain of causes.
And as a rule, I don’t like second-guessing the experts who do accident investigations.
The entity that will eventually provide the definitive analysis of the blackout is ENTSO-E, the European Network of TSOs.
I try to be a good Bayesian.
That means I’m willing — even plan on — adjusting a hypothesis as new evidence comes in.
So in that spirit, I’m going to look at what we know about Monday’s blackout in Spain and Portugal.
But first, the Iberian blackouts should serve as a reminder of just how vital the electrical grid is in the contemporary world. For example:
People were trapped in between floors in elevators.
Trains — such as the intercity one in the photo above — coasted to halt. Airports and air traffic control went dark a least for a few minutes.
Subway trains in Madrid stopped in the tunnels, and passengers had to hike out along the tracks.
It could have been far worse. The death toll was thankfully low: five. A woman in Valencia died after her oxygen concentrator stopped.
Four other deaths resulted from post-blackout coping. Three members of a family in Galicia died from carbon monoxide poisoning after using an emergency generator indoors. A man in Madrid died in a fire caused by his candle.
For everybody else, there was inconvenience: cell phones, ATMs, and cash registers stopped working, and so on.
In a chart, here’s what happened about one-half hour after Noon (1033 GMT / 12:35 PM CEST) on Monday :
The timing of the event is our first big clue.
Noon: solar.
Sure enough, late Monday, Red Eléctrica de España, which manages the Spanish national grid, said it was “very possible” that the fault originated from a solar farm or farms (note the plural) in the country’s south-west.
Which is not too helpful in fingering an individual culprit. That’s where most of Spain’s solar farms are. Some 70% of Spain’s solar generation is in the southern provinces of Andalucia (24.3%), Extremadura (24.3%) and Castilla La Mancha (22.2%):
The statements by REE's System Operations Chief Eduardo Prieto are to date the only source of ‘inside’ knowledge about the causes of the blackout.
Monday evening, Prieto held in a conference call, in Spanish, with various reporters.
Javier Blas, an excellent bilingual reporter for Bloomberg, posted the notes he took during that call on X on Tuesday. I’ve reproduced them in this footnote1
Prieto also held a short press conference on Tuesday whose propose was to quash speculation about the blackout being the result of a cyberattack. It’s on YouTube (in Spanish) here.
In both statements, Prieto definitely said the ‘oscillations’ in the grid originated in southwest Spain:
Un evento en el sistema eléctrico compatible con la pérdida de generación en la región suroeste del sistema peninsular español fue identificado…
[Google translate]: An event in the electrical system compatible with the loss of generation in the southwest region of the Spanish peninsular system was identified
The geographical location of the initial disturbance is important: there’s an alternative theory that the original point of failure was on of one of the smaller interconnector lines between that runs between Perpignan, Spain, and Narbonne, France. That’s in the northeast.
Until there’s good evidence for it, I’m not buying the downed power line theory.
The grid-following inverters (GFL) used at solar farms are well-known to be capable of producing the ‘oscillations’ mentioned by Prieto of REE. GridRadar got a picture of the two bouts of oscillations (to the left) before the crash:
More about grid following versus grid-forming inverters (GFM) below.
Any accident will have a proximate cause and a list of ‘but for’ causes. The ‘but fors’ are usually more interesting, and typically form a long chain.
The proximate cause of the Space Shuttle Challenger exploding was a cracked O-ring that burned through.
But for the Challenger being launched —against advice — on a very cold day in January 1986, the O-ring would not have been brittle, and the Challenger would not have exploded.
The ‘but-fors’ for the Spanish blackout are many.
For ‘but fors’, it’s better to work backwards. These are the preexisting conditions which, in a chain, give the proximate cause its unexpected punch:
Spain produces a surplus of solar electricity, especially in the middle of the day.
Spain exports (I’m tempted to say ‘dumps’) its surplus solar to France. Here’s the chart of Spanish electricity exports to France a day ahead of the blackout:
So Spanish solar was presumably running full tilt. They were getting something for it from France.
Although perhaps not very much. On Sunday, when demand was low, electricity prices in France were negative. Demand on a weekday at noon will be higher, but of course solar supply will likewise be high.
There are only 5 transmission lines going from Spain to France, 4 old ones (2 x 380 KV and 2 x 220 KV) and one modern, high-capacity one (2,000 megawatts HVDC).
That’s ~3,700 MW of theoretical capacity.
The HVDC interconnect is close to maxed out, being used 84% of the time. It’s congested some 40% of the time.
It’s also underground, which rules out weather-related problems. It would take a backhoe to sabotage it.
Those grid constraints put the total operational capacity around 2,800 MW. You can see an excursion to 3,300 MW or so at 16:30 in the chart above.
Spain knows all this, and is planning to improve its ability to transmit electricity to France with a Biscay Gulf project, hoping to add 2,000 MW by 2028:
The islanding effect
As with any network, more connections work to diffuse anomalies. Inadequate interconnections turn a grid into an ‘island’, making it more susceptible to self-generated swings.
A very similar island effect happened in South Australia on 28 September 2016, when a thunderstorm took down several transmission towers on its tie line to Victoria.
The generation mix that day in South Australia — before the tie line went down — was 18% synchronous generation (330 MW), 48% wind generation (883 MW), and 34% import.
That was low enough synchronous generation that the South Australia grid then destabilized and went off the rails.
Frequency
In a bit of electromagnetic magic, generators on the grid at different locations come to run together in sync, like 10-people on the same bicycle:
In Europe, grid operators have a statutory requirement to maintain the frequency at 50 Hz ± 1%. That’s a very tight tolerance. Here it’s being breached in a December 2023 incident in the UK:
The grid running at too low or too high a frequency can damage devices connected to the grid. Most can sense this can sense when this happens and will disconnect themselves.
An imbalance of supply and demand on the grid works to change the AC frequency on the grid. If there is more generation than demand, the frequency will increase. If there is more demand than generation, the frequency will fall.
A key metric is called the rate-of-change of frequency (ROCOF). Its units are Hz per second. 3 Hz/s is a high rate of change.
Classically, devices serving an electrical load will automatically cut off if the frequency falls too low. This is called underfrequency load shedding (UFLS).
UFLS is meant to be a semi-orderly — if unpleasant to customers — thing the grid does to protect itself and keep portions of itself alive. That can obviate the dreaded ‘Black Start’ — powering the grid up from zero, which can take days.
Things generating power — such as solar farms — will also switch off to protect themselves if the grid frequency gets too erratic.
Synchronous inertia
Rotating devices on a grid have inertia. Because of their large size and weight, they resist changes to their speed of rotation.
Old-school power grids have synchronous inertia that comes naturally from the kinetic energy stored in the rotating masses of generators.
Those rotating masses can be pretty massive. Here’s a photo of those at the Diablo Canyon nuclear plant in California. Inside of each of those semicircles is 500 tons rotating at 30 times per second:
There are (relatively expensive) electronic things that can produce synthetic inertia.
Flywheels — that don’t generate electricity — are a mid-tech option that can create inertia.
Grid-forming inverters (GFM), usually combined with battery (BESS) systems, actively try to manage the grid’s inertia.
Of course, the algorithms in a GFM must be right, and speed is of the essence. That gets called fast frequency response (FFR).
Shrinking inertia
With increasing renewables penetration, grid inertia has declined. This from the UK:
Spain is now getting on average about 70% of its power from renewables. At the time of the blackout, it was 78%
By my arithmetic, not much more than 18% of Spain’s electricity was coming from technologies that use conventional rotating generators. I’m adding nuclear (11.63%) with natural gas combined cycle (3.37%) and guessing about hydro.
Spain has some hydropower (8% of its total installed generation capacity) and coal.
Of Spain's 5 hydroelectric generation sources, 3 were offline on Monday for maintenance. That, in hindsight, was clearly a mistake.
Spain in June 2020 shut down 7 of its 15 coal power stations, It now has 4 left. None were running at the time of the incident. Spain plans to end all coal-fired power generation by 2030.
The best info for Spain’s generation mix just prior to the incident comes also comes from Javier Blas:
My prime suspect: solar grid-following inverter (GFL) oscillation
We don’t know yet what caused the solar farm (or farms) in south-west Spain to go offline.
Electronic inverters at a solar farm are essential for converting the direct current (DC) generated by solar panels into alternating current (AC) that can be fed into the electrical grid.
Most inverters at solar farms are "grid-following," meaning they adjust their output to match the grid’s voltage and frequency. The inverter’s control system adjusts its output.
Solar farms typically have multiple inverters operating in parallel. These need to be synchronized and their control systems coordinated.
If the inverters’ (plural) response is too aggressive or poorly tuned, it can overshoot, leading to a cycle of overcorrections — that is, oscillations, which appear as voltage fluctuations at the connection point.
Some rapid changes in solar output — a passing cloud — can trigger the inverters to adjust quickly. If uncoordinated or overly responsive, this can initiate oscillations.
An additional wildcard is that the output filters on the inverters just might resonate with grid components at certain frequencies, amplifying the oscillations.
The stage is finally set. Action!
This was not a classic ‘not enough power’ blackout, at least at first.
While it turned into a classic underfrequency load shedding cascade, it didn’t start off that way
The problem not not with the load at noon, but with generation at noon.
There was a power loss from some device in response to the first of the two oscillations.
The Spanish grid should have been able to handle a single solar farm suddenly going offline.
And it was, at first. There was enough synchronous inertia on grid to dampen the first oscillation. The grid ‘recovered’ in milliseconds.
It’s an open question as to whether the ‘oscillations’ were contagious. That is, if the first led to the second.
Probably, it did. But it’s possible they shared some same external source. The same passing cloud.
The drop in the power supply when the first device went offline would have lowered the grid frequency.
More important, probably, was just that it altered it.
Some second device presumably sensed frequency instability on the grid and turned itself off for self-protection. That would have been about 1.5 seconds later.
The big event, 3.5 seconds later, was when the interconnectors between Spain to France shut themselves off. That ‘islanded’ Spain and ruled out recovery from French nuclear synchronous inertia.
A part of the French grid near the border was affected, but recovered quickly.
The Spanish grid did not. After the cascade completed, generation fell to zero.
Lessons learned
The obvious one, which I won’t belabor, is that Spain should never have causally assumed it could replace conventional generation with wind and solar.
One — slightly stupid — quick fix might be to lower the protection settings on generating devices on its grid.
The most likely fix, given that Spain has already invested so much in solar, will be more electronic gear than can provide synthetic inertia and fast frequency response (FFR).
The role of France is all this is interesting. With its nuclear power, France is turning into both Europe’s battery — and its sponge. I hope France is buying low and selling high.
I’d like to get this out quickly, so I’m going to cut myself off.
Javier Blas (@JavierBlas) on X, Apr 29 2025:
Red Eléctrica de España, the grid operator, offered the first and tentative timeline of the massive blackout that hit the Iberian Peninsula on Monday. The said they have ruled out a cyber-attack on the grid and denied that an atmospheric or weather event triggered the backout. On a conference call with reporters in Spanish, they presented the following chain of events:
1) at 12.33pm, the Spanish grid suffers an “event” akin to a loss of generation in the country’s south-west. It was “very likely” that the generation affected was “solar,” but grid officials cautioned they couldn’t say for certain yet.
2) in milliseconds, the grid self-stabilized and appeared to recover
3) however, about 1.5 seconds later, a second “event” akin to another loss of generation in Spain hits the grid, further destabilizing it. Grid officials didn’t say whether the first event triggered the second, but that’s quite likely.
4) about 3.5 seconds later, the instability in the Iberian Peninsula grid reaches a level that interrupts the border interconnectors between Spain and France (at maximum capacity, about 2.5 GW can flow through them).
5) Immediately after, a “massive” loss of renewable generation hits the grid (officials didn’t say why such a loss of renewables production occurred)
6) the cascading loss of generation further destabilizes the grid, forcing every producing unit - nuclear, gas-fired, hydro -- to disconnect, until the grid complete collapse. At the worst point, generation in Spain plunges to zero (initial data on Monday suggested that 10 GW remained, compared to the about 25 GW before the blackout, but the grid said on Tuesday the actual level fell to zero).
Important to note that the chain of events is very preliminary, and it isn’t a roo analysis. Red Eléctrica officials emphasized they lack enough data to make final conclusions.
All the notes from me, who listened to the call (it was in Spanish).
Will Bates here. Bloomberg’s editors in New York are rather notoriously pro-renewables. Javier Blas’s email to them with his notes above provides an interesting look behind the media curtain. Blas suggested a headline: “Let's call it ‘The first big blackout of the green electricity era’. ” Bloomberg ended up publishing it as an Opinion piece by Blas (link) that shows up as “Spanish Blackout Shouldn't Trigger a Green Retreat”. The on-the-page headline is “Lessons From the First Blackout of the Green Era” and the subhead is “Spain's blackout shouldn't trigger a retreat from renewables, but an upgraded grid is urgently needed for the energy transition.”