shu/power-market-trading-docs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add topical primers: SCUC/SCED, newsvendor, stochastic optimization

Browse Source 
EN only for now; Chinese versions pending.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Renewable Trading Docs 2026-07-18 18:18:12 -04:00
parent 913d76b858
commit 0d22aed4b8
3 changed files with 251 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

59
primers/newsvendor_en.md Normal file
View File

@ -0,0 +1,59 @@
# The Newsvendor Problem: A Compact Primer for Electricity Markets
## 1. The Model in One Paragraph
A vendor must decide how many newspapers `q` to stock before knowing the day's demand `D`, a random variable with distribution `F`. Each unit of unmet demand costs `c_u` (the underage cost: lost margin, penalty, or replacement at a higher price); each unit of unsold stock costs `c_o` (the overage cost: purchase price minus salvage). The vendor minimizes expected cost:
```
minimize over q : E[ c_u · (D − q)⁺ + c_o · (q − D)⁺ ]
```
This is the smallest interesting stochastic optimization problem: one decision, one random variable, one period, and — crucially — a closed-form solution that carries the central lesson of the entire field.
## 2. The Critical Ratio: Why the Answer Is a Quantile, Not a Forecast
Take the derivative of expected cost in `q` (or argue marginally: the last unit stocked is worth it if the expected underage it avoids exceeds the expected overage it risks). At the optimum, the marginal benefit `c_u · P(D > q)` equals the marginal cost `c_o · P(D ≤ q)`, giving the **critical ratio** (also called the critical fractile):
```
F(q*) = c_u / (c_u + c_o) ⟹ q* = F⁻¹( c_u / (c_u + c_o) )
```
Three things to notice. First, the optimal decision is a **quantile** of the demand distribution — the mean appears nowhere. If underage is three times as painful as overage, stock at the 75th percentile; if overage dominates, stock below the median. Second, the deterministic habit of "plan against the forecast" is optimal only in the knife-edge case `c_u = c_o` *and* a symmetric distribution — asymmetric costs or a skewed distribution make the point forecast systematically the wrong answer. Third, the solution needs the whole distribution (or at least the right quantile of it), not a point estimate: this is the precise sense in which uncertainty quantification has decision value.
The expected cost at the optimum also has a clean interpretation: it grows with the *spread* of `D` around the chosen quantile. Forecast improvement (variance reduction) and decision improvement (choosing the right quantile) are complementary but distinct levers — a lesson organizations routinely blur.
## 3. The Newsvendor in Electricity, Usually Unnamed
Electricity markets are full of one-shot quantity decisions made before uncertainty resolves and settled asymmetrically afterward. Each is a newsvendor once you identify `q`, `D`, `c_u`, and `c_o`.
**Renewable day-ahead offering.** A wind or solar producer offers quantity `q` into the day-ahead market at price `π_DA`, then settles the deviation between actual production `D` and the offer at imbalance prices. In a dual-price balancing market, under-delivery is bought back at a price above `π_DA` (unit underage cost `c_u` = buy-back premium) and over-delivery is sold at a price below it (`c_o` = sell-down discount). The optimal offer is exactly the critical-ratio quantile of the production distribution — a producer facing harsh shortfall penalties should offer conservatively (a low quantile), and vice versa. This result, due to the trading literature of the 2000s (Bremnes; Pinson, Chevallier, and Kariniotakis), is why quantile forecasts, not point forecasts, are the deliverable that matters for renewable traders, and it is the entry point for the richer multi-market offering models in the stochastic optimization primer.
**Reserve sizing.** How many MW of operating reserve should the system hold against net-load forecast error and outages? Holding an unused MW costs the reserve procurement price (overage); being a MW short costs expected load shedding at the value of lost load, or expensive emergency actions (underage). With `c_u` in the thousands of $/MWh (VOLL) and `c_o` in the tens, the critical ratio sits far in the right tail — which is exactly why reserve requirements correspond to high quantiles (e.g., 95th–99.9th percentile) of the net-load error distribution. Probabilistic reserve methodologies and ramping-product demand curves in modern ISOs are institutionalized newsvendor logic.
**Operating reserve demand curves (ORDCs).** Run the logic in reverse: instead of fixing a quantity, price each increment of reserve at its expected marginal value, `VOLL × LOLP(r)` — the value of lost load times the probability of losing load given reserve level `r`. ERCOT's ORDC is the canonical implementation. The demand curve *is* the newsvendor first-order condition traced out over quantities.
**Retail and fuel procurement.** A retailer contracting energy forward for uncertain customer load faces spot-price exposure on the shortfall and resale losses on the surplus; a gas-fired generator nominating day-ahead pipeline capacity against uncertain dispatch faces imbalance penalties both ways. Both reduce to critical fractiles of load or dispatch distributions, complicated in practice by the correlation between quantity and price (you tend to be short exactly when prices are high), which shifts the effective quantile — a first taste of why joint distributions matter.
## 4. The Forecasting Connection: Pinball Loss Is Newsvendor Loss
The link between this decision problem and modern probabilistic forecasting is exact, not analogical. The **pinball (quantile) loss** used to train and evaluate quantile forecasts,
```
L_τ(q, d) = τ·(d − q)⁺ + (1 − τ)·(q − d)⁺ ,
```
is the newsvendor cost function with `c_u = τ` and `c_o = 1 − τ`; minimizing its expectation yields the τ-quantile. So a forecaster minimizing pinball loss at `τ = c_u/(c_u + c_o)` is *directly* solving the trader's or operator's decision problem — forecasting and optimization collapse into one step. This observation underlies the current "decision-focused" or "predict-then-optimize" research agenda: rather than fitting a distribution and optimizing against it, train the forecast on the loss the decision actually incurs. The newsvendor is the one case where this program has a complete, closed-form answer.
## 5. Risk Aversion and Other Extensions
The classical model is risk-neutral. Real participants care about tails, and the standard fix is to replace expected cost with a **CVaR** or mean–CVaR objective. The elegant result: for the newsvendor, CVaR optimization again yields a quantile decision, just at a *shifted* critical ratio — risk aversion tilts the offer toward whichever side has the worse tail. Under the Rockafellar–Uryasev formulation the problem remains a small LP given scenarios, previewing how risk enters the larger models in the stochastic primer without breaking tractability.
Other standard extensions each add one ingredient and point toward a larger literature: **price-dependent demand** (the price-setting newsvendor) toward joint pricing/quantity decisions; **quantity–price correlation** toward portfolio and hedging models; **multi-period inventory with carryover** toward base-stock policies and, in electricity, storage arbitrage (a battery's charge decision against uncertain future prices is a newsvendor chained through time); **unknown distribution** toward the distributionally robust newsvendor — Scarf's 1958 closed-form solution for the worst distribution matching a given mean and variance is one of the earliest DRO results and still a touchstone.
## 6. The Bridge to Two-Stage Stochastic Programming
Formally, the newsvendor *is* a two-stage stochastic program: `q` is the here-and-now decision, and the recourse problem is trivial — buy the shortfall at `c_u` or salvage the surplus at `c_o`, with the recourse cost `Q(q, D) = c_u(D − q)⁺ + c_o(q − D)⁺` available in closed form. Everything in the full stochastic optimization toolkit is this structure with the trivialities removed: replace the scalar `q` with a vector of unit commitments, replace the closed-form recourse with a dispatch LP per scenario, and the closed-form quantile solution disappears — but the *economics* survives intact. Stochastic unit commitment is, at heart, the system operator asking the newsvendor question in a thousand coupled dimensions: how much slow, cheap capacity to commit now, given asymmetric costs of being long (startup and no-load costs wasted) versus short (peakers, scarcity prices, load shedding) against the net-load distribution. Whenever a stochastic model's output seems opaque, the newsvendor is the right sanity check: identify the implicit `c_u/c_o` ratio, and ask whether the model is choosing a sensible quantile.
## 7. Suggested Reading
The model traces to Edgeworth's 1888 bank-reserve analysis and Arrow, Harris, and Marschak (1951) in its modern form; Porteus's *Foundations of Stochastic Inventory Theory* and the survey by Qin et al. (2011) cover the classical theory and extensions. Scarf (1958) is the distribution-free origin. For electricity: Bremnes (2004) and Pinson, Chevallier, and Kariniotakis (2007) establish the quantile-offering result for wind; Morales, Conejo, Madsen, Pinson, and Zugno, *Integrating Renewables in Electricity Markets*, chapter 7, gives the full trading treatment; Hogan's ORDC papers develop the scarcity-pricing incarnation; Rockafellar and Uryasev (2000) supply the CVaR machinery reused throughout.

88
primers/scuc_sced_en.md Normal file
View File

@ -0,0 +1,88 @@
# A Primer on SCUC and SCED: The Optimization Engines of Wholesale Electricity Markets
## 1. What These Acronyms Actually Are
Security-Constrained Unit Commitment (SCUC) and Security-Constrained Economic Dispatch (SCED) are the two optimization problems that clear organized wholesale electricity markets in the United States and, in closely related forms, around the world. Every locational price you see published by PJM, MISO, CAISO, ERCOT, SPP, NYISO, or ISO-NE is a byproduct — literally a dual variable or its mixed-integer analogue — of one of these two problems. Understanding them means understanding how roughly two-thirds of US electricity is scheduled and priced.
The two problems answer different questions on different timescales. **SCUC answers "which units should be on?"** — a commitment decision involving startup costs, minimum run times, and hours of lead time, solved primarily in the day-ahead market. **SCED answers "given the units that are on, how much should each produce right now?"** — a dispatch decision solved every five minutes in real time. The "security-constrained" prefix, common to both, means the solution must respect the transmission network not only under normal conditions but also under a defined list of contingencies: the system must survive the sudden loss of any single major element. This is the engineering criterion known as N-1 security, and embedding it in the market optimization is what distinguishes modern LMP markets from simple single-node auctions.
## 2. The Physical and Institutional Setting
Three physical facts shape everything. First, supply and demand must balance continuously — frequency deviates within seconds of an imbalance, and large deviations cascade into blackouts. Second, power flows distribute themselves across the network according to Kirchhoff's laws, not according to contracts; you cannot route power, so congestion on one line is determined by injections everywhere. Third, generators are lumpy and inflexible: a large thermal unit may need 8–24 hours notice to start, cost tens of thousands of dollars per startup, be unable to run below 40–50% of capacity, and be limited in how fast it can ramp.
Institutionally, an Independent System Operator (ISO) or Regional Transmission Organization (RTO) runs a two-settlement market. The **day-ahead market** closes in the morning of the day before the operating day: participants submit offers (generators) and bids (loads and financial traders), and the ISO runs SCUC to produce hourly commitments, schedules, and day-ahead LMPs at which those schedules are financially settled. In the **real-time market**, SCED runs every five minutes against actual conditions, producing real-time LMPs; deviations between real-time delivery and day-ahead schedules settle at real-time prices. Between the two, a **reliability unit commitment (RUC)** pass re-runs a SCUC-like model against the ISO's own load forecast to commit any additional capacity needed for reliability that the financial day-ahead market did not procure.
## 3. The SCED Formulation
SCED is the simpler problem and the right place to start. In its DC-linearized form it is a linear program:
```
minimize Σ_g C_g(p_g) (offer-based cost)
subject to Σ_g p_g = Σ_n d_n (system power balance) → dual λ
P_g^min ≤ p_g ≤ P_g^max (dispatch limits of committed units)
|p_g − p_g^prev| ≤ R_g · Δt (ramp limits over the interval)
−F_k ≤ Σ_n PTDF_{k,n} · inj_n ≤ F_k (base-case line flow limits) → duals μ_k
(same flow constraints under each contingency c in the monitored set)
reserve constraints (co-optimized regulation, spin, non-spin)
```
The cost functions C_g are piecewise-linear, built from each generator's offer curve (up to ten price-quantity segments in most ISOs). The network is represented through Power Transfer Distribution Factors (PTDFs) — linear sensitivities mapping nodal injections to line flows under the DC approximation, which neglects losses and reactive power (losses are typically added back through marginal loss factors). Contingency security enters by adding flow constraints computed with post-contingency sensitivities (LODFs, line outage distribution factors): for each monitored line k and each contingency c, the post-outage flow must remain within the line's emergency rating. In practice only a screened "active set" of binding or near-binding constraint pairs is included, because the full set (thousands of lines × thousands of contingencies) is enormous and almost entirely slack.
Modern SCED co-optimizes energy with **operating reserves and regulation**: a unit's headroom can be sold as reserve instead of energy, and the LP chooses the split that minimizes total cost against reserve demand curves. It typically also runs in a short **look-ahead** mode (several 5–15 minute intervals) so that ramp constraints are managed proactively rather than myopically.
## 4. Locational Marginal Prices
The elegance of SCED is that its dual solution *is* the market price. The LMP at node n decomposes as:
```
LMP_n = λ + loss component − Σ_k μ_k · PTDF_{k,n}
```
that is, the system marginal energy price, plus the marginal cost of losses at that location, plus a congestion component built from the shadow prices of binding transmission constraints. When nothing is congested, all nodes see (nearly) the same price. When a constraint binds, prices separate: nodes whose injections would relieve the constraint are paid more, nodes that would aggravate it are paid less — the price system internalizes Kirchhoff's laws. Because SCED is a convex LP, these prices support the dispatch: every generator dispatched between its limits is exactly indifferent at the margin, and no committed generator loses money on its incremental energy at the LMP. Congestion rents collected by the ISO (load pays more than generation receives, in aggregate, when lines bind) fund Financial Transmission Rights, the hedging instruments for congestion risk.
## 5. The SCUC Formulation
SCUC wraps the dispatch problem in binary commitment decisions over a 24–48 hour horizon:
```
minimize Σ_t Σ_g [ SU_g · v_{g,t} + NL_g · u_{g,t} + C_g(p_{g,t}) ]
subject to u_{g,t} ∈ {0,1} (on/off status)
v_{g,t} ≥ u_{g,t} − u_{g,t−1} (startup logic)
minimum up and down times (once on, stay on ≥ UT_g hours, etc.)
P_g^min · u_{g,t} ≤ p_{g,t} ≤ P_g^max · u_{g,t}
ramp limits, startup/shutdown ramp rates
hourly power balance, reserve requirements
base-case and contingency transmission limits
```
The objective now includes **startup costs** (SU, often dependent on how long the unit has been off — hot, warm, cold starts) and **no-load costs** (NL, the cost of running at zero net output), alongside incremental energy. The binary variables and minimum up/down time logic make SCUC a large mixed-integer linear program: a large RTO's day-ahead SCUC spans roughly 1,000+ generators, 36–48 hourly periods, tens of thousands of network constraints after screening, and on the order of 10⁵ binary variables. ISOs solve it within strict market timelines (roughly 1–4 hours of wall-clock budget) to an optimality gap tolerance, typically around 0.1%.
Several modeling refinements matter enormously in practice. **Combined-cycle units** are modeled with configuration-based transitions (1x1, 2x1, etc.), each configuration its own pseudo-unit with transition logic. **Tight MIP formulations** of the up/down-time and ramping polytopes (e.g., the Rajan–Takriti facets and their successors) dramatically reduce solve times — formulation strength has been as important as solver progress in making large SCUC tractable. And **demand bids, virtual bids (INCs/DECs), and price-sensitive load** enter the day-ahead problem symmetrically with generation, which is what makes the day-ahead market a genuinely financial, two-sided auction rather than a pure engineering schedule.
## 6. The Pricing Problem SCUC Creates: Non-Convexity and Uplift
SCUC breaks the clean duality story of SCED. With binary variables, the problem is non-convex, and no uniform set of prices generally supports the solution: a unit committed for reliability may, at the resulting LMPs, fail to recover its startup and no-load costs; conversely a unit might see prices at which it would prefer a different schedule. Markets patch this with **make-whole payments (uplift)**: side payments guaranteeing that any resource following ISO instructions recovers its as-offered costs over the day. Uplift is discriminatory (different resources receive different effective prices) and non-transparent (it is not hedgeable the way LMPs are), so market designers work to minimize it.
Two pricing responses are in active use. **Fast-start pricing** relaxes the binary constraints of online fast-start units so their startup and no-load costs can set the LMP when they are marginal — several ISOs adopted variants after FERC pushed the issue in the late 2010s. More ambitiously, **Extended LMP / convex-hull pricing** seeks prices that minimize total uplift by pricing off the convex hull of the non-convex cost functions; MISO runs an approximation of this. The theoretically clean object — the convex-hull price — is the Lagrangian dual optimum of the commitment problem, but computing it exactly at scale remains hard, and the debate over how far prices should deviate from marginal-cost logic to reflect commitment costs is one of the liveliest in market design.
## 7. How the Pieces Fit Together Across the Operating Timeline
A useful mental model is a cascade of the same core problem re-solved with shrinking horizons and hardening decisions. Day-ahead SCUC (financial, bid-in demand) sets hourly commitments and day-ahead settlements. RUC (reliability pass, forecast demand) tops up physical commitment. Intraday or hour-ahead SCUC processes re-commit fast and intermediate units as forecasts update (e.g., ERCOT's HRUC, CAISO's short-term unit commitment). Real-time SCED, every five minutes, dispatches the committed fleet and sets real-time prices, with a look-ahead of an hour or so; fast-start units can still be started by a real-time commitment layer. Each stage inherits the previous stage's decisions as fixed and re-optimizes what remains flexible — the cascade itself is an implicit, rolling form of recourse against uncertainty, even though each individual solve is deterministic.
This is also where the connection to stochastic optimization sits: production SCUC/SCED are deterministic in their treatment of load and renewable uncertainty (a single forecast, exogenous reserve requirements), while handling *equipment* uncertainty robustly through the N-1 contingency set. Stochastic SCUC — scenarios for net load, commitment as here-and-now, dispatch as recourse — is the research extension of exactly this problem, used today in advisory look-ahead tools and in setting reserve and ramp-product requirements rather than in the clearing engine itself, chiefly because of the pricing, cost-recovery, and transparency questions that scenario-based clearing raises.
## 8. Computation
Day-ahead SCUC at RTO scale is among the largest MIPs solved on a daily industrial cadence. The practical toolkit: state-of-the-art commercial MIP solvers (branch-and-cut with presolve, cutting planes, heuristics); tight formulations as noted above; **lazy-constraint handling of transmission security** — solve with a small active set of network constraints, check the full N-1 set against the solution, add violated constraints, and iterate, which typically converges in a handful of passes because so few constraints ever bind; decomposition legacies (Lagrangian relaxation was the pre-MIP workhorse and still informs pricing theory; Benders variants appear in extensions); and warm-starting from the previous day's solution. SCED, being an LP (or nearly so) with an active-set network model, solves in seconds — its challenge is data pipeline latency and robustness at a five-minute cadence, not optimization difficulty. A notable research thread applies machine learning to accelerate SCUC by predicting active constraint sets, good commitment patterns, or screening contingencies.
## 9. Variations Outside the US
The SCUC/SCED architecture is characteristic of US-style centralized pool markets with locational pricing. European markets differ structurally: day-ahead clearing (EUPHEMIA algorithm) is a zonal, portfolio-based auction in which unit commitment is left to the participants and non-convexities enter through block orders; congestion within zones is managed afterward by TSO redispatch. Australia's NEM runs a five-minute regional dispatch resembling SCED without a day-ahead commitment market. The trade-offs between nodal/centralized-commitment and zonal/self-commitment designs — pricing accuracy and congestion management versus simplicity and participant autonomy — are a standing debate, with Europe gradually adding flow-based coupling and discussions of finer granularity, and the US designs serving as the reference point for locational pricing.
## 10. Current Pressure Points
Several forces are reshaping both problems. **Storage** requires state-of-charge dynamics inside SCUC/SCED and raises subtle questions about opportunity-cost-based offers and whether the ISO or the owner should manage the state of charge. **Renewables and net-load volatility** stress the deterministic-forecast assumption, driving ramping products, more frequent intraday commitment, and interest in stochastic and robust look-ahead tools. **Distributed energy resources and demand flexibility** (FERC Order 2222 in the US) push aggregations of small resources into the same optimization, multiplying model size. **Co-optimization scope** keeps expanding — energy, multiple reserve classes, ramping products, and in some designs day-ahead ancillary demand curves — enlarging the already-large MIP. And **pricing reform** (convex-hull approximations, fast-start pricing, scarcity pricing via operating reserve demand curves) continues to evolve the mapping from these optimizations to the prices that drive investment.
## 11. Suggested Reading
For the market-design foundations: Schweppe, Caramanis, Tabors, and Bohn, *Spot Pricing of Electricity* (1988), the origin of LMP theory; and Hogan's papers on contract networks and FTRs. For formulations: Knueven, Ostrowski, and Watson, "On Mixed-Integer Programming Formulations for the Unit Commitment Problem" (2020) is the modern survey; O'Neill et al., "Efficient market-clearing prices in markets with nonconvexities" (2005) and Gribik, Hogan, and Pope's 2007 paper on ELMP frame the pricing problem; Rajan and Takriti's 2005 report gives the classic tight up/down-time polytope. For practice, the ISOs' own business practice manuals (e.g., PJM Manual 11, MISO's energy and operating reserve market documentation) are unusually informative primary sources on how these models are actually configured and solved.

104
primers/stochastic_optimization_en.md Normal file
View File

@ -0,0 +1,104 @@
# A Primer on Stochastic Optimization in Electricity Markets
## 1. Why Uncertainty Is the Central Problem in Power Systems
Electricity is unusual among commodities: it must be produced at nearly the same instant it is consumed, storage remains expensive relative to demand, and the physics of the network constrain how power flows. Market operators must therefore commit resources — deciding which power plants to turn on, how much reserve to hold, how to schedule hydro reservoirs — *before* they know exactly what conditions will materialize. The gap between when decisions are made and when uncertainty resolves is what makes electricity markets a natural home for stochastic optimization.
The principal sources of uncertainty are load (demand fluctuates with weather, behavior, and increasingly with distributed resources behind the meter), renewable generation (wind and solar output can swing by large fractions of installed capacity within hours), equipment availability (generators and transmission lines fail unexpectedly), fuel and electricity prices (relevant for market participants making offering and hedging decisions), and inflows (for hydro-dominated systems, reservoir inflows over weeks to years). Historically, load uncertainty was modest and manageable with deterministic forecasts plus reserve margins. The rapid growth of wind and solar changed that calculus: net load (demand minus renewable output) is far more volatile and harder to forecast, which has pushed both system operators and market participants toward decision-making frameworks that represent uncertainty explicitly rather than papering over it with heuristic margins.
## 2. From Deterministic to Stochastic Formulations
The workhorse problems of electricity markets — economic dispatch, unit commitment, market clearing — are traditionally posed as deterministic optimization problems. Economic dispatch minimizes generation cost subject to a power balance constraint and generator limits, typically as a linear or quadratic program. Unit commitment adds binary on/off decisions, minimum up/down times, startup costs, and ramping limits, making it a mixed-integer program (MIP). Both take a single forecast of demand and renewable output as given.
A stochastic formulation replaces the single forecast with a probability distribution, usually approximated by a finite set of scenarios, and restructures the problem around the timing of decisions relative to the resolution of uncertainty. The key conceptual device is the distinction between *here-and-now* decisions, which must be made before uncertainty resolves and are therefore identical across all scenarios, and *wait-and-see* (recourse) decisions, which can adapt once the outcome is observed. In unit commitment, the commitment of slow-start thermal units is here-and-now: a coal or combined-cycle plant needing several hours to start must be committed the day before. Dispatch levels, fast-start unit commitments, and reserve deployment are recourse: they adjust in real time to the realized net load.
## 3. Two-Stage Stochastic Programming and Stochastic Unit Commitment
The canonical model is the two-stage stochastic program with recourse:
```
minimize c'x + E_ω[ Q(x, ω) ]
subject to x ∈ X
where Q(x, ω) = min { q(ω)'y : W y = h(ω) − T(ω) x, y ≥ 0 }
```
Here `x` collects first-stage decisions (unit commitments), `Q(x, ω)` is the optimal recourse cost in scenario `ω` (the dispatch cost given the commitment), and the objective minimizes first-stage cost plus *expected* second-stage cost. With a finite scenario set {ω₁, …, ω_S} with probabilities p_s, the expectation becomes a weighted sum and the whole problem is one large "extensive form" MIP: a copy of the dispatch problem for each scenario, all coupled through the shared commitment variables and *nonanticipativity* constraints (first-stage decisions cannot depend on which scenario occurs).
**Stochastic unit commitment (SUC)** applies exactly this template. The day-ahead commitment of slow units is chosen to minimize expected total cost across scenarios of net load, where each scenario's dispatch respects transmission limits, ramping, and reserves. The value of doing this rather than committing against a point forecast is quantified by two standard metrics. The **Value of the Stochastic Solution (VSS)** compares the expected cost of the stochastic solution against the expected cost of implementing the deterministic (expected-value) solution — it measures what representing uncertainty is worth. The **Expected Value of Perfect Information (EVPI)** compares the stochastic solution against a clairvoyant who solves each scenario separately — it bounds what better forecasting could be worth. Studies on realistic systems typically find VSS on the order of a fraction of a percent to a few percent of production cost, which sounds small until multiplied by the billions of dollars flowing through a large market annually.
A closely related operational use is **reserve determination**: instead of fixing reserve requirements exogenously (e.g., "3% of load plus the largest contingency"), a stochastic model determines reserves endogenously — the model holds back exactly as much flexible capacity as the scenario set makes economical, and it holds it in the right locations relative to transmission constraints.
## 4. Multi-Stage Problems and Stochastic Dual Dynamic Programming
Many electricity problems are inherently multi-stage: uncertainty unfolds gradually and decisions interleave with observations. The paradigm example is **hydrothermal scheduling** in reservoir-dominated systems such as Brazil, Norway, Colombia, and New Zealand. Water stored today can displace expensive thermal generation later, but inflows over the coming months are uncertain. Releasing too much risks scarcity if inflows disappoint; hoarding wastes free energy through spill if inflows are strong. The problem spans months to years, with a decision each week or month after observing that period's inflow.
A scenario tree for such a problem explodes combinatorially — with even ten inflow outcomes per stage and 60 monthly stages, the tree is astronomically large. **Stochastic Dual Dynamic Programming (SDDP)**, developed by Pereira and Pinto in 1991 precisely for the Brazilian system, tames this by exploiting stage-wise independence (or Markovian structure) of the randomness. SDDP approximates the future cost function of dynamic programming — the expected cost-to-go as a function of reservoir storage — with an outer envelope of linear cuts (Benders cuts), refined iteratively through forward simulation passes and backward cut-generation passes. Because the cost-to-go function is convex in storage under linear dispatch models, the cuts converge to the true function. SDDP and its descendants are in production use for national hydro scheduling in several countries, and the marginal water values it produces feed directly into market prices in hydro systems.
## 5. Scenario Generation and Reduction
A stochastic program is only as good as its scenarios. Scenario generation is the modeling step where statistical craft matters most, and it has several requirements: scenarios must capture the marginal distributions of each uncertain quantity, the temporal autocorrelation (a windy hour tends to be followed by a windy hour), the spatial correlation across sites (weather systems are large), and the dependence between variables (cold snaps raise load and can affect both wind output and thermal plant availability).
Common approaches include time-series models (ARMA/GARCH on forecast errors), Gaussian copula methods that impose realistic dependence structure on marginal forecast distributions, sampling from ensemble numerical weather predictions (each ensemble member is a physically consistent scenario), and, increasingly, machine-learning generative models. For discrete events such as generator outages, Monte Carlo draws from Markov availability models are standard.
Because solver effort grows with the number of scenarios, practitioners generate a large scenario set and then apply **scenario reduction** — selecting a small representative subset and reweighting it to stay close (in a probability metric such as the Wasserstein distance) to the original set. The Heitsch–Römisch forward-selection and backward-reduction algorithms are the classical tools. A practical warning: aggressive reduction tends to discard exactly the extreme scenarios that drive reserve and commitment decisions, so reduction criteria are often modified to preserve tails.
## 6. Alternatives and Cousins: Robust, Chance-Constrained, and Distributionally Robust Optimization
Stochastic programming optimizes an expectation, which requires a credible probability distribution and accepts poor outcomes in bad scenarios if they are improbable. Several alternative frameworks address different attitudes toward uncertainty, and all have found electricity applications.
**Robust optimization (RO)** dispenses with probabilities entirely: uncertainty lives in a set (e.g., net load within a polyhedral band with a "budget of uncertainty" limiting how many hours can be simultaneously extreme), and the model minimizes cost under the *worst case* in that set. Two-stage adaptive robust unit commitment became prominent in the early 2010s, with ISO New England among the first system operators to study it seriously. RO's appeal is that it needs no distribution and yields strong reliability guarantees; its drawback is conservatism — the worst case may be wildly expensive to hedge. The budget parameter gives the operator a dial between economy and security.
**Chance-constrained optimization** requires constraints to hold with at least a specified probability — for example, "transmission line flows remain within limits with probability 0.99 given wind forecast errors." Under Gaussian error assumptions, many chance constraints reformulate into tractable second-order cone constraints, which made chance-constrained DC optimal power flow a popular research vehicle. It offers a transparent reliability knob but inherits sensitivity to the assumed distribution.
**Distributionally robust optimization (DRO)** hedges between the two: it optimizes against the worst distribution within an ambiguity set (distributions matching observed moments, or within a Wasserstein ball around the empirical distribution). DRO has become an active research direction for reserve procurement and OPF because it acknowledges that the distribution itself is estimated from limited data.
A rough practical taxonomy: use stochastic programming when you trust your scenarios and care about expected cost; robust optimization when failure is unacceptable and distributions are unreliable; chance constraints when reliability targets are naturally probabilistic; DRO when you have data but not enough to trust it fully.
## 7. Stochastic Market Clearing and the Pricing Problem
Using stochastic optimization inside the *market clearing* itself — not merely as an operator's advisory tool — raises deep economic questions. In a stochastic market clearing, the day-ahead and real-time (balancing) stages are co-optimized: day-ahead schedules are chosen to minimize expected total cost across scenarios, anticipating balancing actions. Research consistently shows this improves expected system cost relative to sequential, deterministic clearing, especially at high renewable penetration, because day-ahead schedules pre-position flexibility where scenarios say it will be needed.
The difficulties are institutional and economic rather than computational:
**Pricing and cost recovery.** In a deterministic convex market, locational marginal prices (LMPs) from the dual variables support the dispatch — every scheduled resource at least breaks even at the price. In stochastic market clearing, no single set of day-ahead prices generally supports the solution scenario-by-scenario: prices that guarantee cost recovery *in expectation* may leave participants losing money in particular realized outcomes, which strains acceptability. Research on "equilibrium-supporting" pricing schemes and revenue-adequacy properties of stochastic clearing remains active, but no major market has adopted full stochastic clearing, in part for this reason.
**Fairness and transparency.** Scenario choices directly shape who gets scheduled and paid. A market operator would have to defend its scenario-generation methodology as a quasi-regulatory matter.
**What markets actually do.** Real markets have adopted uncertainty-aware elements incrementally rather than wholesale: probabilistic reserve requirements and new ramping products (CAISO's flexible ramping product, MISO's ramp capability product) that are sized using distributions of net-load forecast error; robust look-ahead commitment tools running alongside deterministic market engines; and out-of-market operator actions informed by probabilistic forecasts. Stochastic optimization thus influences markets today mostly through *requirement-setting* and *advisory tools* rather than through the clearing engine itself.
## 8. The Participant's Side: Offering, Trading, and Asset Optimization
Market participants face their own stochastic problems, and here adoption has been faster because no institutional consensus is required — a firm simply uses better models to make more money.
**Renewable offering.** A wind producer selling into a day-ahead market must offer a quantity before knowing its output, then settle deviations at imbalance prices. This is a newsvendor-type problem: the optimal day-ahead offer is a quantile of the production distribution determined by the ratio of penalties for under- and over-delivery. Richer versions co-optimize offers across day-ahead, intraday, and balancing markets as multi-stage stochastic programs.
**Storage and hydro valuation.** A battery or pumped-hydro operator arbitraging uncertain prices solves a stochastic dynamic program over the state of charge; the resulting marginal value of stored energy ("opportunity cost curve") is exactly what the asset should bid. For seasonal hydro, SDDP-derived water values serve the same role.
**Virtual bidding and financial trading.** Traders placing virtual (purely financial) bids between day-ahead and real-time markets are effectively trading the distribution of the DA–RT spread, and portfolio construction over such positions is a classic stochastic optimization with risk constraints.
**Risk measures.** Because market participants are rarely risk-neutral, expected cost is usually replaced or augmented by risk measures, most commonly **Conditional Value-at-Risk (CVaR)** — the expected loss in the worst α-fraction of scenarios. CVaR is coherent and, crucially, preserves linearity: it adds only linear constraints and variables to a scenario-based LP/MIP (Rockafellar–Uryasev formulation), so risk-averse stochastic programs remain tractable. A weighted combination of expectation and CVaR is the de facto standard objective in trading and asset-optimization models.
## 9. Computation: Decomposition and Practical Tactics
Extensive-form stochastic MIPs get large fast: a unit commitment with 1,000 units, 36 hours, and 50 scenarios has tens of millions of variables. Three decomposition families dominate:
**Benders decomposition (the L-shaped method)** splits the problem into a master over first-stage variables and scenario subproblems whose duals generate optimality and feasibility cuts. It is natural when the second stage is a continuous LP (dispatch given commitment) and parallelizes trivially across scenarios. SDDP is its multi-stage, sampled cousin.
**Lagrangian relaxation** dualizes coupling constraints — either the nonanticipativity constraints or the system-wide balance constraints — so the problem separates by scenario or by unit. It was the workhorse of unit commitment for decades and provides strong bounds, though recovering primal-feasible solutions requires heuristics.
**Progressive Hedging (PH)** (Rockafellar and Wets) relaxes nonanticipativity with an augmented-Lagrangian penalty: each scenario subproblem is solved independently with a penalty pulling its first-stage decisions toward the scenario-weighted average, iterating to consensus. PH is a heuristic without convergence guarantees in the mixed-integer case but works well in practice on SUC and is embarrassingly parallel.
Beyond decomposition, practitioners rely on **sample average approximation (SAA)** with out-of-sample validation to choose scenario counts, warm-starting from deterministic solutions, tightened MIP formulations of unit constraints (which matter as much as decomposition), and generous parallel hardware. Open-source tooling has lowered the barrier considerably: PySP/mpi-sppy and StochasticPrograms.jl for general stochastic programming, SDDP.jl for multi-stage problems, and power-system-specific platforms such as PowerSimulations.jl, PLEXOS, and PSR's SDDP implementation.
## 10. A Worked Miniature: Two-Scenario Commitment Intuition
To make the recourse logic concrete, consider one hour, two units, and two equally likely net-load scenarios of 400 MW and 700 MW. Unit A is a slow unit: 500 MW capacity, $40/MWh, $10,000 startup, must be committed now. Unit B is a fast peaker: 300 MW, $120/MWh, committable in real time. The deterministic model sees the expected load of 550 MW, commits A, and plans 550 MW from it — but in the high scenario A alone cannot serve 700 MW, so the peaker runs 200 MW at high cost, and in the low scenario A runs part-loaded. The stochastic model evaluates the *actual* expected cost of each commitment plan against both scenarios, correctly weighing A's startup cost against the probability-weighted peaker usage, and would also detect cases where the deterministic plan is infeasible in the tail (e.g., if the high scenario were 900 MW, expected-value commitment would leave the system short). The general lesson generalizes: deterministic models are systematically blind to asymmetries between scenarios, and the cost of that blindness grows with net-load variance — which is exactly what renewable growth delivers.
## 11. Open Frontiers
Several directions define current research and near-term practice. First, **learning-augmented optimization**: using machine learning to warm-start commitments, predict active constraints, or learn cost-to-go functions, cutting solve times enough to make stochastic clearing computationally routine. Second, **distributionally robust and data-driven formulations** that connect the scenario-generation step and the optimization step into one statistically principled pipeline. Third, **pricing theory for non-convex, stochastic markets** — extending convex-hull and extended-LMP ideas to settings with scenarios, so that stochastic clearing could produce defensible settlement prices. Fourth, **multi-stage market designs** (continuous intraday trading, shorter gate closures) that reduce the here-and-now burden by letting more decisions become recourse. And fifth, **demand-side and storage recourse**: as flexible demand and batteries proliferate, the cheapest hedge against uncertainty increasingly sits on the consumption side, and models are evolving to co-optimize it.
## 12. Suggested Reading
For foundations, Birge and Louveaux, *Introduction to Stochastic Programming*, and Shapiro, Dentcheva, and Ruszczyński, *Lectures on Stochastic Programming*. For electricity applications, Conejo, Carrión, and Morales, *Decision Making Under Uncertainty in Electricity Markets*, is the standard reference; Morales, Conejo, Madsen, Pinson, and Zugno, *Integrating Renewables in Electricity Markets*, covers stochastic clearing and renewable offering in depth. Pereira and Pinto's 1991 paper "Multi-stage stochastic optimization applied to energy planning" introduces SDDP; Bertsimas et al.'s "Adaptive robust optimization for the security constrained unit commitment problem" (2013) is the touchstone for the robust alternative; and Rockafellar and Uryasev's "Optimization of conditional value-at-risk" (2000) underpins the risk-averse formulations used throughout the industry.