shu/power-market-trading-docs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
master
power-market-trading-docs/markets/us/field_guide_en.md
Renewable Trading Docs beacc088a3 Reorganize market docs and primers
2026-07-20 06:52:50 -04:00

46 KiB
Raw Permalink Blame History

US Power Markets: A Field Guide

Bidding, trading, and the impact of AI — consolidated from a series of working discussions, July 2026.

Contents

  1. How the market clears
  2. How participants compete
  3. Where edge actually lives
  4. How AI changes the picture
  5. Building a bidding system
  6. China: market structure and deployment
  7. Open questions
  8. Sources and further reading

1. How the market clears

The landscape splits into two worlds: market operators (the seven US ISOs/RTOs — PJM, CAISO, ERCOT, MISO, NYISO, SPP, ISO-NE) who clear the market, and market participants (generators, batteries, traders, hedge funds) who bid into it. Everything in this guide sits on one side or the other of that line.

1.1 The clearing algorithms (ISO/RTO side)

The core is large-scale mixed-integer programming, not ML:

  • SCUC (Security-Constrained Unit Commitment) clears the day-ahead market — a massive MILP solved with commercial solvers (Gurobi / CPLEX / Xpress), deciding unit commitments subject to network and reliability constraints.
  • SCED (Security-Constrained Economic Dispatch) runs every 5 minutes in real time (LP/QP), producing Locational Marginal Prices (LMPs) decomposed into energy + congestion + losses.
  • Energy and ancillary services are co-optimized in the same clearing.

Active frontiers on the clearing side: convex hull pricing / extended LMP for non-convexities; stochastic and robust unit commitment for renewable uncertainty; and ML-accelerated optimization — learned warm starts, active-constraint prediction, transmission constraint screening (pushed by ARPA-E grid optimization competitions).

1.2 The two-settlement structure: day-ahead vs. real-time

All seven ISOs run a two-settlement (day-ahead + real-time) design. Short answer: day-ahead is where the volume and money is; real-time is where the risk is.

Day-Ahead Market (DAM)

  • Primary market in most designs; ~90–95%+ of physical energy settles financially at the DA price in US ISOs
  • Clears once daily via auction with full network modeling
  • Reference price for the forward curve; most hedging keys off it
  • The classic academic electricity-price-forecasting (EPF) problem

Real-Time (Balancing) Market

  • Settles only deviations from day-ahead positions — small volume, huge volatility
  • Prices can move from $30 to $3,000+/MWh (or deeply negative) in minutes
  • Critical for: risk management, batteries, fast-ramping peakers, demand response, DA–RT spread traders
  • Battery operators often earn most energy-arbitrage revenue from RT volatility

Expected RT prices anchor DA prices (virtual bidding arbitrages them on average), but realized RT prices are far noisier. The practical forecasting hierarchy for most participants: (1) day-ahead hourly prices, (2) the DA–RT spread, (3) intraday/real-time — reversed if you operate storage or fast-response assets.

Distinctively US features on the participant side: virtual bidding (INCs/DECs), FTR/CRR auctions, and the fast-growing battery-storage bidding specialty.

2. How participants compete

2.1 Price forecasting: the foundation

Electricity is not storable at scale, so prices reflect instantaneous supply-demand balance and can spike or go negative. Forecasting approaches fall into three families, usually blended:

Fundamental (structural) models simulate the market itself: load forecast vs. the supply stack (merit order). Key inputs: fuel prices (especially gas, which often sets the margin), renewable output forecasts, planned/unplanned outages, transmission constraints, imports/exports. Tools: PLEXOS, Aurora, in-house production-cost / dispatch simulations. Used heavily by utilities, large generators, and ISOs.

Statistical / econometric models — ARIMA/ARIMAX, GARCH for volatility, regressions on temperature, gas, load. They exploit strong seasonality (hour-of-day, day-of-week, season) and are best for "normal" conditions and short horizons.

Machine learning models — gradient-boosted trees, neural nets (LSTMs, transformer-based), trained on weather, load, renewables, prices, fuel, and outage data. Sophisticated shops forecast the full distribution, not just the mean — the tails are where money is made or lost.

In practice: hybrid. Fundamental model for structural shape + ML/statistical corrections + human trader judgment, especially for events models handle poorly (cold snaps, plant trips, unusual grid conditions).

Universal key inputs: weather forecasts (the single biggest driver), natural gas prices, renewable generation forecasts, outage schedules, transmission/congestion conditions.

State of the art. The benchmark progression runs LEAR (regularized linear/LASSO) → deep neural nets → temporal architectures (LSTM, Transformer). Transformer models now forecast DA–RT price spreads in volatile markets (e.g., ERCOT), using load / solar / wind forecasts and temporal features, with walk-forward retraining. Well-tuned linear models remain surprisingly competitive; hybrid linear+nonlinear architectures with online learning are a current frontier. Practical SOTA is increasingly probabilistic — quantile regression, distributional deep nets, conformal prediction — because decisions need the full distribution, not a point forecast.

2.2 Strategic bidding optimization

Two state-of-the-art frameworks for price-makers: bi-level optimization (MPEC-style) and reinforcement learning. Deep RL (DDPG + prioritized experience replay, PPO, SAC) handles continuous state/action spaces and non-convex operating characteristics. But stochastic programming and robust optimization remain the workhorses for co-optimizing energy + ancillary service offers under uncertainty, with CVaR the standard risk overlay.

2.3 Battery storage arbitrage (the hottest area)

  • Classical: stochastic dynamic programming, MPC over price scenarios, degradation costs in the objective.
  • Modern: deep RL for charge/discharge policies; decision-focused learning (forecasts trained on profit, not accuracy — see §4.4).
  • Research frontier: hierarchical / multi-agent RL coordinating DA + RT bidding under the two-settlement structure, with risk-adjusted rewards (fixed policies struggle under regime shifts).
  • Industry: optimizer firms (Habitat Energy, Gridmatic, Tyba, Fluence Mosaic) run ML forecasting + optimization stacks commercially.

2.4 Virtual bidding and financial trading (INCs, DECs, FTRs)

Essentially quant trading: gradient-boosted trees (XGBoost/LightGBM) and deep nets predicting DA–RT spreads at nodal granularity, with features drawn from weather, load/renewable forecast errors, outages, and congestion patterns. Portfolio construction runs under risk limits; FTR auctions add congestion-rent scenario optimization.

2.5 Emerging directions

  • Online / no-regret learning — bidding policies updated directly from market outcomes, with sub-linear regret guarantees.
  • Multi-agent RL — Nash equilibrium approximation for market simulation and market-power analysis.
  • LLM-enhanced trading frameworks — very early, appearing in the literature.
  • Open-source benchmarks — Grid2Op, RL2Grid, new two-settlement bidding environments.

2.6 Industry reality check

Most real money is still made with strong probabilistic forecasting + classical optimization (stochastic MIP/MPC) + human trader judgment. RL is in production mainly for storage dispatch at a handful of sophisticated shops. Deep learning has clearly won the forecast layer; the decision layer remains dominated by optimization with ML inputs.

One-sentence summary of the frontier: DL/transformers have won the forecasting layer; RL is contesting the optimization layer but hasn't displaced stochastic optimization yet.

Two nuances: the two moves are at different maturity stages — DL in forecasting is a completed takeover (and increasingly probabilistic), while RL in decisions is still a challenger vs. deployed stochastic optimization / MPC. And the boundary between the layers is dissolving: decision-focused learning trains forecasts directly on trading profit, collapsing forecast-then-optimize into one learned pipeline — arguably the most interesting current research direction (§4.4).

2.7 Quick reference table

Layer Deployed standard Research frontier
Market clearing (ISO) MILP SCUC + LP SCED, LMP Convex hull pricing, stochastic UC, ML-accelerated optimization
Price forecasting Probabilistic deep nets, GBTs, tuned linear models Transformers, hybrid+online learning, conformal prediction
Bidding/dispatch decisions Stochastic programming, MPC, bi-level MPEC Deep RL (DDPG/PPO/SAC), hierarchical & multi-agent RL, no-regret online learning
Forecast↔decision interface Two-stage predict-then-optimize Decision-focused / end-to-end learning
Storage arbitrage MPC + probabilistic forecasts DFL, risk-aware hierarchical RL
Virtual bidding / FTRs GBT/DNN spread models + portfolio risk limits Transformer spread forecasting, LLM-enhanced frameworks

3. Where edge actually lives

3.1 The hierarchy of durable edge

Any bidding system decomposes into three functional layers: a data and forecasting layer that converts weather, grid, and market data into calibrated probability distributions; an optimization layer that converts distributions into offer curves and positions under risk constraints; and an execution and risk layer that submits bids, monitors positions, settles, and enforces discipline.

The competitive value of these layers is not equal:

  1. Forecast quality and calibration dominate. Two participants running identical optimizers on different forecasts diverge widely in P&L, while two participants running different (competent) optimizers on identical forecasts land close together.
  2. Execution and risk discipline ranks second. In crowded strategies, the shop that re-optimizes every interval, never misses a submission window, and cuts losers per its own rules outperforms the one that doesn't, even with no analytical edge.
  3. Optimizer formulation ranks third. The underlying mathematics (quantile offering, two-stage stochastic programming, CVaR constraints, storage MILPs) is published and commoditized; what differs between shops is craft, not theory.

Build implication: spend the best engineering on forecasting and backtesting, use boring proven math in the middle, and treat operational reliability as a feature.

3.2 The insider-information myth (PJM / SCUC-SCED)

A natural worry: wouldn't a vendor with inside knowledge of PJM's clearing algorithm — or of the bids flowing into it — crush all competitors? The concern dissolves under inspection.

What is NOT really secret. The algorithm: SCUC/SCED are standard MIP/LP formulations, exhaustively documented (PJM Manual 11/12, the OATT, business practice manuals), solved with commercial CPLEX/Gurobi-class solvers. Competent quant teams build "shadow SCED" models from public documentation.

What IS secret — and legally fenced:

Item What it is Why it's decisive
Other participants' bids/offers Released only masked, ~4-month lag DA prices are an emergent function of them
Network state Live state-estimator view: flows, voltages, topology, constraint headroom Knowing which constraint binds next = seeing nodal price divergence before it happens
Derates Real-time reductions in unit capability (e.g., a 900 MW unit down to 600 MW) Supply tighter than the market believes → position ahead of the spike; classic MNPI in FERC cases
Dispatch instructions 5-min SCED setpoints + out-of-market operator actions (reliability commitments, manual dispatch) Reveals decisions invisible to outside models; reliability commitments distort local prices

Using these is a federal crime, not a vendor edge: FERC anti-manipulation rules (post-EPAct 2005) cover trading on material non-public information; PJM staff and systems vendors are bound by confidentiality and trading prohibitions; the Independent Market Monitor (Monitoring Analytics — external to PJM) and FERC's Office of Enforcement screen for anomalous profitability. There is precedent — enforcement cases against individuals trading on non-public grid information, and the Powhatan/UTC saga showed FERC pursues even aggressive rule exploitation.

Why perfect algorithm knowledge wouldn't get you far anyway. SCED is deterministic given its inputs, but the inputs are unknowable in advance — even to PJM. DA prices depend on bids not yet submitted; RT prices depend on forced outages, weather errors, interchange, and binding constraints. Price is an emergent output of thousands of private decisions plus physical randomness.

Where real differentiation lives instead: better weather ensembles and load/renewable models; predicting which transmission constraints will bind (the genuinely hard, high-value problem in nodal markets); modeling bidding behavior from lagged public data; reconstructing network topology (CEII model access, FTR results, historical shadow prices).

The honest residual: EMS/market-software vendors and ex-ISO employees carry legal "soft" insider knowledge — operator behavior, solver quirks, out-of-market actions. It's valuable, common, and legal. The real moat: people who know how the control room actually behaves at 6 PM on a July scarcity day.

3.3 Is accurate price prediction even the most valuable thing?

No — for most participants, point forecasts are an input, rarely the differentiator.

Generators: optionality and risk, not prediction. The right tail of the distribution matters far more than the mean — a peaker earns its year in a handful of scarcity hours; the question is 3 vs. 30 scarcity hours, not ±$2/MWh on the average. Volumetric-price correlation risk dominates hedging: being forced out exactly when prices spike (Winter Storm Uri: forward-sold generators with frozen plants buying back at $9,000/MWh). And the economics run on spreads, not prices: spark spread (gas), dark spread (coal), top-bottom spread (batteries).

LSEs / large buyers: load forecasting beats price forecasting. Exposure is squared — a hot day means high load AND high price. Most valuable: your own load forecast, hedge-ratio policy, and shape-risk management. The ERCOT retailer failures (Griddy) were unhedged structural exposure, not bad price forecasts.

Financial traders: relative value, tails, speed. The money is in spreads and congestion (DA–RT virtuals, UTCs, FTRs, hub-to-node basis) — the skill is predicting which constraints bind. Calibration beats accuracy: well-calibrated tails plus good sizing out-earn a sharper point forecast, and survive the blowups. Speed matters: reacting to unit trips, weather model runs, and interchange changes faster than the market reprices.

What arguably matters more than any forecast:

  1. Risk management and capital discipline — the graveyard is full of risk-control failures, not forecast failures (the GreenHat FTR default, $180M+ socialized; the Uri casualties). Approximately right and definitely solvent wins.
  2. Structural optionality — batteries need a good response policy, not a forecast; market makers earn the spread; structured desks earn risk-transfer margin.
  3. Understanding the plumbing — settlement rules, uplift/make-whole, credit mechanics, FTR auction quirks, emergency operator discretion. Rule knowledge is durable; forecast edges decay.
  4. Weather as the upstream input — many firms' single most valuable proprietary asset is their weather-ensemble processing; price forecasts are largely derivative of weather + fuel + outages.

Synthesis: price forecasting is necessary but low-moat; risk structuring and constraint/network insight are the actual differentiators. The scarce capabilities are (a) calibrated tail distributions, (b) node-level congestion insight, (c) position-sizing discipline to survive being wrong, and (d) rule/plumbing expertise — hence firms pay up for ex-control-room operators, network engineers, and risk managers as eagerly as for forecasters. The forecast is the ante; the other things are the game.

4. How AI changes the picture

The three-layer decomposition of §3.1 is the right lens for evaluating the current AI wave, because recent advances land on the layers very unevenly — and, conveniently, they land hardest on the layer that matters most.

Layer Edge rank AI impact Maturity Adoption posture
Forecasting 1st Step change (AI weather ensembles) Operational now Adopt aggressively; edge is decaying
Execution / risk ops 2nd Large but unglamorous (LLM ops) Production-ready with human review Adopt quietly; compounding advantage
Optimizer 3rd Incremental (decision-focused learning) Research → early practice Experiment; keep auditable structure

4.1 Layer 1 — Forecasting: the genuine step change

The AI weather model transition. The most consequential AI development for power trading is not reinforcement learning or language models; it is the replacement (or augmentation) of physics-based numerical weather prediction (NWP) with learned atmospheric models. The milestone sequence is short but steep: Huawei's Pangu-Weather (Nature, July 2023) demonstrated forecasts roughly 10,000× faster than conventional ensemble systems in peer-reviewed tests; Google DeepMind's GraphCast (Science, December 2023) outperformed ECMWF's flagship HRES deterministic model on roughly 90% of 1,380 verification targets; ECMWF moved its AI-based AIFS model to operational status in 2024 — the first major meteorological agency to run a learned model operationally; Microsoft's Aurora foundation model (Nature, May 2025), pretrained on over a million hours of geophysical data, extended the approach across weather, air quality, and wave prediction at a fraction of traditional compute cost; and DeepMind's GenCast pushed the architecture family into probabilistic ensemble forecasting.

How learned atmospheric models work. Neural networks are trained — mostly on the ERA5 reanalysis, ~45 years of hourly global atmospheric state — to learn the mapping "atmosphere at t → atmosphere at t+6h" directly, replacing numerical integration of physics equations with a single forward pass. Architectures span 3-D vision transformers (Pangu), icosahedral-mesh GNNs (GraphCast), graph/transformer hybrids (AIFS), diffusion models sampling plausible futures (GenCast — natively probabilistic), and pretrained geophysical foundation models (Aurora). Forecasts roll out autoregressively, which compounds errors and — under MSE training — produces progressive smoothing that underestimates extremes. Critical dependency: they still require physics-pipeline initial conditions; data assimilation is not replaced. Global resolution (~25–31 km) means site-level downscaling/calibration matters more, not less.

Why the economics matter more than the leaderboard. Traditional NWP solves differential equations across 3-D grid cells on supercomputers, refreshing two to four times daily at thousands of dollars per cycle. Learned models run on GPUs in seconds to minutes, at accuracy parity in the 12–48h day-ahead window. Three consequences follow:

  • Speed → probability. When a forecast run costs seconds of GPU time, generating dozens or hundreds of ensemble members becomes trivial. The entire downstream bidding pipeline consumes probability distributions — production quantiles for the day-ahead offer, spread distributions for imbalance risk — so ensemble economics translate directly into better-calibrated inputs and therefore better offers. This is the cleanest causal path from "recent AI" to bidding P&L.
  • Speed → freshness. Commercial AI-weather providers now advertise up to 24 runs per day versus the 2–4 of traditional NWP. In a two-settlement market, fresher forecasts matter most between the day-ahead close and real-time delivery: real-time re-offering, intraday position adjustment, and imbalance management for wind and solar portfolios.
  • Speed → iteration (and entry). Cheap reforecasting powers backtests, and open model weights plus public initial-condition data put a near-state-of-the-art global forecast within reach of any shop with GPUs — infrastructure that previously required a national weather service or an expensive vendor contract.

Caveats and failure modes. Three caveats keep this honest:

  1. Extremes are the weak spot. Models trained with mean-squared-error objectives produce smoothed fields that systematically underestimate sharp gradients — peak wind speeds in severe storms, for example. Power markets make and lose money precisely in the tails (scarcity events, ramps, icing), so a model that wins on average RMSE can still be the wrong tool for the hours that dominate annual P&L. Ensemble and generative approaches (GenCast-style) mitigate but do not eliminate this. Any adoption plan should include tail-specific verification against your own asset history, not just headline skill scores.
  2. Vendor claims require independent verification. The commercial AI-weather space is young and marketing-heavy. The practical test is a paid pilot scored against your incumbent provider on your nodes, your variables (hub-height wind, plane-of-array irradiance, temperature-driven load), and your loss function — ideally the downstream trading metric, not meteorological RMSE.
  3. The edge decays. These models are open or cheaply accessible, so the advantage from merely using them erodes as adoption spreads. Public AI raises the floor for everyone. What it cannot commoditize is what you combine it with: proprietary asset telemetry for local calibration, downscaling to your specific sites, and — above all — the translation from weather to nodal price, which runs through congestion.

Downstream of weather: prices, load, congestion. Price forecasting has moved from classical time-series methods to gradient-boosted trees and increasingly to transformers and time-series foundation models; the practical gains are largest in probabilistic (quantile) forecasting of the DA/RT spread, the input that prices a renewable's imbalance risk. Load forecasting benefits from the same architectures plus improved temperature inputs. Congestion and nodal-basis prediction remains the hardest and least commoditized problem — learning the mapping from system conditions to binding transmission constraints — and, precisely because almost nobody does it well, it is where forecast-layer investment retains the longest-lived edge. Graph neural networks that encode grid topology are the active research direction; no vendor sells a turnkey solution worth having.

Layer 1 synthesis: adopt AI weather ensembles early and aggressively, verify tails independently, and reinvest the freed budget into the two things public models cannot provide — local calibration against your own telemetry, and nodal congestion modeling.

4.2 Layer 2 — Execution, operations, and risk: the quiet LLM win

This is the layer vendor marketing ignores, which is exactly why it is underrated. It is unglamorous back-office work — but it attacks the second-ranked source of edge, carries low model risk because a human reviews the output, and its benefits compound.

Market-rule intelligence. US ISO participation is governed by thousands of pages of tariffs, business practice manuals, and a continuous stream of market notices, protocol revisions, and FERC filings. Engineers building bidding systems consistently identify this — not the mathematics — as the dominant cost. It is a nearly ideal LLM workload: a retrieval corpus over the tariff and manual set for each ISO you trade; automated triage of daily market notices, flagging anything that touches your bid parameters, settlement formulas, or ancillary product definitions; and change-diffing of protocol revisions against the assumptions encoded in your optimizer. The failure mode to engineer against is hallucinated rule citations — mitigations are standard (retrieval-grounded answers, mandatory citation to source paragraphs, human sign-off on anything that changes system behavior).

Compliance documentation and the hedge/spec boundary. For any shop running both physical assets and financial positions, FERC manipulation risk makes documentation a first-order concern: every physical bid should be independently defensible as profit-maximizing for the asset, on the record. LLMs are well suited to generating that record — drafting daily bid-rationale documentation from the optimizer's own inputs and outputs (forecast quantiles used, constraints binding, deviation from the neutral baseline and why) in consistent, auditable language. This converts a compliance burden that trading shops chronically under-resource into a largely automated byproduct of the bidding run, and materially strengthens the architectural separation between hedge and speculative books (§5.2).

Settlement, monitoring, and incident response. Shadow settlement — independently recomputing what the ISO owes you and disputing discrepancies — is high-value, detail-heavy work mixing structured data with unstructured rule text: again a natural LLM-plus-tools workload (parsing settlement statements, reconciling against internal calculations, drafting dispute filings with rule citations). Overnight operations monitoring (telemetry anomalies, missed dispatch instructions, forecast-feed failures ahead of submission deadlines) similarly benefits from an agentic layer that triages, summarizes, and escalates rather than paging a human for everything.

Engineering velocity. A second-order but real effect: agentic coding tools compress the build timeline of the entire system — ISO API integrations, backtesting harnesses, data pipelines — which disproportionately benefits small teams competing against incumbents with large engineering staffs. The state-of-the-art bidding shop of 2026 is not necessarily the one with the most exotic model; it is often the one whose five engineers ship like twenty.

Layer 2 synthesis: deploy LLMs as retrieval-grounded analysts and drafters across market rules, compliance, settlement, and monitoring, always with human review at the point of action. The advantage is quiet, defensible, and — because competitors under-invest in exactly these functions — durable.

4.3 Layer 3 — The optimizer: incremental by design, and that's fine

The optimization mathematics was never the bottleneck. Newsvendor-style quantile offering for renewables, two-stage stochastic programs with CVaR, and MILP/dynamic-programming formulations for storage are published, taught, and tractable with commercial or open solvers. They are also auditable: every offer can be traced to a forecast input and a constraint, which matters enormously for both internal risk governance and regulatory defense. Any AI proposal for this layer must beat well-tuned classical methods and preserve explainability and bound its failure modes. Two candidate technologies are worth tracking, at very different maturity levels.

4.4 Decision-focused learning: the credible upgrade

The conventional pipeline trains forecasts to minimize statistical error (MSE, pinball loss) and then optimizes against them independently — without considering how forecast errors propagate into decision quality. Decision-focused learning (DFL, also called value-oriented forecasting or "smart predict-then-optimize") integrates the downstream optimization into the training loop, so the forecaster is trained against a regret-style loss measuring the sub-optimality of the decisions its forecasts induce. Founding results: Donti, Amos, and Kolter's task-based end-to-end learning (NeurIPS 2017) and Elmachtoub and Grigas's "Smart Predict-then-Optimize" (Management Science 2022); the energy literature has applied the framework to day-ahead scheduling against energy and reserve markets, storage arbitrage and predict-then-bid frameworks, and robust microgrid operation.

The core insight is directly relevant to bidding: the most accurate forecast is not necessarily the most valuable one. A price forecast can improve its average error while getting worse in exactly the high-priced hours where offer decisions have consequences; the literature has repeatedly shown that a specific quantile choice, not the most accurate point forecast, maximizes trading value. DFL formalizes and automates that intuition. Critically — unlike end-to-end RL — it preserves the auditable optimizer: you change what the forecast is trained for, not who makes the decision.

Production-readiness: not broadly — late-stage research / early adoption. What's mature: established frameworks (smart predict-then-optimize, learning-by-experience, black-box differentiable optimizers), and implicit differentiation through convex optimization layers demonstrated for economic dispatch and storage arbitrage, consistently beating accuracy-trained forecasts on realized profit for simple convex problems. Why it hasn't crossed into broad production:

  1. Fragile differentiation through the optimizer — no closed form for the backward pass; unrolling has accuracy/efficiency issues; analytical methods impose rigid problem-form requirements; mixed-integer/nonconvex bidding needs surrogates, subgradients, or perturbation estimators (brittle in practice).
  2. Partial uncertainty coverage — most DFL handles uncertainty in the objective only, assuming constraint parameters are known; real trading has uncertainty everywhere.
  3. Task-specificity — the model is welded to one decision problem; any change (asset size, market, bid format) forces retraining. Two-stage pipelines are modular: one forecast feeds many decisions.
  4. Interpretability / risk control — profit-trained forecasts are deliberately biased and hard to explain to risk committees; two-stage lets you audit forecast and decision separately.
  5. Data scarcity and regime-shift fragility — policies trained on historical profit can degrade badly exactly when markets shift.

Recent evaluation work adds honest counter-evidence: DFL's benefits are application-dependent and do not always translate into higher economic value, while typically demanding substantially more computation. Adoption barriers are structural too — third-party forecast vendors cannot train against every client's private loss function, and cost-oriented forecast targets lack the intuitive interpretation (mean, median, quantile) that human reviewers rely on.

Rule of thumb:

  • Small convex inner problem + simple asset (one battery, one market): DFL is production-viable today for a strong technical team.
  • Integer commitments, multi-market co-optimization, ISO bid curves: two-stage probabilistic forecasting + optimization remains the deployed standard, likely for several more years.
  • Pragmatic middle ground (what many shops actually do): "decision-aware" training — keep the two-stage architecture but weight forecast loss by economic consequence. Much of the benefit, little of the fragility.

DFL is worth piloting because you own your full pipeline (an advantage of building over buying), evaluated against a strong quantile-forecast baseline on realized trading P&L, and expected to pay off most where the decision problem is asymmetric — storage bidding and imbalance-exposed renewable offers — rather than uniformly.

4.5 Reinforcement learning: shadow mode, not production

End-to-end RL — a learned policy directly emitting offer curves — remains overwhelmingly academic despite a large literature (comprehensive reviews cover 150+ papers). The structural obstacles have not moved: live trial-and-error is impossible at real-market cost, so policies train in simulators whose fidelity gap to the real market has never been closed; day-ahead auctions offer roughly 365 independent samples per year, a brutal sample-efficiency regime; and an unexplainable policy that misbehaves during a scarcity event is both a financial and a regulatory catastrophe.

Where RL-adjacent methods are in production is instructive: approximate/stochastic dynamic programming — value-function methods over sequential decisions, mathematically RL's sibling — is standard inside commercial storage optimizers, and learned policies are credibly deployed in European continuous intraday markets, where thousands of order-placement decisions per day give RL the interaction density it needs.

The sensible roadmap for a US system mirrors that pattern: classical stochastic optimization for the day-ahead auction; RL experiments confined to real-time re-offering and intraday adjustment, where decisions are frequent; mandatory shadow-mode evaluation against the production optimizer before any capital exposure; and, if RL earns its way in, deployment as a bounded residual correction on top of the optimizer's output rather than a replacement for it.

4.6 Vendor "AI bidding" claims: substance vs. hype

Substance: automated 24/7 bidding demonstrably works and is now independently measurable (Modo Energy leaderboards for storage optimizers); ML forecasting genuinely outperforms older methods; operational consistency alone beats human desks.

Hype patterns: "AI" frequently means a standard GBM-forecast + LP-optimizer dressed up; uplift claims lack stated counterfactuals; and the edge from mere automation compresses as adoption spreads — ERCOT ancillary-services saturation being the clear example.

Due-diligence questions for any vendor: Where do you rank on an independent benchmark? What exactly is the model architecture, and what counterfactual sits behind your uplift numbers? How did the system behave during Winter Storm Uri / the last scarcity event? What is the fee structure (revenue-share vs. fixed)?

4.7 Cross-cutting dynamics

The floor rises; the ceiling is private. Nearly every AI advance discussed here is public or purchasable: open weather-model weights, foundation time-series models, commodity LLMs, published DFL methods. Their diffusion compresses the edge available from any single technology — visible already in ERCOT battery markets, where algorithmic saturation of ancillary services shifted revenue toward energy arbitrage and shrank the gap between top and median operators. Durable differentiation migrates to what cannot be bought: proprietary telemetry and its use in local calibration, nodal congestion modeling, execution reliability, and the institutional discipline of a well-run risk process.

Auditability is a feature, not a constraint. The regulatory environment (FERC manipulation doctrine, ISO market-monitor scrutiny) and internal risk governance both reward architectures whose decisions can be explained after the fact. This is a genuine, often-overlooked argument for forecast-then-optimize over end-to-end learned alternatives — and it is why the highest-value applications of the newest AI (weather ensembles, LLM operations, DFL) are precisely the ones that strengthen the classical architecture rather than replacing it.

Independent benchmarking is changing vendor dynamics. Third-party leaderboards for storage optimizers have replaced self-defined vendor benchmarks with a shared reference point, and the same discipline should be applied internally: every AI adoption in the stack deserves a counterfactual (what would the incumbent method have earned?) and a tail-event stress test (how would this component have behaved during Winter Storm Uri or the most recent scarcity event?).

4.8 Adoption roadmap

Sequenced by expected risk-adjusted return on effort:

  1. Now — AI weather ensembles (Layer 1). Pilot one or more AI-weather feeds against the incumbent, scored on your assets and your trading loss function with explicit tail verification. Low integration risk, direct P&L path, and a decaying edge that rewards early movers.
  2. Now — LLM operations layer (Layer 2). Retrieval-grounded market-rule assistant, automated bid-rationale documentation, shadow-settlement triage, and agentic engineering tooling. Human review at every point of action.
  3. Next two quarters — local calibration and congestion modeling (Layer 1). Reinvest weather-layer savings into asset-specific downscaling and nodal congestion prediction; this is where forecast edge survives commoditization.
  4. Next two quarters — DFL pilot (Layer 3). Retrain the production-forecast loss against realized bidding regret for one asset class (storage or a single wind portfolio), evaluated against the quantile baseline on trading P&L, not forecast error.
  5. Opportunistic — RL in shadow mode (Layer 3). Real-time re-offering only; promotion to production contingent on sustained shadow-mode outperformance and bounded-action deployment.

The through-line: recent AI does not overturn the architecture of a well-built bidding system. It makes the boring architecture better — sharper distributions in, cheaper discipline around, and a modestly smarter objective inside — which, given where the secrets actually live, is exactly the outcome a builder should want.

5. Building a bidding system

5.1 Architecture for a renewables + wholesaler bidding system

Five-stage pipeline: data ingestion → probabilistic forecasting → bid optimization → submission → settlement/monitoring, with a feedback loop retraining models on settlement outcomes.

Key design decisions:

  • Forecasting: quantile forecasts (P10–P90) for production and load, built on NWP ensemble inputs blended with asset telemetry; DA LMP, RT LMP, and crucially the DA–RT spread distribution.
  • Optimization core: for a price-taking renewable, the optimal DA offer is a newsvendor quantile of the production distribution set by the expected DA/RT price ratio, wrapped in a two-stage stochastic program with CVaR. Tractable LP per asset per day (Gurobi/CPLEX/HiGHS). The wholesaler side is the mirror image: DA-vs-RT load procurement plus a hedging overlay (forwards, FTRs).
  • Operations: automated ISO submission with rule validation, shadow settlement, and a serious backtesting harness — the most commonly under-built component.
  • Gotchas that dominate real P&L: negative prices and PTC/REC-driven offer floors; nodal basis and congestion (often the highest-ROI forecasting work); PPA/hedge structures reshaping incentives; per-ISO rule differences.

Note for hybrid assets: storage co-location changes the optimizer from an LP to a sequential problem (see §2.3).

5.2 Separating hedging from speculation

Separation is necessary for three reasons:

  1. Performance attribution — the two books have opposite definitions of success.
  2. Risk governance — bounded physical exposure vs. leveraged financial loss distributions (GreenHat as the cautionary tale).
  3. FERC manipulation risk — physical bids must be independently defensible; joint optimization of physical bids and financial positions is a manipulation-allegation generator.

But complete separation is impossible: every DA offer quantile is a spread view. Resolution: define a neutral baseline bid (e.g., P50 offer / 100% DA load), log every deviation as "embedded alpha," and decompose P&L into baseline + embedded view + pure spec book. Architecturally: share data, forecasting, and backtesting; split optimizers, mandates, limits, and P&L; keep coupling one-directional — the physical optimizer never reads the financial book's positions.

The two books consume the same forecast stack differently: the hedge book needs calibrated mid-distribution quantiles; the spec book needs tails and spread skew.

Strategic note: virtuals/FTRs are crowded quant markets; embedded-view alpha inside a well-run asset book is often the better initial return on modeling effort.

6. China: market structure and deployment

6.1 Market structure: different design, weaker safeguards

The market operator is NOT independent — the fundamental difference from the US. Provincial spot markets are run by dispatch centers and trading centers inside the grid companies (State Grid, China Southern Grid). Trading centers are nominally independent via shareholding reform, but dispatch remains within grid organizations. Grid companies historically had commercial interests in outcomes — the "insider" is structural, not hypothetical. Dispatch/trading separation has been a reform demand since Document No. 9 (2015).

Transparency: improving but well below PJM. A "1+6" national rule framework has been built since 2023: Basic Rules for Market Operation + spot market (2023), information disclosure (2024), medium/long-term trading, registration, ancillary services (2025), metering/settlement (2025). Provinces (Shanxi, Guangdong, Shandong) publish clearing methodology; several use centralized SCUC/SCED-style clearing, some nodal/zonal. Gaps vs. PJM: no bid-level data (even masked/lagged), little constraint-level detail, clearing-model internals undisclosed — shadow-SCED reconstruction is far harder. Rules are revised frequently by administrative notice and differ substantially by province; price caps/floors are tight, and administrative interventions are more common and less documented.

Laws and enforcement: embryonic. No independent market monitor analogous to Monitoring Analytics — surveillance is done by trading centers/dispatch (the grid companies) plus thinly-staffed NEA bureaus. No developed body of power-market manipulation case law; general laws are untested on spot-power conduct. SOE-dominated participation and administrative dispute resolution mean little deterrence-by-precedent.

Net assessment. All three PJM safeguards (§3.2) are weaker in China: partial algorithm disclosure, structural data asymmetry favoring grid-affiliated entities, nascent enforcement. The realistic information edge comes from proximity to grid/dispatch institutions, not rogue vendors. Mitigants: heavy medium/long-term contract coverage (~47% of consumption in 2024) plus price caps mean less money at stake in spot per unit of information advantage, and the reform direction clearly points toward disclosure and oversight. Watch: whether dispatch is ever truly separated from the grid companies. National unified market targeted ~2029; seven provinces in full spot operation as of 2025 (Shanxi, Guangdong, Shandong, Gansu, W. Inner Mongolia, Hubei, Zhejiang).

6.2 Weather data sourcing for a China deployment

Baseline reviewed stack — ECMWF HRES+EPS backbone, GFS/GEFS secondary, CMA regional models + observations, Himawari/Fengyun satellite irradiance, customer met masts, buy-first on WRF downscaling — is conventionally sound. Recommended changes:

  1. Add AI weather models — the glaring omission. AIFS open data plus open-weight models (GraphCast/Pangu) densify the ensemble at near-zero marginal cost; China's domestic AI-weather ecosystem (Pangu, Fuxi, FengWu, CMA-integrated) is both cheap and a compliance asset; GPU-cheap models unlock the intraday refresh cadence the licensed stack can't provide.
  2. Add ERA5/reanalysis on day one — the training substrate for calibration models and backtesting.
  3. Make the blending/calibration layer explicit with its own budget line — multi-model ensembles calibrated per-site against ground truth is where the money is; source diversity is calibration fuel.
  4. Specify a fallback hierarchy (EPS → AIFS open → GFS → persistence) as an engineering requirement.
  5. Two-way compliance review early: ECMWF redistribution terms × Chinese regulations on foreign-data commercial use and domestic-observation export — this constrains where the blending layer can physically run.
  6. Add inverter/SCADA data as ground truth (free, dense, bakes in real plant behavior), plus a QC pipeline for customer sensors.
  7. Reframe "self-run WRF" as "self-run downscaling," method open — learned downscaling will likely beat WRF on cost by decision time.
  8. Design the satellite-nowcast → NWP blending handover in the 0–4h window explicitly, to avoid a discontinuity in the intraday trading horizon.

7. Open questions

Carried forward from the discussions:

  • Which market(s)? US ISOs vs. Chinese provincial spot pilots — gate-closure times and settlement rules change the forecast-refresh requirements.
  • Asset mix? Solar/wind/hybrid-with-storage — storage co-location changes the optimizer from an LP to a sequential problem.
  • Risk mandate? Appetite for a standalone financial (virtuals/FTR) book vs. embedded-alpha-only inside the asset book.

8. Sources and further reading

Prepared July 2026. Vendor performance claims cited in the underlying research should be independently verified against your own assets before procurement decisions.