power-market-trading-docs/pjm/design_models_en.md

Forecasting Models for Renewable Day-Ahead Bidding in PJM

Working design document — v0.1 (July 2026)

Context: We are building a bid-optimization business (Model 2: SaaS, with a likely migration path toward Model 3: asset management / market participant services) that helps renewable generators submit day-ahead (DA) offers into PJM. This document captures the background and technical design of the three core forecasting models that feed the bid optimizer, for further discussion and iteration.

---

0. How the three models fit together

The product's core loop is a newsvendor-style quantity decision: the optimal DA commitment for a renewable plant is a quantile of its production forecast distribution, where the quantile is chosen based on the expected DA–RT spread and the asymmetry of settlement outcomes, corrected for the covariance between the plant's production error and the spread.

                 ┌─────────────────────┐
                 │ Model B: Production  │──── production quantiles (per plant, per hour)
                 └─────────────────────┘                    │
                                                            ▼
┌─────────────────────┐      regime features      ┌──────────────────┐
│ Model A: DA Price    │──────────────────────────▶│  Bid Optimizer   │──▶ DA offer curve
│ (level, coarse)      │                           │  (newsvendor +   │    (quantity + price legs)
└─────────────────────┘                           │   covariance)    │
                 ┌─────────────────────┐          └──────────────────┘
                 │ Model C: DA–RT       │──── spread quantiles (per node, per hour)
                 │ Spread               │
                 └─────────────────────┘

Division of precision requirements:

Model	Precision needed	Role in the bid
A. DA price level	Low (regime/coarse)	Regime conditioning for Model C; negative-price / offer-floor tail probabilities; revenue expectations
B. Production	High (calibrated quantiles)	The quantity axis of the bid
C. DA–RT spread	High (conditional distribution, esp. tails)	Selects the production quantile to commit

Key insight driving this architecture: for a quantity-bidding renewable, the DA–RT spread and own-production distribution jointly determine the optimal bid; the absolute price level matters only (a) as a conditioning regime for the spread, (b) in the tail near the plant's offer floor (negative prices / economic curtailment), and (c) for reporting/hedging.

---

1. Model A — DA Price Level (coarse / regime model)

1.1 Purpose

Not a desk-grade price forecast. Three narrowly-scoped jobs:

1. Regime classification of tomorrow: e.g. {mild / normal / tight / scarce} and {gas-marginal / coal-marginal / renewable-surplus}, used as conditioning features by Model C.
2. Offer-floor tail probabilities: P(nodal LMP < floor) per hour, for both DA and RT — the negative-price classifier. Determines the dispatch-probability weighting of production (truncation of Model B's distribution) and the price legs of the offer curve.
3. Expected price levels for client revenue reporting and hedge support.

1.2 Why we do NOT replicate SCUC

The DA price is the output of PJM's security-constrained unit commitment / economic dispatch, so full structural replication (Dayzer / PROMOD-class) is conceptually "correct" but practically dominated for next-day horizons:

• Offers are confidential at bid time (published ~4 months later, masked). Marginal-unit cost reconstruction errors land exactly where prices are set.
• DA is a financial equilibrium, not a pure physical SCUC: virtual transactions (INCs/DECs/UTCs) push DA toward the market's RT expectation. A physical replica misses this layer entirely.
• Network model is CEII-restricted; monitored constraints and operating limits vary daily with operator judgment. Congestion — the nodal differentiator — is the hardest component to replicate.
• SCUC is combinatorial: small input errors flip discrete commitment decisions; even PJM's own solution is tolerance-gapped, not unique.
• Out-of-market operator actions are unobservable in advance.

Empirical consensus: statistical/ML models beat structural replicas for tomorrow's prices; structural models win for counterfactuals, congestion anatomy, long horizons, and new regimes.

1.3 Our approach: "mimic SCUC's logic, not its solution" (hybrid-lite)

Build a reduced supply-stack view to generate features, then let a statistical model learn the residual mapping:

• Stack tightness: PJM load forecast minus available thermal capacity (outage-feed adjusted) minus fleet renewable forecast. Strongly nonlinear — matters above ~93% utilization.
• Implied marginal fuel / regime flag from the reconstructed stack and gas signals.
• Constraint watch-list per client region: reconstructed from PJM's historical binding-constraints feed + planned transmission outages (OASIS), not from a power-flow model.

Statistical layer: gradient-boosted models for expected level; a dedicated binary classifier for negative-price hours (features: low load forecast, high fleet solar/wind forecast, weekend/holiday, spring/fall, local congestion conditions).

1.4 Data

Data	Source	Cost
DA/RT LMPs (energy/congestion/loss components)	PJM Data Miner (da_hrl_lmps, rt_hrl_lmps, rt_fivemin_hrl_lmps)	Free
Load forecast, metered load	PJM Data Miner	Free
Fleet wind/solar forecasts + actuals	PJM Data Miner	Free
Generation outages (aggregate MW)	PJM Data Miner	Free
Binding constraints	PJM Data Miner	Free
Planned transmission outages	PJM OASIS	Free
Nuclear unit status	NRC daily power reactor status	Free
Gas: Henry Hub spot, NYMEX settlements	EIA, CME	Free-ish
Gas: eastern basis (TETCO M3, Transco Z6, Dominion S, TCO)	Platts / NGI / ICE	Paid — highest-value paid dataset
Weather forecasts (temp, wind, irradiance) + ensemble spread	NOAA (GFS/HRRR/NDFD), ECMWF open data	Free (self-archive!)
Neighboring ISO prices/loads (MISO, NYISO)	Their public portals	Free

1.5 Known structural breaks to encode

• Fleet turnover: retirements, new entry (a 2019-trained model misprices 2026).
• Reserve/ORDC market rule changes (flag dates; distrust pre-change history).
• PTC/ITC fleet composition drift (see §4): as neighboring PTC plants age out of their 10-year windows, historical negative-price frequency becomes a biased predictor.

---

2. Model B — Probabilistic Production Forecast (per plant)

2.1 Purpose and target

For every hour of tomorrow, by ~10:00 AM ET (ahead of the 10:30 DA close): a calibrated quantile set of the plant's available production, e.g. P10 / P25 / P50 / P75 / P90. The bid optimizer commits a quantile of this distribution — a point forecast is structurally insufficient.

Horizon: ~14–38 hours at bid time ⇒ NWP (numerical weather prediction) carries essentially all the signal; persistence is worthless at this range.

2.2 Architecture: two-stage pipeline

Stage 1 — Weather-to-power model (per site). Learn the mapping NWP forecast → SCADA output with gradient-boosted trees (or similar). This absorbs the plant's true power curve, wake losses, inverter clipping, terrain effects, soiling, and the NWP model's local biases.

Critical rule: train on forecast weather, not measured weather. The model must learn NWP's error characteristics end-to-end; training on met-mast actuals then predicting from NWP degrades live performance.

Stage 2 — Distributional layer. Three interchangeable routes:

1. Quantile regression (pinball loss, one model per quantile) — default starting point; native in modern GBM libraries.
2. NWP ensembles (e.g. ECMWF 51-member): push each member through Stage 1 → 51 scenarios → empirical quantiles. Preserves temporal correlation across hours (valuable for multi-hour bid coupling).
3. Analog / error-dressing: dress the point forecast with the historical error distribution from similar conditions.

2.3 Calibration — the core quality bar

A forecast is calibrated if actuals fall below the stated P10 ~10% of the time, etc. (verify with reliability diagrams). Miscalibration translates one-for-one into settlement losses, because the entire bid strategy is "commit quantile q."

• Vendor recalibration layer: even with purchased forecasts, recalibrate in-house via quantile mapping against our own SCADA archive. Small project, high ROI. Archive every vendor forecast ever received.
• Curtailment contamination: SCADA records produced, not producible. Reconstruct available power (turbine-anemometry / unconstrained inverter capability signals); flag curtailed intervals; never calibrate on contaminated ground truth.
• Availability separation: outage-driven shortfalls are not weather error; scale ground truth to available capacity or feed availability as a feature. Note the asymmetry — unplanned outages only subtract (skews the lower tail).

2.4 Build vs. buy

• Buy (vendors: Solargis, Meteomatics, UL, DNV, many others; ~$1–3k/site/month) when starting; always keep the in-house recalibration + evaluation layer. Score vendors quarterly with pinball loss on our SCADA; running two vendors in parallel and blending often beats either.
• Build at portfolio scale: pipeline cost amortizes; enables custom targets (available power under our curtailment logic), guaranteed point-in-time archive, direct optimizer integration.
• Business-model advantage (cross-client learning): with many plants' SCADA under management, weather-to-power models transfer across sites, NWP bias is calibrated regionally, and fleet-wide forecast-error days (the spread driver) become observable. Contracts must permit pooled/anonymized model training from day one.

2.5 Accuracy expectations (day-ahead horizon)

• Solar point forecasts: ~5–10% of capacity RMSE (climate-dependent).
• Wind: ~8–15% of capacity (cube-law amplification of speed errors).
• But judge the distribution: pinball loss across quantiles + tail-focused reliability, benchmarked vs. climatology-dressed persistence.

---

3. Model C — DA–RT Spread Model (per node)

3.1 Purpose

The money model. Produces per-hour, per-node conditional quantiles of S = DA − RT, which select the production quantile to commit (newsvendor), with a covariance correction for the client's own production error.

3.2 Statistical character of the target

• Near-zero unconditional mean by construction: virtuals arbitrage away any persistent gap; what remains is a small conditional risk premium (DA tends to run rich into expected scarcity) plus transient inefficiencies. Low signal-to-noise; expect modest R². Value lives in conditioning and tails.
• Violently asymmetric tails: DA is a smoothed expectation; RT is the spiky realization (scarcity adders → $850+ prints; renewable surplus → negative RT). Fat left tail (RT spike above DA), moderate right tail (RT crash). Gaussian assumptions are disqualifying.
• Weak day-over-day autocorrelation, strong conditional structure (hour, season, tightness, weather uncertainty).

3.3 Decomposition (model the two components separately)

S_node = S_system (energy/hub component) + S_congestion (ΔDA−RT congestion at node)

• S_system: total supply-demand — load forecast error, fleet renewable forecast error, post-DA forced outages, reserve scarcity. Built once, shared across all clients.
• S_congestion: constraints binding in RT but not priced DA (or vice versa) — transmission forced outages, unexpected flows. Often dominant for renewable pockets and the more predictable component (RT congestion persistence during outages). Per-node work; our defensible IP layer.

Directly computable from Data Miner's LMP component breakdown for both markets.

3.4 Architecture: regime classifier × conditional distribution × covariance

Stage A — Spike/regime classifiers (discrete events that dominate P&L):

• P(RT spike above DA): tightness, reserve margin, extreme-temperature forecasts and their ensemble uncertainty, high fleet renewable forecast (underdelivery risk), recent forced outages, day type.
• P(RT crash / negative RT): fleet renewable forecast, low load, inflexible baseload share, local constraint state. Shares infrastructure with Model A's negative-price classifier.

Stage B — Conditional spread quantiles, given regime probabilities: quantile GBM on continuous features, or analog draws from regime-matched historical spread distributions.

Stage C — Production–spread covariance correction (business-specific edge): fleet-overproduction days crash RT exactly when our clients overproduce — the production error and S are negatively... [correlated such that overscheduling is punished]. Implement by conditioning the spread model on the client's own production forecast error, or estimate within-regime correlation and adjust the newsvendor quantile analytically. Generic price shops skip this; we must not.

3.5 Feature set, ranked by expected alpha

1. Tightness (load forecast − available capacity), nonlinear above ~93%.
2. Renewable forecast level and revision velocity across recent NWP cycles (late revisions ⇒ DA cleared on stale info ⇒ spread opportunity).
3. Weather forecast uncertainty (ensemble spread) — RT volatility fuel.
4. Trailing RT-vs-DA congestion at the node + constraint binding frequency, cross-referenced with planned transmission outages.
5. Aggregate cleared virtual volumes (published with lag) — arbitrage efficiency regime.
6. Calendar interactions (hour × season) + structural-break flags (rule changes).

3.6 Evaluation protocol

• Pinball loss vs. the brutal baseline S ≡ 0 ("market is efficient"). Beating it consistently out-of-sample under point-in-time discipline is hard; a huge backtest win ⇒ hunt for leakage first.
• Economic backtest: full newsvendor loop vs. naive P50 bidding; target metric is $/MWh uplift (good implementations: ~$0.5–2/MWh).
• Tail calibration specifically (P5/P95), and regime-sliced evaluation — average-fine models are often terrible in the ~30 days/year that drive annual P&L.

3.7 Portfolio risk (existential, not statistical)

Clients are collectively long renewables ⇒ model errors are correlated across the book. A wrong regime call on a fleet-overproduction day is wrong for every wind client simultaneously. First-class risk metric from day one: aggregate MWh leaning long DA across clients, stress-tested against spike scenarios.

---

4. Cross-cutting: offer floors, tax credits, and the price legs

The quantity bid (from Models B + C) pairs with price legs set by each asset's true marginal cost, which is determined by its tax-credit election:

Asset type	Marginal cost	Rational offer floor	Runs at negative prices?
PTC (in 10-yr window)	≈ −(PTC × tax gross-up)	~−$25 to −$35/MWh	Yes, to the floor
ITC	≈ $0	~$0/MWh	No
Post-PTC-window (yr 11+)	≈ $0	~$0/MWh	No

• PTC (≈$27.50–30/MWh, inflation-adjusted, wage/apprenticeship-compliant, paid on generation for 10 years) shifts the floor negative; ITC (30%+ of capex, paid on investment) does not affect marginal cost.
• IRA made credits tech-neutral (45Y/48E) from 2025 — new high-CF solar increasingly elects PTC ⇒ solar fleet also develops negative floors (structural shift). OBBBA (July 2025) accelerated wind/solar phase-out (placed-in-service generally by end-2027, with begin-construction safe harbor) — existing plants keep locked-in credits. Verify current guidance per project.
• Interaction with Model A's tail job: expected production for bidding is dispatch-probability-weighted: E[dispatched output] ≈ physical forecast × P(LMP ≥ floor), computed per hour, separately for DA and RT (the DA-negative/RT-positive and DA-positive/RT-negative cases settle very differently). Naive use of the physical forecast overcommits most on exactly the high-output days when curtailment risk peaks.
• Deviation settlement nuance: underdelivery against a DA position during negative RT prices can be profitable (sell DA positive, buy back negative) — which is precisely why deviation charges and must-offer rules exist. Any autobidder logic near this line needs compliance review (Market Monitor attention risk). Verify current PJM settlement/deviation rules; they change.

4.1 BORD-style deviation charges (first-class optimizer input)

"BORD" = PJM's Balancing Operating Reserve Deviation charges — a cost- allocation mechanism that functions as a de facto penalty on deviating from the day-ahead position.

• Mechanism. PJM incurs uplift (make-whole payments to units committed or dispatched whose LMP revenues don't cover offered costs). The balancing portion — costs arising after DA close, i.e., from RT diverging from the DA plan — is allocated substantially by cost causation to deviations: generators off their DA schedule (either direction), load off its bid, virtuals (which never deliver by construction). Each deviation-MWh attracts a $/MWh charge rate that varies daily with uplift incurred and total deviation-MWh.
• Effect on the bid. Adds a second term to the deviation leg of the newsvendor objective: every MWh of |Q_rt − Q_da| picks up the charge. It is a friction/transaction cost on leaning away from expected production — the optimal quantile shifts back toward P50 as the expected charge rate rises. The optimizer therefore needs the deviation charge rate (or a forecast of it) alongside the spread forecast.
• Loophole patch. Deviation charges (plus Market Monitor scrutiny) are what make systematically engineering profitable underdelivery at negative RT prices costly — see the nuance bullet above.
• Caveat. PJM's uplift allocation has been litigated and reformed repeatedly (cost buckets, deviation definitions, intermittent-resource exemptions, netting). The settlement module must pull current parameters from the live tariff, never from memory. "BORD-style" is shorthand for the category: ex-post, cost-causation-based charges on DA-vs-RT deviations.
• Contrast (for the China work). China's 两个细则 achieves deviation discipline through administrative performance assessment — penalties against regulated forecast-accuracy/schedule-compliance thresholds, largely independent of actual system balancing cost. PJM prices the externality (optimize deviations against a forecastable stochastic price); China grades the homework (engineer forecasts against a fixed rulebook). This is also why China's forecasting market became a compliance-procurement market with no PJM equivalent.

4.2 Hedge book vs. spread book: separability and its limits

Most client plants carry a hedge (VPPA, futures strip, bank hedge). The bid optimizer's relationship to that hedge rests on a three-layer statement: separable in accounting and first-order decisions, correlated in risk, coupled in second-order (risk-adjusted) decisions.

Layer 1 — additive separability (the "yes"). Hourly revenue:

R = Q_da·P_da + (Q_rt − Q_da)·P_rt     (market legs — bid optimizer's domain)
  + (P_c − P_ref)·Q_c                  (hedge leg — fixed CfD)

The hedge leg contains no daily decision variable: Q_c, P_c were set at signing; P_ref is a hub index the plant's bid does not move. Q_da appears only in the market legs. Spread alpha therefore adds on top of hedge P&L, and each book can be evaluated independently without double-counting. This is the license for treating the hedge as a static given throughout §1–3.

Layer 2 — correlated outcomes (the first "no"). Both legs are driven by the same prices: rearranged, the market legs contain Q_da·(P_da − P_rt) and the hedge leg contains −P_da (for DA-settled references). On a scarcity day, an underscheduled spread position and a short-at-P_c hedge lose together. Independent decisions, correlated P&L — portfolio risk reporting must treat them jointly.

Layer 3 — risk-adjusted coupling (the second "no"). Separability of decisions holds only under expected value. Under mean–variance/CVaR, the optimal DA lean depends on the plant's residual price exposure after the hedge: heavily hedged plants can lean harder on the spread; merchant plants should shade conservative because the spread bet stacks on large open price risk. The hedge book enters the bid optimizer through the risk term, not the P&L identity — a required client-onboarding input (hedge volume, tenor, settlement point and index).

Caveats that can break even Layer 1:

• Contract form. Fixed-volume financial hedges separate cleanly. Unit-contingent / as-generated PPAs set Q_c = Q_rt (hedge leg co-varies with production, though still not with Q_da); physical PPAs with delivery obligations can constrain the bidding problem directly; proxy revenue swaps absorb shape/volume risk and change what remains for the spread book to manage.
• Settlement reference. DA-settled vs RT-settled hedges leave different residual exposures: an RT-settled hedge plus DA scheduling exposes the hedged volume to the DA–RT spread too — equivalent to an embedded virtual position, sometimes intended, sometimes an accident. The optimizer must know which it is.

(Chinese-market transfer of this argument, with four deltas: see the China product design doc §2.7.)

---

5. Data and operational infrastructure (shared foundations)

5.1 Point-in-time discipline (non-negotiable, applies to all three models)

Every feature must reflect the information set available before the 10:30 AM ET DA close (in practice, snapshot at ~10:00):

• Weather: the NWP cycle actually available then (e.g. 00Z run), not later cycles, never realized weather.
• PJM feeds: as-of snapshots (feeds get revised).
• Vendor forecasts: archive as-received (vendors overwrite).
• Plant availability: as known at bid time.

Any post-close information in training data ⇒ backtests flatter, live underperforms "mysteriously." The as-of archive is a compounding business asset that cannot be reconstructed retroactively — competitors can't buy it.

5.2 PJM access and licensing (business-model obligations)

• Data Miner data is internal-use only; redistribution of data or derivatives requires a PJM redistribution license (Associate Membership at minimum). Whether our forecasts count as "derived data" must be resolved with PJM + counsel before product launch.
• Rate limits: 6 connections/min (non-member) vs. 600 (member) — membership is operationally necessary at portfolio scale anyway.
• Bid submission (Model 2 "software agent" variant / Model 3): Markets Gateway API (separate, authenticated), credentialing, account-security rules for third-party agents — confirm current requirements with PJM member services.
• Commercial data (gas indices, weather vendors): internal-use vs. redistribution tiers priced very differently; contract accordingly.

5.3 Settlement feedback loop

Clients' actual PJM settlement statements are ground truth for validating Model C and the economic backtest. Contract for access to settlement data (validation + model training rights) in every client agreement.

---

6. Build sequence (recommended)

Phase	Deliverable	Models touched
1	Empirical spread atlas: per node-cluster/hour/season spread distributions, negative-RT and spike frequencies. Zero ML; sellable as analytics; seeds Stage-B analog library.	C
2	Point-in-time archive pipeline (PJM feeds, NOAA/ECMWF, self-archived from day one).	A, B, C
3	System regime classifiers (spike/crash) on free PJM + NOAA data; negative-price classifier (shared A/C).	A, C
4	Nodal congestion layer: trailing RT-vs-DA congestion features + transmission-outage cross-reference for actual client nodes.	C
5	Production pipeline: vendor-based with in-house recalibration; migrate to in-house Stage-1/Stage-2 at portfolio scale.	B
6	Joint production–spread covariance correction per client (needs their SCADA).	B + C
7	Newsvendor bid optimizer + economic backtesting harness vs. P50-naive baseline.	All
8	Portfolio-correlation risk dashboard (aggregate DA lean, spike stress tests).	C / risk
9	Second-order upgrades: distributional deep learning, joint scenario generation, structural stack refinement.	All

---

7. Open questions for next discussion

1. Newsvendor math in full: closed-form optimal quantile with the covariance correction and deviation-charge asymmetry — worth deriving and unit-testing.
2. Target client nodes/regions: western wind (ComEd/AEP) vs. Mid-Atlantic solar have very different congestion stories; prioritizes the Phase-4 constraint library.
3. Vendor bake-off design for production forecasts (which two vendors, scoring protocol, blend rule).
4. Hybrid/storage clients: brings the price-level shape forecast back as first-order (arbitrage is a level-shape problem) — roadmap trigger point.
5. Intraday/rebidding scope: PJM rebid windows and RT strategy are out of scope for v1 but affect architecture (temporal correlation needs from Model B).
6. Compliance review checklist for autobidder logic near deviation-settlement edges; agent-framework clarification with PJM member services.
7. Must-offer / capacity-resource obligations per client asset — shifts optimal quantiles and constrains offer flexibility; needs current-rules verification.

---

Appendix: Glossary

Term	Definition
LMP	Locational Marginal Price — nodal energy price, decomposed into energy + congestion + loss components
DA / RT	Day-ahead market (cleared ~10:30 AM ET for next day, hourly) / real-time market (5-minute dispatch and settlement)
DA–RT spread (S)	S = DA − RT price; the object of Model C; near-zero mean by arbitrage, fat asymmetric tails
Newsvendor bid	Optimal DA quantity = a quantile of the production distribution, where the quantile is set by the expected spread and settlement asymmetry
Quantile / pinball loss	Value below which the outcome falls X% of the time / the loss function that trains and scores quantile forecasts
Calibration	Property that stated probabilities are honest (P10 exceeded ~90% of the time); checked with reliability diagrams
SCUC	Security-Constrained Unit Commitment — the mixed-integer optimization the ISO runs to clear the DA market
Virtuals (INC/DEC/UTC)	Purely financial DA positions that never deliver physically; arbitrage DA toward expected RT
Uplift	Make-whole payments to units whose market revenues don't cover offered costs (startup, no-load, min-run)
BORD	Balancing Operating Reserve Deviation charges — PJM's allocation of real-time balancing uplift to DA-vs-RT deviations (§4.1)
ORDC / scarcity adders	Operating Reserve Demand Curve — administrative price adders when reserves run short; drives RT spikes
FTR	Financial Transmission Right — hedge/speculative instrument on DA congestion between two nodes
PTC / ITC	Production Tax Credit ($/MWh generated, 10 years; sets negative offer floors) / Investment Tax Credit (% of capex; floor ≈ $0) — see §4
Offer floor	The price leg below which the plant prefers curtailment: ≈ −(PTC × tax gross-up) for PTC assets, ≈ $0 for ITC assets
Must-offer / RPM	Obligation of capacity resources to offer into the DA market / PJM's capacity market (Reliability Pricing Model)
BTM	Behind-the-meter (e.g., rooftop solar netted out of observed load)
NWP	Numerical Weather Prediction (GFS, HRRR, ECMWF); ensembles = many perturbed runs, source of forecast-uncertainty features
SCADA	Plant supervisory control and data acquisition — ground-truth production, availability, curtailment records
Point-in-time discipline	Every training/backtest feature must reflect only information available before the 10:30 AM DA close (§5.1)
Data Miner 2 / Markets Gateway	PJM's public data API / PJM's authenticated bid-submission API
QSE / scheduling agent	Entity with market-participant infrastructure that submits offers and handles settlement on an asset's behalf