power-market-trading-docs/pjm/design_models_en.md

# Forecasting Models for Renewable Day-Ahead Bidding in PJM

**Working design document — v0.1 (July 2026)**

Context: We are building a bid-optimization business (Model 2: SaaS, with a likely
migration path toward Model 3: asset management / market participant services) that
helps renewable generators submit day-ahead (DA) offers into PJM. This document
captures the background and technical design of the three core forecasting models
that feed the bid optimizer, for further discussion and iteration.

---

## 0. How the three models fit together

The product's core loop is a **newsvendor-style quantity decision**: the optimal DA
commitment for a renewable plant is a quantile of its production forecast
distribution, where the quantile is chosen based on the expected DA–RT spread and
the asymmetry of settlement outcomes, corrected for the covariance between the
plant's production error and the spread.

```
                 ┌─────────────────────┐
                 │ Model B: Production  │──── production quantiles (per plant, per hour)
                 └─────────────────────┘                    │
                                                            ▼
┌─────────────────────┐      regime features      ┌──────────────────┐
│ Model A: DA Price    │──────────────────────────▶│  Bid Optimizer   │──▶ DA offer curve
│ (level, coarse)      │                           │  (newsvendor +   │    (quantity + price legs)
└─────────────────────┘                           │   covariance)    │
                 ┌─────────────────────┐          └──────────────────┘
                 │ Model C: DA–RT       │──── spread quantiles (per node, per hour)
                 │ Spread               │
                 └─────────────────────┘
```

Division of precision requirements:

| Model | Precision needed | Role in the bid |
|---|---|---|
| A. DA price level | **Low** (regime/coarse) | Regime conditioning for Model C; negative-price / offer-floor tail probabilities; revenue expectations |
| B. Production | **High** (calibrated quantiles) | The quantity axis of the bid |
| C. DA–RT spread | **High** (conditional distribution, esp. tails) | Selects the production quantile to commit |

Key insight driving this architecture: for a quantity-bidding renewable, the DA–RT
spread and own-production distribution jointly determine the optimal bid; the
absolute price level matters only (a) as a conditioning regime for the spread,
(b) in the tail near the plant's offer floor (negative prices / economic
curtailment), and (c) for reporting/hedging.

---

## 1. Model A — DA Price Level (coarse / regime model)

### 1.1 Purpose

Not a desk-grade price forecast. Three narrowly-scoped jobs:

1. **Regime classification** of tomorrow: e.g. {mild / normal / tight / scarce} and
   {gas-marginal / coal-marginal / renewable-surplus}, used as conditioning
   features by Model C.
2. **Offer-floor tail probabilities**: P(nodal LMP < floor) per hour, for both DA
   and RT — the negative-price classifier. Determines the dispatch-probability
   weighting of production (truncation of Model B's distribution) and the price
   legs of the offer curve.
3. **Expected price levels** for client revenue reporting and hedge support.

### 1.2 Why we do NOT replicate SCUC

The DA price is the output of PJM's security-constrained unit commitment /
economic dispatch, so full structural replication (Dayzer / PROMOD-class) is
conceptually "correct" but practically dominated for next-day horizons:

- **Offers are confidential** at bid time (published ~4 months later, masked).
  Marginal-unit cost reconstruction errors land exactly where prices are set.
- **DA is a financial equilibrium, not a pure physical SCUC**: virtual
  transactions (INCs/DECs/UTCs) push DA toward the market's RT expectation. A
  physical replica misses this layer entirely.
- **Network model is CEII-restricted**; monitored constraints and operating limits
  vary daily with operator judgment. Congestion — the nodal differentiator — is
  the hardest component to replicate.
- **SCUC is combinatorial**: small input errors flip discrete commitment
  decisions; even PJM's own solution is tolerance-gapped, not unique.
- **Out-of-market operator actions** are unobservable in advance.

Empirical consensus: statistical/ML models beat structural replicas for
tomorrow's prices; structural models win for counterfactuals, congestion anatomy,
long horizons, and new regimes.

### 1.3 Our approach: "mimic SCUC's logic, not its solution" (hybrid-lite)

Build a **reduced supply-stack view** to generate features, then let a statistical
model learn the residual mapping:

- **Stack tightness**: PJM load forecast minus available thermal capacity
  (outage-feed adjusted) minus fleet renewable forecast. Strongly nonlinear —
  matters above ~93% utilization.
- **Implied marginal fuel / regime flag** from the reconstructed stack and gas
  signals.
- **Constraint watch-list** per client region: reconstructed from PJM's historical
  binding-constraints feed + planned transmission outages (OASIS), not from a
  power-flow model.

Statistical layer: gradient-boosted models for expected level; a dedicated binary
classifier for negative-price hours (features: low load forecast, high fleet
solar/wind forecast, weekend/holiday, spring/fall, local congestion conditions).

### 1.4 Data

| Data | Source | Cost |
|---|---|---|
| DA/RT LMPs (energy/congestion/loss components) | PJM Data Miner (`da_hrl_lmps`, `rt_hrl_lmps`, `rt_fivemin_hrl_lmps`) | Free |
| Load forecast, metered load | PJM Data Miner | Free |
| Fleet wind/solar forecasts + actuals | PJM Data Miner | Free |
| Generation outages (aggregate MW) | PJM Data Miner | Free |
| Binding constraints | PJM Data Miner | Free |
| Planned transmission outages | PJM OASIS | Free |
| Nuclear unit status | NRC daily power reactor status | Free |
| Gas: Henry Hub spot, NYMEX settlements | EIA, CME | Free-ish |
| Gas: eastern basis (TETCO M3, Transco Z6, Dominion S, TCO) | Platts / NGI / ICE | Paid — highest-value paid dataset |
| Weather forecasts (temp, wind, irradiance) + ensemble spread | NOAA (GFS/HRRR/NDFD), ECMWF open data | Free (self-archive!) |
| Neighboring ISO prices/loads (MISO, NYISO) | Their public portals | Free |

### 1.5 Known structural breaks to encode

- Fleet turnover: retirements, new entry (a 2019-trained model misprices 2026).
- Reserve/ORDC market rule changes (flag dates; distrust pre-change history).
- PTC/ITC fleet composition drift (see §4): as neighboring PTC plants age out of
  their 10-year windows, historical negative-price frequency becomes a biased
  predictor.

---

## 2. Model B — Probabilistic Production Forecast (per plant)

### 2.1 Purpose and target

For every hour of tomorrow, by ~10:00 AM ET (ahead of the 10:30 DA close): a
**calibrated quantile set** of the plant's *available* production, e.g. P10 / P25 /
P50 / P75 / P90. The bid optimizer commits a quantile of this distribution — a
point forecast is structurally insufficient.

Horizon: ~14–38 hours at bid time ⇒ NWP (numerical weather prediction) carries
essentially all the signal; persistence is worthless at this range.

### 2.2 Architecture: two-stage pipeline

**Stage 1 — Weather-to-power model (per site).**
Learn the mapping NWP forecast → SCADA output with gradient-boosted trees (or
similar). This absorbs the plant's true power curve, wake losses, inverter
clipping, terrain effects, soiling, and the NWP model's local biases.

> **Critical rule: train on forecast weather, not measured weather.** The model
> must learn NWP's error characteristics end-to-end; training on met-mast actuals
> then predicting from NWP degrades live performance.

**Stage 2 — Distributional layer.** Three interchangeable routes:

1. **Quantile regression** (pinball loss, one model per quantile) — default
   starting point; native in modern GBM libraries.
2. **NWP ensembles** (e.g. ECMWF 51-member): push each member through Stage 1 →
   51 scenarios → empirical quantiles. Preserves temporal correlation across
   hours (valuable for multi-hour bid coupling).
3. **Analog / error-dressing**: dress the point forecast with the historical error
   distribution from similar conditions.

### 2.3 Calibration — the core quality bar

A forecast is calibrated if actuals fall below the stated P10 ~10% of the time,
etc. (verify with reliability diagrams). Miscalibration translates one-for-one
into settlement losses, because the entire bid strategy is "commit quantile q."

- **Vendor recalibration layer**: even with purchased forecasts, recalibrate
  in-house via quantile mapping against our own SCADA archive. Small project,
  high ROI. Archive every vendor forecast ever received.
- **Curtailment contamination**: SCADA records *produced*, not *producible*.
  Reconstruct available power (turbine-anemometry / unconstrained inverter
  capability signals); flag curtailed intervals; never calibrate on contaminated
  ground truth.
- **Availability separation**: outage-driven shortfalls are not weather error;
  scale ground truth to available capacity or feed availability as a feature.
  Note the asymmetry — unplanned outages only subtract (skews the lower tail).

### 2.4 Build vs. buy

- **Buy** (vendors: Solargis, Meteomatics, UL, DNV, many others; ~$1–3k/site/month)
  when starting; always keep the in-house recalibration + evaluation layer. Score
  vendors quarterly with pinball loss on our SCADA; running two vendors in
  parallel and blending often beats either.
- **Build** at portfolio scale: pipeline cost amortizes; enables custom targets
  (available power under our curtailment logic), guaranteed point-in-time
  archive, direct optimizer integration.
- **Business-model advantage (cross-client learning)**: with many plants' SCADA
  under management, weather-to-power models transfer across sites, NWP bias is
  calibrated regionally, and fleet-wide forecast-error days (the spread driver)
  become observable. Contracts must permit pooled/anonymized model training from
  day one.

### 2.5 Accuracy expectations (day-ahead horizon)

- Solar point forecasts: ~5–10% of capacity RMSE (climate-dependent).
- Wind: ~8–15% of capacity (cube-law amplification of speed errors).
- But judge the **distribution**: pinball loss across quantiles + tail-focused
  reliability, benchmarked vs. climatology-dressed persistence.

---

## 3. Model C — DA–RT Spread Model (per node)

### 3.1 Purpose

The money model. Produces per-hour, per-node **conditional quantiles of
S = DA − RT**, which select the production quantile to commit (newsvendor), with
a covariance correction for the client's own production error.

### 3.2 Statistical character of the target

- **Near-zero unconditional mean by construction**: virtuals arbitrage away any
  persistent gap; what remains is a small conditional risk premium (DA tends to
  run rich into expected scarcity) plus transient inefficiencies. Low
  signal-to-noise; expect modest R². Value lives in conditioning and tails.
- **Violently asymmetric tails**: DA is a smoothed expectation; RT is the spiky
  realization (scarcity adders → $850+ prints; renewable surplus → negative RT).
  Fat left tail (RT spike above DA), moderate right tail (RT crash). Gaussian
  assumptions are disqualifying.
- **Weak day-over-day autocorrelation, strong conditional structure** (hour,
  season, tightness, weather uncertainty).

### 3.3 Decomposition (model the two components separately)

```
S_node = S_system (energy/hub component) + S_congestion (ΔDA−RT congestion at node)
```

- **S_system**: total supply-demand — load forecast error, fleet renewable
  forecast error, post-DA forced outages, reserve scarcity. Built once, shared
  across all clients.
- **S_congestion**: constraints binding in RT but not priced DA (or vice versa) —
  transmission forced outages, unexpected flows. Often dominant for renewable
  pockets and the *more predictable* component (RT congestion persistence during
  outages). Per-node work; **our defensible IP layer**.

Directly computable from Data Miner's LMP component breakdown for both markets.

### 3.4 Architecture: regime classifier × conditional distribution × covariance

**Stage A — Spike/regime classifiers (discrete events that dominate P&L):**
- P(RT spike above DA): tightness, reserve margin, extreme-temperature forecasts
  *and their ensemble uncertainty*, high fleet renewable forecast (underdelivery
  risk), recent forced outages, day type.
- P(RT crash / negative RT): fleet renewable forecast, low load, inflexible
  baseload share, local constraint state. **Shares infrastructure with Model A's
  negative-price classifier.**

**Stage B — Conditional spread quantiles**, given regime probabilities: quantile
GBM on continuous features, or analog draws from regime-matched historical spread
distributions.

**Stage C — Production–spread covariance correction (business-specific edge):**
fleet-overproduction days crash RT exactly when our clients overproduce — the
production error and S are negatively... [correlated such that overscheduling is
punished]. Implement by conditioning the spread model on the client's own
production forecast error, or estimate within-regime correlation and adjust the
newsvendor quantile analytically. Generic price shops skip this; we must not.

### 3.5 Feature set, ranked by expected alpha

1. **Tightness** (load forecast − available capacity), nonlinear above ~93%.
2. **Renewable forecast level and *revision velocity*** across recent NWP cycles
   (late revisions ⇒ DA cleared on stale info ⇒ spread opportunity).
3. **Weather forecast uncertainty** (ensemble spread) — RT volatility fuel.
4. **Trailing RT-vs-DA congestion at the node** + constraint binding frequency,
   cross-referenced with planned transmission outages.
5. **Aggregate cleared virtual volumes** (published with lag) — arbitrage
   efficiency regime.
6. **Calendar interactions** (hour × season) + structural-break flags (rule
   changes).

### 3.6 Evaluation protocol

- **Pinball loss vs. the brutal baseline S ≡ 0** ("market is efficient"). Beating
  it consistently out-of-sample under point-in-time discipline is hard; a huge
  backtest win ⇒ hunt for leakage first.
- **Economic backtest**: full newsvendor loop vs. naive P50 bidding; target
  metric is $/MWh uplift (good implementations: ~$0.5–2/MWh).
- **Tail calibration specifically** (P5/P95), and **regime-sliced** evaluation —
  average-fine models are often terrible in the ~30 days/year that drive annual
  P&L.

### 3.7 Portfolio risk (existential, not statistical)

Clients are collectively long renewables ⇒ model errors are **correlated across
the book**. A wrong regime call on a fleet-overproduction day is wrong for every
wind client simultaneously. First-class risk metric from day one: aggregate MWh
leaning long DA across clients, stress-tested against spike scenarios.

---

## 4. Cross-cutting: offer floors, tax credits, and the price legs

The quantity bid (from Models B + C) pairs with **price legs** set by each asset's
true marginal cost, which is determined by its tax-credit election:

| Asset type | Marginal cost | Rational offer floor | Runs at negative prices? |
|---|---|---|---|
| PTC (in 10-yr window) | ≈ −(PTC × tax gross-up) | ~−$25 to −$35/MWh | Yes, to the floor |
| ITC | ≈ $0 | ~$0/MWh | No |
| Post-PTC-window (yr 11+) | ≈ $0 | ~$0/MWh | No |

- PTC (≈$27.50–30/MWh, inflation-adjusted, wage/apprenticeship-compliant, paid on
  generation for 10 years) shifts the floor negative; ITC (30%+ of capex, paid on
  investment) does not affect marginal cost.
- IRA made credits tech-neutral (45Y/48E) from 2025 — new high-CF solar
  increasingly elects PTC ⇒ solar fleet also develops negative floors (structural
  shift). OBBBA (July 2025) accelerated wind/solar phase-out (placed-in-service
  generally by end-2027, with begin-construction safe harbor) — existing plants
  keep locked-in credits. **Verify current guidance per project.**
- **Interaction with Model A's tail job**: expected production for bidding is
  *dispatch-probability-weighted*: E[dispatched output] ≈ physical forecast ×
  P(LMP ≥ floor), computed per hour, separately for DA and RT (the
  DA-negative/RT-positive and DA-positive/RT-negative cases settle very
  differently). Naive use of the physical forecast overcommits most on exactly
  the high-output days when curtailment risk peaks.
- Deviation settlement nuance: underdelivery against a DA position during
  negative RT prices can be *profitable* (sell DA positive, buy back negative) —
  which is precisely why deviation charges and must-offer rules exist. Any
  autobidder logic near this line needs compliance review (Market Monitor
  attention risk). Verify current PJM settlement/deviation rules; they change.

### 4.1 BORD-style deviation charges (first-class optimizer input)

"BORD" = PJM's **Balancing Operating Reserve Deviation** charges — a cost-
*allocation* mechanism that functions as a de facto penalty on deviating from
the day-ahead position.

- **Mechanism.** PJM incurs uplift (make-whole payments to units committed or
  dispatched whose LMP revenues don't cover offered costs). The *balancing*
  portion — costs arising after DA close, i.e., from RT diverging from the DA
  plan — is allocated substantially by cost causation to **deviations**:
  generators off their DA schedule (either direction), load off its bid,
  virtuals (which never deliver by construction). Each deviation-MWh attracts
  a $/MWh charge rate that varies daily with uplift incurred and total
  deviation-MWh.
- **Effect on the bid.** Adds a second term to the deviation leg of the
  newsvendor objective: every MWh of |Q_rt − Q_da| picks up the charge. It is
  a friction/transaction cost on leaning away from expected production — the
  optimal quantile shifts back toward P50 as the expected charge rate rises.
  The optimizer therefore needs the deviation charge rate (or a forecast of
  it) alongside the spread forecast.
- **Loophole patch.** Deviation charges (plus Market Monitor scrutiny) are
  what make systematically engineering profitable underdelivery at negative RT
  prices costly — see the nuance bullet above.
- **Caveat.** PJM's uplift allocation has been litigated and reformed
  repeatedly (cost buckets, deviation definitions, intermittent-resource
  exemptions, netting). The settlement module must pull current parameters
  from the live tariff, never from memory. "BORD-style" is shorthand for the
  category: **ex-post, cost-causation-based charges on DA-vs-RT deviations.**
- **Contrast (for the China work).** China's 两个细则 achieves deviation
  discipline through *administrative performance assessment* — penalties
  against regulated forecast-accuracy/schedule-compliance thresholds, largely
  independent of actual system balancing cost. PJM prices the externality
  (optimize deviations against a forecastable stochastic *price*); China
  grades the homework (engineer forecasts against a fixed *rulebook*). This is
  also why China's forecasting market became a compliance-procurement market
  with no PJM equivalent.

---

## 5. Data and operational infrastructure (shared foundations)

### 5.1 Point-in-time discipline (non-negotiable, applies to all three models)

Every feature must reflect the information set available **before the 10:30 AM ET
DA close** (in practice, snapshot at ~10:00):

- Weather: the NWP cycle actually available then (e.g. 00Z run), not later cycles,
  never realized weather.
- PJM feeds: as-of snapshots (feeds get revised).
- Vendor forecasts: archive as-received (vendors overwrite).
- Plant availability: as known at bid time.

Any post-close information in training data ⇒ backtests flatter, live
underperforms "mysteriously." The **as-of archive is a compounding business
asset** that cannot be reconstructed retroactively — competitors can't buy it.

### 5.2 PJM access and licensing (business-model obligations)

- Data Miner data is **internal-use only; redistribution of data or derivatives
  requires a PJM redistribution license** (Associate Membership at minimum).
  Whether our forecasts count as "derived data" must be resolved with PJM +
  counsel **before product launch**.
- Rate limits: 6 connections/min (non-member) vs. 600 (member) — membership is
  operationally necessary at portfolio scale anyway.
- Bid submission (Model 2 "software agent" variant / Model 3): Markets Gateway
  API (separate, authenticated), credentialing, account-security rules for
  third-party agents — confirm current requirements with PJM member services.
- Commercial data (gas indices, weather vendors): internal-use vs. redistribution
  tiers priced very differently; contract accordingly.

### 5.3 Settlement feedback loop

Clients' actual PJM settlement statements are ground truth for validating Model C
and the economic backtest. Contract for access to settlement data (validation +
model training rights) in every client agreement.

---

## 6. Build sequence (recommended)

| Phase | Deliverable | Models touched |
|---|---|---|
| 1 | **Empirical spread atlas**: per node-cluster/hour/season spread distributions, negative-RT and spike frequencies. Zero ML; sellable as analytics; seeds Stage-B analog library. | C |
| 2 | **Point-in-time archive pipeline** (PJM feeds, NOAA/ECMWF, self-archived from day one). | A, B, C |
| 3 | **System regime classifiers** (spike/crash) on free PJM + NOAA data; negative-price classifier (shared A/C). | A, C |
| 4 | **Nodal congestion layer**: trailing RT-vs-DA congestion features + transmission-outage cross-reference for actual client nodes. | C |
| 5 | **Production pipeline**: vendor-based with in-house recalibration; migrate to in-house Stage-1/Stage-2 at portfolio scale. | B |
| 6 | **Joint production–spread covariance correction** per client (needs their SCADA). | B + C |
| 7 | **Newsvendor bid optimizer** + economic backtesting harness vs. P50-naive baseline. | All |
| 8 | Portfolio-correlation risk dashboard (aggregate DA lean, spike stress tests). | C / risk |
| 9 | Second-order upgrades: distributional deep learning, joint scenario generation, structural stack refinement. | All |

---

## 7. Open questions for next discussion

1. Newsvendor math in full: closed-form optimal quantile with the covariance
   correction and deviation-charge asymmetry — worth deriving and unit-testing.
2. Target client nodes/regions: western wind (ComEd/AEP) vs. Mid-Atlantic solar
   have very different congestion stories; prioritizes the Phase-4 constraint
   library.
3. Vendor bake-off design for production forecasts (which two vendors, scoring
   protocol, blend rule).
4. Hybrid/storage clients: brings the price-level *shape* forecast back as
   first-order (arbitrage is a level-shape problem) — roadmap trigger point.
5. Intraday/rebidding scope: PJM rebid windows and RT strategy are out of scope
   for v1 but affect architecture (temporal correlation needs from Model B).
6. Compliance review checklist for autobidder logic near deviation-settlement
   edges; agent-framework clarification with PJM member services.
7. Must-offer / capacity-resource obligations per client asset — shifts optimal
   quantiles and constrains offer flexibility; needs current-rules verification.

---

## Appendix: Glossary

| Term | Definition |
|---|---|
| LMP | Locational Marginal Price — nodal energy price, decomposed into energy + congestion + loss components |
| DA / RT | Day-ahead market (cleared ~10:30 AM ET for next day, hourly) / real-time market (5-minute dispatch and settlement) |
| DA–RT spread (S) | S = DA − RT price; the object of Model C; near-zero mean by arbitrage, fat asymmetric tails |
| Newsvendor bid | Optimal DA quantity = a quantile of the production distribution, where the quantile is set by the expected spread and settlement asymmetry |
| Quantile / pinball loss | Value below which the outcome falls X% of the time / the loss function that trains and scores quantile forecasts |
| Calibration | Property that stated probabilities are honest (P10 exceeded ~90% of the time); checked with reliability diagrams |
| SCUC | Security-Constrained Unit Commitment — the mixed-integer optimization the ISO runs to clear the DA market |
| Virtuals (INC/DEC/UTC) | Purely financial DA positions that never deliver physically; arbitrage DA toward expected RT |
| Uplift | Make-whole payments to units whose market revenues don't cover offered costs (startup, no-load, min-run) |
| BORD | Balancing Operating Reserve Deviation charges — PJM's allocation of real-time balancing uplift to DA-vs-RT deviations (§4.1) |
| ORDC / scarcity adders | Operating Reserve Demand Curve — administrative price adders when reserves run short; drives RT spikes |
| FTR | Financial Transmission Right — hedge/speculative instrument on DA congestion between two nodes |
| PTC / ITC | Production Tax Credit ($/MWh generated, 10 years; sets negative offer floors) / Investment Tax Credit (% of capex; floor ≈ $0) — see §4 |
| Offer floor | The price leg below which the plant prefers curtailment: ≈ −(PTC × tax gross-up) for PTC assets, ≈ $0 for ITC assets |
| Must-offer / RPM | Obligation of capacity resources to offer into the DA market / PJM's capacity market (Reliability Pricing Model) |
| BTM | Behind-the-meter (e.g., rooftop solar netted out of observed load) |
| NWP | Numerical Weather Prediction (GFS, HRRR, ECMWF); ensembles = many perturbed runs, source of forecast-uncertainty features |
| SCADA | Plant supervisory control and data acquisition — ground-truth production, availability, curtailment records |
| Point-in-time discipline | Every training/backtest feature must reflect only information available before the 10:30 AM DA close (§5.1) |
| Data Miner 2 / Markets Gateway | PJM's public data API / PJM's authenticated bid-submission API |
| QSE / scheduling agent | Entity with market-participant infrastructure that submits offers and handles settlement on an asset's behalf |