- pjm/design_models (EN/ZH): new §4.2 — three-layer statement (separable in accounting/first-order decisions, correlated in risk, coupled in risk-adjusted decisions), with contract-form and settlement-reference caveats; hedge book enters the optimizer through the risk term - china/product_design (EN/ZH): new §2.7 — the argument transfers to MLT CfDs, with four deltas: same-settlement-price correlation, bidirectional living-book coupling, mandate-band corner solutions, and portfolio-scale price-impact as both modeling correction and compliance red line; 两个细则 as the third P&L stream coupling to the declaration
520 lines
29 KiB
Markdown
520 lines
29 KiB
Markdown
# Forecasting Models for Renewable Day-Ahead Bidding in PJM
|
||
|
||
**Working design document — v0.1 (July 2026)**
|
||
|
||
Context: We are building a bid-optimization business (Model 2: SaaS, with a likely
|
||
migration path toward Model 3: asset management / market participant services) that
|
||
helps renewable generators submit day-ahead (DA) offers into PJM. This document
|
||
captures the background and technical design of the three core forecasting models
|
||
that feed the bid optimizer, for further discussion and iteration.
|
||
|
||
---
|
||
|
||
## 0. How the three models fit together
|
||
|
||
The product's core loop is a **newsvendor-style quantity decision**: the optimal DA
|
||
commitment for a renewable plant is a quantile of its production forecast
|
||
distribution, where the quantile is chosen based on the expected DA–RT spread and
|
||
the asymmetry of settlement outcomes, corrected for the covariance between the
|
||
plant's production error and the spread.
|
||
|
||
```
|
||
┌─────────────────────┐
|
||
│ Model B: Production │──── production quantiles (per plant, per hour)
|
||
└─────────────────────┘ │
|
||
▼
|
||
┌─────────────────────┐ regime features ┌──────────────────┐
|
||
│ Model A: DA Price │──────────────────────────▶│ Bid Optimizer │──▶ DA offer curve
|
||
│ (level, coarse) │ │ (newsvendor + │ (quantity + price legs)
|
||
└─────────────────────┘ │ covariance) │
|
||
┌─────────────────────┐ └──────────────────┘
|
||
│ Model C: DA–RT │──── spread quantiles (per node, per hour)
|
||
│ Spread │
|
||
└─────────────────────┘
|
||
```
|
||
|
||
Division of precision requirements:
|
||
|
||
| Model | Precision needed | Role in the bid |
|
||
|---|---|---|
|
||
| A. DA price level | **Low** (regime/coarse) | Regime conditioning for Model C; negative-price / offer-floor tail probabilities; revenue expectations |
|
||
| B. Production | **High** (calibrated quantiles) | The quantity axis of the bid |
|
||
| C. DA–RT spread | **High** (conditional distribution, esp. tails) | Selects the production quantile to commit |
|
||
|
||
Key insight driving this architecture: for a quantity-bidding renewable, the DA–RT
|
||
spread and own-production distribution jointly determine the optimal bid; the
|
||
absolute price level matters only (a) as a conditioning regime for the spread,
|
||
(b) in the tail near the plant's offer floor (negative prices / economic
|
||
curtailment), and (c) for reporting/hedging.
|
||
|
||
---
|
||
|
||
## 1. Model A — DA Price Level (coarse / regime model)
|
||
|
||
### 1.1 Purpose
|
||
|
||
Not a desk-grade price forecast. Three narrowly-scoped jobs:
|
||
|
||
1. **Regime classification** of tomorrow: e.g. {mild / normal / tight / scarce} and
|
||
{gas-marginal / coal-marginal / renewable-surplus}, used as conditioning
|
||
features by Model C.
|
||
2. **Offer-floor tail probabilities**: P(nodal LMP < floor) per hour, for both DA
|
||
and RT — the negative-price classifier. Determines the dispatch-probability
|
||
weighting of production (truncation of Model B's distribution) and the price
|
||
legs of the offer curve.
|
||
3. **Expected price levels** for client revenue reporting and hedge support.
|
||
|
||
### 1.2 Why we do NOT replicate SCUC
|
||
|
||
The DA price is the output of PJM's security-constrained unit commitment /
|
||
economic dispatch, so full structural replication (Dayzer / PROMOD-class) is
|
||
conceptually "correct" but practically dominated for next-day horizons:
|
||
|
||
- **Offers are confidential** at bid time (published ~4 months later, masked).
|
||
Marginal-unit cost reconstruction errors land exactly where prices are set.
|
||
- **DA is a financial equilibrium, not a pure physical SCUC**: virtual
|
||
transactions (INCs/DECs/UTCs) push DA toward the market's RT expectation. A
|
||
physical replica misses this layer entirely.
|
||
- **Network model is CEII-restricted**; monitored constraints and operating limits
|
||
vary daily with operator judgment. Congestion — the nodal differentiator — is
|
||
the hardest component to replicate.
|
||
- **SCUC is combinatorial**: small input errors flip discrete commitment
|
||
decisions; even PJM's own solution is tolerance-gapped, not unique.
|
||
- **Out-of-market operator actions** are unobservable in advance.
|
||
|
||
Empirical consensus: statistical/ML models beat structural replicas for
|
||
tomorrow's prices; structural models win for counterfactuals, congestion anatomy,
|
||
long horizons, and new regimes.
|
||
|
||
### 1.3 Our approach: "mimic SCUC's logic, not its solution" (hybrid-lite)
|
||
|
||
Build a **reduced supply-stack view** to generate features, then let a statistical
|
||
model learn the residual mapping:
|
||
|
||
- **Stack tightness**: PJM load forecast minus available thermal capacity
|
||
(outage-feed adjusted) minus fleet renewable forecast. Strongly nonlinear —
|
||
matters above ~93% utilization.
|
||
- **Implied marginal fuel / regime flag** from the reconstructed stack and gas
|
||
signals.
|
||
- **Constraint watch-list** per client region: reconstructed from PJM's historical
|
||
binding-constraints feed + planned transmission outages (OASIS), not from a
|
||
power-flow model.
|
||
|
||
Statistical layer: gradient-boosted models for expected level; a dedicated binary
|
||
classifier for negative-price hours (features: low load forecast, high fleet
|
||
solar/wind forecast, weekend/holiday, spring/fall, local congestion conditions).
|
||
|
||
### 1.4 Data
|
||
|
||
| Data | Source | Cost |
|
||
|---|---|---|
|
||
| DA/RT LMPs (energy/congestion/loss components) | PJM Data Miner (`da_hrl_lmps`, `rt_hrl_lmps`, `rt_fivemin_hrl_lmps`) | Free |
|
||
| Load forecast, metered load | PJM Data Miner | Free |
|
||
| Fleet wind/solar forecasts + actuals | PJM Data Miner | Free |
|
||
| Generation outages (aggregate MW) | PJM Data Miner | Free |
|
||
| Binding constraints | PJM Data Miner | Free |
|
||
| Planned transmission outages | PJM OASIS | Free |
|
||
| Nuclear unit status | NRC daily power reactor status | Free |
|
||
| Gas: Henry Hub spot, NYMEX settlements | EIA, CME | Free-ish |
|
||
| Gas: eastern basis (TETCO M3, Transco Z6, Dominion S, TCO) | Platts / NGI / ICE | Paid — highest-value paid dataset |
|
||
| Weather forecasts (temp, wind, irradiance) + ensemble spread | NOAA (GFS/HRRR/NDFD), ECMWF open data | Free (self-archive!) |
|
||
| Neighboring ISO prices/loads (MISO, NYISO) | Their public portals | Free |
|
||
|
||
### 1.5 Known structural breaks to encode
|
||
|
||
- Fleet turnover: retirements, new entry (a 2019-trained model misprices 2026).
|
||
- Reserve/ORDC market rule changes (flag dates; distrust pre-change history).
|
||
- PTC/ITC fleet composition drift (see §4): as neighboring PTC plants age out of
|
||
their 10-year windows, historical negative-price frequency becomes a biased
|
||
predictor.
|
||
|
||
---
|
||
|
||
## 2. Model B — Probabilistic Production Forecast (per plant)
|
||
|
||
### 2.1 Purpose and target
|
||
|
||
For every hour of tomorrow, by ~10:00 AM ET (ahead of the 10:30 DA close): a
|
||
**calibrated quantile set** of the plant's *available* production, e.g. P10 / P25 /
|
||
P50 / P75 / P90. The bid optimizer commits a quantile of this distribution — a
|
||
point forecast is structurally insufficient.
|
||
|
||
Horizon: ~14–38 hours at bid time ⇒ NWP (numerical weather prediction) carries
|
||
essentially all the signal; persistence is worthless at this range.
|
||
|
||
### 2.2 Architecture: two-stage pipeline
|
||
|
||
**Stage 1 — Weather-to-power model (per site).**
|
||
Learn the mapping NWP forecast → SCADA output with gradient-boosted trees (or
|
||
similar). This absorbs the plant's true power curve, wake losses, inverter
|
||
clipping, terrain effects, soiling, and the NWP model's local biases.
|
||
|
||
> **Critical rule: train on forecast weather, not measured weather.** The model
|
||
> must learn NWP's error characteristics end-to-end; training on met-mast actuals
|
||
> then predicting from NWP degrades live performance.
|
||
|
||
**Stage 2 — Distributional layer.** Three interchangeable routes:
|
||
|
||
1. **Quantile regression** (pinball loss, one model per quantile) — default
|
||
starting point; native in modern GBM libraries.
|
||
2. **NWP ensembles** (e.g. ECMWF 51-member): push each member through Stage 1 →
|
||
51 scenarios → empirical quantiles. Preserves temporal correlation across
|
||
hours (valuable for multi-hour bid coupling).
|
||
3. **Analog / error-dressing**: dress the point forecast with the historical error
|
||
distribution from similar conditions.
|
||
|
||
### 2.3 Calibration — the core quality bar
|
||
|
||
A forecast is calibrated if actuals fall below the stated P10 ~10% of the time,
|
||
etc. (verify with reliability diagrams). Miscalibration translates one-for-one
|
||
into settlement losses, because the entire bid strategy is "commit quantile q."
|
||
|
||
- **Vendor recalibration layer**: even with purchased forecasts, recalibrate
|
||
in-house via quantile mapping against our own SCADA archive. Small project,
|
||
high ROI. Archive every vendor forecast ever received.
|
||
- **Curtailment contamination**: SCADA records *produced*, not *producible*.
|
||
Reconstruct available power (turbine-anemometry / unconstrained inverter
|
||
capability signals); flag curtailed intervals; never calibrate on contaminated
|
||
ground truth.
|
||
- **Availability separation**: outage-driven shortfalls are not weather error;
|
||
scale ground truth to available capacity or feed availability as a feature.
|
||
Note the asymmetry — unplanned outages only subtract (skews the lower tail).
|
||
|
||
### 2.4 Build vs. buy
|
||
|
||
- **Buy** (vendors: Solargis, Meteomatics, UL, DNV, many others; ~$1–3k/site/month)
|
||
when starting; always keep the in-house recalibration + evaluation layer. Score
|
||
vendors quarterly with pinball loss on our SCADA; running two vendors in
|
||
parallel and blending often beats either.
|
||
- **Build** at portfolio scale: pipeline cost amortizes; enables custom targets
|
||
(available power under our curtailment logic), guaranteed point-in-time
|
||
archive, direct optimizer integration.
|
||
- **Business-model advantage (cross-client learning)**: with many plants' SCADA
|
||
under management, weather-to-power models transfer across sites, NWP bias is
|
||
calibrated regionally, and fleet-wide forecast-error days (the spread driver)
|
||
become observable. Contracts must permit pooled/anonymized model training from
|
||
day one.
|
||
|
||
### 2.5 Accuracy expectations (day-ahead horizon)
|
||
|
||
- Solar point forecasts: ~5–10% of capacity RMSE (climate-dependent).
|
||
- Wind: ~8–15% of capacity (cube-law amplification of speed errors).
|
||
- But judge the **distribution**: pinball loss across quantiles + tail-focused
|
||
reliability, benchmarked vs. climatology-dressed persistence.
|
||
|
||
---
|
||
|
||
## 3. Model C — DA–RT Spread Model (per node)
|
||
|
||
### 3.1 Purpose
|
||
|
||
The money model. Produces per-hour, per-node **conditional quantiles of
|
||
S = DA − RT**, which select the production quantile to commit (newsvendor), with
|
||
a covariance correction for the client's own production error.
|
||
|
||
### 3.2 Statistical character of the target
|
||
|
||
- **Near-zero unconditional mean by construction**: virtuals arbitrage away any
|
||
persistent gap; what remains is a small conditional risk premium (DA tends to
|
||
run rich into expected scarcity) plus transient inefficiencies. Low
|
||
signal-to-noise; expect modest R². Value lives in conditioning and tails.
|
||
- **Violently asymmetric tails**: DA is a smoothed expectation; RT is the spiky
|
||
realization (scarcity adders → $850+ prints; renewable surplus → negative RT).
|
||
Fat left tail (RT spike above DA), moderate right tail (RT crash). Gaussian
|
||
assumptions are disqualifying.
|
||
- **Weak day-over-day autocorrelation, strong conditional structure** (hour,
|
||
season, tightness, weather uncertainty).
|
||
|
||
### 3.3 Decomposition (model the two components separately)
|
||
|
||
```
|
||
S_node = S_system (energy/hub component) + S_congestion (ΔDA−RT congestion at node)
|
||
```
|
||
|
||
- **S_system**: total supply-demand — load forecast error, fleet renewable
|
||
forecast error, post-DA forced outages, reserve scarcity. Built once, shared
|
||
across all clients.
|
||
- **S_congestion**: constraints binding in RT but not priced DA (or vice versa) —
|
||
transmission forced outages, unexpected flows. Often dominant for renewable
|
||
pockets and the *more predictable* component (RT congestion persistence during
|
||
outages). Per-node work; **our defensible IP layer**.
|
||
|
||
Directly computable from Data Miner's LMP component breakdown for both markets.
|
||
|
||
### 3.4 Architecture: regime classifier × conditional distribution × covariance
|
||
|
||
**Stage A — Spike/regime classifiers (discrete events that dominate P&L):**
|
||
- P(RT spike above DA): tightness, reserve margin, extreme-temperature forecasts
|
||
*and their ensemble uncertainty*, high fleet renewable forecast (underdelivery
|
||
risk), recent forced outages, day type.
|
||
- P(RT crash / negative RT): fleet renewable forecast, low load, inflexible
|
||
baseload share, local constraint state. **Shares infrastructure with Model A's
|
||
negative-price classifier.**
|
||
|
||
**Stage B — Conditional spread quantiles**, given regime probabilities: quantile
|
||
GBM on continuous features, or analog draws from regime-matched historical spread
|
||
distributions.
|
||
|
||
**Stage C — Production–spread covariance correction (business-specific edge):**
|
||
fleet-overproduction days crash RT exactly when our clients overproduce — the
|
||
production error and S are negatively... [correlated such that overscheduling is
|
||
punished]. Implement by conditioning the spread model on the client's own
|
||
production forecast error, or estimate within-regime correlation and adjust the
|
||
newsvendor quantile analytically. Generic price shops skip this; we must not.
|
||
|
||
### 3.5 Feature set, ranked by expected alpha
|
||
|
||
1. **Tightness** (load forecast − available capacity), nonlinear above ~93%.
|
||
2. **Renewable forecast level and *revision velocity*** across recent NWP cycles
|
||
(late revisions ⇒ DA cleared on stale info ⇒ spread opportunity).
|
||
3. **Weather forecast uncertainty** (ensemble spread) — RT volatility fuel.
|
||
4. **Trailing RT-vs-DA congestion at the node** + constraint binding frequency,
|
||
cross-referenced with planned transmission outages.
|
||
5. **Aggregate cleared virtual volumes** (published with lag) — arbitrage
|
||
efficiency regime.
|
||
6. **Calendar interactions** (hour × season) + structural-break flags (rule
|
||
changes).
|
||
|
||
### 3.6 Evaluation protocol
|
||
|
||
- **Pinball loss vs. the brutal baseline S ≡ 0** ("market is efficient"). Beating
|
||
it consistently out-of-sample under point-in-time discipline is hard; a huge
|
||
backtest win ⇒ hunt for leakage first.
|
||
- **Economic backtest**: full newsvendor loop vs. naive P50 bidding; target
|
||
metric is $/MWh uplift (good implementations: ~$0.5–2/MWh).
|
||
- **Tail calibration specifically** (P5/P95), and **regime-sliced** evaluation —
|
||
average-fine models are often terrible in the ~30 days/year that drive annual
|
||
P&L.
|
||
|
||
### 3.7 Portfolio risk (existential, not statistical)
|
||
|
||
Clients are collectively long renewables ⇒ model errors are **correlated across
|
||
the book**. A wrong regime call on a fleet-overproduction day is wrong for every
|
||
wind client simultaneously. First-class risk metric from day one: aggregate MWh
|
||
leaning long DA across clients, stress-tested against spike scenarios.
|
||
|
||
---
|
||
|
||
## 4. Cross-cutting: offer floors, tax credits, and the price legs
|
||
|
||
The quantity bid (from Models B + C) pairs with **price legs** set by each asset's
|
||
true marginal cost, which is determined by its tax-credit election:
|
||
|
||
| Asset type | Marginal cost | Rational offer floor | Runs at negative prices? |
|
||
|---|---|---|---|
|
||
| PTC (in 10-yr window) | ≈ −(PTC × tax gross-up) | ~−$25 to −$35/MWh | Yes, to the floor |
|
||
| ITC | ≈ $0 | ~$0/MWh | No |
|
||
| Post-PTC-window (yr 11+) | ≈ $0 | ~$0/MWh | No |
|
||
|
||
- PTC (≈$27.50–30/MWh, inflation-adjusted, wage/apprenticeship-compliant, paid on
|
||
generation for 10 years) shifts the floor negative; ITC (30%+ of capex, paid on
|
||
investment) does not affect marginal cost.
|
||
- IRA made credits tech-neutral (45Y/48E) from 2025 — new high-CF solar
|
||
increasingly elects PTC ⇒ solar fleet also develops negative floors (structural
|
||
shift). OBBBA (July 2025) accelerated wind/solar phase-out (placed-in-service
|
||
generally by end-2027, with begin-construction safe harbor) — existing plants
|
||
keep locked-in credits. **Verify current guidance per project.**
|
||
- **Interaction with Model A's tail job**: expected production for bidding is
|
||
*dispatch-probability-weighted*: E[dispatched output] ≈ physical forecast ×
|
||
P(LMP ≥ floor), computed per hour, separately for DA and RT (the
|
||
DA-negative/RT-positive and DA-positive/RT-negative cases settle very
|
||
differently). Naive use of the physical forecast overcommits most on exactly
|
||
the high-output days when curtailment risk peaks.
|
||
- Deviation settlement nuance: underdelivery against a DA position during
|
||
negative RT prices can be *profitable* (sell DA positive, buy back negative) —
|
||
which is precisely why deviation charges and must-offer rules exist. Any
|
||
autobidder logic near this line needs compliance review (Market Monitor
|
||
attention risk). Verify current PJM settlement/deviation rules; they change.
|
||
|
||
### 4.1 BORD-style deviation charges (first-class optimizer input)
|
||
|
||
"BORD" = PJM's **Balancing Operating Reserve Deviation** charges — a cost-
|
||
*allocation* mechanism that functions as a de facto penalty on deviating from
|
||
the day-ahead position.
|
||
|
||
- **Mechanism.** PJM incurs uplift (make-whole payments to units committed or
|
||
dispatched whose LMP revenues don't cover offered costs). The *balancing*
|
||
portion — costs arising after DA close, i.e., from RT diverging from the DA
|
||
plan — is allocated substantially by cost causation to **deviations**:
|
||
generators off their DA schedule (either direction), load off its bid,
|
||
virtuals (which never deliver by construction). Each deviation-MWh attracts
|
||
a $/MWh charge rate that varies daily with uplift incurred and total
|
||
deviation-MWh.
|
||
- **Effect on the bid.** Adds a second term to the deviation leg of the
|
||
newsvendor objective: every MWh of |Q_rt − Q_da| picks up the charge. It is
|
||
a friction/transaction cost on leaning away from expected production — the
|
||
optimal quantile shifts back toward P50 as the expected charge rate rises.
|
||
The optimizer therefore needs the deviation charge rate (or a forecast of
|
||
it) alongside the spread forecast.
|
||
- **Loophole patch.** Deviation charges (plus Market Monitor scrutiny) are
|
||
what make systematically engineering profitable underdelivery at negative RT
|
||
prices costly — see the nuance bullet above.
|
||
- **Caveat.** PJM's uplift allocation has been litigated and reformed
|
||
repeatedly (cost buckets, deviation definitions, intermittent-resource
|
||
exemptions, netting). The settlement module must pull current parameters
|
||
from the live tariff, never from memory. "BORD-style" is shorthand for the
|
||
category: **ex-post, cost-causation-based charges on DA-vs-RT deviations.**
|
||
- **Contrast (for the China work).** China's 两个细则 achieves deviation
|
||
discipline through *administrative performance assessment* — penalties
|
||
against regulated forecast-accuracy/schedule-compliance thresholds, largely
|
||
independent of actual system balancing cost. PJM prices the externality
|
||
(optimize deviations against a forecastable stochastic *price*); China
|
||
grades the homework (engineer forecasts against a fixed *rulebook*). This is
|
||
also why China's forecasting market became a compliance-procurement market
|
||
with no PJM equivalent.
|
||
|
||
### 4.2 Hedge book vs. spread book: separability and its limits
|
||
|
||
Most client plants carry a hedge (VPPA, futures strip, bank hedge). The bid
|
||
optimizer's relationship to that hedge rests on a three-layer statement:
|
||
**separable in accounting and first-order decisions, correlated in risk,
|
||
coupled in second-order (risk-adjusted) decisions.**
|
||
|
||
**Layer 1 — additive separability (the "yes").** Hourly revenue:
|
||
|
||
```
|
||
R = Q_da·P_da + (Q_rt − Q_da)·P_rt (market legs — bid optimizer's domain)
|
||
+ (P_c − P_ref)·Q_c (hedge leg — fixed CfD)
|
||
```
|
||
|
||
The hedge leg contains no daily decision variable: Q_c, P_c were set at
|
||
signing; P_ref is a hub index the plant's bid does not move. Q_da appears
|
||
only in the market legs. Spread alpha therefore adds on top of hedge P&L,
|
||
and each book can be evaluated independently without double-counting. This
|
||
is the license for treating the hedge as a static given throughout §1–3.
|
||
|
||
**Layer 2 — correlated outcomes (the first "no").** Both legs are driven by
|
||
the same prices: rearranged, the market legs contain Q_da·(P_da − P_rt) and
|
||
the hedge leg contains −P_da (for DA-settled references). On a scarcity day,
|
||
an underscheduled spread position and a short-at-P_c hedge lose *together*.
|
||
Independent decisions, correlated P&L — portfolio risk reporting must treat
|
||
them jointly.
|
||
|
||
**Layer 3 — risk-adjusted coupling (the second "no").** Separability of
|
||
decisions holds only under expected value. Under mean–variance/CVaR, the
|
||
optimal DA lean depends on the plant's *residual* price exposure after the
|
||
hedge: heavily hedged plants can lean harder on the spread; merchant plants
|
||
should shade conservative because the spread bet stacks on large open price
|
||
risk. **The hedge book enters the bid optimizer through the risk term, not
|
||
the P&L identity** — a required client-onboarding input (hedge volume,
|
||
tenor, settlement point and index).
|
||
|
||
**Caveats that can break even Layer 1:**
|
||
- *Contract form.* Fixed-volume financial hedges separate cleanly.
|
||
Unit-contingent / as-generated PPAs set Q_c = Q_rt (hedge leg co-varies
|
||
with production, though still not with Q_da); physical PPAs with delivery
|
||
obligations can constrain the bidding problem directly; proxy revenue
|
||
swaps absorb shape/volume risk and change what remains for the spread
|
||
book to manage.
|
||
- *Settlement reference.* DA-settled vs RT-settled hedges leave different
|
||
residual exposures: an RT-settled hedge plus DA scheduling exposes the
|
||
*hedged* volume to the DA–RT spread too — equivalent to an embedded
|
||
virtual position, sometimes intended, sometimes an accident. The optimizer
|
||
must know which it is.
|
||
|
||
(Chinese-market transfer of this argument, with four deltas: see the China
|
||
product design doc §2.7.)
|
||
|
||
---
|
||
|
||
## 5. Data and operational infrastructure (shared foundations)
|
||
|
||
### 5.1 Point-in-time discipline (non-negotiable, applies to all three models)
|
||
|
||
Every feature must reflect the information set available **before the 10:30 AM ET
|
||
DA close** (in practice, snapshot at ~10:00):
|
||
|
||
- Weather: the NWP cycle actually available then (e.g. 00Z run), not later cycles,
|
||
never realized weather.
|
||
- PJM feeds: as-of snapshots (feeds get revised).
|
||
- Vendor forecasts: archive as-received (vendors overwrite).
|
||
- Plant availability: as known at bid time.
|
||
|
||
Any post-close information in training data ⇒ backtests flatter, live
|
||
underperforms "mysteriously." The **as-of archive is a compounding business
|
||
asset** that cannot be reconstructed retroactively — competitors can't buy it.
|
||
|
||
### 5.2 PJM access and licensing (business-model obligations)
|
||
|
||
- Data Miner data is **internal-use only; redistribution of data or derivatives
|
||
requires a PJM redistribution license** (Associate Membership at minimum).
|
||
Whether our forecasts count as "derived data" must be resolved with PJM +
|
||
counsel **before product launch**.
|
||
- Rate limits: 6 connections/min (non-member) vs. 600 (member) — membership is
|
||
operationally necessary at portfolio scale anyway.
|
||
- Bid submission (Model 2 "software agent" variant / Model 3): Markets Gateway
|
||
API (separate, authenticated), credentialing, account-security rules for
|
||
third-party agents — confirm current requirements with PJM member services.
|
||
- Commercial data (gas indices, weather vendors): internal-use vs. redistribution
|
||
tiers priced very differently; contract accordingly.
|
||
|
||
### 5.3 Settlement feedback loop
|
||
|
||
Clients' actual PJM settlement statements are ground truth for validating Model C
|
||
and the economic backtest. Contract for access to settlement data (validation +
|
||
model training rights) in every client agreement.
|
||
|
||
---
|
||
|
||
## 6. Build sequence (recommended)
|
||
|
||
| Phase | Deliverable | Models touched |
|
||
|---|---|---|
|
||
| 1 | **Empirical spread atlas**: per node-cluster/hour/season spread distributions, negative-RT and spike frequencies. Zero ML; sellable as analytics; seeds Stage-B analog library. | C |
|
||
| 2 | **Point-in-time archive pipeline** (PJM feeds, NOAA/ECMWF, self-archived from day one). | A, B, C |
|
||
| 3 | **System regime classifiers** (spike/crash) on free PJM + NOAA data; negative-price classifier (shared A/C). | A, C |
|
||
| 4 | **Nodal congestion layer**: trailing RT-vs-DA congestion features + transmission-outage cross-reference for actual client nodes. | C |
|
||
| 5 | **Production pipeline**: vendor-based with in-house recalibration; migrate to in-house Stage-1/Stage-2 at portfolio scale. | B |
|
||
| 6 | **Joint production–spread covariance correction** per client (needs their SCADA). | B + C |
|
||
| 7 | **Newsvendor bid optimizer** + economic backtesting harness vs. P50-naive baseline. | All |
|
||
| 8 | Portfolio-correlation risk dashboard (aggregate DA lean, spike stress tests). | C / risk |
|
||
| 9 | Second-order upgrades: distributional deep learning, joint scenario generation, structural stack refinement. | All |
|
||
|
||
---
|
||
|
||
## 7. Open questions for next discussion
|
||
|
||
1. Newsvendor math in full: closed-form optimal quantile with the covariance
|
||
correction and deviation-charge asymmetry — worth deriving and unit-testing.
|
||
2. Target client nodes/regions: western wind (ComEd/AEP) vs. Mid-Atlantic solar
|
||
have very different congestion stories; prioritizes the Phase-4 constraint
|
||
library.
|
||
3. Vendor bake-off design for production forecasts (which two vendors, scoring
|
||
protocol, blend rule).
|
||
4. Hybrid/storage clients: brings the price-level *shape* forecast back as
|
||
first-order (arbitrage is a level-shape problem) — roadmap trigger point.
|
||
5. Intraday/rebidding scope: PJM rebid windows and RT strategy are out of scope
|
||
for v1 but affect architecture (temporal correlation needs from Model B).
|
||
6. Compliance review checklist for autobidder logic near deviation-settlement
|
||
edges; agent-framework clarification with PJM member services.
|
||
7. Must-offer / capacity-resource obligations per client asset — shifts optimal
|
||
quantiles and constrains offer flexibility; needs current-rules verification.
|
||
|
||
---
|
||
|
||
## Appendix: Glossary
|
||
|
||
| Term | Definition |
|
||
|---|---|
|
||
| LMP | Locational Marginal Price — nodal energy price, decomposed into energy + congestion + loss components |
|
||
| DA / RT | Day-ahead market (cleared ~10:30 AM ET for next day, hourly) / real-time market (5-minute dispatch and settlement) |
|
||
| DA–RT spread (S) | S = DA − RT price; the object of Model C; near-zero mean by arbitrage, fat asymmetric tails |
|
||
| Newsvendor bid | Optimal DA quantity = a quantile of the production distribution, where the quantile is set by the expected spread and settlement asymmetry |
|
||
| Quantile / pinball loss | Value below which the outcome falls X% of the time / the loss function that trains and scores quantile forecasts |
|
||
| Calibration | Property that stated probabilities are honest (P10 exceeded ~90% of the time); checked with reliability diagrams |
|
||
| SCUC | Security-Constrained Unit Commitment — the mixed-integer optimization the ISO runs to clear the DA market |
|
||
| Virtuals (INC/DEC/UTC) | Purely financial DA positions that never deliver physically; arbitrage DA toward expected RT |
|
||
| Uplift | Make-whole payments to units whose market revenues don't cover offered costs (startup, no-load, min-run) |
|
||
| BORD | Balancing Operating Reserve Deviation charges — PJM's allocation of real-time balancing uplift to DA-vs-RT deviations (§4.1) |
|
||
| ORDC / scarcity adders | Operating Reserve Demand Curve — administrative price adders when reserves run short; drives RT spikes |
|
||
| FTR | Financial Transmission Right — hedge/speculative instrument on DA congestion between two nodes |
|
||
| PTC / ITC | Production Tax Credit ($/MWh generated, 10 years; sets negative offer floors) / Investment Tax Credit (% of capex; floor ≈ $0) — see §4 |
|
||
| Offer floor | The price leg below which the plant prefers curtailment: ≈ −(PTC × tax gross-up) for PTC assets, ≈ $0 for ITC assets |
|
||
| Must-offer / RPM | Obligation of capacity resources to offer into the DA market / PJM's capacity market (Reliability Pricing Model) |
|
||
| BTM | Behind-the-meter (e.g., rooftop solar netted out of observed load) |
|
||
| NWP | Numerical Weather Prediction (GFS, HRRR, ECMWF); ensembles = many perturbed runs, source of forecast-uncertainty features |
|
||
| SCADA | Plant supervisory control and data acquisition — ground-truth production, availability, curtailment records |
|
||
| Point-in-time discipline | Every training/backtest feature must reflect only information available before the 10:30 AM DA close (§5.1) |
|
||
| Data Miner 2 / Markets Gateway | PJM's public data API / PJM's authenticated bid-submission API |
|
||
| QSE / scheduling agent | Entity with market-participant infrastructure that submits offers and handles settlement on an asset's behalf |
|