NYC transit operations research brief · retrospective model review
Recent line history helps rank disruption risk—but not reliably enough to alert riders.
A gradient-boosted model ranks line-hours by whether a new qualifying MTA service-alert disruption will begin in the next 60 minutes. The ranking signal is real, modest, and mostly a reflection of recent line history.
Exact prediction target
For one subway line at the start of an hour, did the MTA first publish a new, unplanned delay or suspension alert during that hour? This measures alert publication—not individual train lateness.
Key finding · structure versus recent conditions
The model mainly learned which lines have been experiencing disruptions lately.
Line identity contributes some separation, but recent line history contributes roughly twice as much.
These four scores are recomputed through the embedded 300-tree model—not duplicated as a second calculation.
The checked-in bundle returns 0.011 for the shuttle's real-history case at three decimals; the project notes record 0.012. The visualization reports the live bundle result to preserve model fidelity.
Explore thresholds
Every alerting rule trades false alarms for missed disruptions.
There is no objectively correct threshold. This explorer recomputes exact held-out counts from exported score histograms; choose based on the cost of each error.
The vertical rule marks your current threshold. 0.58 is the F1-optimal point chosen on validation and reported in the tables above — note that F1 is itself an arbitrary tiebreaker, weighting a missed disruption exactly as costly as a false alarm.
Try the model
Score an invented line-hour with the real fitted model.
The browser traverses all 300 exported trees in feature order. The result is a model score for ranking—not a probability or a claim about train lateness.
Why the numbers differ: class weighting makes the raw score useful for ranking but not calibrated. The observed frequency comes from the relevant held-out calibration bin and is the appropriate historical context.
Attribution is leave-one-group-out: each bar is how much the log-odds move when that group of inputs is reset to the reference hour while everything else stays as you set it. Groups interact, so the bars do not sum to the total — this is an honest sensitivity readout, not a SHAP decomposition.
Evidence and limitations
The full evaluation, with technical detail on demand.
Open each section for charts, accessible data tables, and the caveats that constrain responsible use.
Model comparison and precision–recall evidence
Model comparison
Evaluated once on the held-out test set. Each threshold was tuned separately on validation.
| Model | PR-AUC | ROC-AUC | Precision | Recall | Threshold |
|---|
Precision–recall curves
Every operating point; the dashed rule is the 14.4% test base rate.
Curve data table
| Recall | Baseline | Logistic | Gradient boosting |
|---|
Calibration context
Model score against observed frequency. The diagonal shows equality, not a claimed probability.
Scores are not calibrated probabilities
Hours scored near 0.76 saw disruption about 31% of the time. Ranking is useful; probability claims require a separate calibration step.
Feature importance and disruption patterns
Feature importance
Drop in average precision when a feature is shuffled on a 20,000-row test sample.
Disruption prevalence by line
Share of eligible line-hours followed by a qualifying alert. Route bullets identify lines; magnitude is also labeled in the table.
Line prevalence data
| Line | Eligible hours | Disrupted hours | Rate |
|---|
Hour-of-day pattern
Observed disruption frequency by hour, weekday versus weekend.
Monthly drift
The shaded band is the held-out test period; training occurred in a calmer era.
Methodology and responsible-use limitations
The model mostly learned structure, not conditions
The two dominant features are the line's own trailing 30-day rate and the hour of day. It has learned that the A is chronically disrupted and that disruption clusters at certain hours — durable facts you could write on an index card. Features describing evolving conditions contribute far less, which is why gradient boosting beats logistic regression by only 0.006 PR-AUC.
The base rate nearly doubled across the snapshot
Monthly rates run about 7.7% in early 2021 against 15.4% by April 2026. The chronological split is the correct choice for forecasting, but it means training happens on a quieter period than testing. The tuned threshold transfers here only because validation (14.2%) and test (14.4%) happen to be close; that will not hold for a future period without retuning.
The label is a proxy for alerts, not for lateness
A positive means the MTA published a qualifying unplanned alert for that line in that hour — not that any specific train ran late. Changes in the agency's alerting practice move this target independently of actual service, and are a plausible contributor to the drift above.
This is retrospective only
Every number here comes from a static historical snapshot. There is no live ingest, no incremental feature computation, and no serving path — so nothing on this page is a live prediction, and at 23.8% precision a customer-facing "delays expected" alert would be wrong roughly three times out of four.
Methodology details
Reproducible data
Immutable, hash-manifested alert snapshot with auditable SQL layers and scheduled-service eligibility.
Leakage-safe features
Every input is computed strictly before prediction hour t.
Chronological evaluation
70% train, 15% validation, 15% held-out test; threshold tuned only on validation.
Exact static inference
All 300 trees are embedded and traversed locally; export refuses disagreement with scikit-learn above 1e-5.