Static scorecard, regenerated locally on 2026-09-02. warncal is an offline, deterministic backtester; these four pages are its own HTML output, unmodified apart from this note. Nothing here is live or updating.

Data: a synthetic hazard index generated by the project itself, not real weather. It exists to exercise the scoring machinery on a signal whose ground truth is known by construction.

warncal scores warning rules. It is a measurement tool, not a decision authority: it is not medical, clinical or safety advice, it is not an emergency service, and a favourable verdict here does not make a rule safe to deploy. A human who understands the hazard must make that call.
← all four scorecards

synthetic hazard index -- probabilistic

WATCH · 2023-01-01 to 2024-12-31 · 268 events · 187 warning episodes

SHIP
Clears every gate for a WATCH.

Gates

PASSsample size268 observed events (need >= 20)
PASSskill (PSS)PSS = 0.177 (need > 0); grid slot 1:00:00
PASSWATCH objective (recall)recall = 1.000 (need >= 0.80)
PASSrecall lower CI bound95% CI 1.000 [1.000, 1.000] (lower bound must reach 0.80)
PASSmedian lead timemedian 13.0 h, IQR 13.0-13.0 h (need >= 1 h)
PASSfrequency biasbias = 0.70 (want 0.5-3); 187 warning episodes for 268 events

Metrics

metricvalue (95% CI)counted overdefinition
recall1.000 [1.000, 1.000]eventsa/(a+c) of the events, the share that were warned
precision0.781 [0.724, 0.837]episodesa/(a+b) of the warnings issued, the share that verified
far0.219 [0.163, 0.276]episodesb/(a+b) false alarm RATIO (= 1 - precision)
f10.877 [0.840, 0.911]mixedharmonic mean of precision and recall
csi0.867 [0.828, 0.904]mixeda/(a+b+c) critical success index / threat score
bias0.698 [0.594, 0.809]episodes/events(a+b)/(a+c) >1 over-warns, <1 under-warns
pofd0.823 [0.792, 0.853]gridb/(b+d) false alarm RATE (needs correct negatives)
hss0.007 [0.005, 0.008]gridHeidke skill score, 0 = chance, 1 = perfect
pss0.177 [0.147, 0.208]gridPeirce skill score = POD - POFD, base-rate insensitive
ets0.003 [0.003, 0.004]gridequitable threat score (CSI corrected for chance hits)

Object counts

counted overtotalrightwrongdrives
events268268 hits0 missesrecall / POD
warning episodes187146 verified41 false alarmsprecision / FAR
raw alerts9622merged into 187 episodesnothing directly — merging prevents double counting

Multiplicity 1.84 hits per verified episode. At 1.00 the two denominators agree and every object score is unambiguous; well above 1.00 means long episodes are sweeping up several events at once.

Contingency tables

tablehitsfalse alarmsmissescorrect negatives
object, classical mixed268410n/a by design
grid (slot 1:00:00)2681420203050

Lead time

median 13.0 h, IQR 13.0–13.0 h, range 12.0–13.0 h over 268 hits.

Probabilistic

binnmean forecastobserved freq95% CI
0.00–0.1010930.0650.070[0.057, 0.087]
0.10–0.2011820.1660.178[0.157, 0.200]
0.20–0.305990.2570.269[0.235, 0.306]
0.30–0.4023500.3730.352[0.333, 0.372]
0.40–0.5011990.4600.459[0.431, 0.487]
0.50–0.6031990.5290.506[0.488, 0.523]
0.60–0.700–––
0.70–0.800–––
0.80–0.900–––
0.90–1.000–––

Brier 0.20734 · Brier skill 0.098 · ROC AUC 0.682 · 9622 grid slots, base rate 0.35793

Diagnostics

verification diagnostics for synthetic hazard index -- probabilistic

Notes

Plain text scorecard

==============================================================================
  warncal scorecard  --  synthetic hazard index -- probabilistic
  role: WATCH   period: 2023-01-01 to 2024-12-31
==============================================================================

  VERDICT: SHIP
    Clears every gate for a WATCH.

------------------------------------------------------------------------------
  GATES
    [PASS] sample size
           268 observed events (need >= 20)
    [PASS] skill (PSS)
           PSS = 0.177 (need > 0); grid slot 1:00:00
    [PASS] WATCH objective (recall)
           recall = 1.000 (need >= 0.80)
    [PASS] recall lower CI bound
           95% CI 1.000 [1.000, 1.000] (lower bound must reach 0.80)
    [PASS] median lead time
           median 13.0 h, IQR 13.0-13.0 h (need >= 1 h)
    [PASS] frequency bias
           bias = 0.70 (want 0.5-3); 187 warning episodes for 268 events

------------------------------------------------------------------------------
  OBJECT COUNTS (overlapping alerts merged into episodes)

    events            268   =  268 hits + 0 misses        -> recall
    episodes          187   =  146 verified + 41 false alarms  -> precision
    raw alerts       9622   merged into 187 episodes
    multiplicity     1.84   hits per verified episode (1.00 = clean units)

    Classical mixed table, for comparison with published studies:
                   event yes   event no
    warned yes          268          41
    warned no             0         n/a

  OPPORTUNITY GRID (slot = 1:00:00) -- the only source of correct negatives

                   event yes   event no
    warned yes          268       14202
    warned no             0        3050

------------------------------------------------------------------------------
  METRICS   (95% block-bootstrap CI where available)

    metric       value [95% CI]               counted over
    recall       1.000 [1.000, 1.000]         events
    precision    0.781 [0.724, 0.837]         episodes
    far          0.219 [0.163, 0.276]         episodes
    f1           0.877 [0.840, 0.911]         mixed
    csi          0.867 [0.828, 0.904]         mixed
    bias         0.698 [0.594, 0.809]         episodes/events
    pofd         0.823 [0.792, 0.853]         grid
    hss          0.007 [0.005, 0.008]         grid
    pss          0.177 [0.147, 0.208]         grid
    ets          0.003 [0.003, 0.004]         grid

    base rate    0.01530                      grid

  PROBABILISTIC
    Brier        0.20734
    Brier skill  0.098   (vs sample climatology)
    ROC AUC      0.682
    decomposition: reliability 0.00058 - resolution 0.02305 + uncertainty 0.22982

    reliability (forecast bin -> observed frequency)
      bin         n     mean p    obs freq   95% CI
      0.00-0.10    1093     0.065      0.070   [0.057, 0.087]
      0.10-0.20    1182     0.166      0.178   [0.157, 0.200]
      0.20-0.30     599     0.257      0.269   [0.235, 0.306]
      0.30-0.40    2350     0.373      0.352   [0.333, 0.372]
      0.40-0.50    1199     0.460      0.459   [0.431, 0.487]
      0.50-0.60    3199     0.529      0.506   [0.488, 0.523]
      0.60-0.70      0        -           -        -
      0.70-0.80      0        -           -        -
      0.80-0.90      0        -           -        -
      0.90-1.00      0        -           -        -

------------------------------------------------------------------------------
  LEAD TIME (event onset minus episode issue time, hits only)
    median 13.0 h   IQR 13.0 to 13.0 h   range 12.0 to 13.0 h
    median 95% CI [13.0, 13.0] h
   12.0 h |▁          █| 13.0 h   (n=268)

------------------------------------------------------------------------------
  NOTES
    - Each verified episode covered 1.84 events on average. Recall counts
      events and precision counts episodes, so they are not two views of the
      same denominator here; CSI and F1 mix the two and should be read with
      that in mind.

------------------------------------------------------------------------------
  warncal scores warning rules. It is a measurement tool, not a decision
  authority: it is not medical, clinical or safety advice, it is not an
  emergency service, and a favourable verdict here does not make a rule safe
  to deploy. A human who understands the hazard must make that call.
==============================================================================
Not advice. warncal scores warning rules. It is a measurement tool, not a decision authority: it is not medical, clinical or safety advice, it is not an emergency service, and a favourable verdict here does not make a rule safe to deploy. A human who understands the hazard must make that call.