logger

logger — per-step log accumulator + dotted-key NPZ IO

Purpose

Logger accumulates per-step payloads into a RecursiveNamespace of LogEntry series; LogStore
serialises/deserialises them to dotted-key NPZ files and enforces the scoreboard-completeness
contract (a log that can’t be judged is a hard-fail on write, a loud warn on read).

Role in the system

• The L in the one-way pipeline pre_run_loader → runner → logger → ((NPZ)) → data_analyzer → plotter / star_reporter.
• The controller (run_control_loop) hands a Package per step to Logger.append_payload; runner / orchestrator own the call.
• LogStore.save_raw_log is the only sanctioned NPZ writer; LogStore.load_log_from_npz the only reader.
• On load it reads the per-series payload straight from the NPZ (dropping RAW_LOG_EXCLUDED_KEYS) and rebuilds a data_analyzer LogView-shaped Logger.
• The completeness bar is the scoreboard set from ScoreboardSpecLoader (parameter_loader / pre_run_loader), not the full metrics.yaml catalog.

Inputs / Outputs

• In (write): a live Logger whose .logs tree holds LogEntry series (per-step samples, optional paired _desired).
• Out (write): a dotted-key NPZ (np.savez), per-step time series only — no scalars/config knobs; a RawLogSaveResult(path, skipped_keys).
• In (read): one NPZ path (lists are rejected — map comparison lists onto contexts first).
• Out (read): a reconstructed Logger, with a coverage warning if scoreboard metrics are missing.

Key methods/functions

• scoreboard_metric_coverage — split raw keys into (present, missing) vs the scoreboard set, by leaf name — analysis/logger.py:19
• Logger.append_payload — recursively fold a step payload into the log tree (Package nests, None skips) — analysis/logger.py:81
• LogStore.save_raw_log — write the dotted-key NPZ; require_complete=True hard-fails on gaps — analysis/logger.py:228
• LogStore.load_log_from_npz — load one NPZ (schema-compat applied) → Logger — analysis/logger.py:112
• LogStore.raw_log_payload — flatten a Logger to {dotted-key: array}, pairing _desired secondaries — analysis/logger.py:253
• LogStore.populate_logger_from_raw_payload — rebuild the tree, pairing <key>_desired as <key>’s secondary — analysis/logger.py:144
• LogStore.log_entry_for_path — resolve/create the LogEntry leaf for a dotted key — analysis/logger.py:174
• LogStore.raw_series_values — return a key’s series array; raise on a 0-d (scalar metadata) entry — analysis/logger.py:212
• LogStore.iter_log_entries — walk the tree, yielding (path-tuple, LogEntry) per leaf — analysis/logger.py:238

Footguns

_desired keys are loaded as secondaries, not as their own entries

populate_logger_from_raw_payload skips any key ending in _desired during iteration — it is
attached as the paired (secondary) series of its primary <key>. Loading it again would create a
duplicate LogEntry. (analysis/INSIGHTS.md [footgun])

The dotted key space must be a proper tree — LogEntry only at leaves

log_entry_for_path raises RuntimeError if a key would nest under an existing LogEntry (e.g.
a.b exists as a LogEntry, then a.b.c is requested). (analysis/INSIGHTS.md [footgun])

NPZ logs are per-step series ONLY — no scalars

raw_series_values hard-rejects 0-d entries with a clear error. Scalars/config knobs (suppress, dt,
cap, wall) live in the run-spec YAML / config_overrides / parameters.yaml and are rebuilt from cfg
at analysis time. (analysis/INSIGHTS.md [io])

Completeness bar = the scoreboard set, NOT the full catalog

Coverage is matched against YAMLs_by_domain/scoreboard.yaml (the judgment contract), by leaf name.
The full metrics.yaml catalog has include:true metrics that are config-conditional (e.g. reactive
scorer scores absent in POSE mode), so it can’t serve as a universal bar. require_complete=True on
write blocks any log that the scoreboard can’t fully judge. (analysis/INSIGHTS.md [io])

Pseudocode

# write (run → NPZ)
raw = raw_log_payload(logger)              # walk leaves; skip t, empty, RAW_LOG_EXCLUDED_KEYS
  for each LogEntry leaf:
    stack samples → array                  # ragged / object-dtype → record skip, drop key
    if has_secondary: emit <key>_desired   # paired actual/desired
  if require_complete and missing: raise    # scoreboard contract
np.savez(path, **raw) → RawLogSaveResult(path, skipped_keys)

# read (NPZ → run)
raw = {k: d[k] for k in np.load(path).files if k not in RAW_LOG_EXCLUDED_KEYS}  # per-series payload
report_metric_coverage(raw)                # warn (not raise) on missing scoreboard metrics
for key, series in raw:                    # _desired folded into its primary's secondary
    log_entry_for_path(key).append(...)

Related

runner · orchestrator · data_analyzer · detached_run · parameter_loader · infra · terminology