Quartz 5

metric_alignment

Properties2
tagsanalysis
aliasesmetric_alignment

3 min read

metric_alignment — metric-catalog verifier + payload alignment smoke

Purpose

Two-pass guard for the metric pipeline. Pass 1 verifies metrics.yaml is structurally sound
(no null placeholders, parseable reductions/deltas, every leaf has a variable) and that the
catalog actually loads with no duplicate variables. Pass 2 (the alignment smoke) checks that a
live payload’s observed variable leaves match what the catalog requests. Also hosts deprecated
legacy fabricated-payload smokes.

Role in the system

  • Verifies metrics.yaml (the variable catalog) — what is measured — built by the
    MetricSpecLoader in pre_run_loader; judgment semantics live in scoreboard.yaml, not here.
  • Pulls its catalog/spec/reduction symbols from data_analyzer (Reductions, Deltas, DataSummary)
    and metric_catalog (catalog_by_key, catalog_specs) — all packed into one namespace by
    metric_spec_loader_tools.
  • Run as a script (python -m analysis.metric_alignment) it prints both verifier reports and exits
    non-zero on failure — the cheap, simulation-free catalog gate referenced in .claude/rules/ANALYSIS.md.
  • The alignment smoke consumes pipeline payloads (a StepLog / guidance log); not called by
    orchestrator in the live loop — it is a boundary smoke, not a pipeline stage.

Inputs / Outputs

  • In (verifiers): metrics.yaml (or an in-memory metric namespace / selected-key list via metrics=).
  • In (smoke): one or more live payloads (nested records with named-series leaves).
  • Out: Package reports with ok plus structured findings — YAML issues (level/check/path/message),
    pipeline duplicates, or alignment buckets missing / ambiguous / path_mismatches /
    length_mismatches. Each report has a matching print_* for the console.

Key functions

  • metric_spec_loader_tools — pack catalog/spec/reduction symbols into one namespace — analysis/metric_alignment.py:36
  • metric_yaml_node_issues — recursive structural verifier over the metric tree — analysis/metric_alignment.py:175
  • metric_yaml_verification_report — Pass 1: parse + structure + loader-build — analysis/metric_alignment.py:202
  • metric_pipeline_verification_report — catalog loads, reduction items build, no duplicate variables — analysis/metric_alignment.py:283
  • duplicate_metric_variables — variables bound to more than one catalog path — analysis/metric_alignment.py:271
  • metric_payload_paths — flatten a payload into {dotted leaf: sample count} — analysis/metric_alignment.py:381
  • metric_alignment_smoke — Pass 2: bucket payload-vs-catalog mismatches — analysis/metric_alignment.py:448

Footguns

The # noqa: F401 guards are load-bearing — never strip them

metric_spec_loader_tools imports symbols only to hand their string names to
Package.pack(scope=locals()), which resolves them by name. Linters see them as unused; a Ruff pass
once silently deleted them (commit de0b788) and broke the verifier with no immediate error.
(analysis/INSIGHTS.md [footgun])

FALLBACK is advisory; only MISSING / AMBIGUOUS / LENGTH fail the smoke

The smoke buckets are MISSING (variable absent), AMBIGUOUS (name found but catalog path absent among
multiple candidates), FALLBACK (name found, catalog path unmatched — one candidate), LENGTH
(sample count ≠ expected). FALLBACK does not set ok=False. (analysis/INSIGHTS.md [config])

The five legacy smokes are deprecated fabricators

guidance_metric_alignment_smoke, empty_controller_diagnostics, empty_guidance_diagnostics,
empty_ee_controller_step_log, ee_controller_metric_alignment_smoke predate the YAML-driven
pipeline; each emits a DeprecationWarning via warn_legacy_smoke. Verify against real payloads
with metric_alignment_smoke + metric_yaml_verification_report instead. Open question:
empty_ee_controller_step_log hand-builds a zero-filled StepLog and should call empty-path
constructors. (analysis/INSIGHTS.md [history] / [footgun])

Pseudocode

# Pass 1 — metrics.yaml structure + loader
load_spec(metrics.yaml)            # parse_ok / parse_error
walk metric tree:                  # metric_yaml_node_issues, recursive
  null non-metadata child          -> ERROR missing_fields
  reductions not a parseable list  -> ERROR / WARN
  nested delta missing method/etc  -> WARN / ERROR
  leaf w/o children & no variable  -> ERROR missing_fields   (default* exempt)
MetricSpecLoader(...).load_metric_catalog()   # loader_ok
ok = parse_ok and loader_ok and no ERROR issues

# Pass 2 — payload vs catalog (the alignment smoke)
observed = flatten payloads -> {path: samples}, indexed by trailing variable name
for spec in catalog.specs:
  no provided counts            -> MISSING
  catalog path absent, >1 cand  -> AMBIGUOUS
  catalog path absent, 1 cand   -> FALLBACK (advisory)
  samples != expected           -> LENGTH
ok = no MISSING and no AMBIGUOUS and no LENGTH

pre_run_loader · metric_catalog · data_analyzer · orchestrator · ee_guidance · terminology

Graph View

Backlinks