CHI / CHIS reference starter kit
An original, standard-library Python 3.10+ implementation of a proposed measurement framework. No external packages, model calls, proprietary code, or network access. This is an executable specification and synthetic demonstration, not an empirically validated benchmark or a consciousness test.
Run
From this folder:
python -m unittest -v
python chi_chis.py demo --seed 42 --output demo
python chi_chis.py score examples/score.json
python chi_chis.py sweep examples/sweep.json
python chi_chis.py log examples/event.json --output events.jsonl
demo/config.json, demo/summary.json, and demo/trajectory.jsonl are included. Regenerating with the same seed reproduces these files. The standalone config hashes to the config_hash saved in the summary and logs. The demo has eight independent synthetic run clusters, five CHIS steps per run, and a separate five-load sweep. Logged wall times are simulated seconds, not performance measurements. The demo generator assumes run independence; that assumption must be justified for real data.
Definitions and API
Axis order is R, M, C, A, I. Capacity profile p and human-similarity profile h are distinct, each bounded in [0,1]. A high capacity need not be human-like. Labels and task-level estimators must be fixed in a measurement protocol; this module does not infer them.
| Function | Contract |
|---|---|
profile(values) |
Requires exactly the five axes, finite numbers in [0,1]; rejects booleans. |
chi(h, weights=None) |
100 times the weighted geometric mean of h, not p. Positive weights normalize to sum one, equal by default. Any zero produces exactly zero. Missing reference (h=None) produces None / JSON null. |
human_similarity(z, mu, scales) |
Proposed exp(-0.5 * mean(((z-mu)/scales)^2)); requires equal nonempty vectors and positive, fixed scales. Call separately per axis. |
q_m(factors, weights=None) |
Geometric mean of r,t,s,u,i,1-f; six equal positive weights by default. All supplied values validate independently. Missing factors produce None, never an invented aggregate. |
eimc(scores, threshold=.8) |
Mapping of positive integer tested N to Q_M. Returns largest passing tested N; empty/no passing gives null. No interpolation. |
cluster_bootstrap(clusters, statistic, samples=2000, seed=0, confidence=.95) |
Percentile interval resampling whole independent clusters. Callback receives a list of clusters, including duplicates, and must recompute the complete estimand. Fewer than two clusters returns null. |
wilson(successes, total, confidence=.95) |
Bounded Wilson interval for a simple independent Bernoulli proportion; zero total returns null. |
make_event(...) |
Validates profiles, computes CHI, hashes config/history/state, returns a serializable record. |
demo(output_dir, seed=42) |
Writes seeded synthetic JSON and JSONL; returns summary. |
The feature formula is a proposed similarity score, not a calibrated probability, and no real human reference is supplied. In real use, predefine features, obtain a suitable human reference, estimate reference means and fixed positive scales on reference/training data, and freeze those choices before evaluation. Passing human-reference is a caller assertion, not independent verification. Demo similarities are marked synthetic-demo-only throughout.
Q_M factors must come from distinct operational measurements. This kit combines the supplied estimates; it cannot verify that their experimental measurements were independent, valid, or free of information leakage. The synthetic fixture simply generates six distinct draws for each run/load and is not a task implementation.
For the lure factor, use the conservative failure-adjusted false-memory score f = (unsupported affirmative outcomes + invalid/timeout lure outcomes) / all lures, with mutually exclusive outcome categories. Only a valid abstention passes a lure. In a real benchmark, also report f_raw = unsupported affirmative outcomes / all lures and the invalid/timeout rate separately. For example, six valid abstentions, two unsupported affirmatives, and two invalid/timeouts among ten lures give f=.4, f_raw=.2, and invalid rate=.2. The API takes an already-scored f and does not classify raw outcomes; with no lures, omit f and leave Q_M unavailable.
EIMC reports lower_bound/right_censored when the largest tested N passes. Otherwise a successful result is largest_passing_tested, which does not prove failure at every untested larger N. sustained_prefix_eimc is an additional conservative diagnostic requiring every smaller tested load to pass; it never replaces the requested maximum. Increasing quality at a larger tested N triggers a nonmonotonicity warning. Classification uses point estimates; sweep intervals are descriptive and are not a confidence interval on EIMC.
Uncertainty
Choose the independent unit before analysis: typically a complete run or agent trajectory, not a log event. If agents share run-level randomness, resample their common run together. Serial events from one trajectory are not independent replicates. The demo computes CHI from the cluster-balanced mean similarity profile and recomputes that nonlinear aggregate for every bootstrap sample. Q_M intervals likewise recompute Q_M from sampled factor means. This estimand differs from mean per-event CHI; select and report one explicitly. Per-event CHI in JSONL is descriptive and has no asserted CI.
Wilson is appropriate for a simple independent binary proportion. Do not apply it to geometric aggregates or treat events within a trajectory as independent trials. The kit's percentile bootstrap is deliberately small and transparent; it does not address reference-estimation uncertainty, hierarchical sampling, multiple testing, or small-cluster coverage. Eight synthetic clusters demonstrate mechanics only.
Inputs and logging
score accepts capacities, optional human_similarity, reference_kind, and optional five-axis weights. Capacities alone produce null CHI. sweep accepts {"scores":{"1":0.9,"2":0.7},"threshold":0.8}. Load keys must be canonical positive integers: leading zeros, signs, whitespace, non-ASCII digits, and zero are rejected. Duplicate JSON object keys are rejected before decoding can silently discard values. log accepts the keyword arguments of make_event and appends one JSONL record; it does not deduplicate events or provide concurrent-writer locking. Its synthetic flag must be a JSON boolean, not a string or number.
Required event arguments: config, run, agent, seed, step, wall_time, history, state, capacities. Optional: human_similarity, reference_kind, synthetic, budgets. Similarity requires human-reference or synthetic-demo-only; the latter also requires synthetic=true. Records contain version, SHA-256 config/history/state hashes, run/agent/seed, step and wall time, capacity and similarity profiles, CHI, reference label, synthetic flag, and budgets. Hashes use sorted compact JSON; hash identity assumes the same JSON representation and does not prove experimental reproducibility. Save the original configuration and reference metadata alongside logs. Do not put secrets in history/state; hashes are not anonymization.
The demo writes 40 trajectory records and summary sweep estimates. It makes no claim that synthetic CHI, EIMC, or uncertainty measures any real system. CHIS here means the logged sequence of profiles and CHI over five steps; optional derived distance, drift, and continuity metrics are intentionally left to a specified protocol.
License
MIT; see LICENSE. Python standard library only.