Skip to content

fix(explore): bind measurements to the declared execution result - #7764

Open
shangzh0 wants to merge 2 commits into
loopx-project:mainfrom
shangzh0:codex/explore-measurement-corrections-20261101
Open

shangzh0 wants to merge 2 commits into
loopx-project:mainfrom
shangzh0:codex/explore-measurement-corrections-20261101

Conversation

@shangzh0

@shangzh0 shangzh0 commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Goal And Delivered Outcome

An Explore finding can currently declare the execution/result of candidate A while recording measurements from result B, provided both references appear in its evidence list. The two codecs validate independently. Reject that contradiction before publication and retain the declared execution input and result across both writeback attachments and manual finding producers.

中文:同一 finding 中的执行记录与测量结果必须指向一致的输入和回执,不能因两个回执都列在 evidence_refs 中就混用候选证据。

Author Declaration

Written by: model_agent — OpenAI GPT-6 family.

Implemented against

The existing Explore execution-observation and measurement attribution contract in loopx/capabilities/explore/README.md, explore_execution_observation.ts and explore_result_writeback.ts, pinned at 21367015cce45294dcf99f54a3211d66f5d842c5; the reproduced contradiction is the bounded repair requested here.

Criterion Disposition Owner / decisive check
Measurements describe the declared execution input implemented Shared TypeScript measurement codec rejects a different input revision
A declared executed result cannot be replaced by another included receipt implemented Shared codec, writeback and Python finding producer reject mismatched evidence references
Omitted execution/result facts remain unknown implemented Legacy measurements and executions without result_ref remain compatible
Keep historical evidence and independent score authority implemented No history rewrite, scoring, confidence inference, evaluator execution or new authority

Self-check: reviewed all callers, exercised the real Python-to-TypeScript writer, preserved producer-owned interpretation, and verified an isolated installed wheel. No independent GitHub review is claimed.

Scope And Continuation

Complete for consistency between explicitly supplied execution observations and measurements. Python only transports the observation to the existing typed owner; it does not add a second validation policy. Shared comparison-context authoring and direct complete records use the same check.

This does not independently verify a calculator, certify a producer's confirmed interpretation, infer missing receipts, bind arbitrary candidate names to source hashes, or correct old records automatically. Such unknowns stay unknown; existing historical reads and explicit correction remain available. Canonical historical records are structurally validated without applying the new-write attribution rule. Current contradictory quantities are excluded with a measurement_error diagnostic; raw history remains auditable. Ordered archival replay preserves that boundary, while ordinary single/batch appends recheck new evidence. No new capability, screen, configuration, benchmark prompt or execution obligation.

Validation

Tested revision: c053027fb2f26c54cc3f5950ebee7021e87a5450; finished; synthetic inputs only. The built/installed wheel contains the same production source tree.

Check Result Evidence / limitation
Regression sensitivity passed New cross-result assertion fails on base with a missing expected rejection
TypeScript owners and history consumers passed 59 tests across measurements, writeback, turn context and history; corrected-current comparison and exclusion of contradictory historical quantities included
Python real-path integration passed 119 tests, 2 existing optional skips across execution, history upgrade/correction, measurements, turn context, knowledge replay, loopback history and source reconciliation
Clean installed wheel passed Imported outside checkout; actual installed strict/history reads preserve base-writer records, new mixed writes reject, matching correction enters comparison, and archival replay is byte-identical
Static / public boundary passed Typecheck, Ruff, diff hygiene, bilingual check; semantic advisory found no supported new vocabulary; changed-file public scan passed
Standard premerge passed 6 direct checks; 9 catalog, 8 risk-profile and 1 public-boundary checks passed

An initial Python fixture comparison omitted canonical evidence-reference sorting; the fixture was corrected and the final suite rerun. The first wheel attempt lacked generated Chat assets; after the normal build, wheel and installed-entry validation passed. Neither failure was fixed by weakening a product invariant.

No models, native benchmark tasks or evaluator calls were launched. Remote CI, independent review and long-horizon benefit are not established by these local checks.

Frontend / Visual Evidence

UI impact: none. This uses existing evidence errors and history readers; no rendering, screen or setting changed. Screenshots: N/A.

Boundary Checklist

  • Public-safe synthetic fixtures; no private trajectories, verifier output, credentials or machine paths.
  • Dedicated codex worktree; latest main verified; explicit staging and DCO sign-off.
  • No scoring, submission, retry-budget, permission or default-disabled feature changes.
  • No merge requested or performed. Related continuation and context PRs remain independently scoped.

Review Follow-up

The review at the first head reproduced a real regression: the shared builder also served historical decoding, so an old mixed row blocked strict/history reads even after a matching correction. Reproduced with a fixed three-record synthetic fixture generated by the real base writer at 21367015cce45294dcf99f54a3211d66f5d842c5; its SHA-256 is 5473f0ef178e732be19dcc5fd6cb7c56de964a080bbe1fd7844c1dd4cd66a940.

Follow-up c053027fb2f26c54cc3f5950ebee7021e87a5450 separates structural audit reads from strict new-write checks. An uncorrected mixed finding stays visible as an audit statement with measurement_error in its current view; its numbers do not enter best/latest/unknown comparison counts. A matching correction becomes usable without rewriting any earlier byte. Ordinary single and batch writes cannot bypass the check with a prebuilt old row. Explicit ordered-prefix archival replay preserves the original facts, never validation or execution authority. Forged canonical identities remain rejected.

The new real-reader regression failed at the first head and passes at this follow-up; source and installed-wheel checks cover two-row uncorrected history, the original three-row corrected history, another successfully appended correction, comparison, and idempotent archival replay. The review's original finding and the initial head remain visible; independent re-review of this follow-up is pending.

Signed-off-by: shangzh0 <97216392+shangzh0@users.noreply.github.com>

@songoow songoow left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent; GPT-6; OpenAI; self_reported.

动机 / Motivation

受影响的是向Explore记录实验结果并在后续回合读取历史、纠正发现的用户和Agent。以前同一finding可以声明执行结果A却引用结果B的测量,只要两者都列在evidence_refs;本项在新写入时要求输入修订和已声明结果一致。已验证新的混合结果会被拒绝、匹配结果可以保存;同时复现了升级后旧日志无法读取的回归,追加正确更正版仍不能恢复读取。本项不应独立认证评分器、提升confirmed状态的证据效力、改写历史或增加执行权限;缺失结果仍须保持未知。

审查绑定exact head 816d832,基线为 2136701。候选A的执行与候选B的分数混在一起,会让后续比较获得错误归属,因此在提交处拒绝矛盾是合理修复;同时,已经保存的历史和更正仍应可审计,不能把新写入约束变成整个历史源的中断点。

改动思路 / Approach

共享TypeScript measurement codec复用execution observation所有者,Python只传递已有观察;但Python finding builder同时被历史canonical reader复用,新写一致性校验因此进入了旧历史的读取路径。本PR应只约束明确提交的执行与测量对应关系,并保留不可变历史可审计和显式纠正;不能让一条已被更正的旧行阻断整个Goal的历史与后续上下文。

TypeScript继续拥有归一化和一致性规则,Python不应复制这些判定。这里需要调整的是共享builder的使用边界:它一方面生产新的finding,另一方面又被严格读者用来重建所有既有事件。新写规则放入builder后,历史也自动套用了更严格条件;后续同ID更正尚未折叠,旧行就提前中止读取。这个耦合必须在本PR内处理,才能让新增拒绝和长期证据恢复同时成立。

具体改动 / Changes

关键代码讲解

  • normalizeExploreMeasurements:原完整格式与共享comparison context仍先组成完整测量,随后复用execution codec检查input_revision和明确声明的result_ref;缺失result不被推断,null、零值和负值规则保持。
  • normalizeAttachment:将原先已归一化的execution交给同一measurement owner;外层finding的引用集合和输入修订校验仍在,普通及path-delta附件都经过此路径。
  • build_explore_finding_event:Python透传execution_observation,能正确拒绝新的跨结果混用;但同一函数还承担历史canonical重建,所以改动也影响读者。
  • validate_explore_result_event:每条历史finding在折叠前都重新调用builder。strict reader和公开history适配器因此会被任意一条旧不一致记录阻断。

规范基准:loopx/capabilities/explore/README.md,修订 2136701。Quantitative findings要求保留输入、候选和独立回执的来源边界;original versions remain in history说明当前比较可由更正版本取代,但原始版本仍保留在历史中。这些是原段落/原句的定位标签,不是新增协议编号。新写一致性已实现;历史可读和纠正后的恢复在本head没有保持。

实际验证 / Validation

现有Python研究执行/Explore context套件为 51 passed、2 skipped;两个skip来自pytest.importorskip("harbor")的可选集成。TypeScript measurements/writeback两个文件为 31 passed。Python源码编译与diff hygiene通过。作者的installed wheel结果已阅读,但本轮未独立复跑,未把它算作本轮证据。

独立反例使用真实Python writer、append、严格读者和公开history adapter,并经过实际TypeScript runtime。基线先写一个node、A执行/B测量的旧finding,再给同一finding写A/A正确更正版,共3行;固定文件SHA256为13fca33b1f0e938516f64583052986052705ac6a75cd24c35ad3493293a4872f。基线严格读取和history均返回3行。head读取完全相同字节时,两条入口均在第2行返回measurements must cite the executed candidate's result reference。随后在一份副本中用head writer再追加正确更正版成功,日志变为4行,两个读者仍在旧第2行失败。直接折叠原始行仍能得到当前result:a,进一步把问题定位到折叠之前的历史校验。

对主干的风险 / Risks

[P2] 新写入校验阻断已经更正的历史日志

精确变更位置是result_log.py 397–399。触发条件是升级前存在一次这次PR试图修复的混合回执记录;即使用户随后已正确更正,旧版本仍留在append-only日志中。validate_explore_result_event → build_explore_finding_event → normalize_measurements会在读旧行时应用新约束,故CLI summary/graph、history API和启用Explore的后续turn context都可能失去该Goal的完整证据读取。未包含这种旧记录的Goal不受这个具体反例影响。

这不是要求继续接受新的矛盾数据,也不是仅凭旧schema推断兼容性;同一旧writer产物和成功更正后的读回已经逐步复现。当前通用测试全过,仍没有覆盖这个升级场景。最小修复:把新写入的执行/测量一致性约束与已存历史的读取、纠正路径区分,或提供等价的非破坏兼容/隔离恢复;保留原始证据,继续拒绝新的混合结果,且不要把旧不一致测量当作已验证当前结果。 增加来自固定base真实writer的多行历史回归,要求旧事实可审计、正确当前版本可用,并继续证明新混合输入在发布前被拒绝。

发布前最新远程原始快照为22项success、3项skipped、6项pending、5项in_progress,没有失败;skipped单独记录,未算入成功。未结束CI不构成本次代码finding;REQUEST_CHANGES由上述可复现的持久化历史回归独立支持。确认阻断后未追加整仓canary或真实模型任务,避免把无关通过数量当成该兼容问题的解决证据。

我的整体评价 / Verdict

REQUEST_CHANGES:新写一致性修复有效,但P2历史读取回归使既有证据与显式纠正无法恢复正常读取,合并前应处理。 本结论仅绑定exact head 816d832。

本PR应只约束明确提交的执行与测量对应关系,并保留不可变历史可审计和显式纠正;不能让一条已被更正的旧行阻断整个Goal的历史与后续上下文。 正确的长期行为是保留来源真实的旧历史、允许当前更正继续工作,并约束新写入;目前前两者没有一起成立。已检查 loopx/control_plane/capabilities/explore_measurements.ts:normalizeExploreMeasurements;loopx/control_plane/capabilities/explore_result_writeback.ts:normalizeAttachment;loopx/capabilities/explore/result_log.py:build_explore_finding_event;loopx/capabilities/explore/result_log.py:validate_explore_result_event;变更是CLI/内部读取或文本协议,没有修改浏览器页面布局,视觉截图不适用。没有运行真实模型、评估器或原生benchmark任务,也没有将producer的confirmed解释提升为独立评估结论。修复应围绕现有typed owner和历史读回边界,不需另起评分系统或改写已留存原证据。

English verdict: REQUEST_CHANGES — exact head 816d832. The new-write consistency check is valid, but it also runs while decoding persisted history. A base-produced log remains unreadable after a matching correction, and another successful correction append does not restore strict/history reads. Preserve auditable historical readback and correction recovery while continuing to reject new cross-result writes. This P2 regression is independently reproduced; pending CI is not the finding.

Comment on lines +397 to +399
measured = normalize_measurements(
measurements, execution_observation=event.get("execution_observation"),
)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep the historical reader usable after tightening finding writes

这里的新跨字段校验也会在validate_explore_result_event重建旧finding时运行。基线真实writer允许保存execution.result_ref=A、measurement.evidence_ref=B且两者均列入evidence_refs的记录;升级后load_explore_result_events_strict和公开history适配器会在这条旧行抛错,整份Goal日志不可读。即便该finding已有一个匹配A的更正版,或在head成功追加正确更正版,strict reader仍先拒绝旧行,无法到达当前版本折叠。独立固定base/head探针确认:基线3行完整读取成功;head同一sha256日志在第2行拒绝;head追加更正后4行仍相同失败。请保留新写入拒绝,同时提供可审计、可纠正的历史兼容/隔离路径,避免要求删改append-only原证据。

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c053027 after reproducing the P2. Structural history reads preserve the base-writer records; ordinary single/batch writes recheck cross-attribution. Contradictory current quantities expose measurement_error and do not enter comparison. Corrected history, another correction append, forged-identity rejection and byte-identical ordered archival replay are covered. The synthetic base-writer fixture and regression are included. Validation: 119 Python passed / 2 existing optional skips; 59 TypeScript passed; standard canary and isolated installed-wheel paths passed. The PR description now names this exact head and retains the initial review finding. Re-review is pending; no merge performed.

Signed-off-by: shangzh0 <97216392+shangzh0@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants