Repository navigation
Conversation
Signed-off-by: shangzh0 <97216392+shangzh0@users.noreply.github.com>
songoow
left a comment
There was a problem hiding this comment.
Reviewer: model_agent; GPT-6; OpenAI; self_reported.
动机 / Motivation
受影响的是向Explore记录实验结果并在后续回合读取历史、纠正发现的用户和Agent。以前同一finding可以声明执行结果A却引用结果B的测量,只要两者都列在evidence_refs;本项在新写入时要求输入修订和已声明结果一致。已验证新的混合结果会被拒绝、匹配结果可以保存;同时复现了升级后旧日志无法读取的回归,追加正确更正版仍不能恢复读取。本项不应独立认证评分器、提升confirmed状态的证据效力、改写历史或增加执行权限;缺失结果仍须保持未知。
审查绑定exact head 816d832,基线为 2136701。候选A的执行与候选B的分数混在一起,会让后续比较获得错误归属,因此在提交处拒绝矛盾是合理修复;同时,已经保存的历史和更正仍应可审计,不能把新写入约束变成整个历史源的中断点。
改动思路 / Approach
共享TypeScript measurement codec复用execution observation所有者,Python只传递已有观察;但Python finding builder同时被历史canonical reader复用,新写一致性校验因此进入了旧历史的读取路径。本PR应只约束明确提交的执行与测量对应关系,并保留不可变历史可审计和显式纠正;不能让一条已被更正的旧行阻断整个Goal的历史与后续上下文。
TypeScript继续拥有归一化和一致性规则,Python不应复制这些判定。这里需要调整的是共享builder的使用边界:它一方面生产新的finding,另一方面又被严格读者用来重建所有既有事件。新写规则放入builder后,历史也自动套用了更严格条件;后续同ID更正尚未折叠,旧行就提前中止读取。这个耦合必须在本PR内处理,才能让新增拒绝和长期证据恢复同时成立。
具体改动 / Changes
关键代码讲解
- normalizeExploreMeasurements:原完整格式与共享comparison context仍先组成完整测量,随后复用execution codec检查input_revision和明确声明的result_ref;缺失result不被推断,null、零值和负值规则保持。
- normalizeAttachment:将原先已归一化的execution交给同一measurement owner;外层finding的引用集合和输入修订校验仍在,普通及path-delta附件都经过此路径。
- build_explore_finding_event:Python透传execution_observation,能正确拒绝新的跨结果混用;但同一函数还承担历史canonical重建,所以改动也影响读者。
- validate_explore_result_event:每条历史finding在折叠前都重新调用builder。strict reader和公开history适配器因此会被任意一条旧不一致记录阻断。
规范基准:loopx/capabilities/explore/README.md,修订 2136701。Quantitative findings要求保留输入、候选和独立回执的来源边界;original versions remain in history说明当前比较可由更正版本取代,但原始版本仍保留在历史中。这些是原段落/原句的定位标签,不是新增协议编号。新写一致性已实现;历史可读和纠正后的恢复在本head没有保持。
实际验证 / Validation
现有Python研究执行/Explore context套件为 51 passed、2 skipped;两个skip来自pytest.importorskip("harbor")的可选集成。TypeScript measurements/writeback两个文件为 31 passed。Python源码编译与diff hygiene通过。作者的installed wheel结果已阅读,但本轮未独立复跑,未把它算作本轮证据。
独立反例使用真实Python writer、append、严格读者和公开history adapter,并经过实际TypeScript runtime。基线先写一个node、A执行/B测量的旧finding,再给同一finding写A/A正确更正版,共3行;固定文件SHA256为13fca33b1f0e938516f64583052986052705ac6a75cd24c35ad3493293a4872f。基线严格读取和history均返回3行。head读取完全相同字节时,两条入口均在第2行返回measurements must cite the executed candidate's result reference。随后在一份副本中用head writer再追加正确更正版成功,日志变为4行,两个读者仍在旧第2行失败。直接折叠原始行仍能得到当前result:a,进一步把问题定位到折叠之前的历史校验。
对主干的风险 / Risks
[P2] 新写入校验阻断已经更正的历史日志
精确变更位置是result_log.py 397–399。触发条件是升级前存在一次这次PR试图修复的混合回执记录;即使用户随后已正确更正,旧版本仍留在append-only日志中。validate_explore_result_event → build_explore_finding_event → normalize_measurements会在读旧行时应用新约束,故CLI summary/graph、history API和启用Explore的后续turn context都可能失去该Goal的完整证据读取。未包含这种旧记录的Goal不受这个具体反例影响。
这不是要求继续接受新的矛盾数据,也不是仅凭旧schema推断兼容性;同一旧writer产物和成功更正后的读回已经逐步复现。当前通用测试全过,仍没有覆盖这个升级场景。最小修复:把新写入的执行/测量一致性约束与已存历史的读取、纠正路径区分,或提供等价的非破坏兼容/隔离恢复;保留原始证据,继续拒绝新的混合结果,且不要把旧不一致测量当作已验证当前结果。 增加来自固定base真实writer的多行历史回归,要求旧事实可审计、正确当前版本可用,并继续证明新混合输入在发布前被拒绝。
发布前最新远程原始快照为22项success、3项skipped、6项pending、5项in_progress,没有失败;skipped单独记录,未算入成功。未结束CI不构成本次代码finding;REQUEST_CHANGES由上述可复现的持久化历史回归独立支持。确认阻断后未追加整仓canary或真实模型任务,避免把无关通过数量当成该兼容问题的解决证据。
我的整体评价 / Verdict
REQUEST_CHANGES:新写一致性修复有效,但P2历史读取回归使既有证据与显式纠正无法恢复正常读取,合并前应处理。 本结论仅绑定exact head 816d832。
本PR应只约束明确提交的执行与测量对应关系,并保留不可变历史可审计和显式纠正;不能让一条已被更正的旧行阻断整个Goal的历史与后续上下文。 正确的长期行为是保留来源真实的旧历史、允许当前更正继续工作,并约束新写入;目前前两者没有一起成立。已检查 loopx/control_plane/capabilities/explore_measurements.ts:normalizeExploreMeasurements;loopx/control_plane/capabilities/explore_result_writeback.ts:normalizeAttachment;loopx/capabilities/explore/result_log.py:build_explore_finding_event;loopx/capabilities/explore/result_log.py:validate_explore_result_event;变更是CLI/内部读取或文本协议,没有修改浏览器页面布局,视觉截图不适用。没有运行真实模型、评估器或原生benchmark任务,也没有将producer的confirmed解释提升为独立评估结论。修复应围绕现有typed owner和历史读回边界,不需另起评分系统或改写已留存原证据。
English verdict: REQUEST_CHANGES — exact head 816d832. The new-write consistency check is valid, but it also runs while decoding persisted history. A base-produced log remains unreadable after a matching correction, and another successful correction append does not restore strict/history reads. Preserve auditable historical readback and correction recovery while continuing to reject new cross-result writes. This P2 regression is independently reproduced; pending CI is not the finding.
| measured = normalize_measurements( | ||
| measurements, execution_observation=event.get("execution_observation"), | ||
| ) |
There was a problem hiding this comment.
[P2] Keep the historical reader usable after tightening finding writes
这里的新跨字段校验也会在validate_explore_result_event重建旧finding时运行。基线真实writer允许保存execution.result_ref=A、measurement.evidence_ref=B且两者均列入evidence_refs的记录;升级后load_explore_result_events_strict和公开history适配器会在这条旧行抛错,整份Goal日志不可读。即便该finding已有一个匹配A的更正版,或在head成功追加正确更正版,strict reader仍先拒绝旧行,无法到达当前版本折叠。独立固定base/head探针确认:基线3行完整读取成功;head同一sha256日志在第2行拒绝;head追加更正后4行仍相同失败。请保留新写入拒绝,同时提供可审计、可纠正的历史兼容/隔离路径,避免要求删改append-only原证据。
There was a problem hiding this comment.
Fixed in c053027 after reproducing the P2. Structural history reads preserve the base-writer records; ordinary single/batch writes recheck cross-attribution. Contradictory current quantities expose measurement_error and do not enter comparison. Corrected history, another correction append, forged-identity rejection and byte-identical ordered archival replay are covered. The synthetic base-writer fixture and regression are included. Validation: 119 Python passed / 2 existing optional skips; 59 TypeScript passed; standard canary and isolated installed-wheel paths passed. The PR description now names this exact head and retains the initial review finding. Re-review is pending; no merge performed.
Signed-off-by: shangzh0 <97216392+shangzh0@users.noreply.github.com>
Goal And Delivered Outcome
An Explore finding can currently declare the execution/result of candidate A while recording measurements from result B, provided both references appear in its evidence list. The two codecs validate independently. Reject that contradiction before publication and retain the declared execution input and result across both writeback attachments and manual finding producers.
中文:同一 finding 中的执行记录与测量结果必须指向一致的输入和回执,不能因两个回执都列在 evidence_refs 中就混用候选证据。
Author Declaration
Written by: model_agent — OpenAI GPT-6 family.
Implemented against
The existing Explore execution-observation and measurement attribution contract in
loopx/capabilities/explore/README.md,explore_execution_observation.tsandexplore_result_writeback.ts, pinned at21367015cce45294dcf99f54a3211d66f5d842c5; the reproduced contradiction is the bounded repair requested here.Self-check: reviewed all callers, exercised the real Python-to-TypeScript writer, preserved producer-owned interpretation, and verified an isolated installed wheel. No independent GitHub review is claimed.
Scope And Continuation
Complete for consistency between explicitly supplied execution observations and measurements. Python only transports the observation to the existing typed owner; it does not add a second validation policy. Shared comparison-context authoring and direct complete records use the same check.
This does not independently verify a calculator, certify a producer's
confirmedinterpretation, infer missing receipts, bind arbitrary candidate names to source hashes, or correct old records automatically. Such unknowns stay unknown; existing historical reads and explicit correction remain available. Canonical historical records are structurally validated without applying the new-write attribution rule. Current contradictory quantities are excluded with a measurement_error diagnostic; raw history remains auditable. Ordered archival replay preserves that boundary, while ordinary single/batch appends recheck new evidence. No new capability, screen, configuration, benchmark prompt or execution obligation.Validation
Tested revision:
c053027fb2f26c54cc3f5950ebee7021e87a5450; finished; synthetic inputs only. The built/installed wheel contains the same production source tree.An initial Python fixture comparison omitted canonical evidence-reference sorting; the fixture was corrected and the final suite rerun. The first wheel attempt lacked generated Chat assets; after the normal build, wheel and installed-entry validation passed. Neither failure was fixed by weakening a product invariant.
No models, native benchmark tasks or evaluator calls were launched. Remote CI, independent review and long-horizon benefit are not established by these local checks.
Frontend / Visual Evidence
UI impact: none. This uses existing evidence errors and history readers; no rendering, screen or setting changed. Screenshots: N/A.
Boundary Checklist
Review Follow-up
The review at the first head reproduced a real regression: the shared builder also served historical decoding, so an old mixed row blocked strict/history reads even after a matching correction. Reproduced with a fixed three-record synthetic fixture generated by the real base writer at
21367015cce45294dcf99f54a3211d66f5d842c5; its SHA-256 is5473f0ef178e732be19dcc5fd6cb7c56de964a080bbe1fd7844c1dd4cd66a940.Follow-up
c053027fb2f26c54cc3f5950ebee7021e87a5450separates structural audit reads from strict new-write checks. An uncorrected mixed finding stays visible as an audit statement withmeasurement_errorin its current view; its numbers do not enter best/latest/unknown comparison counts. A matching correction becomes usable without rewriting any earlier byte. Ordinary single and batch writes cannot bypass the check with a prebuilt old row. Explicit ordered-prefix archival replay preserves the original facts, never validation or execution authority. Forged canonical identities remain rejected.The new real-reader regression failed at the first head and passes at this follow-up; source and installed-wheel checks cover two-row uncorrected history, the original three-row corrected history, another successfully appended correction, comparison, and idempotent archival replay. The review's original finding and the initial head remain visible; independent re-review of this follow-up is pending.