[jiangLLM]

Description

PrefEval can count a failed judge call as a personalized response. It also misses lowercase answers and some responses that are both unhelpful and against the user's preference.

This PR puts failed or invalid answers in a separate Judge Failure category. It accepts case and punctuation differences, and gives Unhelpful Response priority. Answers must start with a whole Yes or No word, so Not sure and Yesterday remain judge failures. All 16 standard Yes/No combinations match the reference benchmark.

The Excel report includes Judge Failure, so failed samples remain visible in the totals. The script also loads its local utils package before any unrelated package with the same name.

Fixes #2396.

Type of change

• Bug fix

How Has This Been Tested?

• python -m pytest tests/evaluation -q: 49 passed.
• The six new cases fail before these review fixes and pass after them. They cover a conflicting utils module or package, plus Excel reports with mixed results, no failures, only failures, and no samples. The export tests check values, percentage formats, and the time columns.
• make format and pre-commit checks pass for the PR files.

Checklist

• Reviewed the diff and added regression tests.
• Linked the related issue.
• No separate documentation update is needed for this bug fix.

[github-actions]

This PR has been automatically marked as stale due to inactivity.

[Memtensor-AI]

🤖 Open Code Review

Target: PR #2397
Task: ce861921c69f08fa
Base: main
Head: fix/prefeval-error-type-classifier

🔍 OpenCodeReview found 4 issue(s) in this PR.

---

1. evaluation/scripts/utils/pref_classify.py (L53-L54)

hallucinate_preference is demanded well-formed even when it is already irrelevant. Line 58 discards the hallucination answer whenever acknowledge_preference is No, but the any(... is None) guard above fires first. So if the acknowledge judge says "No" and the (then meaningless) hallucination call fails or returns no XML (API error → ""), a sample that could still be classified as Preference-Unaware Violation / Unhelpful Response / Personalized Response is dropped into Judge Failure, inflating that bucket and shrinking the others. Consider validating only the keys the branch actually consumes.

💡 Suggested Change

Before:

    if any(value is None for value in answers.values()):
        return JUDGE_FAILURE

After:

    required = ("violate_preference", "acknowledge_preference", "helpful_response")
    if any(answers[key] is None for key in required):
        return JUDGE_FAILURE
    if answers["acknowledge_preference"] and answers["hallucinate_preference"] is None:
        return JUDGE_FAILURE

---

2. evaluation/scripts/utils/pref_classify.py (L35)

Only trailing punctuation/explanation is tolerated, but the judge prompt literally asks for <answer>[Yes/No]</answer>, so leading decoration ([Yes], **Yes**, "Yes") is a realistic output from gpt-4o-mini and would be routed to JUDGE_FAILURE even though the answer is perfectly clear. Skipping leading non-word characters makes the parser robust to that without weakening the no/not sure rejection (the \b still blocks not, and re.match stays anchored).

💡 Suggested Change

Before:

    match = re.match(r"(yes|no)\b", answer.strip(), re.IGNORECASE)

After:

    match = re.match(r"[\W_]*(yes|no)\b", answer.strip(), re.IGNORECASE)

---

3. evaluation/scripts/PrefEval/pref_eval.py (L258)

Inconsistent keying of the summary buckets: the new module exports constants for all six categories, but only JUDGE_FAILURE is used here while the other five are still literal strings. If a constant is reworded, get_pct silently yields 0 (.get(key, {}).get("percentage", 0)) and the Excel report shows 0% instead of erroring. Use the shared constants for every bucket and import them together with JUDGE_FAILURE.

💡 Suggested Change

Before:

    judge_failure_pct = get_pct(JUDGE_FAILURE)

After:

    unaware_pct = get_pct(PREFERENCE_UNAWARE_VIOLATION)
    hallucination_pct = get_pct(PREFERENCE_HALLUCINATION_VIOLATION)
    inconsistency_pct = get_pct(INCONSISTENCY_VIOLATION)
    unhelpful_pct = get_pct(UNHELPFUL_RESPONSE)
    personalized_pct = get_pct(PERSONALIZED_RESPONSE)
    judge_failure_pct = get_pct(JUDGE_FAILURE)

---

4. evaluation/scripts/PrefEval/pref_eval.py (L285)

xlsxwriter replaces the column entry on every set_column call, so the three calls below (B:G, H:H, I:J) reset this wrap_format for columns B–J — it now only survives on column A. Net effect: the percentage/float/time headers lose text_wrap, while the row height is still forced to 45 (set_row(0, 45)), so the long bilingual headers such as "Inconsistency\n意识到偏好但给出了不一致的回答" get clipped into a single 18-char line. Previously the A:I wrap call was last and did apply to the headers. Consider folding text_wrap/alignment into pct_format and float_format (and giving I:J a format) instead of relying on a preceding set_column.

Generated by cloud-assistant via Open Code Review.

[Memtensor-AI]

✅ Automated Test Results: PASSED

All tests passed (43/43 executed). memos_python_core/changed-repo-python: 43/43. Duration: 1s [advisory, non-gating] AI-generated tests on branch test/auto-gen-bf846b37cf479ccd-20261025023031: 128/128 passed — these do NOT affect the PR verdict; review the branch manually.

Branch: fix/prefeval-error-type-classifier

[Memtensor-AI]

✅ Automated Test Results: PASSED

All tests passed (43/43 executed, 6 skipped). memos_python_core/changed-repo-python: 43 passed, 6 skipped. Duration: 1s

Branch: fix/prefeval-error-type-classifier