Skip to content

Typed evaluation errors with their own codes (422/424/500) - #15

Merged
m-messer merged 2 commits into
mainfrom
feat/typed-evaluation-errors
Oct 20, 2026
Merged

m-messer merged 2 commits into
mainfrom
feat/typed-evaluation-errors

Conversation

@m-messer

Copy link
Copy Markdown
Member

Summary

  • Add EvaluationFunctionError base class; InvalidSubmissionError now subclasses it, plus new UpstreamTimeoutError and UpstreamUnavailableError. All exported from lf_toolkit.
  • Each class sets its JSON-RPC code (rpc_code) and file IO code (file_code):
    • InvalidSubmissionError → 422 / INVALID_SUBMISSION
    • UpstreamTimeoutError → 424 / UPSTREAM_TIMEOUT
    • UpstreamUnavailableError → 424 / UPSTREAM_UNAVAILABLE
    • EvaluationFunctionError → 500 / INTERNAL_ERROR
  • Over JSON-RPC the file code is also sent as the error's data ({"code": ...}), so the two 424s can be told apart. The message reaches the caller as-is, without the "Error calling user handler" prefix.
  • Upstream failures use 424 (Failed Dependency), the same as shimmy, because the function itself is working.
  • Stream server logs no longer print request/response contents or exception messages (these can contain student submissions). They print only sizes, the exception type and the traceback.
  • README: new "Reporting Errors" section.

Test plan

  • New/updated tests in tests/io/ (handler, rpc_handler, file_server, stream_io); 213 passed locally
  • CI green

🤖 Generated with Claude Code

…dardized error classes, and added extensive test coverage. Updated documentation for error reporting.
…to include `data` field, revised tests, and expanded documentation.
@m-messer
m-messer merged commit 8ee1360 into main Oct 20, 2026
2 checks passed
@m-messer
m-messer deleted the feat/typed-evaluation-errors branch October 20, 2026 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant