Skip to content

Fix byte-array projections from JSON collections - #4067

Merged
roji merged 3 commits into
mainfrom
roji-fix-json-byte-array-projection
Oct 22, 2026
Merged

roji merged 3 commits into
mainfrom
roji-fix-json-byte-array-projection

Conversation

@roji

@roji roji commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Fixes #3566.

json_to_recordset/jsonb_to_recordset do not decode the base64 representation used for byte[] properties inside JSON. Collection expansion therefore returned the base64 text's bytes, even though full-document materialization and indexed property access returned the original payload.

Related upstream fix: dotnet/efcore#33440, which fixed dotnet/efcore#33435. That addressed the same base64-versus-binary-conversion mismatch in SQL Server's scalar JSON access by rewriting JSON_VALUE to OPENJSON ... WITH. This PR addresses PostgreSQL's separate collection-recordset path; Npgsql's scalar JSON decoding already works. The late JSON postprocessing approach follows the same upstream precedent while retaining flat PostgreSQL SQL.

Extract affected recordset columns as text and wrap their references in decode(..., 'base64'). This is applied after structural projection expansion, which requires column expressions, and avoids adding SQL subqueries. Existing scalar JSON decoding is unchanged.

Add sync/async regression coverage using complex JSON collection mapping in the shared JSON translation fixture for both json and jsonb, comparing full materialization with projected bytes and checking server-side binary equality. Include non-ASCII bytes, empty arrays, and null values.

Confirmed with a standalone console repro on released providers 8.0.0, 9.0.4, 10.0.0, 10.0.3, and unmodified main. The equivalent collection projection works with SQL Server 10.0.0. The original repro returns matching bytes after the fix. Manual checks also confirmed owned and complex JSON mapping.

Validation: dotnet test --project test/EFCore.PG.FunctionalTests --filter-class '*Json*' — 1,757 passed, 9 skipped, using an isolated PostgreSQL Testcontainer. After switching the regression to complex mapping, the affected JsonTranslationsNpgsqlTest suite passed all 12 cases.

This is a bug fix targeting main (11.0), not a new JSON feature or a serialization-format change. No backports are included.

Decode base64 text extracted by json_to_recordset and jsonb_to_recordset before using binary properties in projections or predicates. Add sync/async regression coverage for json and jsonb, including null and empty payloads.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: dbf67848-d1a3-45a1-a202-1b7161746a53
Copilot AI balanced review requested due to automatic review settings October 21, 2026 15:11
github-code-quality[bot]

This comment was marked as outdated.

This comment was marked as outdated.

roji

This comment was marked as outdated.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: dbf67848-d1a3-45a1-a202-1b7161746a53
Copilot AI balanced review requested due to automatic review settings October 22, 2026 12:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The targeted rewrite and regression coverage address the reported inconsistency, with no blocking issues identified.

No open issues 🎉

Review effort: Balanced

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: dbf67848-d1a3-45a1-a202-1b7161746a53
Copilot AI balanced review requested due to automatic review settings October 22, 2026 12:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused rewrite and regression coverage address the reported mismatch with no unresolved blocking findings; tests were not independently rerun.

No open issues 🎉

Review effort: Balanced

@roji
roji merged commit c1ac3e1 into main Oct 22, 2026
14 checks passed
@roji
roji deleted the roji-fix-json-byte-array-projection branch October 22, 2026 12:35
roji added a commit that referenced this pull request Oct 22, 2026
Backport the complete merged change from #4067 (c1ac3e1), adapting its complex JSON regression to the EF10 ad-hoc JSON suite.

Fixes #3566

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 187745a9-eff9-41fb-8678-a531666c27f6
roji added a commit that referenced this pull request Oct 22, 2026
Backport #4067, retaining the late JSON postprocessor and its comments. Adapt regression coverage to EF9 owned ToJson collection mapping for json/jsonb, sync/async, null/empty payloads, and binary equality.

Fixes #3566

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f9d39831-0c6c-4de0-9216-259844517ec4
roji added a commit that referenced this pull request Oct 22, 2026
Backport #4067 using EF8 table identities and owning selects to decode base64 recordset fields without wrapper subqueries. Add owned-JSON regression coverage, including correlated references.

Fixes #3566

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c8a1b67d-06d1-4128-b3e6-3679f8634267
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants