Byte array in owned entity mapped to JSON returns different value when using projection

Hi, I encountered a problem while working with a byte array in an owned entity mapped to JSON - it looks like the materialization behaves differently when getting the whole entity vs getting only the specific property.
I would like to use this for encrypted columns, below is a simplified example.

Looks similar to dotnet/efcore#33435.

Sample Code

using Microsoft.EntityFrameworkCore;

await using var context = new BloggingContext();
await context.Database.EnsureCreatedAsync();
var blog = new Blog
{
    Posts =
    [
        new Post { Content = "This is the content of the first post."u8.ToArray() }
    ]
};


await context.Blogs.AddAsync(blog);
await context.SaveChangesAsync();

var blogs1 = await context.Blogs.ToListAsync(); 
var blogs2 = await context.Blogs
    .SelectMany(b => b.Posts.Select(p => p.Content))
    .ToListAsync();

var content1 = blogs1[0].Posts[0].Content; 
var content2 = blogs2[0];
var decodedViaSql = await context.Database
    .SqlQueryRaw<byte[]>( "SELECT decode((b.\"Posts\"->0->>'Content'), 'base64') AS \"Value\" FROM \"Blogs\" AS b LIMIT 1")
    .FirstAsync();

Console.WriteLine($"content1 len={content1.Length}, bytes={string.Join(",", content1)}"); 
Console.WriteLine($"content2 len={content2.Length}, bytes={string.Join(",", content2)}"); 
Console.WriteLine($"decodedViaSql len={decodedViaSql.Length}, bytes={string.Join(",", decodedViaSql)}"); 
Console.WriteLine("content1 equals content2: " + content1.SequenceEqual(content2)); 
Console.WriteLine("content1 equals decodedViaSql: " + content1.SequenceEqual(decodedViaSql));
Console.WriteLine("content2 equals decodedViaSql: " + content2.SequenceEqual(decodedViaSql));

public class BloggingContext : DbContext
{
    public DbSet<Blog> Blogs => Set<Blog>();

    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        => optionsBuilder.UseNpgsql("Host=localhost;Port=5432;Username=postgres;Password=password;Database=demo");

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<Blog>()
            .OwnsMany<Post>(b => b.Posts, b =>
            {
                b.ToJson();
                b.WithOwner();
            });
    }
}

public class Blog
{
    public int BlogId { get; set; }
    public List<Post> Posts { get; set; } = [];
}

public class Post
{
    public required byte[] Content { get; set; }
}

Thanks for looking into this.

Versions:
.Net: 9, 10
Npgsql.EntityFrameworkCore.PostgreSQL: v9.0.4, v10.0.0

Thanks for the minimal repro - I can indeed see the bug happening. It also indeed seems PG-specific (does not repro with SQL Server). I'll try to take a look at this soon.

@roji

Hi @roji, just a heads up that this still reproduces in EfCore.Pg v10.

Looking into this closer, I adapted the repro code a bit to see what might be happening under the hood and this is my conclusion:

Full entity materialization correctly yields the base64-decoded payload.
Projection returns the UTF-8 bytes of the base64 string itself, i.e., no decode is happening in the projection path.

AI Triage

The below is an AI-generated analysis and may contain inaccuracies.

Confirmed as a PostgreSQL provider bug in the collection-expansion path. Using the reported OwnsMany(...).ToJson() model, full entity materialization returns the original bytes, but SelectMany(b => b.Posts.Select(p => p.Content)) returns the UTF-8 bytes of their base64 representation.

Reproduced with Npgsql.EntityFrameworkCore.PostgreSQL 8.0.0, 9.0.4, 10.0.0, 10.0.3, and current main (3931c57). This is not a new 9.x/10.x regression: the same collection query already fails in 8.0.0. The equivalent SelectMany query returns the correct bytes with SQL Server 10.0.0, confirming provider specificity for the reported query.

The failing SQL is:

SELECT p."Content"
FROM "Blogs" AS b
JOIN LATERAL ROWS FROM (
    jsonb_to_recordset(b."Posts") AS ("Content" bytea)
) WITH ORDINALITY AS p ON TRUE

jsonb_to_recordset converts the string to PostgreSQL bytea, but does not decode base64. The provider's existing VisitJsonScalar base64 handling is not used by these recordset column references. Direct indexed access (b.Posts[0].Content) already generates decode(..., 'base64') and works with Npgsql. A fix needs to extract the recordset field as text and explicitly decode it, including when used in server-side predicates.

Related upstream reports are dotnet/efcore#33435, fixed for SQL Server by dotnet/efcore#33440, and dotnet/efcore#33522 for SQLite. These concern analogous provider-specific conversions, not an unresolved EF Core issue blocking this fix. No duplicate Npgsql issue or implementing PR was found. The existing Bug issue type is appropriate.

minimal repro

Use Npgsql.EntityFrameworkCore.PostgreSQL 10.0.3 and Testcontainers.PostgreSql 4.15.0 in a console project with a running Docker-compatible runtime:

using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Logging;
using Testcontainers.PostgreSql;

await using var postgres = new PostgreSqlBuilder("postgres:17").Build();
await postgres.StartAsync();
Environment.SetEnvironmentVariable("REPRO_CONNECTION", postgres.GetConnectionString());

await using var context = new BloggingContext();
await context.Database.EnsureCreatedAsync();
context.Blogs.Add(new Blog
{
    Posts = [new Post { Content = "This is the content of the first post."u8.ToArray() }]
});
await context.SaveChangesAsync();
context.ChangeTracker.Clear();

var blog = await context.Blogs.AsNoTracking().SingleAsync();
var content = await context.Blogs.SelectMany(b => b.Posts.Select(p => p.Content)).SingleAsync();
Console.WriteLine(Convert.ToHexString(blog.Posts[0].Content));
Console.WriteLine(Convert.ToHexString(content));
Console.WriteLine(blog.Posts[0].Content.SequenceEqual(content)); // False

public class BloggingContext : DbContext
{
    public DbSet<Blog> Blogs => Set<Blog>();

    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        => optionsBuilder
            .UseNpgsql(Environment.GetEnvironmentVariable("REPRO_CONNECTION"))
            .LogTo(Console.WriteLine, LogLevel.Information)
            .EnableSensitiveDataLogging();

    protected override void OnModelCreating(ModelBuilder modelBuilder)
        => modelBuilder.Entity<Blog>().OwnsMany(b => b.Posts).ToJson();
}

public class Blog
{
    public int BlogId { get; set; }
    public List<Post> Posts { get; set; } = [];
}

public class Post
{
    public required byte[] Content { get; set; }
}

[10.0] Fix byte-array projections from JSON collections (#4073)

[9.0] Fix byte-array projections from JSON collections (#4074)

[8.0] Fix byte-array projections from JSON collections (#4075)

Navigation Menu

Repository navigation

Byte array in owned entity mapped to JSON returns different value when using projection #3566

Sample Code

AI Triage

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Byte array in owned entity mapped to JSON returns different value when using projection #3566

Description

Sample Code

Activity

roji commented on Jul 4, 2025

drapka commented on Dec 10, 2025

roji commented on Oct 21, 2026

AI Triage

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions