Skip to content

GH-1337: Skip unknown fields in FlightData parsing (ArrowMessage.frame) - #1338

Open
laurentgo wants to merge 1 commit into
apache:mainfrom
laurentgo:claude/arrow-java-unknown-fields-74336a
Open

laurentgo wants to merge 1 commit into
apache:mainfrom
laurentgo:claude/arrow-java-unknown-fields-74336a

Conversation

@laurentgo

Copy link
Copy Markdown
Contributor

What's Changed

ArrowMessage.frame() ignored unknown fields by reading their tag but not their payload, so the payload was parsed as the next tag. The default case now skips the value according to its wire type, as protobuf requires:

  • varint: up to 10 bytes (64-bit values);
  • fixed64 / fixed32: 8 / 4 bytes;
  • length-delimited: the length is read with the existing readFieldLength (which keeps the KnownLength bounds check), then skipped without allocating; short InputStream.skip results are handled and EOF is an error;
  • groups: skipped recursively up to a nesting limit (default 100, like protobuf-java's recursion limit; override with -Darrow.flight.max_protobuf_skip_group_depth or ARROW_FLIGHT_MAX_PROTOBUF_SKIP_GROUP_DEPTH), with the end-group field number checked against the start-group;
  • invalid wire types and malformed groups: IOException("Malformed FlightData frame: ...").

A known field number sent with a different wire type than expected is treated as an unknown field and skipped too, as protobuf does. The zero-copy read path for known fields is unchanged. This follows apache/arrow#26885 (ARROW-10962), which aligned the parser with protobuf semantics for missing fields.

Tests are in TestArrowMessage: unknown varint (10, 18, 26, 10-byte), length-delimited (0, 10, 18, 26 with tag-like bytes), fixed64, fixed32, groups and nested groups, and known field numbers with wrong wire types, each placed before, between and after the known fields on both known-length and compressed streams. Also covered: the group depth limit, malformed or unterminated groups, invalid wire types and truncated fields. The new tests fail without the fix.

Closes #1337.

The code and the tests were drafted with Claude Code; the author has reviewed both.

…e.frame)

ArrowMessage.frame() ignored unknown fields by reading their tag but not
their payload, so the payload was parsed as the next tag. Skip the value
according to its wire type instead, as protobuf requires: varints, fixed32,
fixed64, length-delimited fields and groups (recursively, up to a
configurable nesting depth).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@laurentgo laurentgo added the bug-fix PRs that fix a big. label Oct 11, 2026
@github-actions github-actions Bot added this to the 20.0.0 milestone Oct 11, 2026
// ignore unknown fields.
// Anything else is an unknown field, including a known field number sent with another
// wire type than expected. As in protobuf, it is skipped, not rejected.
skipField(stream, tag);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe a tag of 0x00 (field number 0) ends up here. Its wire type is VARINT, so skipField consumes the next byte as its value.

protobuf rejects field number 0, so we could do the same at the top of skipField:

private static void skipField(InputStream stream, int tag, int depth) throws IOException {
  if (WireFormat.getTagFieldNumber(tag) == 0) {
    throw new IOException("Malformed FlightData frame: invalid tag " + tag);
  }
  switch (WireFormat.getTagWireType(tag)) {

The tag == 0 check in skipGroup can then be removed.

}
try {
final int depth = Integer.parseInt(value.trim());
if (depth >= 0) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should set an upper bound here, and skip group which is recursive (skipField -> skipGroup -> skipField). With a large setting (like -Darrow.flight.max_protobuf_skip_group_depth=10000000), a peer can send a deeply nested message and overflow the stack.

frame() only catches Exception, so the StackOverFlowError skips the cleanup: appMetadata and body are never released and the allocator leaks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug-fix PRs that fix a big.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ArrowMessage.frame() does not skip unknown FlightData fields

3 participants