Skip to content

plugin: axiomcode-index decodes quoted IR fields, so a literal with quotes is found (#1835) - #1967

Open
swapnilpaliwal-sd wants to merge 1 commit into
perf/perf-staging-indexfrom
fix/index-decode-quoted
Open

swapnilpaliwal-sd wants to merge 1 commit into
perf/perf-staging-indexfrom
fix/index-decode-quoted

Conversation

@swapnilpaliwal-sd

@swapnilpaliwal-sd swapnilpaliwal-sd commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Closes #1977

Part of #1835.

Stacked on #1965, which replaced this reader; review that first. This PR adds one commit on top of it.

The parser writes the IR as RFC 4180 over a tab: EntityUtils.escapeTsv turns
newlines, CRs and tabs into two-character escapes first, then quotes a field
that holds a double quote and doubles the quote. axiomcode-index read it with
QUOTE_NONE semantics, so the bundle decoded these fields and the index did not:
a literal Order "A-1" Is Late was stored with doubled quotes, impact
on that string said it appears in no literal, impact KEY printed
'a ""b"" c', a Java annotation was stored as @route("""/x"""), and a C#
member read on a string literal was named Length".

ir_rows() now decodes a field that opens with a quote exactly as the bundle's
readRfc4180 does (a doubled quote is one, the closing quote ends the field, a
quote elsewhere is text). A line with no quote is still a plain tab split, so
the reader costs the same. A quoted field left open at a line end takes the
next line, as the bundle's reader does; escapeTsv never writes one.

The hand-written decoders that ran on the quoted text go: decorations'
unquoted() (Python decorator text starts with @, so it can no longer fire), and
the C# literal un-doubling, replaced by cs_string_text(), which reads the C#
token: regular, verbatim (a doubled quote is one) and raw (as many quotes close
it as open it). On decoded text the old un-doubling read the raw string
""""a "" b"""" as "a " b" and left a verbatim string's doubled quotes.
C# verbatim and raw strings are now literals. ax_registration's '"""' replace
stays: a Python triple-quoted decorator path still needs it.

INDEX_VERSION 10. On four measured projects every table but literals,
comments and decorations is identical, and those now match an index read with
csv QUOTE_MINIMAL row for row.

Tests: tests/index_reader.py holds ir_rows to csv.DictReader and to the built
bundle reader, with the decoded values written out; tests/cases
quoted-literal-text for python, java and csharp, with controls.

Results:

  • tests/index_reader.py: 73/73 (40/73 on the base).
  • tests/cases/*/quoted-literal-text: 20/20; the 9 non-control checks fail on the base.
  • run.py --lang python 451/451, --lang java 378/380, --lang csharp 325/327. The 4 failures are one library-discovery case that fails the same way on the untouched base.
  • fastpath.py: 8/8 for python, java and csharp.

@swapnilpaliwal-sd swapnilpaliwal-sd added the bug Something isn't working label Oct 15, 2026
@swapnilpaliwal-sd swapnilpaliwal-sd added the plugin the axiomcode plugin: skill + MCP layer label Oct 15, 2026
@swapnilpaliwal-sd swapnilpaliwal-sd added bug Something isn't working python Python java Java plugin the axiomcode plugin: skill + MCP layer csharp C# labels Oct 15, 2026
…uotes is found

The parser writes the IR as RFC 4180 over a tab: EntityUtils.escapeTsv turns
newlines, CRs and tabs into two-character escapes first, then quotes a field
that holds a double quote and doubles the quote. axiomcode-index read it with
QUOTE_NONE semantics, so the bundle decoded these fields and the index did not:
a literal `Order "A-1" Is Late` was stored with doubled quotes, `impact`
on that string said it appears in no literal, `impact KEY` printed
'a \""b\"" c', a Java annotation was stored as @route("""/x"""), and a C#
member read on a string literal was named `Length"`.

ir_rows() now decodes a field that opens with a quote exactly as the bundle's
readRfc4180 does (a doubled quote is one, the closing quote ends the field, a
quote elsewhere is text). A line with no quote is still a plain tab split, so
the reader costs the same. A quoted field left open at a line end takes the
next line, as the bundle's reader does; escapeTsv never writes one.

The hand-written decoders that ran on the quoted text go: decorations'
unquoted() (Python decorator text starts with @, so it can no longer fire), and
the C# literal un-doubling, replaced by cs_string_text(), which reads the C#
token: regular, verbatim (a doubled quote is one) and raw (as many quotes close
it as open it). On decoded text the old un-doubling read the raw string
""""a "" b"""" as "a " b" and left a verbatim string's doubled quotes.
C# verbatim and raw strings are now literals. ax_registration's '"""' replace
stays: a Python triple-quoted decorator path still needs it.

INDEX_VERSION 10. On four measured projects every table but literals,
comments and decorations is identical, and those now match an index read with
csv QUOTE_MINIMAL row for row.

Tests: tests/index_reader.py holds ir_rows to csv.DictReader and to the built
bundle reader, with the decoded values written out; tests/cases
quoted-literal-text for python, java and csharp, with controls.

Refs #1835
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working csharp C# java Java plugin the axiomcode plugin: skill + MCP layer python Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant