Repository navigation
plugin: axiomcode-index decodes quoted IR fields, so a literal with quotes is found (#1835) - #1967
Open
swapnilpaliwal-sd wants to merge 1 commit into
Open
swapnilpaliwal-sd wants to merge 1 commit into
swapnilpaliwal-sd wants to merge 1 commit into
Conversation
…uotes is found The parser writes the IR as RFC 4180 over a tab: EntityUtils.escapeTsv turns newlines, CRs and tabs into two-character escapes first, then quotes a field that holds a double quote and doubles the quote. axiomcode-index read it with QUOTE_NONE semantics, so the bundle decoded these fields and the index did not: a literal `Order "A-1" Is Late` was stored with doubled quotes, `impact` on that string said it appears in no literal, `impact KEY` printed 'a \""b\"" c', a Java annotation was stored as @route("""/x"""), and a C# member read on a string literal was named `Length"`. ir_rows() now decodes a field that opens with a quote exactly as the bundle's readRfc4180 does (a doubled quote is one, the closing quote ends the field, a quote elsewhere is text). A line with no quote is still a plain tab split, so the reader costs the same. A quoted field left open at a line end takes the next line, as the bundle's reader does; escapeTsv never writes one. The hand-written decoders that ran on the quoted text go: decorations' unquoted() (Python decorator text starts with @, so it can no longer fire), and the C# literal un-doubling, replaced by cs_string_text(), which reads the C# token: regular, verbatim (a doubled quote is one) and raw (as many quotes close it as open it). On decoded text the old un-doubling read the raw string """"a "" b"""" as "a " b" and left a verbatim string's doubled quotes. C# verbatim and raw strings are now literals. ax_registration's '"""' replace stays: a Python triple-quoted decorator path still needs it. INDEX_VERSION 10. On four measured projects every table but literals, comments and decorations is identical, and those now match an index read with csv QUOTE_MINIMAL row for row. Tests: tests/index_reader.py holds ir_rows to csv.DictReader and to the built bundle reader, with the decoded values written out; tests/cases quoted-literal-text for python, java and csharp, with controls. Refs #1835
Open
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1977
Part of #1835.
Stacked on #1965, which replaced this reader; review that first. This PR adds one commit on top of it.
The parser writes the IR as RFC 4180 over a tab: EntityUtils.escapeTsv turns
newlines, CRs and tabs into two-character escapes first, then quotes a field
that holds a double quote and doubles the quote. axiomcode-index read it with
QUOTE_NONE semantics, so the bundle decoded these fields and the index did not:
a literal
Order "A-1" Is Latewas stored with doubled quotes,impacton that string said it appears in no literal,
impact KEYprinted'a ""b"" c', a Java annotation was stored as @route("""/x"""), and a C#
member read on a string literal was named
Length".ir_rows() now decodes a field that opens with a quote exactly as the bundle's
readRfc4180 does (a doubled quote is one, the closing quote ends the field, a
quote elsewhere is text). A line with no quote is still a plain tab split, so
the reader costs the same. A quoted field left open at a line end takes the
next line, as the bundle's reader does; escapeTsv never writes one.
The hand-written decoders that ran on the quoted text go: decorations'
unquoted() (Python decorator text starts with @, so it can no longer fire), and
the C# literal un-doubling, replaced by cs_string_text(), which reads the C#
token: regular, verbatim (a doubled quote is one) and raw (as many quotes close
it as open it). On decoded text the old un-doubling read the raw string
""""a "" b"""" as "a " b" and left a verbatim string's doubled quotes.
C# verbatim and raw strings are now literals. ax_registration's '"""' replace
stays: a Python triple-quoted decorator path still needs it.
INDEX_VERSION 10. On four measured projects every table but literals,
comments and decorations is identical, and those now match an index read with
csv QUOTE_MINIMAL row for row.
Tests: tests/index_reader.py holds ir_rows to csv.DictReader and to the built
bundle reader, with the decoded values written out; tests/cases
quoted-literal-text for python, java and csharp, with controls.
Results:
tests/index_reader.py: 73/73 (40/73 on the base).tests/cases/*/quoted-literal-text: 20/20; the 9 non-control checks fail on the base.run.py --lang python451/451,--lang java378/380,--lang csharp325/327. The 4 failures are one library-discovery case that fails the same way on the untouched base.fastpath.py: 8/8 for python, java and csharp.