Repository navigation
Conversation
When Unicode support was first added to Python, there was no Unicode property identifying whitespace, so we approximated it by putting together a couple of other properties. Now there is a White_Space property, so let's use it. Happily, the difference from our original approximation is only in the four rare control characters 001C..001F. As a bonus, `isspace` now joins all similar methods in giving exactly matching results for ASCII characters represented as `str` or as `bytes`. Add a test for that nice property.
Numerlor
left a comment
Numerlor
left a comment
There was a problem hiding this comment.
Ran into an issue with this when I noticed int's stripping is different to str's, so it's nice to see that there's already a PR for a fix.
The introduced changes seem good to my eye that's not entirely familiar with the codebase, apart from the outdated version notices.
I've also noticed that the added docstrings to test_unicode are using single quote marks while others are using double quotes so changing that to be consistent would be nice
| ((bidirectional in ('WS', 'B', 'S') | ||
| or category == 'Zs') | ||
| and codepoint not in range(0x1c, 0x20))) |
There was a problem hiding this comment.
Is this guaranteed to hold in future unicode versions? i.e. should it be tested on the string method if it's only testing the unicode database parsing and generation?
|
This PR is stale because it has been open for 30 days with no activity. |
serhiy-storchaka
changed the title
github-actions
Bot
removed
the
stale
Resolve conflicts by taking upstream versions of the regenerated Objects/unicodetype_db.h and test checksum, and by adapting the PropList parsing hunk to the current makeunicodedata.py structure. Co-authored-by: Muse Code (Meta Muse Spark)
Regenerate Objects/unicodetype_db.h at Unicode 18.0.0 with the White_Space property (25 whitespace code points; U+001C..U+001F no longer whitespace). Retarget the docs versionchanged and test notes from 3.9 to 3.16, rename the NEWS entry to the current gh-issue convention, and update the unicodedata method checksums for the intended behavior change. Co-authored-by: Muse Code (Meta Muse Spark)
Documentation build overview
5 files changed ·
|
Required by the blurb-no-space-core-and-builtins lint hook. Co-authored-by: Muse Code (Meta Muse Spark)
|
This needs a decision since the behavior change is wider than the docs and NEWS say: everything built on We'd also have to drop U+001C–U+001E from splitlines() in a separate sister change. |
|
@malemburg, what do you think? Those characters are used as delimiters in real-world data like RFC 7464 JSON text sequences, GS1 barcoes, MARC 21 library records, HL7 MLLP framing, probably more. Even if we add a "Porting to 3.16" section and explain the difference, silent breakage of existing software is a real risk. |
When Unicode support was first added to Python, there was no Unicode
property identifying whitespace, so we approximated it by putting
together a couple of other properties.
Now there is a White_Space property, so let's use it.
Happily, the difference from our original approximation is only in the
four rare control characters 001C..001F.
As a bonus,
isspacenow joins all similar methods in giving exactlymatching results for ASCII characters represented as
stror asbytes. Add a test for that nice property.https://bugs.python.org/issue18236