Repository navigation
improve SOFA attribute and string variable handling and testing thereof - #149
Open
f-brinkmann wants to merge 5 commits into
Open
f-brinkmann wants to merge 5 commits into
f-brinkmann wants to merge 5 commits into
Conversation
Open
f-brinkmann
requested review from
a team,
ahms5 and
hoyer-a
and
a balanced review from Copilot
October 24, 2026 16:45
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
UTF-8 byte-array comparisons can raise decoding errors, and custom netCDF string variables lack dimension metadata needed for reliable read/write round trips.
2 open issues
What changed in this PR
This PR addresses #147 by improving UTF-8 handling in SOFA file I/O, writing text as standard-compliant netCDF character data while accepting netCDF string data when reading.
Changes:
- Updates text encoding, decoding, comparison, and byte-length calculation.
- Adds tests for UTF-8 data and netCDF attribute types.
- Raises minimum requirements to Python 3.11 and netCDF4 1.7.5, removes the NumPy upper bound, and updates CI and documentation.
| File | Description |
|---|---|
| tests/test_io.py | Adds encoding and netCDF type checks. |
| sofar/utils.py | Changes string comparisons to Unicode. |
| sofar/sofastream.py | Accepts netCDF string variables during verification. |
| sofar/sofa.py | Calculates string lengths in UTF-8 bytes. |
| sofar/io.py | Updates text serialization and decoding. |
| README.md | Updates the minimum Python version. |
| pyproject.toml | Revises Python and dependency requirements. |
| HISTORY.rst | Documents text handling and support changes. |
| .circleci/config.yml | Removes unsupported Python versions from CI. |
Review effort: Balanced
Comment on lines
+178
to
+179
| if file[var].dtype in ["S1", str]: | ||
| dtype = "string" |
| try: | ||
| assert np.all( | ||
| np.squeeze(a).astype("S") == np.squeeze(b).astype("S")) | ||
| np.squeeze(a).astype("U") == np.squeeze(b).astype("U")) |
hoyer-a
approved these changes
Oct 29, 2026
hoyer-a
left a comment
hoyer-a
left a comment
Member
There was a problem hiding this comment.
As far as i can judge, this looks good!
Co-authored-by: Anton Hoyer <156099087+hoyer-a@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue(s) are closed by this pull request?
Closes #147 and fixes #140. Also fixes failing tests in #148
Changes proposed in this pull request:
Improved writing of SOFA attributes and string variables to be fully conform with the SOFA standard
chararray as required by the SOFA standardsofar.io._format_value_for/from_netcdfand the use of the netCDF functionsstringtocharandchartostringthat caused the error in the first place is now avoided.stringto be user friendly and backwards compatibleAll changes are fully backwards compatible because they only changed how data are handled internally in
SofaandSofaStream.AI disclaimer:
The fixes were done by an LMM. I checked the SOFA standard before hand. I discussed and instructed the LMM with care, and checked all output.