Skip to content

fix: Python 3 leftovers in protobuf_json and keepkeyctl - #231

Open
pattems wants to merge 2 commits into
keepkey:masterfrom
pattems:py313-fix-protobuf-json-make-the-module-import
Open

pattems wants to merge 2 commits into
keepkey:masterfrom
pattems:py313-fix-protobuf-json-make-the-module-import

Conversation

@pattems

@pattems pattems commented Oct 28, 2026

Copy link
Copy Markdown

Two commits that remove Python 2 only code that still fails on Python 3.

1. fix(protobuf_json): make the module importable and usable on Python 3

keepkeylib/protobuf_json.py still contains Python 2 only code, so on any Python 3:

>>> import keepkeylib.protobuf_json
ModuleNotFoundError: No module named 'types_pb2'

and once that is fixed, the long / unicode names in the type tables raise NameError. With protobuf 7 the field.label == FD.LABEL_REPEATED checks also fail, because FieldDescriptor.label was removed (only is_repeated is left).

This commit:

  • imports types_pb2 relative to the package (from . import types_pb2 as types),
  • uses int / str instead of long / unicode,
  • decodes the hexlified bytes in pb2json() so its result can go straight into json.dumps() (json2pb() already accepts str for unhexlify),
  • adds a small _is_repeated() helper that uses FieldDescriptor.is_repeated and falls back to label == LABEL_REPEATED on older protobuf, so protobuf>=3.0.0 from setup.py keeps working.

Regenerating the bundled *_pb2.py files for protobuf 7 is separate work and not part of this PR.

2. fix(keepkeyctl): replace Python 2 leftovers in command handlers

Several keepkeyctl commands fail or print garbage on Python 3:

  • ethereum_sign_tx and nano_sign_tx use str.decode('hex') / bytes.encode('hex') and raise AttributeError: 'str' object has no attribute 'decode'. Now bytes.fromhex() / bytes.hex().
  • firmware_update --url calls urllib.urlretrieve / urllib.urlcleanup (AttributeError: module 'urllib' has no attribute 'urlretrieve'), opens the download in text mode and compares the first bytes against a str. Now urllib.request.*, binary mode, and a bytes comparison.
  • ethereum_get_address, ethereum_sign_tx, eos_get_public_key, get_entropy and encrypt_keyvalue put binascii.hexlify() output (bytes) into strings, giving 0xb'ab...'. In ethereum_sign_tx that broken string is also the from address for the JSON-RPC calls and the raw transaction passed to eth_sendRawTransaction. With -j, the bare bytes results make json.dumps() fail. Now bytes.hex(). sign_message returned base64 bytes the same way; they are now decoded.
  • The dead Python 2 branches in encrypt_keyvalue / decrypt_keyvalue are removed.
  • decrypt_keyvalue returned the decrypted bytes (printed as b'...'; -j raised TypeError). It now
    returns the UTF-8 text that encrypt_keyvalue stored, or hex for anything that is not UTF-8.

tests/unit/test_keepkeyctl.py loads the script and runs these handlers against a mocked client (no device). ethjsonrpc and rlp are stubbed in sys.modules, so the test needs no extra dependencies.

Testing

protobuf 7.36.2, run from the repo root with PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python, which the current bundled pb2 files need on protobuf >= 4.

protobuf_json: round trip pb2json() -> json.dumps() -> json.loads() -> json2pb() of a TxInputType with repeated uint32, uint64, bytes, enum, a nested message, and repeated bytes. The test asserts that the result equals the original message.

keepkeyctl: PYTHONPATH=. python -B tests/unit/test_keepkeyctl.py (8 tests). I ran it with unittest directly because tests/conftest.py imports requests.

Python protobuf_json on master protobuf_json with this PR test_keepkeyctl on master test_keepkeyctl with this PR
3.12.13 ModuleNotFoundError: types_pb2 OK 5 failures, 3 errors 8 OK
3.13.14 ModuleNotFoundError: types_pb2 OK 5 failures, 3 errors 8 OK
3.14.3 not run OK 5 failures, 3 errors 8 OK

The protobuf_json round trip also passes on 3.12.13 with protobuf 3.20.3, which has no is_repeated and so uses the label fallback.

keepkeylib.protobuf_json still had Python 2 only code: a bare
"import types_pb2" (an implicit relative import, so the module fails
to import on Python 3) and the long/unicode builtins. Use a package
relative import and int/str, and decode hexlified bytes so pb2json()
output can be passed to json.dumps().

FieldDescriptor.label was removed in protobuf 7; use
FieldDescriptor.is_repeated when available and fall back to
label == LABEL_REPEATED on older protobuf.
Several keepkeyctl commands still used Python 2 only idioms and fail
or print garbage on Python 3:

- ethereum_sign_tx and nano_sign_tx used str.decode('hex') and
  bytes.encode('hex'), which raise AttributeError; use bytes.fromhex()
  and bytes.hex().
- firmware_update --url called urllib.urlretrieve/urlcleanup, which
  only exist as urllib.request.* on Python 3, opened the download in
  text mode and compared the first bytes against a str; use
  urllib.request, open it in binary mode and compare against bytes.
- ethereum_get_address, ethereum_sign_tx, eos_get_public_key,
  get_entropy and encrypt_keyvalue formatted binascii.hexlify() output
  (bytes) into strings, producing "0xb'...'" (also in the address used
  for the JSON-RPC calls and in the published raw transaction) or bytes
  that json.dumps() rejects with -j; use bytes.hex(). sign_message
  likewise returned base64 bytes; decode them, and decrypt_keyvalue
  returned the decrypted bytes (printed as b'...', TypeError with -j);
  return the UTF-8 text encrypt_keyvalue stored, or hex otherwise.
- Drop the dead Python 2 branches in encrypt_keyvalue/decrypt_keyvalue.

Add unit tests that run these handlers against a mocked client.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant