Skip to content

Jumbodiff to do more decoding - #67

Closed
ckuethe wants to merge 6 commits into
cdump:masterfrom
ckuethe:master
Closed

ckuethe wants to merge 6 commits into
cdump:masterfrom
ckuethe:master

Conversation

@ckuethe

@ckuethe ckuethe commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Sorry about the huge diff, I've been working on this for a while, accumulating lots of little changes.

I finally figured out what the eid=1 messages in the data buffer are; they're a batch of messages like rtdata or doseratedb. I updated my toolkit to take advantage of that. https://github.com/ckuethe/radiacode-tools/blob/main/src/rcrtlog.py

More flags are decoded, so when you're reading the databuf you can tell whether you're getting old records or new ones (here's one of my devices catching up):

{"host_time": "2026-10-31T12:12:10", "message_time": "2026-10-31T11:22:53.376514", "serial": "RC-103G-001371", "type": "DoseRateDB", "count": 319, "count_rate": 3.323, "dose_rate": 4.248e-06, "dose_rate_err": 35.0, "flags": {"value": 45056, "save_to_database": true, "old_data": true, "range_current": false, "power_source": "USB", "temp_status": "Not defined"}}
{"host_time": "2026-10-31T12:12:10", "message_time": "2026-10-31T11:22:53.376514", "serial": "RC-103G-001371", "type": "DoseRateDB", "count": 321, "count_rate": 3.344, "dose_rate": 3.995e-06, "dose_rate_err": 36.1, "flags": {"value": 45056, "save_to_database": true, "old_data": true, "range_current": false, "power_source": "USB", "temp_status": "Not defined"}}
{"host_time": "2026-10-31T12:12:10", "message_time": "2026-10-31T11:22:53.376514", "serial": "RC-103G-001371", "type": "DoseRateDB", "count": 295, "count_rate": 3.073, "dose_rate": 5.086e-06, "dose_rate_err": 33.3, "flags": {"value": 45056, "save_to_database": true, "old_data": true, "range_current": false, "power_source": "USB", "temp_status": "Not defined"}}
{"host_time": "2026-10-31T12:12:10", "message_time": "2026-10-31T11:22:53.376514", "serial": "RC-103G-001371", "type": "DoseRateDB", "count": 305, "count_rate": 3.177, "dose_rate": 4.537e-06, "dose_rate_err": 35.8, "flags": {"value": 45056, "save_to_database": true, "old_data": true, "range_current": false, "power_source": "USB", "temp_status": "Not defined"}}
{"host_time": "2026-10-31T12:12:10", "message_time": "2026-10-31T12:11:12.385346", "serial": "RC-103G-001371", "type": "RealTimeData", "count_rate": 3.133, "count_rate_err": 6.9, "dose_rate": 4.09e-06, "dose_rate_err": 21.6, "flags": {"value": 8256, "save_to_database": false, "old_data": false, "range_current": false, "power_source": "USB", "temp_status": null}, "real_time_flags": {"value": 0, "set": []}}
{"host_time": "2026-10-31T12:12:10", "message_time": "2026-10-31T12:13:19.635346", "serial": "RC-103G-001371", "type": "RawData", "count_rate": 4.0, "dose_rate": 1.846e-05}
{"host_time": "2026-10-31T12:12:10", "message_time": "2026-10-31T12:13:19.635346", "serial": "RC-103G-001371", "type": "RawData", "count_rate": 2.0, "dose_rate": 3.906e-07}
{"host_time": "2026-10-31T12:12:11", "message_time": "2026-10-31T12:13:21.136335", "serial": "RC-103G-001371", "type": "RawData", "count_rate": 0.0, "dose_rate": 0.0}
{"host_time": "2026-10-31T12:12:11", "message_time": "2026-10-31T12:11:14.387111", "serial": "RC-103G-001371", "type": "RealTimeData", "count_rate": 3.113, "count_rate_err": 6.9, "dose_rate": 4.085e-06, "dose_rate_err": 21.6, "flags": {"value": 8256, "save_to_database": false, "old_data": false, "range_current": false, "power_source": "USB", "temp_status": null}, "real_time_flags": {"value": 0, "set": []}}

Figured out the FlashArea, and then I found https://github.com/WeebDataHoarder/PhytonUtils/ who's apparently done some of the same analysis.


Dose Counter
More event details
Multi-part messages id databuf

Add SPEC_RESET/SPEC_DIFF decoders and spectrogram diff accessor

  • SPEC_RESET (0x207) header is ts(I) only (no calibration), unlike SPEC_ACCUM/SPECTRUM which use ts+a0+a1+a2.
  • SPEC_DIFF (0x206) is the per-channel count diff; version 1 decodes as a v1-compressed 1024-channel list, version 2 is an experimental (channel<<4|count) word list.
  • Add spectrum_diff()/spectrum_reset_data() methods and a log-analysis tool.

Fix buffered timestamps: base is per-read host now, drop +128s offset

Buffer ts_offset is in 0.01s units relative to read time. The connect-time base of now+128s pushed timestamps ~128s too late and went stale on long sessions. Now refresh the base to host now on each data_buf() read.

Skip malformed buffer records and resync instead of stopping

  • data-buffer decoder now skips malformed records and re-syncs to the next plausible record (two-record lookahead) instead of discarding the rest of the buffer; multi-sample blocks are pre-validated against the remaining buffer so a corrupt samples_num can't greedily consume the tail
  • seq discontinuities are non-fatal
  • document the malformed-record handling; add missing COMMAND ids (RD_VIRT_STRING_BATCH, WR_VIRT_STRING_BATCH, SET_BT_EXCHANGE_ID) and the verified firmware-descriptor / response layouts from app captures

Add RD_FLASH_AREA decoder and FlashArea descriptor

RD_FLASH_AREA returns the flash region encrypted with a self-seeded LCG keystream (seed*22695477+1, XOR with seed>>16). Add decode_RC_RD_FLASH_AREA plus a FlashArea dataclass and RadiaCode.flash_area() method. Document the wire layout in PROTOCOL.md and correct the earlier note that the area was opaque/undecrypted. BootSignat is a bit-packed date/time, not a timestamp.

Split flags into per-context typed classes; drop non-detector bits

The record 'flags'/'real_time_flags' fields use a different bit table from GET_STATUS, established via baksmali disassembly of cs1 and the log-6 trace. Add RecordFlags (power source, temp state, save-to-db/old-data/range bits) and RealTimeFlags (alarm-sound/background bits); fix status() tuple bug. Remove generic platform bits (SpO2, valves, pumps, pressure) that never apply to a radiation detector from DeviceFlags.

Complete docs; confirm SPEC v1 encoding and drop unobserved v2 SPEC format

  • Docs: add missing public API members and record types to the reference; document spectrum_diff/spectrum_reset_data, DoseCounter, FlashArea and flags.
  • Protocol.md: document sequence base/echo, write-command retcodes, init sequence, extra VS ids, and the SET_LOCAL_TIME + periodic DEVICE_TIME=0 time-sync pattern.
  • SPEC_DIFF confirmed as version-1 run-length across 11 capture logs (incl. a 360-sample plane spectrogram); ts is cumulative live-time and channel 1023 is the overflow bin.
  • Remove the never-observed version-2 SPEC format from the decoder and clamp SpecFormatVersion to the supported {0,1} set.

Decode SET_EXCHANGE interface/connect status; add record alarm-state bits

SET_EXCHANGE now decodes IF_TYPE (1=USB, 2=BLE) and IF_READY (ready/busy) into a SetExchange descriptor; connect raises if the interface is busy. RecordFlags gains dose/count alarm-level bits (0-1, 8-9) and drops the unobserved bit-16 'Range: Current' flag.

Document firmware-update commands; generalize log references

Add FW_IMAGE_GET_INFO/WRITE/UPDATE (0x0012/13/14)

Dose Counter
More event details
Multi-part messages id databuf

Add SPEC_RESET/SPEC_DIFF decoders and spectrogram diff accessor

- SPEC_RESET (0x207) header is ts(I) only (no calibration), unlike
  SPEC_ACCUM/SPECTRUM which use ts+a0+a1+a2.
- SPEC_DIFF (0x206) is the per-channel count diff; version 1 decodes as a
  v1-compressed 1024-channel list, version 2 is an experimental
  (channel<<4|count) word list.
- Add spectrum_diff()/spectrum_reset_data() methods and a log-analysis tool.

Fix buffered timestamps: base is per-read host now, drop +128s offset

Buffer ts_offset is in 0.01s units relative to read time. The connect-time
base of now+128s pushed timestamps ~128s too late and went stale on long
sessions. Now refresh the base to host now on each data_buf() read.

Skip malformed buffer records and resync instead of stopping

- data-buffer decoder now skips malformed records and re-syncs to the next
  plausible record (two-record lookahead) instead of discarding the rest of
  the buffer; multi-sample blocks are pre-validated against the remaining
  buffer so a corrupt samples_num can't greedily consume the tail
- seq discontinuities are non-fatal
- document the malformed-record handling; add missing COMMAND ids
  (RD_VIRT_STRING_BATCH, WR_VIRT_STRING_BATCH, SET_BT_EXCHANGE_ID) and the
  verified firmware-descriptor / response layouts from app captures

Add RD_FLASH_AREA decoder and FlashArea descriptor

RD_FLASH_AREA returns the flash region encrypted with a self-seeded LCG
keystream (seed*22695477+1, XOR with seed>>16). Add decode_RC_RD_FLASH_AREA
plus a FlashArea dataclass and RadiaCode.flash_area() method. Document the
wire layout in PROTOCOL.md and correct the earlier note that the area was
opaque/undecrypted. BootSignat is a bit-packed date/time, not a timestamp.

Split flags into per-context typed classes; drop non-detector bits

The record 'flags'/'real_time_flags' fields use a different bit table from
GET_STATUS, established via baksmali disassembly of cs1 and the log-6 trace.
Add RecordFlags (power source, temp state, save-to-db/old-data/range bits)
and RealTimeFlags (alarm-sound/background bits); fix status() tuple bug.
Remove generic platform bits (SpO2, valves, pumps, pressure) that never apply
to a radiation detector from DeviceFlags.

Complete docs; confirm SPEC v1 encoding and drop unobserved v2 SPEC format

- Docs: add missing public API members and record types to the reference;
  document spectrum_diff/spectrum_reset_data, DoseCounter, FlashArea and flags.
- Protocol.md: document sequence base/echo, write-command retcodes, init
  sequence, extra VS ids, and the SET_LOCAL_TIME + periodic DEVICE_TIME=0
  time-sync pattern.
- SPEC_DIFF confirmed as version-1 run-length across 11 capture logs
  (incl. a 360-sample plane spectrogram); ts is cumulative live-time and
  channel 1023 is the overflow bin.
- Remove the never-observed version-2 SPEC format from the decoder and clamp
  SpecFormatVersion to the supported {0,1} set.

Decode SET_EXCHANGE interface/connect status; add record alarm-state bits

SET_EXCHANGE now decodes IF_TYPE (1=USB, 2=BLE) and IF_READY (ready/busy)
into a SetExchange descriptor; connect raises if the interface is busy.
RecordFlags gains dose/count alarm-level bits (0-1, 8-9) and drops the
unobserved bit-16 'Range: Current' flag.

Document firmware-update commands; generalize log references

Add FW_IMAGE_GET_INFO/WRITE/UPDATE (0x0012/13/14)
@ckuethe

ckuethe commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

Conveniently, the radiacode app was just updated today, which allowed me to capture and analyze the firmware update procedure too. Not that I recommend anyone try it, but hey you do what you want with your own hardware. 🤷🏻‍♂️

@SergeyZh

SergeyZh commented Nov 7, 2026

Copy link
Copy Markdown

Thanks for this PR — the eid=1 multi-sample decoding and the flags table helped me fix a
Home Assistant integration after a firmware update. Some data points from an RC-103 on
firmware 4.15 (target 4.15 "Oct 15 2026 10:18:10", boot 4.0), over BLE:

1. RealTimeData payload is 16 bytes on 4.15, not 15.
After the usual <ffHHHB there is one extra byte (always 0x00 so far). Evidence: the
next record's seq follows immediately after 16 bytes, and with 16 every DATA_BUF read
(43 consecutive reads over ~3.5 min) parses exactly to the end of the buffer; with 15
the parser desyncs right after the first record. Example (RealTime → RawData block):

bf 00 00 a4f24002 | 00c0ce40 8bfd0537 3000 a000 4000 00 | 00 | c0 01 01 e90c4102 7a00 32000000 ...
seq eid gid ts      | cr       dr       cre  dre  flg  rt   | ?? | next record (seq 0xc0)

RealTime is the first record of every read, so a strict parser loses everything after
it (in my case all RareData → temperature/battery 0). Your resync probably recovers, but
_SINGLE_SIZE[0] and the §8 table in PROTOCOL.md describe 4.14 only. I don't know yet
whether the byte is padding or a widened rt_flags. On 4.14 it was 15 bytes as documented.

2. "Old data" records can come after current ones in the same read.
After the device had been disconnected for a while, one DATA_BUF read held the current
records first, followed by replayed history with flags bit 15 set (0x9000 / 0x9040),
including a RareData whose duration was ~2800 s older. So "last record wins" picks stale
values; consumers should skip old_data records (or compare duration/ts_offset).

3. Confirmations on 4.15

  • RareData arrives both as eid=0 and as eid=1 multi-sample blocks (2 samples, 14 B each).
  • RawData comes as eid=1, gid=1 blocks with smpl_time_ms=50.
  • GET_VERSION layout matches your §5; the target date string has a trailing NUL (len 21).

I can share raw DATA_BUF captures if useful.

@ckuethe

ckuethe commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for testing. I don't think I upgraded all of my devices, but I can probably roll back to an older firmware for testing.

one thing I found useful was to turn on debug logs in the android app, then pull the logs from /sdcard/Android/data/com.almacode.radiacode/files/ (or something like that) via ADB. I have script that parses the app logs and its decodes to compare them with the decoders in this library.

@ckuethe

ckuethe commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

I dumped the history from all my devices so I have v4.14 and v4.15 captures. Won't have time to inspect them for a couple of days, but I'll also have time to accumulate side by side measurements over that interval. Once that's done I can compare the traffic from two devices that should have recorded extremely similar measurements.

Comment thread src/radiacode/decoders/databuf.py Resolved
)
return exchange

def status(self) -> DeviceFlags:

@cdump cdump Nov 7, 2026 •

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

breaks the spectrum viewer - show-spectrum.py still calls rc.status().split(':'), update it

next release will required major version bump because interface changed, it's ok.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds like a good opportunity to add a test suite. It's a little bit evil, but https://github.com/ckuethe/radiacode-log-analysis imports the decoders directly to test whether a payload decodes to the same thing as the android app. I think we can do something similar to test that decoders at least return meaningful dataclasses.

Comment thread src/radiacode/decoders/databuf.py Resolved
Comment thread src/radiacode/decoders/databuf.py Resolved
Comment thread examples/specdiff.py
v = struct.unpack_from('<B', data, pos)[0]
pos += 1
elif vlen == 2:
v = struct.unpack_from('<H', data, pos)[0]

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing last +.
But it should be replaced with existing decode_counts_v1 from spectrum.py, which did't have that bug.
It's non production script, so you are free to use not-so-fancy ways to access that function if it's not exported to the public library API, and btw maybe it's better to move this script from examples to docs/

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Speaking of API exports, I'd like to make _batch_read_vsfrs a public function.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have nothing against it

@ckuethe

ckuethe commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks. I've got a day of data from firmware 4.14 and 4.15; will see if I can tighten up those decodes.

edit: bunch of captures from the android app, and a script to feed the captures to the python decoders ... https://github.com/ckuethe/radiacode-log-analysis

Firmware 4.15 appends one byte after rt_flags in gid-0 (RealTimeData) records, growing the body from 15 to 16 bytes; the first 15 bytes keep the same layout. decode_VS_DATA_BUF now auto-detects the body size per buffer (the two sizes are mutually exclusive) and threads it through record sizing/resync/decode, so one decoder handles both firmware versions.
Multi-sample blocks (eid=1) carry a per-sample interval (smpl_time) that the decoder read but ignored, stamping every expanded sample with the group's dt. smpl_time is in the same 10ms units as ts_offset (verified: DoseRateDB blocks tile the single-sample grid exactly at smpl_time*10ms), and samples run oldest->newest from the group timestamp. Timestamp sample i at dt + i*smpl_time*10ms.
The 'off + 7 > n' header guard lets off+7 == n through, so a gid==7 record read data[off+7] == data[n] and raised IndexError. _probe/_detect_realtime_size run outside the decode loop's try/except (and _resync scans raw bytes), so a truncated buffer whose tail decodes as eid=0,gid=7 could crash the decoder. Require off+8 <= n before reading the event byte.
To be revisited later; it does trigger a few times, and recovers a
relatively small number of messages in my test loads
@ckuethe

ckuethe commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

I'm going to try untangle this, and make new PRs.

@ckuethe ckuethe closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants