Data retrieval tool with socketcan forwarding

[pavel-kirienko]

We have a data ingestion tool, now we want a retrieval tool.

⚠️ THIS IS SUBJECT TO REVIEW/IMPROVEMENT BASED ON INITIAL FEEDBACK

The idea is simple: the user says ./letopisec_fetch.py --server=... and gets a list of all device tags, and for each device the oldest (min seqno) and the newest (max seqno) record, everything is neatly JSON-formatted with pretty indentation and coloring if stdout is a tty, otherwise compact.

Invoking ./letopisec_fetch.py --server=... --device_tag=... (i.e., when the tag is given), the user gets the records printed in seqno order from min to max in JSON lines format, one record per line; the tool should request them with default response size limit (1000 records) and print them out as they arrive (since there may be many records, we don't use JSON indentation, but still color if stdout is a tty). The records must expand can_id such that it only contains the ID and the encoded flags (RTR/EXT/ERR) are exposed as separate boolean JSON fields.

Optional --min-seqno=.. can be provided to ignore old seqnos, plus we need a --follow option that reads all existing seqnos ASAP and then blocks on HTTP long polling, returning new records as they come, thus implementing real-time bridging.

We need to design the JSON schemas for this.

Option --boot=.. restricts output to specific boot IDs only (there may be several or a range like 1,2,3,7-15,56-88).

Option --forward=<socketcan-iface-name> instructs the tool to forward all read records into the specified local socketcan interface, eg --forward=vcan0; crucially, the frames should be forwarded respecting their boot timestamps (e.g., if frames were captured at times 1000, 1100, 4500 us since boot, appropriate delays will be inserted during forwarding; note that we must avoid phase slip); the forward option requires --boot= because timestamps are discontinuous across reboots.

Option --output-socketcan also requires --boot= and its effect is that the stdout format matches the candump log format (see candump --help) instead of JSON.

NOTE: When paging GET /cf3d/api/v1/records, use seqno_min as the cursor:

• lossless paging: seqno_min = last_returned_record.seqno + 1;
• catch-up paging (may skip unseen records from a truncated page): seqno_min = latest_seqno_seen + 1.