Firmware integration · Phase 1 backend · live
Scooter Firmware Bring-Up
The backend is running in production and a simulated scooter already talks to it end to end. This guide takes the real firmware the same way, from the first TLS handshake to a confirmed unlock. Do the stages in order: each one ends with a check against the live system, and every later stage assumes the earlier checks pass.
- Broker
- mqtt.xzcode.me:8883
- Transport
- TLS 1.2/1.3 · MQTT 3.1.1
- Identity
- scooter-{id}, one per unit
- Byte order
- little endian
- Integrity
- CRC-16/CCITT-FALSE
- Report every
- 10 s (default)
How a reading and a command travel
The scooter only ever talks to the MQTT broker. It has no HTTP client and never learns whether the backend accepted a frame. Everything after the broker belongs to the backend team, but knowing the path tells you where to look when something goes missing.
c/scooter-1/ prefix, using the identity the scooter logged in with. That prefix is how the backend catches one unit publishing under another unit's id.- No application-level receipt for telemetry. MQTT QoS 1 PUBACK means the broker has it, nothing more. Rejected frames are stored server-side with the raw bytes and a reason.
- A command succeeds only when your ACK arrives. The backend never treats "published to MQTT" as done. No matching ACK within 15 s means
TIMEOUT. - The wire contract lives in one place:
packages/protocol/src/. The server and the simulator both import it, so when this guide and that code disagree, the code wins. Tell the backend team.
Get an identity for each unit
Every scooter has its own credential. A shared fleet key is forbidden: one extracted key would let anyone impersonate every scooter and receive every scooter's commands. The backend team provisions each unit and gives you four values.
| Value | Example | Where it goes |
|---|---|---|
| Scooter id | 1 | The topic path: sc/1/…. Decimal, no leading zeros. |
| Username | scooter-1 | MQTT CONNECT username |
| Client id | scooter-1 | MQTT CONNECT client identifier. It must match the provisioned value exactly. |
| Password | 32 random chars | MQTT CONNECT password. Shown once at provisioning; the server keeps only a bcrypt hash. |
- Store the credential in EEPROM or flash, per unit. Never compile one into a shared image.
- Never print the password on a debug UART. A lost password means reprovisioning, not recovery.
- Two units with the same client id will repeatedly kick each other off the broker. Only one session per client id can exist at a time.
Before touching firmware, prove the credential works from a laptop. Stop any simulator or unit using the same client id first, or this connection will take over its session.
mosquitto_sub -h mqtt.xzcode.me -p 8883 -V mqttv311 \
--cafile isrg-root-x1.pem \
-i scooter-1 -u scooter-1 -P '<password>' \
-t 'sc/1/cmd' -q 1 -d
You see CONNACK (0) and SUBACK with granted QoS 1. CONNACK (5) means the credential is wrong or disabled.
Connect over TLS
| Setting | Value | Why |
|---|---|---|
| Host / port | mqtt.xzcode.me : 8883 | The only listener devices can reach. Port 1883 is internal to the server. |
| Protocol | MQTT 3.1.1 (level 4) | What the backend is tested against for devices. |
| Clean session | true | The device re-subscribes on every connect (below). |
| Keepalive | 60 s suggested | Keep it well under the carrier's NAT idle timeout. A dead link is detected after about 1.5× keepalive. |
| Max packet | 1 KB | The broker drops anything larger. The largest frame is 20 bytes. |
| Client certificate | not required yet | Password auth today. mTLS depends on your module. See Still to agree. |
Trusting the server
The broker presents a Let's Encrypt certificate for mqtt.xzcode.me. Let's Encrypt replaces it about every 60–90 days, so load the ISRG Root X1 CA into the modem (and ISRG Root X2, so a change of key type doesn't strand units). Never pin the leaf certificate. A pinned leaf bricks the whole fleet's connectivity on the next renewal.
Certificate validation needs the time. After a cold boot, get network time from the modem (NITZ, or AT+CCLK? on 3GPP modules) before opening TLS. The same clock stamps telemetry, which the backend rejects if it's more than 5 minutes off.
The CONNECT packet
Register a Last Will so the broker announces the scooter offline if the link dies without a clean disconnect:
| Will field | Value |
|---|---|
| Topic | sc/1/status |
| Payload | 0x00 (1 byte) |
| QoS / retain | 1 / true |
On every successful CONNACK, in this order
- Publish
0x01tosc/1/status, QoS 1, retained. - Subscribe to
sc/1/cmdat QoS 1. Clean session drops subscriptions, so this happens every time. - Publish a
BOOTevent if this is the first connect since power-up. - Replay any buffered readings with bit 7 set (stage 07).
- Resume live telemetry on the normal interval.
If the connect fails, back off with jitter: for example 2 s, 4 s, 8 s … capped at 5 minutes, plus a random 0–30%. A fleet that reconnects in lockstep after a cell outage floods the broker all at once.
Ask the backend team to run this while the unit is powered. It should show connected=true, clean_start=true and subscriptions=1:
docker exec <emqx-container> emqx ctl clients show scooter-1
The scooter's connection badge on the dashboard reads Online once the first live telemetry frame lands (next stage).
Build frames byte for byte
Every frame is a fixed-size binary record: a version byte 0x01, the fields, then a CRC over everything before it. Multi-byte fields are little endian, the AVR's native order, so a memcpy of a uint32_t is already correct. The CRC goes out low byte first too.
CRC-16/CCITT-FALSE
Poly 0x1021, init 0xFFFF, no reflection, no final XOR. Check value: CRC of the ASCII string "123456789" is 0x29B1. Test this before anything else; a wrong CRC is the most common bring-up failure.
/* Bit-wise: no table, fits comfortably on an ATmega328. */ uint16_t crc16_ccitt_false(const uint8_t *data, uint8_t len) { uint16_t crc = 0xFFFF; while (len--) { crc ^= (uint16_t)(*data++) << 8; for (uint8_t i = 0; i < 8; i++) crc = (crc & 0x8000) ? (crc << 1) ^ 0x1021 : (crc << 1); } return crc; } /* Sealing a 20-byte telemetry frame: CRC over bytes 0..17, stored LE. */ uint16_t crc = crc16_ccitt_false(frame, 18); frame[18] = crc & 0xFF; frame[19] = crc >> 8;
Telemetry · 20 bytes · sc/{id}/tel
The bottom row is the golden vector from packages/protocol/test/telemetry.spec.ts: msg_id 4660, latitude 30.0444, longitude 31.2357, battery 87 %, speed 23 km/h, status 0x1A, CRC 0x469D. Its timestamp decodes to a 1970 date, so use it to test byte layout only. Sent live, it would be rejected as CLOCK_SKEW.
status_bits · byte 17
int32 tops out near ±214.7°. Past that the value wraps to a different coordinate that looks valid. Compute with int32_t arithmetic from fixed-point GPS fields, never through a float that can drift.A host-side unit test of your encoder, fed the golden values above (including timestamp 16909060 and status 0x1A), produces exactly:
01 34 12 04 03 02 01 60 69 E8 11 88 30 9E 12 57 17 1A 9D 46
Run the same source on the target and compare over the UART. A difference there, but not on the host, usually means a struct was padded or an integer promoted.
Send live telemetry
Telemetry is also the heartbeat. There's no separate keepalive frame, so publish every interval even when nothing has changed. Publish at QoS 1, not retained, with bit 7 clear.
What the backend does with each frame
Checks run in this order. A reject is refused and stored with the raw bytes. A tag is accepted but flagged.
| Check | Outcome | Rule |
|---|---|---|
| BAD_LENGTH | reject | Payload not exactly 20 bytes. No padding, no trailing newline. |
| BAD_VERSION | reject | Byte 0 isn't 0x01. |
| BAD_CRC | reject | CRC over bytes 0–17 doesn't match bytes 18–19. |
| TOPIC_CLIENTID_MISMATCH | reject | The client id you logged in with doesn't own the scooter id in the topic. Also logged as a security incident, and the unit is disconnected. |
| UNKNOWN_SCOOTER | reject | The scooter id isn't in the fleet. |
| BAD_BATTERY | reject | Outside 0–100. |
| BAD_SPEED | reject | Above 40 km/h. |
| CLOCK_SKEW | reject | Live frame more than ±300 s from server time. |
| no GPS fix | tag | Bit 1 clear: frame accepted, coordinates ignored, battery/lock/speed still used. Send these frames. Don't hold them back. |
| BAD_LAT / BAD_LNG | reject | With a fix: latitude outside ±90 or longitude outside ±180. |
| position jump | tag | Moved faster than 200 m/s since the last frame: accepted, marked suspect_jump. |
duplicate msg_id | ignored | Same msg_id seen within 600 s: processed once, silently. This is what makes QoS 1 redelivery safe. |
msg_id rules
- Increment by one per frame, wrapping at 65 535. At 10 s, a wrap takes about 7.5 days, far outside the 600 s dedup window.
- Don't reset it on reconnect. Rapid reconnects would reuse ids inside the window, and real readings would be discarded as duplicates.
- Don't restart it from 0 after a reboot either. A unit that reboots and reconnects within 10 minutes hits the same trap. Save the counter to EEPROM every few hundred frames and skip ahead on boot, or seed it randomly.
- A QoS 1 retransmission reuses the same bytes and the same
msg_id. That's correct.
With the unit on the bench, the dashboard or the rider app shows battery, lock and last seen updating every interval, and the badge holds at Online. The backend team confirms telemetry_rejected has no new rows for your scooter id. If it does, each row's reason, detail and raw hex usually pinpoint the bug.
Execute commands and answer with ACKs
Commands arrive on sc/{id}/cmd as 12-byte frames. The backend matches your ACK to the command by command_id alone, so mirror it exactly.
Command · 12 bytes · received on sc/{id}/cmd
Example: UNLOCK (0x02), command_id 42, issued at 1789313734, CRC 0x0491.
ACK · 10 bytes · publish to sc/{id}/ack
The ACK for the command above, result SUCCESS. The same ACK with result ALREADY_DONE is 01 02 2A 00 00 00 02 00 66 C3.
| Opcode | Command | What the device does |
|---|---|---|
| 0x01 | LOCK | Engage the lock, confirm with the sensor, ACK, then send one telemetry frame straight away. |
| 0x02 | UNLOCK | Same as LOCK, other direction. |
| 0x03 | PING | ACK SUCCESS. Nothing else. |
| 0x04 | REBOOT | Publish the ACK, wait for its PUBACK, then reset. Reset first and the command times out. |
| 0x05 | SET_INTERVAL | issued_at carries the new interval in seconds (1–3600). Apply it, then ACK. |
| Result | Name | Backend records | Send it when |
|---|---|---|---|
| 0x00 | SUCCESS | ACKED | Done and confirmed by the sensor. |
| 0x01 | FAILED | FAILED | Tried and failed. Put a device-specific reason in err. |
| 0x02 | ALREADY_DONE | ACKED | This command_id was already executed. It's a retry. |
| 0x03 | INVALID | FAILED | Unknown opcode or bad parameter. |
| 0x04 | NO_FEEDBACK | FAILED | Actuator driven but the confirmation sensor didn't answer. The operator is told the lock state is unverified. |
Device rules for commands
- Keep the last ~8 executed
command_ids in RAM (32 bytes). If a command's id is already there, don't actuate again; answerALREADY_DONE. Retries always reuse the original id, so this is what prevents a double unlock. - Don't send an ACK you can't back up. Silence becomes a clean
TIMEOUTthe operator can retry; a falseSUCCESSleaves a rider standing at a locked scooter. - An ACK that arrives after the 15 s timeout is ignored. It doesn't bring the command back.
- The backend never sends two in-flight commands of the same opcode to one scooter, so you won't see overlapping UNLOCKs.
- Handle the frame as it arrives: check length, version and CRC first. A command failing those gets no ACK.
Send a PING through the API (the rider app's Ping button does the same):
curl -X POST https://api.xzcode.me/api/v1/scooters/1/commands \ -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ -d '{"opcode":"PING"}' # → 202 {"command_id": 42, "state": "PENDING", ...} curl -H "Authorization: Bearer $TOKEN" https://api.xzcode.me/api/v1/commands/42 # → "state": "ACKED", "result_name": "SUCCESS" within about a second
Then UNLOCK and LOCK from the rider app: the lock icon flips as soon as the ACK arrives, with no page reload. The token comes from POST /api/v1/auth/login with an operator account.
Events and the status byte
Event · 8 bytes · sc/{id}/evt
| Type | Event | Send it |
|---|---|---|
| 0x01 | LOW_BATTERY | Once, when the battery crosses your low threshold. Not on every reading. |
| 0x02 | BOOT | First connect after power-up or reset. |
| 0x03 | FALL_DETECTED | Reserved for a later phase. |
| 0x04 | TAMPER | Tamper switch or unexpected motion while locked. |
| 0x05 | CHARGING_START | Charger connected. |
| 0x06 | CHARGING_END | Charger removed. |
Any other type is rejected as UNKNOWN_EVENT_TYPE, not ignored.
Status · 1 byte · sc/{id}/status
Just 0x01 (online) or 0x00 (offline), with no version byte and no CRC. Publish 0x01 retained after every connect. The broker publishes the retained 0x00 for you through the Last Will when the link drops. Before a deliberate power-down, publish 0x00 retained yourself, then send DISCONNECT. A clean disconnect doesn't trigger the Will.
Cut the modem's power on the bench. The scooter turns Offline within about 1.5× the keepalive (Last Will) or 30 s (missed telemetry), whichever comes first. Power it back and it returns to Online on the first live frame, and a BOOT event appears in the scooter's event list.
Survive outages with bit 7
This is the behaviour that matters most. While the link is down, keep sampling and store the readings. When the link returns, replay them with bit 7 set. Bit 7 means one thing: this reading is not current.
| Mistake | What the operator sees |
|---|---|
| Bit 7 never set on replayed frames | A scooter back from a two-hour outage jumps to two-hour-old positions shown as current. Someone gets sent to where it was. |
| Bit 7 left set on live frames | The scooter streams happily but stays Offline and never appears on the live map. |
- Replayed frames take new
msg_ids from the normal sequence. Don't reuse the ids the readings would have had. - Keep each reading's original timestamp. The backend accepts buffered frames up to 7 days old, but still rejects any stamped more than 5 minutes in the future.
- Replay oldest first, then go back to live frames. If storage fills, drop the oldest readings, not the newest.
- Pace a long replay (for example 5–10 frames per second). The broker has no trouble with it, but the modem's send buffer may.
Disconnect the antenna for two minutes while the scooter moves on the bench GPS simulator, then reconnect. The live map doesn't jump backwards, the history for those two minutes fills in with the device's timestamps, and the badge returns to Online on the first live frame after the replay.
When it doesn't work
Before blaming firmware, confirm the backend is healthy by running the simulator with a scooter id that isn't your hardware. If the simulated scooter works and yours doesn't, the difference is in the device.
npm run sim -- --count 1 --start-id 2 --interval 10 # known-good peer on the live broker npm run sim -- --count 1 --start-id 2 --ack-mode silent # forces the 15 s TIMEOUT path npm run sim -- --count 1 --start-id 2 --buffered-burst # outage + bit 7 replay # every command it receives is printed: [sim] scooter 2 <- cmd #42 UNLOCK [sim] scooter 2 -> ack #42 SUCCESS (locked=false)
For a full reference implementation of connect, status, subscribe, telemetry, ACK and replay, read apps/simulator/src/virtual-scooter.ts.
| Symptom | Most likely cause |
|---|---|
| CONNACK 5, not authorised | Wrong username or password, or the credential was revoked. |
| TLS handshake fails | Clock not set yet, or ISRG Root X1 missing from the modem. |
| Connects, then drops every few seconds | Another unit or a test tool is using the same client id. |
| Connects, publishes once, gets disconnected | Published outside the five allowed topics (wrong id, leading zero, typo in the channel). |
| Connected but nothing on the dashboard | Check in this order: CRC check value, little endian, exact length, clock, topic id. telemetry_rejected gives the reason. |
Every frame is TOPIC_CLIENTID_MISMATCH | Client id doesn't belong to the scooter id in the topic. |
| Frames vanish with no rejected row | Duplicate msg_id inside 600 s, usually a counter reset on reconnect or reboot. |
| Shows Offline while streaming | Bit 7 set on live frames, or the unit's interval is longer than the one configured for it. |
Commands always end TIMEOUT | Not subscribed to sc/{id}/cmd after reconnect, ACK published to the wrong topic, or command_id not mirrored byte for byte. |
| Lock changes only after a delay | No telemetry frame sent right after the ACK. It shows at the next interval. |
| A command runs twice | The recent command_id history isn't checked, so a retry wasn't recognised. |
| Position lands somewhere impossible | int32 overflow or float rounding in the ×10⁷ conversion. |
Still to agree between the teams
| Question | Backend today | Needs from firmware |
|---|---|---|
| Mutual TLS | Server certificate only; devices log in with a password. | Does the 4G module support per-device X.509 client certificates? If yes, the broker switches to verify_peer. |
| Buffered age limit | Replays accepted up to 7 days old; the ±5 min rule applies to live frames only. | Confirm 7 days covers the longest outage you'll store, and how many readings fit in storage. |
| Reporting interval | 10 s default, changeable per unit with SET_INTERVAL. | Confirm 10 s fits the data plan and battery budget. |
| REBOOT ordering | Expects an ACK before the reset. | Confirm the module can deliver a PUBACK-confirmed ACK before resetting. |