Initial public release
This commit is contained in:
@@ -0,0 +1,179 @@
|
||||
Developed by Grant Lanier
|
||||
|
||||
# M5Stack Keyboard Emulator
|
||||
|
||||
A node-based macro editor for the M5Stack AtomS3. Build keyboard/mouse/serial automation as visual flow graphs, push them to the device over USB, optionally sync runtime variables over BLE.
|
||||
|
||||
## What it is
|
||||
|
||||
The AtomS3 is an ESP32-S3 board with an LCD that doubles as a button. With this firmware on it, the host PC sees it as a USB HID keyboard + mouse + CDC serial device. Whatever it sends via that USB connection is determined by routines authored in the desktop app and uploaded to flash.
|
||||
|
||||
A routine is a directed graph rather than a flat key sequence. Nodes can branch (user picks an option on the device's screen), loop a user-supplied number of times, send RS-232 over the device's UART pins, probe the host with a Num Lock toggle, or pull a variable dict from the desktop app over BLE and interpolate it into typed text. Once uploaded, the device runs all of this on its own, the desktop app is only needed for authoring and for serving BLE variables.
|
||||
|
||||
## How to use it
|
||||
|
||||
**Flashing a new ATOMS3.** Run `CompileAndUpload/upload.bat` and follow the prompts, it walks you through putting the AtomS3 into download mode and flashes the firmware (see [Flashing a fresh device](#flashing-a-fresh-device) for details). Then run `GUI.bat`. Click **Connect** in the toolbar if the device doesn't auto-connect; the AtomS3 is found by USB VID + a JSON ping handshake.
|
||||
|
||||
**Profiles.** A profile is a named set of routines. The toolbar has a profile dropdown with `+` (new) and `×` (delete). Whatever profile is currently selected is the "active" upload target. The profile named `Sub-Routines` is reserved: routines there don't show in the device menu, they exist only to be called from other routines via the Sub-Routine node.
|
||||
|
||||
**Routines.** The left pane lists routines in the current profile. Use `+` to add, `×` to delete, the arrows to reorder (order = order on the ATOM S3's menu). Click the camera icon to the right of the routine name in the left sidebar to change its image, this image will show on-device. You can rename a routine by opening it and left-clicking on the "Start" block.
|
||||
|
||||
**Modifying Routines.** Once you have selected a routine to edit, right-click the canvas to add a node, drag between ports to connect them (outputs are red and inputs are green). Every routine must have a single **Start** node - its output is the entry point. The right pane shows properties for whatever block is currently selected. Multi-select with shift (or by dragging with right-click), pan with left-click, zoom with the wheel.
|
||||
|
||||
**Settings.** Moving up to the toolbar, the Settings dialog covers global timings (key hold, type delay, combo pre/post, probe timeout, etc.) and screen orientation. These ship with the project and are applied device-side on upload.
|
||||
|
||||
**Upload.** Clicking "Upload Profile" in the toolbar serializes the currently selected profile / settings / sub-routines to JSON and writes it to the device's LittleFS partition.
|
||||
|
||||
**Running routine.** Universally a short press of the screen is "next" / "scroll" and a long press is "select." If at any point you want to cancel a running routine, you can unplug the device or click the physical button on the left side of the ATOM S3.
|
||||
|
||||
**Variables (optional).** Open the Variables window, edit the dict (e.g. `{"ticket": "PROJ-1234"}`). When a running macro hits a Variables node in BLE pull/push/request mode, the device powers on its radio just long enough to fetch the current dict from the desktop app. Any Type Text node with `(VAR{ticket})` in it will then interpolate the value (case-insensitive — `(VAR{TICKET})` matches the same entry). Requires the desktop app to be running and the Bluetooth adapter installed in the desktop to support BLE, you will know that the Python app is ready for BLE connections based off of the "BLE: Ready" status in the toolbar. The legacy `(BLE{name})` syntax is still recognized for backward compatibility.
|
||||
|
||||
**Backups & recovery.** The Backups button in the toolbar opens a snapshot history of the project. Profiles are auto-saved to `config/profiles/` and recovered on next launch.
|
||||
|
||||
# Technical details
|
||||
|
||||
## Flashing a fresh device
|
||||
|
||||
A new AtomS3 has no firmware on it. From the project root, run:
|
||||
|
||||
```
|
||||
CompileAndUpload\upload.bat
|
||||
```
|
||||
|
||||
The script is interactive and figures out the COM port itself.
|
||||
|
||||
You can define the COM port manually using this format: `upload.bat COM7` if needed.
|
||||
|
||||
The toolchain expects `arduino-cli` on PATH; `esptool.exe` is resolved from the Arduino15 cache automatically.
|
||||
|
||||
### AtomS3 vs AtomS3 Lite (one universal binary)
|
||||
|
||||
The AtomS3 (128×128 LCD) and the AtomS3 Lite (no screen, one RGB LED) are the same ESP32-S3 module and run the **same firmware** — the build detects the board at boot. Settings has two flash buttons: *Flash Firmware — AtomS3* and *Flash Firmware — AtomS3 Lite (no screen)*. They flash identical code; the Lite button only switches the flasher's manual-recovery wording (the Lite has no screen to watch).
|
||||
|
||||
On a Lite, everything the LCD would show maps to the RGB LED instead. Every state is a distinct color + motion — smooth crossfades mark the calm "what am I / who has me" states, while sharper blinks and bursts mark action:
|
||||
|
||||
| State | LED |
|
||||
| --- | --- |
|
||||
| Boot | white pulse |
|
||||
| Idle — **BLE** mode | soft white↔blue crossfade |
|
||||
| Idle — **Mesh** mode | soft white↔amber crossfade |
|
||||
| (either idle) | a click flashes white then blinks N times for the slot position |
|
||||
| Mode switched (5 s hold) | triple burst in the new accent — blue (BLE) / amber (Mesh) |
|
||||
| Running a routine | steady green (brightens as text types; breathes during a delay) |
|
||||
| Pause node | steady yellow (untimed) / yellow blink that speeds up in the last 3 s (timed) |
|
||||
| Branch / loop selector | magenta / orange blink-bursts counting the current choice or value |
|
||||
| Error | red triple-blink; failed check = red blink |
|
||||
| Live joined (idle) | slow cyan breathe (heartbeat); flickers white as keystrokes arrive |
|
||||
| Reconnecting to host | cyan 1 Hz blink |
|
||||
| Lost the hub mid-session | cyan/red blink |
|
||||
| Identify (host labeling it) | fast blue/white strobe |
|
||||
| Mesh hub mode | steady purple |
|
||||
| BLE variables exchange | blue breathe (brief, inside a routine) |
|
||||
|
||||
Because the idle color now encodes the live transport (blue = BLE, amber = Mesh), you can tell a resting headless node's mode at a glance without a screen. A Lite still supports the full feature set including button-driven macro selection (click to cycle, hold to run; hold 5 s to toggle the transport), but with no screen it's best to keep your most-used routines in the first slots so the position blink-count stays easy to read. Lites shine as headless live-keyboard nodes.
|
||||
|
||||
## Live keyboard — two transports
|
||||
|
||||
The **Keyboard** window streams your laptop's keyboard and mouse to many devices at once. Clicking the toolbar's **Keyboard** button first asks which transport to use for the session; the picker shows each mode's soft device cap:
|
||||
|
||||
- **Bluetooth (BLE)** — a direct BLE link to each device, no USB hub required. Windows only holds ~3-4 reliable concurrent BLE links, so this **soft-caps at 4 devices**.
|
||||
- **ESP-NOW hub** — the M5Stack plugged into the laptop over USB is switched into an **ESP-NOW hub** and broadcasts the input stream to every other device over connectionless Wi-Fi — no router, no access point, no SoftAP. It sidesteps the BLE ceiling and **soft-caps at 12 devices** (the hub roster can track more; 12 is just where the UI starts warning).
|
||||
|
||||
Both caps are soft: the window keeps working past them but shows a non-blocking warning, since throughput degrades. Streaming, recording, replay, the trackpad, and profiles behave identically in either mode.
|
||||
|
||||
How it works in practice:
|
||||
|
||||
- Open **Keyboard** and pick a mode. For hub mode, plug one device into the laptop over USB first — the app finds it and makes it the hub automatically.
|
||||
- Click **Discover** — idle in-range devices answer (over the mesh in hub mode, or a BLE scan in Bluetooth mode) and you pick which to add. (Leave each node idle on its selector; a running routine owns its radio for HID.)
|
||||
- The device list shows a **Lag** column: how many keystrokes each node is behind the live stream (hub mode only — BLE mode shows a dash). It should sit at 0; if it climbs across the fleet, the Wi-Fi channel is congested — change **Mesh Wi-Fi channel** in Settings (every device must share the same channel) to a clearer one.
|
||||
|
||||
Zero input loss is guaranteed by the transport: every keystroke/mouse-button/wheel event carries a sequence number, the hub keeps a retransmit buffer, and nodes NACK any gap and ACK their progress. Pure mouse *moves* are the only thing that may be coalesced (absolute positions are self-correcting). Every frame is AES-256-GCM encrypted under a per-session key, the same crypto the BLE channels use. The hub holds no key — it only routes — and reverts to a normal node a few seconds after you close the window.
|
||||
|
||||
The encryption keys are keyed by each device's MAC, which is the same MAC the mesh uses, so **no re-provisioning is needed** — devices you've already uploaded a profile to just work. Old BLE profiles are migrated automatically on load.
|
||||
|
||||
## Wiping a device
|
||||
|
||||
A flashed device has two persistent regions: NVS at `0x009000` (20 KB, runtime settings) and LittleFS at `0x670000` (1.5 MB, the uploaded project). To clear both without reflashing the firmware itself:
|
||||
|
||||
```
|
||||
python wipe_device.py # auto-detect port
|
||||
python wipe_device.py COM7 # or specify it
|
||||
```
|
||||
|
||||
`wipe_device.py` finds the running device by USB VID `0x303A` and a JSON `ping` handshake, sends `{"cmd":"bootloader"}` to drop it into ROM download mode, waits for the USB-JTAG port to re-enumerate, then calls `esptool` to erase the two partitions. Useful before reflashing or when stale settings are causing weird behavior.
|
||||
|
||||
## Running the editor
|
||||
|
||||
```
|
||||
pip install -r requirements.txt
|
||||
python main.py
|
||||
```
|
||||
|
||||
Dependencies are: `pyserial` for USB CDC, `bleak` for BLE. Tkinter is stdlib. `GUI.bat` is just a double-clickable wrapper around `python main.py`.
|
||||
|
||||
A connected device exposes one of two COM ports depending on what mode it's in: TinyUSB CDC while the firmware is running, USB-JTAG while the ROM bootloader is active. The Python app talks to the former, `esptool` talks to the latter. If the device is in USB-JTAG mode and you want to bring it back to USB-CDC a simple reset using the side button is safe.
|
||||
|
||||
## Project layout
|
||||
|
||||
```
|
||||
app.py MacroPadApp - main window, wires everything together
|
||||
main.py entry point
|
||||
serial_manager.py USB CDC discovery + JSON command protocol
|
||||
ble_server.py bleak client; pushes variable dict on demand
|
||||
live_protocol.py shared live-keyboard packing + mesh/hub framing
|
||||
mesh_link.py serial bridge to the ESP-NOW hub device
|
||||
mesh_manager.py mesh keyboard session (drop-in for the old BLE one)
|
||||
wipe_device.py standalone NVS + LittleFS eraser
|
||||
test_serial.py smoke-test for the JSON protocol without the GUI
|
||||
node_editor/ canvas, node rendering, port routing
|
||||
widgets/ MacroList, PropertiesPanel, Toolbar, dialogs
|
||||
models/ Macro, NodeData, Project, Settings, profile/backup mgrs
|
||||
utils/constants.py NODE_TYPES, HID lookup tables, canvas styling
|
||||
firmware/MacroPad/ Arduino sketch (the on-device interpreter)
|
||||
CompileAndUpload/ compile.bat / upload.bat / wipe_device.bat wrappers
|
||||
```
|
||||
|
||||
## Communication model
|
||||
|
||||
**USB (always-on).** JSON over TinyUSB CDC at 115200 baud. The host drives the conversation: `ping`, `bootloader`, profile upload/download, wipe. `serial_manager.py` is the single chokepoint for everything that goes over the wire.
|
||||
|
||||
**BLE (on-demand).** The radio is off by default. When a running routine hits a BLE Variables node, the device brings up NimBLE, advertises, accepts a write to the variables characteristic, then tears the stack down. `ble_server.py` runs a continuous bleak scan loop on the host so it's ready the instant the device shows up. BLE and USB stacks fight each other on the ESP32-S3 so this was necessary.
|
||||
|
||||
## Project / profile / backup model
|
||||
|
||||
- A **Project** is `Settings` + the BLE variables dict + every routine across every profile, JSON-serialized.
|
||||
- **Profiles** are saved to `config/profiles/` and auto-recovered on launch (`profile_manager.py`).
|
||||
- **Backups** are timestamped snapshot copies (`backup_manager.py`); not git, just a versioned history of the project.
|
||||
- The profile named **`Sub-Routines`** is reserved; routines there don't appear in the device menu and exist solely to be called from other routines via the Sub-Routine node.
|
||||
|
||||
## Operators (node types)
|
||||
|
||||
Routines are graphs. Every routine starts with a **Start** node and flows through any of the following:
|
||||
|
||||
| Node | What it does |
|
||||
|---|---|
|
||||
| **Start** | Entry point. Owns the routine name and its label color in the device list. |
|
||||
| **Type Text** | Sends a typed string. Pop-out editor for long blocks; supports cmd/PowerShell hinting. |
|
||||
| **Key Combo** | Modifier+key chord. Per-node Custom Timings override the global pre/post hold (defaults are ~3× faster than device defaults). |
|
||||
| **Pause** | Halts execution until the user clicks the device, or for a fixed duration. Display text is shown on the LCD. |
|
||||
| **Delay** | Plain `delay(ms)`. |
|
||||
| **Branch** | N-way fork rendered as a dropdown on the device. User picks a path. |
|
||||
| **Loop** | Repeats N times. Has a tied loop-back input. Count can come from the last **Loop Selector** that ran instead of being static. |
|
||||
| **Loop Selector** | Prompts on-device for a loop count (min/max/step/default). Optionally re-prompts each iteration. |
|
||||
| **Iteration Branch** | Inside a loop, dispatches per-iteration. References a specific Loop node by ID. "Skip on final iteration" exists because some side-effect chains (e.g. "switch to next KVM target") have nothing meaningful to do on the last pass. |
|
||||
| **Aggregator** | Many-to-one merge (1–16 inputs → 1 output). Keeps the canvas readable when several branches converge. |
|
||||
| **Mouse Click** | left/right/middle × click/double/press/release. |
|
||||
| **Media Key** | Volume, mute, play/pause, next/prev, stop, brightness. |
|
||||
| **Macro** | Plays back a recorded HID stream — raw `[t_ms, action, usage_code]` tuples captured by `MacroRecorderDialog`. Unlike Type Text / Key Combo, this preserves real chords and exact timing because it doesn't re-synthesize from keysyms. |
|
||||
| **RS232 Send** | Serial out from the device's UART pins: configurable baud / data / stop / parity / line-ending, optional response-wait with timeout, optional post-send delay (handy when the command kills USB power, e.g. a KVM switch — the device persists resume state first). |
|
||||
| **PC Alive Check** | Toggles Num Lock and watches the LED report to decide whether the host is alive. Branches true/false; optional retry loop. |
|
||||
| **Variables** | Reads/writes the on-device variable store; five modes. **Pull BLE** — device fetches the current dict from the host (scope `Universal` shared across devices, or `This device` keyed by eFuse MAC). **Push BLE** — device uploads its local store to the host, written into that device's profile only; Universal is never clobbered (use this to round-trip values that were set on-device back to the desktop app). **Request BLE** — device names the variables it wants and the host pops a fill-in dialog (optional notification sound) before returning the user's edits. **Set Variables** — local name=value writes, no radio. **Get Variables** — runs a host-side script via HID and decodes the answer from Num Lock toggle counts → outcome values, either Manual (raw PowerShell) or Semi-Auto (build a check sequence in a visual editor, up to 5 outcomes; an optional Win+R bootstrap can launch an elevated terminal first). All three BLE modes bring NimBLE up only for the exchange and tear it back down. Use `(VAR{name})` inside Type Text to interpolate a stored value (case-insensitive; `(BLE{name})` is still recognized as a legacy alias). |
|
||||
| **Sub-Routine** | Calls a routine from the reserved `Sub-Routines` profile and resumes when it returns. |
|
||||
| **Note** | Canvas annotation. Never serialized to the device. |
|
||||
|
||||
green = input, red = output. Nodes can be visually flipped (ports swap sides) by selecting the node and pressing "f" which is purely cosmetic.
|
||||
|
||||
## Things worth knowing
|
||||
|
||||
- The HID lookup table lives in `utils/constants.py` as `TKKEYSYM_TO_HID`. That's how the recorder maps Tk keysyms to USB HID usage codes. Add entries there if you find a key that doesn't record correctly.
|
||||
- `test_serial.py` is a quick smoke-test for the JSON protocol without launching the GUI.
|
||||
Reference in New Issue
Block a user