348 lines
21 KiB
Markdown
348 lines
21 KiB
Markdown
# 05 — Input Handling (Python xkbcommon + Browser WASM)
|
||
|
||
This chapter covers input handling in the MIT approach: keyboard (with
|
||
xkbcommon), pointer, touch, focus management, and how input events travel
|
||
between browser and native apps.
|
||
|
||
## Architecture (same as Greenfield, but with improvements)
|
||
|
||
The browser is the source of truth for the keymap (same as Greenfield).
|
||
The keymap is compiled in the browser using @gfld/compositor-wasm's xkbcommon
|
||
WASM module, then shipped to the Python proxy as a real native shm fd via
|
||
the `/mkstemp-mmap` WebSocket endpoint. The proxy then sends it to native
|
||
apps via libwayland-server's SCM_RIGHTS FD passing.
|
||
|
||
**Improvements over Greenfield**:
|
||
1. **Real touch input** — Greenfield has a 46-line stub that never fires
|
||
2. **Primary selection** — middle-click paste, which Greenfield lacks
|
||
3. **Adaptive keyboard layout** — easier to switch layouts at runtime
|
||
|
||
## Browser side — Custom TypeScript (clean-room)
|
||
|
||
### Seat abstraction
|
||
|
||
Same architecture as Greenfield (clean-room from spec + Wayland docs):
|
||
- `Seat.ts` — `wl_seat` global, multi-seat support, focus management
|
||
- `Keyboard.ts` — `wl_keyboard` impl, keymap, modifiers, key events
|
||
- `Pointer.ts` — `wl_pointer` impl, motion, buttons, axis
|
||
- `Touch.ts` — `wl_touch` (REAL implementation, not stub)
|
||
- `Xkb.ts` — xkbcommon WASM wrapper (REUSED @gfld/compositor-wasm)
|
||
- `InputQueue.ts` — event coalescing + RAF dispatch
|
||
- `InputOutput.ts` — FD abstraction
|
||
- `events.ts` — KeyEvent, ButtonEvent, AxisEvent wrappers
|
||
|
||
### xkbcommon WASM (REUSED)
|
||
|
||
`@gfld/compositor-wasm` (MIT) bundles:
|
||
- libxkbcommon 1.5.0 compiled to WASM via emsdk 3.1.46
|
||
- Embedded xkeyboard-config database
|
||
- Exports `init()` + `lib.xkbcommon` singleton with keymap/state functions
|
||
|
||
This is REUSED AS-IS. No custom code needed.
|
||
|
||
### DOM → Wayland key event translation (clean-room)
|
||
|
||
Same approach as Greenfield (verified correct Wayland behavior):
|
||
1. `event.code` (e.g. `"KeyA"`) looked up in EvDevKeyCode reverse-name-map → evdev keycode (e.g. 30)
|
||
2. `x11KeyCode = evdevKeyCode + 8` (Linux input protocol offset)
|
||
3. `capsLock`/`numLock` read from `event.getModifierState(...)`
|
||
4. DOM `.key` glyph IGNORED — only physical `.code` used (mirrors evdev)
|
||
|
||
### Modifier tracking (REUSED WASM)
|
||
|
||
Via @gfld/compositor-wasm's xkbcommon state machine:
|
||
1. `Seat.updateModifierState` calls `xkb.keyDown(x11)` or `xkb.keyUp(x11)`
|
||
2. `_xkb_state_update_key` invoked internally
|
||
3. `notifyModifiers` reads `xkb.modsDepressed` / `modsLatched` / `modsLocked` / `group`
|
||
4. `Keyboard.sendModifiers(modsDepressed, modsLatched, modsLocked, group)` sends to native client
|
||
|
||
### Touch input (NEW vs Greenfield)
|
||
|
||
Greenfield has a 46-line stub for `Touch.ts`. Our MIT approach implements
|
||
real touch:
|
||
|
||
```typescript
|
||
// Pseudocode for Touch.ts (clean-room)
|
||
export class Touch {
|
||
private touchPoints: Map<number, TouchPoint> = new Map();
|
||
|
||
constructor(public seat: Seat) {
|
||
// Listen to DOM pointer events with pointerType === 'touch'
|
||
document.addEventListener('pointerdown', this.onPointerDown.bind(this));
|
||
document.addEventListener('pointermove', this.onPointerMove.bind(this));
|
||
document.addEventListener('pointerup', this.onPointerUp.bind(this));
|
||
document.addEventListener('pointercancel', this.onPointerCancel.bind(this));
|
||
}
|
||
|
||
private onPointerDown(ev: PointerEvent) {
|
||
if (ev.pointerType !== 'touch') return;
|
||
// Map to wl_touch.down event
|
||
this.touchPoints.set(ev.pointerId, { x: ev.offsetX, y: ev.offsetY });
|
||
this.sendDown(ev.pointerId, ev.offsetX, ev.offsetY);
|
||
}
|
||
|
||
// ... etc
|
||
}
|
||
```
|
||
|
||
### Keymap FD workaround (REUSED approach)
|
||
|
||
Same as Greenfield (see `../greenfield-analysis/05-input-handling.md`):
|
||
|
||
1. Browser compiles keymap via @gfld/compositor-wasm's xkbcommon
|
||
2. `inputOutput.mkstempMmap(new Blob([utf8(keymapString)]))` opens a
|
||
`/mkstemp-mmap` WebSocket, sends the blob
|
||
3. Python proxy creates a real native shm fd via `os.memfd_create()` (replaces Greenfield's `createMemoryMappedFile` native addon)
|
||
4. Returns `ProxyFD { handle: <native_fd>, type: 'shm', host: baseURL }`
|
||
5. Browser sends `wl_keyboard.keymap(format, ProxyFD, size)`
|
||
6. Proxy deserializes, looks up native fd via `proxyFDtoNativeFD`, ships via
|
||
libwayland-server's SCM_RIGHTS
|
||
|
||
## Python side — pywayland + python-xkbcommon
|
||
|
||
### python-xkbcommon (MIT)
|
||
|
||
The `xkbcommon` PyPI package (by Stephen Early, MIT-licensed) provides
|
||
Python bindings to libxkbcommon. Useful if the proxy needs to compile
|
||
keymaps (e.g., for XWayland apps that don't speak Wayland natively).
|
||
|
||
For most use cases, the proxy doesn't need xkbcommon — it just forwards
|
||
the keymap from browser to native apps. The browser does the compilation.
|
||
|
||
### Pointer (clean-room)
|
||
|
||
Same approach as Greenfield:
|
||
- DOM `pointermove` / `pointerdown` / `pointerup` / `wheel` events
|
||
- Mapped to `ButtonEvent` / `AxisEvent` wrappers
|
||
- `InputQueue` coalesces and dispatches on next RAF
|
||
- `DefaultPointerGrab.focus()` runs on every motion: `Renderer.pickView`
|
||
walks the view stack front-to-back, returns first view whose
|
||
`Surface.isWithinInputRegion(point)` (pixman-WASM region) is true
|
||
- On change, `Pointer.setFocus` emits `wl_pointer.leave` + `frame` to old
|
||
surface, then `wl_pointer.enter` + `frame` (+ modifiers if keyboard is
|
||
grabbed) to new surface
|
||
|
||
### Pointer capture / window leave (REUSED approach)
|
||
|
||
- `canvas.setPointerCapture` on `pointerdown`
|
||
- `pointerleave` calls `notifyKeyboardFocusOut` (saves keyboard focus,
|
||
releases depressed keys, sets `needFocusInit=true` for re-sync),
|
||
resets `buttonCount=0`
|
||
- DnD suppresses this handler mid-drag
|
||
|
||
### Cursor images (REUSED approach)
|
||
|
||
CSS `cursor: url(data:...) x y, auto`:
|
||
1. Native client calls `wl_pointer.set_cursor`
|
||
2. Cursor surface commits a buffer
|
||
3. `CursorRole` draws the bitmap to a hidden canvas
|
||
4. `toDataURL()` produces a data URL
|
||
5. Set on `document.body.style.cursor`
|
||
|
||
OS composites the cursor. No rotation/scale transforms.
|
||
|
||
**NEW**: Use SVG cursor sprites (e.g., from the Phosphor icon set, MIT)
|
||
instead of bundled PNGs (Greenfield's bundled PNGs are AGPL-contaminated
|
||
since they're inside `@gfld/compositor`).
|
||
|
||
### Scroll wheel / touchpad (REUSED approach)
|
||
|
||
`WheelEvent.deltaMode`:
|
||
- `DOM_DELTA_LINE × 12` — discrete wheel clicks
|
||
- `DOM_DELTA_PAGE × surface size` — page-sized scrolls
|
||
- `DOM_DELTA_PIXEL ÷ 3` — fine-grained touchpad
|
||
|
||
Each non-zero axis → `wl_pointer.axis(time, axis, Fixed)`. Wayland version
|
||
≥5 also gets `axis_source(wheel)` + `frame()` (groups events).
|
||
|
||
### Linux button codes (REUSED approach)
|
||
|
||
DOM `event.button` (0=left, 1=middle, 2=right, 3=Back, 4=forward) mapped
|
||
to Linux input protocol button codes via the `linuxInput` table.
|
||
|
||
## Focus management (REUSED approach)
|
||
|
||
### DOM focus
|
||
|
||
The `<canvas>` owns keyboard focus permanently (`tabIndex=1`).
|
||
`Keyboard.setFocus(surface)` emits `wl_keyboard.leave` / `enter` between
|
||
native surfaces. Activation is driven by `XdgToplevel` / `ShellSurface`
|
||
activation handlers.
|
||
|
||
### DOM → Wayland focus
|
||
|
||
- Browser DOM `focus` event → `seat.notifyKeyboardFocusIn()` (restores
|
||
saved focus, re-syncs modifiers)
|
||
- Browser DOM `blur` event → `seat.notifyKeyboardFocusOut()` (saves
|
||
focus, releases depressed keys, sets `needFocusInit=true`)
|
||
|
||
### Surface-level focus
|
||
|
||
- Click on a surface → `XdgToplevel` activation request →
|
||
`desktopSurface.activate()` → `gainFocus()` (refcounted via
|
||
`focusCount`, fires `surfaceActivationUpdated` event to the user shell
|
||
on 0↔1 transitions)
|
||
- Popup grabs take keyboard + pointer focus via `seat.popupGrabStart`,
|
||
validating the serial against the keyboard/pointer `grabSerial`
|
||
|
||
## Input queue (REUSED approach)
|
||
|
||
`InputQueue.ts` coalesces events and dispatches them on the next RAF
|
||
(`InputQueue.drainQueues`). Batches bursts of input events (e.g. fast
|
||
mouse motion) into a single dispatch, avoiding redundant work.
|
||
|
||
FIFO per event type (motion, axis, key, button). Each dispatch calls the
|
||
appropriate `seat.notify*` method.
|
||
|
||
## ASCII event-flow diagram
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────────────────┐
|
||
│ BROWSER (DOM) │
|
||
│ │
|
||
│ <canvas tabIndex=1> ─┐ │
|
||
│ pointermove/dn/up ───┤── createButtonEventFromMouseEvent() ── ButtonEvent │
|
||
│ wheel ────┤── createAxisEventFromWheelEvent() ── AxisEvent │
|
||
│ keydown / keyup ────┤── createKeyEventFromKeyboardEvent() ── KeyEvent │
|
||
│ pointerdown (touch) ─┤── createTouchEventFromPointerEvent() ── TouchEvent (NEW) │
|
||
│ │ • evdevKeyCode = EvDevKeyCode[event.code] │
|
||
│ │ • x11KeyCode = evdevKeyCode + 8 │
|
||
│ │ • capsLock/numLock = event.getModifierState(...) │
|
||
│ pointerleave / blur ──┤── seat.notifyKeyboardFocusOut() (saves focus, resets mods) │
|
||
│ focus ────┴── seat.notifyKeyboardFocusIn() (restores saved focus) │
|
||
└──────────────────────────────┬──────────────────────────────────────────────────────┘
|
||
│ session.inputQueue.queue{Motion,Axis,Key,Button,Touch}
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────────────────────┐
|
||
│ BROWSER COMPOSITOR (InputQueue.drainQueues, scheduled on next animation frame) │
|
||
│ │
|
||
│ seat.notifyMotion / notifyAxis / notifyButton / notifyKey / notifyFrame / notifyTouch│
|
||
│ │ │
|
||
│ ├─► Seat.notifyKey ─► Keyboard.grab.key(event) │
|
||
│ │ └─ Keyboard.sendKey(event) │
|
||
│ │ └─ keyboardResource.key(serial,time,evdev,st) │
|
||
│ │ │
|
||
│ ├─► Seat.notifyButton ─► runButtonBindings + grab.button(event) │
|
||
│ │ └─ Pointer.sendButton │
|
||
│ │ └─ pointerResource.button(serial,time, │
|
||
│ │ linuxInput[btn], state) │
|
||
│ │ │
|
||
│ ├─► Seat.notifyMotion ─► grab.motion ─► Pointer.moveTo / sendMotion │
|
||
│ │ └─ pointerResource.motion(time, sx_f, sy_f) │
|
||
│ │ │
|
||
│ ├─► Seat.notifyAxis ─► grab.axis ─► Pointer.sendAxis │
|
||
│ │ deltaMode→scale: line=×12, page=×surface-size, pixel=÷3 │
|
||
│ │ └─ pointerResource.axis(time, axis, Fixed) │
|
||
│ │ + axisSource(wheel) + frame() if v>=5 │
|
||
│ │ │
|
||
│ ├─► Seat.notifyTouch (NEW) ─► Touch.sendDown/Motion/Up │
|
||
│ │ └─ touchResource.down/motion/up/cancel │
|
||
│ │ │
|
||
│ ├─► Seat.notifyFrame ─► pointerResource.frame() (groups events) │
|
||
│ │ │
|
||
│ └─► Seat.notifyModifiers ─ xkb.keyDown/Up(x11) ─► xkb_state_serialize_mods │
|
||
│ └─ Keyboard.sendModifiers(modsDepressed,Latched, │
|
||
│ Locked,group) │
|
||
│ │
|
||
│ Keymap path (once per WlKeyboardResource): │
|
||
│ Seat.notifyUpdateKeymap(nrmlvo) → createFromNames(rules,model,layout,variant,opt) │
|
||
│ → xkb_context_new + xkb_keymap_new_from_names (@gfld/compositor-wasm, MIT) │
|
||
│ → xkb_keymap_get_as_string ──► keymapString │
|
||
│ Keyboard.sendKeymap(resource): │
|
||
│ inputOutput.mkstempMmap(new Blob([utf8(keymapString)])) ──► ProxyFD │
|
||
│ (Python proxy creates memfd via os.memfd_create()) │
|
||
│ keyboardResource.keymap(xkbV1, ProxyFD, size) │
|
||
└──────────────────────────────┬──────────────────────────────────────────────────────┘
|
||
│ client.connection.onFlush(wireMessages)
|
||
│ ─ one big Uint32Array, FDs serialized as ProxyFD JSON
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────────────────────┐
|
||
│ WIRE (ARQChannel = KCP-over-WebSocket, custom TS, clean-room) │
|
||
│ Browser ⇄ ReconnectingWebSocket ⇄ Internet ⇄ aiohttp on proxy :8081 │
|
||
│ Message layout (Uint32Array): │
|
||
│ [0] outOfBandOpcode (0 = in-band) │
|
||
│ [1] eventSerial │
|
||
│ [2] = nFds, then for each fd: [len][JSON bytes, 4-byte aligned] │
|
||
│ then concatenated Wayland wire messages │
|
||
│ + Authorization: Bearer <jwt> │
|
||
└──────────────────────────────┬──────────────────────────────────────────────────────┘
|
||
│ NativeWaylandClientSession.onMessage (Python)
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────────────────────┐
|
||
│ PYTHON PROXY (aiohttp + pywayland) │
|
||
│ │
|
||
│ SessionController endpoints: /mkfifo /mkstemp-mmap /read-fd /close-fd │
|
||
│ /read-fd-as-stream /write-fd-as-stream │
|
||
│ /signal /channel /request-keyframe │
|
||
│ │
|
||
│ mkstempMmap(ws): │
|
||
│ receives Blob → os.memfd_create() → write bytes → return fd │
|
||
│ returns ProxyFD { handle: <native_fd>, type: 'shm', host: baseURL } │
|
||
│ │
|
||
│ onWireMessageEvents: │
|
||
│ for each ProxyFD in message: │
|
||
│ nativeFd = webFS.proxyFDtoNativeFD(proxyFD) # identity when host matches │
|
||
│ └─ if foreign host → spawn bridging pipe + WebSocket │
|
||
│ pywayland per-resource dispatcher fires │
|
||
│ # For seat events, dispatcher is a no-op (events just pass through) │
|
||
│ # libwayland-server sends via wl_resource_post_event_array │
|
||
│ flush_clients() │
|
||
└──────────────────────────────┬──────────────────────────────────────────────────────┘
|
||
│ libwayland-server sends wire msg + SCM_RIGHTS FDs
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────────────────────┐
|
||
│ NATIVE (Wayland client, sandboxed in bubblewrap) │
|
||
│ wl_display.fd readable → libwayland-client demarshals │
|
||
│ wl_keyboard.keymap(fd) → mmap(fd) → xkb_keymap_new_from_string (native xkbcommon) │
|
||
│ wl_keyboard.key / enter / leave / modifiers → client app │
|
||
│ wl_pointer.enter / motion / button / axis / frame → client app │
|
||
│ wl_touch.down / motion / up / cancel → client app (NEW: actually works) │
|
||
└──────────────────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Key answers (quick reference)
|
||
|
||
| Question | Answer |
|
||
|---|---|
|
||
| DOM→Wayland key event translation | `event.code` → EvDevKeyCode → evdev keycode → +8 = x11KeyCode. DOM `.key` glyph ignored. |
|
||
| Modifier tracking | @gfld/compositor-wasm xkbcommon state machine (REUSED MIT) |
|
||
| Focus: DOM ↔ native surfaces | `<canvas>` owns keyboard focus. `Keyboard.setFocus(surface)` emits leave/enter between native surfaces. |
|
||
| Pointer capture / window leave | `canvas.setPointerCapture` on pointerdown. `pointerleave` saves focus + releases keys. |
|
||
| Touch event mapping | **NEW: REAL implementation** via `pointerdown`/`pointermove`/`pointerup` with `pointerType === 'touch'`. Greenfield has only a stub. |
|
||
| Pointer enter/leave between surfaces | `DefaultPointerGrab.focus()` runs on every motion. `Renderer.pickView` walks view stack. |
|
||
| Pointer acceleration / cursor images | No acceleration. CSS `cursor: url(data:...) x y, auto`. NEW: SVG sprites (MIT) instead of bundled PNGs (AGPL). |
|
||
| Scroll wheel / touchpad → `wl_pointer.axis` | `WheelEvent.deltaMode`: LINE×12, PAGE×size, PIXEL÷3. v≥5 gets `axis_source(wheel)` + `frame()`. |
|
||
| WASM xkbcommon? | Yes — @gfld/compositor-wasm (MIT, REUSED). libxkbcommon 1.5.0 + embedded xkeyboard-config. |
|
||
| Keymap FD workaround for WebSocket | Browser → `/mkstemp-mmap` WebSocket → Python proxy `os.memfd_create()` → ProxyFD JSON → libwayland SCM_RIGHTS to native app. |
|
||
| Primary selection (middle-click paste)? | **NEW: implemented** via `zwp_primary_selection_device_manager_v1` (Greenfield lacks). |
|
||
|
||
## File reference
|
||
|
||
| File (custom, clean-room) | Role |
|
||
|---|---|
|
||
| `packages/compositor/src/input/Seat.ts` | `wl_seat` global |
|
||
| `packages/compositor/src/input/Keyboard.ts` | `wl_keyboard` impl |
|
||
| `packages/compositor/src/input/Pointer.ts` | `wl_pointer` impl |
|
||
| `packages/compositor/src/input/Touch.ts` | `wl_touch` impl (REAL, not stub) |
|
||
| `packages/compositor/src/input/Xkb.ts` | @gfld/compositor-wasm wrapper (REUSED MIT) |
|
||
| `packages/compositor/src/input/InputQueue.ts` | Event coalescing |
|
||
| `packages/compositor/src/input/InputOutput.ts` | FD abstraction |
|
||
| `packages/compositor/src/input/events.ts` | Event wrappers |
|
||
| `packages/compositor/src/browser/input.ts` | DOM keyboard events |
|
||
| `packages/compositor/src/browser/pointer.ts` | DOM pointer events |
|
||
| `packages/compositor/src/browser/touch.ts` | DOM touch events (NEW) |
|
||
|
||
### Reused (MIT)
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `libs/compositor-wasm/src/index.ts` | xkbcommon + pixman WASM (REUSED) |
|
||
|
||
### Python side
|
||
|
||
| Component | Role |
|
||
|---|---|
|
||
| `pywayland.server.Client` | wl_client wrapper |
|
||
| `pywayland.server.Resource` | wl_resource base (per-resource dispatchers) |
|
||
| `xkbcommon` PyPI package | Optional, only needed for XWayland keymap compilation |
|