# 03 — HTML5 Rendering Pipeline (Custom WebGL2 + Workers + OffscreenCanvas) This chapter describes the **custom browser-side rendering pipeline** that replaces Greenfield's AGPL `@gfld/compositor`. It's the largest single component of the MIT approach (~3,000 LOC TypeScript, ~5 dev-weeks). ## Critical: clean-room requirement **You CANNOT read Greenfield's `@gfld/compositor` source and reimplement it.** Even "inspired-by" code is risky under the *Computer Associates v. Altai* Abstraction-Filtration-Comparison test. AGPL §13 network-use clause would force you to open-source your entire Odoo fork if a court finds your compositor is an AGPL derivative. **Right approach**: True clean-room. - Person A (spec author, may read AGPL source) writes a behavioral spec (inputs/outputs/edge cases, NO code). - Person B (implementer, never opens `packages/compositor/src/`) implements from spec + MIT references only: - Wayland protocol spec (https://wayland.freedesktop.org/docs/html/) - wlroots C source (MIT, https://gitlab.freedesktop.org/wlroots/wlroots) - Weston C source (MIT, https://gitlab.freedesktop.org/wayland/weston) - Smithay Rust source (MIT, https://github.com/Smithay/smithay) - WebGL2/WebCodecs/KCP MDN docs Maintain `SPEC.md` and `CLEAN_ROOM_LOG.md` as legal-defensibility artifacts. ## Modernization wins over Greenfield Greenfield's browser compositor uses: - WebGL1 (no OffscreenCanvas, no WebGL2) - Main-thread WebGL (jank on heavy frames) - WebCodecs with hardware decode explicitly disabled (`// FIXME hardware decoding frame offset is wrong`) - FFmpeg Wasm fallback (GPL — we can't use it) Our MIT approach modernizes to: - **WebGL2** — uniform buffer objects, transform feedback, simplified YUV→RGB shaders - **OffscreenCanvas + Web Workers** — render in a worker, never block main thread - **WebCodecs with hardware decode enabled** — fix Greenfield's "frame offset is wrong" bug - **OpenH264 BSD Wasm fallback** — replaces Greenfield's GPL FFmpeg Wasm - **Adaptive bitrate** — consume `EncoderFeedback` data Greenfield produces but doesn't use - **Real touch input** — Greenfield has a 46-line stub that never fires ## Proposed file structure ``` packages/compositor/ # PROPRIETARY, clean-room TS ├── SPEC.md # clean-room behavioral spec ├── CLEAN_ROOM_LOG.md # who read AGPL, who didn't, when ├── src/ │ ├── index.ts # public API: createCompositorSession() │ ├── core/ │ │ ├── Session.ts # top-level session │ │ ├── Compositor.ts # wl_compositor global │ │ └── Globals.ts # Wayland globals registry │ ├── surface/ # ~1,200 LOC, 3 weeks │ │ ├── Surface.ts # wl_surface, double-buffered state │ │ ├── SurfaceState.ts # state struct │ │ ├── SurfaceRole.ts # role abstraction │ │ ├── SurfaceTree.ts # parent/child tree │ │ ├── Subsurface.ts # wl_subsurface │ │ ├── Subcompositor.ts # wl_subcompositor global │ │ ├── Region.ts # wl_region (pixman-WASM) │ │ └── FrameCallback.ts # wl_callback │ ├── render/ # ~1,800 LOC, 3 weeks │ │ ├── Renderer.ts # 4-phase render pipeline │ │ ├── Scene.ts # one per output (canvas) │ │ ├── View.ts # positioned surface for compositing │ │ ├── Output.ts # wl_output, OffscreenCanvas binding │ │ ├── RenderState.ts # per-view-per-scene texture + transform │ │ ├── Texture.ts # WebGL2 texture wrapper │ │ ├── ShaderProgram.ts # WebGL2 program wrapper │ │ ├── Loop.ts # requestAnimationFrame integration │ │ └── shaders/ │ │ ├── YUVA2RGBA.glsl # BT.601 YUVA → RGBA │ │ ├── YUV2RGB.glsl # YUV → RGB (no alpha) │ │ ├── RGBandA2RGBA.glsl # RGB+Alpha → RGBA │ │ └── SceneShader.glsl # final pass: textured quads │ ├── buffer/ # ~600 LOC, 1.5 weeks │ │ ├── BufferContents.ts # buffer abstraction (mimeType) │ │ ├── BufferImplementation.ts # interface │ │ ├── Shm.ts # wl_shm, BMP-header hack │ │ ├── ImageBitmapBuffer.ts # web-bitmapbuf fast path │ │ └── EncodedFrameBuffer.ts # H.264/PNG from proxy │ ├── transport/ # ~3,500 LOC, 4 weeks │ │ ├── Channel.ts # ARQChannel + SimpleChannel │ │ ├── kcp.ts # ported fresh from MIT C ref │ │ ├── ReconnectingWebSocket.ts # vendored from pladaria (MIT) │ │ ├── BufferStream.ts # serial → frame promise resolution │ │ ├── StreamingBuffer.ts # per-surface encoded-frame buffer │ │ ├── EncodedFrame.ts # wire format parser │ │ ├── EncoderFeedback.ts # 4-byte decode feedback, 1 Hz │ │ ├── RemoteInputOutput.ts # FD-over-WebSocket for native apps │ │ ├── RemoteAppLauncher.ts # channel setup per app │ │ ├── RemoteOutOfBandChannel.ts # OOB opcode dispatch │ │ └── decoders/ │ │ ├── FrameDecoder.ts # interface │ │ ├── WebCodecsFrameDecoder.ts # primary: VideoDecoder (HW-accelerated) │ │ └── OpenH264FrameDecoder.ts # fallback: OpenH264 BSD Wasm │ ├── input/ # ~2,500 LOC, 3 weeks │ │ ├── Seat.ts # wl_seat global │ │ ├── Keyboard.ts # wl_keyboard, keymap, modifiers │ │ ├── Pointer.ts # wl_pointer, motion, buttons, axis │ │ ├── Touch.ts # wl_touch (REAL implementation, not stub) │ │ ├── Xkb.ts # xkbcommon WASM wrapper │ │ ├── InputQueue.ts # event coalescing + RAF dispatch │ │ ├── InputOutput.ts # FD abstraction │ │ └── events.ts # KeyEvent, ButtonEvent, AxisEvent │ ├── roles/ # ~1,800 LOC, 3 weeks (HIGH RISK) │ │ ├── XdgWmBase.ts # xdg_wm_base global, ping │ │ ├── XdgSurface.ts # xdg_surface, configure/ack │ │ ├── XdgToplevel.ts # xdg_toplevel, window state │ │ ├── XdgPopup.ts # xdg_popup, transient windows, grabs │ │ ├── XdgPositioner.ts # popup positioning math │ │ ├── Shell.ts # legacy wl_shell (optional) │ │ └── ShellSurface.ts # legacy wl_shell_surface │ ├── data/ # ~600 LOC, 1.5 weeks │ │ ├── DataDeviceManager.ts # wl_data_device_manager │ │ ├── DataSource.ts # wl_data_source │ │ ├── DataOffer.ts # wl_data_offer │ │ ├── WaylandDataSource.ts # native source wrapper │ │ ├── PrimarySelectionDeviceManager.ts # NEW: zwp_primary_selection_v1 │ │ └── PrimarySelectionSource.ts # NEW: middle-click paste │ ├── desktop/ # ~800 LOC, 1.5 weeks │ │ ├── Desktop.ts # Floating vs AlwaysFullScreen │ │ ├── DesktopSurface.ts # abstraction │ │ ├── FloatingDesktopSurface.ts # default mode │ │ └── FullscreenDesktopSurface.ts # kiosk mode │ ├── math/ # ~500 LOC, 0.5 weeks │ │ ├── Mat4.ts # 4×4 matrix │ │ ├── Vec4.ts # 4-vector │ │ ├── Point.ts # 2D point │ │ ├── Size.ts # 2D size │ │ ├── Rect.ts # 2D rectangle │ │ └── Transformations.ts # affine transforms │ ├── userapi/ # ~120 LOC, 0.5 weeks │ │ └── UserShellApi.ts # OWL shell integration seam │ ├── browser/ # ~750 LOC, 2 weeks │ │ ├── input.ts # DOM keyboard events │ │ ├── pointer.ts # DOM pointer events │ │ ├── dnd.ts # DnD bridge │ │ ├── selection.ts # clipboard bridge (native → DOM) │ │ ├── text-selection.ts # Firefox fallback (text only) │ │ ├── attributes.ts # canvas attributes │ │ └── capabilities.ts # browser feature detection │ └── (optional, Phase 6) │ ├── webapp/ # ~500 LOC, 1 week (WASM web apps) │ │ ├── WebAppLauncher.ts │ │ ├── WebConnectionHandler.ts │ │ ├── WebInputOutput.ts │ │ └── WebBuffersFactory.ts │ └── xwayland/ # ~5,000 LOC, 8 weeks (X11 apps) │ ├── XWindowManager.ts │ ├── XWindow.ts │ ├── XWindowFrame.ts │ └── ... (uses @gfld/xtsb MIT) └── test/ ├── unit/ # ~1,000 LOC, Jest unit tests ├── protocol/ # ~2,000 LOC, Weston test clients ported to TS └── e2e/ # ~500 LOC, Playwright E2E with real Linux apps ``` ## Modern rendering pipeline (WebGL2 + OffscreenCanvas + Workers) ``` ┌─────────────────────────────────────────────────────────────────────────┐ │ MAIN THREAD (Odoo OWL shell + DOM events) │ │ │ │ element owned by OWL component │ │ │ │ │ │ canvas.transferControlToOffscreen() → OffscreenCanvas │ │ │ │ │ │ postMessage({canvas: offscreenCanvas}, [offscreenCanvas]) │ │ ▼ │ │ ┌────────────────────────────────────────────────────────────────┐ │ │ │ Render Worker (WebGL2 + OffscreenCanvas) │ │ │ │ │ │ │ │ Receives: │ │ │ │ - OffscreenCanvas (transferred) │ │ │ │ - EncodedVideoChunk / ImageBitmap (transferred from decode) │ │ │ │ - Surface state updates (postMessage) │ │ │ │ - Frame callback requests (postMessage) │ │ │ │ │ │ │ │ Runs the 4-phase render pipeline: │ │ │ │ PHASE 1 — pixel upload (per view per scene) │ │ │ │ PHASE 2 — requestAnimationFrame │ │ │ │ PHASE 3 — draw (per scene: viewport + clear + drawArrays) │ │ │ │ PHASE 4 — fire frame callbacks │ │ │ │ │ │ │ │ Sends: │ │ │ │ - Frame callback timestamps (postMessage to main thread) │ │ │ │ │ │ │ └────────────────────────────────────────────────────────────────┘ │ │ │ │ ┌────────────────────────────────────────────────────────────────┐ │ │ │ Decode Worker (WebCodecs VideoDecoder, hardware-accelerated) │ │ │ │ │ │ │ │ Receives: │ │ │ │ - EncodedVideoChunk (transferred from transport) │ │ │ │ │ │ │ │ Decodes: │ │ │ │ - VideoDecoder.decode(chunk) │ │ │ │ - VideoFrame output (zero-copy GPU texture source) │ │ │ │ │ │ │ │ Sends: │ │ │ │ - VideoFrame (transferred to render worker) │ │ │ │ - Decode duration telemetry (for adaptive bitrate) │ │ │ │ │ │ │ └────────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────────┘ ``` ## Why WebGL2 over WebGL1 - **Uniform Buffer Objects** — share state between shaders efficiently - **Transform feedback** — could be used for vertex skinning (not needed for compositor) - **Simplified YUV→RGB shaders** — `texelFetch` for direct texel access - **3D textures** — could simplify YUV planar upload (Y, U, V as 3 slices) - **Better browser support** — 97%+ global browser support per caniuse ## Why OffscreenCanvas + Web Workers - **Never block main thread** — heavy decode + render happens off-thread - **Smooth UI** — Odoo shell stays responsive during video playback - **Multi-output scaling** — one worker per output for parallel rendering - **Recovery** — if render worker crashes, restart without losing Odoo session ## WebCodecs vs FFmpeg Wasm | Aspect | WebCodecs (primary) | OpenH264 Wasm (fallback) | |---|---|---| | License | W3C standard (MIT-equivalent) | BSD-2-Clause | | Hardware decode | Yes (Intel/AMD/NVIDIA) | No (CPU only) | | Latency | ~5-10ms per frame | ~30-50ms per frame | | CPU usage | Low (offloaded to GPU) | High (full decode on CPU) | | Browser support | Chrome 94+, Edge 94+, Safari 16+, Firefox 130+ | All browsers with Wasm | | Codecs | H.264, VP9, AV1, H.265 (where supported) | H.264 only | **Recommended approach**: Probe `VideoDecoder.isConfigSupported()` at session startup. If supported, use WebCodecs. Otherwise, fall back to OpenH264 Wasm (loaded lazily, only when needed). ## Wire format for encoded frames (REUSED from Greenfield) Same wire format as Greenfield (see `../greenfield-analysis/09-hardware-accel.md` for byte layout). The browser side parses `{contentSerial, mimeType, size, encodedSize, opaque, alpha?}` and dispatches to the right decoder. ## Engineering observations 1. **Vestigial damage tracking in Greenfield** — `damageRects` are accumulated but not consumed by the renderer. Our MIT approach should actually use them for scissoring (per-rect partial repaints). 2. **Multi-output texture cost** — each view has one texture per scene. Our approach should share textures across scenes where possible. 3. **SHM xrgb8888 BMP-header hack** — cleverly offloads pixel conversion to the browser's BMP decoder. Worth re-implementing. 4. **Encoder feedback telemetry** — Greenfield produces it but doesn't use it. Our approach should consume it for adaptive bitrate. 5. **Cursor bypass** — cursor is rendered as CSS `cursor: url(dataURL)`, not via WebGL. Same approach in our MIT version. 6. **Frame-callback coalescing** — multiple `wl_surface.frame` requests within one render pass fire on the same RAF tick. Same in our version. 7. **Subsurface sync cascade** — three-layer mechanism (cachedState, commitCache, effectiveSync). Hard to get right; clean-room from spec. ## Effort estimate | Component | LOC | Weeks (1 dev) | Risk | |---|---|---|---| | Surface state stack | 1,200 | 3 | Med | | Rendering pipeline (WebGL2 + OffscreenCanvas) | 1,800 | 3 | Med | | Buffers (SHM + BMP hack) | 600 | 1.5 | Low | | Transport (incl. fresh KCP port from MIT C ref) | 3,500 | 4 | **High** | | Input (seat/kbd/ptr/touch/xkb) | 2,500 | 3 | Med | | Roles (xdg-shell state machine) | 1,800 | 3 | **High** | | Data device / clipboard / DnD + primary selection | 600 | 1.5 | Low | | Desktop policy | 800 | 1.5 | Low | | Math | 500 | 0.5 | Low | | User shell API | 120 | 0.5 | Low | | Browser glue | 750 | 2 | Med | | H.264 decoder (WebCodecs + OpenH264 BSD) | 500 | 1 | Low | | Core (Session/Globals) | 200 | 0.5 | Low | | **MVP TOTAL** | **~13,400** | **~25 weeks** | | | XWayland (optional Phase 6) | 5,000 | 8 | High | | WASM web app (optional) | 500 | 1 | Low | | Test suite + hardening | — | 3 | — | | **GRAND TOTAL** | **~18,900** | **~30-36 weeks** | | With 2 senior devs in parallel: **~15-18 weeks for MVP**.