# 09 — Hardware Acceleration, Encoding, WebCodecs, GPU Textures This chapter covers the encoding/decoding/WASM pipeline in the MIT approach. **Critical changes from Greenfield**: - **No x264enc** (GPL-2.0+) — replaced with `vaapih264enc`/`nvh264enc`/`openh264enc` - **No FFmpeg Wasm** (GPL-3.0 in Greenfield's build) — replaced with WebCodecs primary + OpenH264 BSD fallback - **Hardware decode enabled** (Greenfield disables it with a FIXME) - **Adaptive bitrate** (consumes `EncoderFeedback` data Greenfield produces but doesn't use) ## What's reused (MIT) ### Browser side | Component | License | Role | |---|---|---| | `@gfld/compositor-wasm` | MIT | pixman 0.42.2 + xkbcommon 1.5.0 WASM (region math + keymap) | ### Vendored C (MIT, used in Python proxy via cffi) | File | Lines | Role | |---|---|---| | `westfield-dmabuf.c` | ~200 | DMA-BUF import via EGL/GBM | | `westfield-egl.c` | ~900 | EGL zero-copy setup (EGLDisplay, EGLContext, EGLImage) | | `wlr_drm.c` | ~244 | `wl_drm` global impl | | `wlr_linux_dmabuf_v1.c` | ~1,110 | `zwp_linux_dmabuf_v1` global impl | | `drm_format_set.c` | ~239 | DRM format helpers | ## What's replaced (proprietary, clean-room) ### Browser side | Component | Replaces | Why | |---|---|---| | Custom `WebCodecsFrameDecoder.ts` | Greenfield's `webcodec-buffer-decoder.ts` (AGPL) | Clean-room; **enables hardware decode** (Greenfield disables with FIXME) | | Custom `OpenH264FrameDecoder.ts` | Greenfield's `wasm-buffer-decoder.ts` + `H264NALDecoder.worker.ts` (AGPL) + `@gfld/compositor-ffmpeg-h264` (GPL) | Clean-room; uses **OpenH264 BSD Wasm** instead of FFmpeg GPL Wasm | | Custom `YUVA2RGBA` / `RGBXandA2RGBA` shaders | Greenfield's (AGPL) | Clean-room; **WebGL2** (Greenfield uses WebGL1) | | Custom `Renderer` / `Scene` / `SceneShader` | Greenfield's (AGPL) | Clean-room; **OffscreenCanvas + Web Workers** (Greenfield is main-thread) | ### Python side | Component | Replaces | Why | |---|---|---| | Custom `gst_encoder.py` (~700 LOC) | Greenfield's `proxy-encoding-addon.node` (AGPL, 290 lines C) + `libproxy-encoding.so` (AGPL, ~1400 lines C) | Pure Python via PyGObject. Same GStreamer, no N-API addon. | | `vaapih264enc` / `nvh264enc` / `openh264enc` | `x264enc` (GPL) | GPL-free H.264 encoding | ## Encoder selection (per-frame, deterministic) Same logic as Greenfield, but **without x264enc**: | name | element | when used | split_alpha | coded-size multiple | License | |---|---|---|---|---|---| | `vaapih264` | `vaapih264enc` | >256×256 + ARGB/XRGB/dmabuf, Intel/AMD GPU | true (parallel alpha H.264 stream) | 128 | LGPL + MIT (libva) | | `nvh264` | `nvh264enc` | same, NVIDIA GPU | true | 128 | BSD plugin + proprietary NVENC blob | | `openh264` | `openh264enc` | same, no GPU (CPU fallback) | true | 128 | BSD-2-Clause (libopenh264) | | `png` | `pngenc` | ≤256×256 + ARGB/XRGB/dmabuf | false (PNG has native alpha) | 16 | LGPL | **Rule**: pixel_count ≤ 256×256 → PNG; otherwise the configured H.264 encoder (selected by `--encoder=vaapih264|nvh264|openh264`). **NOTE**: This means the **default encoder is hardware-accelerated** (vaapih264 on Intel/AMD, nvh264 on NVIDIA). Greenfield defaults to x264 (software, GPL). ## Canonical H.264 pipeline (Python equivalent) Same GStreamer pipeline as Greenfield, ported to PyGObject: ```python # gst_encoder.py (clean-room, ~700 LOC) import gi gi.require_version("Gst", "1.0") gi.require_version("GstGL", "1.0") gi.require_version("GstEGL", "1.0") # if available from gi.repository import Gst, GstGL, GLib class H264Encoder: def __init__(self, egl_display_ptr, egl_context_ptr, encoder_type="vaapih264"): # Create GLDisplay + GLContext from existing EGL context (zero-copy) self.gl_display = GstGL.GLDisplayEGL.new_with_egl_display(egl_display_ptr) self.gl_context = GstGL.GLContext.new_wrapped( self.gl_display, egl_context_ptr, GstGL.GLPlatform.EGL, GstGL.GLLAPI.GLES2, ) # Encoder pipeline encoder_element = { "vaapih264": "vaapih264enc", "nvh264": "nvh264enc", "openh264": "openh264enc", }[encoder_type] self.pipeline = Gst.parse_launch( f"appsrc name=src ! " f"glupload ! glcolorconvert ! " f"glshader name=scale ! " # pads to macroblock multiple f"glcolorconvert ! video/x-raw(memory:GLMemory),format=NV12 ! " f"gldownload ! queue ! " f"{encoder_element} tune=zerolatency bitrate=12000 vbv-buf-capacity=160 " f"bframes=0 rc-lookahead=0 sliced-threads byte-stream=true ! " f"video/x-h264,profile=high,stream-format=byte-stream,alignment=au ! " f"appsink name=sink" ) # Wire up appsink callback (zero-copy external Buffers) sink = self.pipeline.get_by_name("sink") sink.emit("set-property", "emit-signals", True) sink.connect("new-sample", self._on_new_sample) # Alpha pipeline (separate, for transparent windows) self.alpha_pipeline = self._build_alpha_pipeline(encoder_element) # GLib MainLoop in separate thread (GStreamer requires it) self.glib_loop = GLib.MainLoop() self.glib_context = GLib.MainContext.new() GLib.MainContext.push_thread_default(self.glib_context) import threading self.glib_thread = threading.Thread(target=self.glib_loop.run, daemon=True) self.glib_thread.start() def encode_frame(self, buffer_id, content_serial, creation_serial, dma_buf_fd, w, h): # Push buffer into appsrc (zero-copy via GstBuffer wrapping the dmabuf) # ... marshalled from asyncio via GLib.idle_add def _on_new_sample(self, appsink): sample = appsink.emit("pull-sample") buffer = sample.get_buffer() # Marshal back to asyncio asyncio.run_coroutine_threadsafe( self._deliver_frame(buffer, self.current_buffer_id), self.asyncio_loop, ) ``` ### Bitrates (same as Greenfield) - Opaque: 12000 (vaapih264 / nvh264 / openh264) - Alpha: 1200 (10% of opaque) ### Adaptive bitrate (NEW — Greenfield lacks) Consume `EncoderFeedback` data (refreshInterval + decode duration): ```python # adaptive_bitrate.py (clean-room, ~150 LOC) class AdaptiveBitrate: def __init__(self, encoder): self.encoder = encoder def on_feedback(self, refresh_interval_ms, avg_decode_ms): # If decode is taking too long, reduce bitrate if avg_decode_ms > refresh_interval_ms * 0.8: new_bitrate = max(self.current_bitrate * 0.7, MIN_BITRATE) self.encoder.set_property("bitrate", new_bitrate) # If decode is fast and we have headroom, increase bitrate elif avg_decode_ms < refresh_interval_ms * 0.3: new_bitrate = min(self.current_bitrate * 1.3, MAX_BITRATE) self.encoder.set_property("bitrate", new_bitrate) ``` ## Wire format (REUSED from Greenfield) Same binary format as Greenfield (see `../greenfield-analysis/09-hardware-accel.md`): ``` struct packed props { u32 buffer_id, creation_serial, content_serial, u32 frame_encoding_type (0=h264, 1=png), u32 width, height, coded_width, coded_height } u32 opaque_length u8 opaque[opaque_length] // H.264 AU or PNG blob u32 alpha_length // 0 if no alpha split u8 alpha[alpha_length] // H.264 AU of alpha-only stream ``` ## Browser decode (clean-room TS) ### Preferred path: WebCodecs `VideoDecoder` (with HW decode enabled) ```typescript // WebCodecsFrameDecoder.ts (clean-room, ~250 LOC) export class WebCodecsFrameDecoder implements FrameDecoder { private opaqueDecoders: Map = new Map(); private alphaDecoders: Map = new Map(); async isSupported(): Promise { const config = { codec: 'avc1.64001f', optimizeForLatency: true, // NEW: enable hardware acceleration (Greenfield disables this) hardwareAcceleration: 'prefer-hardware', }; const result = await VideoDecoder.isConfigSupported(config); return result.supported; } async decode(surface: Surface, encodedFrame: EncodedFrame): Promise { let opaqueDecoder = this.opaqueDecoders.get(surface.id); if (!opaqueDecoder) { opaqueDecoder = new VideoDecoder({ output: (frame) => this.onDecoded(surface, frame), error: (e) => this.onError(surface, e), }); opaqueDecoder.configure({ codec: 'avc1.64001f', optimizeForLatency: true, hardwareAcceleration: 'prefer-hardware', // NEW }); this.opaqueDecoders.set(surface.id, opaqueDecoder); } const chunk = new EncodedVideoChunk({ type: encodedFrame.isKeyframe ? 'key' : 'delta', data: encodedFrame.opaque, timestamp: encodedFrame.timestamp, }); opaqueDecoder.decode(chunk); // Alpha path (if present) if (encodedFrame.alpha) { // Same pattern for alphaDecoder } } } ``` **Key difference from Greenfield**: We use `hardwareAcceleration: 'prefer-hardware'` (Greenfield uses `'prefer-software'` with a FIXME about "frame offset is wrong"). This needs testing, but should give significant CPU savings. ### Fallback path: OpenH264 BSD Wasm (NOT FFmpeg — FFmpeg is GPL) ```typescript // OpenH264FrameDecoder.ts (clean-room, ~250 LOC) // Uses OpenH264 BSD-2-Clause Wasm build (Cisco's binary release or self-built) // Available from: https://github.com/cisco/openh264 (BSD-2-Clause) ``` OpenH264 is BSD-2-Clause — fully proprietary-compatible. Less featureful than FFmpeg (no VP9/AV1), but sufficient for H.264 fallback. ## WebGL2 upload (clean-room TS, modernized) WebGL2 with `desynchronized:true` + OffscreenCanvas in Web Worker. Per Scene (canvas/output), three GL helpers (same as Greenfield, but WebGL2 instead of WebGL1): ### `YUVA2RGBA` FBO pass: uploads Y/U/V/alphaY as **4× LUMINANCE textures** (WebGL1 compatible) OR **1× R8UI texture array** (WebGL2, more efficient), renders into the surface's RGBA texture via `YUVA2RGBAShader` (BT.601 coefficients). ### `RGBXandA2RGBA` FBO pass for the `DualPlaneRGBA*` WebCodecs outputs (opaque + alpha planes composited via `RGBandA2RGBAShader`). ### `SceneShader` Final pass: draws every view's RGBA texture to the canvas with `gl.blendFunc(ONE, ONE_MINUS_SRC_ALPHA)`. **NEW**: WebGL2 `texStorage2D` for immutable textures (faster upload). **NEW**: `VideoFrame` directly as a WebGL texture source (zero-copy from WebCodecs to GPU, no `copyTo` needed). ## End-to-end pipeline ``` ┌──────────────────── NATIVE CLIENT PROCESS ────────────────────┐ │ App renders via GTK/Qt/raw EGL into │ │ ┌────────────────────────────────────┐ │ │ │ wl_shm buffer (ARGB8888/XRGB8888)│ ─── or ─────────┐ │ │ │ zwp_linux_dmabuf_v1 (DMA-BUF) │ │ │ │ │ wl_drm create_prime_buffer │ │ │ │ └────────────────────────────────────┘ │ │ └────────────────────────┬──────────────────────────────────┼────┘ │ Wayland socket (UNIX domain) │ ▼ │ ┌────────────────────── PYTHON PROXY (aiohttp + pywayland + GStreamer)─┐ │ pywayland (Apache-2.0) │ │ └─ Display → Client → Resource (buffer) │ │ └─ display.init_shm(), init_drm("/dev/dri/renderD128") │ │ └─ Vendored westfield-egl.c (MIT, via cffi): │ │ EGLDevice + EGLDisplay + EGLContext │ │ │ │ gst_encoder.py (~700 LOC Python, clean-room) │ │ ┌─ frame_buffer {type: SHM|DMA, w, h, fmt} │ │ │ │ │ │ GLib MainLoop in separate thread (GLib.idle_add) │ │ │ │ │ │ │ ▼ │ │ │ gst_parse_launch: │ │ │ ┌─ Encoder selection (per-frame): │ │ │ │ pixel_count ≤ 256×256 ─► PNG pipeline │ │ │ │ > 256×256 + ARGB/XRGB/dmabuf ─► configured h264 │ │ │ │ │ │ │ │ H.264 pipelines (parallel, NO x264 — GPL): │ │ │ │ opaque: appsrc → glupload → glcolorconvert → │ │ │ │ glshader(scale→coded_size) → │ │ │ │ glcolorconvert(NV12) → gldownload → │ │ │ │ queue → vaapih264enc|nvh264enc|openh264enc │ │ │ │ → byte-stream AU → appsink │ │ │ │ alpha: same source, fragment shader │ │ │ │ `vec4(pix.a,pix.a,pix.a,0.)` → │ │ │ │ second H.264 stream │ │ │ │ │ │ │ │ PNG pipeline (single): │ │ │ │ appsrc → glupload → glcolorconvert → glshader → │ │ │ │ RGBA → gldownload → pngenc → appsink │ │ │ │ │ │ │ └─ asyncio.run_coroutine_threadsafe for cross-thread │ │ │ │ │ └─ surface_buffer_encoding.py │ │ └─ frame_data_channel.send(buffer) │ │ │ │ channel.py: │ │ ARQChannel (kcp PyPI package, MIT, over WebSocket) │ │ SimpleChannel (raw WebSocket, for FEEDBACK only) │ │ ▲ │ │ │ frame_feedback.py: 16-bit rAF interval + 16-bit avg │ │ │ decode duration → used to pace wl_frame.done AND │ │ │ drive adaptive_bitrate.py (NEW vs Greenfield) │ └───────┼───────────────────────────────────────────────────────┘ │ 4 logical channels per client: PROTOCOL / FRAME / XWM / FEEDBACK ▼ ┌────────────────────── BROWSER (Custom Compositor TS, clean-room) ──────┐ │ WebSocket → ReconnectingWebSocket (vendored MIT) │ │ ↓ │ │ channel.ts: ARQChannel.input(kcp) → onMessage │ │ ↓ │ │ RemoteAppLauncher.setupFrameDataChannel → │ │ BufferStream.deliverContentToBufferStream (parse serials) │ │ ↓ │ │ StreamingBuffer.getContents(surface, serial) │ │ ↓ │ │ Session.frameDecoder.decode(surface, encodedFrame) │ │ │ │ │ ├─ PNG path (both decoders): │ │ │ new Blob([opaque], {type:'image/png'}) → │ │ │ createImageBitmap(blob, sx, sy, w, h) // crop padding │ │ │ → SinglePlane { bitmap: ImageBitmap } │ │ │ │ │ └─ H.264 path: │ │ (preferred) WebCodecs VideoDecoder │ │ config: avc1.64001f, prefer-hardware (NEW!), │ │ optimizeForLatency │ │ opaqueDecoder + alphaDecoder per surface │ │ EncodedVideoChunk {type:key|delta, data:nal} │ │ → VideoFrame.output → │ │ if I420: copyTo → DualPlaneYUVAArrayBuffer │ │ else: keep VideoFrame → DualPlaneRGBAVideoFrame │ │ (fallback) OpenH264 BSD Wasm in 2 Web Workers │ │ (NOT FFmpeg — GPL contamination) │ │ worker.postMessage(nal, [nal.buffer]) // transferable │ │ OpenH264._decode → I420 planes │ │ postMessage back with [y.buffer, u.buffer, v.buffer] │ │ → DualPlaneYUVASplitBuffer │ │ │ │ Renderer.render() (rAF-driven, in Web Worker via OffscreenCanvas)│ │ ↓ │ │ Scene.render(viewStack) │ │ ├─ Scene['image/png'] → texImage2D(ImageBitmap) │ │ ├─ Scene['image/bitmap'] (web-bitmapbuf path) → texImage2D │ │ └─ Scene['video/h264']: │ │ YUVA2RGBA.convertYUVASplitBufferInto / │ │ convertYUVAArrayBufferInto: │ │ Y/U/V/alphaY → 4× LUMINANCE textures (WebGL1 compat) │ │ OR 1× R8UI texture array (WebGL2, more efficient) │ │ bind FBO → renderState.texture (RGBA) as color attach │ │ YUVA2RGBAShader (BT.601) → drawArrays → RGBA in tex │ │ or RGBXandA2RGBA.convertInto (WebCodecs RGBA): │ │ VideoFrame/ImageBitmap → texImage2D directly │ │ ↓ │ │ SceneShader.draw(): drawArrays with surface's RGBA texture, │ │ blend gl.ONE / gl.ONE_MINUS_SRC_ALPHA, desynchronized canvas │ │ ↓ │ │ Browser composites canvas into the page │ └─────────────────────────────────────────────────────────────────┘ Same-frame web app fast path (web-bitmapbuf, REUSED): ┌───────────────┐ MessagePort.postMessage([view, ...fds], ┌──────────────┐ │ Web app │ ──► [imageBitmap]) ──────────────────────────► │ Compositor │ │ (iframe) │ web_bitmapbuf_factory.create_buffer(id, fd) │ WebConnHandl │ └───────────────┘ → wl_buffer → ImageBitmapBuffer └──────────────┘ (no encoding, no WebSocket, no GStreamer) ``` ## Key answers (quick reference) ### Why H.264 over PNG? H.264 at 12 Mbps × 1/60s ≈ 25 KB/frame for 1080p, vs PNG 100 KB – 3 MB. PNG wins for ≤256×256 because H.264 I-frame overhead and alpha-split doubling dominate at small sizes. ### WebCodecs? Yes — preferred path. **Hardware decode enabled** (Greenfield disables with FIXME). OpenH264 BSD Wasm is the fallback (NOT FFmpeg — GPL). ### Trade-offs? - H.264 = small wire / decode cost - PNG = medium wire / browser-native fast decode (`createImageBitmap`) - raw SHM = **not used** between proxy and browser — every surface commit is encoded. ### Per-surface policy? No — per-frame deterministic. Only configurable knob is `--encoder=vaapih264|nvh264|openh264`. PNG is always automatic fallback. ### Cursors? Always PNG (32×32 ≤ 256×256). Decoded to ImageBitmap and set as the CSS cursor via `setBrowserCursor` — never enters the WebGL compositor. ### Alpha pre-multiplication? Implicit in shaders (same as Greenfield). Works acceptably for typical Wayland content. ### OffscreenCanvas / Web Workers? **Yes** — Render Worker (WebGL2 + OffscreenCanvas) + Decode Worker (WebCodecs VideoDecoder). Greenfield uses only 2 workers for the WASM fallback path; our MIT approach uses Workers for ALL rendering. ### Adaptive bitrate (NEW)? **Yes** — `adaptive_bitrate.py` consumes `EncoderFeedback` data and adjusts `vaapih264enc`/`nvh264enc`/`openh264enc` bitrate at runtime. Greenfield produces the feedback data but doesn't use it. ## Notable design trade-offs for the MIT approach 1. **Hardware-accelerated encoding by default** (vaapih264 on Intel/AMD, nvh264 on NVIDIA). Greenfield defaults to x264 (software, GPL). 2. **Hardware-accelerated decoding by default** (WebCodecs `prefer-hardware`). Greenfield uses `prefer-software` with a FIXME. 3. **No GPL anywhere in the encoding/decoding pipeline** (no x264, no FFmpeg Wasm). 4. **Adaptive bitrate** consumes the feedback data Greenfield ignores. 5. **OpenH264 BSD Wasm** as fallback (Cisco's binary release or self-built). Sufficient for H.264 fallback when WebCodecs is unavailable. 6. **OffscreenCanvas + Web Workers** for all rendering (Greenfield is main-thread). 7. **WebGL2** instead of WebGL1 (uniform buffer objects, simplified shaders). 8. **VideoFrame directly as WebGL texture source** (zero-copy from WebCodecs to GPU, no `copyTo` needed). 9. **Same alpha-split H.264 approach** as Greenfield (doubles cost for transparent windows) — could be replaced with AV1 with native alpha support in the future. 10. **Future: AV1/VP9 support** via WebCodecs (when browser support is widespread) — sidesteps H.264 MPEG-LA patent royalties entirely. ## File reference ### Browser side (custom, clean-room TS) | Path | Role | |---|---| | `packages/compositor/src/transport/decoders/WebCodecsFrameDecoder.ts` | WebCodecs VideoDecoder (HW-accelerated) | | `packages/compositor/src/transport/decoders/OpenH264FrameDecoder.ts` | OpenH264 BSD Wasm fallback | | `packages/compositor/src/transport/decoders/FrameDecoder.ts` | Decoder interface | | `packages/compositor/src/transport/EncodedFrame.ts` | Wire format parser for encoded frames | | `packages/compositor/src/transport/EncoderFeedback.ts` | Sends 4-byte decode feedback at 1 Hz | | `packages/compositor/src/transport/StreamingBuffer.ts` | Per-surface encoded-frame buffer | | `packages/compositor/src/transport/BufferStream.ts` | Manages serial → frame promise resolution | | `packages/compositor/src/render/Renderer.ts` | 4-phase render pipeline (WebGL2 + OffscreenCanvas) | | `packages/compositor/src/render/Scene.ts` | One per output, manages view stack + textures | | `packages/compositor/src/render/YUVA2RGBA.ts` | YUVA→RGBA FBO pass | | `packages/compositor/src/render/YUVA2RGBAShader.ts` | BT.601 YUVA→RGBA shader | | `packages/compositor/src/render/RGBXandA2RGBA.ts` | RGB+Alpha → RGBA FBO pass (WebCodecs output) | ### Browser side (REUSED MIT) | Path | Role | |---|---| | `libs/compositor-wasm/src/index.ts` | pixman + xkbcommon WASM (region math + keymap) | ### Python side (custom, clean-room) | Path | Role | |---|---| | `proxy/encoding/gst_encoder.py` | GStreamer H.264/PNG encoder (~700 LOC) | | `proxy/encoding/adaptive_bitrate.py` | Adaptive bitrate controller (NEW) | | `proxy/surface_buffer_encoding.py` | Encoding pipeline orchestration | | `proxy/proxy_buffer.py` | Proxy-side buffer representation | | `proxy/frame_feedback.py` | Receives 4-byte decode feedback | | `proxy/encoding/glib_thread.py` | GLib MainLoop in separate thread + asyncio marshalling | ### Vendored Greenfield C files (MIT, used via cffi) | Path | Role | |---|---| | `native/westfield-egl.c` | EGL zero-copy setup | | `native/westfield-dmabuf.c` | DMA-BUF import via EGL/GBM | | `native/wlr_drm.c` | `wl_drm` global impl | | `native/wlr_linux_dmabuf_v1.c` | `zwp_linux_dmabuf_v1` global impl | | `native/drm_format_set.c` | DRM format helpers | ### Reused (LGPL/BSD/MIT) | Component | Role | |---|---| | PyGObject (`gi.repository.Gst`) | GStreamer Python bindings (LGPL-2.1+) | | GStreamer core | Media pipeline framework (LGPL-2.0+) | | gst-plugins-base, gst-plugins-good | PNG, appsrc, appsink, videoconvert (LGPL) | | gst-vaapi | `vaapih264enc` for Intel/AMD (LGPL + MIT libva) | | gst-plugins-bad (BSD subset) | `nvh264enc` for NVIDIA, `openh264enc` for CPU fallback | | libopenh264 | OpenH264 BSD Wasm fallback decoder (BSD-2-Clause) |