22 KiB
09 — Hardware Acceleration, Encoding, WASM, GPU Textures
This chapter covers Greenfield's hardware acceleration / encoding / WASM pipeline: how native buffers get encoded on the proxy side, transported, decoded in the browser, and turned into WebGL textures or canvas images.
What Greenfield actually ships
Two Wasm libraries under libs/
compositor-wasm builds pixman 0.42.2 + xkbcommon 1.5.0
(Emscripten 3.1.46, -msimd128 -O3 -flto). Used by the browser for
region math (damage tracking) and keyboard layout — not for
pixel format conversion or H.264 decode.
compositor-ffmpeg-h264 is a stripped-down FFmpeg n5.1.2
(--disable-everything --enable-decoder=h264) compiled to a 1.7 MB
single-file Wasm worker. A 30-line decoder.c wrapper exposes
create_codec_context / decode / close_frame / destroy_codec_context.
Decodes to I420 (YUV 4:2:0 planar) and returns plane pointers; planes
are postMessage'd back to the main thread as transferable
ArrayBuffers. No threads (single-threaded, SIMD only).
Native encoder addon (packages/compositor-proxy/native/encoding/)
A GStreamer wrapper exposed via Node-API. The TS side (Encoder.ts)
calls appEndpointNative.encodeFrame(encoder, bufferId, contentSerial, creationSerial). The C side (node_encoder.c) sniffs the wl_resource:
zwp_linux_dmabuf_v1→ DMA path- Legacy
wl_drm→ DMA path wl_shm_buffer→ SHM path
For DMA-BUFs the proxy wraps the dma-buf as an EGLImage and then as a
GstGLMemoryEGL, so pixels stay on GPU through the color-convert
stages.
Threading
Encoding runs on a dedicated GLib main-loop thread fed by an
eventfd-signalled GAsyncQueue. Output returns to Node via a
napi_threadsafe_function with zero-copy external Buffers (finalised
back to C on GC).
Encoder selection (per-frame, deterministic)
gst_frame_encoder.c::frame_encoder_descriptions[] declares four
pipelines:
| name | element | when used | split_alpha | coded-size multiple |
|---|---|---|---|---|
x264 |
x264enc (software, default) |
>256×256 + ARGB/XRGB/dmabuf | true (parallel alpha H.264 stream) | 128 |
nvh264 |
nvh264enc (Nvidia NVENC) |
same, --encoder=nvh264 |
true | 128 |
vaapih264 |
vaapih264enc (Intel/AMD VA-API, "experimental") |
same, --encoder=vaapih264 |
true | 128 |
png |
pngenc (fallback) |
≤256×256 + ARGB/XRGB/dmabuf | false (PNG has native alpha) | 16 |
Rule: pixel_count ≤ 256×256 → PNG; otherwise the configured H.264 encoder. YUV420/NV12/NV21/Y444 SHM formats are not supported on the H.264 path and only by PNG at small sizes — effectively unsupported for video-class content.
Canonical H.264 pipeline
appsrc → glupload → glcolorconvert → glshader(scale→coded_size) →
glcolorconvert → video/x-raw(memory:GLMemory),format=NV12 → gldownload →
queue → x264enc tune=zerolatency bitrate=12000 vbv-buf-capacity=160
bframes=0 rc-lookahead=0 sliced-threads byte-stream=true →
video/x-h264,profile=high,stream-format=byte-stream,alignment=au → appsink
The glshader element does two jobs:
- Pad up to the encoder's macroblock multiple (128 for H.264, 16 for PNG)
using a
u_transformationmatrix. - For the alpha pipeline, run
vec4(pix.a, pix.a, pix.a, 0.)so alpha is encoded as a grayscale H.264 stream — H.264 has no alpha, so Greenfield ships two parallel AU streams.
Bitrates: opaque 12000 / alpha 1200 (x264), 12000 / 1200 (nvh264).
Wire format
Single contiguous blob, little-endian:
struct packed props {
u32 buffer_id, creation_serial, content_serial,
u32 frame_encoding_type (0=h264, 1=png),
u32 width, height, coded_width, coded_height
}
u32 opaque_length
u8 opaque[opaque_length] // H.264 AU or PNG blob
u32 alpha_length // 0 if no alpha split
u8 alpha[alpha_length] // H.264 AU of alpha-only stream
Parsed by remote/EncodedFrame.ts::createEncodedFrame.
Transport (Channel.ts)
Four logical channels per client multiplexed over WebSockets:
| Channel | Type | Purpose |
|---|---|---|
| PROTOCOL | ARQ (KCP) | Wayland wire protocol + out-of-band ops |
| FRAME | ARQ (KCP) | Encoded surface frames |
| XWM | ARQ (KCP) | XWayland WM messages |
| FEEDBACK | SIMPLE (raw WS) | 4-byte feedback messages |
ARQ = a JS KCP implementation layered over WebSocket. Tuning:
MTU=64000, MAX_BUFFERED_AMOUNT=2949120 (~2.8 MB — the back-pressure
ceiling), SND_WINDOW=128, RCV_WINDOW=1024,
setNoDelay(1, 100, 0, 1).
Browser decode (packages/compositor/src/remote/)
At session startup Session.create() probes
VideoDecoder.isConfigSupported({codec:'avc1.64001f', optimizeForLatency:true, hardwareAcceleration:'prefer-software'}):
WebCodecs preferred
Software-only; the prefer-hardware path is implemented but commented out
with // FIXME hardware decoding frame offset is wrong.
FFmpeg Wasm fallback
If WebCodecs unavailable: two persistent Workers (h264NALDecoderOpaque,
h264NALDecoderAlpha), one AVCodecContext per surface per worker,
keyed by surfaceH264DecodeId. Encoded NALs are transferred zero-copy
into the worker; decoded I420 planes are transferred back.
Per-surface decoder context is lazily created on first H.264 frame.
PNG path (both decoders)
Identical in both branches: new Blob([opaque], {type:'image/png'}) →
createImageBitmap(blob, sx, sy, w, h) (the sx/sy crops the encoder
padding back to logical size).
WebGL upload (render/)
WebGL 1 with desynchronized:true for low-latency presentation. Per Scene
(canvas/output), three GL helpers:
YUVA2RGBA
FBO pass: uploads Y/U/V/alphaY as 4× LUMINANCE textures, renders into
the surface's RGBA texture via YUVA2RGBAShader (BT.601 coefficients,
alpha recovered by running the alpha-Y plane through the same YUV→RGB
matrix and taking R).
RGBXandA2RGBA
FBO pass for the DualPlaneRGBA* WebCodecs outputs: opaque + alpha planes
composited via RGBandA2RGBAShader.
SceneShader
Final pass: draws every view's RGBA texture to the canvas with
gl.blendFunc(ONE, ONE_MINUS_SRC_ALPHA) (note: expects premultiplied
alpha but the YUVA shader outputs straight alpha — a minor inconsistency).
PNG and web-bitmapbuf paths skip the FBO and call
texImage2D(ImageBitmap) directly.
Alpha pre-multiplication
Not done explicitly anywhere. The proxy extracts alpha as a Y plane; the browser shader recombines; the canvas blend mode assumes premultiplied but receives straight alpha — works acceptably because Wayland surfaces rarely use partial alpha.
End-to-end pipeline
┌──────────────────── NATIVE CLIENT PROCESS ────────────────────┐
│ App renders via GTK/Qt/raw EGL into │
│ ┌────────────────────────────────────┐ │
│ │ wl_shm buffer (ARGB8888/XRGB8888)│ ─── or ─────────┐ │
│ │ zwp_linux_dmabuf_v1 (DMA-BUF) │ │ │
│ │ wl_drm create_prime_buffer │ │ │
│ └────────────────────────────────────┘ │ │
└────────────────────────┬──────────────────────────────────┼────┘
│ Wayland socket (UNIX domain) │
▼ │
┌────────────────────── COMPOSITOR PROXY (Node + N-API addons)─┐
│ wayland-server-addon.node │
│ └─ wl_display → wl_client → wl_resource (buffer) │
│ └─ initShm(), initDrm("/dev/dri/renderD128") │
│ └─ westfield-egl.c: EGLDevice + EGLDisplay + EGLContext │
│ │
│ proxy-encoding-addon.node (per-surface Encoder) │
│ ┌─ node_encoder.c → frame_buffer {type: SHM|DMA, w, h, fmt} │
│ │ │
│ │ gst_main_loop.c (own GLib thread, eventfd work queue) │
│ │ │ │
│ │ ▼ │
│ │ gst_frame_encoder.c │
│ │ ┌─ Encoder selection (per-frame): │
│ │ │ pixel_count ≤ 256×256 ─► PNG pipeline │
│ │ │ > 256×256 + ARGB/XRGB/dmabuf ─► configured h264 │
│ │ │ │
│ │ │ H.264 pipelines (parallel): │
│ │ │ opaque: appsrc → glupload → glcolorconvert → │
│ │ │ glshader(scale→coded_size) → │
│ │ │ glcolorconvert(NV12) → gldownload → │
│ │ │ queue → x264enc|nvh264enc|vaapih264enc → │
│ │ │ byte-stream AU → appsink │
│ │ │ alpha: same source, fragment shader │
│ │ │ `vec4(pix.a,pix.a,pix.a,0.)` → │
│ │ │ second H.264 stream │
│ │ │ │
│ │ │ PNG pipeline (single): │
│ │ │ appsrc → glupload → glcolorconvert → glshader → │
│ │ │ RGBA → gldownload → pngenc → appsink │
│ │ │ │
│ │ └─ frame_encoding_result_to_encoded_frame(): │
│ │ packed struct {buffer_id, creation_serial, │
│ │ content_serial, type(0=h264,1=png), w, h, │
│ │ coded_w, coded_h} + opaque_len + opaque + │
│ │ alpha_len + alpha → Node Buffer (zero-copy ext.) │
│ │ │
│ └─ SurfaceBufferEncoding.ts │
│ └─ frameDataChannel.send(buffer) │
│ │
│ Channel.ts: │
│ ARQChannel (KCP-reliable over WebSocket, MTU 64 KB) │
│ SimpleChannel (raw WebSocket, for FEEDBACK only) │
│ ▲ │
│ │ FrameFeedback.ts: 16-bit rAF interval + 16-bit avg │
│ │ decode duration → used only to pace wl_frame.done │
└───────┼───────────────────────────────────────────────────────┘
│ 4 logical channels per client: PROTOCOL / FRAME / XWM / FEEDBACK
▼
┌────────────────────── BROWSER (compositor) ────────────────────┐
│ WebSocket → ReconnectingWebSocket │
│ ↓ │
│ Channel.ts: ARQChannel.input(kcp) → onMessage │
│ ↓ │
│ RemoteAppLauncher.setupFrameDataChannel → │
│ BufferStream.deliverContentToBufferStream (parse serials) │
│ ↓ │
│ StreamingBuffer.getContents(surface, serial) │
│ ↓ │
│ Session.frameDecoder.decode(surface, encodedFrame) │
│ │ │
│ ├─ PNG path (both decoders): │
│ │ new Blob([opaque], {type:'image/png'}) → │
│ │ createImageBitmap(blob, sx, sy, w, h) // crop padding │
│ │ → SinglePlane { bitmap: ImageBitmap } │
│ │ │
│ └─ H.264 path: │
│ (preferred) WebCodecs VideoDecoder │
│ config: avc1.64001f, prefer-software, optimizeForLatency│
│ opaqueDecoder + alphaDecoder per surface │
│ EncodedVideoChunk {type:key|delta, data:nal} │
│ → VideoFrame.output → │
│ if I420: copyTo → DualPlaneYUVAArrayBuffer │
│ else: keep VideoFrame → DualPlaneRGBAVideoFrame │
│ (fallback) FFmpeg Wasm in 2 Web Workers │
│ worker.postMessage(nal, [nal.buffer]) // transferable │
│ libavH264._decode → I420 planes │
│ postMessage back with [y.buffer, u.buffer, v.buffer] │
│ → DualPlaneYUVASplitBuffer │
│ │
│ Renderer.render() (rAF-driven, queueCancellableMicrotask) │
│ ↓ │
│ Scene.render(viewStack) │
│ ├─ Scene['image/png'] → texImage2D(ImageBitmap) │
│ ├─ Scene['image/bitmap'] (web-bitmapbuf path) → texImage2D │
│ └─ Scene['video/h264']: │
│ YUVA2RGBA.convertYUVASplitBufferInto / │
│ convertYUVAArrayBufferInto: │
│ Y/U/V/alphaY → 4× LUMINANCE textures │
│ bind FBO → renderState.texture (RGBA) as color attach │
│ YUVA2RGBAShader (BT.601) → drawArrays → RGBA in tex │
│ or RGBXandA2RGBA.convertInto (WebCodecs RGBA): │
│ VideoFrame/ImageBitmap → texImage2D directly │
│ ↓ │
│ SceneShader.draw(): drawArrays with surface's RGBA texture, │
│ blend gl.ONE / gl.ONE_MINUS_SRC_ALPHA, desynchronized canvas │
│ ↓ │
│ Browser composites canvas into the page │
└─────────────────────────────────────────────────────────────────┘
Same-frame web app fast path (web-bitmapbuf):
┌───────────────┐ MessagePort.postMessage([view, ...fds], ┌──────────────┐
│ Web app │ ──► [imageBitmap]) ──────────────────────────► │ Compositor │
│ (iframe) │ web_bitmapbuf_factory.create_buffer(id, fd) │ WebConnHandl │
└───────────────┘ → wl_buffer → ImageBitmapBuffer └──────────────┘
(no encoding, no WebSocket, no GStreamer)
Key answers
Why H.264 over PNG?
H.264 at 12 Mbps × 1/60s ≈ 25 KB/frame for 1080p, vs PNG 100 KB – 3 MB. PNG wins for ≤256×256 because H.264 I-frame overhead and alpha-split doubling dominate at small sizes.
WebCodecs?
Yes — preferred path, software-only (prefer-hardware is wired but
disabled with FIXME "frame offset is wrong"). FFmpeg Wasm is the fallback.
Trade-offs
- H.264 = small wire / decode cost
- PNG = medium wire / browser-native fast decode (
createImageBitmap) - raw SHM = not used between proxy and browser — every surface commit is encoded.
Per-surface policy?
No — per-frame deterministic. Only configurable knob is
--encoder=x264|nvh264|vaapih264. PNG is always automatic fallback.
Cursors?
Always PNG (32×32 ≤ 256×256). Decoded to ImageBitmap and set as the CSS
cursor via setBrowserCursor — never enters the WebGL compositor.
Alpha pre-multiplication?
Implicit in shaders. Proxy extracts alpha as a Y plane (no premult);
browser recombines via BT.601 YUVA→RGBA shader outputting straight alpha;
canvas blend is ONE, ONE_MINUS_SRC_ALPHA (expects premult) — minor
inconsistency, not visually problematic for typical Wayland content.
OffscreenCanvas / Web Workers?
Workers yes (two for Wasm decoder, shared across surfaces). OffscreenCanvas
no — renderer is main-thread WebGL with desynchronized:true.
WebGPU/WebGL2 not used.
Notable design trade-offs for your compositor
- Always-encode policy (no raw SHM escape hatch even on LAN).
- One encoder per surface, EOS-and-recreate on resize/format change.
- Alpha as a separate H.264 stream — doubles cost for transparent windows.
- Per-surface decoder context, no decoder pooling.
- Only aggregate feedback (1 Hz rAF interval + decode duration); no
per-frame ACKs, no bitrate adaptation;
requestKeyUnitplumbing exists end-to-end but the browser never calls it. - KCP over WebSocket (not WebRTC datachannels) — simpler infra, no STUN/TURN, but no real congestion control.
- Emscripten SIMD (not threads) — avoids COOP/COEP requirements.
- Two parallel GL pipelines (opaque + alpha) for H.264.
- PNG fallback boundary empirically at 256×256 pixels.
- WebGL 1, not 2/WebGPU — maximum browser compat at the cost of manual LUMINANCE-texture YUV juggling and a hidden FBO intermediate.
File reference
Wasm libraries
| Path | Role |
|---|---|
libs/compositor-wasm/build_wasm.sh |
Builds pixman 0.42.2 + xkbcommon 1.5.0 to WASM (emsdk 3.1.46) |
libs/compositor-wasm/emscripten-build.ini |
Emscripten build config |
libs/compositor-wasm/src/index.ts |
TS wrapper, init function |
libs/compositor-wasm/xkbcommon/build.sh |
xkbcommon build (clones 1.5.0, embeds xkeyboard-config DB) |
libs/compositor-wasm/pixman/build.sh |
pixman build |
libs/compositor-ffmpeg-h264/build_wasm.sh |
Builds FFmpeg n5.1.2 to WASM (single-file, decoder-only) |
libs/compositor-ffmpeg-h264/native/decoder.c |
30-line wrapper exposing decode API |
libs/compositor-ffmpeg-h264/src/H264Decoder.ts |
TS wrapper around the Wasm module |
libs/compositor-ffmpeg-h264/src/H264Worker.ts |
Worker entry point |
libs/compositor-ffmpeg-h264/src/index.ts |
Exports |
Native encoder addon
| Path | Role |
|---|---|
packages/compositor-proxy/native/encoding/src/node_encoder.c |
Node-API addon (290 lines) |
packages/compositor-proxy/native/encoding/src/gst_frame_encoder.c |
GStreamer H.264/PNG encoder (~1400 lines) |
packages/compositor-proxy/native/encoding/src/gst_main_loop.c |
Dedicated GLib main loop for encoding |
packages/compositor-proxy/native/CMakeLists.txt |
Build config |
packages/compositor-proxy/src/SurfaceBufferEncoding.ts |
TS-side encoding orchestration |
packages/compositor-proxy/src/ProxyBuffer.ts |
Proxy-side buffer representation |
packages/compositor-proxy/src/FrameFeedback.ts |
Receives 4-byte decode feedback |
packages/compositor-proxy/src/encoding/ |
High-level encoding files |
Browser decode
| Path | Role |
|---|---|
packages/compositor/src/remote/EncodedFrame.ts |
Wire format parser for encoded frames |
packages/compositor/src/remote/webcodec-buffer-decoder.ts |
WebCodecs VideoDecoder path |
packages/compositor/src/remote/wasm-buffer-decoder.ts |
FFmpeg Wasm fallback path |
packages/compositor/src/remote/H264NALDecoder.worker.ts |
Worker entry, runs FFmpeg Wasm decoder |
packages/compositor/src/remote/EncoderFeedback.ts |
Sends 4-byte decode feedback at 1 Hz |
packages/compositor/src/remote/StreamingBuffer.ts |
Per-surface encoded-frame buffer |
packages/compositor/src/remote/BufferStream.ts |
Manages serial → frame promise resolution |
WebGL rendering
| Path | Role |
|---|---|
packages/compositor/src/render/Renderer.ts |
4-phase render pipeline |
packages/compositor/src/render/Scene.ts |
One per output, manages view stack + textures |
packages/compositor/src/render/SceneShader.ts |
Final pass: textured quads to canvas |
packages/compositor/src/render/Texture.ts |
WebGL texture wrapper |
packages/compositor/src/render/RenderState.ts |
Per-view-per-scene texture + transform |
packages/compositor/src/render/YUVA2RGBA.ts |
YUVA→RGBA FBO pass |
packages/compositor/src/render/YUVA2RGBAShader.ts |
BT.601 YUVA→RGBA shader |
packages/compositor/src/render/YUV2RGBShader.ts |
YUV→RGB shader (no alpha) |
packages/compositor/src/render/RGBXandA2RGBA.ts |
RGB+Alpha → RGBA FBO pass (WebCodecs output) |
packages/compositor/src/render/RGBandA2RGBAShader.ts |
Shader for above |
packages/compositor/src/render/Program.ts |
WebGL program wrapper |
packages/compositor/src/render/ShaderCompiler.ts |
Shader compile/link helper |
packages/compositor/src/render/ShaderSources.ts |
GLSL source strings |
Protocol extensions
| Path | Role |
|---|---|
protocol/web-bitmapbuf-unstable-v1.xml |
Greenfield's custom Wayland extension for browser JS apps |
protocol/wayland-drm.xml |
Standard wl_drm for DMA-BUF (proxy-only) |
packages/compositor/src/web/WebBuffersFactory.ts |
Browser-side impl of web_bitmapbuf_factory |