feat: M6 web client QoL — clipboard, multi-monitor, quality, reconnect, H.264, docs
Builds on the M6a-g MVP (d07e98e) with five user-facing features that
take the in-browser remote-control client from "demo toy" to "workable
for daily use", plus user-facing documentation.
M6h — Text clipboard sync (both directions)
- Host → browser via Clipboard{format: Text} → navigator.clipboard.writeText.
- Browser → host via Cmd-V/Ctrl-V keydown intercept → navigator.clipboard
.readText → Message{clipboard}, sent BEFORE the V keystroke so the
host's paste hotkey lands on the freshly-synced text.
- Handles both single-format Clipboard (older peers) and MultiClipboards
wrapper (peers ≥ 1.3.0; gated by clipboard.rs:is_support_multi_clipboard).
- Switched printable-with-modifier hotkeys (Ctrl-C etc.) from Unicode to
Chr payload — host's process_unicode does key_sequence(char) which
ignores modifiers, breaking copy/cut; process_chr respects them.
- Firefox refuses navigator.clipboard.readText() by default — accepted as
a known browser limitation, host → browser direction works regardless.
M6i — Multi-monitor switching
- HUD picker shown when peer_info.displays > 1.
- On change: SwitchDisplay + CaptureDisplays{set:[idx]} two-message dance
— required for clients ≥ 1.2.4 (we send "1.4.0"). Without the follow-
up, switch_display_to leaves both video services subscribed and
switching display 0 → 1 → 0 doesn't restore display 0.
- Mouse coords offset by the active display's virtual-desktop origin
(DisplayInfo.x, .y). Without this, clicks on display 2 landed on
display 1 because both share canvas (0,0) but only display 1 has
origin (0,0) in virtual-desktop space.
M6j — Quality / FPS / mute controls
- Image quality preset (Low/Balanced/Best) → Misc{option: {image_quality}}.
- Custom FPS (15/30/60) → Misc{option: {custom_fps}}; host caps at 30
unless allow_more_fps is advertised.
- Mute toggle additionally sends Misc{option: {disable_audio: Yes/No}}
so the peer stops encoding audio while muted (saves CPU + bandwidth).
M6k — Auto-reconnect on transient drops
- session.recv() throw → reconnect with exponential backoff: 1s, 2s, 4s,
8s, 16s, 30s, 30s, capped at 30s, max 10 attempts.
- Dim overlay sits on top of the canvas during retry; canvas keeps
last-known frame for visual continuity.
- Auth errors (password/signature) bail immediately — no point retrying.
- User options (mute, image_quality, custom_fps, current display)
re-applied to host on each successful reconnect, since host treats
every session as fresh and resets to defaults.
- Architecture: `session` is a let-binding mutated on reconnect; HUD
button closures read it at click-time so they automatically retarget.
Input modules (mouse/keyboard/clipboard) get a Proxy that forwards
method calls to whatever session is current — avoids re-binding
window/canvas listeners on each reconnect.
M6l — H.264 video decode (Annex-B + SPS-derived codec string)
- decode/bitstream.ts: iterate Annex-B NAL units, derive avc1.PPCCLL
from the keyframe's inline SPS (host's hwcodec defaults to high
profile; a hardcoded baseline string would make WebCodecs refuse the
stream).
- Defer H.264 decoder configure until first keyframe arrives.
- VP9 codec string corrected from level 1.0 (vp09.00.10.08) to level
5.0 (vp09.00.50.08) — wrong level was probably forcing software
decode in some browsers.
- Default prefer flipped to VP8 (cheapest software encoder; H.264 path
stays implemented for hosts with hwcodec/nvenc).
M6m — docs/CONFIGURATION.md "Web client" section: routes, browser
matrix, network requirements (relay reachability + reverse-proxy WS
upgrade), feature status table, codec selection rationale, the
recv/dec/draw HUD diagnostic, build commands.
Bundle: 535 KB / ~75 KB gzipped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -317,6 +317,119 @@ for the dashboard with no separate session model.
|
||||
|
||||
---
|
||||
|
||||
## Web client
|
||||
|
||||
Admins can remote-control any peer directly from the browser — no desktop
|
||||
RustDesk install required. The Devices page row dropdown surfaces a
|
||||
**Connect** button that opens `/admin/connect/<peer_id>` in a new tab.
|
||||
The page is a TypeScript SPA bundled into hbbs (`include_bytes!` from
|
||||
`web_client/dist/`), so there's no separate process or service to run.
|
||||
|
||||
### Routes
|
||||
|
||||
| Route | Auth | Purpose |
|
||||
|---|---|---|
|
||||
| `/admin/connect/:peer_id` | cookie + admin | Server-rendered HTML wrapper that injects per-request config (rendezvous host, relay host, server pk, peer id, admin display name) into a `<script id="custom-config">` tag, then loads `bundle.js`. |
|
||||
| `/admin/connect/assets/bundle.js` | cookie + admin | The compiled SPA. ~535 KB, ~75 KB gzipped. |
|
||||
| `/admin/connect/assets/bundle.css` | cookie + admin | Minimal dark theme + reconnect overlay. |
|
||||
|
||||
### How it talks to peers
|
||||
|
||||
The browser opens **WebSockets directly** to the existing rendezvous
|
||||
(`hbbs:21118`, the `ws_port`) and relay (`hbbr:21119`, `port + 2`)
|
||||
sockets. The wire shape — protobuf `RendezvousMessage` + secretbox-
|
||||
encrypted `Message` stream — is identical to what the desktop client
|
||||
uses; the SPA is just a from-scratch reimplementation of the protocol
|
||||
talking through WS frames instead of TCP.
|
||||
|
||||
Browsers can't do UDP NAT-punching, so the SPA always sets `force_relay
|
||||
= true` in `PunchHoleRequest` and follows the `RelayResponse` branch
|
||||
unconditionally. **Direct peer-to-peer is never attempted.**
|
||||
|
||||
### Browser requirements
|
||||
|
||||
- **Secure context** — page must be served over HTTPS or `http://localhost`. WebCodecs `VideoDecoder` / `AudioDecoder` are gated to secure contexts. Plain `http://lan-ip:21114` will fail; either use TLS in front of hbbs or do `ssh -L 21114:localhost:21114 hbbs-host` and connect via `http://localhost:21114`.
|
||||
- **Browser version** — Chrome 94+, Edge 94+, Firefox 130+, Safari 16.4+. Older browsers lack WebCodecs and will throw "VideoDecoder unavailable" on session start.
|
||||
- **Clipboard permission** — Firefox refuses `navigator.clipboard.readText()` by default, so admin → host paste is silently no-op there. Host → admin paste works (it uses `writeText`, which only needs a recent click). Chrome/Safari prompt once.
|
||||
|
||||
### Network requirements
|
||||
|
||||
- The **relay host** advertised to clients (`--relay-servers=<HOSTS>` on hbbs) must resolve and be reachable from the end-user's browser on port 21119. The relay is what carries the actual session bytes — if a user's browser can't open `ws://<relay-host>:21119/`, the session dies after the rendezvous step. A common gotcha: setting `--relay-servers=hbbr-internal.local` works for desktop clients on the LAN but breaks for browsers off-LAN.
|
||||
- **Reverse proxies** must forward the WebSocket upgrade for both 21118 (rendezvous) and 21119 (relay). Caddy: `reverse_proxy /ws/* hbbs:21118` plus equivalent for 21119; nginx: the standard `proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade";` block.
|
||||
- Audit rows are written under the admin's cookie via the existing `/api/audit/conn` endpoint; no new server endpoint.
|
||||
|
||||
### Features
|
||||
|
||||
| Feature | Status |
|
||||
|---|---|
|
||||
| Video (VP8 / VP9 / H.264) | ✅ — preference is VP8 (lightest software encoder; H.264 path also implemented for hosts with hwcodec) |
|
||||
| Audio (Opus) | ✅ |
|
||||
| Mouse + keyboard input | ✅ — Legacy keyboard mode; Translate mode silently drops Unicode/ControlKey payloads on the host |
|
||||
| Text clipboard sync (both directions) | ✅ — handles both single-format `Clipboard` and `MultiClipboards` (peers ≥ 1.3.0) |
|
||||
| Multi-monitor switching | ✅ — `SwitchDisplay` + `CaptureDisplays` two-message dance for hosts ≥ 1.2.4; mouse coords offset by display's virtual-desktop origin |
|
||||
| Image quality presets (Low/Balanced/Best) | ✅ |
|
||||
| Custom FPS (15/30/60) | ✅ — host caps at 30 unless `allow_more_fps` is advertised |
|
||||
| Mute toggle | ✅ — also tells host to stop encoding audio (saves CPU + relay bandwidth) |
|
||||
| Ctrl+Alt+Del | ✅ — Windows hosts only (server-side `#[cfg(windows)]`) |
|
||||
| Auto-reconnect on transient drops | ✅ — up to 10 attempts, exponential backoff 1s → 30s, dim overlay during retry, user options re-applied on success |
|
||||
| File transfer | ❌ deferred — separate `FileAction` family, double the surface area |
|
||||
| AV1 / H.265 decode | ❌ deferred — VP8/VP9/H.264 covers the common cases |
|
||||
| IME / compose input | ❌ deferred — needs `compositionend` + `KeyEvent.seq` |
|
||||
| Touch gestures | ❌ deferred |
|
||||
| Cursor sprite rendering | ❌ deferred — host-side cursor visible in the video; we don't draw a separate one |
|
||||
|
||||
### Codec selection
|
||||
|
||||
The SPA advertises VP8/VP9/H.264 decode and prefers VP8. The host's
|
||||
codec picker (`libs/scrap/src/common/codec.rs`) honours the preference
|
||||
when the host has the matching encoder available, else falls back to
|
||||
its "auto" path (H.265 → H.264 → AV1/VP9/VP8).
|
||||
|
||||
VP8 is the default because it's the cheapest software encoder; on a
|
||||
host without hwcodec, VP9 software-encode caps screen sharing at single
|
||||
digits FPS, while VP8 keeps headroom for the screen-capture pipeline.
|
||||
On a host *with* hwcodec H.264 (nvenc/qsv on most Windows boxes), flip
|
||||
the preference in `web_client/src/transport/session.ts` to
|
||||
`PreferCodec.H264` — the SPA's H.264 path parses SPS to derive the
|
||||
correct `avc1.PPCCLL` codec string for WebCodecs.
|
||||
|
||||
### Performance gotchas
|
||||
|
||||
The HUD shows three live numbers: `recv` (frames/sec arriving from the
|
||||
relay), `dec` (frames/sec the browser decoded), `draw` (frames/sec
|
||||
painted to canvas). Use them to localise FPS issues:
|
||||
|
||||
- All three low → host is encoding slowly. Either CPU-bound (no
|
||||
hwcodec, VP9 chosen) or QoS-throttled by host based on `TestDelay`
|
||||
RTT measurements.
|
||||
- `recv` high, `dec` low → browser fell back to software decode. Check
|
||||
the codec string at the end of the HUD line; mismatched profile/level
|
||||
(e.g. `avc1.42E01E` for a high-profile stream) forces software.
|
||||
- `dec` high, `draw` low → main thread is overwhelmed (very rare with
|
||||
hardware decode).
|
||||
|
||||
### Building the bundle
|
||||
|
||||
The bundle is committed under `web_client/dist/` so a Cargo build
|
||||
doesn't need a Node toolchain. To regenerate after editing the SPA:
|
||||
|
||||
```sh
|
||||
cd web_client
|
||||
npm install # one-time
|
||||
npm run build # → dist/bundle.{js,css}
|
||||
git add dist/
|
||||
cd .. && cargo build --release -p hbbs
|
||||
```
|
||||
|
||||
To regenerate protobuf bindings after a `libs/hbb_common/protos/*.proto`
|
||||
change:
|
||||
|
||||
```sh
|
||||
cd web_client && npm run protogen
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Database
|
||||
|
||||
SQLite, file `db_v2.sqlite3` in hbbs's working directory. Tables created
|
||||
|
||||
Reference in New Issue
Block a user