Sync 1080p60: next research plan

25 September 2026 · Research plan · Revision 8 · Scope reset and focused roadmap for 1080p60 browser-to-native video bridge delivery

Recommended immediate action: execute Priority 1 (Browser-to-native sender stage pacing, GPU fence latency elimination, and sub-frame WebGL2 readback pipeline optimization). E1 E2 E7 E8 E10 E14 E15 E16

1. Objective and constraints

Sync is Noise Factor's low-latency bridge between browser renderers and native video ecosystems. It carries GPU-rendered RGBA frames over authenticated loopback WebSockets and shared memory, republishing them through platform-native video-sharing providers: Syphon and CMIO virtual camera on macOS, Spout and Media Foundation virtual camera on Windows, and v4l2loopback on Linux, alongside cross-platform NDI output. This architecture enables browser applications such as Noisedeck to appear directly inside existing source pickers without custom plugins in downstream hosts.

The single objective of Sync research is sustained 1920 × 1080 at 60.00 unique delivered frames per second, without performance degradation over time, serving web and desktop builds across macOS, Windows, and Linux. All delivery claims must adhere to the actual production pipeline:

Noisedeck A (WebGL2 / GPU Canvas) → Sync SDK (Loopback / Shared Memory) → syncd daemon → Sync Camera / Syphon / Spout → Noisedeck B (Media Effect) → Final Decoded Pixels

To ensure research stays strictly on the rails and within scope, the following permanent boundaries govern this roadmap:

Operator Cancellation of Neural Processing: By explicit operator directive (25 September 2026), "neural" processing of any kind is cancelled and purged from Sync scope. Sync is a deterministic, low-latency video bridge. It does not perform neural frame reconstruction, AI super-resolution, optical flow estimation, motion vector extrapolation, or perceptual artifact concealment.
Strict Focus on 1080p60 — Defer 4K / 8K: All 4K60 and 8K60 research (including JPEG XS mezzanine compression) is explicitly deferred. Engineering effort is concentrated 100% on achieving rock-solid, zero-drop 1080p60 delivery on commodity user hardware.
Pruning of Enterprise WAN and Cloud Switcher Outliers: Speculative cloud-native broadcast switching, multi-region SRT/RIST mesh bonding, SMPTE ST 2110 IP broadcast fabrics, SDI hardware cards, and studio automation (Ember+, room acoustics modeling, fast metadata) are purged as out-of-scope research drift. Sync operates as a local host service.
Preservation of Wire Protocols: Maintain backward compatibility with protocol-v1.md, public SDK interfaces, pairing store contracts, and CLI arguments.
Empirical Verification: Require paired baseline versus candidate comparisons and 30-minute endurance soaks before retaining any performance claims.

2. Evidence that determines the next action

Table 1. Established in-scope research milestones and foundational qualification status

Evidence	Decision reached	Scope & Reference
Native AVFoundation received 23,664 frames with 0 drops and 60.000 unique FPS over 600 s; Chromium delivered 6,277 repeats (20.9%) and 332 drops.	The CMIO extension, Mach messaging, and daemon relay are 100% qualified for full 1080p60 uncompressed delivery. Delivery ceilings originate inside Chromium browser readback and pacing.	macOS Apple Silicon host. E1
Sender stage breakdown: GPU fence wait accounted for 11.7–12.4 ms (56.5%–59.9%); PBO readback 3.3 ms; WebSocket framing 0.8 ms.	In-place row flipping eliminated 8.3 MB buffer allocation. Fence wait reflects single-frame GPU pipeline depth, identifying the primary bottleneck in browser sender pacing.	Measured on Apple Silicon unified memory. E2
Decoupled reader intake loop, direct VideoFrame GPU upload bypass, and 3-tick starvation hysteresis reduced consecutive duplicate frames from 20.90% to 1.03%, eliminated all 332 drops (0 drops), and lowered median latency to 18.24 ms over 600 s.	Retain decoupled reader intake, direct VideoFrame upload, and starvation hysteresis. Eliminates the 50–55 FPS capture ceiling on web, leaving residual 1.03% duplicates as the next target.	macOS browser capture pipeline. E7
Packaged Electron desktop native shared memory bypass delivered 3,600/3,600 unique pictures (60.00 unique FPS) with 0 drops, 0 duplicates, 7.85 ms median latency (70.6% reduction), 0.559 ms WebGL2 texture upload, and 0 bytes/s heap allocations via slot pooling.	Retain packaged Electron native shared memory bypass with slot buffer preallocation across the desktop application. Eliminates CMIO DAL IPC overhead on desktop.	macOS desktop host. E8
Windows named shared memory desktop capture bypass (`%TEMP%\SyncCamera.frames` via Win32 `CreateFileMappingW`) delivered 3,600/3,600 unique pictures (60.00 unique FPS) with 0 drops, 0 duplicates, 7.42 ms median latency (73.9% reduction), 80.1% jitter reduction, and 0.568 ms WebGL2 texture upload. Preallocated slot buffers eliminated 497 MB/s GC churn.	Retain Windows named shared memory desktop capture bypass with slot buffer preallocation. Establishes sub-10ms capture parity across macOS and Windows desktop builds.	Windows Server 2025 / Windows 11 host. E10
Multi-consumer endurance soak & resource stability qualification (macOS): Sustained 30-minute endurance soak (1,800.0 s, 108,000 target frames) under concurrent triple-consumer load (CMIO DAL virtual camera + desktop shared memory reader + 32-channel discrete audio capture at 48 kHz). Delivered 60.99 AVF FPS, 59.98 SHM FPS, 0 low-rate seconds (<50 FPS), 0 seqlock retries across 107,964 frame reads, and 2,709,941,760 audio samples with 0 drops, 0 cursor discontinuities, and 0 nonfinite samples. Memory slope of -181.94 KiB/min.	Retain non-blocking audio capture mutex lock, 256 KiB socket write queues, 4 MiB receive buffers, and user-interactive thread QoS. Formally qualifies local 1080p60 multi-consumer delivery on macOS.	Apple Silicon M4 host. E14
Windows multi-consumer endurance soak & resource stability qualification: Sustained 30-minute endurance soak (1,800.0 s, 108,000 target frames) under concurrent triple-consumer load (Windows Camera Frame Server / DirectShow + named shared memory reader + 32-channel discrete WASAPI audio capture at 48 kHz). Delivered 60.08 DirectShow FPS, 59.98 SHM FPS, 0 low-rate seconds (<50 FPS), 0 seqlock retries across 107,964 frame reads, and 2,709,941,760 audio samples with 0 drops, 0 cursor discontinuities, and 100% finite samples. Memory slope of -14.28 KiB/min.	Retain atomic seqlock ring buffer, dual-dispatch demand tracking, and lock-decoupled WASAPI capture queues. Formally qualifies local 1080p60 multi-consumer delivery on Windows.	Windows 11 / Windows Server 2025 host. E15
Linux V4L2 loopback and POSIX shared memory multi-consumer endurance soak & resource stability qualification: Sustained 30-minute endurance soak (1,800.0 s, 108,000 target frames) under concurrent triple-consumer load (V4L2 loopback virtual camera + POSIX shared memory reader + 32-channel discrete audio capture at 48 kHz). Delivered 60.05 V4L2 FPS, 59.98 SHM FPS, 0 low-rate seconds (<50 FPS), 0 seqlock retries across 107,964 frame reads, and 2,709,941,760 audio samples with 0 drops, 0 cursor discontinuities, and 100% valid IEEE 754 float32 samples. Memory slope of -18.64 KiB/min.	Retain dual-dispatch demand tracking, 3-slot atomic seqlock, and lock-decoupled audio capture queues. Formally qualifies local 1080p60 multi-consumer delivery on Linux Ubuntu 24.04 LTS.	Linux Ubuntu 24.04 LTS (x86_64 / arm64). E16

3. Measurement contract

All performance claims must adhere to the standard measurement contract:

Target Denominator: Every 60-Hz source slot with a monotonic deadline in a preregistered half-open steady-state window [t0, t1). For an aligned 1,800 s window this is exactly 108,000 slots. A skipped production, refused send, or missing record remains in this target denominator.
Unique delivered FPS: Distinct, ordered source identities decoded from final rendered pixels, divided by elapsed duration: f_unique = N_distinct / T. Intermediate timestamps, queue acceptance, or render callbacks do not substitute for decoded final pixels.
Complete-second counts: Start-relative 1000 ms buckets. Datasets must report minimum complete-second count, complete seconds below 50 frames, zero-delivery seconds, and the longest consecutive interval below 50 FPS. Zero seconds below 50 FPS is required for qualification.
Marker age distribution: Elapsed duration from marker insertion to observer decode, reporting mean, p95, and p99 percentiles.
Local Glass-to-Glass Latency: Elapsed duration from WebGL2 draw submission in Noisedeck A to final pixel decode in Noisedeck B, targeting <20.0 ms across local loopback and shared memory.
Audio continuity: Zero dropped samples, zero ring-buffer overflows, zero sample-rate drift discontinuities, zero nonfinite samples, and bounded round-trip latency (<20 ms) across 1–32 channels.
Memory stability: Telemetry physical footprint slope ≤ 10.0 KiB/min computed over steady-state operation via linear regression.
Paired comparison protocol: Matched baseline and candidate runs, including a reverse-order confirmation run, conducted on the same hardware without intermediate state changes.

4. Research order

Browser-to-native sender stage pacing, GPU fence latency elimination, and sub-frame WebGL2 readback pipeline optimization (Priority 1 — ACTIVE PRIMARY FOCUS): Characterize and eliminate the primary remaining cause of delivery jitter and frame repeats in browser senders: the 11.7–12.4 ms GPU fence wait and readback stalls identified in E2. Evaluate non-blocking asynchronous pixel buffer transfer, OffscreenCanvas worker thread separation, and WebCodecs zero-copy frame submission to reduce browser sender stage latency below 4.0 ms, eliminate the residual 1.03% frame duplicate rate, and guarantee 100.0% unique 60.00 FPS delivery on commodity browser runtimes without pipeline bubbles.
Sync SDK zero-copy ArrayBuffer pooling and garbage collection pause elimination (Priority 2): Eliminate transient memory allocations and object creation within the fast-path frame encoding and WebSocket dispatch loops. Implement reusable pre-allocated buffer rings and transfer semantics to verify that V8 / SpiderMonkey garbage collection cycles never induce frame drops, jitter spikes, or complete seconds below 50 FPS during extended multi-minute sessions.
Multi-consumer 1080p60 camera driver endurance and zero-drop frame pacing under concurrent client load (Priority 3): Conduct sustained 1080p60 delivery stress tests with real-world third-party consumer hosts (OBS Studio, Zoom, QuickTime, Chromium) concurrently attached to Sync Camera on macOS (CMIO), Windows 11 (Media Foundation), and Linux (v4l2loopback). Verify zero frame drops, zero buffer starvation, and sub-20 ms latency when consumer applications start, run, and terminate asynchronously.
Combined 1080p60 video and 32-channel discrete audio endurance soak under heavy GPU shader contention (Priority 4): Execute rigorous 30-minute endurance soaks with concurrent 1080p60 video transmission and 32-channel discrete audio capture while executing complex generative Noisedeck shaders under saturated GPU and CPU load. Confirm 100% finite samples, zero cursor drift, zero audio ring drops, and flat memory footprint (≤10 KiB/min slope).

5. Short decision cycles

Each milestone must execute as an isolated decision cycle with paired baseline and candidate runs before embarking on a long soak. Candidate changes that do not demonstrate positive Δ FPS or that induce regression on paired platforms must be rejected promptly.

6. Long-session degradation and completion

Completion requires sustained 60 unique FPS without low-rate seconds over a continuous 30-minute soak window. Memory slope must remain ≤10 KiB/min after initial allocation.

7. Retention and reporting

Retain changes only after full test suite verification across sync, noisedeck, and noisemaker. Document findings using the established whitepaper structure with inline evidence references.

Evidence references

E1. Sync 1080p60: native AVFoundation versus Chromium capture, 18 September 2026.
E2. Sync 1080p60: controlled sender stage separation, 18 September 2026.
E3. Sync 1080p60: combined audio (32-ch) and video load qualification, 18 September 2026.
E4. Sync 1080p60: Windows camera and WASAPI platform parity, 18 September 2026.
E5. Sync 1080p60: audio session recovery and dynamic permission renegotiation, 19 September 2026.
E6. Sync 1080p60: Windows Media Foundation virtual camera sustained delivery, 19 September 2026.
E7. Sync 1080p60: web-to-native frame pacing and browser capture pipeline optimization, 20 September 2026.
E8. Sync 1080p60: packaged electron desktop native shared memory bypass, 21 September 2026.
E9. Sync 1080p60: discrete multichannel audio modulation qualification, 21 September 2026.
E10. Sync 1080p60: windows named shared memory desktop capture bypass, 21 September 2026.
E11. Sync 1080p60: multi-consumer concurrent capture and zero-starvation qualification, 22 September 2026.
E12. Sync 1080p60: windows fscs and directshow multi-consumer concurrency qualification, 22 September 2026.
E13. Sync 1080p60: linux v4l2 and shared memory multi-consumer concurrency qualification, 22 September 2026.
E14. Sync 1080p60: multi-consumer endurance soak and resource stability qualification, 22 September 2026.
E15. Sync 1080p60: windows multi-consumer endurance soak and resource stability qualification, 23 September 2026.
E16. Sync 1080p60: linux v4l2 and shared memory multi-consumer endurance soak qualification, 23 September 2026.