Building a Custom HTML5 Canvas + Synced Video Engine for Real-Time Combat Dialogue (V.E.T.O.R — A Ruptura)

I am building V.E.T.O.R — A Ruptura, an independent Brazilian action-adventure PC/Windows game running on a custom client engine built with HTML5 Canvas + synchronized MP4 video scenes instead of Unity, Unreal, or Godot.

Here is a breakdown of the technical architecture and two recent gameplay milestones showing real-time companion dialogue and multi-language voice switching in combat.

1. Custom 2D Canvas + Synced MP4 Scene Deck (1280×720)

Rather than rendering complex lighting and hand-animated cutscenes through heavy 2D shader pipelines, the client layers an HTML5 Canvas loop on top of a synchronized deck of 1280×720 MP4 clips:

• Proximity-based preloading: scenes and transitions preload ahead of player movement to minimize hitching when crossing boundaries.
• Frame-aligned visual overlays: foreground props (vegetation, cover obstacles, seam patches) mask and composite dynamically over the active video frame.
• Client-side combat physics: movement, knockback/recoil, hit windows, guard timing, HUD bars, and target focus run synchronously in Canvas over the video layer.

2. Real-Time Combat Dialogue & Companion Assistance

Instead of static branching menus, NPCs, allies, and bosses respond dynamically to player input during encounters. A tolerance and context controller translates tone and dialogue into combat consequences (triggering escalation, opening tactical windows, or prompting weak-point scans from your floating robot companion VETOR).

In this recent encounter against Tharok, ally Albert and VETOR interact mid-fight:

3. Live 7-Language Dubbing Pipeline

The game already supports voice acting and UI across 7 languages (Brazilian Portuguese, English, Spanish, French, German, Italian, and Dutch), switching languages coherently inside the same battle sequence:

Current Project Status & Questions

• V.E.T.O.R — A Ruptura is in active development for PC/Windows (no public demo yet).
• Project overview and additional clips: https://vetorlogico.com

For anyone who has worked with Canvas + real-time media or dynamic NPC dialogue:

1. How do you balance combat pacing with readable real-time companion/boss dialogue without overwhelming player attention?
2. What memory or synchronization edge cases would you monitor most closely when combining Canvas gameplay loops with synced video streams?