Fivefold HighVis R&D

Unreal + AI:
Keeping Control, Gaining Realism

9 April 2026 — Day One Summary

The Starting Point

Unreal Engine gives a production team precise control over camera position, scene composition, spatial layout, and timing. The limitation is that producing a photorealistic result from Unreal requires significant time, specialist artists, and iterative refinement of materials and lighting. AI image generation works the other way: it produces photographic-looking output quickly and cheaply, but the composition is largely outside the operator's control. Prompt adjustments shift the look rather than the framing.

What We Tested

Two scenes were tested in a single day, each using a different type of Unreal output as the composition source. The first was a spaceship interior rendered as an untextured grey pass. The second was a Riyadh desert cityscape output as a depth map. Both followed the same principle: Unreal locks the composition, AI applies the look.

Spaceship — Untextured Pass

The plain grey render was passed directly into Nano Banana Pro with a prompt describing the target look and materials. The spatial structure from Unreal stayed intact while the AI applied surface detail, lighting, and atmosphere. Three video generation models were then run against the same source animation to compare motion quality.

Cityscape — Depth Map Pass

A depth map was chosen over a flat colour render because depth gradients encode surface orientation and scene geometry more explicitly, giving the AI more spatial information to work from. Flux 2 Pro produced the strongest base image for the Riyadh night scene. A placement issue with one tower required a two-step edit before the image went to Magnific and then Kling for video generation.

Challenges & Flags

Composition Drift in Image Gen

When a depth map and a photographic style reference were combined in the Weavy interface, Nano Banana began reinterpreting the composition from the depth map rather than preserving it. The platform appeared to treat the style reference as a layout input rather than appearance-only. This is most likely a Weavy-specific behaviour. Direct NB prompting without Weavy as an intermediary should be tested to confirm whether the separation can be maintained cleanly.

Videogame Look on First Pass

First-pass cityscape outputs looked rendered rather than photographic. A depth map carries spatial information but no material cues, and the model defaulted to a CG appearance. Adding a photographic style reference image alongside the depth map and including explicit hard constraints in the prompt — not CGI, not illustrated, not a video game — resolved this for Flux 2 Pro. The constraint language was required; the style reference alone was not enough.

Structural Edits Need Two Steps

Attempting to remove and reposition a tower in a single prompt failed on both Flux and NB. Neither model produced a clean result when asked to do both operations at once. Splitting the instruction into two passes — remove first, reintroduce second — worked on the first attempt. Single-prompt structural edits are unreliable. Two-step editing should be the default for any structural change.

Video Generation Cap: 10 Seconds

All three video generation tools capped output at approximately 10 seconds, regardless of input clip length. Camera moves will need to be designed within that constraint, or the pipeline will need an additional stitching step to cover longer durations.

Results & Learnings

Depth Map vs Untextured Render

The depth map gave the AI more geometry information than the plain grey render. Surface orientation and spatial depth are implicit in depth gradients in a way that a flat grey pass does not provide. The cityscape result was stronger on first pass than the spaceship, though the spaceship pipeline had fewer steps and required no structural editing. Higher-fidelity Unreal outputs — normal maps, lit renders, beauty passes — are worth testing to see how much the source quality affects the final image.

Model Comparison — Video Gen

Kling o1, Luma Modify, and Wan 2.7 were all tested on the spaceship scene. All three preserved the Unreal camera motion in the output. Temporal stability and image quality varied between models. The Comparisons page shows all three side by side against the original Unreal render — the differences are visible and worth evaluating in context rather than in isolation.

Near-Realtime: Where We Are

The full pipeline — Unreal render to final video — ran within a single working session. Individual stages each took minutes. The bottleneck was not the tools but the iteration: prompt refinement, multi-step structural edits, and switching between platforms added time that a validated workflow would eliminate. Building a prompt library — verified prompts per scene type, per tool — is the most direct path to reducing that overhead.

Recommendations for Next Session

1. Test Nano Banana directly outside Weavy with depth map and style reference combined. Confirm whether the composition drift is a Weavy interface issue or an NB model behaviour.

2. Design all camera moves for 10s maximum until a clip-stitching workflow has been validated and tested for seam artefacts.

3. Build a validated prompt library — one confirmed prompt per scene type (interior, exterior, day, night) significantly reduces first-pass iteration time.

4. Test higher-fidelity Unreal source outputs: lit render, normal map, and beauty pass. The quality of the source affects how much work the AI has to do to reach a photographic result.