Live Depth

Arkestra can lift a live 2D picture into 3D while it plays. Run a video, a camera or a generator on one track, and see it as a field of Gaussian splats or a point cloud on a 3D track, with real depth.

Depth can come from three places:

Depth	What it uses
Luma	Brightness as height. Instant, no setup
Depth Texture	A second texture that already holds depth — a depth-capable camera, or your own depth pass
Estimated (ML)	Depth estimated from the picture itself by a machine-learning model running on the Neural Engine

Live texture splats

1. Add a Gaussian Splat object to a 3D track.
2. In its card, set Input to Live Texture.
3. In the LIVE TEXTURE section, pick the Source — any track, chain or effect output.
4. Choose a Depth mode.

Every frame, the source is cut into a grid and each cell becomes one splat: a flat disc that lies on the depth surface and tilts with it, so neighbouring splats keep touching on slopes. Everything else on the splat card still applies — lighting, the HSV colorizer, displacement and Render as Point Cloud.

While Input is Live Texture, the splat playlist pauses and its SPLAT PLAYLIST, SPLAT DATA and PLACEMENT sections are hidden. Switch back to Splat Files to return to the playlist.

Controls

Control	Description
Source	The track, chain or effect output to lift
Depth	Luma, Depth Texture or Estimated (ML)
Depth Source	Depth Texture only. The texture to read depth from. Brighter is nearer
Sync	Estimated (ML) only. See Sync
Resolution	Splats along the source’s longer side: Low (128), Medium (256), High (384), Very High (512) or Ultra (640). The shorter side follows the source’s aspect ratio
Depth (slider)	How deep the scene is from nearest to farthest
Perspective	0 is flat relief. Higher values spread far areas out and pull near areas in, as a real camera would
Edge Separation	Where depth jumps (a person in front of a wall): 0 stretches splats across the gap like a rubber sheet, 1 lets the foreground separate cleanly from the background
Splat Overlap	Splat size relative to the grid spacing. Raise it to close gaps, lower it for a dotted look
Depth Smoothing	Smooths depth over time. Hides flicker in estimated depth; set to 0 for an instant response
Luma Alpha	0 draws every splat opaque, 1 makes dark areas transparent
Black Clip	Splats darker than this are dropped, so black backgrounds fall away

Higher Resolution costs more GPU time and fill rate. High (384) is a good default; lower it on slower Macs or when the splats cover the whole screen.

Sync

Estimated depth is computed a moment after the frame it came from. Sync chooses how colour lines up with it:

Sync	Behaviour
Locked (default)	Colour comes from the same frame the depth was estimated from, so they always match. The splats update at the depth rate. Camera moves, lighting and displacement still run at full frame rate
Live Color	The current frame’s colour on the latest depth. Smoother motion, but silhouettes trail slightly behind the picture, and a hard cut shows the new image on the old shape for a moment

Depth on a point cloud

A point cloud in Texture input mode can use estimated depth too. In the XYZ MAPPING section, set Z Axis (or Relief on a wrapped shape) to Depth (ML). X and Y can use it as well.

A point cloud and a splat reading the same source share one depth estimate, so using both costs no extra.

Estimated depth

Estimated (ML) uses Depth Anything V2, a depth-estimation model, running on your Mac’s Neural Engine rather than the GPU, so it doesn’t slow down rendering.

• Quality: pick how much detail to trade for speed. The setting applies across the whole app.

  Quality	Detail	Updates per second (approx.)
  Fast	Softer edges	Every frame
  Balanced (default)	Close to Detailed on real footage	~85
  Detailed	Finest	~45

• Stabilize: Off, Low, Medium (default) or High. Holds depth steady from frame to frame: it stops the whole scene from pumping in and out, and smooths flicker in areas that aren’t moving, while things that do move still follow quickly. Like Quality, it applies across the whole app.

• First use: each quality’s model (about 50 MB) downloads once, then is optimised for your Mac. This needs an internet connection and can take a little while. The Model row shows the progress: Downloading, Optimizing for this Mac…, Loading…, then Ready. If it fails, press Retry.

• Until it’s ready, depth falls back to Luma, so the scene never goes empty.

• After that, it loads from disk in a moment, with no download.

• Settings → General → Downloaded Models lists every model Arkestra has downloaded, with its size and what uses it. Delete frees the space; the model downloads again the next time you use that quality.

• Depth Rate shows how often depth updates and how long each estimate takes. Depth updates on its own schedule, independently of the frame rate.

The model estimates relative depth from a single picture: it knows what is in front of what, not real distances, and its sense of scale can shift from frame to frame. Expect some wobble on busy or fast-changing content. Stabilize removes most of it; on splats, Depth Smoothing adds further smoothing.

Tip

Estimated depth works best on footage with clear foreground and background — people, objects, landscapes. Flat graphics and abstract generators have no real depth to find; use Luma for those.

Related

• Point Clouds — texture input mode and XYZ mapping
• Point Cloud Displacement — displacement also applies to live splats