Vision Tracking
Arkestra uses Apple’s Vision framework to detect faces, hands, and body pose from a live camera feed or any texture in your project: a track, a layer, or a single effect’s output. All detected values are normalised 0–1 floats that can be mapped to any parameter exactly like an LFO or audio signal.
Detection runs on the Neural Engine where possible at ~15 fps, and the channels glide smoothly between detections at render rate.
Opening the Vision Tracking panel
Section titled “Opening the Vision Tracking panel”Click the Vision Tracking icon (figure silhouette) in the right sidebar. The panel is titled Vision Tracking.
Input source
Section titled “Input source”At the top of the panel, choose where Vision reads its frames from:
| Mode | What it analyses |
|---|---|
| Camera | A connected camera or webcam |
| Texture | A texture rendered in the current project: a track, a layer or one effect inside it, an offscreen renderer, or the projection output |
Click Camera or Texture to switch. The choice is saved per project — camera connections and tracking state are stored in the project file and restored automatically when you reopen it. If a previously used camera is no longer available, Arkestra warns you so you can pick another.
Camera mode
Section titled “Camera mode”Pick a camera from the Camera dropdown. Use the refresh button next to it if a newly connected camera does not appear. Quality sets the capture preset (Medium is enough for tracking alone; High is better when the camera is also used as a texture).
- Mirror Camera flips the camera image horizontally. It is a setting of the camera itself: it is remembered per camera and shared with every camera track that uses the same camera. The built-in FaceTime camera and Continuity Camera start mirrored (like FaceTime and Photo Booth); external cameras, capture devices and Desk View start unmirrored.
- Tracking always follows the image as it is shown, so the overlay and the X channels line up with the mirrored or unmirrored picture without any extra setting.
- If Arkestra does not have camera access, the panel says so and offers Open Privacy Settings.
- If the selected camera is unplugged, the panel shows it as disconnected and tracking resumes automatically when the same camera comes back.
Texture mode
Section titled “Texture mode”Under Source, pick what Vision analyses instead of a camera. The menu is the same texture picker used elsewhere in Arkestra:
- Tracks › Output — the track’s own rendered output, before its opacity and blend mode (other tracks are not included).
- Tracks › a layer or effect — the image at that point in the track’s chain, before anything after it is applied.
- Offscreen Renderers and Projection — when the project has them.
Use this to react to your own visuals — for example, drive a shader parameter from the motion of a performer in a video track.
If you draw the detections back onto the same track with the Vision Overlay effect, pick the video layer (or the effect just before Vision Overlay) as the source. If you pick the whole track, Vision would also see its own overlay lines.
Preview
Section titled “Preview”The Preview shows exactly the frames Vision analyses, in both Camera and Texture mode, with the detections drawn on top:
- Every detected face, hand and body is drawn faintly, so you can see everyone Vision found.
- The person driving the channels is drawn brightly: face box and centre dot (with a mouth-open bar when Landmarks are on), left hand in the Vision colour and right hand in purple (curl-sized fingertip dots, a yellow thumb–index line when pinching), and the body skeleton in orange.
Click Hide / Show next to the Preview header to collapse it while keeping tracking active. The preview costs nothing while hidden.
Active Detectors
Section titled “Active Detectors”Enable only the detectors you need:
| Toggle | What it enables | Default |
|---|---|---|
| Face | Face position, size, head angles, presence | On |
| Landmarks | Eye-open and mouth-open values (requires Face) | Off |
| Hand | Both left and right hand — wrist, fingertips, finger curl, pinch, presence | Off |
| Body | Body centre, wrists and the full skeleton, presence | Off |
Below the detectors, one status line per running detector shows how it is doing, for example Hands 15 Hz · ANE · 6.2 ms · 2 found: the measured analysis rate, the compute device (ANE = Neural Engine), the average analysis time and how many were detected in the last frame. Errors appear here in orange.
Follow
Section titled “Follow”When several people are in view, Follow chooses which one drives the Face and Body channels:
| Option | Follows |
|---|---|
| Most Stable (default) | The person tracked the longest. Someone walking into frame never takes over. |
| Largest | The biggest face or body, usually whoever is closest to the camera. |
| Left-most / Right-most | The person furthest to that side of the image. |
Switching only happens when the new candidate is clearly ahead, so two similar people do not make the target flicker. The hand channels are always the left and right hand of whoever Vision sees.
Invert X
Section titled “Invert X”All X channels follow the image as shown: X = 0 is the left edge of the picture, X = 1 the right edge. Turn on Invert X to flip every X channel (1 − x), for example when your projection faces the opposite way from the camera.
Projects saved before this option existed keep the X direction they had: Arkestra turns Invert X on for them where the old default flipped X. Turn it off in those projects to follow the image.
Smoothing
Section titled “Smoothing”Vision detects at about 15 Hz, while Arkestra renders at up to 120 fps. Smoothing (0–95%) does two things:
- It removes jitter with a filter per value that holds still values steady but follows fast movements with almost no lag. Higher settings hold still values steadier.
- It glides each channel between detections at render rate, so mapped parameters move continuously instead of stepping 15 times a second. This adds about one detection interval (~65 ms) of delay.
At Off (0%) channels are raw detections: no filtering, no gliding, lowest latency. 60% is the default. For slow, musical movement (a long rise or fall), use the smoothing on the individual mapping instead.
Live Values
Section titled “Live Values”The Live Values section shows a real-time readout of every active channel as a labelled bar chart. Active groups glow when a subject is detected. Use this to understand and tune your mappings without needing to open the parameter editor.
Mapping Vision to a parameter
Section titled “Mapping Vision to a parameter”- Select a track or effect and open the Parameter Inspector.
- Click the source pill on any parameter and choose Vision.
- A three-row picker appears for the mapping type:
| Row | Parent type | Extended sub-type |
|---|---|---|
| Face | Face (position, size, angles) | + Landmarks (eye open, mouth open) |
| Hand | L. Hand (wrist, index, pinch) | L. Fingers (all tips + per-finger curl) |
| Hand | R. Hand | R. Fingers |
| Body | Body (centre, both wrists) | Skeleton (full joint positions) |
Select the sub-type row, then pick a specific landmark from the grid below.
Available channels
Section titled “Available channels”All values are 0–1. Coordinate origin: X=0 is the left of the image as shown, X=1 the right (swapped with Invert X), Y=0 is bottom, Y=1 is top.
| Channel | Label | Notes |
|---|---|---|
| X | Horizontal position | 0 = left, 1 = right |
| Y | Vertical position | 0 = bottom, 1 = top |
| Size | Bounding box area (√width×height) | Larger when closer to camera |
| Roll | Head tilt left/right | 0.5 = upright |
| Yaw | Head turn left/right | 0.5 = facing forward |
| Pitch | Head tilt up/down | 0.5 = level |
| Presence | Whether a face is detected | 1 = detected; fades to 0 over about a second after it is lost |
Face Landmarks (requires + Landmarks)
Section titled “Face Landmarks (requires + Landmarks)”| Channel | Label | Notes |
|---|---|---|
| Eye L | Left eye openness | 0 = closed, 1 = wide open |
| Eye R | Right eye openness | |
| Mouth | Mouth openness | 0 = closed, 1 = open |
L. Hand / R. Hand
Section titled “L. Hand / R. Hand”| Channel | Label |
|---|---|
| Wrist X / Y | Wrist position |
| Index X / Y | Index fingertip position |
| Pinch | Distance between thumb tip and index tip — 0 = pinching, 1 = open |
| Presence | Detection confidence |
L. Fingers / R. Fingers (extended)
Section titled “L. Fingers / R. Fingers (extended)”| Channel | Label |
|---|---|
| Mid X / Y | Middle fingertip |
| Ring X / Y | Ring fingertip |
| Little X / Y | Pinky tip |
| Thumb Curl | 0 = extended, 1 = fully curled |
| Index Curl | |
| Mid Curl | |
| Ring Curl | |
| Little Curl |
| Channel | Label |
|---|---|
| Center X / Y | Hip midpoint |
| L.Wrist X / Y | Left wrist |
| R.Wrist X / Y | Right wrist |
| Presence | Detection confidence |
Skeleton (extended body)
Section titled “Skeleton (extended body)”Full joint set: Neck, Left/Right Shoulder, Left/Right Elbow, Left/Right Knee, Left/Right Ankle — each as X/Y pairs.
- Start with Face only — it is the cheapest detector and covers most interactive use cases.
- Use Presence as a modulator to drive a parameter to zero when no subject is in frame, avoiding frozen values.
- Pinch (0 = pinching, 1 = open) can gate effects with hand gestures. Invert it in the mapping if you want a pinch to drive a value up.
- In Texture mode, try pointing Vision at a feedback loop or particle track to create visual feedback that reacts to its own motion.
- The Live Values panel is the fastest way to verify a mapping before performing.