What's New in visionOS 27 for Spatial Devs

visionOS started with one way onto the platform that mattered: build a native app with SwiftUI and RealityKit, or check the “Designed for iPad” box and ship a panel. visionOS 27 adds a third doorway and widens the second. You can now extend spatial content straight from a Mac app with no visionOS build at all, stream a full OpenXR experience from a PC over Wi-Fi, and put a real 3D environment behind a website in Safari. The release reads less like a feature list and more like Apple answering the question of where your existing work already lives, then meeting it there.¹

The pattern across the announcements is reach. Object tracking reaches iOS with the same trained reference objects. Spatial Preview reaches Vision Pro from Preview on macOS. Foveated Streaming reaches a Windows PC. The Spatial Web reaches a webpage. Apple spent this cycle reducing the distance between content you already have and a spatial presentation of it, and the four new entry points are the proof. The sections below take each one in turn, with the session moment where Apple demonstrates it and the attributed claim that anchors it.

In session 287, Apple, the Vision Products Group, describes the M5-generation Apple Vision Pro as desktop-class compute with over 4K pixels per eye and 90Hz hand tracking, then frames the new third path as a way to bring existing macOS and PC experiences onto the platform rather than rebuilding them.²

TL;DR / Key Takeaways

• Spatial Preview, a new macOS 27 framework, sends spatial photos, Apple Immersive Video frames, PDFs, and live USD scenes from a Mac to Vision Pro through Quick Look, with no visionOS app required and SharePlay review built in.³⁴
• Foveated Streaming (introduced in visionOS 26) connects Vision Pro to a PC to stream OpenXR content over Wi-Fi, compressing the stream by gaze using NVIDIA CloudXR, with hand and controller input sent back automatically.⁵
• Object tracking gains high-frame-rate tracking, an extended Create ML training mode for hand-held objects, and a metric-space pose API for measurement, and it now runs on iOS with the same reference objects.⁶
• Anyone can now build a spatial accessory: a board with tracking LEDs, an IMU, and Bluetooth, discovered through GCSpatialAccessory and trained with a Create ML bundle into a reference accessory file.⁶
• The Spatial Web grows up: Safari windows widen and curve, Web Environments are on by default, and a JavaScript requestImmersive API (modeled on Fullscreen) lets a webpage transport visitors inside a 3D environment.²⁷
• RealityKit adds Projective Textures, Cloth Simulation, a Custom Reverb Mesh, and Gaussian Splatting; Reality Composer Pro 3 adds an AI assistant, Animation Graph, Script Graph, and navigation meshes.²

The Third Path: Spatial Preview From the Mac

The most quietly significant addition in the release needs no visionOS code at all. Spatial Preview is a macOS 27 framework that lets a Mac app push spatial content onto a paired Vision Pro and update it live. Quick Look receives the content on the visionOS side, so a creator can move around a 3D scene at full scale, swap camera viewpoints, apply material overrides, and leave annotations without anyone writing a line of visionOS UI.³

In session 282, Apple’s visionOS team explains that the framework offers two session types: document preview sessions for spatial photos, videos, and document types like PDF, and USD preview sessions for 3D content, with Quick Look launching on Vision Pro and no code required on the visionOS side.⁸ The endpoint can be the device already connected through Mac Virtual Display, or any nearby Vision Pro on the same iCloud account chosen through a device picker.⁸

The 3D path is where the framework earns its keep. A USD preview session takes a USD stage, opens it in a volumetric view on Vision Pro, and lets the person step into a full-scale immersive view. Edits flow both directions over a live USD stage, so a layout change made in the Mac app appears spatially on Vision Pro, and an annotation added in the headset reflects back on the Mac. Apple notes that the framework optimizes heavy USD scenes before sharing, with mesh decimation, texture downsampling, and full scene reconstruction when necessary, and that a reconstructed scene stays viewable and annotatable even when it is no longer editable.⁹ SharePlay support means a second reviewer can join the same live session, which turns asset review from a render-and-email loop into a shared room. Apple cites Cinema 4D and SketchUp as the kind of content tools the workflow targets.³⁹

Foveated Streaming: A PC Inside the Headset

The second widened path runs the other way: instead of pushing content from a Mac, you stream a full OpenXR experience from a Windows PC. Foveated Streaming, which Apple introduced in visionOS 26, lets Vision Pro connect to an external device, send input back (hands, controller positions, microphone), and receive video and audio rendered remotely.⁵ The name describes the trick: the system compresses the stream by gaze, sending the region you are focused on at higher detail and the periphery at lower bandwidth, using Vision Pro’s eye tracking.⁵

In session 286, Apple’s engineering team states that NVIDIA CloudXR is built into visionOS, performant enough to stream over Wi-Fi from a local PC or the cloud with no cables, and that a team can get an OpenXR application streaming in about a day and add visionOS-only capabilities within a week.¹⁰ The architecture has two halves: a visionOS receiver app built on the FoveatedStreamingSession API, and an OpenXR host app that implements Apple’s Foveated Streaming Protocol (a lightweight TCP connection that handles authentication and barcode pairing) and uses the CloudXR OpenXR runtime.¹⁰

The detail that makes this more than a remote desktop is the integration with native frameworks. The receiver presents streamed content through an ImmersiveSpace, and Apple recommends a progressive immersion style so the experience appears through a portal grounded in the room.¹⁰ A message-channels API on the session carries opaque data both ways, so a SwiftUI level-select can trigger a load on the PC, and ARKit data from the headset can align the streamed scene to physical equipment. X-Plane 12 from Laminar Research uses exactly that: ARKit finds a physical flight simulator, message channels sync its position to the PC, and the real and virtual cockpits line up.⁵¹⁰ Supplying a depth buffer even lets streamed content and on-device RealityKit content occlude each other.¹⁰

Immersive Environments and the Spatial Web

Two sessions cover the environment story from opposite ends: the craft of building a photorealistic environment, and the new ability to put one behind a webpage. Both matter because visionOS 27 turns Web Environments on by default, so a Safari window can now sit inside a background the way a native app does.²

In session 320, Apple’s visionOS Safari team introduces an immersive API modeled on the JavaScript Fullscreen API: where you would call requestFullscreen on a video, you call requestImmersive on an HTML model element to transport it beyond the browser bounds while the page stays visible.⁷ The engineer walks through a venue site that previews the view from a chosen seat inline, then enters the theater immersively on a tap, and stresses two reference-frame rules worth internalizing: an inline model is centered in its layer at CSS scale, while an immersive model is anchored at the person’s feet at real-world scale, and the immersive environment opens from behind the Safari window, so you keep the main focus where the window does not hide it.⁷ A practical optimization closes the loop: setting display: none on the model element hides the inline preview and defers downloading and decoding the asset until the immersive request fires, saving bandwidth for visitors who never enter.⁷

The design discipline behind a convincing environment is its own subject, and the principles transfer directly to web environments.

In session 234, Apple’s environment design team notes that a fully immersed viewer sees approximately 81 degrees of the scene in their field of view, that an ideal 360-degree panorama targets 14,400 by 7,200 pixels (sharp at 40 pixels per degree), and that expensive motion (evolving clouds, swaying palm fronds, breaking waves in the Bora Bora environment) gets approximated cheaply through UV flow maps, scrolling shadow masks, and layered sine-wave vertex animation rather than real-time simulation.¹¹ The session’s closing advice doubles as a rule for any spatial work: be intentional about everything you add or remove, and try to break your own design before trusting it.¹¹

Object Tracking Grows Up and Crosses to iOS

Object tracking, which arrived in visionOS 2.0, turns a physical object into a virtual anchor: you start with a USDZ model, train a reference object in Create ML on a Mac, and pass it to the tracking API to receive position and orientation updates.⁶ visionOS 27 sharpens the capability in three ways and extends it to a second platform.

In session 283, Apple’s visionOS team details the three upgrades: high-frame-rate tracking for objects moving through space, an extended Create ML training mode that improves accuracy and robustness for hand-held objects (at the cost of significantly longer training time), and a coordinate-space correction API that returns a pose either rendered (with display corrections so virtual content aligns visually with the object) or in true metric space with no corrections, which is what measurement use cases like a surgical-training probe require.¹² The session also confirms a useful fallback: when you cannot get a photorealistic 3D model, you can 3D-print a marker, mount it on the object, and use the marker as the reference object.¹²

The cross-platform move is the headline. Object tracking now runs on iOS through ARKit, and because Create ML training is platform-agnostic, the same trained reference object works on both iOS and visionOS with the same tracking quality.⁶¹² On iOS you load reference objects into a world-tracking configuration, assign each to either stationary detection or high-frame-rate tracking, and handle the resulting object anchors in your session delegate.¹²

The companion story is spatial accessories. visionOS 26 shipped the Logitech Muse and PSVR2 Sense controllers; visionOS 27 lets anyone build one. A spatial accessory is a board carrying a constellation of tracking LEDs, an IMU for orientation and acceleration, and a Bluetooth chip, plus optional buttons, touchpads, and haptics, installed into any object you want to track.⁶ You validate the build with an ARKit accessory-tracking debug view in developer settings, train it with a Create ML bundle (a USDZ annotated with LED and IMU positions) into a reference accessory file, then discover it through the new GCSpatialAccessory class and run an accessory tracking provider.¹² Apple says manufacturers DFRobot and MikroE will ship off-the-shelf reference hardware and development kits later this year, so plug-and-play parts exist for teams that do not want to fabricate a board.²¹² The tradeoff against object tracking is clear from the session: object tracking wins on measurement precision, while a spatial accessory wins on refresh rate and latency for fast-moving objects, and adds physical buttons and haptics.¹²

RealityKit and Reality Composer Pro 3

The native path did not stand still. RealityKit gains Projective Textures (textured spotlights for effects like stained-glass projection or underwater caustics), Cloth Simulation that drapes and folds fabric in real time, a Custom Reverb Mesh that models how sound scatters off wood, metal, and stone in a space, and Gaussian Splatting to scan a real object and bring it in as a 3D splat.² Reality Composer Pro 3 is a larger refresh: an AI assistant that generates 3D models from a description, an Animation Graph state machine for runtime transitions, a node-based Script Graph for interaction logic with live preview on Vision Pro, and navigation meshes for characters to path around obstacles, all without opening Xcode.² For teams already invested in the native stack, the editor work is the part that changes day-to-day velocity most.

What to Adopt First

A release with four new entry points rewards triage. Start where the distance from your existing work is shortest.

1. If you have a Mac content app, prototype Spatial Preview. It ships in the macOS Preview app, exposes the same features as an API, and needs no visionOS build, so the cost of a first spatial review workflow is close to zero.³⁸
2. If you ship a web product, turn on a Web Environment and test requestImmersive. Web Environments are on by default, the immersive API mirrors the Fullscreen API you already know, and a single model element plus one call is enough to drop a visitor into a scene.²⁷
3. If you have an OpenXR or PC experience, stream it. Apple’s open-source samples and reference protocol implementation put a Foveated Streaming connection within a day’s reach, and native SwiftUI, ARKit, and RealityKit integration is additive from there.¹⁰
4. If your app already does object tracking, take the metric-space pose and the iOS port. The coordinate-space correction API enables measurement, and the same reference objects now run on iPhone and iPad with no retraining.⁶¹²
5. Treat custom spatial accessories as a later bet. Building a board is the highest-effort path; wait for the DFRobot and MikroE reference kits unless fast-motion input with haptics is core to your experience.¹²

The throughline matches the release: adopt the path that meets your existing content where it lives, and defer the one that asks you to build new hardware.

FAQ

What is the Spatial Preview framework in visionOS 27?

Spatial Preview is a macOS 27 framework that lets a Mac app send spatial content (spatial photos, Apple Immersive Video frames, PDFs, standard images, and live USD 3D scenes) to a paired Apple Vision Pro, where Quick Look displays it with no visionOS app required.³⁸ It offers document preview sessions for media and document types and USD preview sessions for 3D content, supports live two-way editing of a USD stage, and includes SharePlay so collaborators can review the same scene together.⁸⁹

How does Foveated Streaming work and what does it need?

Foveated Streaming connects Vision Pro to an external device such as a Windows PC to stream OpenXR content as video and audio, while visionOS sends input (hands, controller positions, microphone) back automatically.⁵ It compresses the stream by gaze using Vision Pro’s eye tracking, runs on NVIDIA CloudXR built into visionOS, and works over Wi-Fi with no cables. You build a visionOS receiver on FoveatedStreamingSession and a host app that implements Apple’s Foveated Streaming Protocol and the CloudXR OpenXR runtime.⁵¹⁰

Can I use object tracking on iPhone now?

Yes. visionOS 27 brings object tracking to iOS through ARKit, and because Create ML training is platform-agnostic, a reference object you trained for visionOS works on iOS with the same tracking quality.⁶¹² On iOS you load reference objects into a world-tracking configuration, assign each to stationary detection or high-frame-rate tracking, and handle the object anchors in your session delegate.¹²

How do I put a 3D environment behind my website in Safari?

Use the HTML model element with a USDZ asset and call the JavaScript requestImmersive API in response to a user interaction, which mirrors the Fullscreen API pattern.⁷ Web Environments are enabled by default in Safari on visionOS 27, so a webpage can have a 3D background like an app.² Remember that an immersive model anchors at the person’s feet at real-world scale and opens from behind the Safari window, and that display: none on the model defers asset download until the immersive request fires.⁷

What is a spatial accessory and do I have to build the hardware?

A spatial accessory is an electronic device with a constellation of tracking LEDs, an IMU, and a Bluetooth chip (plus optional buttons, touchpads, and haptics) that Apple Vision Pro tracks at high frequency with low latency.⁶¹² visionOS 27 lets anyone build one, discovered through GCSpatialAccessory and trained into a reference accessory file with a Create ML bundle.¹² You do not have to fabricate a board: Apple says DFRobot and MikroE will ship off-the-shelf reference hardware and development kits later this year.²¹²

The full Apple Ecosystem cluster: the visionOS spatial patterns that name the Windows, Volumes, and Immersive Spaces these new paths render into; the RealityKit spatial mental model underneath the 3D content layer Spatial Preview and Foveated Streaming compose with; the discipline of shipping one SwiftUI app across five Apple platforms that object tracking’s iOS-and-visionOS reach now extends; and the Apple platform matrix that maps where each capability lands. The hub is at the Apple Ecosystem Series. For broader iOS-with-AI-agents context, see the iOS Agent Development guide.

References