On-Device AI Across iOS 27: Spotlight and Media

In iOS 26, the on-device large language model lived inside your app. You opened a LanguageModelSession, attached tools, and the model reasoned over whatever context you handed it¹. iOS 27 moves the same model up a level. Apple now wires it into Core Spotlight so an app’s existing search index becomes a grounding source for the model², and into AVFoundation so the device transcribes and translates subtitles during playback without the app writing a line of inference code³. The shift is the headline: the on-device model is becoming a system service, not only an in-app API. The same intelligence that you call directly from your code also runs underneath search and underneath media, reachable through frameworks you already use.

This post covers two of those system surfaces from their WWDC26 sessions: LLM search through Core Spotlight (session 246) and Apple AI-generated subtitles with on-screen style preview (session 256). If you have not met the framework primitives behind the first one, the Foundation Models on-device LLM explainer and the iOS 27 tool-calling post set the stage.

TL;DR

• SpotlightSearchTool adopts the Foundation Models Tool protocol so a language model can search your app’s Core Spotlight index directly for contextual response generation. Apple ships it on iOS, iPadOS, macOS, and visionOS².
• You configure the tool in roughly one line, attach it to a LanguageModelSession, and the model generates its own queries, runs them against your index, and reasons over the results².
• A new index-delegate method, searchableItems(forIdentifiers:), lets you recover the full CSSearchableItem so the model sees metadata the compact search index cannot return².
• Guidance profiles scope the tool’s search capabilities for smaller models; custom pipeline stages let the index run computation (counts, averages, custom scores) on the model’s behalf².
• Apple AI-generated subtitles run live and locally during playback, with no app code required. They cover two paths: speech transcription from audio, and language translation from existing subtitles³.
• AVPlayerViewController gets generated subtitles and the new in-player style preview for free; AVPlayerLayer and AVCaptionRenderer expose the style-preview API for custom player UIs³.

What moved into the system

The iOS 26 story put one model in your process and gave you the Tool protocol to extend it¹. iOS 27 keeps that model and adds two places where the system reaches for it on your behalf. Core Spotlight becomes a retrieval surface the model can query, which turns the search index you already donate into grounded context. AVFoundation becomes a transcription and translation surface, which means subtitles in languages the original media never shipped.

The two features look unrelated until you notice the shared pattern. In both, Apple owns the model and the heavy lifting, and your app owns the content and the surface. You donate searchable items; the model writes the queries. You play a video; the device generates the captions. The intelligence is a shared system capability, and your job shifts from implementing it to feeding it the right content and presenting the result well.

LLM search through Core Spotlight

The setup, in Apple’s framing, is a hiking-trails app that already browses state parks and trails and lets the user write personal notes after each hike. The developer wants to ask the model open questions about those hikes. A bare LanguageModelSession answers from the model’s own world knowledge, which is the wrong source: the answers should come from hikes the app actually knows about. The app has already indexed every trail into a Core Spotlight search index, so the fix is to let the model reach into that index through tool-calling².

That is what SpotlightSearchTool does. Apple presents it as a tool that adopts the Tool protocol to let a language model directly search your app’s content in Core Spotlight for contextual response generation, available on iOS, iPadOS, macOS, and visionOS². The prerequisite is the work you would already have done: donate searchable content with Core Spotlight, the subject of Apple’s prior “Supporting semantic search with Core Spotlight” session²⁴. Once your items are donated, you import CoreSpotlight and FoundationModels, and Apple says the tool is ready to search your index in one line of code. You choose a model, whether the SystemLanguageModel or one supplied through the new Model Provider APIs, then add a SpotlightSearchTool instance to your session².

import CoreSpotlight
import FoundationModels

// Apple: "in one line of code, the tool is ready to search
// your app's Core Spotlight index."
let session = LanguageModelSession(tools: [SpotlightSearchTool()])

let response = try await session.respond(to: "What hikes have I gone on?")

The trajectory, not the query

The detail that earns attention is who writes the search. You do not. Apple describes the trajectory of a question like “What hikes have I gone on?” as the model deciding it needs SpotlightSearchTool, generating the query itself, Spotlight executing that query and returning a description of the result set, and the model reasoning over that output to generate its final response². The closing line of the session states the shift plainly: developers are not writing search queries anymore; they provide the content and let intelligence do the rest².

One gap shows up early, and Apple is candid about it. Some metadata in the Spotlight index, such as text content and HTML, is stored in a highly compact representation that can be searched but not recovered in a form a language model can read². The model finds the right items without being able to read everything donated for them. The fix lives on the index delegate you already implement to handle reindex requests. Apple adds a method to recover the full CSSearchableItem by its unique identifier, so the model can manage responses over potentially millions of results without loading them all².

// On your CSSearchableIndex delegate.
func searchableItems(
    forIdentifiers identifiers: [String]
) async -> [CSSearchableItem] {
    // Return the complete item for each id. The callback is also the
    // place to attach metadata you would not donate for search but
    // want the model to reason over.
    identifiers.compactMap { fullItem(for: $0) }
}

Apple makes the second point explicitly: if your app holds metadata that does not make sense to donate for search but would help the model reason, the delegate callback is the place to set those extra attributes on the item for the model to see².

Displaying results

Apple draws a clean line between two display modes. The session response is a concise description over the result set, which is what an assistant-style interface wants to show. For a list-style display, the results themselves are available directly on SpotlightSearchTool, which Apple calls the best way to access searchable items when the result set is large². Search replies arrive as an async sequence, each potentially carrying a batch of results until the tool call completes. Because the model may call the tool more than once before its final answer, Apple says to use the queryToken on each reply to decide when the UI should refresh².

Customizing for the model you choose

SpotlightSearchTool exposes a wide capability set, from semantic search over text to structured search over dates, persons, and locations². Apple gives three levers to fit that surface to the model you run. The first is the guidance profile. Handing a small model the tool’s entire capability set burns context it does not have, so a GuidanceProfile scopes guidance to only what the app needs, down to the exact list of metadata attributes the model should consider, with a dynamic guide level set when you create the tool². Apple’s advice for on-device models, which carry a more restricted context size, is focused guidance for simpler search capabilities².

The second lever is reference resolution. If the app donates person relationships and the user asks about other people on a trail, the model needs to know who “that person” refers to. Apple’s answer is a contact resolver that returns contact information related to the user’s identity, which the tool matches against metadata in the index².

The third lever is the one that turns search into computation. For a complex request like “how many trails have I hiked this year, and for each month, how many miles on average?”, Apple says the model may forgo a simple query in favor of a pipeline search that brings together index queries plus computation over the result set². The model breaks the question into stages: a search for completed hikes, a counting stage that builds a table by month, then a stage that computes an average over the counts². Pipeline stages are Generable, so the model generates a stage on demand from the prompt, and an app can register its own custom stages². Apple’s example is a happiness score computed over each item’s notes, perhaps via sentiment analysis or a five-star rating, with Guide properties telling the model which results to prefer².

// Apple's example: a Generable pipeline stage that scores items so the
// model reasons over the top-scoring results, not the raw set.
@Generable
struct HappinessScoredTrail {
    let item: CSSearchableItem
    @Guide(description: "Prefer higher scores; 5-star hikes rank highest.")
    let happinessScore: Double
}
// Register the stage on the tool's configuration.

Apple closes the section on verification rather than implementation, pointing at the Evaluations framework to measure how well the model calls the tool and how meaningful its responses are, with result coverage as the example metric for the hiking app². That evaluation loop is its own subject, covered in the Apple Evaluations framework post.

Generated subtitles, on device

The media surface shows the same principle from a different angle. Apple AI-generated subtitles are created live, locally on the device, as the media plays³. The framing in session 256 is accessibility first: subtitles matter for people who are deaf or hard of hearing, for people following spoken dialogue, and for anyone who simply cannot hear the audio in the moment. When content ships without a language a viewer understands, the device can fill the gap³.

Apple describes two paths. The first is speech transcription: source audio goes into the on-device speech-to-text model and subtitles come out³. The second is language translation: existing subtitles (English, in Apple’s example) go into the on-device translation model, which produces subtitles in another language such as Italian³. Authored subtitles, the ones a content creator wrote by hand, stay preferred and unchanged; generated subtitles only add languages³. The same on-device translation capability surfaces directly to developers in the Translation framework.

The part that matters for adoption: you implement nothing to turn this on. Apple states generated subtitles are available automatically during video playback³. They cover HTTP live streaming including live TV channels, video on demand, live events like sports, and file-based content such as app-bundled or downloaded media³. Both professional content and customer-created content (iPhone camera capture, social videos) qualify³.

The launch matrix is narrow and worth reading precisely. Starting in iOS and macOS 27, English subtitles can be generated from English audio, also supported on tvOS and visionOS 27³. On top of that, multiple subtitle languages can be generated from English subtitles on iOS and macOS³. So transcription is English-to-English at first across four platforms, and translation fans out from English on the two largest ones.

Where your work actually is

Generated subtitles cost nothing to enable, so your effort goes into selection UI. Apple’s guidance is direct: provide subtitle selection UI during playback³. Three options, in descending order of how much Apple hands you:

• AVPlayerViewController on iOS (and AVPlayerView on macOS) fully implements subtitle selection and player controls. You do nothing extra³.
• AVLegibleMediaOptionsMenuController presents the selection controls and behavior without player controls, which fits when you already have a player UI and only need the menu³.
• Custom controls for media selection, when you want the menu to match the rest of your app³.

In Apple’s demo, the generated options carry a sparkle symbol and the word “Translated” so users can tell them apart from authored tracks³.

The style preview

The second feature in the session is presentation. The Settings app has let people pick and create caption styles for years, but Apple’s point is that changing the style mid-video, with a live preview, is both easier and more accessible than leaving playback to hunt through Settings³. The same styles a user defined in Settings, including custom ones, now appear in the player’s style menu with a preview of each³.

AVPlayerViewController implements the style preview and player controls outright³. For custom players, AVPlayerLayer exposes a style-preview API, and AVCaptionRenderer can provide the preview if you take on rendering it yourself³. Apple’s AVPlayerLayer flow: each system style carries a profile ID, so you fetch all styles by their profile IDs, populate your UI with their names, and call the preview function when a user selects one. The new subtitle shows in the chosen style while any existing subtitles hide automatically so they do not interfere. Passing nil for the text parameter shows localized system text, and a position parameter offsets the preview to clear your controls. You call the function again for each new style, stop the preview when selection is done (which restores the active subtitles), and set the chosen style for all subtitles on the system³.

How to adopt

For Spotlight search:

1. Donate your content to Core Spotlight first; the tool searches the index, so no donation means nothing to search²⁴.
2. Attach SpotlightSearchTool() to a LanguageModelSession and pick your model, the SystemLanguageModel or one from the Model Provider APIs².
3. Implement searchableItems(forIdentifiers:) on your index delegate to return full items and attach model-only metadata that the compact index cannot recover².
4. For an assistant view, show the session response; for a list view, read CSSearchableItem results off the tool and refresh on each reply’s queryToken².
5. If you run an on-device model, scope capabilities with a GuidanceProfile, and reach for pipeline stages only when a request needs counting or scoring across a large set².

For generated subtitles:

1. Nothing turns the feature on; it runs automatically during playback³.
2. Make sure your player exposes subtitle selection UI through AVPlayerViewController, AVLegibleMediaOptionsMenuController, or your own controls³.
3. Add the style preview through AVPlayerViewController for free, or wire AVPlayerLayer’s preview API into a custom player³.
4. Read the support matrix before you promise languages: English-to-English transcription across four platforms, English-source translation on iOS and macOS³.

FAQ

What is SpotlightSearchTool in iOS 27?

It is a Foundation Models tool that adopts the Tool protocol so a language model can search your app’s Core Spotlight index directly and use the results for response generation. Apple ships it on iOS, iPadOS, macOS, and visionOS. You attach a SpotlightSearchTool instance to a LanguageModelSession, and the model generates the search queries itself rather than you writing them².

Do I have to write the search queries?

No. Apple’s framing is that developers are not writing search queries anymore; they provide the content and let the model do the rest. The model decides it needs the tool, generates a query, Spotlight runs it and returns a description of the result set, and the model reasons over that output to produce its answer².

Why can’t the model see all my donated metadata?

Some Spotlight metadata, such as text content and HTML, is stored in a highly compact form that can be searched but not recovered in a readable way. To give the model the full picture, implement searchableItems(forIdentifiers:) on your index delegate to return the complete CSSearchableItem, and use that callback to attach any extra model-only attributes².

Do I need to implement anything to get generated subtitles?

No. Apple states generated subtitles are available automatically during video playback, created live and locally on the device. Your work is the selection UI and, optionally, the in-player style preview, not the inference³.

Which languages do generated subtitles support at launch?

Starting in iOS and macOS 27, English subtitles can be generated from English audio, also on tvOS and visionOS 27. Separately, multiple subtitle languages can be generated from English subtitles on iOS and macOS. So transcription is English-to-English across four platforms, and translation fans out from English on iOS and macOS³.

Are generated subtitles and Spotlight search private?

Both run on device. Generated subtitles are created live and locally as media plays³, and SpotlightSearchTool queries your app’s local Core Spotlight index through the on-device model¹². Apple supplying the system implementation changes who maintains it, not where it runs.

The full Apple Ecosystem cluster: the Foundation Models on-device LLM and the Tool protocol that SpotlightSearchTool adopts; the iOS 27 tool-calling control that governs how aggressively the model calls tools like this one; the parallel App Intents in iOS 27 surface for background execution, sync, and Spotlight; and Core AI: running models on Apple Silicon for the layer beneath the system model. The hub is at the Apple Ecosystem Series. For broader iOS-with-AI-agents context, see the iOS Agent Development guide.