Which native app tasks can on-device LLMs take on?

What can people finish more quickly after a model is added?

An on-device LLM gives an application a way to process text without a model-server connection. Calling the model is different from shortening a user’s task. Summarizing a note, extracting an event candidate from email, or adding classification tags provides a concrete input, expected result, and place to correct errors.

Apple’s Foundation Models task documentation lists summarization, entity extraction, editing, and classification among on-device uses. These can become actions inside existing note and editing interfaces. I would use such defined features as the unit for deciding whether to adopt an on-device LLM.

Even “small model” needs qualification. In Apple’s June announcement of its third-generation family, AFM 3 Core is a three-billion-parameter dense model, while AFM 3 Core Advanced has 20 billion total parameters and activates one to four billion depending on the request. Total and active size differ. A model-family announcement also does not establish that every app can select every model. Check the API’s actual model and device support, then evaluate the feature.

Android offers dedicated tasks and custom prompting

ML Kit’s GenAI Prompt API accepts custom requests to Gemini Nano for text and image tasks. Its documentation contrasts flexible prompting with APIs tuned for summarization, proofreading, rewriting, and image description. Custom prompting increases prompt-design and quality-assurance work; dedicated APIs restrict the task and output while lowering integration effort.

The Summarization API, for example, produces one to three bullets from articles or conversations and includes Japanese among supported languages. Actual feature/language availability still depends on device configuration and downloaded models. Extracting only meeting decisions with owners and deadlines is a different requirement from a fixed bullet summary.

I would first test a dedicated API where its output fits the user’s task. Move to custom prompting when the application needs its own extraction or classification behavior. Greater flexibility also increases responsibility for defining errors and their correction interface. A common model entry point does not equalize the work required to complete different features.

Typed values still need semantic verification

Displaying a raw response makes dates and categories difficult to handle consistently. Apple’s guided generation generates application-defined Swift types. It reduces manual parsing and supports displaying event candidates, names, and summaries in separate fields.

Android’s Structured Output API uses Kotlin types and annotations. At this article’s cutoff, its documentation labels it alpha, while Prompt API is beta. Structured output has a separate availability check. Treating both as the same release stage would obscure whether a required feature actually works on the target device.

A result matching a type can still misrepresent the source. A properly formatted date may have no basis in the original text. Easier connection to displays and storage makes the remaining responsibility clearer: verify correspondence with the input. Structured output establishes neither semantic accuracy nor completion of an application action.

Consider one feature: turning a note into an event candidate

This is a hypothetical implementation example. A note says, “Online meeting with Tanaka tomorrow at 15:00.” The model extracts a person, time, and location candidate. The application displays the original text alongside it and identifies the reference date and time zone used for “tomorrow.” If note creation and processing occurred on different days, silently choosing a reference and saving would be inappropriate.

The user corrects the date and selects Save. Only after the calendar operation succeeds does the application report that it added the event. The model provides a candidate; application code establishes the save outcome. Text containing no event, or a vague wish to talk sometime next week, should support no extraction or an unresolved candidate rather than a fabricated definite date.

This separates text recognition, candidate generation, correction, and calendar storage for evaluation. High extraction accuracy may still fail to save time if checking the result takes too long. Preserving the original text and providing a route back to ordinary editing are part of the value of native integration.

Local inference differs from an entirely local feature

The current Foundation Models framework supports on-device models, Private Cloud Compute, and third-party server models through the same Swift API. Dynamic Profiles can change models, instructions, and tools within a session. The framework’s name therefore cannot establish that all input stays on the device.

Tool calling can invoke application code to retrieve information or perform actions. A local model calling a network-connected tool still creates an external data path. Analytics and synchronization also require examining where source text, generated results, and conversation history go.

Local inference has a concrete benefit: it can process input without sending it to a model server. A product promising this should match its actual data path to that promise. Silently replacing an unavailable local feature with cloud processing changes the commitment. If cloud processing is offered, make its transmitted data and switching conditions understandable and selectable.

Unavailability and preparation belong in the normal interface

Apple documents availability checks covering device and region support, Apple Intelligence settings, and model download, with a fallback experience when unavailable. ML Kit’s setup guide likewise distinguishes unavailable, downloadable, downloading, and available states. Owning a supported device does not establish immediate readiness.

The ML Kit overview notes that Nano versions differ across devices and may produce different outputs from the same prompt. It also specifies per-app inference and battery-use quotas. Inference is allowed only in the top foreground app; a foreground service does not permit use while the app itself is elsewhere. That affects a design intended to process a large collection of notes after the screen closes.

When summarization is unavailable, people should still be able to read and edit the note. Show preparation honestly and avoid duplicating requests after repeated taps. If someone moves to another note while generation continues, a late result must not overwrite the new note. Model accuracy does not measure these behaviors, yet ordinary app usability depends on them.

Measure Japanese accuracy and time through correction

Apple’s language documentation provides runtime locale checks. Language support is not evidence of sufficient accuracy for a feature’s inputs. For Japanese event extraction, I would include relative dates, Japanese eras, honorifics, omitted subjects, multiple people’s events, and text with no event.

Google’s evaluation guidance recommends task-specific inputs and expected outputs evaluated across Nano versions. Apple’s Evaluations framework also addresses feature datasets and metrics. Measure incorrect dates and people for extraction, or omitted conditions for summaries. Conforming to a type or producing natural prose does not measure the user’s correction burden.

Distinguish first visible output from finishing verification and the intended action. Apple’s Instruments demonstration traces input processing, generation, tool calls, and total latency. Real-device evaluation should also cover initial preparation, long inputs, repeated use, and battery consumption. This article performs no device comparison and does not conclude that local inference is always faster or more efficient.

I would compare one feature with manual entry or an existing dedicated process, keeping failures, unavailability, cancellations, and human corrections in the record. Measure completed tasks and total time. The value of an on-device LLM depends on whether the application helps people verify, correct, and act on its output, alongside the model’s size.