Apple Intelligence — all the new AI features coming to iOS 18, iPadOS 18 and macOS Sequoia
When you buy through links on our articles, Future and its syndication partners may earn a commission.
Apple unveiled Apple Intelligence, its take on generative AI, at WWDC 2024 — and now it's finally come to developer betas for iOS 18.1, iPadOS 18.1 and macOS Sequoia 15.1. Described as a form of “personal intelligence,” it “makes your most personal products more useful and delightful,” according to Tim Cook.
Built on the top of multiple large language and diffusion models running on device, Apple Intelligence also has access to your personal information, calendar, Mail and other data stored on your various Apple devices.
Similar to Microsoft Copilot in Windows, Apple's interpretation of AI is able to perform tasks within the operation system, as well as generate text or images inside any app on your phone, tablet or laptop.
Apple says it secures data on device and only sends information out to the cloud where the cloud servers are built using Apple Silicon chips.
All of the Apple Intelligence features are system wide, and work across every Apple device running either an M-series chip or an A17 or above on iPhone. So while older mobile devices have to miss out, there are plenty of devices capable of joining in. Even if it is only in beta for the time being.
Rewrite and generating text
One of the more notable features of Apple Intelligence is the ability to have it create or rewrite your text to make it more engaging or legible.
Rewrite and Writing Tools can be used to adjust the tone of an email or speech to suit a different audience, add humor to a party invite or clean up class notes.
These are system wide features so will be accessible in all Apple and third-party apps. For example you can use it in Mail, Messages or Notes as well as Keynote, Powerpoint or Word.
The other feature is Proofread that can check grammar, word choice and sentence structure while also suggesting edits and explaining those edits.
Finally Summarize lets you select text and have that text recapped in the form of an easily digestible paragraph, bullet list or even a table.
Creating images
As well as creating text from a prompt, Apple is adding generative images through diffusion models similar to DALL-E or Midjourney.
This includes creating emojis and gifs within Notes or Messages. "If you wish a friend a happy birthday you could create an image using generative AI," the company said.
Generating images will work across a range of apps including Notes, Pages and Keynotes as, like text generation, will be system wide.
A new app called Image Playground will let users create "fun images in seconds" across three styles — animation, illustration and sketch. It is built into a range of apps such as Messages but will also be its own dedicated app.
All the images are created using a locally running diffusion model, so nothing is sent off the device. You can choose from a range of concepts and themes. Like any AI image tool you type a description to define the image, choice someone from your photo library to include in the image and pick the favorite style.
In Messages you can create personalized concepts related to the conversation your having, such as showing a sketch of a friend on a hike if you're talking about an upcoming hiking trip.
This isn't going to replace the photorealism of Midjourney and is more like the quick image creation of MetaAI in Instagram or WhatsApp, but the fact it runs locally is impressive.
Photo editing
The Photos app itself is getting a major upgrade, with a new layout and the ability to create custom categories and galleries. It is also getting better image editing capabilities thanks to AI... sorry Apple Intelligence.
A new Clean Up tool lets users identify and remove any distracting objects from the background of a photo without having to do any more than tapping a button.
Photos is also getting a better Memories feature that lets you create a story by typing a natural language description, Apple Intelligence will then find the relevant images and make the memory, even pulling in a song from Apple Music.
A new Siri
Siri is also getting a makeover and the ability to understand and respond to more natural language. For example, if you stutter or change your mind mid-sentence, the new Siri will still be able to understand what you were saying.
Siri will also be able to make calls out to ChatGPT or other AI models in the future where it needs more compute power and AI intelligence, such as analyzing an image or generating complex responses for stories or recipes.
The new Siri also includes information on products and settings within the Apple ecosystem. This means it could make changes based on a rough description, such as writing a message now and sending it later, without you having to dig through multiple menus to find that option.
Some of these features are live in the iOS 18.1 beta, but some features won't be ready until later this year.
Agent behavior
What is most remarkable is the agent-like behavior. Apple Intelligence can create actions across your system and apps, carrying out tasks on your behalf. For example, you could ask it to pull up files sent by a friend or play a podcast shared by a partner.
Apple says this is grounded in personal information and context, able to retrieve and analyze personal data such as emails, calendar invite or documents.
Apple Intelligence can process personal details, understand documents you've been sent, meeting times in calendar and other events. It can then offer information on how to adapt your schedule based on the situation.
"With Apple Intelligence, Siri will be able to take hundreds of new actions in and across Apple and third-party apps," the company explained. "For example, a user could say, 'Bring up that article about cicadas from my Reading List,” or “Send the photos from the barbecue on Saturday to Malia,' and Siri will take care of it."
Apple Intelligence will also be able to tailor its actions to the user. For example it can be context aware, knowing what messages you've had from friends and whether they recommended content. You could ask Siri to play the podcast Jamie recommended and it will locate that podcast and play it without you having to open the Mail or Messages apps.
Keeping it secure
Apple says Apple Intelligence goes "hand in hand with powerful privacy". The company says its new AI implementation was "built with privacy at the core".
This is because, to be a truly helpful AI assistant it has to have access to personal and private information. This is data most users wouldn't want sent off device.
"A cornerstone of Apple Intelligence is on-device processing, and many of the models that power it run entirely on device," the company says.
But not every task can be carried out on device due to its complexity, requiring AI tools like ChatGPT rather than a small local model. So they launched Private Cloud Compute, which Apple claims "extends the privacy and security of Apple devices into the cloud to unlock even more intelligence."
They say this is the result of new Apple Silicon server chips that secure data in a similar way to the chips on iPhones and Macs.
The company says it only sends data relevant to task to Apple Silicon servers, not every piece of data. It cryptographically limits the ability of data to go to a server that isn't protected and open to privacy inspection.
ChatGPT
Apple is integrating OpenAI's flagship GPT-4o into its ecosystem, although not to the extent many hoped. It won't replace Siri as the main assistant, rather offer up a way to provide additional resources, reasoning and information.
Users will be able to send an image, document or text prompt off to ChatGPT without having to leave the tool they are currently using. You can just open Apple Intelligence and "send to ChatGPT".
this same functionality is coming to Siri as well, Siri will be able to tap into the power of GPT-4o to answer questions or provide resources it can't do using its own local models. For example you could upload a photo of your fridge and ask for a recipe, it would send the image to ChatGPT to identify the contents and create the recipe.
You wouldn't know it had left Siri as you won't need to switch apps, the full functionality is integrated into the Siri UI.
"Privacy protections are built in for users who access ChatGPT — their IP addresses are obscured, and OpenAI won’t store requests. ChatGPT’s data-use policies apply for users who choose to connect their account," Apple said.
Availability
Apple Intelligence will be available for free in iOS 18, iPadOS 18 and macOS Sequoia but won't be available until the Fall and only in U.S. English. It's unclear when the software will roll out to the public versions of the software, and recent rumors suggest iPhone users will have to wait until the release of iOS 18.1.
However a developer-only beta has been released for iPhones, iPads and Macs so long as they have the right hardware. It will require a Mac or iPad running an Apple Silicon chip, so any M-series Mac or iPad and no Intel Macs. Apple Intelligence will only be available on iPhone 15 Pro and iPhone 15 Pro Max or late as it requires a minimum of an A17 chip. You also need an Apple developer account to be able to download this version of the beta.
