Learning Generative AI #6: Apple Intelligence

Apple recently introduced the first features of its new AI platform, Apple Intelligence. In this issue, I'll explore Apple Intelligence and illustrate some of the more important features.

This is the eighth in my series on generative AI and the sixth on Learning Generative AI. If you are interested in the others saved on Substack, here are the links.

Learning Generative AI #5: An Invaluable Investing Tool

Learning Generative AI #4: Planning Your Travel

Learning Generative AI: #3 Getting Useful Medical Information

Learning Generative AI: #2 Satisfying Your Curiosity

Learning to Use Generative AI: #1 Writing and Editing

Explanation and Access to Leading Generative AI Systems

A Primer on Generative AI

If you are interested in my other articles on autonomous vehicles and other emerging technologies, click this link to access them.

Apple Intelligence is a suite of advanced artificial intelligence (AI) and machine learning (ML) technologies integrated into Apple’s ecosystem to enhance user experiences across devices. Central to this intelligence are features like Siri, Visual Lookup, and Live Text, which allow users to interact with their devices naturally and intuitively. Leveraging on-device processing and neural networks, Apple delivers these capabilities while prioritizing user privacy. By performing most tasks locally on hardware like the A-series and M-series chips, Apple ensures fast, reliable performance without compromising sensitive data.

A standout application of Apple Intelligence is its use in computer vision. Features like Visual Lookup and Live Text empower users to recognize objects, identify landmarks, and interact with text directly from images or the camera. These capabilities are designed to be accessible, enhancing productivity, learning, and creativity for users of all skill levels.

Apple’s AI also powers personalized experiences, from tailoring music recommendations in Apple Music to predictive text suggestions on the keyboard. With its commitment to accessibility, Apple Intelligence supports voice commands, real-time translation, and assistive tools like VoiceOver. This ecosystem reflects Apple’s vision of AI as a tool to simplify and enrich daily life, blending cutting-edge technology with thoughtful design.

How to Initiate Apple Intelligence

First, you will need an iPhone 16 or 15 Pro and iOS 18.2. To get Apple Intelligence working on your iPhone, go to Settings, select Apple Intelligence and Siri, and you will see the following screen. Then, turn on Apple Intelligence and any related settings you want.

This will enable various Apple Intelligence features on your devices, as long as you have an iPhone 15 Pro or newer and are running iOS 18.2.

Siri With ChatGPT

The first time you use Siri to access ChatGPT, you must authorize it with a permissions screen that will come up—ChatGPT functions as an extension for Siri. You can see it by going to Settings, Apple Intelligence, Siri, and ChatGPT.

The screen on an iPad looks like this:

Access to ChatGPT from Siri is free, but as you can see, there are daily limits on how much you can use ChatGPT.

When you make a request for Siri that can use ChatGPT for a response, it will ask you if you want to use ChatGPT. Here is an example response:

Intelligent Object Recognition with Visual Lookup.

With Visual Lookup, you can identify objects in photos, such as plants, animals, landmarks, etc. Apple’s intelligent object recognition uses advanced machine learning and computer vision technologies. Here's how it works in simple terms:

• When the camera captures an image, it sends the visual data (pixels) to the device's processor for analysis.
• The image is processed to identify edges, shapes, and textures.
• Lighting, colors, and contrast are adjusted to enhance recognition.
• Apple uses deep learning models trained on massive datasets to recognize patterns in the image.
• These models can detect objects, landmarks, and more based on their learned features, animals, and text.
• The system first identifies regions of interest (parts of the image likely to contain objects).
• It then classifies the object within those regions (e.g., “dog,” “tree,” or “chair”).
• For specific objects like landmarks or known items, the system matches the

How to Use It

Here are the steps to using this Apple intelligence feature on your iPhone:

1. Open a Photo: Open the Photos app and select an image.
2. Look for the Information Button: If the object in the photo is recognized, a small star (*) will appear on the "i" (info) button at the bottom of the screen.
3. Tap the Info Button: Tap the "i" button to open the detailed information panel.
4. Access Visual Lookup: Scroll to see suggestions related to the object. For example:

— For a plant: Learn its name and species. — For a landmark: Get details and history. — For an animal: See its classification.

1. Explore Further: Tap the lookup link or information card to learn more.

Examples

In this example, you see a photo of a dog, and you want to identify what breed it is. You see the icon (circled in red) at the bottom of the first photo with the small *, which means the system can identify the object in the picture. You tap the “I” button and get the second photo that indicates it’s a dog, and you can look it up (circled in red), so you select that and get the third screen identifying it as a Labradoodle.

Here is another example identifying a bird on a telephone pole as a Red-Shouldered Hawk.

Apple Intelligence in eMail

Apple's iOS 18 introduces several enhancements to the Mail app, focusing on improved organization, efficiency, and user experience. Key mail features include:

1. On-Device Categorization

Mail now automatically sorts incoming emails into distinct categories, streamlining inbox management:

• Primary: Essential emails from known contacts and time-sensitive messages.
• Transactions: Receipts, order confirmations, and financial statements.
• Promotions: Marketing materials, sales offers, and newsletters.
• Updates: General notifications and other communications.

This categorization helps users prioritize important emails and declutter their inboxes.

2. Priority Messages

Time-sensitive emails, such as same-day invitations or boarding passes, are highlighted at the top of the inbox, ensuring users can quickly access urgent information without searching through their emails.

3. Summarize Emails

Apple intelligence can summarize your emails for you. This can be particularly helpful in something like newsletter emails to help you decide whether you should read it. In the following example, I highlighted the email I received to show where the summarize button is located. The second example illustrates the summarization by Apple Intelligence.

4. Thread and Preview Summaries

Apple Intelligence generates concise summaries for email threads, allowing users to grasp the main points without opening each message. These summaries appear in the email preview pane, replacing the first few lines of the email, and at the top of email threads, facilitating efficient email management.

5. Smart Reply

The Mail app offers contextually relevant, AI-generated quick reply suggestions, enabling users to respond promptly with options like "Approve," "I agree," or "Let me check." This feature streamlines communication by reducing the time needed to craft responses.

6. Email Writing Tools

Integrated AI-powered writing tools assist users in composing emails by offering features such as rewriting text for clarity, adjusting tone, and proofreading. These tools help enhance users' writing to make it more friendly, concise, or professional, similar to Grammarly's AI writing features.

I drafted the email on the left and activated the highlighted Apple Intelligence symbol in this email example. This gives you the next screen, which provides alternative writing tools to rewrite the email.

The following two pictures show the rewritten email in the “friendly” and “professional” styles.

The two rewrites have improved the email's initial version. You can choose which version to use.

7. Improved Search Functionality

The search feature in Mail has been enhanced with AI capabilities, providing more accurate and contextually relevant results. Users can find specific emails more efficiently, improving overall productivity.

8. Privacy enhancements

Building on previous privacy features, iOS 18 protects user data by preventing senders from knowing when an email is opened and hiding IP addresses, thereby enhancing user privacy and security.

These updates in iOS 18 aim to make email management more intuitive and efficient, leveraging AI to enhance user experience and productivity.

Apple Intelligence Writing Tools

Apple Intelligence's Writing Tools offer a suite of features to enhance writing across Apple devices. These tools are available in iOS 18.1, iPadOS 18.1, and macOS 15.1 or later on compatible devices. Here's an overview of the main features with examples:

Proofreading

This tool identifies and corrects grammatical and spelling errors. Example:

• Highlighting a misspelled word and suggesting the correct spelling
• Identifying a misused phrase and offering a more appropriate alternative

Rewriting

Allows users to rephrase text and adjust its tone.Examples:

• Transforming a casual email into a more professional one
• Making a formal document more friendly and approachable

Summarizing

Creates concise versions of longer texts in various formats.Examples:

• Condensing a lengthy article into key bullet points
• Creating a brief paragraph summary of a long email thread

Tone Adjustment

Users can modify the tone of their writing to suit different contexts. Examples:

• Changing text to a "friendly" tone for informal communication
• Adjusting to a "professional" tone for business correspondence
• Using "concise" mode to shorten verbose paragraphs

Accessing Writing Tools

To use Apple Writing Tools in Apple Intelligence:

1. Select the text you want to modify.
2. On iPhone/iPad: Tap the selection and choose "Writing Tools" from the menu. You may need to select the “….”.
3. On Mac: Click the blue icon next to the text, right-click, and select "Writing Tools," or use Edit > Writing Tools from the menu bar.

Here is an example. On the left is a rough draft with spelling and grammar errors. It highlights the selection of “Writing Tools.” On the right is the rewritten note with a corrected and more professional tone.

These tools are integrated across Apple's ecosystem, including Mail, Messages, Notes, and Safari, enhancing the writing experience in various contexts. There are some problems with using it with MS Word for Mac because of conflict issues with MS Co-Pilot.

Compose with ChatGPT

You can also have ChatGPT compose or write anything for you, so you don’t need to do a draft.

To use Compose with ChatGPT in Apple iOS 18.2:

1. Select text in any app that supports Writing Tools.
2. Tap "Writing Tools" from the pop-up menu. You may need to tap ">" to see this option.
3. In the Writing Tools menu, swipe up and select "Compose."
4. Tap "Compose with ChatGPT"
5. Type your prompt or request in the field provided
6. Tap the Send button to generate the text
7. Review the generated content. You'll see options like "Refine with ChatGPT" and "Rewrite" below the text.
8. Use the "Suggested by ChatGPT" section to add details or modify aspects of the text.

This feature is available in various apps, including Notes, Mail, and Messages. Based on text prompts, you can generate original text, edit existing content, or create images.

Example

As an example, I used it to compose a Happy New Year email, and this is what it wrote:

Subject: Cheers to a New Year! 🎉

Dear [Recipient's Name],

I hope this message finds you well! As we stand on the brink of another year, I am filled with gratitude for the moments we've shared and the memories we've created. Thank you for being a part of my life.

Here's to a bright New Year and a fond farewell to the old; here's to the things that are yet to come and to the memories that we hold. May this year bring new happiness, new goals, new achievements, and a lot of new inspirations in your life. Wishing you a year fully loaded with happiness.

Happy New Year!

Warm regards,

[Your Name]'

That's not bad at all! I wish I could write that well, but I guess I can, and we all can now.

Managing Photos

Apple Intelligence introduces several powerful features for managing photo albums in iOS 18, enhancing the user experience in the Photos app. Here are the key features and examples:

1. Natural Language Search

This feature allows users to search for photos using descriptive phrases. Examples:

• "Maya skateboarding in a tie-dye shirt"
• "Sadie cartwheeling on the grass"

Users can also search for specific moments within videos. For example:

• Searching for a particular scene in a long video clip

2. Memory Movies

Users can create personalized video compilations based on text descriptions. Apple Intelligence in iOS 18 creates memory movies with a narrative arc using photos and videos from your library. Here's how it works:

1. In the Photos app, look at the tabs on the left side (if they don't appear, click on the tabs icon at the top left).

• Then select the icon labeled Memories.
• Look at the top right corner for a button named Create.
• When you select that, you will get a page asking you to create a Memory Movie.
• Type in a description of the memory movie you want it to create where you see the Apple Intelligence Icon and room for text.

1. Apple Intelligence then:

• Selects the best photos and videos based on the description
• Crafts a storyline with unique chapters
• Arranges the content into a movie with a narrative arc
• Sets it to music

The narrative arc typically follows a classic structure with a beginning, middle, and end, which helps maintain viewer interest and effectively deliver the message.

Apple Intelligence likely uses metadata, facial recognition, and scene analysis to organize photos and videos chronologically and thematically. It may identify key moments, locations, and people to create logical and emotional chapters. The system aims to evoke emotions and create a compelling story by selecting images and clips that best represent the narrative arc, much like a skilled editor would craft a memorable video story.

Here is an example:

• I used it to create a wonderful Memory Movie of Christmas over the years. Since I had so many Christmas photos, I asked it to create similar ones, and it used different photos.

Other Apple Intelligence Features in Photos

1. Trips Organization — Automatically organizes travel photos into collections. Example: Viewing photos from a recent vacation grouped by location and date
2. People and Pets — AI-organized albums of individuals and animals.Example:
3. Featured Photos — AI-curated showcase of best shots, updated regularly. Example: Highlighting a well-composed landscape photo in the Featured Photos section.
4. Clean Up Tool — This feature lets users remove unwanted objects or people from photos. Example: Tapping to remove a photobomber from the background of an image

These Apple Intelligence features in iOS 18 significantly enhance photo management, making it easier for users to organize, find, and create content from their photo libraries.

Image Playground

Image Playground is an AI-powered image generation tool introduced by Apple as part of iOS 18.2 and macOS Sequoia 15.2. It allows users to create custom illustrations and cartoon-like images based on text descriptions or existing photos. Here's how it works and some examples of its use:

How Image Playground Works

Image Playground is a new Apple app that will automatically appear as an icon.

1. Text-based generation: Users can type descriptions or prompts to generate images
2. Photo-based creation: Images can be generated based on existing photos from the user's library.
3. On-device processing: All image creation is done on the device, ensuring privacy and allowing unlimited experimentation.
4. Multiple styles: Users can choose between "Animation" (3D-like images) and "Illustration" (flat, cartoon-style) modes.

Iterative refinement: Users can modify results by adding or removing descriptors and swiping through multiple generated options.

Examples of Image Playground Use

There are many ways, you can use Image playground, here is a list of different ideas and examples.

1. Creating personalized avatars: Generate unique profile pictures for social media based on photos of yourself or friends.
2. Designing custom emojis: Create playful, personalized emojis for messaging apps.
3. Illustrating creative concepts: Generate images of imaginative scenarios like "a crocodile wearing a tutu" or "a skunk hula hooping on top of an elephant.”
4. Enhancing note-taking: Use Image Wand in the Notes app to transform rough sketches into complete AI-generated images.
5. Visualizing ideas: Create illustrations for brainstorming sessions or presentations.
6. Personalizing messages: Generate unique images to accompany text in messaging apps.
7. Creating themed collections: Develop a series of images around a specific theme, such as "space scenes" or "futuristic cityscapes.”
8. Transforming pet photos: Generate cartoon-style images of pets for fun or personalized pet-related content.
9. Illustrating stories: Create visual elements for creative writing or storytelling projects.
10. Designing custom stickers: Generate unique stickers for various messaging and social media platforms.

My Example

Well, here is an image I created from a photo of myself and the descriptions to make it look like Christmas with a Santa hat. It is somewhat close, but it is fun, which is the purpose of this app.

Summary

Apple Intelligence is only getting started, but already has many valuable features, including the following released through iOS 18.2:

1. ChatGPT integration with Siri and Writing Tools
2. Genmoji: Create custom emojis using text prompts
3. Image Playground: Generate or modify images using AI
4. Visual Intelligence (iPhone 16 series only): Analyze objects and text in photos
5. Enhanced Writing Tools with custom rewrites
6. Clean Up tool in Photos: Remove unwanted objects from images
7. Memory Movies in Photos: Create personalized video compilations
8. Natural Language Search in Photos
9. Notification summaries
10. Reduce Interruptions Focus mode
11. Intelligent Breakthrough & Silencing in Focus
12. Priority messages in Mail
13. Smart Reply in Mail and Messages
14. Summaries in Mail and Messages
15. Siri enhancements:
16. Image Wand in Apple Pencil tool palette
17. Compose with ChatGPT: Draft original text in any app

These features are available on compatible devices running iOS 18.2, iPadOS 18.2, or macOS Sequoia 15.2.

Share How Technology is Transforming Our Lives