Best new features coming to Apple Vision Pro with visionOS 2

Photos on Vision Pro are getting a big update. Spatial photos and videos — media that has depth to it — have been a big marketing point for Apple. Now, using “advanced machine learning”, Apple has made it so that normal images can be transformed into spatial images in the Photos app. Whether videos can get this as well was not mentioned. My guess would be no, because that would require a lot more processing power and time.

To go along with this, Apple is bringing SharePlay to the Photos app as well. You can share media, as well as spatial media, with friends and family. Other people can also appear in your view as Personas: Apple’s digital recreations of people in MR.

New gestures

Apple introduced two new and improved gestures for getting around visionOS. Turning your hand with the palm facing up and then making a pinching gesture will open the home screen. Doing so with the palm facing down will open up the time and allow you to access notifications.

Mac Virtual Display — the option to use your Vision Pro as an external monitor for your Mac — is getting improvements too. With visionOS 2 you will be able to enlarge your external display to insane sizes, going even so far as to curve the display around your head.

To facilitate the processing requirements of such a feat, foveated rendering will be carried out on your Mac. Your Vision Pro will use its eye tracking to let the Mac know what to do. Foveated rendering means areas outside of your direct line of sight will be rendered at a lower resolution. This will help the headset run a gigantic external monitor without sacrificing performance.

Travel Mode for trains

Travel Mode lets AR headsets like the Vision Pro work inside moving vehicles without the tracking going haywire. However, Travel Mode on Vision Pro only supported air travel. With visionOS 2 this compatibility extends to trains as well, which should be useful for large metropolitan areas with extensive public transit.

Third-party mouse support

Up till now the Vision Pro has only supported the Apple Magic Trackpad. Other computer mice, and even Apple’s own Magic Mouse, were not compatible. VisionOS 2 finally brings the ability to connect and use whichever mouse you want to.

This is especially useful for people who use the Vision Pro for work because the Magic Trackpad is just a trackpad. And if mouse support were limited to the Magic Mouse, it would’ve made for a very unergonomic experience.

AirPlay casting to Vision Pro

With visionOS 2 you can now cast content from your other Apple devices to your Vision Pro headset. Apple AirPlay will be used for this and allow you to seamlessly send over content to your Vision Pro for a more immersive viewing experience.

Live Captions

Live captions are coming to the Apple Vision Pro. With visionOS 2, your headset will be able to transcribe in real-time the dialogue happening around you. Live captions will also work with audio and video playing on the headset. I think Apple might go the way of Google’s Project Astra and Meta’s Project Nazare with this in the future.

Other new features in visionOS 2

Those were some of the big new features coming to Vision Pro with visionOS 2, but there are lots of other things as well. These are:

• A new dedicated Vimeo app
• Customizable app menu like on iOS and iPadOS
• Cinema Mode will now support web videos
• Videos in Cinema Mode can now be tilted upwards
• Visible external keyboard in immersive backgrounds
• Ability to watch multiple sports games together on Apple TV
• Support for panoramic viewing of panoramic photos on the web
• Quick Look will now let you alter existing models instead of having to load new ones for minor changes
• Apple Communication Safety is coming to Vision Pro
• Mindfulness will detect your respiration and adjust its visualizations accordingly
• Look-to-Dictate (looking at the microphone icon to bring up speech-to-text) will be coming to Messages
• Personas will be getting more accurate skin tones and more vibrant clothing
• Personas in windowed mode will now be able to choose their background image
• Guest User will remember a guest’s calibration for up to a month

In our review of Apple Vision Pro we said it was an absolutely amazing machine. VisionOS 2 is poised to make the headset even better when Apple rolls out the new update in Fall.