By Joe Rosensteel

June 19, 2024 1:38 PM PT

How Sandwich streamed The Talk Show Live in 3D on Vision Pro

During last week’s WWDC festivities, John Gruber interviewed Apple executives on stage for The Talk Show Live, as he’s done for years. This time it was different because people at home with a Vision Pro could watch the event live from the Theater app by Sandwich Vision, streamed by SpatialGen. (The stream is still available to watch after the fact in the Theater app.)

Sandwich is Adam Lisagor’s media empire specializing in commercial production, and Sandwich Vision is the Vision Pro development arm. I had the chance to talk to Adam, Andy Roth, and Dan Sturm. Andy is the developer for Sandwich Vision’s Television and Theater apps. Dan is the visual effects supervisor for Sandwich.

Disclosure: I am friends with Dan, and have worked for Sandwich as a freelance compositor on some projects, but I am not connected to Television, Theater, or The Talk Show Live in any capacity. The following was lightly edited for clarity and length.

For those that haven’t watched it in a Vision Pro, how would you describe the experience of viewing The Talk Show Live in the Theater app?

Adam: The experience is entirely unique. It’s a blend of different immersive styles and definitions that combine to create a unique kind of immersion that’s more than the sum of its parts.

• The user is immersed in an immersive space within visionOS (the theater inside the app, surrounded by theater seating, with a sense of scale and perspective—and the equivalent of a 76′ screen in front of them, so it’s the feeling of a typical huge AMC style multiplex theater with few enough cues from lighting, shape, and texture to break the illusion.

• The user sees a human-scaled “portal” to the stereoscopic capture of humans on a stage, separated forward from the big screen in z-depth about the same distance as the actual humans would be in a real theater environment. So the human scale is immersive, and the stereo capture is immersive.

• The user hears spatialized audio of the humans on stage combined with the audience captured in a stereo image, creating a sense of immersion in sound within the environment (as well as a sense of place in the community). This is a real psychological effect that happens when a person sits within a large group that’s having a communally similar reaction—we get a sense of overwhelm from the uncommonly emergent scale of the group of which we’re now a member.

• The user experiences the event in real time, which, in combination with the other immersion styles, is almost never experienced—we watch broadcast TV of live events all the time, but we never experience live events in real time with multiple styles of immersion.

All of this combined leads to a sense of nowness and thereness that is, as some social media users described, “magical.”

Continue reading “How Sandwich streamed The Talk Show Live in 3D on Vision Pro”…

By Jason Snell

June 13, 2024 4:29 PM PT

Last updated October 24, 2024

OK, fine, here’s Apple Intelligence

It was clear that Apple didn’t want to be a part of this.

For years now, the company has resisted using the phrase Artificial Intelligence to describe what it does. It’s facile and misleading and not what Apple is really about at all. So it’s been working on advanced computing techniques and the hardware to drive them, all using phrases like machine learning and Neural Engine.

Then, a couple of years ago, the AI buzzword exploded into the consciousness of… if not the general public, then the tech industry and press. The image creation and chatbot demonstrations were, yes, facile and misleading, but also quite a party trick. The narrative shifted: Was this the future of computing?

And there was Apple, churning away on its ML photo tagging in Photos and eye tracking in visionOS and image segmentation for silly video effects, suddenly caught flatfooted. The area it had looked on with some disdain—and not without reason—was suddenly the next big thing in tech.

And, of course, the scary part was: What if it was the next big thing? What if—after investing billions in visionOS and a car project that never even saw the light of day after a decade, all as a hedge against anything that might eclipse the iPhone—the technology that it had pooh-pooed was the real existential threat?

According to multiple reports, there was a moment in late 2022 or early 2023 when Apple realized that it needed to aggressively embrace AI or risk being seen as old and out of touch—not just by its industry but by the general public. The result was a prioritization of AI features all around the company, and the first fruits of that prioritization were on display at Monday’s WWDC keynote.

Three kinds of AI

It seems to me that on Monday, we saw three different kinds of AI features. Apple won’t ever say which was which, but the rest of us can guess at it. The three categories are:

• Features Apple would’ve done anyway. The company has been rolling machine-learning stuff into its software for ages now. Would smarter photo tagging and better Inbox organization in Mail have been announced regardless of Apple’s new, big AI focus? Almost certainly. Apple has always been about leveraging advanced technology in the service of customer needs.

• Features Apple would’ve done… eventually. It sure seems like Apple’s shift of priorities led to some features being announced that seem very much like the sort of thing Apple would’ve done, but at a slower pace and more conservatively. There seem to be more WWDC announcements than usual that ask for our patience because they won’t be ready until later this year or even sometime next year. But if it was an AI-related feature and was on the drawing board, it probably got pulled forward for introduction at WWDC 2024. Ready or not, here they come.

• Features Apple would never have done. There are some areas where Apple shows the strain of fearing they’ll be seen as burying their heads in the sand and resisting the trends. ChatGPT support and Image Playgrounds both feel like areas where Apple is stepping far out of its comfort zone in order to make it seem more tuned in to the zeitgeist.

We’ll see how these different features end up faring over the course of time.

The difficult choice

Last week, I wrote that Apple should see itself as the grown-up in the tech industry’s rush toward AI at any cost. For all of AI’s potential, it can be unreliable, the training data sources can be questionable, and it all just feels overhyped. (To me, AI manages to be simultaneously overhyped and potentially world-changing. Great potential… lots of snake oil. All at once.)

Apple’s WWDC announcements show that the company wrestles with this idea too. In its most thoughtful moments, it has expressed the desire to approach AI features with clear, important guidelines. The tools are meant to be intuitive, personalized, deeply integrated into Apple’s product experiences—they’re features, not tech demos—and build with privacy in mind.

In most cases I think Apple’s announcements showed those principles in action. This is a company that realized it was not going to be able to perform all the AI actions it wants to from a user’s device, so it built its own server hardware and operating system and is hosting them in its own data centers, all so that it can ensure that private user data isn’t ever misused. Nobody else does that, but Apple knew it had to.

Most of what Apple announced is really about solving problems for users. Rather than creating a model that has eaten the entire Internet but has a troubling tendency to hallucinate, Apple has focused on serving a user’s personal contexts. It seems to have avoided buying into the broader tech industry fallacy that the future of computers is asking questions to AI chatbots that respond with clichés and dissembling and hallucinatory answers.

Well. Avoided for the most part.

Apple has always been good at building features that play to its strengths, both technical and market wise. It owns your phone, it’s got your data in its ecosystem, it’s invested in building neural processing into its chips, and now it can put that all together to build a whole new Siri that knows your personal context and can put it into action.

But at the same time… all the cool kids are making weird AI art in Midjourney and Stable Diffusion, I guess? And even if Apple doesn’t have a chatbot that’s been trained on “world knowledge” (the official buzzphrase of WWDC 2024), I suppose you’re not in with the in crowd unless you’ve got a chatbot?

This is where Apple has compromised, a little, at a distance: Its Image Playgrounds feature makes weird AI-looking art, built around a nicer interface than any fill-in-the-text prompt, but still kind of weird and off-putting. And Apple has wired in ChatGPT, because if Siri can’t help you, maybe a chatbot can.

Warning labels and defaults

Which brings me to Apple’s ChatGPT integration, which seems utterly necessary from the standpoint of making the iPhone not feel hopelessly behind Google, which is rolling its Gemini chatbot into Android. People who despise AI hype in general and chatbots in particular will probably be disgusted by Apple’s choice, but if you look closer, Apple’s clearly not comfortable with any aspect of the partnership.

Think about it. Apple made a much-ballyhooed introduction of its partnership with ChatGPT by showing off that every single time Siri offers to use it; it throws up a (scary?) alert dialog that asks if you’re sure you want to use the chatbot. When the ChatGPT results return, they’re appended with a warning label advising that you check all the information if it’s important!

That was just in the keynote. Before the day was done on Monday, Apple execs had also made it clear that the feature was turned off by default and that ChatGPT was just the first partnership of many. Google Gemini was mentioned by name as if it was only a matter of time before ChatGPT’s archrival would be integrated. They likened it to offering users different, interchangeable search engines in Safari. They also suggested repeatedly that the ultimate goal of the “external chatbot” feature wasn’t really general knowledge at all but chatbots trained on specific areas of knowledge, like medicine.

Now Bloomberg’s Mark Gurman reports that Apple’s not even paying OpenAI for ChatGPT. I get why OpenAI would do this deal—its biggest competitor has the home-field advantage on the world’s only other notable mobile operating system—but it really looks like they’re the one over the barrel here, not Apple.

I think that might say it all. The conventional wisdom going into WWDC was that Apple was the company that was flailing and desperate, trying to catch up to the giants of artificial intelligence and retain some level of relevance. What we saw this week, however, was a company that seems surprisingly confident in its own AI prowess, and one that continues to largely follow its own playbook rather than compromising its ideals.

The hottest company in tech, OpenAI, just gave its crown jewels to Apple for free, and Apple responded by introducing its integration as optional and tagged with numerous warning labels. So, who’s behind and flailing, exactly?

The truth is that Apple right now is like a duck: serene on the surface but paddling furiously underneath. It was clearly complacent about the pace of AI innovation and allowed itself to get a bit too comfortable, and now it’s hurrying to keep up. It will undoubtedly make a few mistakes along the way. We also don’t know how good Apple’s stuff is—note that most of the Siri section in the WWDC keynote was aggressively in the future tense—and have to take the company’s word for how private its cloud services really are.

There’s a lot to interrogate here, and if Apple can’t fulfill its promises, it will be in real trouble. That’s a story that will be written over the next year, but right now, I wouldn’t bet against them.

By Dan Moren

June 13, 2024 11:06 AM PT

Excluding your website from Apple’s AI crawler

One of the more contentious announcements from Apple this week is that it trained its foundation models used as the basis for its forthcoming Apple Intelligence features via, among other content, the open web.

Obviously that raised a lot of eyebrows among those of us that publish content to the web. Using copyrighted material to train AI falls under fair use is a question still being hotly debated, and one that may ultimately be highly dependent on the exact circumstances. It’s also a behavior that people feel justifiably uncomfortable with.

Setting aside feelings on that issue for just a moment, it’s worth looking at the mechanics behind this. Apple also said during its announcement that it’s providing a way for publishers to exclude their sites from being used for training its AI models, via a long established system built originally for search engines: robots.txt.

Robots.txt or not

If you’re not familiar with robots.txt, it’s a text file placed at the root of a web server that can give instructions about how automated web crawlers are allowed to interact with your site. This system enables publishers to not only entirely block their sites from crawlers, but also specify just parts of the sites to allow or disallow.¹

Apple’s own Applebot web crawler has existed for some time; it’s used to power search features in Spotlight, Siri, and Safari. For example, whenever you see a Top Hit in Safari or Spotlight, that information is coming from a search index created using Applebot. Podcasts are also fed by Applebot, though in those cases only via specific URLs registered with Apple Podcasts.

If you’re worried that blocking Applebot from crawling your site might impact your site showing up in traditional search results, good news: the new AI training element of Applebot uses a separate identifier, allowing you to block only that functionality without affecting your site’s appearance in Apple’s search features.

How to stop Apple from using your content in AI training

Apple provides a detailed support document about Applebot and the various directives to control how it interacts with your site.

To specifically exclude your whole site from being used for Apple’s AI training features, you can add the following to your robots.txt file:

User-agent: Applebot-Extended
Disallow: /

To test this out, I’ve added those directives to my personal site. This turned out to be slightly more confusing, given that my site runs on WordPress², which automatically generates a robots.txt file. Instead, you have to add the following snippet of code to your functions.php file by going to the administration interface and choosing Appearance > Theme File Editor and selecting functions.php from the sidebar. (You can also do this via a plugin like Code Snippets, which I use.)

add_filter('robots_txt', 'my_robots_commands', 99, 2); // filter to add robots

function my_robots_commands($output, $public) {
  $output .= "User-agent: Applebot-Extended\nDisallow: /";
  return $output;
}

If you want to go beyond Apple, this same general idea works for other AI crawling tools as well. For example, to block ChatGPT from crawling your site you would add a similarly formatted addition to the robots.txt file, but swapping in “GPTBot” instead of “Applebot-Extended.”³

Google’s situation is more complex: while the company does have a Googlebot-Extended that powers some of its AI tools, like Gemini (née Bard), blocking that won’t necessarily remove your site’s content from being crawled for use in Google’s AI search features. To do that, you’d need to block Googlebot entirely, which would have the unfortunate effect of removing your site from its search indexes as well.

Fait accompli

One challenge with these AI tools, however, is whether or not the damage has already been done. Many of these AI models have, of course, already been trained, and it’s not as though you can remove training data from them by blocking these crawlers now—it’s very much closing the barn doors after the horses have gotten out.

However, it does mean that you can prevent your site and content from being used for training going forward, so any material you publish from that point on will be excluded. If you feel like that’s cold comfort, you’re not alone.

Update: An earlier version of this article misstated the user agent for ChatGPT’s crawler: it’s GPTBot, not ChatGPT-User.

[Dan Moren is the East Coast Bureau Chief of Six Colors, as well as an author, podcaster, and two-time Jeopardy! champion. You can find him on Mastodon at @dmoren@zeppelin.flights or reach him by email at dan@sixcolors.com. His next novel, the sci-fi adventure Eternity's Tomb, will be released in November 2026.]