It’s hard to trust Google’s new AI-powered search vision when it can’t even correctly summarize the details of its own products.
At its Google I/O developer conference on Wednesday, Google laid out the foundation for revamping its iconic search engine, which has followed the same basic formula for years. The company aims to power Google search up with conversational and generative AI — and, most importantly, put those responses before traditional links and everything else. For the most popular search engine in the world, this is a monumental change.
A Google example of generative AI results atop traditional search links.
The problem with that approach is that AI chatbots often make confident-sounding mistakes — and, true to form, Google couldn’t even get the details of its latest Google Pixel Fold and Pixel Tablet products exactly right, including the price. We used Google Bard for our tests, not Google Search, but with the same AI eventually underlying both, we think the results are worth highlighting.
Google makes a bold choice when laying out its latest search vision: It puts its AI results right on top. Microsoft and its AI-powered Bing search engine also includes a conversational AI-powered chat interface, but places its AI in a small box to the right of traditional links, at about the height (and priority) of the top search result. In certain cases, such as asking both search engines for the specifications of the Google Pixel Fold, Bing includes an array of specifications from a third-party Web site. In Google’s mockups, it doesn’t.
Further reading: AI chatbots aren’t search engines, they’re crypto bros
Bing’s right; Bard isn’t
On the advice of my editor, I asked both Google Bard and Bing Chat for the specifications of both the Google Pixel Fold and Pixel Tablet. Bard was much more verbose — but, unfortunately, also more wrong.
Both search engines, to be fair, returned mostly accurate results. We even gave Bard the benefit of the doubt, asking it for the specifications of, first, the “Google Fold” and then the “Google Pixel Fold.” But Bard still returned incorrect information about the exterior display, reporting first that its outer 5.8-inch display had a 1080×2670 display and an inner 7.6-inch display of 1920×1080. It doesn’t; the exterior display has a resolution of 2092×1080 and an inner display of 2208×1840. (Both run at up to 120Hz, another fact Bard left out.) When we asked it a second time, it didn’t even distinguish between the inner and outer display, simply stating that it has a 7.6-inch foldable display with a resolution of 1920×1080.
(You can see the results of our query below. You may need to open the images in a separate tab. Google lists the specifications of the Pixel Fold on its shopping page.)
Mark Hachman / IDG
Bard also waffled on the camera specs, first reporting that the Pixel Fold’s front camera was 8 megapixels, and its rear camera was 48MP (main) / 12MP (ultrawide) / and 10MP (telephoto). It then decided that the front selfie camera was 12MP, and the rear was 50MP/12MP/10MP. (The correct specifications are a 8MP selfie camera and a 48MP/10.8MP/10.8MP rear camera.)
Bard was at least consistent on the price, the battery size, the dimensions and the wireless capabilities, but couldn’t decide whether the Fold runs Android 13 or the upcoming Android 14. (Google doesn’t say.)
Google also goofed when asked about the new Google Pixel Tablet. Its most significant error? The price, which Bard listed as simply $599. In fact, according to Google, the base configuration is $499, a full $100 less. (There is a $599 option, which increases the base 128GB of storage to 256GB, but more notably, a swathe of early Pixel Tablet rumors pegged it as that price.) Bard also continued to have problems with the camera specs, incorrectly characterizing the front and rear cameras as 8MP and a 13MP (main) and 5MP (ultrawide) rear camera. The Pixel Tablet includes 8MP front and rear cameras, with no options for an ultrawide.
Again, we’ve reproduced our queries below. You can compare them to the official Google Pixel Tablet specifications.
Mark Hachman / IDG
Bing, meanwhile, skated by reporting that the Pixel Fold has a “1080p” exterior screen. Otherwise, though, it was entirely factually correct on both Google devices, when Google itself wasn’t.
Google is moving to an updated PaLM 2 large language model, with tighter integrations to other Google services. Whether that gives Bard and the underlying Bard/Search AI better accuracy is unknown. And we’d be remiss in not noting the following caveat:
“There are known limitations with generative AI and LLMs, and Search, even today, will not always get it right,” Elizabeth Reid, the Google general manager for search, wrote in a blog post. “We’re taking a responsible and deliberate approach to bringing new generative AI capabilities to Search. We’ve trained these models to uphold Search’s high bar for quality, and we will continue to make improvements over time.”
Still, the basic premise holds true: If Google wants to lead its search results with AI-generated responses, it had better be certain that those AI responses are absolutely correct. Right now, Google’s AI answers aren’t.
