Every Google Gemini AI-powered feature for Pixel phones so far

At Google I/O in May, the company debuted its new multimodal AI known as Google Gemini. During that launch, Google talked about what Gemini can do and how it will modify many of the company’s products. However, it wasn’t until early December that we actually got to see Gemini in action, and how it’ll start to modify Google’s Pixel phones.

Now that Gemini is out there in the world on the Pixel 8 series and with more to come, we’ve made a list of all the things it can do on a Pixel phone so far, and a few brief ideas for features we hope to see in the future.

Gemini is Google’s next-generation AI architecture. It has been designed from the ground up to exchange Google’s current AI model known as PaLM2. Today, PaLM2 is what works under the hood for various Google AI features and products, such as Google Bard, Duet AI in Workspace applications, etc. Think of Gemini as being similar to GPT-4, the underlying architecture that powers the popular ChatGPT.

The big difference with Gemini when compared to PaLM2 or event GPT-4 is how it was built. Most multimodal models are built piecemeal — you build one thing to do Task A and then another thing to do Task B and then put them together so the system can achieve Task C. However, Gemini, according to Google, is “natively multimodal, pre-trained from the start on different modalities.” This should, in theory, make it faster, more accurate, and more efficient when it comes to performing its tasks. It should also allow it to perform tasks that other systems cannot.

There are three versions of Gemini, with each designed for specific types of work. Gemini Ultra is the BIG one. This is the multimodal model that will do incredibly complex tasks. Gemini Pro will power Google’s consumer-level products that work in the cloud. Think Google Bard and other PaLM2 products. And finally, there’s Gemini Nano, which is designed to work on a device natively and is what appears on Pixel phones. This is the least powerful version of Gemini, but it could be the most practical to use since it will not necessitate a data connection and can be incorporated directly into consumer-level products. You can read more about the differences between the three Gemini versions in our explainer at the link.

So what can Pixel phones do with Gemini Nano? Let’s find out!

On the Google Pixel 8 Pro, you can use a Gemini Nano-based feature inside the Recorder app. When you open a transcript of a recording, you will see a new Summarize button. Tapping this will use the power of Gemini to produce a quick transcript summary.

For this to work, you must be using a Pixel 8 Pro on the December 6, 2023 update or later. You’ll also need the newest version of the Recorder app.

Because this is a Gemini Nano feature, your phone’s processor handles the summary. That means you don’t need to be connected to the internet for the Summarize button to work.

This is also only supported on the Pixel 8 Pro using the December 2023 patch, at least for now. It also is a feature that’s in developer preview, which means you need to switch it on manually.

Essentially, this allows you to acquire suggestions for replies within your WhatsApp chats. It works by scanning the most recent chats in a thread and then providing you with a suggested reply that would be appropriate for that moment. Instead of manually typing out this reply, you’d just hit the suggestion, and Gemini will do the work for you.

Obviously, this is not unlike the years-old Smart Reply. But since this uses Gemini Nano, it will work faster, better, and use your phone’s processor rather than the cloud.

To turn this on, you’ll need access to Developer Options. To activate this on your Pixel 8 Pro, go to Settings > About phone and scroll down to the very bottom to find your software build number. Tap this number seven times and then enter your PIN. Now, go to Settings > System > Developer options and scroll down until you find AICore Settings. Tap this and make sure the Enable AICore Persistent toggle is on. This will give you access to Gboard’s Smart Reply within WhatsApp as powered by Gemini Nano.

On December 6, 2023, Google formally switched on Gemini Pro inside Google Bard. According to Google, this will make Bard “far more capable at things appreciate understanding and summarizing, reasoning, coding, and planning.” If you transmit with Bard with text in English today, you will use Gemini. Currently, it does not work for anything other than text-based prompts, so uploading images or voice recordings to Bard will still use PaLM2.

This technically isn’t a Pixel feature, but you can access Bard on a Pixel through your browser. You can try it out for yourself at Bard’s web portal.

So far, we haven’t seen much difference between Gemini Pro Bard and PaLM2 Bard. However, it’s early days yet, so we could see more substantial results in the future.

We’ve only seen a few features integrating Gemini, but there’s undoubtedly much more on the way. Below, we have a few ideas as to cool new tricks Pixel phones could do. To be clear, these are not rumored features or things Google has told us might be on the way — these are just ideas for what we hope to see.

Photography and editing features

Google already uses a lot of AI for photography, videography, and media editing on Pixels. These tools help Pixels to be some of the best camera phones you can buy. Gemini could take these features up a notch and produce photography and videography the likes of which we’ve never seen from a phone.

As an example of what this could mean, the Pixel 8 Pro has a new feature called Video Boost. This processes each frame of a video using the same principles behind the processing a Pixel does to a captured photo. To use Video Boost, though, you need to shoot that video on the Pixel 8 Pro, upload it to Google, foresee for Google to process it, and then download it. Maybe, at some point in the future, this process could happen on-device with the power of Gemini. Going advance, in the distant future, this could maybe even happen in real-time as you record the clip.

Likewise, we already have Magic Editor for Pixel phones. Right now, though, it’s pretty simplistic and also needs a data connection. With Gemini, Magic Editor could become the next Photoshop but be so easy to use that anyone could do it.

Given how popular Pixels are for photos and videos, we are certain Google has a lot of big plans for Gemini in this realm.

Complex mapping

Google Maps has a ton of capabilities already when it comes to mapping and directions. However, even on desktop, you can’t get too complex with it. For example, you can’t incorporate variables in your route appreciate weather reports, concerts or other events that could provoke significant spikes in traffic, or your own personal driving habits.

With Gemini, though, these variables could all come into play. You could inquire directions to a location, and Google could pull in all those different factors to give you the ideal route. It could even give you a hyper-accurate window of time for your arrival.

With the upcoming Google Assistant with Bard, you could even make the creation of complex navigation completely natural. For example, you could say, “Hey Google, I need directions to work that will bypass Route 101 and get me there by my usual time.” That’s a command that is way too complex for Maps/Assistant to comprehend now, but Gemini could make it possible.

Quick productivity

One thing that we’re starting to see Bard be able to do is create tables in Sheets, papers in Docs, and emails in Gmail using only voice prompts. With Gemini, though, this could become incredibly powerful.

visualize being able to record a meeting with Recorder and then ask Bard/Assistant to create a table of all the major data points discussed. Or, visualize having Gmail automatically unsubscribe you from any email list that you haven’t opened in the past 90 days. You could even draft a report in Docs and then ask for it to be formatted in a specific style, appreciate Chicago-style or Associated Press Stylebook, which could save you a ton of time.

What Gemini features are you most excited to see on Pixel? What features are you hoping to see in the future? Let us know in the comments!