If Google really wants AI to take over everything, it needs to put Gemini within the tool that all of us have – smartphones. Google is now doing that and it’ll replace Google Assistant on your devices.
By putting Gemini directly into Android, Google can use its AI to deeply integrate into the OS and be more context aware and understand the apps you are using. It’s bringing a new level of power to an on-device assistant experience.
What does that mean for you? For now, Google has teased several features. Here’s a short list of them that happen through an overlay:
- Drag and drop AI images: By using the Gemini overlay, you could ask it to produce an AI-generated image that can be dragged and dropped onto your conversations in Google Messages, Gmail, and more.
- “Ask this video”: With a YouTube video playing, you could say “Ask this video” in Gemini to get specific info from the video.
- “Ask this PDF”: This is a paid feature for Gemini Advanced, but you could ask Gemini to find specific information within a lengthy PDF that it could find and tell you where in the document it exists.
Google also said that Pixel devices later this year will get Gemini Nano with Multimodality. This ramps up the understanding of “context like sights, sounds and spoken language.” This happens on-device too, so that means localized fun, with examples like:
- Scam alerts while on calls: Let’s say you receive a call and it sounds somewhat legit, but you can’t really tell and are worried it might be someone trying to scam you. Google things it can use Gemini Nano on your device to recognize the scam and then warn you in the middle of the call that you should probably hang up. It looks for patterns it thinks could be scams, so if someone asks for PINs or passwords or to make transfers, it could trigger the warning.
- Clearer descriptions with Talkback: Talkback is going to utilize Gemini Nano to help people who experience blindness or low vision by giving them richer and clearer descriptions of what’s happening in an image.
It sounds like Google is going to kick this all off by building Gemini into Android 15 first. We’ll know more soon.