The problem with testing AI chatbot subscriptions like Google’s Gemini Advanced and OpenAI’s ChatGPT Plus is their generality. The same tool is used for disparate applications; the same software service that developers in San Francisco are using to build their latest app might also be used by parents in Kansas to plan a Paw Patrol birthday. Even though companies often tout esoteric benchmarks to prove their chatbot’s superiority, it can be hard to discern how a chatbot’s technical prowess translates into a better experience for you, the user.
Google is the latest company to offer one of its best AI chatbots as a subscription product; in early February, the company began offering access to Gemini Advanced for $20 per month. In doing so, Google was following the precedent set by OpenAI, which sells access to its GPT-4-powered chatbot for $20 a month. Additionally, Microsoft sells subscriptions to its top tool, Copilot Pro (which is also powered by ChatGPT-4) for the same price. But, do you really need to factor another pricey subscription into your budget? After hours of testing these subscription chatbots and prodding at their limitations, my two core takeaways remain the same in 2024 as they were last year when these services first arrived.
First, most people are fine with the free option. If you have a specialized need for the tool, like coding, or want to experiment with powerful AI models and features currently available, then Gemini Advanced or ChatGPT Plus might be worth the $20-a-month cost. For the average chatbot user, who may utilize AI to craft emails at work and Rick and Morty fan fiction at home, the basic versions of ChatGPT and Gemini are free, competent, and wildly more powerful than anything available in the recent past.
My second key takeaway? Don’t immediately trust the output. It’s been said a million times, and I’m here to say it again: chatbots love to lie. For example, in previous tests ChatGPT’s image analysis feature confidently mislabeled my daily multivitamin as a prescription pill for erectile dysfunction, a potentially dangerous mix-up.
Are you still interested in subscribing to an AI chatbot tool, but not sure which one is the best fit for you? Here’s some helpful context about how Gemini Advanced and ChatGPT Plus compare—and what sets each subscription apart.
Gemini Advanced from Google: As a package deal, Gemini Advanced offers the most to users on top of an impressive chatbot. Yes, you receive access to Google’s best AI model, Gemini Ultra 1.0, with the $20 per month AI Premium plan, but you also get everything offered with the company’s Google One subscription included in that price, which includes 2 terabytes of cloud storage. The company is expected to add a Gemini integration for Gmail and Docs as part of the subscription. Google just announced another new Gemini model, Gemini Pro 1.5, that can process more data than the current iteration, but this is not yet available to the public.
ChatGPT Plus from OpenAI: If you’ve experimented with AI chatbots in the past, odds are you’re familiar with using ChatGPT, which makes the transition to ChatGPT Plus with GPT-4 and Dall-E 3 quite simple. While OpenAI’s subscription does not include ancillary perks like cloud storage, it does have one exclusive, innovative feature: the GPT store. Here you can build and share custom versions of ChatGPT that have been optimized for different situations.
Copilot Pro from Microsoft: Similar to ChatGPT Plus, you get unfettered access to GPT-4 and Dall-E 3 when you subscribe to Copilot Pro. Built on top of OpenAI’s technology, Copilot Pro’s core differentiator is its integration with Microsoft’s suite of productivity software. The AI tools can be used directly inside of Excel, Outlook, and PowerPoint, if you’re also an active Microsoft 365 subscriber.
Even though we have experience testing a variety of chatbots at WIRED and putting fresh AI features to the test, keep in mind that these comparisons are designed to give you an overview of how the tools work. My tests are not all-encompassing, bro. (For example, I have too much respect for coders to pretend like I could gauge the worthiness of an AI tool for software development.) Also, since Microsoft’s offering uses the same generative AI models as OpenAI’s service, you can expect similar results from both tools. For this reason, I just compared the results of ChatGPT Plus and Gemini Advanced.
To start off, chatbots are often positioned as a productivity tool for white-collar workers. So, I tried to see how well ChatGPT Plus and Gemini Advanced would be at a basic meeting summary. After uploading a transcript from an interview with a video game developer, I asked the chatbots, “Could you please summarize this meeting transcript into five bullet points?”
Of all the tests, this was the one where the chatbots showed the most similarities and promise. Both chatbots did a great job of catching key moments and distilling them down. Comparing the bullet points, four out of the five highlights generated by the chatbots featured the same parts of the sprawling conversation, and the fifth bullet point was negligibly different.
Another common, in-office application for chatbots is to improve email correspondence. Curious how well these tools are at rephrasing nasty emails into a more professional tone, I composed a scathing message for my editor (who’s an absolute angel and I would never be mean to) and asked both chatbots to make it appropriate for the workplace.
Gemini Advanced succeeded in the rephrasing test. Not only was the chatbot’s rewording appropriate for work, it was composed well enough to send without any adjustments. Also, the bot offered multiple tips for making my emails less mean in the future. Technically, ChatGPT Plus was able to put the email into the right tone, but the writing was stilted and relied too much on formal structures, starting off with, “I hope this message finds you well.”
What about non-work uses for the chatbots? Mimicking one of Google’s prompts for the Gemini Advanced demonstration, I submitted a cute photo of me and my partner hiking in Yosemite National Park and asked it for a pithy, compelling Instagram caption with multiple emoji and no hashtags.
Both chatbots flopped at this task. ChatGPT Plus wrote, “Finding our happy trail 🌲☀️💚 Sometimes the best path isn’t a path at all. Just us, the whispers of the forest, and the promise of another shared adventure. 🥾❤️🗺️” Not exactly the usual connotation for “happy trail.” In contrast, Gemini Advanced refused to write anything about the image, because real people were visible, and the chatbot currently has guardrails to prevent targeted harassment.
Let’s not forget that both of these chatbots include image generation features. To try out that aspect, I asked for a blank party invitation for my imaginary six-year-old’s birthday party. I asked for the invitation to have a Peppa Pig theme and make use of their favorite colors, pink and gold.
ChatGPT Plus did a better job of designing a cool looking invitation, although it made a few cursed Peppas with three eyes and misshapen faces. So, not including a recognizable character in the prompt would likely produce improved results. Neither chatbot was able to create completely legible text for the invite, but results from Gemini Advanced were worse—closer to scribbles than actual words.
Lastly, I wanted to see how good these chatbots were at playing pretend. I asked both tools to roleplay as an ancient space wizard who’s come down to Earth in search of a cool Dungeons & Dragons group to join, but people are low-key afraid to interact with them.
Both chatbots did surprisingly well at this, and the responses were quite entertaining. ChatGPT Plus did a fabulous job of sticking to the character and adding humorous quips like, “Are those your latest wizard robes, or did you raid the wardrobe of a colorblind peacock? The fabric is so bright, I’m afraid we’ll need to cast a spell of shade just to look at you!” Although Gemini Advanced did break character, it was more lyrical and engaging when it came to the fantastical writing elements.
While conversations with a chatbot may feel like a one-on-one affair, it’s never a completely private experience. You should avoid sharing sensitive, private information with any of the publicly available chatbots.
The biggest reason is that the service providers can use your conversations to help train their machine intelligence algorithms. OpenAI does allow users to opt out of having their ChatGPT conversations train the algorithm, but permissions are enabled by default and you have to choose to turn off your chat history. Even in that situation, your chats are not completely ephemeral. According to one of OpenAI’s FAQ pages, “To monitor for abuse, we will retain all conversations for 30 days before permanently deleting.”
That might not sound ideal, but it is better than what Google currently offers to Gemini users. Similar to ChatGPT, if you choose to turn off your Gemini Apps activity, then conversations will not be used to train other interactions of the AI model. Although, if your Gemini chat is randomly selected for human review, it sticks around on Google’s servers, even if you decide to delete it. A conversation that’s selected for review is disassociated from your account, but it could be saved by the company for up to three years.
Nope, but there is a catch. While ChatGPT Plus and Gemini Advanced are available in many international markets, the language you’d like to use when interacting with the chatbot does matter. English is prioritized by both companies.
ChatGPT supports languages other than English, but with limited success. Gemini Advanced at launch was just designed for queries in English, and support for other languages, like Japanese and Korean, is starting to roll out for Google’s chatbot.
Maybe one day, but not in 2024.
Yes, many of the AI tools are currently overhyped. Consumer expectations of AI-fueled hyperproductivity may not match what even top tier chatbots are currently capable of achieving. Even with that in mind, the tools provide real value for power users and curious early adopters, who may feel comfortable paying a monthly subscription for access to the nascent technology.
Even if chatbots eventually disappear from the spotlight—whether it be due to a loss of public interest or loss of copyright court cases—the underlying technology is likely to be foundational to the next wave of web browsers, search engines, and operating systems. And sure, maybe one day AI developers will achieve their grand ambitions to create a divinely powerful, semi-sentient algorithm that changes society forever. But for now, you can pay $20, and it’ll help you write better emails at work.
https://www.wired.com/feed/rss
Reece Rogers
