Last week, OpenAI released its highly anticipated research product, Search ChatGPTto take on Google. The industry has been preparing for this moment for months, prompting Google to inject AI-generated responses into its core product earlier this year, and producing some embarrassing hallucinations in the process. This incident led many people to believe that OpenAI’s search engine was truly a “Google killer.”

But after using ChatGPT Search as your default search engine (you can also do this with The OpenAI extension) for about a day, I quickly returned to Google. OpenAI’s search product was impressive in some ways and offered a glimpse of what an AI search interface might one day look like. But for now, it’s still too impractical to use as a daily driver.

ChatGPT Search was sometimes useful for surfacing real-time answers to questions that I otherwise would have had to wade through lots of ads and SEO-optimized articles to find. Ultimately, it presents concise answers in a nice format: you get links to the news sources on the right side, with headlines and a short excerpt that confirms that the AI-generated text you just to read is correct.

However, this often seemed impractical for everyday use.

In its current form, ChatGPT Search is unreliable for what people use Google for most: short navigation queries. Queries of less than four words represent most of the searches on Google; it’s often just a few keywords that take you to the right web page. This is the kind of search that most people barely realize they’re doing all day, and it’s what Google tends to do very well.

I’m talking about “Celtics score”, “cotton socks”, “library hours”, “San Francisco weather”, “coffee shops near me” and other queries that make of Google the gateway to the Internet for billions of people.

My test with ChatGPT Search was quite frustrating at times, and it made me realize how many keyword searches I perform per day. I couldn’t reliably find information using short queries, and for the first time in years I really wanted Google Search.

Make no mistake, Google’s quality has declined over the last decade, largely because it has been flooded with ads and SEO. Yet, I kept opening Google in a separate window during my test because ChatGPT Search couldn’t provide me with a correct answer or web page.

Who would win: ChatGPT search or short queries?

I typed in “Nuggets score” to check how a live NBA game between the Denver Nuggets and the Minnesota Timberwolves was going. ChatGPT told me the Nuggets were winning even though they were actually losing, and showed the Timberwolves score 10 points lower than it actually was, according to a Google result at the same time.

Another time, I tried “earnings today” to check companies reporting quarterly results that could affect stock prices on Friday. ChatGPT told me that Apple and Amazon were releasing their results on Friday, even though both companies had already released it a day earlier. In other words, he was hallucinating and making up information.

In another test, I typed in the name of a tech lead to find their contact information. ChatGPT showed me a summary of the person’s Facebook profile and hallucinated a link to their LinkedIn page, which produced an error message when I clicked on it.

Another time, I typed in “baggy jeans,” hoping to go shopping. ChatGPT Search first described to me what baggy jeans were (a definition I didn’t need) and recommended I go to Amazon.com to buy a nice pair.

I could go on, but you get the idea. Broken links, hallucinations, and random responses defined my first day of using ChatGPT Search.

Maybe a “Google Killer” One Day, But Not Today

This was not a trivial launch for OpenAI. Sam Altman praised the feature for being “really good,» even though he is known for downplaying his startup’s AI capabilities. The reason this time is different may have something to do with the fact that Search is one of the biggest companies on the Internet, and OpenAI’s release could pose a real threat to its biggest competitor, Google.

To be honest, Google Search is a 25 year old product and ChatGPT Search is brand new. In a blog post, OpenAI says it plans to improve the feature based on user feedback in the coming months, and it seems more than likely that this could be a significant area of ​​investment for the startup.

To its credit, ChatGPT Search is pretty good at answering long written search questions. Something like: “Which American professional sports league has the most diversity?” isn’t a question you can easily answer with Google, but ChatGPT Search is pretty good at pulling multiple websites and getting you a decent answer in just a few seconds. (Perplexity is also very good at answering these questions, and its search product has been around for over a year.)

Compared to the traditional version of ChatGPT, which already had access to the web, the search function looks like a better interface for browsing the web. There are now clearer links to the sources from which ChatGPT obtains its information: for news, ChatGPT will use the media companies it has consulted. enter into all these licensing agreements with.

The problem is that most Google searches aren’t that long of a question. To truly replace Google, OpenAI needs to improve on these more convenient, shorter searches that people already perform throughout their day.

OpenAI is quick to acknowledge that ChatGPT Search struggles with short queries.

“With ChatGPT search, we have observed that users tend to start asking questions in a more natural way than in the past with other search tools,” OpenAI spokesperson Niko Felix said. in an emailed statement to TechCrunch. “At the same time, web browsing queries, which tend to be short, are quite common. We plan to improve the experience for these types of queries over time.

That said, these short keyword queries have made Google indispensable, and until OpenAI achieves this, Google will remain the mainstay for many people.

There are several reasons why OpenAI might have difficulty with these short queries. The first is that ChatGPT relies on Microsoft Bing, which is widely considered an inferior engine to Google. The second reason is that large language models may not be well suited to these short prompts. LLMs generally need fully written questions to produce effective answers. May need re-prompting (running short queries through an LLM as a longer question) before ChatGPT Search can properly perform such searches.

Although OpenAI has only now released its research product, Perplexity’s own AI research tool is already answering 100 million search queries per week. Perplexity has also been touted as a “Google killer,” but it has the same problems with short queries.

Aravind Srinivas, CEO of Perplexity, explained how people use its product differently compared to Google Search during TechCrunch Disrupt earlier this week: “The median number of words in a Google query is between two and three. In Perplexity, that’s about 10-11 words. So clearly the usage in Perplexity is more about people coming in and being able to directly ask a question. On the other hand, at Google, you type a few keywords to instantly access a certain link.

I think the fact that people aren’t using these products for web browsing presents a bigger problem than OpenAI or Perplexity let on. This means that ChatGPT Search and Perplexity do not replace Google Search for the task it is best for: web browsing.

Instead, these AI products fill a new niche, surfacing insights that are buried in traditional search. Don’t get me wrong, that’s valuable in its own right.

Both OpenAI and Perplexity say they will work to improve at these short queries. Until then, I don’t think any of these products can completely replace Google. If OpenAI wants to replace the gateway to the Internet, it needs to create a better one.