It's the unfounded promises that this will be solved because the tech will only get better that really upset me. Because sure, it will get better, I'm pretty certain of that. They'll have additional capabilities, they'll have access to more-recent data, etc. But "better" does not necessarily equate to "will fix the lying problem." That's a problem that is BAKED INTO the technology, and requires some kind of different approach to solve -- you can't just keep making a hammer bigger and bigger in the hopes that one day it'll turn into a screwdriver.

Before LLMs really took off, we were in the middle of an "AI winter", where there just weren't any promising techs, at least none with sufficient funding attached to them. And it's WORSE now. LLMs have sucked all the air out of the room, and all of the funding out of other avenues of research. Technologies that were "10-20" years away now might be 30-40, because there's fewer people researching them, with less money, and they might even be completely different people trying to restart the research after the old ones got recruited away to work on LLMs!

We lost that battle back when we collectively decided that sales and marketing is respectable work.

And suddenly this type of quality is becoming "normal" and acceptable now?

The notion that "computers are never wrong" has been engrained in society for at least a century now, starting with scifi, and spreading to the rest of culture.

It's an idea that has caused more harm than good.

> Normally this would never fly, but nowadays it's kind of OK?

We started down this path ever since obvious bugs were reframed as "hallucinations".

Complain to it enough times, remain resilient and you’ll eventually figure it out (that’s a wild card though). Or find someone who has and take their word for it (except you can’t because they’re probably indistinguishable from the ‘bot’ now according to the contradictory narrative). Iterate. Spiral. No one should have to go through that though. Be merciful.

I've recently started wondering what the long term impacts of AI slop is going to be. Will people get so sick of the sub-par quality that there will be a widespread backlash, and a renewed focus on handmade or artisinal products made by hand? Or will we go the other way where everyone will accept the status-quo and everything will just get shittier, and we will just have multiple cycles of AI slop trained on AI slop?

Suddenly? That's the level of quality that is standard in all software projects I've ever seen since I've started working in IT.

Enshittification is all around us and is unstoppable. Because we have deadlines to hit and goals to shows we reached to the VP. We broke everything and the software is just half working? Come on that's an issue for the support and ops teams. On to the next beautiful feature we can put on marketing slides!

>Why should false and or inaccurate results be accepted?

The typical response is "because humans are just as bad, if not worse."

And how quickly the bar is being lowered

When were search results 100% fact checked and accurate??

Half my browser extensions have sole purpose of removing shit from sites I visit.

HN is like a unicorn that havent made me block a single thing yet.

We can't expect the vast majority of regular users to have any of that skill.

What is this going to lead to? fascinating times.

This is a very very good point. If this was happening with different queries that we never used, or a new type of questions/queries than I would have some patience. But it happens exactly with the formulations that were giving you the best results in the SERP before!

I'm a member of a few car groups on Facebook and the misinformation coming from Google is infuriating, because people treat it as gospel and then you have to explain to them that the AI slop they were shown as the top result in Google is not - in fact - correct.

As a simple example - someone googled "how to reset sensus system in Volvo xc60" and Google told them to hold the button under the infotainment screen for 20 seconds and they came to the group confused why it doesn't work. And it doesn't work because that's not the way to do it, but Google told them so, so of course it must be true.

Ad supported search has been awful for a few years now, just buy a Kagi subscription and you'll be like me: horrified but mildly amused with a dash of "oh that explains a lot" when people complain about Google

That was true before AI too (I know, I did such searches myself). Google results has been drowning in slop for over a decade now - it was just human-generated slop, aka. content marketing and SEO stuff.

I'm not defending the AI feature here, just trying to frame the problem: the lies and hallucinations were already there, but nobody cared because apparently people don't mind being constantly lied to by other people.

I the have had a lot of luck with copilot conversations to research stocks and trading strategies. I am always skeptical of results and verify everything with various sources but it does help me find/get on the right track.

I think it's just much easier for an LLM to learn how to be convincing than it is to actually be accurate. It just has to convince RLHF trainers that it's right, not actually be right. And the first one is a general skill that can be learned and applied to anything.

https://arxiv.org/html/2409.12822v1

> eg i ask for a source, it gives me a quote

It should give you both - the quote should be attributed to where it was found. That's, generally, what people mean when they ask or search for "a source" of some claim.

As for general point - using LLMs as "better search" doesn't really look like those Google quick AI answers. It looks like what Perplexity does, or what o3 in ChatGPT does when asked a question or given a problem to solve. I recommend checking out the latter; it's not perfect, but good enough to be my default for nontrivial searches, and more importantly, it shows how "LLMs for search" should work to be useful.

> LLMs as a replacement for search

Some people expect LLMs as part of a better "search".

LLMs should be integrated to search, as a natural application: search results can heavily depend on happy phrasing, search engines work through sparse keywords, and LLMs allow to use structured natural language (not "foo bar baz" but "Which foo did a bar baz?" - which should be resistant to terms variation and exclude different semantics related to those otherwise sparse terms).

But it has to be done properly - understand the question, find material, verify the material, produce a draft reply, verify the draft vis-a-vis the material, maybe iterate...

Some chatbots plan a query and summarize what a search returns instead of trying to produce an answer on their own; I use perplexity a lot which always performs a search, I think ChatGPT et al have some kind of classifier to decide if web search is necessary. I especially use it when I want a suggestion without sifting through pages of top ten affiliate listicles (why is there a list of top 10 microwaves? I only need one microwave!)

Its good to be shown direction. When I only have a vauge idea of what I want, AI usually helps me frame it into searchable terms I had no clue existed.

I find LLMs are often better for X vs Y questions where search results were already choked by content farm chaff. Or at least LLMs present more concise answers, surrounded by fewer ads and less padding. Still have to double check the claims of course.

Maybe that's because we're conditioned by the UX of search.

But another thing I find even more surprising is that, at least initially, many expected that the LLMs would give them access to some form of higher truth.

Gemini was underwhelming until 2.5 Pro came along, which is very good. But in my experience all of the Google models are far worse than everything else when it comes to hallucination.

I switched to DDG over seven years ago and just realized it had been that long when I read your comment. Google started wasting my time and I had to shift.

fyi, you can remove any and all “ai” assistant bs etc from DDG if you use the noai subdomain (in case you wanna avoid their stuff, although it’s much less prominent anyway) https://noai.duckduckgo.com/

What’s the word?

Many of them try to be mindful of their climate impact.

I've tried to explain it in those terms as well: every medium-sized prompt on these large models consumes roughly one phone battery charge worth of energy. You have a phone with a calculator.

I'd ask them to do the math on how much energy they're wasting asking stupid things of these systems, but I'm too afraid they'd ask ChatGPT to do the math.

> Let's use the word machine for this number problem!

You know, that's a thought process that makes internal sense.

You have someone who's terrible at math. They want something else to do math for them.

Will they prefer to use a calculator, or a natural language interface?

How do you use a calculator without knowing what you're doing?

I'm so lazy, I have chat bots do all kinds of complex calculations for me. I even use it as a stock screener and the poor thing just suffers, burning fuck tons of electricity.

That's just plain wrong, and I'm a smoker. LLMs won't affect the ones around you, unless you engage with them in some way. Sit next to me while I smoke and you'll be affected by passive smoking regardless of how much you engage or not. Not really a accurate comparison :)

Wish I could edit, but I was referring to the bill splitting math specifically here. I didn't mean to quote the rest.

When it comes to facts that actually matter, people need to know to verify the output.

Yes, more popups, retention metrics be damned. Even 2 years since ChatGPT, many people still think it's omniscent which is what's causing trouble.

That always struck me as pretty overblown, given that before map apps, people got lost all the goddam time. It was a rare trip with any complexity that a human map reader wouldn’t make a mistake or two.

LLMs aren’t competing with perfect, they are competing with websites that may or may not be full of errors, or asking someone that may or may not know what they are talking about.

Yeah, but we're all used to having software integrated into our lives now. And we all now how shitty and broken software often is...

Apple maps currently insists that there's a hotel and restaurant across the street from me.

According to the address on the business website that Apple Maps itself links to, the business is 432 km away from me.

Unfortunately, getting bit in cases such as publishing misinformation or false legal citations waste everyone time, not just their own.

Which local models are you using, that do not output loop garbage at temperature 0?

What do you get at very low temperature values instead of 0?

Although I broadly agree, I wouldn't go quite as far as where you say:

> All the "no true Scotsman" stuff about what it's doing behind the scenes is irrelevant, because we have no idea what human brains are doing behind the scenes either.

Computers and transistors have a massive speed advantage over biological brains and synapses — literally, not metaphorically, the same ratio as the speed difference between how far you walk in a day and continental drift, with your brain being continental draft — which means they have the possibility of reading the entire Internet in a few weeks to months to learn what they know, and not the few tens to hundreds of millenia it would take a human.

Unfortunately, the method by which they acquire information and knowledge, is sufficiently inefficient that they actually need to read the entire Internet to reach the skill level of someone who has only just graduated.

This means I'm quite happy to *simultaneously* call them extremely useful, even "artificial general intelligence", and yet also agree with anyone who calls them "very very stupid".

If we actually knew how our brains did this inteligence thing, we could probably make AI genuinely smart as well as absurdly fast.

Their point wasn’t that it’s not useful. It’s that it isn’t artificial intelligence like the masses consider the term.

You wouldn’t say Intellisense isn’t useful but you also wouldn’t call it “AI”. And what LLMs are like is basically Intellisense on steroids (probably more like a cocktail of speed and acid, but you get my point)

It matters if we are making a distinction between essence and output.

On the output side, it functionally doesn’t really have a difference. At least in terms of more abstract things like writing code. Although I would argue that the output AI makes still doesn’t match the complexity and nuance of an individual human being, though, and may never do so, simply because the AI is simulating embodiment and existing in the world. It might need to simulate an Earth equivalent to truly simulate a human’s personal output.

In the essence side, it’s much more of a clear distinction. We have numerous ways of determining if a thing is human or not - biology, for one. It would take some serious sci-fi until we get to the point where an android is indistinguishable from a human on the cellular level.

Historically, there's been some discussion about that:

https://en.wikipedia.org/wiki/Chinese_room

LLMs can't write code.

They don't have capacity to understand logical or temporal relationships, which is the core competency of coding.

They can form syntactically valid strings in a formal language, which isn't the same thing as coding.

How could you think this hard and be so far off. Google is in a hyper strong position here and I don’t even like them.

They can refine grounded results over time and begin serving up increasingly well reasoned results over time as Models improve cost effectively. Then that drives better vectors for ads.

Like what about this is hard to understand?

Most folks just want confirmation. They don't want to have their views/opinions changed. LLM are good at trying to give folks what they're looking for.

Repent.

You are not there to "love what gives you the kicks". That's a kind of love that should not exit the bedroom (better, the bathroom).

> Most people use visual body cues to determine whether someone is lacking in confidence in their answer

Do they?

And a rational interlocutor, besides conversational shortcuts, replies in the form "The best estimations from sources like S0, S1 and S2, publish a value between V1 and V2".

The one that appears when you search with Google.

I have a system prompt that gives me probability estimates to everything the LLM claims. Its a simple fix for your problem.

We consider LLMs an "intuition" machine that can talk and partially understand, and do not let it retrieve information from its faulty memory and will, but force it to use an implemented memex to produce any output.

"Tell us about X (X='PS/2 Models'); here are your encyclopedias: extract and formulate".

While you argue that showcasing a 'faulty assistant' like Google's is 'informative', particularly for those tracking AI progress, the typical LLM-got-it-wrong post often doesn't provide that deeper insight. It usually presents an isolated error without context or analysis of the system's architecture, training data limitations, or the specific type of reasoning failure. This makes its informative value quite shallow, quickly becoming repetitive rather than truly enlightening about 'progresses in the area' beyond the surface-level observation that LLMs are imperfect.

Regarding the 'constructive' aspect and the 'damaging replacement of experts,' I agree this is a critical concern. However, the genre of simply posting screenshots of LLM errors is rarely constructive in addressing this complex socio-technical issue. It highlights a symptom (LLMs making mistakes) but typically fails to constructively engage with the causes or potential solutions for deskilling, corporate responsibility in AI deployment, or the nuances of human-AI collaboration. True constructive engagement would require more than just pointing out a wrong answer; it would demand analysis, discussion of best practices, or calls for better system design and oversight, which this genre seldom provides.

Dismissing my observation with 'then don't read it' sidesteps the core issue. My point isn't about my personal reading habits, but about the low signal-to-noise ratio of this content genre. While you argue these stories are 'needed' because people blindly trust LLMs, especially with integrations like Gemini in search, these posts rarely offer more than the simplistic, already widely understood caution: 'don't blindly trust LLMs.' This is precisely the 'usual adage' I mentioned. The genre often lacks depth, failing to provide nuanced understanding or genuinely new information about why these systems fail in specific ways or how users can develop better critical assessment skills beyond mere distrust. If the goal is genuine education due to increased LLM exposure, the content needs to evolve beyond just showcasing errors.

Gemini doesn't refer to a specific model so perhaps the one on the search page is weaker than the ones it offers in the app.

> You should know that LLMs are very weak in procedural operations.

Indeed! That's why they are LLMs, not AI!

Why are new hypes always re-defining terms? 20 years ago, "AI" was actually about intelligence. Just like "crypto" was about cryptography instead of money scams.

> The phenomenon is part of all the "obvious" things that are not in the awareness of very large masses.

I can't imagine a single HN reader who is not aware that LLMs make mistakes. This is not the general public. (And even most of the general public has heard about this minor issue by now.)