Back

That's part of the problem with "I tried an LLM and it worked, therefore LLMs are useful." It takes a lot more than trying things out to reach that conclusion.

I've used the analogy of a Vegas slot machine before; if all you see are people getting good results out of LLMs, it's easy to believe that's the typical mode of operation.

There's the old saw about "the plural of anecdote is not data," but I think that's bullshit. It's absolutely data. Just maybe not complete, fairly sampled, or completely understood data.

If someone like me points out that the claim that LLMs are useful hasn't been substantiated empirically, pointing out personal anecdotes where they have been useful to you isn't nothing, but it's not enough to establish the extraordinary claim, either.

At this point, if you've read this thread or anything else I've posted for the past approximately two years, you might well accuse me of engaging in high-school debate logic. I'd argue that's a bit unfair, I'm at least at the college debate level here, but still.

I do think debating and talking through this is important, though. AI proponents have proposed and started enacting the project of restructuring society to be convenient for AI producers.

Copyright law and policy, climate action, water usage policy, land usage for data centers, compensation for software engineers involved in and adjacent to AI production, hell, OpenAI has even made *foreign policy* arguments on the basis of the claim that their AI products are immensely useful to society writ large.

Those are all massive changes to society, undertaken on the basis of an extraordinary and unproven claim.

So, like, yeah. Maybe we should be a bit more careful about our empiricism than relying only on personal anecdotes and citation cliques. Maybe we should be clear where the burden of proof lies. Maybe we should be suspicious of claims made by financially motivated actors.