That's part of the problem with "I tried an LLM and it worked, therefore LLMs are useful." It takes a lot more than trying things out to reach that conclusion.
I've used the analogy of a Vegas slot machine before; if all you see are people getting good results out of LLMs, it's easy to believe that's the typical mode of operation.