There are useful things too.
Once you set up a RAG system, it kind of allows for custom training your own statistical models, which can be incredibly useful and significantly improve behaviour and reduce hallucinations. It also gets a lot of benefit from clear task structure that's built in a way that's token-efficient to parse, which is a whole discipline to learn in and of itself...
I have found it useful to help me generate the first 80% of work that I probably would have blown off as unimportant before. Stuff like status reports or transforming my notes into a design doc for others to use. Only the first 80% though, because it's still not even close to a replacement for a human that knows what they're talking about. It's also cool for doing some deep context searches of an unknown codebase (though it still hallucinates pretty heavily here, and you need to be vigilant).
Without RAG it's at best a teenager* on their first week on the job equipped with OG google search from 2002-2010 (when it was good enough to basically do the same thing AI is doing now for research, before Google needed to make the product worst to improve their monetization of it). You can make it do stuff for you, but you wouldn't trust it with anything except the clearest instructions.
It's a cool tool that people that are good at building systems can use to build systems differently. Clever people can now do clever things slightly faster (maybe, evidence is pointing to that it's actually slower than just doing it by hand). And purpose built statistical models will, shock, continue to do amazing things (see all the AI "breakthroughs" in sciences)...none of that's really new though. That's the same thing we've been seeing for decades, packaged differently.
People that are not good at building systems will build horrifying monstrosities with it that will require much engineering to undo in the coming decades.
It's also wildly socially toxic because it's being touted as actual AI and a replacement for people. And as noted above, loads of people are offloading critical thinking to it because it's easier and it does get things right a good percentage of the time. Especially simple things with well documented truth-based evidence that's extant on the internet.
Unfortunately, the important parts of life are rarely simple things things with well-documented truth-based evidence extant on the internet.
ETA: *A properly dim teenager that keeps forgetting the corrections you have to constantly give them.
)
