We are thinking in the old paradigm of AI creating software in the old syntax. It's like genetically modifying horses instead of inventing a car for more powerful transportation.

Perhaps, this or next AI model will still be tied to the same software engineering paradigm, but true breakthrough will happen when a supersystem of the current "programming systems" will be consciously build. Roughly, some kind of programming language "Unicode" will be standardized so there will be no need to reinvent code (using AI or not). Then AI can help to be the "search" in that universal repository. Also some kind of verification capabilities needed because LLMs can't produce correct algorithms for all but trivial problems, and trivial problems will not even need to be written if said repository will be there.

AI is NOT a mathematician, and will never become true one. Gödel's completeness theorem covers the case. AI model is just a very big formula after all.

Until that time AI will serve to generate a lot of crap until stakeholder (who were not wise enough to rely on AI for "programming" too much) will realize their mistakes.