LLMs create insecure software as of today

LLMs have a bit of a general issue, which is that they don't produce correct, high-quality code in one-shot. The solution is to have other LLMs review the code in a feedback loop to resolve the issues. Only after that can you have some confidence that the output is correct or good enough.

Why is this a problem?

Well, the problem is that LLMs don't produce secure code in one-shot either. The solution to have other LLMs review the code for security defects. Unfortunately, this is problematic, because to get high security you really need adversarial security review, but that is blocked by frontier AI companies.

The AI companies correctly reason that if you can do proper security reviews against the code you've produced, you can also use that capability against code (or systems) that you don't own, which would allow users to carry out cybersecurity attacks. There is no easy way around this. AI companies can't reliably detect if the code was "originally created" by their own AI, because there would be many ways around such detection.

You can, of course, try to apply for elevated access, but the vast majority of developers using AI can't or won't do that, which means that this is not an option for the vast majority of developers.

Another practical workaround is using open-weight Chinese models that don't have such a restriction. However, Chinese models might start to have such a "block" too, as K3.1 Pro's open-weight version, for example, lacks cybersecurity data. And many companies don't allow the use of Chinese or other open-weight models.

We can't solve the security issues with human review either. The vulnerabilities that LLMs will produce are not trivial issues but something that almost no human can spot. However, any attacker who has access to a frontier model (without cybersecurity restrictions) can easily reveal them.

It seems that we have hit a roadblock for security. Could there be a way to ensure high security for software but not allow attackers to break other systems?

My solution is as follows: AI providers can host the software in a feature called "secure cloud". Companies will submit their packages to the AI company, which does the adversarial security review and patching themselves in a "black box", never revealing the vulnerabilities. The AI providers then offer a standard cloud hosting service and don't let the customer access the patched version of software. This, of course, would be incredibly profitable for the AI companies.