AI Wrote 162 Python Packages. Only One Worked. Here's What That Means for Open Source

Originally published on tamiz.pro.

An experiment that set an AI loose on the Python ecosystem just returned a brutal verdict: out of 162 packages it generated, only one actually worked. That's a 0.6% success rate. It's not just a statistic — it's a signal about the gap between code that looks correct and code that is correct, and it has profound implications for how we trust, vet, and secure the open source ecosystem.

The Experiment: What Actually Happened

The setup was straightforward: a researcher (or a team) prompted an LLM to generate Python packages that would be useful to developers. The AI produced 162 packages, each with a setup.py, some source files, and a README. Then the team ran a standard quality gate: install the package in a clean environment, run its test suite (if any), check for common security anti-patterns, and verify that the public API actually works as documented.

Only one package passed all checks. The other 161 failed for reasons that range from the mundane to the catastrophic:

• Broken imports: The package imported modules or functions that don't exist in the Python standard library or in the declared dependencies.
• Incorrect API usage: The code called a library function with the wrong signature, or assumed a method existed that was removed in a newer version.
• Missing dependencies: The install_requires list was incomplete, so the package crashed at runtime when a transitive dependency wasn't present.
• Syntax errors: The generated code wasn't even valid Python — a basic python -m py_compile would have caught it, but nobody ran it.
• Security anti-patterns: Hard-coded API keys, eval() on user input, subprocess calls with shell=True, and unsafe pickle deserialization.
• Semantic drift: The package "worked" in isolation but broke when used with realistic data or in a real-world context.

The one that worked? It was probably a trivial utility — a function that does one thing and does it correctly. That's not a coincidence.

Why AI-Generated Packages Fail

The 0.6% success rate isn't a failure of the AI's "intelligence" — it's a failure of the evaluation criteria. LLMs are trained to produce text that looks like code, not code that behaves correctly. They don't have a mental model of the runtime environment, the dependency graph, or the actual semantics of the libraries they invoke. They're pattern matchers, not reasoners.

Here's a deeper look at the technical failure modes:

Hallucinated APIs

The most common failure is hallucination. The model invokes a function like requests.get_json() — which doesn't exist — because it's a plausible combination of requests.get() and .json(). It looks right to a human skimming the code, but it crashes the moment you run it.

Version Blindness

LLMs have a training cutoff. They know about libraries as of a certain date, but they don't know that pandas changed its API in version 2.0, or that numpy deprecated np.float. When the model generates a package, it pins a version in setup.py that may not even exist, or it uses an API that's incompatible with the version it declares.

No Error Handling

Generated code rarely includes robust error handling. It assumes the happy path. No try/except, no validation of inputs, no handling of edge cases. This means that even if the package installs, it will fail in production under real-world conditions.

Security by Default

This is the scariest one. The model doesn't have a security model. It will happily generate subprocess.run(f"rm -rf {path}", shell=True) because that's a common pattern in the training data. It will embed credentials in source code because that's what it's seen. It doesn't know that these are anti-patterns.

The Open Source Supply Chain Problem

Open source is built on trust. When you pip install a package, you're implicitly trusting that the maintainer wrote the code, ran the tests, and didn't introduce a backdoor. AI-generated code breaks that trust model in two ways.

First, it creates a volume problem. A single attacker can use an LLM to generate thousands of plausible-looking packages in minutes. The packages won't all work — but they don't have to. They just have to look legitimate enough for a developer to install them. This is a perfect breeding ground for typosquatting: an AI can generate requests2, requestss, and requests-utils — all with realistic READMEs and setup files — and only the ones that get installed matter.

Second, it creates a verification problem. The one-in-162 success rate is actually the wrong metric to focus on. The real problem is that the 161 failures look just as legitimate as the one success. A human reviewing a package's source code can't easily tell the difference between an AI-generated package that will crash and an AI-generated package that contains a hidden backdoor. Both have the same superficial quality: plausible code, a README, a license, and a setup.py.

This is why we need stronger provenance in package registries. We need to know who or what wrote the code. We need package signing, reproducible builds, and a clear audit trail. Without that, AI-generated code will become a vector for supply chain attacks at a scale we've never seen.

What Maintainers and Developers Should Do

The practical takeaway is simple: treat AI-generated code as untrusted input. Here's what that means in practice:

1. Never install a package without checking its source. Look at the actual code, not just the README. If the code looks like it was generated by an AI (and you can usually tell — it's too uniform, too clean, too "correct" in a superficial way), be suspicious.
2. Run security scans. Use tools like pip-audit, safety, and bandit to check for known vulnerabilities and anti-patterns. This should be a standard part of your CI pipeline.
3. Test in a sandbox. Before you add a dependency to your project, install it in a clean virtual environment and run your test suite. If it breaks, you've caught it early.
4. Pin exact versions and use lock files. Don't let a transitive dependency update to a version that was generated by an AI. Lock files are your first line of defense.
5. Verify checksums. Use pip install --require-hashes to ensure that the package you're installing is exactly the one you reviewed.

Here's a quick example of how to audit a package before you commit to it:

# Create a clean venv and install the package
python -m venv /tmp/test-env
source /tmp/test-env/bin/activate
pip install --require-hashes -r requirements.txt

# Audit for known vulnerabilities
pip-audit

# Scan for security anti-patterns
bandit -r src/

If any of these steps fail, don't use the package. It's not worth the risk.

The Future of AI-Assisted Open Source

This experiment shouldn't be read as a condemnation of AI in software engineering. It should be read as a clarion call for better tooling and better process.

The future isn't "AI writes packages." The future is "AI writes tests, AI reviews code, AI suggests patches, AI flags potential vulnerabilities." In other words, AI should be a copilot, not an autopilot. The moment you let AI-generated code into your production dependency tree without human verification, you're gambling with your supply chain.

We also need better benchmarks. The current evaluation criteria for AI code generation — "does it compile?" or "does it pass a unit test?" — are insufficient. We need benchmarks that measure real-world correctness: does the code handle edge cases? Does it handle security? Does it actually solve the problem it claims to solve?

The 0.6% success rate is a wake-up call. It tells us that AI is not yet ready to write production-grade software on its own. But it also tells us that AI is capable of generating a huge volume of code — and that volume, if left unchecked, is a serious risk to the open source ecosystem.

Key Takeaways

• AI-generated code has a ~0.6% success rate in a realistic quality gate — that's a 99.4% failure rate.
• The failures are technical: hallucinated APIs, version mismatches, missing error handling, and security anti-patterns.
• The risk is supply chain: AI can generate thousands of plausible-looking packages, enabling typosquatting and malware at scale.
• The mitigation is process: never trust AI-generated code without human review, automated security scanning, and strict version pinning.
• The future is copilot, not autopilot: AI should assist, not replace, human judgment in open source.

For more on the intersection of AI and software engineering, check out Tamiz's Insights and tamiz.pro.