My Leak Scanner Reported 16 Bugs in CPython. I Read All 16: Every One Was a Handoff.
I wrote a resource-leak scanner for Python. Its own documentation claims it was calibrated on real code and comes back clean: "CPython standard library, 736 files, 1 HIGH β and that one is a genuine leak." This week I p
I wrote a resource-leak scanner for Python. Its own documentation claims it was calibrated on real code and comes back clean: "CPython standard library, 736 files, 1 HIGH β and that one is a genuine leak."
This week I pointed the same tool at a bigger target: the whole Lib/ tree of CPython 3.14, including the test package.
It reported 16 HIGH severity leaks. I read all 16, line by line, and then wrote down what each one actually was.
All 16 were handoffs. Not one was a leak. Every single HIGH came from the test tree, and production code came back at zero.
That is a useful result, but not the one the summary line implies. Here is the full accounting, the seven handoff shapes, and the bug my first fix introduced.
The run
The tool is a single standard-library file. No install, no dependencies, AST only, never executes the code it scans:
python3 resource_leak_doctor.py scan C:/Python314/Lib --json > report.json
Numbers on that corpus:
| Metric | Value |
|---|---|
.py files scanned |
1,851 |
of which under Lib/test/
|
1,131 |
constructor call sites (open, io.open, socket.socket, sqlite3.connect, Popen) |
2,417 |
| findings total | 105 |
| HIGH | 16 |
| MEDIUM | 16 |
| LOW | 73 |
HIGH located in Lib/test/
|
16 of 16 |
| HIGH located in production code | 0 |
So the honest summary is not "there are 16 leaks in CPython". It is: my HIGH rule fires almost exclusively where handoff is the normal idiom β test code.
The 16, adjudicated
Every finding was an assignment of the shape name = <constructor>(...), followed by no close(), no with, and no return of that name within the same function. All true as stated. All misleading as conclusions.
Grouping them by what the handle was passed to:
| Shape | Count | Example site |
|---|---|---|
| mocked constructor (no real handle exists) | 7 | test_unittest/testmock/testwith.py |
detach() β ownership explicitly pushed out |
3 |
test_httplib.py:1502, test_socket.py
|
ownership-sink API (loop.connect_read_pipe) |
2 | test_asyncio/test_events.py |
cleanup registry (addCleanup(sock.detach)) |
1 | test_socket.py:6152 |
| callee closes it (thread target) | 1 | test_asyncio/utils.py:285 |
enclosing scope owns it (nonlocal) |
1 | test_socket.py:5295 |
the failure is the assertion (assertEqual(e.errno, EPERM)) |
1 | test_posix.py:1138 |
Take the one that looks most like a real bug:
# Lib/test/test_asyncio/utils.py:285
@contextlib.contextmanager
def run_udp_echo_server(*, host='127.0.0.1', port=0):
addr_info = socket.getaddrinfo(host, port, type=socket.SOCK_DGRAM)
family, type, proto, _, sockaddr = addr_info[0]
sock = socket.socket(family, type, proto)
sock.bind((host, port))
sockname = sock.getsockname()
thread = threading.Thread(target=lambda: echo_datagrams(sock))
thread.start()
sock is never closed in this function. It is closed in echo_datagrams when the sentinel arrives:
def echo_datagrams(sock):
while True:
data, addr = sock.recvfrom(4096)
if data == b'STOP':
sock.close()
break
The handle crossed a function boundary, into a thread. A per-function ledger cannot see that. The finding is arithmetically correct and semantically useless.
test_posix.py:1138 is even worse in the other direction β open() is supposed to fail there, because the test asserts that an immutable file returns EPERM. Flagging a guarded open() as a leak is like flagging assertRaises as an uncaught exception.
Fix #1: I made it worse
My first fix was a post-filter over the JSON report. Ninety lines, text regex on the enclosing function: if the file contains name.detach(), waive. If it contains addCleanup(name.close), waive. Ready in twenty minutes.
I ran it against a regression corpus I built by hand β seven files, some real leaks, some handoff decoys:
HIGH=5 waived=3 kept=2
WAIVED detach() real_leak_comment.py:2 <-- wrong
KEEP expected_failure.py:5 <-- wrong
real_leak_comment.py is a genuine leak: f = open(p), read, return, never closed. It got waived because it contains this comment:
def read_all(p):
f = open(p)
# later we could f.detach() but we never do
data = f.read()
return data
A comment saying "we never detach" was read as evidence of detaching. And expected_failure.py was kept because my assertRaises-and-except OSError predicates both had to match in the same function body β they matched in CPython's version, not in mine.
Two bugs, one corpus, twenty minutes. This is exactly how a "precision improvement" turns a noisy tool into a confidently wrong one: false negatives are silent.
Fix #2: name the handoff with the AST
The rewrite asks one question per finding β was the handle handed off, and by which mechanism β and answers it on nodes, not text:
-
detach()β aCallwhosefuncisAttribute(attr='detach')and whose receiver is that exactName. Comments cannot satisfy this. - cleanup registry β
addCleanup(name.close)/addCleanup(name.detach)/atexit.register(name.close). - callee closes it β the name appears in an argument position (including inside tuples, lists and lambdas:
threading.Thread(target=lambda: echo_datagrams(sock))), and the resolved callee's parameter is closed inside its own body. - enclosing scope β
nonlocal namein the assignment's function; the outer scope closes it later. - expected failure β the assignment sits inside a
Trywhose handler catchesOSError/PermissionErrorand contains an assertion. - ownership sink β passed to
connect_read_pipe/connect_write_pipe/fdopen, APIs that document taking the handle. - mocked constructor β the file builds the call with
mock_open(...)or patches the called name.
Result on the same 1,851 files:
HIGH=16 waived=16 kept=0
WAIVED ownership-sink(connect_read_pipe) test/asyncio/test_events.py:1590
WAIVED ownership-sink(connect_read_pipe) test/asyncio/test_events.py:1659
WAIVED callee-closes(echo_datagrams) test/asyncio/utils.py:285
WAIVED detach() test/test_httplib.py:1502
WAIVED expected-failure test/test_posix.py:1138
WAIVED detach() test/test_socket.py:2046
WAIVED enclosing-scope-owns test/test_socket.py:5295
WAIVED cleanup-registry test/test_socket.py:6152
WAIVED detach() test/test_socket.py:7435
WAIVED mocked-constructor test/test_unittest/testmock/testwith.py:194
... (7 mocked-constructor total)
And the part that matters more, the regression corpus, where two real leaks must survive:
HIGH=5 waived=3 kept=2
WAIVED expected-failure expected_failure.py:5
WAIVED callee-closes(serve) handoff_callee2.py:14
WAIVED mocked-constructor mocked.py:6
KEEP None real_leak.py:2
KEEP None real_leak_comment.py:2
The comment decoy now survives. That is the whole point: the AST version has fewer false positives and no new false negatives on the same corpus, and both numbers are printed, not asserted.
Four rules I now ship with
1. Every waiver carries a name. The filter does not delete findings, it annotates them. enclosing-scope-owns and mocked-constructor are auditable claims; "waived: heuristic" is not. When a waiver is wrong you can see which belief was wrong.
2. "Unexplained" is not "clean". The adjudicator exits 2 if anything survives, 0 if everything is explained, and it never conflates the two. The scanner itself already returns 3 when it finds zero .py files β because a gate that only checks "exit code != 0" becomes decoration the day someone passes a wrong path.
3. A HIGH severity is a claim about reviewer minutes, not about the defect. Sixteen HIGH in a codebase means sixteen review minutes spent by someone who trusts the label. If the tool cannot distinguish "leak" from "handoff", the correct output is a handoff candidate at the severity of a question, not a HIGH.
4. Precision is a property of the rule Γ the corpus, and the corpus moves. My earlier calibration reported 1 HIGH on 736 files. Same rule, 1,851 files: 16. The rule did not get worse; the corpus grew a test tree full of deliberate ownership handoffs. Any "precision: 99%" claim that does not name its corpus and its file list is not a measurement β it is a vibe with a decimal point.
Reproduce it
The scanner is one stdlib file: scan, rules, selftest (40 assertions, exit code is the verdict), --json for CI consumption. The adjudicator is two modules and only needs the report plus the source root.
If you run a linter as a merge gate, the useful test is not "does it pass on my code" but: take a corpus you did not write, run it, and read every HIGH by hand. I have not met a rule yet that survived that without learning a new handoff shape.
The write-ups for this whole series of instrument-calibration experiments live in my repo: github.com/Felixwang007/awesome-content-tools. The scanner itself β with the calibration tables and an evaluation_cases.md boundary statement β is packaged as a skill: θ΅ζΊζ³ζΌδΈε₯ζζΈ
ηδ½ζ£ε©ζ. The seven adjudication predicates are spelled out above rather than pasted, because the shapes are the transferable part; each one is a dozen lines of ast and you should be able to reimplement and re-measure them on your own corpus in an afternoon.
If you have a scanner that reports zero on your codebase, I would like to know what its corpus was before it was your codebase.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.