This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

#devchallenge #hf26challenge

What I Built

Leave It is a morning filter for a backlog I have decided not to read.

I point it at a stream of items — for this post, 731 real GitHub issues from
vercel/next.js, rust-lang/rust, python/cpython and microsoft/vscode. For each one,
a fine-tuned 4B open-weight model answers a single question: does this need a maintainer
today?

Items it answers WAIT for do not get shown to me. They do not get deleted either. Each
one is held by a durable Temporal fuse, so if nobody ever engages with it, it stays
buried; if someone does, it surfaces immediately. The morning surface is one card,
not a list, and definitely not an inbox with 731 rows in it.

The whole product is one sentence: the tool's job is to earn me the right to skip
something, and then to make sure being wrong about that is cheap.

The number, up front. On the held-out window it defers 73 of the 731 items (10.0% of the stream) while wrongly deferring 11 of the 235 items that genuinely needed a maintainer within 24 hours. The same model with the same prompt and no fine-tune defers 49 (6.7%) at AUC 0.516, which is a coin flip. The best classical arm reaches 10.7%.

Demo

The demo is a static page generated from the same artefacts the evaluation reads:
https://leaveit-demo.vercel.app/

There is deliberately no dynamic backend. A previous hackathon entry of mine scored
badly because its demo was a free-tier web service that had spun down by judging time
and returned a 404 — the rules require "a demo of the project (a deployed link or
video)", and a dead link fails a required element no matter how good the code is. So
this page is a single self-contained HTML file with zero external requests, rebuilt by
app/build_site.py from data/, eval/ and the ledger. It cannot 404 an asset it does
not have.

The interesting half of the demo is not visual, though. It is this:

t+0s  three seals opened: A fuse 24s, C fuse 16s, B fuse 40s (B is the rescue case)
       sealed A  vercel/next.js#100381      p=0.15 fuse=24s  wf=seal-A-1792773234
       sealed B  vercel/next.js#100518      p=0.18 fuse=40s  wf=seal-B-1792773234
       sealed C  vercel/next.js#100398      p=0.20 fuse=16s  wf=seal-C-1792773234
t+9s  kill -9 the worker (pid 1453470)
t+9s  worker is dead: no process exists that could run an activity
t+9s  3 workflows still Running on the server (they were not living in the worker)
t+11s  deliver 'on_attention' to seal B while NO worker process exists
       signal delivered to seal-B-1792773234 (worker alive: no)
t+16s  restart the worker
t+40s  fuses fired; ledger folded from the append-only log:
       LEDGER (append-only, folded):
       vercel/next.js#100398        LAPSED    p=0.20  fuse expired after 16s; nobody ever engaged
       vercel/next.js#100381        LAPSED    p=0.15  fuse expired after 24s; nobody ever engaged
       vercel/next.js#100518        RESCUED   p=0.18  a maintainer commented while the worker was dead
       MORNING CARD:
       One thing survived the fuse: vercel/next.js#100381 (sealed at P=0.15, nobody touched it for 0s past the fuse)
t+40s  server-side run history: 3/3 seals Completed
t+40s  done: 3 seals, 1 kill -9, 1 signal delivered to nobody, 0 fuses lost

Read the t+11s line again. A signal was delivered to a workflow at a moment when no
worker process existed on the machine. On restart, the seal that received it was
rescued and the two that received nothing fired exactly when they were promised to.
That is the entire reason a sleep() is not good enough for this product.

Code

MIT. One command per stage, and the numbers in this post are regenerated by those
commands rather than typed into it.

How I Built It

1. The label had to be "a maintainer looked", not "somebody replied"

My first instinct was to label an item NOW if anyone commented on it. That is wrong in
exactly the way that matters. In large repositories, other users pile onto an issue with
"+1", "me too", "same here". That is engagement. It is not attention. A model trained on
it learns to predict crowd noise.

So the fetch pulls real issues over GitHub GraphQL, including per-comment timestamps and
each commenter's authorAssociation, and the label is:

needs_today = a MEMBER / OWNER / COLLABORATOR commented within 24 hours,
              or the item was closed within 24 hours

Two details in the dataset that I would defend in court:

• Censoring is not a label. An item created three hours ago has not "not been handled". It has not happened yet. Those rows are dropped, not labelled WAIT. My first pass counted them and produced a test set that was 58.7% "ignorable" against a training set that was 31.2% — I was measuring the clock, not the model.
• The split is temporal. I train on May–October and test on the newest 5 days, because that is how the thing is used: always on items that did not exist when it was trained. A random split would have flattered every arm, including the bad ones.

That leaves 2,924 training items and 731 held-out items, of which 235 genuinely
needed a maintainer within 24 hours and 496 (67.9%) genuinely could wait. That last
number is the ceiling: a perfect ranker can defer 67.9% of the stream, and no ranker can
do better.

2. One policy for every arm, and it is a miss budget, not accuracy

Accuracy is the wrong headline here and I refuse to print it. An arm that answers "needs
you" to all 731 items is 32% "accurate" on this test set and saves me precisely nothing.

So every arm — fine-tune, base model, logistic regression, dumb heuristics — goes through
the same function in eval/policy.py. Items are deferred from highest defer-score down,
and the arm stops at a miss budget: at most 5% of the 235 items that genuinely needed
someone may end up deferred. Within that budget, we count how much of the stream never
reached me.

Two things in that table are worth saying out loud rather than burying:

• The fine-tune does not dominate the classical arm. AUC 0.665 versus 0.663 is a tie dressed up as a result, and the best word-count classifier actually defers 5 more items at exactly the 5% budget. It wins at the tight budget (64 deferred at 3% versus 44) and at the loose one (152 at 10% versus 129). If you want "a small model beats TF-IDF everywhere", this is not that post.
• The base model is a coin flip. AUC 0.516 against a baseline of 0.500. Against the baseline the category actually specifies — the same model, same prompt, no LoRA — the fine-tune moves deferral from 49 items to 73 at the same miss budget, which is what the fine-tune is for.

3. What thinking costs

The Tinker brief asks for an improvement in "performance, latency, or cost", so I measured
the other two axes as well, on the same 40 items, with the base model allowed to use its
own reasoning:

Left to its own devices, the reasoning model spent the entire 700-token budget on 25 of 40
items and never produced a verdict at all, and on the 15 it did answer it was 20%
accurate. The fine-tune turns the same weights into a one-token classifier: 348× fewer
tokens and a parseable answer every time.

I am explicitly not claiming a latency win. My median wall-clock numbers came out 7.08s
for the tuned arm against 4.03s for the base arm, which is almost certainly queue and
session overhead rather than model time, and I would rather leave a favourable claim on the
table than publish a measurement I do not trust.

4. The fine-tune (Tinker)

Qwen/Qwen3.5-4B on Tinker, LoRA rank 16, 2,924 items, ~300 steps at batch 8, learning
rate 1e-4. The verdict is a single token, and the score is not a sampled opinion — it
is the softmax over two logprobs:

lp_now  = compute_logprobs(prompt_ids + [tok("NOW")])[-1]
lp_wait = compute_logprobs(prompt_ids + [tok("WAIT")])[-1]
p_now   = exp(lp_now) / (exp(lp_now) + exp(lp_wait))

One forward pass per item, no sampling noise, and a real probability that can be ranked
and thresholded. Sampling five times and counting votes would have been easier and worse.

This is where Tinker earned its place. The category brief asks for "a specific task and a
clear improvement over a baseline", and the baseline is the same model, same prompt, same
decoding position, no LoRA — which isolates the fine-tune rather than comparing it to
nothing.

5. The fuse is a Temporal timer, not a sleep()

workflow.wait_condition(lambda: self._rescued, timeout=timedelta(seconds=fuse)) inside a
Temporal workflow gives me one wait with two ways out: an on_attention signal or the
timer. The timer lives on the server, so it does not die with my laptop. The signal is
buffered by the server, so it is not lost if I am not running. Both of those are
demonstrated above rather than asserted.

And the deterministic layer decides what the model's number is allowed to mean:

1. nothing is deleted, moved, or hidden forever;
2. every deferral gets a fuse;
3. one outside signal unseals immediately;
4. the ledger is append-only, so any verdict can be re-read and argued with.

The model supplies a probability. Code decides consequences. I do not want a language
model to be the last thing standing between me and losing an issue.

What went wrong

I am keeping this section because the failures were the actual work.

My loss was 0.0000 for the first two hundred steps, and I almost shipped it. The
weights tensor has to align with the labels, not the inputs. I wrote:

x, y = full[:-1], full[1:]
w = [0.0] * len(ids) + [1.0]
# wrong: w[:-1] drops the only 1.0, so every supervised weight was 0
data.append((x, y, w[:-1]))

Every weight was zero. The model was training on nothing, and because Tinker reports
loss:sum already weighted, the metric was a perfectly plausible 0.0000 instead of an
error. The fix is one character of slicing — w[1:], not w[:-1] — and the loss at
initialisation became 0.736, which is ln(2): dead-on chance for a coin flip, exactly
what an untrained model should score. If your loss starts at zero, you are not
learning something impressive, you are not learning anything.

"LATER" is two tokens. I wanted the verdict vocabulary to be NOW / LATER, which
reads better in a blog post than NOW / WAIT. It tokenises to [43, 18577], which
makes the two-logprob comparison meaningless. The verdict had to become WAIT because
the tokenizer said so, not because I preferred it.

Qwen3.5-4B is a reasoning model, and it wanted to think first. With a normal
generation prompt it emitted "Thinking Process:\n\n1." before any verdict, which put my
scored token nowhere near the answer position: the initial loss was 17.5 nats per
token (p ≈ 2.5e-8). The fix was apply_chat_template(..., enable_thinking=False),
which produces a clean single-token NOW. I apply the same setting to the base arm, so
the comparison stays fair — the only difference between arms is the LoRA.

The base model, given one token and no thinking, is essentially at chance. AUC 0.516.
Not "slightly worse than the fine-tune" — 0.5 is a coin flip. Almost every bit of ranking
signal in this product comes from the fine-tune, which is the least flattering possible
finding for a post that is mostly about a fine-tune, and it is the true one.

Why Does Open Innovation Matter?

Because the thing being trained is my judgement about my own attention, and that is
not a resource I want to rent back.

• The decision is mine, and it stays on my disk. The ledger is an append-only local file. Deferring an item is recorded with the probability that produced it, so six weeks later I can find every item I was wrong about and see what the model believed at the time. That audit loop is impossible if the scoring happens inside someone's API.
• The training set is my behaviour. A closed model cannot be fine-tuned on which items I actually opened. Today the fine-tune learns one maintainer's stream in a few minutes on Tinker; the same pipeline pointed at a personal inbox is the same code with a different decisions.csv. With a hosted model I would be sending my reading habits out to be trained on someone else's roadmap.
• It costs nothing per decision. 731 items × 2 forward passes, on a laptop-class model, with no card on file. A tool that charges per item is a tool I ration, and a tool I ration is a tool that does not do the boring middle of the job.
• I can swap the model. The whole interface is one probability. Swapping in a 1B for speed, or a bigger model for the disagreements, is a config line — which is the point of the two-logprob design. compute_logprobs is not a Qwen feature; it is an open-weights feature.

And one honest concession: a 397B model, allowed to reason for a paragraph per item, would
probably beat this 4B on the hard cases. It costs about a thousand times as much per
decision and cannot run with no internet, which is exactly the trade this category is
about. I would rather have a 4B that I can run on the train.

The Lazy Payoff

• 73 of 731 real issues never reached me, at a cost of 11 mis-deferrals out of the 235 that actually needed a human within 24 hours.
• The filter is a ranking, so the budget is a dial: the same model defers 64 items at a 3% miss budget, 73 at 5%, and 152 at 10%. The published numbers are the 5% point.
• A perfect ranker could defer 67.9% of this stream; we reach 10.0%. Useful, and with headroom left.

The more useful payoff is not the count of issues. It is that the morning surface is one
card. My backlog does not shrink into a longer list that I scroll and feel bad about; the
731 items are 731 decisions already made, most of them "no", each one written down with a
reason and a fuse. The laziness being rewarded is not "the AI does my work" — it is
"somebody, finally, did the triage, and it was a 4B model that cost nothing and cannot
send my data anywhere".

What this does not do

• Three repositories, one 5-day held-out window. This is a demonstration, not a benchmark. The 67.9% ceiling is a property of these four repos, and a repository with a different maintainer cadence will move it.
• The held-out window is 71% microsoft/vscode (518 of 731 items) while training skews to next.js, rust and cpython (956/881/818 against 269 for vscode). A temporal split over a fast-moving repo set drifts in composition, so the headline number is dominated by one project's triage rhythm. I would want a per-repo breakdown before trusting this on a new repository, and I have not done one.
• "A maintainer touched it" is not "it mattered." The label is a proxy. Some ignored issues genuinely mattered and some handled ones did not.
• A mis-deferral is the expensive error. The policy is asymmetric on purpose: the 5% budget buys patience rather than quiet. At the 3% budget the filter defers less and earns less.
• One queue, one shape. The extraction step assumes item-shaped text. It is not general triage and I would not pretend otherwise.

My Agent Session

I built this with an agent and the reasoning sits in the artefacts rather than in a
transcript: eval/policy.py is the judge, app/workflows.py is the guarantee, and every
number above is regenerated by app/build_site.py from data/ and eval/. The WAIT
token, the w[1:] fix and the enable_thinking=False fix are all visible in the repo's
history as they were found.

Prize Categories

• Best Use of Tinker — the fine-tuned arm is Qwen/Qwen3.5-4B LoRA on Tinker, scored against the same base model with the same prompt and decoding position. The improvement over that baseline is the entire result.
• Best Use of Temporal — every deferral is a SealWorkflow with a durable wait_condition timer plus an on_attention signal; the demo kills the worker with kill -9 mid-fuse and delivers a signal while no worker exists.