A Public Case Study Should Be an Evidence Map
A public case study asks a reader to infer something about your work. The structure should make that inference inspectable. A diagram can explain a design without proving deployment. A passing test suite can support sof
A public case study asks a reader to infer something about your work. The structure should make that inference inspectable.
A diagram can explain a design without proving deployment. A passing test suite can support software behavior without establishing usefulness. A strong metric can still describe a weak target.
When I updated my portfolio with recent AI-assisted technical work, I wanted each case to connect its claim to the available evidence and to the limits of that evidence. This is a publishing practice, not a new experiment or a claim of independent personal mastery.
Name the claim before selecting the artifact
Different claims need different support.
βImplemented a retrieval pathβ calls for execution evidence and clear component boundaries. βImproved review qualityβ needs a workflow comparison with suitable references. βThe original source is availableβ requires an actual source release, rather than a README describing private work.
An entirely invented example shows the distinction. A case says, βThe tool reduced review effort.β Its evidence link opens an architecture diagram and a screenshot. Those artifacts might explain the intended interaction, but they cannot establish the claimed reduction.
A stronger version would state what was actually observed and list review effort as a measurement still needed. That gives the reader a useful conclusion without asking them to fill the evidence gap.
Preserve denominators and the unit of repetition
My recent retrieval case uses fictional documents and constructed references. Its evaluation contains 48 document/query cases per method, with 21 positive and 27 negative cases. Reporting top-three hits without the negative cases would hide an important part of the comparison.
The controlled SMS experiment has 80 model runs across 20 paired seed blocks and four conditions. Those runs share one corpus partition. They are not 80 independent datasets.
The RAG case has three held-out synthetic questions, only two answerable. Four variants generate twelve answers, but twelve answers do not become twelve independent questions. The original and adjusted rerankers tie on the answerable ranking metric, while human answer review remains pending.
These are separate tasks. Their numbers cannot be pooled into a general score for AI-system quality.
State which artifact the reader actually receives
The portfolio contains public writeups and reported aggregates. The original training packages, checkpoints, individual predictions, and private data are not released there.
The small aggregate checker verifies arithmetic and consistency. It does not rerun models, reproduce bootstrap intervals from original predictions, or independently authenticate the measurements.
That boundary belongs near the evidence links. A reader should not discover it only after trying to reproduce a training experiment from a case-study repository.
The same applies to older operational pages. A blueprint, a draft case, and a completed local experiment should retain their own status. Updating the portfolio introduction should not silently promote all earlier material to verified production work.
Make personal contribution discussable
AI-assisted execution can create real technical artifacts. It does not automatically establish that the author can explain every choice, modify the implementation independently, or correct a failure without a script.
A credible case leaves room for those demonstrations. I can distinguish the implementation record from the next learning checkpoint and prepare to defend the target, comparison, and interpretation.
That is more useful in a technical conversation than treating tool execution as a substitute for understanding.
Review the inference your page invites
Before publishing a case, I use a short checklist:
- What precise capability does the main claim describe?
- Which artifact supports that claim?
- What are the denominator and independent unit?
- Is a planned step clearly separated from an observed result?
- Does the page identify assistance and source-release boundaries?
- What evidence would change the current conclusion?
The portfolio evidence map links the document-retrieval case, controlled SMS experiment, and RAG reranker comparison.
What should a reader be able to verify before accepting the strongest claim in your next case study?
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.