I Built a YouTube Credibility Scanner. Its Worst Bug Was Accusing Honest Creators

A while ago a creator replied to a TruthScore report on his video. His message was short: the tool had flagged his links as an affiliate funnel. They were links to his own business.

He was right, and I had shipped that bug to real people.

What the tool does

TruthScore takes a YouTube URL and scores how trustworthy the video looks. It combines:

• Engagement patterns (likes, comments, ratio to views)
• Estimated dislikes, since YouTube hides the real count
• Comment sentiment
• A channel trust score
• Manipulative language in the title and description
• Affiliate and funnel detection in description links

Stack:JavaScript, HTML, CSS.

The bug

My first affiliate detector treated every outbound link in a description as one signal. Tracking parameters, shortened URLs, and links to a creator’s own site all fed the same “funnel” score.

That’s wrong for an obvious reason: a creator linking to their own product is not the same as one stacking five referral links.

The fix: hard vs. soft signals

I split detection into two tiers:

Hard signals: known affiliate network patterns, referral parameters, discount-code formats
Soft signals: shorteners, redirects, link density

Here’s the idea in simplified form:
funnel_score = 0

if hard_signals_present:
funnel_score += weight(hard_signals)
funnel_score += small_boost(soft_signals)
else:
funnel_score += 0 # soft signals alone never flag a funnel

Soft signals can strengthen a case but can’t make one. A link to someone’s own site no longer counts against them.

I made two other changes in the same rebuild. Comment sentiment is now counted per comment, so one angry thread doesn’t wipe out a mostly positive section. And I raised the score baseline, so a video isn’t treated as suspect before the scan finds anything.

What happened when I ran it on 20 videos

After the fix, I ran the scanner on 20 new “make money with AI” videos, mostly dropshipping, digital products, and AI side hustles.

• 15 came back “Be Careful”
• 5 came back “Likely Legit”

The pattern was boringly consistent. The flagged videos usually had a specific income number in the title (“$2,000 in 24 hours”), a “free” course or call in the description, and an application form that never showed a price.

The five that came back clean were mostly founders showing case studies or selling software, not courses. Their income comes from a product, not from the next funnel step.

This is not a random sample. I picked new uploads in one hype-heavy niche. Don’t read it as “75% of YouTube is fake.” Read it as “this is what the scanner found in the videos I chose.”

What it still gets wrong

Two things bother me.

First, one video came back “Be Careful” even though the web research found no scam reports, no refund complaints, and no suspicious links. A warning label with zero evidence behind it is a false alarm, and I’m still working out why the score landed there.

Second, YouTube hides dislike counts, so the tool works from an estimate. On several videos that estimate came back as 0.0%, which obviously isn’t real. It’s a weak signal, and I’m wary of letting it carry weight.

Scoring other people’s credibility with code means you will be wrong sometimes, in public, about real people. The only defense I’ve found is to say where you’re wrong before someone else does.

What I’d tell anyone building a “trust score”

1. A single noisy signal is worse than no signal. The affiliate bug came from treating ambiguous evidence as proof.
2. Absence of evidence shouldn’t look like evidence. If the research finds nothing, the report should say so, not hand down a warning.
3. Let your first public mistake be small. I found my bug because a creator told me. I’d rather hear it early from one person than late from many.

Try it, break it

Paste any YouTube link into truthscore.online. If the report looks wrong to you, send me the link. Every mistake I can reproduce makes the scoring better.

And a question for the developers here: how would you tell a creator’s own links from affiliate links without maintaining a giant blocklist?