I ran an SEO experiment on my side project and broke my own control group

In September I had a weird problem with my side project, a tarot card reference in four languages: https://tarot-magic.com. The Ukrainian card pages sat at positions 6-9 in Google and got almost no clicks. Over three months they had 2,614 impressions and 2 clicks.

On September 21 I rewrote all 78 Ukrainian meta descriptions. The old ones were keyword dumps cut off mid-sentence. The new ones are one plain sentence about what the page actually covers. I left the other three languages alone as a control and promised myself to wait two weeks before looking.

Two weeks are up. Here is what happened, including the part where I ruined the experiment myself.

The numbers

I compared two equal 14-day windows in Search Console, filtered to /uk/card/.

Impressions doubled while average position got worse. That looks contradictory until you check the page list: Google started showing pages that had no impressions before, and they enter at positions 20-35. They drag the average down even though part of the existing pages held their positions.

If I keep only pages averaging position 10 or better, I get 157 impressions and 2 clicks, a CTR of 1.3%. The previous window under the same filter had 197 impressions and zero clicks.

1.3% is roughly what you would expect at positions 6-8 on mobile. But it is two clicks. I am not going to claim anything from two clicks.

How I broke it

Before starting I wrote down one rule: compare CTR only if positions stay stable, because CTR at position 7 and at position 30 are different metrics.

Positions did not stay stable. The single biggest Ukrainian page in the top 10, "Six of Wands", went from 7.4 to 37.5 in exactly this window.

Worse, on the same day the descriptions changed, an FAQ block was added to all 312 card pages in all four languages. It started as a six-page pilot and got rolled out everywhere at once. So the "control" languages were not a control anymore. Russian card pages dropped from 379 impressions to 133 in the same period, and I have no way to tell whether that was the FAQ, normal ranking churn for a young domain, or something else.

Lesson for next time: a control group must not be touched by anything, including changes that look harmless.

The accidental winner

The best result came from a page that was not part of the experiment. On September 20 the French "card of the day" page got the main local query, "Tarot du jour", added to its title. Over the next two weeks it got 18 impressions and 5 clicks. That is half of all clicks on the site for the period.

My current guess is that the title decides whether someone looks at the result at all, and the description only matters after that. That is the next thing I want to test, with a control group I will not touch this time.

Bonus: Yandex

Yandex had excluded 76 of 88 Russian pages as "low value". On October 1 I sent all Russian URLs for recrawl through Yandex Webmaster. Pages in search: 12 on Oct 1, 21 on Oct 2, 30 on Oct 5. No new links, no analytics changes, just a fresh visit from the crawler. The same update on Oct 5 dropped 20 English card pages with the same "low value" status, all last crawled around Sep 21-23. So if you work with Yandex, recrawl requests seem to matter more than I expected.

Next check is mid-October. If anyone has run title A/B tests on a small site, I would love to hear how many impressions you needed before the difference was real.