I ran 6 free image compressors through 13 tricky images and scored every result

"Save as JPEG, quality 75" is the default advice. I saved five images at quality 75 with ImageMagick and scored them with a perceptual metric. They landed anywhere from 71 to 82. Same setting, very different results, and the setting itself tells you nothing about which image got which.

So I wanted to know what the popular free compressors actually do to images. Not the file size they show you, but how much of the image is left.

I'm biased: I built one of the tools in this test (PicoDiet). I'll show where it loses too.

How to score "looks the same"

File size is easy to compare. Quality is the hard part, and eyeballing doesn't scale to dozens of files.

I used two open metrics from libjxl:

• SSIMULACRA2, which compares a compressed image with the original and gives a score up to 100. It was calibrated against how people rated compressed images. Roughly:
  • 90+: no difference even when flipping between the two at full size
  • 80: no difference side by side
  • 70: artifacts you'd notice only with the original next to it
  • 30 and below: obvious blocks, blur and banding
• Butteraugli, which reports the worst visible difference anywhere in the image. An average score can hide one smeared face or a line of blurry text. Butteraugli doesn't.

Both are command line tools (ssimulacra2 and butteraugli_main), so you can check your own images:

ssimulacra2 original.png compressed.jpg
butteraugli_main original.png compressed.jpg

One more thing: compare colour-managed pixels. An iPhone photo is in Display P3. If a tool converts it badly, the colours shift, and you want the score to catch that.

The test

13 images: 11 public-domain photos and graphics from Wikimedia Commons plus 2 screenshots. Photos, an iPhone portrait, an iPhone HEIC, transparent PNGs (a logo, a cut-out, a 3D icon), an illustration, screenshots and two animated GIFs.

Tools, free web versions, default settings, one run each: TinyPNG, Squoosh, iLoveIMG, compressjpeg.com and its sister sites (compresspng, compressgif and imagecompressor.com gave identical files), optimizepng.com and ezgif. Plus PicoDiet. Every flaw below I also checked by eye.

What broke

This was the part I didn't expect. Five of the six tools broke at least one image.

Squoosh turned transparency black. Its default output is JPEG, and JPEG has no alpha channel, so the transparent background of the 3D icon became a black square. It also kept only one frame of the animated GIF and couldn't open the HEIC.

compresspng.com made the gradients band. It reduced the 3D icon to too few colours, and the smooth pink shading broke into stripes. Score: 52.

iLoveIMG made an iPhone photo dull. It mislabelled the photo's colour profile, so a bright red jacket came out duller and browner. It also put streaks into the animated GIF.

The rest, shortly:

• TinyPNG was the strongest of the others. Nothing badly broken, but it rejects GIF and HEIC, converts iPhone photos to sRGB (you lose the extra-vivid reds and greens on Apple screens), and left some grain in gradients that you see up close.
• ezgif streaked the 3D animation (score 51).
• optimizepng left speckles on an app screenshot.

Default settings: smaller files, less image

At their defaults, every tool aims at a different quality, and none of them tells you which. On photos they scored between 63 and 76. PicoDiet aims at 80 by default, so its files were bigger:

(size · SSIMULACRA2 score)

If you just want the smallest number on the screen, Squoosh and TinyPNG win this table. They get there by throwing more away.

The fair comparison: same quality, then compare size

To compare file sizes fairly, both files have to look equally good. So for each image I took each tool's own score and set PicoDiet to that exact score, then compared bytes:

On PNG, iLoveIMG and compress*.com came out a bit smaller. That happened on images where they lost more detail in one spot, like the flower edges and the app screenshot, than PicoDiet's worst-spot check allows. Their average score matched, but one region didn't.

Where PicoDiet lost outright:

• One GIF. On the retro animation, ezgif and iLoveIMG saved 41-42% at scores of 84-85. PicoDiet couldn't find a smaller GIF that kept 80 in every frame, so it gave back the original.
• Speed. Squoosh runs in your browser and took 1-1.5 s per photo. PicoDiet took 1.5-7.5 s per image, because it tries several encoder settings and scores each one.

What I took from it

• A quality number isn't a quality. "75" in one tool and "75" in another, or on two different images, can look very different. If quality matters, measure the output, don't trust the setting.
• Check the boring cases. Transparency, animation, iPhone colours and HEIC are where tools break, not on the regular photo you test with.
• "X% smaller" means nothing without the quality next to it. Most of the size differences at default settings were just different quality targets.

The full tables, all 13 images and the crops are here: picodiet.com/image-compression-benchmark. Every tool ran at its defaults, so you can repeat any of it with ssimulacra2 and butteraugli_main.

If you've seen a compressor do something weird to your images, I'd like to hear which one and on what kind of image.