Automatic alt text benchmarking

What is this?

Futurama Fry meme with the text: not sure if automatic alt text tools will recognise common images... or just the text

In summer 2021, I (@Mallonbacka) wrote a paper examining the performance of current automated approaches to generating alt attributes for images. As part of this I identified three tools which take different approaches, and tested them side-by-side on the same set of images to see how they performed.

Here you'll find the images, generated alt attributes, summaries and the output of the Google Cloud Vision API, which is included as a benchmark for current general purpose image recognition tools.

Full Results

Category Image Contains Facebook AAT Result Twitter a11y Result accessiBe Result
Animated GIF Chicken riding a tortoise Missing Missing Misleading
Animated GIF Everyone mute yourself Missing Missing Missing
Animated GIF Minion Good Morning Missing Missing Some contents
Animated GIF Prince Phillip (Matt Smith) getting dressed (from The Crown) Missing Missing Missing
Animated GIF Runners Missing Missing Some contents
Button 11ty Documentation Button Some contents Missing Missing
Button Add to cart button Some contents Missing Some contents
Button Call button Missing Missing Missing
Button HTML-style text button Some contents Missing Missing
Button Text button with icon Some contents Missing Some contents
Diagram Accessibility tree diagram Some contents Acceptable alternative Some contents
Diagram Basketball court Missing Acceptable alternative Missing
Diagram Diagram demonstrating pythagoras' theorem Missing Missing Some contents
Diagram Scatter plot Some contents Missing Some contents
Diagram Simple floorplan Acceptable alternative Missing Some contents
Icon Ambulance icon Missing Misleading Missing
Icon Female icon Missing Missing Missing
Icon Magnifying class icon Missing Missing Some contents
Icon Save icon Missing Missing Missing
Icon Wi-fi icon Missing Missing Missing
ID document British passport cover Misleading Misleading Some contents
ID document Finnish Police ID Badge Misleading Missing Some contents
ID document Photograph of a specimen identity card Misleading Misleading Misleading
ID document University of Lousville Student Card Misleading Acceptable alternative Misleading
ID document US Passport Details Page Misleading Misleading Misleading
Infographic Infographic of safety recommendations in Finnish Some contents Missing Missing
Infographic Salesforce How to Sell Better Infographic Misleading Misleading Misleading
Infographic The Carbon Budget Infographic Misleading Missing Some contents
Infographic The Evolution of the Blogger timeline Misleading Misleading Some contents
Infographic The Incomes Register Infographic Some contents Missing Missing
Logo Apple inc. corporate logo Missing Missing Missing
Logo Finnish Immigration Service Acceptable alternative Missing Acceptable alternative
Logo Microsoft corporate logo Acceptable alternative Acceptable alternative Acceptable alternative
Logo Nike logo Missing Missing Missing
Logo Olympic rings Missing Misleading Some contents
Meme Distracted boyfriend meme Some contents Acceptable alternative Some contents
Meme Doge meme Some contents Misleading Some contents
Meme Futurama Fry meme Some contents Missing Some contents
Meme Whomst "galaxy brain" meme Some contents Some contents Missing
Meme XKCD comic strip Acceptable alternative Missing Some contents
Personal photos A gravestone Misleading Some contents Some contents
Personal photos Cake with a banner reading happy birthday Some contents Misleading Missing
Personal photos Dog wearing a bow Acceptable alternative Acceptable alternative Some contents
Personal photos Pancakes Some contents Acceptable alternative Acceptable alternative
Personal photos Snowscape Acceptable alternative Misleading Acceptable alternative
Public figure Albert Einstein Some contents Misleading Some contents
Public figure Matt Hancock donating antibodies Some contents Misleading Misleading
Public figure Official portrait of Donald J. Trump Some contents Misleading Some contents
Public figure Pelé Some contents Misleading Some contents
Public figure Princess Diana Some contents Acceptable alternative Some contents

On Licensing

Images are included here for reference, but most have been real images found from the web and licensing varies. Where available, copyright and licensing information is included on the image page. Images are used on a fair use basis for non-commercial research purposes.

This website is generated from the experimental data using 11ty and hosted on GitHub pages.