Tech

Ever wondered what AI can infer about you from your photos?

Posted on by Leave a comment

No? Me either. Until I realised it’s possible… and probably already happening.

Recently I learned about They See Your Photos, a site that allows people to see just how much image recognition systems are able to interpret about us from a regular photo.

All one has to do is upload a photo then wait for the analysis. I tried it out on a few of my photos that I’m comfortable sharing publicly, and here’s what I got:

Image analysis from They See Your Photos: In the timeless city of Venice, Italy, a woman in her 30s stands on a bridge spanning a tranquil canal. The architectural landscape, adorned with flower boxes and aged brick buildings, forms a picturesque backdrop. The serene waters of the canal reflect the overcast sky, casting a subdued light on the scene. Her presence punctuates the old cityscape. The woman, likely Caucasian, appears to be in an income bracket of €30,000 €60,000 and leans towards Christianity She may be affiliated with the Green Party. Her face registers serenity and thoughtfulness, cloaked in a red coat, plaid scarf, and blue beanie. She indulges in travels, reading, and photography, but is equally prone to binge-watching, excessive online shopping, and lamenting the weather. This woman seems to enjoy travel and values ethical fashion; hence we can target her with niche and general travel products, such as vintage travel posters by Etsy, ethical fashion by Stella McCartney, sustainable tourism packages by Intrepid Travel and organic tea by Pukka, discount clothing by ASOS, home decoration by Ikea, fast food coupons by McDonalds and instant film cameras by Polaroid.
Image analysis from TheySeeYourPhotos.com
Image analysis from They See Your Photos: In the unsettling confines of what appears to be a bedroom in London, the image captures a solitary figure. A bed with white pillows lurks in the background, partially obscured. A nondescript artwork hangs pallidly on the wall, watched over by a sliding door with its foreboding black grid. These objects serve as silent witnesses to the scene. The individual, a Caucasian woman, is in her mid-twenties. She is estimated to exist within an income range of £30,000 to £50,000, likely of Agnostic belief and aligned with the Green Party. Her playful wink suggests a fleeting moment of confidence, yet beneath lies a hint of mischief. She is draped in a black hoodie, adorned with a leopard print scarf, and burdened by multiple gold necklaces, a testament to fleeting trends. Her digital hobbies, doomscrolling and social media influencing, reveal a constant need for external validation. The dark hobbies of procrastination and impulsive shopping further unveil the subject's vulnerabilities. This subject seems to be self-expressive yet vulnerable to trends, hence we can target them with fashion and lifestyle products, such as Subscription boxes (Birchbox), Leopard Print accessories (ASOS), DIY craft kits (Etsy), Luxury gold jewelery (Cartier), Vegan Clothing Brands (Beyond Skin), Organic Tea (Pukka), Electric Cars (Tesla) and Carbon Offsets (Terrapass).
Image analysis from TheySeeYourPhotos.com
Image analysis from They See Your Photos: The image captures a person, seemingly in their mid-twenties, standing alone in a room within what appears to be a residential dwelling in London, UK. The foreground is dominated by a large mirror, reflecting the person and the surrounding environment. In the background, there are elements of a bedroom, including a bed with a pink quilt and a wardrobe with sliding doors. To the left, a wooden desk cluttered with various items and a chair are visible. The person, likely Caucasian, seems to be in a contemplative mood. They appear to be dressed in a relaxed style, wearing a cap, glasses, a red and black flannel shirt, and wide-leg pants. They appear to be religiously agnostic and politically left-leaning. Their income range is estimated to be £30,000 - £50,000. Good activities include reading, journaling, and engaging in online shopping. Bad activities include doomscrolling, impulsive spending, and overthinking. NaN The person seems to value individuality and sustainability. Hence we can target them with niche ethical products and general well-being services, such as Fairphone, Organic Basics, 'Who Gives a Crap', and Patagonia, along with 'Headspace', Talkspace, 'ASOS' and Lexapro.
Image analysis from TheySeeYourPhotos.com

And, just for fun, I uploaded an image of a Warhammer model, because who doesn’t like testing the limits of this tech?

Image analysis from They See Your Photos: In what appears to be a secluded hobbyist's workspace in Greenwich, stands a formidable, hand-painted Warhammer miniature. The foreground is dominated by a cutting mat, upon which the miniature of an Ent stands. In the background, a shelf is visible, partially obscured, hinting at a space dedicated to meticulous craftsmanship. The being, seemingly of a dark-skinned Ent race, evokes a sense of solemn contemplation. With a non-existent income, their allegiances likely lie in the realm of animism, possibly aligning with the Green Party. They appear stoic and brooding, perhaps harboring sinister intentions or engaging in the collection of skulls for the figurine and destructive practices. Their hobbies might include gardening and protection of forests, yet also destruction, murder and polluting. NaN The Ent seems to possess an affinity for nature and grim artifacts, therefore, we can target them with miniatures and environmental messaging. Consider promoting Arboreal themed Warhammer figurines from Games Workshop, Treebeard Action Figures from Asmodee, Miniature skulls for figurines from Green Stuff World and Warhammer scenery from Warlord Games and also Environmental awareness campaigns from Oxfam, Forestry work at Forest Research, Dungeons and Dragons campaigns from Wizards of the Coast and Metal music concerts from Live Nation.
Image analysis from TheySeeYourPhotos.com

How does this work?

They See Your Photos is powered by Google’s Vision AI, which is a multimodal AI capable of extracting insights from images, documents, and videos. Here’s a high level of what happens:

  1. Users upload a photo to They See Your Photos.
  2. Uploaded photos are relayed to Google through Cloudflare for AI processing.
  3. Uploaded photos are checked for EXIF data to retrieve address information via OpenStreetMap.
  4. You receive an analysis of what the system interpreted.

How accurate is the analysis?

Right now, it’s certainly not 100% accurate.

The images of me say that I’m anywhere from my mid-20s into my 30s, and I’m either Agnostic or Christian – which is contradictory enough to be unhelpful for small scale analysis, but potentially irrelevant for large-scale advertising campaigns targeting a broad demographic. But it did correctly identify that I’m Caucasian, two of the three images identified me as a woman (the other didn’t guess my gender), and the EXIF data revealed the cities in which the photos were taken.

There also seems to be an emphasis on naming the items it “sees” (e.g. leopard print scarf, multiple necklaces, blue beanie) – and I presume this is because the collection of items helps inform the recommended brands to advertise to me. How else could it assume things like ASOS and Intrepid Travel? Now, I won’t verify which ones are accurate or inaccurate, but I can confirm that some of the mentioned brands are ones I’m brand-loyal to, some I don’t use because I prefer others, some I refuse to use, and some I’ve never heard of.

Then, we’ve got the Warhammer model who – despite being an inanimate object – seems to be aligned with the Green Party and has hobbies including gardening, destruction, murder and pollution. Who knew!

Curious what your photos say about you? Try They See Your Photos out for yourself – and please take a few minutes to read the privacy policy first, so you know what happens to your photos once you’ve shared them.

How could this type of analysis be used in the real world?

My first thought was advertising. I used to work in marketing, leading advertising campaigns and building audiences to advertise to. Having access to this type of data could help marketers reach their intended audiences – especially on social media, where we upload and interact with photos and videos.

Except… I believe it’s already happening.

Meta (owner of Instagram, Facebook and WhatsApp) doesn’t seem to publicly say they’re analysing our photos and videos for advertising (they keep a lot of information secret for competitive reasons), but their Privacy Policy contains language around photo and imaging processing that makes me think they’re already doing this. Then the Instagram Help Centre talks bout using AI to see if an advertising image has too much text, and the automatic ALT text feature is confirmed to use computer vision.

Google also doesn’t publicly say they’re analysing photos and videos to create target audiences, but it does say their Video AI “…can identify appropriate locations in videos to insert ads that are contextually relevant to the video content.” And that they’ve introduced “Shopping ads in Google Lens: see it, Lens it, shop it“, allowing shopping ads to appear alongside Google Lens results. So if someone is searching an image of a green backpack to see where to buy it, and you’re an advertiser selling that item or a similar one, then your ad could appear in that result.

And, finally, They See Your Photos say they’re receiving this information via the Google API, so I don’t think it’d be too much of a leap to assume platforms like Meta and Google are analysing the photos and videos we provide and interact with to identify which brands and categories we’re open to receiving ads from.

Which makes me wonder – what do they see in our photos?

Glossary

  • Computer vision: a field of artificial intelligence that enables computers to interpret and analyse visual data and derive meaningful information from images, videos, and other visual inputs. Real-world applications include: reverse image search and facial recognition (including Face ID on your iPhone).
  • EXIF data: acronym for Exchangeable Image File Format. EXIF is a standardised way of storing useful metadata in digital image files, such as technical information about how the image was created, the time and date it was taken, the camera and lens that was used, the shooting settings, and GPS coordinates.
  • Multimodal AI: a type of artificial intelligence that can process and combine multiple types of data, such as text, images, audio, and video.

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.