Visual AI Blog A Brief History of Automatic Photo Tagging, Pt 2 Ofer Fryman June 18, 2018 7 Min Read In part two of A Brief History of Automatic Photo Tagging, we’re exploring automatic photo tagging of today and tomorrow in an area that early forays seemed to ignore: ecommerce. The early stages of consumers interacting with automatic photo tagging and photo recognition features seemed underwhelming. An incredible piece of technology was put in play, but the real experiences with it were nowhere near as spectacular or as dystopian as they seemed. This was about to change. Where companies before focused on helping people tag and sort their own personal photos for their private use, they would now shift their focus to a more commercial use. And these shifts are just beginning. Companies are seeing early success with automatic photo tagging and associated technologies — and consumers are excited about this tech again — but many possibilities haven’t made it to market. Yet. Automatic photo tagging makes a move In 2015, the most common uses of automatic photo tagging were personal photos, as on Facebook and personal photo collections. As 2016 and 2017 unfolded, these features became more prominent. Facebook rebranded their automatic tagging feature as a way not just to remove the hassle of individually tagging photos in a mass upload but also to manage a user’s identity. “Powered by the same technology we’ve used to suggest friends you may want to tag in photos or videos, these new [automatic tagging] features help you find photos that you’re not tagged in and help you detect when others might be attempting to use your image as their profile picture,” they wrote in a 2017 update. A photo accompanying Facebook’s 2017 report shows that users can now be notified of pictures of them uploaded by other users. Google, now deep into Google Photos, began offering more types of auto-tagging and auto-sorting as well. By 2017, you could sort not just by date or the place the photo was taken, but by features as nebulous as “skylines”: [Source] They also allowed you to group by faces but, unlike Facebook, Google Photo users manually controlled who was automatically tagged and under what title. This was perhaps a nod to their early failures at tagging where they mistook black people for gorillas, which obviously called for changes. But it also could have been in response to user outrage at Flicker and subsequent research at Cornell that showed that users have a definitive preference over which tags are used for what photos, and what order those tags are presented in. Giving users more power over the naming, ordering and categorization spoke directly to users’ desires to better organize their photos. This is an easy example that shows the widespread commercial uses for automatic photo tagging and facial recognition. In an era where more and more people have years — sometimes even decades — of photos online, keeping track of them and managing how they’re used would be a challenging task for users from the ’00s on. Automatic photo tagging was and will be a critical tool for finding, gathering and sorting your digital likeness for the foreseeable future. Personal photo organization wasn’t just limited to social providers either. Apple included variations on this theme in their iOS 10, grouping photos by who appears in them: [Source] This is a move that keeps Apple’s internal photo ecosystem on par with competitors and encourages users to keep their photos exclusively within the Apple sphere. These updates showcased that more sophisticated technology was ready to be brought consumer-side. That’s important given the size and growth of the photo-sharing industry. In 2016, Deloitte reported that there was a 15% increase in photos shared or stored online, bringing with it a 20% greater network impact. The proliferation of (cheap) digital cameras, photo-sharing sites and social connection through photos, they say, led to users sharing, in one year, 31X the photos taken in an average year of the ’90s. Personal photo tagging and storage have gotten a pretty big boost since 2015. But it’s another space that’s had arguably more movement: ecommerce. More than just a pretty face 2016 and 2017 were big years for automatic photo tagging for ecommerce use as well. A paper out of the Indian Institute of Technology in January of 2016 bluntly stated that it was one of the biggest use cases that automatic photo tagging could help: “The explosive growth of e-commerce products being sold online has made manual annotation infeasible. Without such [automatic] tags it’s impossible for customers to be able to find these products. Hence a scalable approach catering to such large number of product images and allocating meaningful tags is essential and could be used to make an efficient tag based product retrieval system.” Categorizing items based on tags generated from images alone had arrived. The same year, Pinterest rolled out automatic object detection to their visual search after launching a visual search in 2015. Pinterest’s visual search was a huge hit — no surprise, given the visual nature of the platform — and the automatic tagging of items was designed to complement it. [Source] “We wanted to make it as simple as possible to start a discovery experience from any [visual search],” they wrote in their announcement. “…[A]utomatic object detection makes visual search a more seamless experience…” This sentiment is echoed at more traditional ecommerce retailers, like intu, a UK retailer. “It is a new way for consumers to discover products,” says Karen Harris, Managing Director of intu Digital. “We wanted to create better discoverability on our site for our consumers, because we have 500+ retailers integrated with us and over 4 million products.” Deep tagging for ecommerce has started to take over the digital commerce space in the last few years. And even where it isn’t directly implemented, we’re seeing that consumers want to be able to interact with images to get to where they want to go more quickly. Take Instagram’s “shoppable posts.” Users see a picture with a tag of an item that gives more information and makes for easy purchasing: [Source] This is the same idea as using automatic tagging to create shoppability on Pinterest or even on content sites, like Fashion Lover: [Source] Consumers aren’t the only ones who benefit from this trend. Automatic photo tagging is also a great tool for retailers, as it allows them to more easily categorize items for customers without spending the resources to manually tag items. This makes consumer discovery easier and will continue to create a better shopping experience as the AI gets better. Imagining the horizon The development of automatic photo tagging has been steady but somewhat slow. Don’t expect that to last much longer. Retailers, especially in the ecommerce space, are now releasing features that have fleshed out automatic photo tagging capabilities or other variants, like reverse-image search. The truth is, there’s a lot more going on today. Especially for deep learning services, automatic tagging is getting very good and is ready to be deployed. Deep tagging can pull fabric type, colors, faces, objects, texts, species and composition; it can suggest keywords and localize objects. In some sense, this is still raw power that has yet to be focused for consumers. There’s a lot on the table that hasn’t been shaped into features for ecommerce consumers or even widespread back-end features for retailers. But it’s coming up, and fast. We’ve already seen companies pushing out tools, features and internal processes that rely on sophisticated image technology. Whether it’s saving time internally by categorizing items or allowing customers to take furniture measurements via an app, we’re finally on the edge of visual technology revolutionizing the way consumers shop.