Geotagging – one day later

The First Day
When we were doing our projections for how many photos Flickr members would geotag, we though that we’d hit Spiral Jetty a million in the first month, maybe even as fast as two weeks. Instead, 24 hours in, there were 1,234,384 geotagged photos (and now more than 1.6 million geotagged photos as I write this, about 9 hours later). Crazy!

(On the left is a nice example of geotagging in a magic spot – the Robert Smithson land art project, Spiral Jetty on Utah’s Great Salt Lake. See whileseated‘s photos around Spiral Jetty.)

The Tech Behind the Scenes

One of the "little" things that was incredibly complex technically was the integration of location-based searching into our existing tag and text-based search technology. That means you can do things like search for photos matching "food" in southern Asia or architecture in South America. [We’ve opened up comments on this thread: please post some of your favorite searches!] This is already pretty awesome after one day — imagine a year from now. Or ten years from now. Or one hundred :)

But the reason I say it’s complex is that marrying "traditional" search with spatial search in a real-time context is extremely hard, especially at our volumes and rate of growth. More than 228,000,000 photos have been uploaded, with over a million new photos being added on a good day. There are billions of bits of data that go into the search (more than half a billion tags alone), along with privacy controls, group membership, and so on.

This is one of the largest real-time search indexes in the world. In contrast, nearly all web search is done in a "batch" mode with periodic updates, while nearly all real time search is done on a small set of items whcih "expire" after a short period. But new or updated Flickr photos are typically searchable in under a minute.

Luckily, we were able to build on the amazing work of Yahoo!’s Search team. In addition to directly recruiting some of the smartest search scientists and engineers in the world, Yahoo! has rolled up a number of all-star teams through acquisitions over the years, including AltaVista, Inktomi, AlltheWeb, Overture and FAST. We get a lot of support from our brainiac brothers and sisters in the Yahoo! Search Group.

In particular, VESPA (the codename for an internal search technology that we use) enables both real-time indexing at our scale and the ability to retreive photos in any corner of the globe with any tag or text, belonging to any user or group … and it does so fast. I’d be remiss without a special shout-out to the VESPA team: thank you!

Over time, we’ll be able to apply more and more cool technology coming out of the Yahoo! Search group, along with amazing too-hot-for-prime-time projects from Yahoo! Research. (And that’s going to mean a lot to you in 2012 when you have 150,000 or whatever photos on Flickr and are reminiscing back to the days before the invention of direct retinal capture technology.) I guess the acquisition worked out ;)

The Maps Themselves

There’s been a lot of grumbling about the maps detail and coverage of Yahoo! maps outside the U.S. We definitely hear it, and the Yahoo! Local/Maps team, which happens to sit across from the Flickr team hears it (and feels it!) … loud and clear. The maps in much of the world are not good enough yet. We agree.

But, fear not: maps are continually being upgraded and street-level detail for more countries has been in the works for a while already. Satellite imagery upgrades already happen a couple of times each month (the Earth is big, y’know) and in the months since the new Maps beta launched, they’ve already made significant improvements. More are coming.

Right now we got what we got, but the platform underlying it will automatically pick up on improvements as they are made (and in the meantime, one point in the Maps team’s favor is having good labels for countries around the world: check out themexican’s geotagged photos on a map. His travels around western China and central Asia are one of my favorite things captured on Flickr, and seeing the place names makes it all a little more real.

But yes: more, better, soon!

Geo APIs released

Finally, for the hackers reading this, today we’re also releasing extensions to Flickr’s API to enable adding and retreiving geo information, setting privacy permissions, and searching by location: everything you need to roll your own. The docs are under the link above (scroll to photos.geo) and the best place to dig in as on the developers’ mailing list (see today’s announcement).

This also means: "hey, if our maps don’t work for you, use whatever maps you’d like!" We expect the momentum of third party geo-based Flickr "mashups" to continue and hopefully this will let developers take it to the next level since they don’t have to do their own spatial indexing any more, can enable private geo-data  and get the new "accuracy" and search-within-locations features thrown in. Go for it!

And it’s worth a little more props for the Search group here: we expected to have to do a few weeks of stress-testing before releasing the API since we’re going a few steps beyond the cutting edge, but even with the unexpectedly strong uptake of geotagging, it’s clear that the systems can handle it, so we’re releasing earlier than planned.

All in all, a wicked few days for the Flickr team (thanks for your incredible work everyone!) And thanks to everyone who uses Flickr for making it even better, all the time. (Now, post your favorite map searches!)

UPDATE: Bonus link! (I just got sent this.) Yahoo! Research Berkeley’s ZoneTag smart mobile upload client just got a new feature: if you take a photo "near" an event (in time and space), it’ll automatically get tagged with the correct Upcoming event and show up on the corresponding event page without you doing anything. Very snazzy.