Google Goggles – Hot Chips Keynote
August 23, 2010 — Hot Chips conference, Stanford California, the first keynote address at the conference was from David Petrou from Google. His talk entitled “search originating inside and outside your head” described their efforts at developing Google Goggles, an application for mobile visual search.
Goggles is attempting to develop the capability of searching by picture to identify landmarks, street views and other information from an image. Part of their database includes history, a record of photos that have already been taken and deposited into the system. The greatest challenge in this application are the use cases, you don’t know what you’re looking at.
Design principles were to develop a universal image recognition capability, and the challenges were in generating specific results, sorting through ambiguity, minimizing false positives all while searching across more than 1 billion photographs. Image processing is the key technology using that they also are concerned with privacy.
So far, they can return “good” results about one third of the time. They have categories responses and have found objects like signs to be very challenging. Textured objects and clear sets of characters work. They can, relatively accurately, identify packaged goods, unique places and landmarks, and faces. Privacy concerns, however, are keeping the face recognition on hold.
The recognition flow starts with the query at a root functional level. A number of engines analyze the image for textures and interest points. The uses a parallel tree structure to match and label image characteristics. The Nike swoosh symbol is very hard to recognize because there are very few points available, it’s all curves without context. A difficulty is adjusting the identification rate versus the false positive rate, but on a QVGA size image the processing time is about 0.2 seconds per image.
For facial recognition, they can get good matching if they have 17 prior images of that face and return the correct one as one of the top 10 matches. If they have 50 images, they can get that face to the top 5 about half of the time. Text recognition would seem to be an easier task than faces, however text requires additional information and a relaxation of the processing steps.
Optical character recognition needs at least VGA resolution and the recognition depends on whether the text is dense, as in books or newspapers, or sparse, as in signs. At issue are characteristics external to the text. These issues include space and non-frontal or oblique images. Generally, sign recognition is less than 20 percent of images.
Another area of research is augmented reality. Here, they would add other information to an image to add definition and content. This is not the same as visual search because there are different thresholds for the use the value of this additional content. So far they’ve seen a ratio of about 100:1 for internal or user generated search versus external or gratuitously supplied information when people use goggles and augmented reality.
Sources for augmented reality include personal memory and sensors working in the infrared and ultraviolet wavelengths. The overall performance of the system is restricted by the latency of the client, network, and the servers. The problem is users expect high-quality results and universal recognition in a very short time. Returning responses were quickly requires a change in protocols especially on 3G networks. The timing constraints are relaxed a little on WiFi which is faster than 3G.
Ultimately, what they need is faster hardware. Optical character recognition needs better algorithms, but a streaming input may provide more clues to recognition since the search engines can pick a frame to process. Clients need to ingest the photos faster and provide more metadata. On the server side, they have to overcome the memory bandwidth limits and figure ways to increase parallelism.
Google goggles is a ongoing development program with limited availability. They’re working from a separate code base and different operating systems. Looking at alternative data input methods such as HTML 5, especially for camera image capture functions.
