Predicting Visual Memory
February 4, 2013, Electronic Imaging Conference, Burlingame, CA—Aude Oliva from MIT described her work in identifying characteristics of memorable pictures. This work is becoming more important as the number of photos spreads across the Internet.
The number of images continues to grow, and the social media are accelerating the trend. Some estimates are that 90 percent of all Web content will be visual. This progression causes at least two problems, how to search for specific images, and remembering which ones you have seen.
The research crosses many areas, including human responses, physiology, memory, and others. The challenge is how to make an image better and increase its memorable quotient. Obvious applications for this work encompass ads, apps, mnemonic aids, memory training, and memory diagnostics for head trauma and dementia.
This is a new area of study that started with a simple memory game. The game displayed a number of images at 1 second intervals, and the player had to indicate recall. The game has multiple levels and reward structures. Players noted that the game was very engaging. The images were from Web collections and contained 900 categories. Those with people were recalled at a 90 percent rate, scenery with identifiable structure averaged 67 percent recall, and landscapes were mostly forgettable at 40 percent.
This memorabilty is consistent across groups and stable over time. The tests were modified to change timing and the number of intervening images before a repeat. A 15 image span takes about 36 seconds, a 100 image span about 4 minutes, and 1k images takes about 40 minutes. The memorable images retained their high rankings while the indistinct images got worse.
Since faces have a high memorability score, they started another experiment with 10k faces. 2222 of these had memory scores averaging 80 percent correct recall. Some of the faces were forgettable, and some triggered false positive responses. The key attribute seems to be a function of implied attitude.
Now the task is to rank the features that make an image memorable. Subjective judgments of faces is not the same as an actual ability to remember that face. People are poor at predicting how and what they remember, so the researchers are extracting features and scoring them relative to the bulk test results.
A simple scalar measure for parameters like brightness, number of objects, mean hue showed almost no correlation with actual results. Scene category also fared poorly, with correlation of P=0.37. Object features plus name plus size plus position was almost as good as a guess at P=0.47. Pixel extractions, GIST, SIFT ( Scale invariant feature transform), and histograms of oriented gradients also didn’t fare well.
Beautiful landscapes are not very memorable, but images with people, action, and activities are. The common elements are close-ups and people-sized objects. The researchers used functional MRI to check on brain activity for actions related to perception and memory. Some images have inherent triggers, but the possibility of global and regional biases caused the to restrict their populations to US residents.
They checked short-term retention, but image sequences of over 30 seconds are outside of the short-term range. They didn’t try to measure and characterize long-term memory, as some of the images might last a lifetime. They are working on developing techniques to enhance content to increase memorabilty. As with anything to do with people, expertise in an area and associations affect the results. Novelty seems to help improve memories, but the images and components need some reference.
