Images Shared in Social Media
February 3, 2014, Electronic Imaging Conference, San Francisco—Shih-Fu Chang from Columbia University evaluated the various types of images appearing in social media. These images and collections of images are used to create metadata for further valuation.
People take and share pictures because of their emotional and sentimental value. Increasingly, the number of pictures going into social media is drawing comments. Social media researchers consider sites like Twitter to be the “heartbeat”of the online community, and try to evaluate keywords as a reflection of sentiment.
The web is becoming more visual, and many sites are making picture posting simple with the expectation that the pictures will draw comments. These social visual multimedia websites cover many areas of topical interest. In ’12, the tweet of the year came from @BarackObama “four more years” plus a photo of Barack and Michelle Obama. Another highly reused image was that of the roller coaster at sea on the New Jersey shore after hurricane Sandy.
A picture is supposed to be the equivalent of 1000 words, but also incorporates intent, perception, and a psychological emotion wheel that calls for new taxonomy and lexicons to describe the various gradations of emotions. For simplicity and consistency, they have defined words to describe 24 emotional gradations.
The current process is to crawl through the words and attributes of images on Flickr. They then try to relate the tags and emotions that are linked in the picture with the intended emotions from the photographer. They parse the words for adjectives and tags that relate the photos and the emotions and look for adjective-noun pairs was he adjective addresses emotional qualities and the nouns represent connector entities.
They remove adjective phrases like hotdog, etc. to reduce their search space. After filtering, they found 3000 pairs to represent the levels and relations of words. This ontology enables emotion browsers and maps. The next step is to invoke machine learning and big data to train classifiers in the 3000 pairs and test for accuracy.
They decompose images into features like color, contrast, etc. and use SIFT and other filtering to extract attribute descriptors, detect objects, and identify those features. Some of the aesthetic features they consider are dark hair, color harmony, etc., then these extracted features are compared to human responses to get annotations and characteristics.
A hierarchical texture engine finds objects and attributes and tests candidates and features against priors. For example, searching for “lonely dog” scores 0.4 with the basic dog detector but improves to 0.8 with object and attribute definitions. “Crazy car” in the control set rates 0.2 but improves to 1.0 accuracy with new tools and uses.
The next step is to correlate tweet text and pictures for live sentiment prediction. This multi-model analysis requires even greater accuracy, so they have set up a 3-D mechanical Turk analysis structure. They use a consensus vote of the three analysis engines for ground truth. One challenge is separating the emotions of the publisher and the viewer since there is not always correlation between publisher and viewer.
The analysis of the picture and the tweet consider the differences of the publisher and viewer and some new analysis tools are able to make recommendations, predict viewer responses, and feed a comment robot. This comment robot is a browser plug-in that makes guesses about appropriate comments for any picture.
As a result of these efforts, they can predict with some accuracy comments relating to any pictures. Using big data analytics tools is helping to infer intent but the area still needs to develop clear semantics. Ongoing research is working with existing and developing tools for greater accuracy and ease of use, so more users can mine images and content for further insights.
