Skip to:

Getting a feel for sentiment analysis

An excellent session of the London Text Analytics Group (March 14) contrasted two approaches to sentiment analysis: one proudly (and publicly) ditches grammar, while the other uses grammar to disambiguate content. Both approaches made ambitious claims for their software; which is the best approach?

Stephen Pulman of TheySay, a start up from the University of Oxford, had the more traditional approach.  He pointed out that taking individual words by themselves can lead to great confusion. Just assessing whether something is positive or negative is not so simple: “Bacteria” is negative, and “kill” is negative, but “kills bacteria” is positive.  More complex still, the phrase “never fails to kill bacteria” is highly positive.  A bag-of-words approach is unlikely to pick up all these distinctions.

After this clear, albeit limited explanation, he provided some examples – which, to be honest, were not highly exciting. He admitted that their system struggled with ambiguity (“wicked” can vary from positive to negative depending on the context) and with sarcasm and irony – although, of course, these become less of a problem if you look at text exclusively within a domain; but I would imagine most sentiment analysis tools would have problems here as well.

 

Josef Steinberger, from the University of West Bohemia, described the approach of his company, SentiSquare, and made some big claims for it. SentiSquare is certainly ambitious, working with multiple languages, and abandoning what he called the “dictionary method”, by which he meant trying to disambiguate terms with multiple meanings: he used the examples of “vin” meaning wine, “VIN”, the abbreviation for vehicle information number, used for car ID purposes, and Vin Diesel, the film actor – no need, he said, to disambiguate these terms when they appear.

 

It became a bit clearer why there was no need to disambiguate them during the Q&A session. He revealed that SentiSquare uses a separate corpus for each industry in which it works (e.g. finance sector, automobile sector) and this would explain why disambiguation becomes less important – of the three “VIN” terms above, two are highly unlikely in texts about the car industry. He also revealed the use of “consultants”, which presumably means some human fine-tuning of the results.  

How does SentiSquare work? His presentation was even sketchier than that for TheySay. After what he termed “semantic analysis” of the content, the software then creates a “cluster of prioritized comments”. In his example, an analysis of many thousands of text posts relating to new mothers on maternity leave revealed a high number of comments about “losing weight”, but relatively few about sport or exercise. Hence, on his next slide, the recommendation to the client (Nestle), who were employing SentiSquare to identify topics of interest for their newsletter to new mothers, was to provide information about diets. I found the jump from sentiment analysis to business recommendation something of a leap, but to be fair to SentiSquare, this could not be labelled as a fault of the sentiment analysis -  it was a fault of the conclusions being drawn from it. 

So which of the two approaches was better?  Clearly the grammatical approach is highly language-specific; the advantage of a more purely statistical tool such as SentiSquare is the possibility of using it with multiple languages, but even with SentiSquare Steinberger was cautious about using the tool for languages that did not stem in the way many European languages do.

Both techniques showed the remarkable results that could be obtained, and yet the limitations of sentiment analysis. I was not completely convinced by any of the client screenshots I saw. A screenshot from TheySay with a Sainsbury logo had a big red downward arrow, which looked like a pretty broad-brush result suggesting comments about Sainsbury were becoming more negative. Yes, sure, but why exactly? A slide from SentiSquare suggested that the most popular opinion on the Web about the Volkswagen emissions scandal was a request for car owners to have their cars serviced free for a year. It is understandable why many drivers would say that, but this would by no means necessarily be an effective management response to the VW emission crisis. TheySay showed a beta screen about Brexit, and attitudes for and against it; but the slide only showed how much work still remained to be done (“David Cameron”, “Mr Cameron”, and “UK prime minister” all appeared as separate entities – need for some human input here!)

So in conclusion, both presentations suggested that while extracting sentiment is certainly possible, the action that should be taken with this information is a separate, and important, activity, that the sentiment analysis tool cannot help with.