Sentiment Analysis

Every day an estimated 500 million tweets are posted on Twitter. People report on what they had for breakfast, what they watched on television, how they feel about their elected representatives, the fight they had with the customer service agent for their cable-TV provider, and any other number of topics. They share both their innermost feelings about the state of the world as well as carefully crafted public relations messages intended to sway opinion in a specific direction.

It's worth noting that the half a billion tweets represents information generated on a single platform, which isn't even the largest of the social media networks. That honor is reserved for Facebook. Together, Instagram and Facebook average three billion users per month, more than a third of the world's population.

Social media has become the medium through which people live a significant part of their lives. It's used to connect, share, advertise, and interact. But there is significantly more to social media than just the modern day equivalent of the social square. It is also one of the most important platforms for business, marketing, and public engagement.

Tens of billions of dollars worth of business is transacted either implicitly (and often explicitly) across social media. To highlight its importance, consider (source: Socialpilot.co):

Put succinctly: social media is a powerful channel to connect businesses with consumers. But, that is only part of its value. Social media is essential to companies for a second reason: it provides an invaluable source of data about how organizations and brands are perceived. Buried in those day-to-day interactions is information about what people say and how they feel, often in very candid and personal ways.

Whether it is a Facebook post reviewing a product, a tweet commenting on the service, or a direct message to the brand; that information is highly valuable. Taken in aggregate, it can provide profound insights into how customers interact with the organization and how that relationship has changed over time.

There is just one problem with social media data: much of the most important information is locked up in raw text. Communications, blog articles, Twitter messages, and Instagram postings are written for humans, not computers; and to get actionable intelligence, you often need to consider thousands or millions of interactions.

This leaves a problem: if the information from social media is valuable, but the volume of data is often too large to analyze manually; how do you mine for analytic gold? How do you extract structured data that provides metrics to analyze the interactions people have in the online social square?

Short answer: you use sentiment analysis. Throughout this article we will introduce sentiment analysis, look at how it works, and review some of its most important applications.

How Does Sentiment Analysis Work?

Now that we know what it is, how is it possible that computers gain a measurable insight into human feeling?

Human language is complex: slang, misspellings, and cultural variations all modify the meaning of text and its perception. Beyond structural and regional considerations, language is nuanced. Specific placements and subtle modifiers that might appear in prior sentences or paragraphs can change the meaning of a passage completely. How do you go about trying to quantify sarcasm, for example?

For these reasons, sentiment analysis systems can be amongst the most sophisticated types of machine learning systems. In other cases, it uses brute force to roughly correlate the presence of certain words with emotion. Looking at the process from high altitude, sentiment systems attempt to create numeric scores for:

• polarity: the emotional strength of the sentiment, usually classified between negative and positive
• subjectivity: is the statement intended to represent an objective fact or personal feelings, views, or beliefs with the latter being more associated with "feelings"

There are many approaches to quantifying these two metrics, some more sophisticated than others. But once a machine can understand the polarity and subjectivity of a piece of text, that provides the core of the sentiment. When the same process is run on hundreds of thousands of samples, the resulting computation provides a remarkably accurate quantification of the human feeling expressed.

So how does the magic work? How are such scores created?

Text Preparation: Split, Parse, and Encode

Like many other forms of NLP, the first step is to prepare the text: split input into words or phrases, standardize (or lemmatize), and (optionally) encode.

The process begins by taking the raw text and spliting it into a set of meaningful pieces (sometimes called "tokens" or n-grams) representing words or groups of words. As the input is being split, the computer will perform additional processing to simplify and standardize the text, such as removing punctuation and word variations/modifiers. Processing pipelines can be as simple or complicated as needed, but often include steps such as:

• lemmatization, where word varaiations are modified to a single representation
• changing plural references of a word to a singular (examples: "cats" would become "cat", "churches" would be come "church" and "abaci" would be "abacus.")
• modifying the past or future tense of a verb to present tense (examples: "borrowed" would become "borrow")

Once the input has been prepared, there are some systems that are able to work on it directly. For more sophisticated analysis, however, additional processing is usually needed. If it will be handed off to a machine learning system, for example, the next step is to structure (or encode) the text in a way for efficient computation. Because ML algorithms are not generally able to work on text data, encoding usually means representing words as a set of numbers (usually called a vector).

There are a lot of strategies that can be used to do this such as Bag of Words (BoW), term frequency-invese document frequency (tf-idf), Glove, and Word2Vec. With each, the goal is to generate a consistent set of numbers (indexes) that align with the input text. These numbers then become the predictors (or "features") used by the sentiment algorithm to create the scores. The way the vector is created and encoded is is different depending on the algorithm.

Sentiment Analysis Algorithms

Once data has been prepared, there are many different algorithms capable of computing scores from the features. These include:

• Rule-Based approaches, which utilizes a formal procedure coded as a set of processing steps or "rules"
• Automatic approaches, which leverages labeled data to determine the sentiment directly from the text without requiring an explicit procedure
• Hybrid approaches, which blend the rule-based and automatic techniques

Rule-Based Approach

Rule-based systems perform a distinct transform or encoding operation on a piece of input text. The operation is usually defined by an expert in languages and encoded using a special syntax in a grammar (a programming language intended to make it easier to express processing rules).

Machine-Learning (Automatic) Approach

Machine learning approaches use statistical and mathematical models to try and infer the sentiment directly from a text without being explicitly programmed. This ability to learn makes them very adaptable.

Like when working with other types of structured data, ML sentiment analysis is a process of encoding information so that numerical trends can be found that relate (or correlate) with a desired outcome (or target). It's a type of supervised "classification" (for outcomes with a specific category like good/bad) or "regression" (if the outcome is on a continuous scale).

This means that much of the work involved in creating sentiment models is very similar to other types of ML analysis:

1. Acquire and label data
2. Create a pipeline to prepare the text
3. Train a model instance
4. Iteratively assess the model and refine the inputs until the results are "good enough" for your application

It also means that the same families of algorithms are used to create sentiment ML systems, such as Naive Bayes, Linear Regression, Support Vector Machines (SVM), and neural nets.

Acquire and Label Data

Everything starts with the acquisition and labeling of data. For sentiment analysis, this means that you need to have input data relevant to what you are trying to encode (and lots of it). If looking at customer comments, for example, you would need to have samples similar to what you are hoping to assess and you need to have those samples linked to either a category like happy/sad (for classification) or a score (for regression).

Sometimes that type of information might be submitted at the same time when the comment is acquired, such as when customers create reviews on Amazon and include a rating on a five-star scale. More often, though, the data needs to be reviewed and tagged by an expert. An important part of tagging data is ensuring that the categories and scores are consistent. It's common for two different reviewers to provide ratings that might differ quite a bit on the same text, and such variation can make the sentiment model very inconsistent. This is usually solved by having a single person grade the initial data set used to "train" the model.

Create a Processing Pipeline

Next, you use a pipeline to prepare and encode the data via a process as to what was described above. This usually involves a feature extractor that transfers text into a feature vector that is paired with the target label..

Train the Model

Then you take the labeled data and use that to train a model. What training means depends greatly on the type of machine learning algorithm chosen, but is the process where the computers learns to associate the text to the tag/outcome. The "artifact" that comes out the other side of training is your model.

Iteratively Assess and Improve

Before models are used, they are tested to see how accurately they describe a set of labeled data they have never seen before. If the model gives poor results, it's common to go back to the original data and tweak how the information is encoded or weighted and try again.

Hybrid Model

Hybrid approaches utilize a combination of rules and machine learning in order to generate their scores, often in a procedure that take the output from one step and pipes it into the next. Such pipelines can be as simple or complex as needed for a specific task, and may themselves be part of larger machine learning workflows.

Now that we have a rough idea how sentiment analysis works, where does it get used in the real world?

Sentiment analysis can be used to monitor just about every type of interaction in a company. In addition to the mountains of text available within social media, nearly every application produces or consumes text at some level. This means that sources are everywhere: medical notes, consumer complaint logs, product inquiries, repair records, customer support logs, emails, phone records, and meeting memos are only the beginning.

Sentiment analysis' "superpower" is that it allows you to get "structured" information about how people feel, providing quantitative data which can be considered alongside any other key performance indicators. Moreover, because the same text can be graded along many different scales, the types of analysis you do can become very powerful. It is often:

• Used to assess how changes to a formal processes impacted employee and customer satisfaction. Are customers happier after a new website feature was introduced? Did an advertising or social marketing campaign reach the right people, are they talking about a product, is there excitement or buzz in what they are saying?
• Leveraged as an input to machine learning systems to personalize marketing content based on how people have responded to certain types of information in the past.
• Applied to screen inbound customer support and sales tickets, and then escalate those that are deemed sensitive to ensure a more timely response.

How do you feel?

Sentiment analysis provides the ability to take otherwise unstructured text and turn it into structured data providing information about how people feel. It works through a process that uses specialized text preparation, rule based analysis, and machine learning to generate scores for:

• polarity: the emotional strength of the sentiment
• subjectivity: whether a statement is associated with feeling or attempts to represent objective fact

When combined with the enormous volume of data generated through social media and other applications, sentiment analysis provides a potent lens with which to view nearly every aspect of your organization. Does your product give customers a warm and fuzzy feeling? Are you meeting their expectations with your quality of service? Did your most recent marketing campaign make an impact? Sentiment analysis can answer these types of questions and many more.

It is one of the most impactful tools available to in the data science toolkit, and opens the door to making actionable decisions on the "gut feelings" of your customers. Customer and user feelings are too important to ignore and represents one (if not the most) important metrics of success for many organizations. Using sentiment analysis, all of that information can be at your finger tips, allowing you to mine the equivalent of analytic gold.