Word Lists and Sentiment Analysis

A traditional method of analyzing texts is to compute the proportion of the words have positive connotations, negative connotations or are neutral. This method is commonly referred to as sentiment analysis. The typical approach to sentiment analysis is to see how many words in a text are also in a predefined list of words associated with a sentiment. So “I am having a bad day.” might score a “1” on a negative sentiment scale for the presence of “bad” or a .17 because one of six of the words is negative. Some sentiment systems rank words on a scale, so that “terrific” might be a 5 while “fine” scores a 1.

Some systems go beyond positive and negative. The proprietary LIWC program, for example, extends this to measure dozens of other word attributes, such as “tone”, “analytic thinking”, and “clout”. More generally, these methods can be used whenever you have a list of words, and you want to count their occurrences in a set of texts. They are commonly referred to as “dictionary methods.”

This lesson introduces two different dictionaries that are available in Python, AFINN, and Vader. It concludes by showing how to analyze a text corpus for occurrences on any arbitrary word list.

This lesson assumes your computer has an Anaconda Python 3.7 distribution installed.

AFINN

AFINN is an English word listed developed by Finn Årup Nielsen. Words scores range from minus five (negative) to plus five (positive). The English language dictionary consists of 2,477 coded words.

If this is your first time running this notebook, you may need to install it:

!pip install afinn

from afinn import Afinn

After importing Afinn, you need to set the language, English (en), Danish (da), or emoticon (emoticons).

afinn = Afinn(language='en')

The score method returns the sum of word valence scores for a text string.

afinn.score('Bad day.')

-3.0

afinn.score('Good day.')

3.0

afinn.score('Horrible, bad day.')

-6.0

In all these cases, afinn has preprocessed the text by removing the punctuation, converting all the words to lower-case, and before analyzing it.

Before using a sentiment dictionary, it is useful to see whether it has any face validity. To do that, we can look at a sample of the words from the list.

After importing the pandas library, the cell below will load word list as a pandas dataframe from the tab-delimited version on Afinn’s GitHub page and display a sample of 10 words.

import pandas as pd

afinn_wl_url = ('https://raw.githubusercontent.com'
                '/fnielsen/afinn/master/afinn/data/AFINN-111.txt')

afinn_wl_df = pd.read_csv(afinn_wl_url,
                          header=None, # no column names
                          sep='\t',  # tab sepeated
                          names=['term', 'value']) #new column names

seed = 808 # seed for sample so results are stable
afinn_wl_df.sample(10, random_state = seed)

We can get a sense of the distribution of word values by plotting them:

%matplotlib inline

afinn_wl_df['value'].hist()

<matplotlib.axes._subplots.AxesSubplot at 0x119d0d198>

Overall, the dictionary appears to have more negative words than positive words, but the values for both positive and negative words are rarely extreme, with both two and negative two as the most common values.

Applying the dictionary

We can use Afinn to analyze a more extensive text database. Aashita Kesarwani put together a corpus of comments made to New York Times articles. I sampled 10,000 of these from April 2017 and stored them as a JSON file.

json_url = ('https://github.com/nealcaren/osscabd_2018/'
            'blob/master/notebooks/data/nyt_201704_comments.json?raw=true')

nyt_df = pd.read_json(json_url)

The head method provides an overview of the dataframe.

nyt_df.head()

The column of interest is commentBody.

To estimate the Afinn sentiment score for all of the responses in the dataframe, we can apply the scorer to the commentBody column to create a new column. Applying this function takes a couple of seconds.

nyt_df['afinn_score'] = nyt_df['commentBody'].apply(afinn.score)

describe gives a sense of the distribution.

nyt_df['afinn_score'].describe()

count    10000.000000
mean        -0.283000
std          7.166188
min       -130.000000
25%         -3.000000
50%          0.000000
75%          3.000000
max         42.000000
Name: afinn_score, dtype: float64

It is also useful to sort by afinn_score to get a sense of what is in the extreme scoring comments. In this case, I subset the dataframe to display just the two relevant columns.

columns_to_display = ['commentBody', 'afinn_score']

nyt_df.sort_values(by='afinn_score')[columns_to_display].head(10)

It could be useful to see more of the comment.

pd.set_option('max_colwidth', 100)

nyt_df.sort_values(by='afinn_score')[columns_to_display].head(10)

sample = nyt_df.iloc[3956]['commentBody']
print(sample)

The ultimate weakness of violence is that it is a descending spiral, begetting the very thing it seeks to destroy. Instead of diminishing evil, it multiplies it. Through violence you may murder the liar, but you cannot murder the lie, nor establish the truth. Through violence you murder the hater, but you do not murder hate. In fact, violence merely increases hate ... Returning violence for violence multiples violence, adding deeper darkness to a night already devoid of stars. Darkness cannot drive out darkness; only light can do that.

~ Martin Luther King

By default, the sort is ascending, mean the lowest scoring, or most negative comments, are displayed by head. The comments with the highest score are shown with tail.

nyt_df.sort_values(by='afinn_score')[columns_to_display].tail(10)

sample = nyt_df.iloc[3486]['commentBody']
print(sample)

I applaud the spirit of this column and agree one should approach politics with compassion for others with differing points of view -- a) because they may have little choice, given their life story, to believe what they believe; and b) they may be right.

There is of course a compassionate center-right vision not comfortable with PC or identity politics (of any color or gender) that believes amping up tensions between groups is not a good idea, that believes decentralized markets will solve problems like healthcare in a much more humane way (by better saturating the distribution).

These people might also oppose the attacks on free speech and due process on today's campus, the use of the govt to surveil people, and the way the media often sides in Orwellian fashion with whatever the statist vision is.

These people are eminently sane and favor a longer-road humanism that results in a sustainable society with greater law and order (where people of all races can flourish in peace on calm streets) and wherein govt largesse can be brought in later in the pipeline after true market reforms have occurred to help make systems like education and healthcare more functional via the human desire to compete.

One of the drawbacks to using the raw Afinn score is the that longer texts may yield higher values simply because they contain more words. To adjust for that, we can divide the score by the number of words in the text.

The most straightforward way to count words in a Python string is to use the split method, which splits a string based on white spaces, and then count the length of the resulting list.

def word_count(text_string):
    '''Calculate the number of words in a string'''
    return len(text_string.split())

word_count('This sentence has seven words in it.')

7

You can employ that function on our dataframe to create a new column, word_count using appply to the text column, commentBody.

nyt_df['word_count'] = nyt_df['commentBody'].apply(word_count)

nyt_df['word_count'].describe()

count    10000.000000
mean        73.459700
std         63.508284
min          2.000000
25%         26.000000
50%         53.000000
75%        100.000000
max        296.000000
Name: word_count, dtype: float64

We can divide the original score by the word count to produce afinn_adjusted. This isn’t exactly a percentage variable, since word scores in Afinn can range from -5 to 5, but it is a useful adjustment to control for variable comment length. To make it clearer that this isn’t a percent score, and to make the results more readable, the adjustment is multiplied by 100.

nyt_df['afinn_adjusted'] = nyt_df['afinn_score'] / nyt_df['word_count'] * 100

nyt_df['afinn_adjusted'].describe()

count    10000.000000
mean         0.216934
std         14.222974
min       -100.000000
25%         -6.000000
50%          0.000000
75%          5.882353
max        266.666667
Name: afinn_adjusted, dtype: float64

You can use groupby to see how the sentiment score varies by key characteristics, such as whether or not a New York Times editor highlighted the comment.

nyt_df.groupby('editorsSelection')['afinn_adjusted'].describe()

The above syntax maybe a little complex to decipher. * nyt_df is the dataframe we want to use; * .groupby('editorsSelection') creates a pandas groupby object split by the values of editorsSelection; * ['afinn_adjusted'] is the specific column we want to focus on; * .describe() produces descriptive statistics for each of the groups.

Overall, the findings suggest that editors select comments with more of a negative tone.

Pandas can also be used to create the absolute value of a variable using the abs method. This is useful for exploring to what extent, in this case, editors embrace or avoid comments that are extreme, either positive or negative.

nyt_df['afinn_adjusted_abs'] = nyt_df['afinn_adjusted'].abs()

nyt_df.groupby('editorsSelection')['afinn_adjusted_abs'].describe()

Here, there seems to be some evidence that editors are avoiding comments with extreme sentiment, as values are lower across the board for the editor’s selections.

Vader

A second method for sentiment analysis is vader(Valence Aware Dictionary and sEntiment Reasoner). According to the authors, it is, “a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media.”

As with Afinn, Vader measures both the strength and direction of sentiment. Based on the work of 10 coders, the Vader dictionary includes approximately 7,500 words, emoticons, emojis, acronyms, and commonly used slang.

Unlike Afinn, Vader scores an entire text, not just words. Looking at the whole text allows for the algorithm to adjust for negations, such as “not”, booster words, such as “remarkably”. It also scores words written in all caps as more intense. Vader returns the proportion of a text that is negative, positive, and neutral, along with a combined score.

There is a version included with nltk (from nltk.sentiment.vader import SentimentIntensityAnalyzer) but a more recent version can be separately installed:

!pip install vaderSentiment

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

Vader requires that you set up an analyzer.

analyzer = SentimentIntensityAnalyzer()

The polarity_scores method returns a dictionary with four items. * pos, neu, and neg are the proportions of text that fall in each category. * compound is the normalized, weighted composite score.

analyzer.polarity_scores('Horrible bad day.')

{'neg': 0.875, 'neu': 0.125, 'pos': 0.0, 'compound': -0.7906}

One useful feature of Vader is that it is able to look at works in context and score appropriately.

analyzer.polarity_scores("At least it isn't a horrible book.")

{'neg': 0.0, 'neu': 0.637, 'pos': 0.363, 'compound': 0.431}

It also scores contemporary lingo and emojis.

analyzer.polarity_scores('Today SUX!')

{'neg': 0.779, 'neu': 0.221, 'pos': 0.0, 'compound': -0.5461}

analyzer.polarity_scores('💋')

{'neg': 0.0, 'neu': 0.263, 'pos': 0.737, 'compound': 0.4215}

Since Vader returns a dictionary (unlike Afinn which returns a single value), it is slightly more complicated to use it on an entire pandas dataframe.

First, apply the analyzer on the text column.

sentiment = nyt_df['commentBody'].apply(analyzer.polarity_scores)

Our new object sentiment is a series, where each item is a dictionary. This series can be unpacked into a dataframe.

sentiment_df = pd.DataFrame(sentiment.tolist())

sentiment_df.head()

The new sentiment dataframe can be merged with the original dataframe.

nyt_df_sentiment = pd.concat([nyt_df,sentiment_df], axis = 1)

nyt_df_sentiment.head()

If you intend to do this more than once, it can be useful to wrap the entire process into a single function that takes a dataframe and returns the datframe with the polarity columns appended. To clarify where the sentiment information comes from, the prefix vader_ is added to each of the polarity scores.

def vaderize(df, textfield):
    '''Compute the Vader polarity scores for a textfield.
    Returns scores and original dataframe.'''

    analyzer = SentimentIntensityAnalyzer()

    print('Estimating polarity scores for %d cases.' % len(df))
    sentiment = df[textfield].apply(analyzer.polarity_scores)

    # convert to dataframe
    sdf = pd.DataFrame(sentiment.tolist()).add_prefix('vader_')

    # merge dataframes
    df_combined = pd.concat([df, sdf], axis=1)
    return df_combined

df_vaderized = vaderize(nyt_df, 'commentBody')

Estimating polarity scores for 10000 cases.

df_vaderized.head()

The distribution of the combined variable shows peaks at the extremes and zero.

%matplotlib inline


df_vaderized['vader_compound'].plot(kind='hist')

<matplotlib.axes._subplots.AxesSubplot at 0x115c7abe0>

Plotting the positive and negative scores shows that many comments have both attributes present.

df_vaderized.plot.scatter(x='vader_pos', y = 'vader_neg')

<matplotlib.axes._subplots.AxesSubplot at 0x1166bb160>

Finally, unlike the Afinn score analysis, there’s no strong evidence that the New York Times’ editor selection is associated with the Vader sentiment scores.

sentiment_variables = ['afinn_adjusted', 'vader_neg', 'vader_neu', 'vader_pos']

df_vaderized.groupby('editorsSelection')[sentiment_variables].mean()

Word List

Occasionally, you will have a sentiment list from a different source that you would like to use. Or, more generally, you have a word list about any subject, not just attitudes, and you want to count their occurrences in texts, such as the use of words associated with politics or hypothesis testing.

The final section of the lesson shows the steps for building a function that can analyze texts for the presence of words on any given list. In this case, the sample list will be words associated with men that were assembled by Danielle Sucher.

The list is stored as a csv file. Pandas can be used to read the word list and turn it into a Python list.

male_words_df = pd.read_csv('data/male_words.csv')
male_words_df.sample(10)

male_words_list = male_words_df['term'].values

The function that looks for cooccurences has two parts. A preliminary helper function transforms the original text string into a list of lower case words stripping out any punctuation.

def text_to_words(text):
    '''Transform a string to a list of words,
    removing all punctuation.'''
    text = text.lower()

    p = '!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'
    text = ''.join([ch for ch in text if ch not in p])

    return text.split()

text_to_words('Make this lower case and remove! All? Punctuation.')

['make', 'this', 'lower', 'case', 'and', 'remove', 'all', 'punctuation']

The main function takes two arguments, the text and the word list. First, the text string is transformed to a list using text_to_words. Second, a new list intersection is created which contains only those elements from the text list that are in the word list. Finally, the function returns the length of the intersection.

def count_occurences(text, word_list):
    '''Count occurences of words from a list in a text string.'''
    text_list = text_to_words(text)

    intersection = [w for w in text_list if w in word_list]

    return len(intersection)

count_occurences('He went to the store.', male_words_list)

1

The function can now be applied on the Times dataframe to count ocurrences of male words in the comments. Since the function takes a second arguement, the word list, this is passed to the apply method as a tuple.

nyt_df['male_words'] = nyt_df['commentBody'].apply(count_occurences,
                                                   args=(male_words_list, ))

Most comments do not have words from our male list.

nyt_df['male_words'].describe()

count    10000.000000
mean         1.101600
std          2.126812
min          0.000000
25%          0.000000
50%          0.000000
75%          1.000000
max         19.000000
Name: male_words, dtype: float64

Sorting the dataframe in descending order by the new variable reveals the comments with the highest number of male words. In order to view larger parts of the text field, I adjust the max_colwidth option in pandas.

pd.set_option('display.max_colwidth', 1000)
nyt_df.sort_values(by='male_words', ascending=False)[['commentBody', 'male_words']].head()

To highlight the flexibility of the count_occurences function, load a new list of female words, from the same source, in order to estimate the number of female words in each comment. After the word list is loaded, this is accomplished by supplying the new word list as an argument in the function.

female_words_df = pd.read_csv('data/female_words.csv')
female_words_list = female_words_df['term'].values

nyt_df['female_words'] = nyt_df['commentBody'].apply(count_occurences, args=(female_words_list, ))

Female words are rarer in comments.

gender_words = ['male_words', 'female_words']
nyt_df[gender_words].describe()

Finally, it appears that Editors select comments with more gendered words, both male and female.

nyt_df.groupby('editorsSelection')[gender_words].describe()