Tech Tips: Twitter Analytics with Google Natural Language

Summary

I'll be taking Twitter tweets and processing them through Google's Natural Language APIs in this post. The NL APIs provide the ability to parse text into 'entities' and/or determining 'sentiment' of the entity and surrounding text. I'll be using a combination of the two to analyze some tweets.

Entity + Sentiment Analysis

Lines 4-23: REST call to Google's NL entity-sentiment endpoint. The response from that endpoint is an array of entities in rank order of 'salience' (relevance). Entities have types, such as organization, person, etc. Net, the tweet gets parsed into entities and I'm pulling the #1 entity from the tweet as ranked by salience.

const ENTITY_SENTIMENT_URL = 'https://language.googleapis.com/v1beta2/documents:analyzeEntitySentiment?key=' + GOOGLE_KEY;

    try {
        const response = await fetch(ENTITY_SENTIMENT_URL, {
            method : 'POST',
            body : JSON.stringify(body),
            headers: {'Content-Type' : 'application/json; charset=utf-8'},
        });

        if (response.ok) {
            const json = await response.json();
            const topSalience = json.entities[0];
            const results = {
                'name' : topSalience.name,
                'type' : topSalience.type,
                'salience' : topSalience.salience,
                'entitySentiment' : topSalience.sentiment
            }
            return results;
        }
        else {
            let msg = (`response status: ${response.status}`);
            throw new Error(msg);
        }
    }
    catch (err) {
        ts = new Date();
        let msg = (`${ts.toISOString()} entitySentiment() - ${err}`);
        console.error(msg)
        throw err;
    }

Sentiment Analysis

Lines 1-22 implement a sentiment analysis of the entire tweet text.

    try {
        const response = await fetch(SENTIMENT_URL, {
            method : 'POST',
            body : JSON.stringify(body),
            headers: {'Content-Type' : 'application/json; charset=utf-8'},
        });

        if (response.ok) {
            const json = await response.json();
            return json.documentSentiment;
        }
        else {
            let msg = (`response status: ${response.status}`);
            throw new Error(msg);
        }
    }
    catch (err) {
        ts = new Date();
        let msg = (`${ts.toISOString()} sentiment() - ${err}`);
        console.error(msg)
        throw err;
    }

Blending

Lines 1-16: This function calls upon both of the Google NL API functions above and provides a blended analysis of the tweet text. Below, I took the product of the averages of the magnitude (amount of emotion) and score (positive vs negative emotion) between the entity-sentiment and overall sentiment to arrive at an aggregate figure. There are certainly other ways to combine these factors.

async function analyze(tweet) {
    const esnt = await entitySentiment(tweet);
    const snt = await sentiment(tweet);

    let results = {};
    results.tweet = tweet;
    results.name = esnt.name;
    results.type = esnt.type;
    results.salience = esnt.salience;
    results.entitySentiment = esnt.entitySentiment;
    results.documentSentiment = snt;
    let mag = (results.entitySentiment.magnitude + results.documentSentiment.magnitude) / 2;
    let score = (results.entitySentiment.score + results.documentSentiment.score) / 2;
    results.aggregate = mag * score;
    return results;
}

Execution

Lines 1-4: Simple function for reading a JSON-formatted file of tweets.
Lines 6-17: Reads a file containing an array of tweets and process each through the Google NL functions mentioned above.

async function readTweetFile(file) {
    let tweets = await fsp.readFile(file);
    return JSON.parse(tweets);
}

readTweetFile(INFILE)
.then(tweets => {
    for (let i=0; i < tweets.length; i++) {
        analyze(tweets[i].text)
        .then(json => {
            console.log(JSON.stringify(json, null, 4));
        });
    }
})
.catch(err => {
    console.error(err);
});

Results

Below is an example of a solid negative sentiment from Donald Trump regarding Adam Schiff. Schiff is accurately identified as a 'Person' entity. Note the negative scores + high emotion (magnitude) in both the entity sentiment and overall sentiment analysis.

{
    "tweet": "Shifty Adam Schiff wants to rest his entire case on a Whistleblower who he now
     says can’t testify, & the reason he can’t testify is that he is afraid to do so 
     because his account of the Presidential telephone call is a fraud & 
     totally different from the actual transcribed call...",
    "name": "Adam Schiff",
    "type": "PERSON",
    "salience": 0.6048015,
    "entitySentiment": {
        "magnitude": 3.2,
        "score": -0.5
    },
    "documentSentiment": {
        "magnitude": 0.9,
        "score": -0.9
    },
    "aggregate": -1.435
}

Below is another accurately detected 'person' entity with a positive statement from the President.

{
    "tweet": "Kevin McAleenan has done an outstanding job as Acting Secretary of
Homeland Security. We have worked well together with Border Crossings being way down.
Kevin now, after many years in Government, wants to spend more time with his family and
go to the private sector....",
    "name": "Kevin McAleenan",
    "type": "PERSON",
    "salience": 0.6058554,
    "entitySentiment": {
        "magnitude": 0.4,
        "score": 0
    },
    "documentSentiment": {
        "magnitude": 1.2,
        "score": 0.3
    },
    "aggregate": 0.12
}

Source

https://github.com/joeywhelan/twitterAnalytics

Wednesday, October 16, 2019

Twitter Analytics with Google Natural Language