Trading on Trump's Tweets

Summary

There have been several articles during President Trump's term regarding his use of Twitter and his influence on stock prices.  Below is one such article:

https://fortune.com/2017/02/24/trump-tweet-stocks/ 

This post explores development of a programmatic analysis of Trump's tweets that are focused on publicly traded companies.  I use a variety of APIs to ferret out the tweets of interest and then take action on them.  In this exercise, I simply generate an alert email; however, one could envision automated trading as the action.

This post represents the culmination of the my Twitter API blog series:

• Twitter API Authentication
• Twitter Search API
• Twitter Analytics with Google Natural Language
• Twitter Filtered Streams

Architecture

This is a Node-based architecture that uses various publicly accessible REST APIs.

 

Processing Logic

 

 

Code Excerpt - Tweet Processing

The function below accepts tweet text as input and then sends that text through Google's NL engine for entity analysis. If the top entity, ranked by salience, is an "ORGANIZATION" - then there's a chance the tweet is regarding a company. The next step is to use the IEX Cloud API to determine if the entity does in fact corresponds to a publicly traded company.  If so, then perform full processing of the tweet:  gather further NL + stock analytics and then package them for an email alert.

async function processTweet(tweet) {
    logger.debug(`processTweet()`);
    try {
        const esnt = await entitySentiment(GOOGLE_KEY, ENTITY_SENTIMENT_URL, tweet);

        if (esnt.type === 'ORGANIZATION') { //entity corresponds to a type that might be a company
            let stock;
            if (Array.isArray(symbolArray)) {
                stock = symbolArray.find(obj => {
                    return obj.name.match(esnt.name);
                });
                if (stock) {  //name corresponds to a publicly traded company - fetch full tweet sentiment
                    //and stock data
                    const snt = await sentiment(GOOGLE_KEY, SENTIMENT_URL, tweet);
                    const data = await getStockData(IEX_KEY, STOCK_URL, stock.symbol);

                    let analytics = {};
                    analytics.tweet = tweet;
                    analytics.name = esnt.name;
                    analytics.salience = esnt.salience;
                    analytics.entitySentiment = esnt.entitySentiment;
                    analytics.documentSentiment = snt;
                    let mag = (analytics.entitySentiment.magnitude + analytics.documentSentiment.magnitude) / 2;
                    let score = (analytics.entitySentiment.score + analytics.documentSentiment.score) / 2;
                    analytics.aggregate = mag * score;
                    analytics.symbol = stock.symbol;
                    analytics.data = data;
                    sendEmail(SENDGRID_KEY, SENDGRID_URL, analytics);
                }
            }
        }
    }
    catch(err) {
        logger.error(err);
    }
}

Code Excerpt - Fetch Stock Data

Excerpt below exercises IEX's API to fetch a few simple stock data items.  This API is quite rich.  There is significantly more analytics available that what I've pulled below:  current stock price and previous day's history.

async function getStockData(token, url, symbol) {
    logger.debug(`getStockData() - name:${symbol}`);
    
    let data = {};
    const price = await getPrice(token, url, symbol);
    const previous = await getPrevious(token, url, symbol);
    data.current_price = price;
    data.date = previous.date;
    data.open = previous.open;
    data.close = previous.close;
    data.high = previous.high;
    data.low = previous.low
    return data;
}

Code Excerpt - Test

Test function below submits a tweet President Trump unleashed on Harley-Davidson on June 25, 2018.

async function test() {
    symbolArray = await getSymbols(IEX_KEY, SYMBOL_URL);
    const tweet1 = "Surprised that Harley-Davidson, of all companies, would be the first to wave the White Flag. I
fought hard for them and ultimately they will not pay tariffs selling into the E.U., 
which has hurt us badly on trade, down $151 Billion. Taxes just a Harley excuse - be patient!";
    await processTweet(tweet1);
}

test()
.then(() => {
    console.log('complete');
});

Results

Excerpt below of the raw email text that was generated.

Date: Sun, 27 Oct 2019 17:54:52 +0000 (UTC)
From: twitterTrade@example.com
Mime-Version: 1.0
To: joey.whelan@gmail.com
Message-ID: 
Content-type: multipart/alternative; boundary="----------=_1572198892-24558-282"
Subject: Twitter Trade Alert - Negative Tweet: Harley-Davidson

{
    "tweet": "Surprised that Harley-Davidson, of all companies, would be th=
e first to wave the White Flag. I fought hard for them and ultimately they =
will not pay tariffs selling into the E.U., which has hurt us badly on trad=
e, down $151 Billion. Taxes just a Harley excuse - be patient!",
    "name": "Harley-Davidson",
    "salience": 0.35687405,
    "entitySentiment": {
        "magnitude": 0.4,
        "score": 0
    },
    "documentSentiment": {
        "magnitude": 0.9,
        "score": -0.1
    },
    "aggregate": -0.0325,
    "symbol": "HOG",
    "data": {
        "current_price": 39.39,
        "date": "2019-10-25",
        "open": 38.64,
        "close": 39.39,
        "high": 39.69,
        "low": 38.64
    }
}

Source

https://github.com/joeywhelan/twitterTrade

Copyright ©1993-2024 Joey E Whelan, All rights reserved.

Twitter Filtered Stream

Summary

This post discusses my use of  Twitter Developer Labs (beta) APIs for creating a real-time tweet feed.  The APIs are all HTTP-based.  The actual streaming tweet feed is a HTTP connection that, in theory, never ends.

Architecture

The diagram below depicts the overall flow for this exercise.

• An API token has to be fetched to call any of the Twitter APIs.
• Fetch any existing tweet filter rules
• Delete them
• Add new filtering rules
• Start streaming a tweet feed based on those filtering rules

Fetch API Token

I discussed the steps for that in this post.

Get Existing Filter Rules

The code below fetches any existing filtering rules in place for the given account associated with the bearer token.

const RULES_URL  = 'https://api.twitter.com/labs/1/tweets/stream/filter/rules';
async function getRules(token, url) {
    console.debug(`${(new Date()).toISOString()} getRules()`);
    
    try {
        const response = await fetch(url, {
            method: 'GET',
            headers: {
            'Authorization' : 'Bearer ' + token
            }
        });
        if (response.ok) {
            const json = await response.json();
            return json;
        }
        else {
            throw new Error(`response status: ${response.status} ${response.statusText}`);    
        }
    }
    catch (err) {
        console.error(`${(new Date()).toISOString()} getRules() - ${err}`);
        throw err;
    }
}

Delete Existing Filter Rules

Passing an array of filter IDs, delete that array from Twitter for the account associated with the bear token.

async function deleteRules(token, ids, url) {
    console.debug(`${(new Date()).toISOString()} deleteRules()`);
 
    const body = {
        'delete' : {
            'ids': ids
        }
    };
    try {
        const response = await fetch(url, {
            method: 'POST',
            headers: {
                'Content-Type' : 'application/json',
                'Authorization' : 'Bearer ' + token
            },
            body: JSON.stringify(body)
        });
        if (response.ok) {
            const json = await response.json();
            return json.meta.summary.deleted;
        }
        else {
            throw new Error(`response status: ${response.status} ${response.statusText}`);    
        }
    }
    catch (err) {
        console.error(`${(new Date()).toISOString()} deleteRules() - ${err}`);
        throw err;
    }
}

Add New Filtering Rules

The code below adds an array of filtering rules to a given account.  Example array with a single rule below.  That rule targets tweets from the President and filters out any retweets or quotes.

const RULES = [{'value' : 'from:realDonaldTrump -is:retweet -is:quote'}];
async function setRules(token, rules, url) {
    console.debug(`${(new Date()).toISOString()} setRules()`);
 
    const body = {'add' : rules};
    try {
        const response = await fetch(url, {
            method: 'POST',
            headers: {
                'Content-Type'  : 'application/json',
                'Authorization' : 'Bearer ' + token
            },
            body: JSON.stringify(body)
        });
        if (response.ok) {
            const json = await response.json();
            return json.meta.summary.created;
        }
        else {
            throw new Error(`response status: ${response.status} ${response.statusText}`);    
        }
    }
    catch (err) {
        console.error(`${(new Date()).toISOString()} setRules() - ${err}`);
        throw err;
    }
}

Stream Tweets

Below is an excerpt of the main streaming logic.  A link to the full source repo is at the bottom of this blog.  This excerpt follows happy path of a HTTP 200 response and starts up a theoretically never-ending reader stream to Twitter with tweets that match the filter criteria built up previously.  Twitter sends heartbeats on this connection every 20 seconds.

                g_reader = response.body;
                g_reader.on('data', (chunk) => {
                    try {
                        const json = JSON.parse(chunk);
                        let text = json.data.text.replace(/\r?\n|\r|@|#/g, ' ');  //remove newlines, @ and # from tweet text
                        console.log(`${(new Date()).toISOString()} tweet: ${text}`);
                    }
                    catch (err) {
                        //heartbeat will generate a json parse error.  No action necessary; continue to read the stream.
                        console.debug(`${(new Date()).toISOString()} stream() - heartbeat received`);
                    } 
                    finally {
                        g_backoff = 0;
                        clearTimeout(abortTimer);
                        abortTimer = setTimeout(() => { controller.abort(); }, ABORT_TIMEOUT * 1000);
                    } 
                });

Results

2019-10-24T14:01:01.906Z filter()
2019-10-24T14:01:01.909Z getTwitterToken()
2019-10-24T14:01:02.166Z clearAllRules()
2019-10-24T14:01:02.166Z getRules()
2019-10-24T14:01:02.353Z deleteRules()
2019-10-24T14:01:02.604Z number of rules deleted: 1
2019-10-24T14:01:02.605Z setRules()
2019-10-24T14:01:02.902Z number of rules added: 1
2019-10-24T14:01:02.903Z stream()
2019-10-24T14:01:03.179Z stream() - 200 response
2019-10-24T14:01:23.177Z stream() - heartbeat received
...
2019-10-24T14:20:03.657Z stream() - heartbeat received
2019-10-24T14:20:12.959Z tweet: The Federal Reserve is derelict in its duties if it 
doesn’t lower the Rate and even, ideally, stimulate. Take a look around the World at our 
competitors. Germany and others are  actually GETTING PAID to borrow money. Fed was way too 
fast to raise, and way too slow to cut!
2019-10-24T14:20:23.660Z stream() - heartbeat received

Source

https://github.com/joeywhelan/twitterFilter

Copyright ©1993-2024 Joey E Whelan, All rights reserved.

Twitter Analytics with Google Natural Language

Summary

I'll be taking Twitter tweets and processing them through Google's Natural Language APIs in this post.  The NL APIs provide the ability to parse text into 'entities' and/or determining 'sentiment' of the entity and surrounding text.  I'll be using a combination of the two to analyze some tweets.

Entity + Sentiment Analysis

Lines 4-23:  REST call to Google's NL entity-sentiment endpoint.  The response from that endpoint is an array of entities in rank order of 'salience' (relevance).  Entities have types, such as organization, person, etc.  Net, the tweet gets parsed into entities and I'm pulling the #1 entity from the tweet as ranked by salience.

const ENTITY_SENTIMENT_URL = 'https://language.googleapis.com/v1beta2/documents:analyzeEntitySentiment?key=' + GOOGLE_KEY;

    try {
        const response = await fetch(ENTITY_SENTIMENT_URL, {
            method : 'POST',
            body : JSON.stringify(body),
            headers: {'Content-Type' : 'application/json; charset=utf-8'},
        });

        if (response.ok) {
            const json = await response.json();
            const topSalience = json.entities[0];
            const results = {
                'name' : topSalience.name,
                'type' : topSalience.type,
                'salience' : topSalience.salience,
                'entitySentiment' : topSalience.sentiment
            }
            return results;
        }
        else {
            let msg = (`response status: ${response.status}`);
            throw new Error(msg);
        }
    }
    catch (err) {
        ts = new Date();
        let msg = (`${ts.toISOString()} entitySentiment() - ${err}`);
        console.error(msg)
        throw err;
    }

Sentiment Analysis

Lines 1-22 implement a sentiment analysis of the entire tweet text.

    try {
        const response = await fetch(SENTIMENT_URL, {
            method : 'POST',
            body : JSON.stringify(body),
            headers: {'Content-Type' : 'application/json; charset=utf-8'},
        });

        if (response.ok) {
            const json = await response.json();
            return json.documentSentiment;
        }
        else {
            let msg = (`response status: ${response.status}`);
            throw new Error(msg);
        }
    }
    catch (err) {
        ts = new Date();
        let msg = (`${ts.toISOString()} sentiment() - ${err}`);
        console.error(msg)
        throw err;
    }

Blending

Lines 1-16:  This function calls upon both of the Google NL API functions above and provides a blended analysis of the tweet text.  Below, I took the product of the averages of the magnitude (amount of emotion) and score (positive vs negative emotion) between the entity-sentiment and overall sentiment to arrive at an aggregate figure.  There are certainly other ways to combine these factors.

async function analyze(tweet) {
    const esnt = await entitySentiment(tweet);
    const snt = await sentiment(tweet);

    let results = {};
    results.tweet = tweet;
    results.name = esnt.name;
    results.type = esnt.type;
    results.salience = esnt.salience;
    results.entitySentiment = esnt.entitySentiment;
    results.documentSentiment = snt;
    let mag = (results.entitySentiment.magnitude + results.documentSentiment.magnitude) / 2;
    let score = (results.entitySentiment.score + results.documentSentiment.score) / 2;
    results.aggregate = mag * score;
    return results;
}

Execution

Lines 1-4:  Simple function for reading a JSON-formatted file of tweets.
Lines 6-17:  Reads a file containing an array of tweets and process each through the Google NL functions mentioned above.

async function readTweetFile(file) {
    let tweets = await fsp.readFile(file);
    return JSON.parse(tweets);
}

readTweetFile(INFILE)
.then(tweets => {
    for (let i=0; i < tweets.length; i++) {
        analyze(tweets[i].text)
        .then(json => {
            console.log(JSON.stringify(json, null, 4));
        });
    }
})
.catch(err => {
    console.error(err);
});

Results

Below is an example of a solid negative sentiment from Donald Trump regarding Adam Schiff.  Schiff is accurately identified as a 'Person' entity.  Note the negative scores + high emotion (magnitude) in both the entity sentiment and overall sentiment analysis.

{
    "tweet": "Shifty Adam Schiff wants to rest his entire case on a Whistleblower who he now
     says can’t testify, & the reason he can’t testify is that he is afraid to do so 
     because his account of the Presidential telephone call is a fraud & 
     totally different from the actual transcribed call...",
    "name": "Adam Schiff",
    "type": "PERSON",
    "salience": 0.6048015,
    "entitySentiment": {
        "magnitude": 3.2,
        "score": -0.5
    },
    "documentSentiment": {
        "magnitude": 0.9,
        "score": -0.9
    },
    "aggregate": -1.435
}

Below is another accurately detected 'person' entity with a positive statement from the President.

{
    "tweet": "Kevin McAleenan has done an outstanding job as Acting Secretary of
Homeland Security. We have worked well together with Border Crossings being way down.
Kevin now, after many years in Government, wants to spend more time with his family and
go to the private sector....",
    "name": "Kevin McAleenan",
    "type": "PERSON",
    "salience": 0.6058554,
    "entitySentiment": {
        "magnitude": 0.4,
        "score": 0
    },
    "documentSentiment": {
        "magnitude": 1.2,
        "score": 0.3
    },
    "aggregate": 0.12
}

Source

https://github.com/joeywhelan/twitterAnalytics

Copyright ©1993-2024 Joey E Whelan, All rights reserved.

Twitter Premium Search API - Node.js

Summary

In this post I'll demonstrate how to use the Twitter Premium Search API.   This is a pure REST API with two different search modes:  past 30 days or full archive search since Twitter existed (2006).

The API has a very limited 'free' mode for Developers to try out.  Limits are imposed on usage:  number of API requests, tweets pulled per month and rate of API calls.  To do anything of significance with this API, you're faced with paying for Twitter's API subscription.  That gets pretty pricey quickly with the cheapest tier currently at $99/month.  This post is based on usage of the 'free'/sandbox tier.

Main Loop

Line 1 fetches a bearer token for accessing the Twitter APIs.  I covered this topic in a previous post.

Lines 4-19 implement a while loop that fetches batches of tweets for a given search query.  For the Twitter free/sandbox environment, you can pull up to 100 tweets per API call.  Each tweet in the batch is evaluated to determine if it was 140 or 280 character tweet.  The tweet text is formatted and then that and the created_date are added to a JSON array.  That array is ultimately written to file.

Line 20 is a self-imposed delay on calls to the Twitter API.  If you bust their rate limits, you'll get a HTTP 429 error.

        const token = await getTwitterToken(AUTH_URL);
        let next = null;
        
        do {
            const batch = await getTweetBatch(token, url, query, fromDate, maxResults, next);
            for (let i=0; i < batch.results.length; i++) {  //loop through the page/batch of results
                let tweet = {};
                if (batch.results[i].truncated) {  //determine if this is a 140 or 280 character tweet
                    tweet.text = batch.results[i].extended_tweet.full_text.trim();
                }
                else {
                    tweet.text = batch.results[i].text.trim();
                }

                tweet.text = tweet.text.replace(/\r?\n|\r|@|#/g, ' ');  //remove newlines, @ and # from tweet text
                tweet.created_at = batch.results[i].created_at;
                tweets.push(tweet);
            }
            next = batch.next;
            await rateLimiter(3);  //rate limit twitter api calls to 1 per 3 seconds/20 per minute
        }
        while (next);

Tweet Batch Fetch

Lines 1-26 set up a node fetch to the Twitter REST API end point.  If this was a call with a 'next' parameter (meaning multiple pages of tweets on a single search), I add that parameter to the fetch.

    const body = {
        'query' : query,
        'fromDate' : fromDate,
        'maxResults' : maxResults
    };
    if (next) {
        body.next = next;
    }

    try {
        const response = await fetch(url, {
            method: 'POST',
            headers: {
            'Authorization' : 'Bearer ' + token
            },
            body: JSON.stringify(body)
        });
        if (response.ok) {
            const json = await response.json();
            return json;
        }
        else {
            let msg = (`authorization request response status: ${response.status}`);
            throw new Error(msg);    
        }
    }

Usage

let query = 'from:realDonaldTrump -RT';  //get tweets originated from Donald Trump, filter out his retweets
let url = SEARCH_URL + THIRTY_DAY_LABEL;  //30day search
let fromDate = '201910010000'; //search for tweets within the current month (currently, Oct 2019)
search(url, query, fromDate, 100)  //100 is the max results per request for the sandbox environment 
.then(total => {
    console.log('total tweets: ' + total);
})
.catch(err => {
    console.error(err);
});

Output

Snippet of the resulting JSON array from the function call above.

[
    {
        "text": "We have become a far greater Economic Power than ever before, and we are using that power for WORLD PEACE!",
        "created_at": "Sun Oct 13 14:32:37 +0000 2019"
    },
    {
        "text": "Where’s Hunter? He has totally disappeared! Now looks like he has raided and scammed even more countries! Media is AWOL.",
        "created_at": "Sun Oct 13 14:15:55 +0000 2019"
    },

Source

https://github.com/joeywhelan/twitterSearch

Copyright ©1993-2024 Joey E Whelan, All rights reserved.