Python

The low learning curve Python programming language has grown in popularity over the past decade. Data Scientists, algorithmic developers, quantitative financial professionals, and market enthusiasts have helped this become a strong tool for algorithmic research, development, and trading. Python for the trading industry comes with tools including:

  • Jupyter notebooks
  • NumPy for High-Speed Numerical Processing
  • Pandas for Efficient Data Analysis and Time Series Analysis Techniques
  • Matplotlib for Data Visualization
  • TA-Lib for Technical Analysis
  • Tensor flow

Posts

Yahoo Finance ORCL - Profile Tab

Web Scraping a Stock Symbol’s URL using Yahoo Finance with Python for Alternative Data Links

Alternative Data is growing as a necessary weapon for traders and quantitative investors. Yet there are many barriers to alternative data success.

According to Greenwich Associates, some of the barriers include:

  • Data incompatibility
  • Poor or ineffective Data Sources
  • Human Capital Required

Overcoming Some of these Barriers with Data or Symbology Mapping

Symbology Maps

Every trading firm has some form of symbology map. Most cover the basic use cases for their specific trading. For example, one firm may have maps of Reuters Identification Codes (RICS) to their internally used maps. It is all that firm needs, and therefore that is where most trading firms stop.

Expanding Symbology Maps to Cover the Need of Alternative Data

Data incompatibility for traders and quant researchers begins with understanding the linkage between data sets. The data in the Alternative Dataset most likely is missing the trading symbol that could be used to link the data to the stock market data.

For example, the alternative data may contain a reference to the company name, or to a web site. Consider that your altdata has “www.oracle.com.” The data incompatibility begins with a missing map that takes this URL and maps it to “ORCL,” the trading symbol used by traders. Both the URL and the trading Symbol map to a company name “Oracle Corporation.”

This is a non-traditional Symbology Map.

www.oracle.com = ORCL.

Your alternative data may need other non-standard symbology mappings. Consider some other mappings for example:

  • Corporate Officers
  • Major Vendors
  • Raw Materials
  • Financial Analysts

Symbology maps are needed to overcome AltData barriers. They directly address issues with data incompatibility thereby improving ineffective data sources and decreasing the demand on

Building A URL to Symbology Map

The following Python code can be used to build this non-traditional symbol to URL map. This code (written in Python 3) uses web scraping and basic logic to pull the URL from the Yahoo Finance “profile” page/tab to get the URL for a given list of stock symbols.

Keep in mind – If you use a web service like Yahoo Finance make sure you review their terms of service in collecting data from Yahoo (or any other) site. Be fair. Be Ethical.

Yahoo Finance ORCL - Profile Tab
Yahoo Finance ORCL – Profile Tab

The following code was written using JupyterLab on CloudQuant.ai – the advanced data science platform provided by CloudQuant.

The code loops through a list of symbols [‘SBUX’, ‘MET’, ‘CAT’, ‘JNJ’, ‘ORCL’] and reads the Yahoo finance page to build a map of symbols to company URL.

import bs4 as BeautifulSoup 
from bs4 import SoupStrainer
import re
import urllib.request 
import pandas as pd
import requests
symbols = ['SBUX', 'MET', 'CAT', 'JNJ', 'ORCL']

headers = {'User-agent': 'Mozilla/5.0'}
mySymbols = {}

# looping through all my symbols
for s in symbols:
    vals = {}
    #getting the symbol "profile" from Yahoo finance.
    # The url for the stock appears on this page.
    url = ("https://finance.yahoo.com/quote/{}/profile?p={}".format(s,s))
    webpage = requests.get(url, headers=headers)
    soup = BeautifulSoup.BeautifulSoup(webpage.content) 

    # the title has the company name but has additional information in the format of 
    # (SSS) profile and .... 
    # where SSS is the symbol. We remove this extra title to get the company name.
    title = soup.find("title")
    tmp = title.get_text()
    rxTitle = re.compile(r'\(.*$')
    coName = rxTitle.sub("", tmp)
    
    # looping through all the links in the document.
    # The company web site is the the one that doesn't have yahoo in the reference,
    # and has a blank title.
    for link in soup.find_all('a', href=True):
        try:
            if link['target'] and "" == link['title']:
                m = re.search('yahoo', link['href'], flags=re.IGNORECASE)
                if None == m:
                    
                    url = link['href']
                    webpage = requests.get(url, headers=headers)
                    soup = BeautifulSoup.BeautifulSoup(webpage.content) 
                    
                    vals = {"company":coName, "url":link['href']} 
                    print (s, vals)
                    mySymbols[s] = vals
        except:
            pass

Output

The output from this portion of the script will produce the console prints.

Console Output is OK, but…

Outputting this data to the console is helpful, mainly to easily see what is happening in my code. But what I really want is a Pandas data frame that will allow me to sort, filter, and save the data into a CSV. To put the collected data map into a dataframe I simply use the following code.

#placing my data into a data frame
df = pd.DataFrame.from_dict(mySymbols, orient = 'index')
df

Dataframe Output


Author: Tayloe Draughon, Senior Product Manager, CloudQuant

backtest chart

Conversations: What we wish we knew when we started AlgoTrading

, ,
CloudQuant’s portfolio managers and quantitative algo traders look back on their starts in Algorithmic Trading. This candid overview allows everyone to see the “Things We Wish We Knew When We Started AlgoTrading”.  This is a short collection of the interviews with some of our amazing coders here in the office
Trevor Trinkino Quantitative Trader

FXCM Machine Learning with Trevor Trinkino

On February 8th Trevor Trinkino presented Machine Learning with FXCM in a webinar. During this presentation, he promised to make available his machine learning Python Notebook and the supporting data file. These are available on our Google drive at:
Young women getting started in technology

Young Women: Programming Helps!

,
The first book I got was “Hello World”. The book intimidated me, because I saw programming as a superpower, something only my dad was capable enough to do. I tried to learn it, but I just got bored very quickly. My dad, still persistent, decided it was time to try programming something we loved.
Backtest Research Life Cycle for Trading Strategies

Backtesting Trading Strategies

If you knew your trading strategy would work 50% of the time, would you commit your scarce savings to trade it? What if it worked 75% of the time? Backtesting gives one the confidence to know that your trading strategy will work.
Three Inside Up Technical Pattern shown on $IBM Jan 2, 2018

TA-LIB Three Inside Up Buy Signal – $IBM

,
IBM, the 100-year-old company, is wedging into a tight trade, but it looks like the Bulls are gearing to press a move higher. Just before year-end, the stock set a Three Inside Up Japanese candlestick pattern setting the stage for a rally. Source code using TA-LIB and Python included.
Harami Sell Signal on Google Dec 20, 2017

Harami Sell Signal with Three Inside Down Demonstrated on $GOOG

,
December 19, 2017 Monday Google pressed new highs, but Tuesday closed out the day with a Harami Sell signal. Watch out for today. A negative close today boosts the negative outlook with the emergence of a Three Inside Down pattern. In this event, it will most likely mean that there will be a “little coal in Google stockholders stockings for Christmas”.
RSI

Technical Analysis Library (TA-LIB) for Python Backtesting

Anyone who has ever worked on developing a trading strategy from scratch knows the huge amount of difficulty that is required to get your logic right. … TA-LIB Turbo-Charges Your Research Loop: TA-Lib is widely used by quantitative researchers and software engineers developing automated trading systems and charts. This freely available tool allows you to gather information on over 200 stock market indicators.
Candlestick market data chart

Understanding Candlestick Bars & Market Data for Beginning Algo Programmers

In this video, we introduce you to Candlestick Bars, a store of Historic Market Data, how to access that data via Pythons Lists and how pointers work in lists.
Trend Analysis in a Candlestick Market Data Chart

ZigZag Strategy Suggestion from Quora

A suggested a Zig-Zag trading strategy that bounces back and forth on the stock market to make small profits. Testing shows the strategy wouldn’t work.