Python

The low learning curve Python programming language has grown in popularity over the past decade. Data Scientists, algorithmic developers, quantitative financial professionals, and market enthusiasts have helped this become a strong tool for algorithmic research, development, and trading. Python for the trading industry comes with tools including:

  • Jupyter notebooks
  • NumPy for High-Speed Numerical Processing
  • Pandas for Efficient Data Analysis and Time Series Analysis Techniques
  • Matplotlib for Data Visualization
  • TA-Lib for Technical Analysis
  • Tensor flow

Posts

Yahoo Finance ORCL - Profile Tab

Web Scraping a Stock Symbol’s URL using Yahoo Finance with Python for Alternative Data Links

Alternative Data is growing as a necessary weapon for traders and quantitative investors. Yet there are many barriers to alternative data success.

According to Greenwich Associates, some of the barriers include:

  • Data incompatibility
  • Poor or ineffective Data Sources
  • Human Capital Required

Overcoming Some of these Barriers with Data or Symbology Mapping

Symbology Maps

Every trading firm has some form of symbology map. Most cover the basic use cases for their specific trading. For example, one firm may have maps of Reuters Identification Codes (RICS) to their internally used maps. It is all that firm needs, and therefore that is where most trading firms stop.

Expanding Symbology Maps to Cover the Need of Alternative Data

Data incompatibility for traders and quant researchers begins with understanding the linkage between data sets. The data in the Alternative Dataset most likely is missing the trading symbol that could be used to link the data to the stock market data.

For example, the alternative data may contain a reference to the company name, or to a web site. Consider that your altdata has “www.oracle.com.” The data incompatibility begins with a missing map that takes this URL and maps it to “ORCL,” the trading symbol used by traders. Both the URL and the trading Symbol map to a company name “Oracle Corporation.”

This is a non-traditional Symbology Map.

www.oracle.com = ORCL.

Your alternative data may need other non-standard symbology mappings. Consider some other mappings for example:

  • Corporate Officers
  • Major Vendors
  • Raw Materials
  • Financial Analysts

Symbology maps are needed to overcome AltData barriers. They directly address issues with data incompatibility thereby improving ineffective data sources and decreasing the demand on

Building A URL to Symbology Map

The following Python code can be used to build this non-traditional symbol to URL map. This code (written in Python 3) uses web scraping and basic logic to pull the URL from the Yahoo Finance “profile” page/tab to get the URL for a given list of stock symbols.

Keep in mind – If you use a web service like Yahoo Finance make sure you review their terms of service in collecting data from Yahoo (or any other) site. Be fair. Be Ethical.

Yahoo Finance ORCL - Profile Tab
Yahoo Finance ORCL – Profile Tab

The following code was written using JupyterLab on CloudQuant.ai – the advanced data science platform provided by CloudQuant.

The code loops through a list of symbols [‘SBUX’, ‘MET’, ‘CAT’, ‘JNJ’, ‘ORCL’] and reads the Yahoo finance page to build a map of symbols to company URL.

import bs4 as BeautifulSoup 
from bs4 import SoupStrainer
import re
import urllib.request 
import pandas as pd
import requests
symbols = ['SBUX', 'MET', 'CAT', 'JNJ', 'ORCL']

headers = {'User-agent': 'Mozilla/5.0'}
mySymbols = {}

# looping through all my symbols
for s in symbols:
    vals = {}
    #getting the symbol "profile" from Yahoo finance.
    # The url for the stock appears on this page.
    url = ("https://finance.yahoo.com/quote/{}/profile?p={}".format(s,s))
    webpage = requests.get(url, headers=headers)
    soup = BeautifulSoup.BeautifulSoup(webpage.content) 

    # the title has the company name but has additional information in the format of 
    # (SSS) profile and .... 
    # where SSS is the symbol. We remove this extra title to get the company name.
    title = soup.find("title")
    tmp = title.get_text()
    rxTitle = re.compile(r'\(.*$')
    coName = rxTitle.sub("", tmp)
    
    # looping through all the links in the document.
    # The company web site is the the one that doesn't have yahoo in the reference,
    # and has a blank title.
    for link in soup.find_all('a', href=True):
        try:
            if link['target'] and "" == link['title']:
                m = re.search('yahoo', link['href'], flags=re.IGNORECASE)
                if None == m:
                    
                    url = link['href']
                    webpage = requests.get(url, headers=headers)
                    soup = BeautifulSoup.BeautifulSoup(webpage.content) 
                    
                    vals = {"company":coName, "url":link['href']} 
                    print (s, vals)
                    mySymbols[s] = vals
        except:
            pass

Output

The output from this portion of the script will produce the console prints.

Console Output is OK, but…

Outputting this data to the console is helpful, mainly to easily see what is happening in my code. But what I really want is a Pandas data frame that will allow me to sort, filter, and save the data into a CSV. To put the collected data map into a dataframe I simply use the following code.

#placing my data into a data frame
df = pd.DataFrame.from_dict(mySymbols, orient = 'index')
df

Dataframe Output


Author: Tayloe Draughon, Senior Product Manager, CloudQuant

backtest chart

Conversations: What we wish we knew when we started AlgoTrading

, ,
CloudQuant’s portfolio managers and quantitative algo traders look back on their starts in Algorithmic Trading. This candid overview allows everyone to see the “Things We Wish We Knew When We Started AlgoTrading”.  This is a short collection of the interviews with some of our amazing coders here in the office

CloudQuant presentation for the Students of the University of Chicago Financial Program

Earlier this year Nick Schmandt, one of our Data Scientists, gave an “Introduction to CloudQuant” presentation to the Students of the Financial Program at the University of Chicago. Below is the video of his presentation. His slides are available our Google Drive and his script is available and ready to be cloned and modified by you at app.CloudQuant.com.  

Video Contents

A general overview of the CloudQuant environment is provided, starting with the basic CallBack functions (such as on_start, on_finish, and on_minute_bar). CloudQuant is a Python-based system, there are several public scripts available which can be easily modified. CQ Lite users have access to minute-by-minute stock data, and Elite users have access to Tick and Quote data as well as sentiment and other market-based factor analysis values. The backtesting engine has some of the best market-predicting logic, allowing its users to accurately predict the prices obtained when stocks are bought or sold at different times. In the video, Nick examines three different ways to purchase and sell stocks in the CloudQuant environment. The accuracy of these predictions is one of the greatest assets for the CloudQuant environment. Forums are available to discuss different technical issues and problems with the CloudQuant website, as well as request additional features.

Factoring Model using Momentum (talib.MOM) and RSI – Relative Strength Index (talib.RSI)

Towards the end of the video, Nick discusses an actual model in development using the TA-LIB library which is included in CloudQuant. A Factored Basket model is a zero beta, market-neutral strategy. This is used to insulate a trading environment from market motion by keeping equal amounts of stock invested in long and short positions by correctly selecting stocks that will outperform/underperform. Shares are rebalanced daily based on a chosen algorithms which predict success. Nick shows three Factors each of which achieve different levels of success, a successful strategy could likely be deduced by combining these algorithms or other factors.
Backtest Homepage showing P&L and Sharpe

CloudQuant rolls out Upgrades to Free Stock Market Backtesting System

,
CloudQuant, the trading strategy incubator, announces upgrades to our free stock market backtesting system. The web application allows any market enthusiasts to develop a trading strategy using easy to learn Python programming. Anyone who has ever written a spreadsheet macro or a simple program can easily use the system.
Trevor Trinkino Quantitative Trader

FXCM Machine Learning with Trevor Trinkino

On February 8th Trevor Trinkino presented Machine Learning with FXCM in a webinar. During this presentation, he promised to make available his machine learning Python Notebook and the supporting data file. These are available on our Google drive at:
Trading Strategy Scorecard from CloudQuant

One Minute Trader Podcast with Tayloe Draughon of CloudQuant

, ,
One Minute Trader with Matt Davio recently interviewed Tayloe Draughon to discuss Crowd Sourced Trading Ideas using our trading strategy incubator.
Young women getting started in technology

Young Women: Programming Helps!

,
The first book I got was “Hello World”. The book intimidated me, because I saw programming as a superpower, something only my dad was capable enough to do. I tried to learn it, but I just got bored very quickly. My dad, still persistent, decided it was time to try programming something we loved.
Backtest Research Life Cycle for Trading Strategies

Backtesting Trading Strategies

If you knew your trading strategy would work 50% of the time, would you commit your scarce savings to trade it? What if it worked 75% of the time? Backtesting gives one the confidence to know that your trading strategy will work.
Three Inside Up Technical Pattern shown on $IBM Jan 2, 2018

TA-LIB Three Inside Up Buy Signal – $IBM

,
IBM, the 100-year-old company, is wedging into a tight trade, but it looks like the Bulls are gearing to press a move higher. Just before year-end, the stock set a Three Inside Up Japanese candlestick pattern setting the stage for a rally. Source code using TA-LIB and Python included.
Harami Sell Signal on Google Dec 20, 2017

Harami Sell Signal with Three Inside Down Demonstrated on $GOOG

,
December 19, 2017 Monday Google pressed new highs, but Tuesday closed out the day with a Harami Sell signal. Watch out for today. A negative close today boosts the negative outlook with the emergence of a Three Inside Down pattern. In this event, it will most likely mean that there will be a “little coal in Google stockholders stockings for Christmas”.