A Basic Stock Trading Backtesting System for F# using Ta-Lib and FSharp.Data
- Mar 8, 2014
This article is written for the intermediate F# audience who has a basic familiarity of stock trading and technical analysis, and is intended to show the basics of implementing a backtesting system in F#.
If you’re an F# beginner it may not take too long for you to get up to speed on the concepts if you check out a few resources. In particular:
The backtesting strategy which you will implement is a simple SMA crossover where the fast line is 10 bars and the slow one 25. When the fast line crosses above the slow line, a buy signal will be triggered, and when the fast line cross below the slow line, a sell signal will be triggered. Only one long position will exist at any time, so the system will not trigger another buy order until after the long position is sold.
Feel free to get in touch and send through any improvements, questions or corrections.
Starting Out
Both Ta-Lib and FSharp.Data are available on NuGet, so in the Package Manager Console, execute
Install-Package FSharp.Data
and
Install-Package TA-Lib
Create a new fsx file in the project, and call it SimpleBacktest.fsx
This post will be structured in source code chunks, and can be consecutively paste them into the fsx file.
#r "System.Data.Entity.dll" #r "FSharp.Data.TypeProviders.dll" #r "System.Data.Linq.dll" #r @"C:\Source\StockScreener\packages\FSharp.Data.2.0.3\lib\net40\FSharp.Data.dll" #r @"C:\Source\StockScreener\packages\TA-Lib.0.5.0.3\lib\TA-Lib-Core.dll" open System open System.Collections.Generic open System.Data open System.Data.Linq open FSharp.Data open Microsoft.FSharp.Data.TypeProviders
Visual Studio 2012 has some Intellisense issues with F# FSI files and relative reference paths, so the references to Fsharp.Data.dll and Ta-Lib-Core.dll need to be their full path. Replace C:\Source\StockScreener\ with your project path, so that the full paths point to the appropriate DLLs.
Fetching the Data
type TaLibPrepData =
{ Symbol : string;
Date : DateTime[];
Open : float[];
High : float[];
Low : float[];
Close : float[]; }
module StockData =
type Stocks = CsvProvider<AssumeMissingValues=true,IgnoreErrors=true,Sample="Date (Date),Open (float),High (float),Low (float),Close (float),Volume (int64),Adj Close (float)">
let getHistoricalData symbol (fromDate:DateTime) (toDate:DateTime) =
let url = sprintf "http://ichart.finance.yahoo.com/table.csv?s=%s&a=%i&b=%i&c=%i&d=%i&e=%i&f=%i&g=d&ignore=.csv"
symbol (fromDate.Month - 1) (fromDate.Day) (fromDate.Year) (toDate.Month - 1) (toDate.Day) (toDate.Year)
Stocks.Load(url).Rows |> Seq.toList |> List.rev
let getTaLibData symbol (fromDate:DateTime) (toDate:DateTime) =
let historicalData = getHistoricalData symbol fromDate toDate
{
Symbol = symbol;
Date = historicalData |> Seq.map (fun x -> x.Date) |> Seq.toArray;
Open = historicalData |> Seq.map (fun x -> x.Open) |> Seq.toArray;
High = historicalData |> Seq.map (fun x -> x.High) |> Seq.toArray;
Low = historicalData |> Seq.map (fun x -> x.Low) |> Seq.toArray;
Close = historicalData |> Seq.map (fun x -> x.Close) |> Seq.toArray;
} : TaLibPrepData
Methods in the TA-Lib .NET wrapper expect arrays in their arguments, the TaLibPrepData record type was created so that the data can be mapped into it’s labels, ready to be used in the TA-Lib wrapper methods.
Yahoo Finance data is fetched in CSV format, and is parsed by the FSharp.Data CSV type provider. Note that when CsvProvider is instantiated, a sample is given describing the fields in the CSV returned by Yahoo Finance along with type names in brackets. Using a type provider significantly simplifies fetching the CSV data, cutting down on the amount of boiler plate code.
The URL format is pretty cryptic, so a resource like http://www.gummy-stuff.org/Yahoo-data.htm is important if you want to figure out what the query string parameters mean.
Ta-Lib Wrapper
type TaLibOutReal =
{ OutReal : float array; OutBegIndex : int }
module TaLibWrapper =
open TicTacTec.TA.Library
// getAllocationSize returns an integer value with the exact allocation size needed for the TA-Lib outReal array. See: http://ta-lib.org/d_api/d_api.html
let getAllocationSize (lookback:int) (startIdx:int) (endIdx:int) =
let temp = Math.Max(lookback, startIdx)
if temp > endIdx then 0 else endIdx - temp + 1
// Simple Moving Average. See: http://ta-lib.org/function.html
let sma timePeriod data =
let startIdx = 0
let endIdx = data.Close.Length - 1
let mutable outBegIdx = 0
let mutable outNBElement = 0
let lookback = Core.SmaLookback(timePeriod)
let allocationSize = getAllocationSize lookback startIdx endIdx
let mutable outReal : float array = Array.zeroCreate allocationSize
let retCode = Core.Sma(startIdx, endIdx, data.Close, timePeriod, &outBegIdx, &outNBElement, outReal)
if retCode <> Core.RetCode.Success then
invalidOp (sprintf "AssertRetCodeSuccess")
{ OutReal = outReal; OutBegIndex = outBegIdx }
TaLibWrapper.sma returns SMA data where the timePeriod is in bars. For this example, the SMA is calculated over the bar’s closing price. startIdx and endIdx define the range of the data.Close array that the SMA will be calculated over. In this case, we’re processing the entire array of closing prices.
The variable names such as outReal, outNBElement are intended to be consistent with the TA-Lib source code and examples. outBegIndex represents the bar offset of outReal[0] – i.e. if the SMA has a timePeriod of 10 bars, 10 leading bars of data are needed before the first SMA value can be calculated, so outReal[0] will be the SMA at bar 10. The value of outBegIndex is returned along with outReal as both are used later in this example.
In order to avoid allocating an outReal array any larger than necessary, we calculate the allocation size via getAllocationSize. If we didn’t calculate the allocation size needed for the outReal array, there would be an extra 10 elements at the end of the array.
Order Management
type OrderType =
| Buy
| Sell
type Order =
{ Date : DateTime;
Symbol : string;
Type : OrderType;
Quantity : int;
Price : double; }
member this.Total = this.Price * double this.Quantity
let ninjaTraderDaysOffset = 2.0
let ninjaTraderBarOffset = 1
let createOrder orderType quantity (data : TaLibPrepData) bar =
{ Order.Date = (data.Date.[bar]).AddDays(ninjaTraderDaysOffset); Symbol = data.Symbol; Type = orderType; Quantity = quantity; Price = data.Open.[bar+ninjaTraderBarOffset]; }
type OrderMatch = { Buy:Order; Sell:Order; }
let getOrderMatches (orders:Order seq) =
if Seq.exists (fun (x:Order) -> x.Quantity <> (Seq.head orders).Quantity) orders then
invalidOp "All order quantities must match"
let buyOrders = Seq.filter (fun x -> x.Type = OrderType.Buy) orders
let sellOrders = Seq.filter (fun x -> x.Type = OrderType.Sell) orders
if (Seq.length buyOrders <> Seq.length sellOrders) then
invalidOp "The number of buy orders must match the number of sell orders"
Seq.zip buyOrders sellOrders |> Seq.map (fun x -> { OrderMatch.Buy = fst x; Sell = snd x } ) |> Seq.toList
getOrderMatches matches buy and sell orders and returns a list of trades. The implementation is a simple one and only supports matching where the buy and sell quantity both match. In other backtesting scenarios, the user may want to queue up multiple buy orders, then make a large sell order, or vice versa – getOrderMatches could be extended to support this.
During development, I found it useful to run the same backtesting strategy in NinjaTrader and compare the results for correctness. The values of ninjaTraderDaysOffset and ninjaTraderBarOffset were used in order to get results which matched the output of NinjaTrader. See the section The Equivalent Ninjatrader Strategy below for further details.
The basic strategy for matching orders is to split the buy and sell orders into two separate lists, then use Seq.zip to match them together. The orders are already sorted by date. Note that Seq.zip returns a tuple, where fst x is the first element and snd x is the second element.
Trade Management
type Trade =
{ Symbol : string; Quantity : int;
EntryDate : DateTime; EntryPrice : double;
ExitDate : DateTime; ExitPrice : double;
CumulativeProfit : double; }
member this.EntryTotal = this.EntryPrice * double this.Quantity
member this.ExitTotal = this.ExitPrice * double this.Quantity
member this.Profit = (this.ExitPrice - this.EntryPrice) / this.EntryPrice
let createTrade (buy:Order) (sell:Order) (previousTrade:Trade option) =
if (buy.Quantity <> sell.Quantity)
then invalidOp "All order quantities must match"
let Profit = (sell.Price - buy.Price) / buy.Price;
let cumulativeProfit =
match previousTrade with
| Some previousTrade -> (1.0 + previousTrade.CumulativeProfit) * ( 1.0 + Profit) - 1.0
| None -> Profit
{ Trade.Symbol = buy.Symbol; Quantity = buy.Quantity;
EntryDate = buy.Date; EntryPrice = buy.Price;
ExitDate = sell.Date; ExitPrice = sell.Price;
CumulativeProfit = cumulativeProfit; }
let getTrades (orderMatches:OrderMatch list) =
let rec getRemainingTrades (lastTrade:Trade) (matches:OrderMatch list) (acc:Trade list) =
match matches with
| [] -> List.rev acc
| x::xs ->
let trade = createTrade x.Buy x.Sell (Some lastTrade)
getRemainingTrades (trade) xs (trade::acc)
let firstTrade = createTrade orderMatches.Head.Buy orderMatches.Head.Sell None
firstTrade :: getRemainingTrades firstTrade orderMatches.Tail []
There are two sides to a trade, so an assertion is made at the beginning of createTrade to ensure the buy and sell quantity both match.
In order to calculate the cumulative profit, I need to know the cumulative profit of the previous trade. For the first trade, the cumulative profit is the same as profit, however for subsequent trades, the cumulative profit is calculated via (1.0 + previousTrade.CumulativeProfit) * ( 1.0 + Profit) – 1.0 the result is a percentage value i.e. 0.13 = 13%
The calculation of the cumulative profit is done via tail recursion and an accumulator (acc:Trade list)
Trading Performance
type PerformanceSummary =
{ Symbol : string;
StartDate : DateTime;
EndDate : DateTime;
TotalNetProfit : double;
GrossProfit: double;
GrossLoss : Double;
TotalNumberOfTrades : int;
WinningTrades : int;
LosingTrades : int;
CumulativeProfit : double; }
module TradingPerformance =
let profitOrLoss = (fun (x:Trade) -> x.ExitTotal - x.EntryTotal)
let totalNetProfit (trades:Trade list) = trades |> List.sumBy profitOrLoss
let winningTrade = (fun (x:Trade) -> x.ExitPrice > x.EntryPrice)
let losingTrade = (fun (x:Trade) -> x.ExitPrice < x.EntryPrice)
let getGrossProfit (trades:Trade list) = trades |> List.filter winningTrade |> List.sumBy profitOrLoss
let getGrossLoss (trades:Trade list) = trades |> List.filter losingTrade |> List.sumBy profitOrLoss
let getPerformanceSummary (trades:Trade list) =
let firstTrade = Seq.head trades
let lastTrade = Seq.last trades
{ PerformanceSummary.Symbol = firstTrade.Symbol;
StartDate = firstTrade.EntryDate; EndDate = lastTrade.EntryDate;
TotalNetProfit = totalNetProfit trades; GrossProfit = getGrossProfit trades; GrossLoss = getGrossLoss trades;
TotalNumberOfTrades = List.length trades; WinningTrades = List.filter winningTrade trades |> List.length; LosingTrades = List.filter losingTrade trades |> List.length;
CumulativeProfit = lastTrade.CumulativeProfit}
Strategy Implementation
let barValue (data:TaLibOutReal) bar barsAgo =
data.OutReal.[bar - barsAgo - data.OutBegIndex]
let crossAbove (series1:TaLibOutReal) (series2:TaLibOutReal) bar lookback =
(barValue series1 bar lookback) <= (barValue series2 bar lookback) && (barValue series1 bar 0) > (barValue series2 bar 0)
let crossBelow (series1:TaLibOutReal) (series2:TaLibOutReal) bar lookback =
(barValue series1 bar lookback) >= (barValue series2 bar lookback) && (barValue series1 bar 0) < (barValue series2 bar 0)
module SmaCrossoverStrategy =
let getOrders symbol (fromDate:DateTime) (toDate:DateTime) fastPeriod slowPeriod =
let data = StockData.getTaLibData symbol fromDate toDate
let smaFast = TaLibWrapper.sma fastPeriod data
let smaSlow = TaLibWrapper.sma slowPeriod data
let quantity = 100
let tradeIsOpen = ref false
seq {
for bar in smaSlow.OutBegIndex + 1 .. smaSlow.OutReal.Length-1 do
if (!tradeIsOpen = false) && (crossAbove smaFast smaSlow bar 1) then
tradeIsOpen := true
yield (createOrder OrderType.Buy quantity data bar)
else if (!tradeIsOpen = true) && (crossBelow smaFast smaSlow bar 1) then
tradeIsOpen := false
yield (createOrder OrderType.Sell quantity data bar)
if !tradeIsOpen = true then
let bar = smaSlow.OutReal.Length-1
yield (createOrder OrderType.Sell quantity data bar)
}
The SmaCrossoverStrategy should be fairly self explanatory and is written in an imperative programming style. getOrders iterates over each bar, checking for SMA crossover, and creates an order if there is no current open trade.
Finally, after all the bars have been processed, if there is a trade still open, a Sell order is created with the price at the final bar in the data series.
Displaying Backtesting Performance
let fromDate = new DateTime(2000, 1, 1)
let toDate = new DateTime(2003, 1, 1)
let symbol = "ANZ.AX"
let fastPeriod = 10
let slowPeriod = 25
let orders = SmaCrossoverStrategy.getOrders symbol fromDate toDate fastPeriod slowPeriod |> Seq.toList
let orderMatches = getOrderMatches orders
let trades = orderMatches |> getTrades |> Seq.toList
let tradingPerformance = TradingPerformance.getPerformanceSummary trades
let asCurrency (v:double) = v.ToString("$0.00")
let asIsoDate (v:DateTime) = v.ToString("yyyy-MM-dd")
let asPercentage (v:double) = v.ToString("0.00%")
let printTrade tradeNumber trade =
printfn "%7i | %8i | %-11s | %10s | %10s | %s | %6s | %10s" (tradeNumber+1) trade.Quantity (asCurrency trade.EntryPrice) (asCurrency trade.ExitPrice) (asIsoDate trade.EntryDate) (asIsoDate trade.ExitDate) (asCurrency trade.Profit) (asPercentage trade.CumulativeProfit)
The format specifiers in the printfn statement such as %7i are for controlling width and alignment so that you’ll get a nicely formatted table
If you then then run the following code:
printfn "Trade-# | Quantity | Entry Price | Exit Price | Entry Date | Exit Date | Profit | Cum.Profit " trades |> Seq.iteri (printTrade)
You should see the following output:
Trade-# | Quantity | Entry Price | Exit Price | Entry Date | Exit Date | Profit | Cum.Profit
1 | 100 | $10.30 | $11.64 | 2000-03-15 | 2000-05-13 | $0.13 | 13.01%
2 | 100 | $12.20 | $12.50 | 2000-06-01 | 2000-06-30 | $0.02 | 15.79%
3 | 100 | $12.42 | $12.79 | 2000-07-07 | 2000-09-22 | $0.03 | 19.24%
4 | 100 | $13.51 | $14.25 | 2000-10-06 | 2000-12-22 | $0.05 | 25.77%
5 | 100 | $15.07 | $14.53 | 2001-02-03 | 2001-03-17 | -$0.04 | 21.26%
6 | 100 | $13.65 | $13.70 | 2001-04-18 | 2001-04-19 | $0.00 | 21.71%
7 | 100 | $14.00 | $14.04 | 2001-04-27 | 2001-05-02 | $0.00 | 22.05%
8 | 100 | $14.10 | $15.58 | 2001-05-04 | 2001-07-12 | $0.10 | 34.87%
9 | 100 | $16.46 | $16.00 | 2001-08-05 | 2001-09-14 | -$0.03 | 31.10%
10 | 100 | $16.95 | $17.56 | 2001-10-07 | 2001-11-30 | $0.04 | 35.81%
11 | 100 | $18.57 | $17.07 | 2001-12-27 | 2002-01-16 | -$0.08 | 24.84%
12 | 100 | $17.60 | $17.65 | 2002-02-06 | 2002-03-15 | $0.00 | 25.20%
13 | 100 | $17.86 | $18.02 | 2002-04-11 | 2002-04-14 | $0.01 | 26.32%
14 | 100 | $18.33 | $19.00 | 2002-04-19 | 2002-05-30 | $0.04 | 30.94%
15 | 100 | $19.38 | $19.35 | 2002-06-01 | 2002-06-28 | $0.00 | 30.74%
16 | 100 | $18.65 | $17.60 | 2002-08-18 | 2002-09-29 | -$0.06 | 23.37%
17 | 100 | $18.50 | $18.50 | 2002-10-27 | 2002-11-22 | $0.00 | 23.37%
The Equivalent Ninjatrader Strategy
The output you get from the above backtesting system should give similar output to NinjaTrader’s backtest a strategy feature assuming you only buy long, and maintain one trade at a time.
You can use the following strategy in NinjaTrader, and if you configure the strategy in the same way as the F# code, the Trades tab will show the same output.
#region Using declarations
using System;
using System.ComponentModel;
using System.Diagnostics;
using System.Drawing;
using System.Drawing.Drawing2D;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Data;
using NinjaTrader.Indicator;
using NinjaTrader.Gui.Chart;
using NinjaTrader.Strategy;
#endregion
// This namespace holds all strategies and is required. Do not change it.
namespace NinjaTrader.Strategy
{
/// <summary>
/// Enter the description of your strategy here
/// </summary>
[Description("Enter the description of your strategy here")]
public class SmaCrossoverStrategy : Strategy
{
#region Variables
private int fast = 10;
private int slow = 25;
#endregion
/// <summary>
/// This method is used to configure the strategy and is called once before any strategy method is called.
/// </summary>
protected override void Initialize()
{
SMA(Fast).Plots[0].Pen.Color = Color.Orange;
SMA(Slow).Plots[0].Pen.Color = Color.Green;
Add(SMA(Fast));
Add(SMA(Slow));
CalculateOnBarClose = true;
}
/// <summary>
/// Called on each bar update event (incoming tick).
/// </summary>
protected override void OnBarUpdate()
{
if (CrossAbove(SMA(Fast), SMA(Slow), 1))
EnterLong();
else if (CrossBelow(SMA(Fast), SMA(Slow), 1))
ExitLong();
}
#region Properties
/// <summary>
/// </summary>
[Description("Period for fast MA")]
[GridCategory("Parameters")]
public int Fast
{
get { return fast; }
set { fast = Math.Max(1, value); }
}
/// <summary>
/// </summary>
[Description("Period for slow MA")]
[GridCategory("Parameters")]
public int Slow
{
get { return slow; }
set { slow = Math.Max(1, value); }
}
#endregion
}
}
Where to next?
This sample is intended to show the basics of implementing a backtesting system in F#. There are many ways this can be extended:
- Extend the Ta-Lib wrapper to support ADX, Beta, EMA, MACD, etc. The TA-Lib wrapper could be made a lot more generic to reduce duplicate code when adding support for these methods.
- Extend the order matching system to handle orders of different quantities
- Calculate the performance summary (with beta, # winning trades, # losing trades, etc)
- The system above only allows one long position at a time, extend the system to allow incremental buying based on a pre-set account size.
- Implement other strategies such as:
- Buying when the 50 day EMA crosses above the 200 day EMA and the current price is above the 50 day EMA, sell when the price drops below the 200 day EMA.
- Buying on dips only when the stock is an uptrend and the dip is within 1% of the 200 day EMA
