The past is prologue:  ETF backtesting basics

The ETF business is booming, going from $420 billion at the start of 2007 to $608 billion at its end, a 45% increase.  270 ETFs made their debut in 2007, bring the total number currently traded to 629. (To clarify, in 2006 158 ETFs were launched; in 2005, that number was only 52).

One of the more interesting points to consider is that the “specialty – domestic” ETF style, that is, ETFs which specialize in fundamentally weighted indexes, inverse indexes, etc., had the greatest amount of increase in assets.  Fundamentally weighted ETFs, along with the new quantitative ETFs (via PowerShares) are all new, and do not have the benefit of tracking an index as old or well known as the S&P 500 (Ticker: SPY). 

What many of the ETF providers do is “backtest” their investment strategy, which means to take historical data, apply their trading strategy, and relate their returns to an comparable benchmark.  In a recent meeting with an ETF marketing rep, he did confess that few see a backtest that one would “not like”. 

Backtesting is a form of “statistical marketing”, and while it is indeed an important function of funds to show trading metrics, especially in a passively managed strategy, the average investor should be aware of what backtests entail, how they can differ from actual returns, and how one can objectively consider them.

As stated above, ETFs are booming, but this is a recent phenomenon.  The ETF explosion is due to the investor clamor for more esoteric investment vehicles, and this creates the problem that these esoteric strategies do not typically have a usable index preexisting, at least not over a meaningful time frame. Enter the backtest, which uses historical data, trades, and rebalancing to mimic an index. 

There is something I have noticed while reviewing ETFs which all investors should be aware: on many ETF factsheets, you will see historical returns.  Many of these returns are for the index, which is hypothetical, and certainly not the actual returns of the fund. 

Take for example the Powershares Valueline Timeliness Select ETF, (Ticker: PIV, we personally love Valueline’s service, research materials and analysis). Valueline ranks a company on “timeliness” which is a proprietary review of its fundamentals, its safety and its ability to outperform the general indexes. They then use this information to create an index, and then buy and sell equities to this strategy. If one were to review the index’s returns, one would be star struck. The index beats the S&P 500 as well as the Russell 2000 growth (tracked by; Ticker: IWO) handily over a 1, 3, 5, and 10 year period. Based on backtested data.  

But the reality is that the fund has only existed since 2006, and the results are far from that consistent.  For 2006, the fund returned 4.7%, underperforming the S&P 500 by almost 11%. For 2007 it then beat the S&P 500 by almost 11%, returning 16.4%. Through 2/06/08 of this wild year, the fund is down almost 8% more than the S&P 500, (18%) year to date. Now, granted this fund is still young, but whom can we trust? The reputation of Valueline? The backtest? The stocks in the index? 

Backtests share common statistical metrics (Standard Deviation, Sharpe Ratio, Beta, Alpha, R-Squared, Maximum up/down draw), all of which I will describe in detail in my next research article (set a reminder below). The point of these metrics are to explain how the hypothetical index performs, and more importantly, how much risk is associated with the backtest.

Below is an example of a benchmark, with some data taken out for copyright purposes:

Set a reminder for the follow up article:

>> RSS - My Yahoo
>> RSS - Google
>> RSS - Newsgator
>> RSS - Bloglines