21 May 2015 / Dr Ewan Kirk
Investors in general are sceptical of systematic trading. Why could this be? Long term performance seems
to indicate that the performance of models is at least as good as that of humans. So why the scepticsm?
This short piece was engendered by some recently published research which we will examine later but first let's
examine the problem.
Aversion in action
Over the years we have met many investors and allocators ranging from high net worth individuals
to sovereign wealth funds and pension funds. No two investors are identical and each potential investor's
response to the systematic trading proposition is different. However, their view is often coloured by
considerable scepticism. The questions that most investors ask about model based investment
processes are strikingly similar. How do we know that our models will work? What happens when they
go wrong? How do we know if they have gone wrong?
Furthermore, when models or strategies have a period of poor performance(footnote)
the scepticism and lack of
confidence grows quickly. Is the model broken? Has the environment changed? Is it different this time?(footnote)
Although systematic trading comes in many forms, the concern about things being broken or never working again
seem to bedevil CTAs and other macro strategies more than others.
We would be the first to agree that scepticism is an extremely good heuristic(footnote)
for investors to use when they are evaluating investment styles and funds. However, we are always somewhat taken aback at
how much stronger the scepticism is regarding systematic methods compared to other investment approaches. Surely a robust,
process which has been tested on decades of data and hundreds of different markets should be more believable than a process which depends
on the deep but often inexplicable insights of humans which are prone to well known behavioural biases and errors?
Well, much to our relief, it turns out that the bias against systematic processes and algorithms is a bias in
its own right! In Algorithm Aversion: People
Erroneously Avoid Algorithms after Seeing Them Err"
Dietvorst, Simmons and Massey perform clever experiments to
evaluate their subjects' reactions to both the performance of algorithms and also their reactions when algorithms
appear to underperform human forecasters.
We know that most of the readers of this post will have downloaded the paper and read it thoroughly before
continuing but for those that haven't, we have summarised the paper below.
Before starting the analysis of the results, the authors perform a little thought experiment(footnote)
. Imagine you are driving to work and you unexpectedly encounter a traffic jam. You
decide (predict) that an alternate route will be quicker. Arriving at work 20 minutes later, your co-worker tells you
that you mis-predicted the effect of the traffic and in fact it would have been fine on your normal route. This has happened to us all
whether or not it is driving or deciding to take a bus or the tube. When this happens, you are unlikely to never
trust your judgement of traffic conditions in the future.
However, imagine the scenario if your traffic-aware
had rerouted you to
avoid a traffic jam but it turned out that the traffic cleared more quickly than expected. Many of us would lose confidence in the routing
and would become
more reluctant to trust it in the future.
To quote the authors: “It seems that errors that we tolerate in humans become less tolerable when
machines make them.”(footnote)
The authors perform experiments on undergraduate students and use Amazon's Mechanical Turk
to supply a second cohort. Simplifying a little, both cohorts have an economic incentive to undertake a forecasting
task well. They are allowed either to use a human forecast or an algorithmic forecast. The participants are told that the algorithm has been
developed using statistical techniques. The participants are given some experience in either a human forecast method (themselves) or the
algorithm. They then have to choose what they will use to forecast during the period of the experiment when they will be paid on the basis of
The cunning psychologists arranged the results so that the model outperformed the participants' forecasts by a considerable
margin. Cleverly, the model didn't always
outperform the human forecasters, just most
of the time.
Despite seeing the model outperform in most cases, the subjects chose to bet on the human forecasts much more than one would expect
if they were being rational. Also, those people who selected the model over humans were much quicker to drop the model
predictions when they saw the algorithm make an error compared to when they experienced the human forecaster make an error. The
graph below shows the proportion of the subjects choosing the statistical model both in the case where they did not see the model perform and where they
This is a very surprising result. For example, those people who were more confident in a human would still choose the model 33%
of the time if they hadn't seen the model perform. But if they actually had seen the model perform the number choosing the model
dropped to 10%. This is even though they presumably could have worked out that the model outperformed. Not only did people choose
their own or human forecasts more ex-ante
when they saw a model perform better than a human, they then chose it even less.
It appears that the participants in the experiments over weighted the errors that the model made. Clearly if the model predicted perfectly
then it would take an outstandingly stubborn or luddite human to believe that their own views were better than a perfect machine. But, of course,
no statistical model is perfect and when a good model errs, it is significantly more penalised than a human who errs.
How does this affect investors?
As developers of algorithmic rules for trading, this is fascinating research. It appears that not only do human investors
suffer significant behavioural biases(footnote)
but that there is a significant bias against algorithms themselves.
It is hard to see how one can guard against this bias when choosing an investment. As we said at the outset, surely a rule
or process which can be demonstrated to have worked over 30 years of data and hundreds of markets should be more compelling
than a discretionary track record which at most will span 10 or 15 years? When we have been discussing the performance of
CTAs in the 2011 to 2014 period, we have often mentioned that the track record of CTAs looks good relative to
equities which are the cornerstone of almost all investment portfolios. Discretionary hedge funds are not
simple long only equities positions(footnote)
but often the track record of the manager is shorter than a decade and ex-ante
assume that investors would be more sceptical of discretionary styles of management compared
to systematic styles of investment(footnote)
. Logically, this
should be true but it may be that
“algorithm aversion” is misleading investors into being more sceptical of systematic trading than is
justified both by the historical statistics and the realistic ex-ante
expectations. In addition, investors
are prone to giving up on systematic investment styles more quickly than they should, compared to discretionary styles.
This is likely to have a negative impact on investors' portfolios in the long term since the investors are not only giving
up their high water marks in drawdowns but their portfolios also become sub-optimal because a valuable diversifying source
of return has been removed.