Moving Averages and Harness Engineering for +33% CAGR on Portfolio Optimization
It's not the individual, but the environment, that determines it's success.
Moving averages are the first fool’s errand we make as traders. If only it was smoother, but still responsive. Predictive, not just reactive. And so many hours are lost clicking through TradingView PineScripts or MetaTrader indicators trying to find the Holy Grail amongst muck.
But maybe it’s not the moving average that is the problem. Maybe it’s the way we have yet to quantify what makes one ‘good.’ Yeah, visually it looks pretty and we can tell. But what does that really mean?
Here’s the thing:
It’s usually a matter of proper framing of a problem that leads us to the solution rather than a well thought out chase. In fact, diverse and seemingly random solutions searched in a well structured harness often do better than hand tuned optimized single functions.
And so if you’re not spending a little bit of time thinking higher than your peers, you might just be wasting your time overall.
So let’s change that and follow me as I discover the next state-of-the-art moving average with nothing more than a good harness.
Overview of Moving Averages
The most rudimentary moving average of them all, the Simple Moving Average (SMA) was created by a guy named Udny Yule back in 1909 to analyze time series data. By the mid-1900s, traders were using it to read charts and determine trends. Early inefficient markets, especially commodity markets, could be traded with great profit using this simple indicator.
Soon enough however, as markets got more efficient, traders realized that the SMA lagged too far behind the price to be useful anymore. Researchers from other domains as well in engineering, transportation, sales forecasting, and more also needed something more accurate for their own applications.
In the 1960s, a trader named Pete Haurlan applied an exponential weighting function to the SMA to create the EMA, a superior function that addressed some of the issues of lag that the original function had.
While this was a vast improvement, it was not the end-all-be-all. In a few decades, trades were back at it asking how they could make an even better moving average function with even less lag and more smoothness.
In 2005 a trader named Alan Hull created his own flavor of the indicator named the Hull Moving Average (clever). It uses a sharper correction impulse to reduce lag, which works due to the fractal nature of the markets.
It captured the perfect blend of smoothness to reactivity.
Ever since, it has been hard to beat.
But the Question Still Persists…
The Hull Moving Average isn’t a crystal ball. It’s still a reactive, backwards looking function that only has as much market information as what you put into it. Of course it’s not going to be the perfect money maker.
For that, we need to set out to design the perfect moving average; one that improves upon the central problems of moving averages in general: lag, overcorrection, and noise indicating false trends.
And to Do That Requires The Oracle
We first need to ask the question of what a ‘good’ moving average is. We can look at a chart and say ‘yes that looks better than that one.’ But in optimization and programming land, that isn’t going to cut it. We need to figure out what that “feeling” is and quantify it into some numbers.
To do this, I decided to compare each moving average against a ‘perfect’ moving average. The functions that were closest to this ‘perfect’ function would score the highest and therefore be the winners.
How would I find this perfect moving average? And if I had one already why would I need to write this blog post?
Unfortunately, I do not have such a moving average. But, I do have a filter that looks into the future to paint a line backwards that is ‘perfect’ and we can use this as our oracle (our perfect example to compare against).
Such a function is pointless to us as traders who can’t see the future. But such functions exist and are used to decompose time series data to study underlying trends. I chose the Hodrick–Prescott (HP) Filter because it works the best for me and has strong support in economics research.
As you can see, the HP Filter captures that “feeling” we are looking for. Smooth, but still captures trends. Reactive, such that simply following the slope of the curve would lead to a profitable trading strategy. This is ideal.
The Comparatives
I employed three basic tests to score each moving average:
The Lag Test
How much does a moving average lag behind the perfect oracle?
The Turning Point Test
How well does a moving average react to changes in the trend?
The Trading Test
How well does a moving average work as a basic single indicator trading strategy?
Each of these three test a different part of the function’s ability, giving us a wide sweep of metrics which we can use to generalize if one moving average is better than another.
Normalization
But first, I normalized each moving average. What do I mean by this? Well, rather than comparing HMA(9) to SMA(9), which both use 9 time steps, I wanted to compare the base function. A moving average could use more data and still spit out a curve that looks similar to another that uses less data. And so, fixing the window size for each function isn’t going to lead to a fair test.
I calibrated each function to find the window size that corresponded to the same smoothness as an SMA(50) function and used that window size for that function as the comparative default. You can read the code to see exactly how I did this.
Below are the values that were calculated for each of the baseline functions.
The Lag Test
Three symbols were used across three different markets: SPY, EURUSD, and GLD. Each function was swept across normalized window values from 8 → 96 and then the lag of each of those moving averages to the HP filter was calculated for each time step and averaged.
SMA generally performs the worst here which isn’t surprising given that it was the first function created for this task and all others attempted to improve upon it. Hull does second best to KAMA (Kaufman's Adaptive Moving Average) in this test.
The Turning Point Test
Turning points for all the symbols were calculated and the functions were applied. The confirmation delay and overshoot were averaged for every event aggregated together for each normalized timeframe. Lower confirmation delay means more reactivity of the function. Lower overshoot means the function doesn’t overestimate where the price is headed after the turning point.
In this example, we see that Hull squeaks out at lower timeframes on the confirmation delay, but comes in second to KAMA once again. But both Hull and KAMA pay for this reactivity by often overshooting drastically, which makes sense. You can’t have a free lunch.
The Trading Test
Finally, a simple strategy was simulated using each of the functions as the only indicator. Exposure for either a short-long portfolio (-1 → 1) and cash-long (0 → 1) was calculated by taking the slope of the curve applied to a rolling normalization function (tanh, clip, or rank). The result was 6 configurations per function and each function strategy only used their SMA(50) equivalent values (HMA(78) for Hull Moving Average).
Short-Long Results
This test is where Hull pulls away from the rest. It is especially good in non-trending or mean reverting markets, like EURUSD, and that gives it the leg up it needs against KAMA to pull ahead.
Long-Only Results
Similarly, long-only strategies, which generally are more stable, give Hull the advantage across both SPY and EURUSD. Hull is still performant on GLD, but does not pull ahead.
Final Scores
Taking all of the trading systems into account, Hull is the clear winner, nearly sweeping the board. That combined with its 2nd best performance against KAMA in the previous tests makes it the overall robust leader: it has the best performance with solid principles behind it.
Cool, But Where’s the Holy Grail
I said I was going to find the state-of-the-art and instead I just proved that Hull is pretty good. Right, but that is to set up this empirical harness so that we can throw any function at it and get a definitive score on how good that moving average is.
Now that the harness is complete, it is time to embark on quest to victory.
We already have our evaluation system constructed. All that is left is throwing arbitrary functions at it. So I need a source of arbitrary functions that are aligned to the task.
Per usual, I used an LLM to help me generate said functions. I first had it generate ‘families’ and then solutions under those families so that I could force a level of novelty and explore semantically different formulas than just ‘SMA plus this small change of detail’.
And while you may think that it is naive to just ‘spray and pray’ a bunch of functions from an LLM and see what works, the research actually says that novel and broad searches lead to better and more robust solutions than ones that are trained iteratively against a single optimization function. Click that link and read the paper for a fascinating brain blast.
361 Formulas vs. Hull - Round 1
Over 300 unique moving averages were developed. Each one was evaluated against the gauntlet and scored accordingly.
Note that different metrics describe different things. And so some metrics should be maximized while others minimized to calculate the score. For example, Sharpe should be higher, lag vs HP trend should be lower, etc.
You can see clear distributions around Hull with some beating them. But are those functions the same, or is one good at one metric and terrible at the rest? To determine this, I plotted the elites (best overall candidates) and marked them in squares against the baseline circles below.
The elites clearly cluster around Hull and beat it in some cases. Let’s look more closely at those elites and toss out the rest.
8 Formulas vs. Hull - Round 2
For this round, I used larger price datasets from EODHD (who graciously give me complete access for free, so check them out. They are my go-to data source due to their ease of use and value. Use my affiliate link so they know I sent you.)
I then employed the following tests:
Test A — Unseen history. EODHD daily data 2000 → mid-2021: two decades the stage-1 ranking never touched (used daily up to current day), across six assets (SPY, QQQ, IWM, TLT, GLD, USO) — dot-com bust, 2008, the 2010s bull, the 2020 crash.
Test B — Parameter sweeps. The full (smoothness × slope-window) Sharpe surface, scored by its plateau (robustness), not its peak; plus the cost capacity curve — at what transaction cost does the edge die?
Test C — Strategy construction. Three strategy templates (continuous slope-sizing, fast/slow crossover with hysteresis, multi-timescale ensemble) — an MA that wins as a slope signal can lose as a crossover system.
Statistical honesty. Block-bootstrap confidence intervals on every headline Sharpe, because a 21-year point estimate still has error bars.
Transaction fees are modeled at 5 basis points.
Test A
The Stage 1 trading strategy simulation was recreated and transaction fees were applied. Each function was averaged across a 6-asset portfolio and the tanh long-only slope rule was used, as it showed the most stability across all the function previously as a trading strategy test.
Reflux came out as the winner on Sharpe with total return.
Test B
The Sharpe ratio was calculated on average for every function at different transaction costs to see which ones held up the best against fees. Reflux did the best here, maintaining performance across the board besides early on from fees up to 5 basis points.
Below is a sweep of the Sharpe surfaces for each of the elites where higher general Sharpe ratios in all cases means more robust performance of the function.
Reflux is the clear winner here again with plateau reaching 40% meaning that it is highly robust in all cases.
Test C
In the strategy construction test, we again see Reflux as the winner on both continuous slope-sizing (Stage-1 strategy test) and a multi-timescale ensemble to test variability of each function.
Final Scores
Reflux beats Hull in 5 out of 6 tests, more than all other elites, making it the clear winner. This proves that the harness works and that we have a truly better function out there.
But I had one last test to make. Can we use Reflux as a drop in replacement for an existing strategy? This would be the ultimate test to see if we have an actual useable function for making money.
Replace EMA for Reflux to Make OLMAR Even Better
Awhile ago I implemented the "On-Line Moving Average Reversion" (OLMAR) portfolio selection strategy which does very well when applied across large baskets of assets on a daily trading basis.
The moving average used in that algorithm is the Exponential Moving Average. So, if Reflux really is better, I should be able to pop it out and get better results, right? Let’s see how it worked.
With just a function simple replacement, Reflux improved OLMAR by a massive leap forward. We maintained the same risk profile (maximum drawdown) while gaining a 33% increase in CAGR (which is insane).
On 0.2x leverage to reduce downside risk, we still see a massive improvement over the original OLMAR strategy. This goes to show that a good evaluation system and harness go a long way when it comes to function optimization and research.
And Moving Forward
So you can go ahead and take the Reflux Moving Average if you’d like and apply it to your own strategies. But what you can also do is apply the concept of good evaluation system construction to then find new solutions to age old problems. In the era of AI, this is becoming the way forward, especially in research.
I don’t see that stopping any time soon, and you know for certain that people making 3x your salary sitting in a skyscraper in Manhattan are having their bosses shove AI mandates down their throats. You might as well start competing.
Get the Code
Get the Reflux Moving Average along with the other 360 functions to supercharge your trading journey in the subscriber Google Drive.
All of the code to reproduce the experiments here are also included in the Google Drive for paid members.
Not a paid member yet? No worries.
Subscribe now to get access to this posts code and all of the previous code for all other posts. Not only does it add value to your own trading journey, but it helps support me on my own as I educate you on complex systems in a simple and easy to digest way.
Consider it. Hope to see you soon, and like always, happy trading!
Stuart
subscribed today, cannot access code
your work is brilliant but I can never access any of the code