Co-Evolving Thousands of Portfolios for Robustness in Portfolio Selection

“Portfolio ensemble optimization” for generalization and out-of-distribution robustness in portfolio selection: a portfolio ensemble from behaviourally diverse (correlation matrix above) high-quality portfolios co-evolved.

Optimizing market neutral robust portfolios that would generalize well to the future is still a big challenge.

This fact has proven itself by the reported fund performances during the most recent stock market crash.

Seven decades ago, Harry Markowitz famously said: “Diversification is the only free lunch in investing.”

Diversification prevents weight concentration to few assets and reduces the exposure to idiosyncratic shocks.

Yet, diversification seems to be not perfectly leveraged by many quantitative finance specialists for a long-time.

Warren Buffett defined diversification as a protection against ignorance that makes a very little sense for experts.

At the initial epoch of the optimization, the ensemble portfolio is equivalent to a pure genotype diversity. Mean-variance portfolios tend to perform even worse than this naive portfolio generally in out-of-sample.

The horizontal middle in above plot is the most recent stock-market crash due to the Coronavirus pandemic, where the initial long-short portfolio is much worse in risk-sensitive rewards of the resulting final ensemble.

In portfolio optimization literature, one can find many methods ensuring the genotype diversity of portfolios.

Per-asset risk diversification metrics (Max. risk contribution, Diversification ratio, Concentration ratio) are flawed.

Although they are less exposed to idiosyncratic shocks, they are vulnerable to systemic shocks by ignoring correlations.

They do not ensure the robustness of a portfolio in any market condition by diversifying hierarchical sources of risks.

To ensure the out-of-sample robustness, a portfolio should consist of a large amount of uncorrelated sub-portfolios.

At early epochs of optimization, the ensemble first over-fits to the training set; thus fails to generalize well. Even if better forecasts could be incorporated, they would not have sufficient accuracy during a black-swan.

The ensemble first learns to contain high quality assets in training but still requires to diversity its exposures, for being robust and obtaining a much lower maximum drawdown at black swan events like this market crash.

Hierarchical Risk Parity from Marcos Lopez de Prado known to result in a better out-of-sample performance.

This is because HRP aims to construct a portfolio that can be hierarchically decomposed into uncorrelated bets.

In other words, HRP tries to heuristically diversify the risk into several Uncorrelated Exposures (as referred by Meucci).

This helps HRP to be less vulnerable to not only idiosyncratic but also systemic shocks compared to other methods.

Monte Carlo Simulation is often used to verify the robustness under different scenarios sampled from the past.

At the end of the optimization, the ensemble consists of many high-quality but behaviorally diverse portfolios. This helps maintaining a good test performance in risk-adjusted metrics while having a lower drawdown.

The ensemble avoids over-fitting and obtains a much lower maximum drawdown during the market crash. As it has been proven by this black-swan event, it is much less vulnerable to any idiosyncratic or systemic shock.

Just like HRP targets, the aim of portfolio selection should be constructing a behaviorally diverse portfolio ensemble.

This is also what meant to happen intrinsically when one optimizes a portfolio using the minimum variance objective.

Ensembles empirically tend to yield better test results when there is diversity among their individual hypotheses.

Penalizing the correlation in-between high-quality sub-portfolios improves the generalization of their ensemble.

In this way, the ensemble portfolio can be robust against black swans, even if some of its constructs perform badly.

Comparison of the proposed method with HRP on a blind test set since the beginning of the market crash. The proposed method is better than HRP in almost all of the risk-adjusted performance metrics at the blind test.

Thus, I constructed an ensemble from behaviorally diverse and high-quality portfolios co-evolved in parallel via GPU.

The members are co-evolved for maximizing their Probabilistic Sharpe Ratio while minimizing their highest correlations.

Although some of the high-quality portfolios of the training set have bad test performances, their ensemble stays robust.

The sub-portfolios are co-evolved using a novel population-based large-scale non-convex global optimization method.

Quadratic optimizers generally produce unreliable solutions as they fail to invert the positive-definite covariance matrix.

The test performance during the whole optimization epochs until early-stopping triggered by training set. If only a single portfolio has been optimized for its quality, it could perform bad on the out-of-sample test set.

The proposed method can also enable co-evolving a population of “tactical investment algorithms” in the same manner.

Such population of optimal individuals for different market regimes can dynamically be ensembled by nowcasting.

In this way, those behaviorally diverse individuals can compete with each other for dynamically allocated resources.

More advanced ensembling methods that can be dynamically applied during live trading should further be investigated.

It is also straightforward to incorporate views (such as forecasted returns, etc) to this method, similar to Black-Litterman.

Conclusion

In this article, I have claimed that robustness in portfolio optimization can be done by co-evolving a diverse ensemble.

Population-based Quality-Diversity optimization (or Illumination) is also becoming popular in other science fields.

This research can hopefully point light to many other fields of optimization where ensuring the robustness is critical.

Ensemble models also shown to improve the accuracy, uncertainty and out-of-distribution robustness of Deep Learning.

Here, you can check the weights of the assets within the ensemble portfolio:
https://gist.github.com/kayuksel/46ffb48b1244eb2eb621d5793abd686b

References

DeMiguel, Victor, Lorenzo Garlappi, and Raman Uppal. “Optimal versus naive diversification: How inefficient is the 1/N portfolio strategy?.” The review of Financial studies 22, no. 5 (2009): 1915–1953.

Meucci, Attilio. “Managing diversification.” Risk (2009): 74–79.

Lopez de Prado, Marcos. “Building diversified portfolios that outperform out of sample.” The Journal of Portfolio Management 42, no. 4 (2016)

Bailey, David H., and Marcos Lopez de Prado. “The Sharpe ratio efficient frontier.” Journal of Risk 15, no. 2 (2012): 13.

Lopez de Prado, Marcos. “Tactical investment algorithms.” Available at SSRN 3459866 (2019).

Lipton, Alex, and Marcos Lopez de Prado. “What Quants Can Learn from the COVID Crisis.” Risk Magazine, April (2020).

Pugh, J. K., Soros, L. B., & Stanley, K. O. (2016). Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, 3, 40.

Fort, Stanislav, Huiyi Hu, and Balaji Lakshminarayanan. “Deep ensembles: A loss landscape perspective.” arXiv preprint arXiv:1912.02757 (2019).