Mathematical Optimizers - Several Evolution Strategies algorithms in Odin

Simplified versions of Mathematical Optimizers for derivative free, no gradient, black box optimization.

Description

I'm trying to study a little bit about different Mathematical Optimizers for Black Box Optimization that don't need derivatives. In this repository there are 4 different types that are are variants of Evolution Strategies. They are all were tested on the Rosembork objective function with 100 dimensions. You can compare the Wall Clock Time and the number of objective function evaluations of the different algorithms.
Note that some of them are simplified versions of the full algorithm because some require a production quality Matrix Library or Linear Algebra Library and currently Odin doesn't have the bindings for one. The full versions of the algorithm, not simplified, converge even faster then the one that I present in this repository.

Simplified iw_es__importance_weighted_evolution_strategies

Paper
Importance Weighted Evolution Strategies
by Víctor Campos, Xavier Giro-i-Nieto, Jordi Torres
https://arxiv.org/abs/1811.04624

Results:

Optimization finished in 2m3.906014196s.
Termination Reason : Max generations
Total Generations  : 100_000
Total Evaluations  : 1_800_000
Best Fitness Found : 8.99688349e-05

N               := 100         // Num Dimensions.
max_generations := 100_000

Simplified Covariance matrix adaptation evolution strategy CMA-ES

Wikipedia
https://en.wikipedia.org/wiki/CMA-ES

Results:

Total generations          : 160_000
Elapsed time               : 27.345488028s
Best fitness found         : 1.08106838e-05
Total function evaluations : 5_440_000

N               := 100       // Dimensions
MAX_GENERATIONS := 160_000

Simplified Limited-Memory Covariance Matrix Adaptation Evolution Strategy LM-CMA-ES

Wikipedia
https://en.wikipedia.org/wiki/Evolution_strategy

Results:

Optimization finished in 18.058169728s.
Termination Reason : Max generations reached
Total Generations  : 180_001
Total Evaluations  : 6_120_000
Best Fitness Found : 1.79380565e-05
    
N               := 100      // Dimensions
max_generations := 180_000

DEO - Differential Evolution Strategies Optimizer

Wikipedia
https://en.wikipedia.org/wiki/Differential_evolution

Results:

Optimization finished in 12m42.897588565s.
Termination Reason : Max generations reached
Total Generations  : 1_200_000
Total Evaluations  : 1_200_001_000
Best Fitness Found : 2.89717857e-04
    
N               := 100        // Dimension
max_generations := 1_200_000

License

MIT Open Source

Have fun

Best regards,
Joao Carvalho