Powerful Mathematical Optimizers for BlackBox Optimization without direct derivatives.

For lower then or equal to 1000 optimized variables or parameters. Converges faster then sNES. but uses more memory and is heavier in it's internal matrix operations. It can easily solve Rosenbrock function for 100 dimensions, with a very small number of fitness function evaluations.

For more then 1000 optimized variables or parameters. It handles 33_000 parameters with ease, depending on how hard is the function to optimize. Converges slower then xNES. but uses much less memory and is much ligher, in it's optimizer internal operations. It can easily solve Rosenbrock functions for 100 dimensions, with a very small mumber of fitness function evaluations, but more then xNES.

The reason is simple, the fitness function to optimize can be what ever you like, it can be a program, it can be a hand made experiment, it can be what ever you need to optimize, provided that you have a way to evaluate how good you solution is has a quantitative f64 value, that you will try to maximize. If your fitness function only allows for minimization, just "return -val", the negative of the value of a minimization problem transforms into a maximization problem.

One that is also considered very good is the Bayesian Optimization Algorithm, internally, it uses Gaussian Processes to module surrogate functions and models also how certain the optimization algorithm is about some value. It optimizes over the surrogate function that is less heavy then the target heavy experiments. It uses very little number of fitness function evaluations, but it has inside of it matrix operations that are O( n ^ 3 ), and there implementation are not as simple has xNES or sNES. They also don't scale, in the number of dimensions, as well as xNES or sNES.

All this work is derived work from the xnes.py ( authored by Tom Schaul, tom@idsia.ch ) and snes.py ( authored by Tom Schaul, tom@idsia.ch ) the original code is from the site,
Natural Evolution Strategies
https://schaul.site44.com/nes.html
And the original code was under BSD License, The derived code that I present in here has the same license, BSD License.

1. I Studied some literature about Natural Evolution Strategies and the OpenAI blog post and paper that said that this was a very powerful optimizer, even used with success optimize in a distributed way in Reinforcement Learning workload by OpenAI with out Deep Reinforcement Leaning, but with competitive results.

2. Studied also SPSA - Simultaneous Perturbation Stochastic Algorithm, also for black box optimization without direct derivatives. Compared it using the Rosenbrock functions for 100 dimensions. sNES and xNES made "circles" around SPSA, and I was not expecting this. Because the theory of SPSA is very solid.

3. The original xnes.py and snes.py where made for Python 2 and with an old version of SciPy, that didn't work for the current version of SciPy. So I converted it to Python 3 and to use NumPy for most matrix operations, and modernized the SciPy calls ex: expm.

4. Did a lot of tests to find if the code continues to converged correctly, and it appears to converge correctly.

5. Ported the code of sNES and xNES to C and made the necessary Makefiles.

6. Did test on the ported code.

7. Optimized the sNES and the xNES C code.

8. Ported the code to Odin.

9. Did some more tests and more optimization.

Best regards,
Joao Carvalho