This is a really fast derivative free optimizer that can minimize 3000 dimensions Rosenbrock in 30 minutes on a slow computer single threaded.

This is an implmentation of TROA - Trust Region Optimizer Algotithm in Odin, this is a derivative free mathematical optimization algorithm, internally it calculates the derivatives by using the central derivative method of any objective function, but the user doesn't need to give it the analitical derivative of the objective function. This is a very simple implementation of it but a very fast one, has you can see in terms of timmings to solve high dimension the Rosenbrock function.

10   Dimensions          0.004 seconds
50   Dimensions          0.090 seconds
100  Dimensions          0.359 seconds
200  Dimensions        1.33 seconds
500  Dimensions       10.25 seconds
600  Dimensions       17.5 seconds
700  Dimensions       26.7 seconds
1000 Dimensions   1 m 13 seconds
2000 Dimensions   9 m 13 seconds
2500 Dimensions  18 m  6 seconds
3000 Dimensions  29 m 36 seconds 

It's a clever method for finding the minimum value of a complex mathematical function, like finding the lowest point in a hilly, fog-covered landscape.

The "Hiker in the Fog" Analogy

Imagine you're a hiker trying to find the lowest point in a valley, but it's incredibly foggy. You can only see a small circular area around you.

• The Map ( Model ):
  You can't see the whole valley, but you can create a simple map of the ground you're standing on. You can tell which direction is steepest downhill from your current position. This "map" is a simplified model of the real terrain. In the code, this is done by calculating the gradient ( g ), which points in the direction of the steepest ascent.

• The Circle of Trust ( Trust Region ):
  The foggy area you can see is your "trust region," with a radius called delta. You trust your simple map to be accurate only within this circle.

• Taking a Step:
  You use your map to decide on the best step ( s ) to take to get lower, but you are not allowed to step outside your circle of trust. The code does this by moving in the opposite direction of the gradient ( -g ) for a distance equal to the trust radius delta.

• Checking Your Progress:
  After taking the step, you check your actual change in altitude. You compare this actual improvement ( ared ) with what your simple map predicted the improvement would be ( pred ). This comparison gives a ratio called rho.

Adjusting to the Fog:

• Good Prediction ( rho is high ):
  If your map was very accurate, you become more confident. You accept the new position and might expand your circle of trust ( delta increases ) for the next step, as the fog seems to be clearing.

• Bad Prediction ( rho is low or negative ):
  If your map was very wrong ( e.g., you predicted you'd go down but you actually went up ), you realize your map is unreliable. You reject the step ( stay where you were) and shrink your circle of trust ( delta decreases ), because the terrain is more complex than you thought and you need a more localized map.

This process is repeated—building a local model, taking a step within a trusted region, and then updating the size of that region based on success—until the circle of trust becomes so small that you can't find a better step, meaning you've likely arrived at a low point ( a local minimum ).

MIT Open Source License

Best Regards,
Joao Carvalho