Regret factor in ForwardBackward

[aldma]

This PR adds the possibility to specify a regret factor to increase the stepsize gamma at every iteration, if adaptive, before backtracking. Recent results provide convergence guarantees even without a maximum value on gamma [Theorem 14, arXiv:2208.00779v2].

This feature was implemented for ForwardBackward only. Although it does not seem a good fit for accelerated methods like ZeroFPR, PANOC, and PANOCplus, at least when based on quasi-Newton directions, it is unclear whether the FastForwardBackward solver could benefit from it. See discussion in #86 .

Practical performance seems to improve with values regret_gamma close to 1. Tests and references have also been updated accordingly.

[aldma]

Based on benchmarks and interest, the default value for regret_gamma could be set greater than, but close to, 1. Another possibility is to take an adaptive regret factor, as common in the convex literature.

[lostella]

@aldma thanks for this! Some comments

• Accelerated algorithm from Nesterov here (page 143) allows for increasing stepsizes
• Would it make sense to have regret > 1 by default?
• I’m not sure about the name “regret”. Maybe “increase” is better? Also I see the backtracking factor is not exposed, but I guess it could be

[aldma]

Thanks @lostella for you comments

• Nesterov's paper focuses on convex problems, whereas the proof here (Thm 14) is for fully nonconvex ones (thanks @AndThem!)
• The denomination "regret" comes from the FOM toolbox (webpage), but it is not widespread nor standard (or so it seems). Perhaps the backtracking should be exposed and the two names somehow consistent?
• I think it makes sense to have r > 1 by default, but I don't see any recommendations more specific than 1 < r < 2. It should perhaps be linked to the backtracking factor too. In FOM they take r = 1/beta, with beta the backtracking factor.

[aplavin]

Would be nice to have it included indeed!
Another useful convenience feature is "at first, increase gamma with by large factor, like 2 or 3; after backtracking happens for the first time, switch to a small r like 1.1". This makes the algorithm insensitive to the initial gamma value in a very wide range, even if it happened to start too small.

[AndThem]

According to the current theory, the price for not imposing an upper bound on the stepsize is the fact that each failed linesearch step incurs one (wasted) extra gradient evaluation (an extra Lipschitz-like check is required). I believe that selecting a (very large) gamma_max, perhaps after the first couple of iterations, is a much more reasonable choice to avoid such overhead.

[lostella]

Looks good, thank you @aldma!