Learning-Rate-Free Learning by D-Adaptation

通过 D-Adaptation 实现无学习率学习

Aaron Defazio, Konstantin Mishchenko

Abstract

D-Adaptation is an approach to automatically setting the learning rate which asymptotically achieves the optimal rate of convergence for minimizing convex Lipschitz functions, with no back-tracking or line searches, and no additional function value or gradient evaluations per step. Our approach is the first hyper-parameter free method for this class without additional multiplicative log factors in the convergence rate. We present extensive experiments for SGD and Adam variants of our method, where the method automatically matches hand-tuned learning rates across more than a dozen diverse machine learning problems, including large-scale vision and language problems. An open-source implementation is available.

摘要

D-Adaptation 是一种自动设定学习率的方法，在最小化凸 Lipschitz 函数时能够渐近达到最优收敛速率，且无需回溯或线搜索，也无需在每一步额外计算函数值或梯度。对于这一类问题，我们的方法是首个无需超参数且收敛速率中不含额外乘法对数因子的方法。我们对该方法的 SGD 和 Adam 变体进行了大量实验。实验表明，在十多个不同的机器学习问题（包括大规模视觉和语言问题）中，该方法都能自动匹配手动调优的学习率。我们提供了开源实现。

语言

1 种语言有译本

AI 译本 · Equalang

新上架

PDF · 2026年11月翻译 · 公式优化

免费试读