From Dev.to
My Gradient Clipping Was Never Clipping. It Was Secretly Multiplying My Weight Decay
Yesterday's experiment ended with a clean verdict: LR warmup is insurance — free at the tuned learning rate, a full payout at an overheated one. Gradient clipping has the exact same folklore job descr...