AdamOptimizer

class AdamOptimizer

Adam gradient-descent optimizer for a compute-graph loss function [1].

Wraps a loss function into a Runtime, runs it forward and backward on each step, and applies the Adam update (beta1 / beta2 moment decay, eps for numerical stability, optional weight_decay) to its requires_grad Context globals. schedule optionally decays the learning rate over step_count steps; grad_clip_threshold, if non-zero, rescales the gradient norm before the update.

References

Public Types

enum LRSchedule

Learning-rate decay applied over the course of training.

Values:

enumerator none

Constant learning rate.

enumerator cosine

Cosine decay from learning_rate to zero over step_count steps.

Public Functions

AdamOptimizer(const Function &function, ContextPtr context, double learning_rate, LRSchedule schedule = LRSchedule::none, std::size_t step_count = 0, double beta1 = 0.9, double beta2 = 0.999, double eps = 1e-8, double grad_clip_threshold = 0.0, double weight_decay = 0.0)
Parameters:

  • function – The loss function to minimize.

  • context – Context whose requires_grad globals are trained.

  • learning_rate – Initial (or, without schedule, constant) learning rate.

  • schedule – Learning-rate decay over training.

  • step_count – Total number of steps, for schedule.

  • beta1 – First-moment decay rate.

  • beta2 – Second-moment decay rate.

  • eps – Numerical-stability constant.

  • grad_clip_threshold – Maximum gradient norm; 0 disables clipping.

  • weight_decay – L2 weight-decay coefficient.

TensorVec step(const TensorVec &inputs)

Run one training step on inputs and apply the Adam update; returns function’s outputs.

void replace_function(const Function &function)

Replace the loss function, keeping the optimizer state.

double learning_rate() const

The current, possibly schedule-decayed, learning rate.

inline const TypeVec &input_types() const

Input types of the loss function.

inline ContextPtr context() const

The context being trained.

inline Tensor parameters() const

The trained globals, packed into one contiguous tensor.

inline const std::vector<std::string> &param_names() const

Names of the trained globals, in the order they appear in parameters.