AdamOptimizer

class madspace.AdamOptimizer(self: madspace._madspace_py.AdamOptimizer, function: madspace._madspace_py.Function, context: madspace._madspace_py.Context, learning_rate: SupportsFloat, schedule: madspace._madspace_py.AdamOptimizer.LRSchedule = <LRSchedule.none: 0>, step_count: SupportsInt = 0, beta1: SupportsFloat = 0.9, beta2: SupportsFloat = 0.999, eps: SupportsFloat = 1e-08, grad_clip_threshold: SupportsFloat = 0.0, weight_decay: SupportsFloat = 0.0)

Bases: pybind11_object

Adam gradient-descent optimizer for a compute-graph loss function [1].

Wraps a loss function into a Runtime, runs it forward and backward on each step, and applies the Adam update (beta1 / beta2 moment decay, eps for numerical stability, optional weight_decay) to its requires_grad Context globals. schedule optionally decays the learning rate over step_count steps; grad_clip_threshold, if non-zero, rescales the gradient norm before the update.

References

Parameters:

  • function – The loss function to minimize.

  • context – Context whose requires_grad globals are trained.

  • learning_rate – Initial (or, without schedule, constant) learning rate.

  • schedule – Learning-rate decay over training.

  • step_count – Total number of steps, for schedule.

  • beta1 – First-moment decay rate.

  • beta2 – Second-moment decay rate.

  • eps – Numerical-stability constant.

  • grad_clip_threshold – Maximum gradient norm; 0 disables clipping.

  • weight_decay – L2 weight-decay coefficient.

class LRSchedule(*args, **kwargs)

Bases: pybind11_object

Learning-rate decay applied over the course of training.

Members:

none

cosine

Overloaded function.

  1. __init__(self: madspace._madspace_py.AdamOptimizer.LRSchedule, value: typing.SupportsInt) -> None

  2. __init__(self: madspace._madspace_py.AdamOptimizer.LRSchedule, name: str) -> None

cosine = <LRSchedule.cosine: 1>
AdamOptimizer.LRSchedule.name -> str
none = <LRSchedule.none: 0>
property value
context(self: madspace._madspace_py.AdamOptimizer) → madspace._madspace_py.Context

The context being trained.

cosine = <LRSchedule.cosine: 1>
input_types(self: madspace._madspace_py.AdamOptimizer) → list[madspace._madspace_py.Type]

Input types of the loss function.

learning_rate(self: madspace._madspace_py.AdamOptimizer) → float

The current, possibly schedule-decayed, learning rate.

none = <LRSchedule.none: 0>
step(self: madspace._madspace_py.AdamOptimizer, inputs: collections.abc.Sequence[object]) → list[madspace._madspace_py.Tensor]

Run one training step on inputs and apply the Adam update; returns function’s outputs.