modelscope-scepter/docs/en/scepter/opt.md

Optimizer (Optimizer)

Overview

1. lr_schedulers
2. optimizers

---

lr_schedulers

Basic Usage

Usage when subclassing lr_schedulers:

from scepter.opt.lr_schedulers import LR_SCHEDULERS
from scepter.opt.lr_schedulers.base_scheduler import BaseScheduler


@LR_SCHEDULERS.register_class()
class XxxLR(BaseScheduler):
    def __init__(self, cfg, logger=None):
        super().__init__(cfg, logger=logger)

Actual usage to start lr_scheduler (optimizer is necessary), refer to task/stable_diffusion/impls/solvers/diffusion_solver.py:

if self.cfg.have("LR_SCHEDULER") and not self.optimizer is None:
    self.lr_scheduler = LR_SCHEDULERS.build(self.cfg.LR_SCHEDULER, logger=self.logger,
                                            optimizer=self.optimizer)

scepter.modules.opt.lr_schedulers.base_scheduler.BaseScheduler

The base class for lr_schedulers, supports registration operations, can be customized as needed;

function __init__()

Parameters(input parameters)

(cfg: scepter.modules.utils.config.Config, logger = None) -> None

config(Common parameters, actual needs vary according to different schedulers, taking StepLR as an example):

• STEP_SIZE
• GAMMA
• LAST_EPOCH

function __call__()

Parameters(input parameters)

(optimizer: scepter.modules.opt.optimizers.OPTIMIZERS) -> None

Sets up the schedule for the passed-in optimizer object;

---

optimizers

Basic Usage

Usage when subclassing optimizers:

from scepter.opt.optimizers.base_optimizer import BaseOptimize
from scepter.opt.optimizers.registry import OPTIMIZERS


@OPTIMIZERS.register_class()
class Xxx(BaseOptimize):
    def __init__(self, cfg, logger=None):
        super(Xxx, self).__init__(cfg, logger=logger)

Actual usage to start optimizers, refer to task/stable_diffusion/impls/solvers/diffusion_solver.py, requires passing in train_parameters:

if self.cfg.have("OPTIMIZER"):
    self.optimizer = OPTIMIZERS.build(self.cfg.OPTIMIZER, logger=self.logger,
                                      parameters=self.train_parameters())

scepter.modules.opt.optimizers.base_optimizer.BaseOptimize

The base class for optimizers, supports registration operations, can be customized as needed;

function __init__()

Parameters(input parameters)

(cfg: scepter.modules.utils.config.Config, logger = None) -> None

config(common parameters, actual needs vary according to different optimizers, taking SGD as an example):

• LEARNING_RATE
• MOMENTUM
• DAMPENING
• WEIGHT_DECAY
• NESTEROV

function __call__()

Parameters(input parameters)

(parameters:dict()) -> None Inputs the train parameters that need gradient updates, in dict format;