modelscope-scepter/docs/en/scepter/sampler.md

Sampler Module (Sampler)

Overview

A sampler defines the method for selecting the necessary data for training, validation, and testing. Samplers are fairly generic and in most cases do not require custom development. Here, several commonly used samplers are provided. Some samplers, such as MixtureOfSamplers, encapsulate a SubSampler.

Of course, samplers also support custom development and registration. Developers just need to inherit from BaseSampler and register their implemented sampler class.

Basic Usage

Predefined samplers:

• TorchDefault
• LoopSampler
• MixtureOfSamplers
• MultiFoldDistributedSampler
• EvalDistributedSampler
• MultiLevelBatchSampler
• MultiLevelBatchSamplerMultiSource

from scepter.modules.data.sampler import (
    LoopSampler,
    MixtureOfSamplers,
    MultiFoldDistributedSampler,
    EvalDistributedSampler,
    MultiLevelBatchSampler,
    MultiLevelBatchSamplerMultiSource)

Custom sampler, taking LoopSampler as an example:

@SAMPLERS.register_class()
class LoopSampler(BaseSampler):
    para_dict = {}
    def __init__(self, cfg, logger):
        super().__init__(cfg, logger)
        rank = we.rank
        self.rng = np.random.default_rng(self.seed + rank)
    def __iter__(self):
        while True:
            yield self.rng.choice(sys.maxsize)
    def __len__(self):
        return sys.maxsize
    @staticmethod
    def get_config_template():
        return dict_to_yaml('SAMPLERS',
                            __class__.__name__,
                            LoopSampler.para_dict,
                            set_name=True)

Implement the iter and len methods to perform sampling. Related configuration parameters can be read from the cfg parameter in init.

---

LoopSampler

An infinite looping sampler for simple data.

function LoopSampler.init

(cfg: scepter.modules.utils.config.Config, logger=None) -> None

function LoopSampler.iter

Iterator, each iteration yields the index of a sample.

MixtureOfSamplers

A sampler for large-scale data with multi-level indexing.

function MixtureOfSamplers.init

(samplers: list[sampler], probabilities: list[float], rank: int = 0, seed: int = 8888)

Parameters

• samplers —— List of samplers for mixing.
• probabilities —— Probabilities associated with each sampler.
• rank —— The rank representing the current process number.
• seed —— The seed for random sampling, obtained globally from data.registry.

function MultiLevelBatchSampler.iter

() Iterator, each iteration yields the index of a sample.

MultiFoldDistributedSampler

A multi-fold sampler that supports repeating data several times within one epoch.

function MultiFoldDistributedSampler.init

(dataset: torch.utils.data.Dataset, num_folds=1, num_replicas=None, rank=None, shuffle=True)

Parameters

• dataset —— An instance of the torch.utils.data.Dataset class.
• num_folds —— Integer, indicating the number of times data is repeated.
• num_replicas —— The number of data partitions, generally consistent with the world size.
• rank —— The rank representing the current process number.
• shuffle —— Whether to shuffle the data or not.

function MultiFoldDistributedSampler.iter

() Iterator, each iteration yields the index of a sample.

function MultiFoldDistributedSampler.set_epoch

(epoch: int) Sets the current epoch. Parameters

• epoch —— The current epoch number.

EvalDistributedSampler

A sampler for evaluation during testing, where you may find that the last rank has less data than other ranks when not using padding mode.

function EvalDistributedSampler.init

(dataset: torch.utils.data.Dataset, num_replicas: Optional[int] = None, rank: Optional[int] = None, padding: bool = False)

Parameters

• dataset —— Instance of the torch.utils.data.Dataset class.
• num_replicas —— The number of data partitions, typically consistent with the world size.
• rank —— The rank representing the current process number.
• padding —— Whether the data needs padding. If true, it can ensure the last rank has the same amount of data as other ranks.

function EvalDistributedSampler.iter

() Iterator, each iteration yields the index of a sample.

function EvalDistributedSampler.set_epoch

(epoch: int) Sets the current epoch.

Parameters

• epoch —— The current epoch number.

MultiLevelBatchSampler

A sampler for large-scale data with multi-level indexing.

function MultiLevelBatchSampler.init

(index_file: str, batch_size: int, rank: int = 0, seed: int = 8888)

Parameters

• index_file —— The index file for multi-level data indexing.
• batch_size —— The size of a batch.
• rank —— The rank representing the current process number.
• seed —— The seed for random sampling, obtained globally from data.registry.

function MultiLevelBatchSampler.iter

() Iterator, each iteration yields the index of a sample.