o "j6@sbddlZddlZddlmZddlmZmZmZGdddeZ GdddZ Gd d d e Z dS) N)IterableDataset) RandomSamplerSamplerSequenceSamplerc@s4eZdZdZ     d ddZddZd d ZdS) BatchSampleraf A base implement of batch sampler used by `paddle.io.DataLoader` which yield mini-batch indices(a list/tuple with length as mini-batch size and holds sample indices) iterably. Batch sampler used by :code:`paddle.io.DataLoader` should be a subclass of :code:`paddle.io.BatchSampler`, BatchSampler subclasses should implement following methods: :code:`__iter__`: return mini-batch indices iterably. :code:`__len__`: get mini-batch number in an epoch. Args: dataset(Dataset, optional): this should be an instance of a subclass of :ref:`api_paddle_io_Dataset` or :ref:`api_paddle_io_IterableDataset` or other python object which implemented :code:`__len__` for BatchSampler to get indices as the range of :attr:`dataset` length. Default None, disabled. sampler (Sampler, optional): this should be a :ref:`api_paddle_io_Sample` instance which implemented :code:`__iter__` to generate sample indices. :attr:`sampler` and :attr:`dataset` can not be set in the same time. If :attr:`sampler` is set, :attr:`dataset` should not be set. Default None, disabled. shuffle(bool, optional): whether to shuffle indices order before generating batch indices. Default False, don't shuffle indices before generating batch indices. batch_size(int, optional): sample indice number in a mini-batch indices. default 1, each mini-batch includes 1 sample. drop_last(bool, optional): whether drop the last incomplete (less than 1 mini-batch) batch dataset. Default False, keep it. see :ref:`api_paddle_io_DataLoader` Returns: BatchSampler: an iterable object for indices iterating Examples: .. code-block:: python >>> import numpy as np >>> from paddle.io import RandomSampler, BatchSampler, Dataset >>> np.random.seed(2023) >>> # init with dataset >>> class RandomDataset(Dataset): ... def __init__(self, num_samples): ... self.num_samples = num_samples ... ... def __getitem__(self, idx): ... image = np.random.random([784]).astype('float32') ... label = np.random.randint(0, 9, (1, )).astype('int64') ... return image, label ... ... def __len__(self): ... return self.num_samples ... >>> bs = BatchSampler(dataset=RandomDataset(100), ... shuffle=False, ... batch_size=16, ... drop_last=False) ... >>> for batch_indices in bs: ... print(batch_indices) [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] ... [96, 97, 98, 99] >>> # init with sampler >>> sampler = RandomSampler(RandomDataset(100)) >>> bs = BatchSampler(sampler=sampler, ... batch_size=8, ... drop_last=True) ... >>> for batch_indices in bs: ... print(batch_indices) [56, 12, 68, 0, 82, 66, 91, 44] ... [53, 17, 22, 86, 52, 3, 92, 33] NFrcCs|dur$|dus Jdt|tsJdt||r Jd||_n,t|tr-Jd|dus5Jdt|tsCJdt||rKt||_nt||_t|trY|dks`Jd|||_ t|tsqJd t|||_ dS) Nz'either dataset or sampler should be setz/sampler should be a paddle.io.Sampler, but got z+shuffle should be False when sampler is setz1dataset should not be a paddle.io.IterableDatasetz'should not set both dataset and samplerz+shuffle should be a boolean value, but got rz1batch_size should be a positive integer, but got z-drop_last should be a boolean value, but got ) isinstancertypesamplerrboolrrint batch_size drop_last)selfdatasetr shuffler rrc/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddle/io/dataloader/batch_sampler.py__init__esJ         zBatchSampler.__init__ccsXg}|jD]}||t||jkr|Vg}q|js(t|dkr*|VdSdSdS)Nr)r appendlenr r)r batch_indicesidxrrr__iter__s   zBatchSampler.__iter__cCs.t|j}|t|j |jd7}||jSNr)rr r rr r num_samplesrrr__len__s  zBatchSampler.__len__)NNFrF)__name__ __module__ __qualname____doc__rrrrrrrrsO ' rc@seZdZdddZddZdS)_InfiniteIterableSamplerrcCs"t|ts Jd||_||_dS)Nz:dataset should be an instance of paddle.io.IterableDataset)rrrr )rrr rrrrs z!_InfiniteIterableSampler.__init__ccs dg|jVq)N)r rrrrrsz!_InfiniteIterableSampler.__iter__N)r)rrr rrrrrrr"s  r"c@s:eZdZdZ    d ddZddZdd Zd d ZdS) DistributedBatchSamplera] Sampler that restricts data loading to a subset of the dataset. In such case, each process can pass a DistributedBatchSampler instance as a DataLoader sampler, and load a subset of the original dataset that is exclusive to it. .. note:: Dataset is assumed to be of constant size. Args: dataset(Dataset): this could be an instance of subclass of :ref:`api_paddle_io_Dataset` or other python object which implemented `__len__` for BatchSampler to get indices of samples. batch_size(int): sample size of each mini-batch. num_replicas(int, optional): porcess number in distributed training. If :attr:`num_replicas` is None, :attr:`num_replicas` will be retrieved from :ref:`api_paddle_distributed_ParallelEnv` . Default None. rank(int, optional): the rank of the current process among :attr:`num_replicas` processes. If :attr:`rank` is None, :attr:`rank` is retrieved from :ref:`api_paddle_distributed_ParallelEnv`. Default None. shuffle(bool, optional): whther to shuffle indices order before genrating batch indices. Default False. drop_last(bool, optional): whether drop the last incomplete(less than a mini-batch) batch dataset size. Default False. Returns: DistributedBatchSampler, return an iterable object for indices iterating. Examples: .. code-block:: python >>> import numpy as np >>> from paddle.io import Dataset, DistributedBatchSampler >>> # init with dataset >>> class RandomDataset(Dataset): ... def __init__(self, num_samples): ... self.num_samples = num_samples ... ... def __getitem__(self, idx): ... image = np.random.random([784]).astype('float32') ... label = np.random.randint(0, 9, (1, )).astype('int64') ... return image, label ... ... def __len__(self): ... return self.num_samples ... >>> dataset = RandomDataset(100) >>> sampler = DistributedBatchSampler(dataset, batch_size=64) >>> for data in sampler: ... # do something ... break NFcCs||_t|tr |dksJd||_t|tsJd||_t|ts(Jdddlm}|durCt|tr;|dks?Jd||_n|j|_|dur]t|trU|dksYJd||_ n|j |_ ||_ d|_ tt t|jd|j|_|j|j|_dS) Nrz'batch_size should be a positive integerz!shuffle should be a boolean valuez$drop_last should be a boolean number) ParallelEnvz)num_replicas should be a positive integerz%rank should be a non-negative integerg?)rrr r r rZpaddle.distributedr%nranks local_rankrepochmathceilrr total_size)rrr Z num_replicasrankrrr%rrrrsF        z DistributedBatchSampler.__init__c#s.tj}t|}jt|}|t|kr#||d|7}n||t|t|d|7}t|jks=JjrQtj j |j d7_ fdd}j dkr`||}t|j ksiJt|}g}|D]}||t|jkr|Vg}qqjst|dkr|VdSdSdS)Nrcsg}jjj}|jdksJ|j}tjjt||jjD]}||||jq+|t||d}||j|jd||S)Nrr)r+r r&ranger'rextend)indicesZsubsampled_indicesZlast_batch_sizeZlast_local_batch_sizeir#rr_get_indices_by_batch_size!s.    zDDistributedBatchSampler.__iter__.._get_indices_by_batch_sizer)rrnpZarangetolistr+r)r*rrandomZ RandomStater(r&riterrr r)rrr/Z padding_sizer1Z _sample_iterrrrr#rrs8      z DistributedBatchSampler.__iter__cCs*|j}|t|j |jd7}||jSr)rr rr rrrrrGs zDistributedBatchSampler.__len__cCs ||_dS)a Sets the epoch number. When :attr:`shuffle=True`, this number is used as seeds of random numbers. By default, users may not set this, all replicas (workers) use a different random ordering for each epoch. If set same number at each epoch, this sampler will yield the same ordering at all epoches. Arguments: epoch (int): Epoch number. Examples: .. code-block:: python >>> import numpy as np >>> from paddle.io import Dataset, DistributedBatchSampler >>> # init with dataset >>> class RandomDataset(Dataset): ... def __init__(self, num_samples): ... self.num_samples = num_samples ... ... def __getitem__(self, idx): ... image = np.random.random([784]).astype('float32') ... label = np.random.randint(0, 9, (1, )).astype('int64') ... return image, label ... ... def __len__(self): ... return self.num_samples ... >>> dataset = RandomDataset(100) >>> sampler = DistributedBatchSampler(dataset, batch_size=64) >>> for epoch in range(10): ... sampler.set_epoch(epoch) N)r()rr(rrr set_epochLs %z!DistributedBatchSampler.set_epoch)NNFF)rrr r!rrrr6rrrrr$s= ,8 r$) r)numpyr2rrr rrrrr"r$rrrrs