o "j$V@sddlZddlmZddlZddlmZGdddZGdddeZGd d d eZd d Z Gd ddeZ GdddeZ GdddeZ dddZ ddfddZGdddeZdS)N)Iterable) frameworkc@(eZdZdZddZddZddZdS) Dataseta An abstract class to encapsulate methods and behaviors of datasets. All datasets in map-style(dataset samples can be get by a given key) should be a subclass of `paddle.io.Dataset`. All subclasses should implement following methods: :code:`__getitem__`: get sample from dataset with a given index. This method is required by reading dataset sample in :code:`paddle.io.DataLoader`. :code:`__len__`: return dataset sample number. This method is required by some implements of :code:`paddle.io.BatchSampler` see :code:`paddle.io.DataLoader`. Examples: .. code-block:: python >>> import numpy as np >>> from paddle.io import Dataset >>> # define a random dataset >>> class RandomDataset(Dataset): ... def __init__(self, num_samples): ... self.num_samples = num_samples ... ... def __getitem__(self, idx): ... image = np.random.random([784]).astype('float32') ... label = np.random.randint(0, 9, (1, )).astype('int64') ... return image, label ... ... def __len__(self): ... return self.num_samples ... >>> dataset = RandomDataset(10) >>> for i in range(len(dataset)): ... image, label = dataset[i] ... # do something cCdSNselfr r ]/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddle/io/dataloader/dataset.py__init__AzDataset.__init__cCtdd|jj)N'{}' not implement in class {} __getitem__NotImplementedErrorformat __class____name__r idxr r r rD zDataset.__getitem__cCr)Nr__len__rr r r r rJrzDataset.__len__Nr __module__ __qualname____doc__r rrr r r r rs ) rc@s0eZdZdZddZddZddZdd Zd S) IterableDataseta An abstract class to encapsulate methods and behaviors of iterable datasets. All datasets in iterable-style (can only get sample one by one sequentially, like a Python iterator) should be a subclass of :ref:`api_paddle_io_IterableDataset` . All subclasses should implement following methods: :code:`__iter__`: yield sample sequentially. This method is required by reading dataset sample in :ref:`api_paddle_io_DataLoader` . .. note:: do not implement :code:`__getitem__` and :code:`__len__` in IterableDataset, should not be called either. see :ref:`api_paddle_io_DataLoader` . Examples: .. code-block:: python :name: code-example1 >>> import numpy as np >>> from paddle.io import IterableDataset >>> # define a random dataset >>> class RandomDataset(IterableDataset): ... def __init__(self, num_samples): ... self.num_samples = num_samples ... ... def __iter__(self): ... for i in range(self.num_samples): ... image = np.random.random([784]).astype('float32') ... label = np.random.randint(0, 9, (1, )).astype('int64') ... yield image, label ... >>> dataset = RandomDataset(10) >>> for img, label in dataset: ... # do something ... ... When :attr:`num_workers > 0`, each worker has a different copy of the dataset object and will yield whole dataset samples, which means samples in dataset will be repeated in :attr:`num_workers` times. If it is required for each sample to yield only once, there are two methods to configure different copy in each worker process to avoid duplicate data among workers as follows. In both the methods, worker information that can be getted in a worker process by `paddle.io.get_worker_info` will be needed. splitting data copy in each worker in :code:`__iter__` .. code-block:: python :name: code-example2 >>> import math >>> import paddle >>> import numpy as np >>> from paddle.io import IterableDataset, DataLoader, get_worker_info >>> class SplitedIterableDataset(IterableDataset): ... def __init__(self, start, end): ... self.start = start ... self.end = end ... ... def __iter__(self): ... worker_info = get_worker_info() ... if worker_info is None: ... iter_start = self.start ... iter_end = self.end ... else: ... per_worker = int( ... math.ceil((self.end - self.start) / float( ... worker_info.num_workers))) ... worker_id = worker_info.id ... iter_start = self.start + worker_id * per_worker ... iter_end = min(iter_start + per_worker, self.end) ... ... for i in range(iter_start, iter_end): ... yield np.array([i]) ... >>> dataset = SplitedIterableDataset(start=2, end=9) >>> dataloader = DataLoader( ... dataset, ... num_workers=2, ... batch_size=1, ... drop_last=True) ... >>> for data in dataloader: ... print(data) # doctest: +SKIP("The output depends on the environment.") Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[2]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[3]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[4]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[5]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[6]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[7]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[8]]) splitting data copy in each worker by :code:`worker_init_fn` .. code-block:: python :name: code-example3 >>> import math >>> import paddle >>> import numpy as np >>> from paddle.io import IterableDataset, DataLoader, get_worker_info >>> class RangeIterableDataset(IterableDataset): ... def __init__(self, start, end): ... self.start = start ... self.end = end ... ... def __iter__(self): ... for i in range(self.start, self.end): ... yield np.array([i]) ... >>> dataset = RangeIterableDataset(start=2, end=9) >>> def worker_init_fn(worker_id): ... worker_info = get_worker_info() ... ... dataset = worker_info.dataset ... start = dataset.start ... end = dataset.end ... num_per_worker = int( ... math.ceil((end - start) / float(worker_info.num_workers))) ... ... worker_id = worker_info.id ... dataset.start = start + worker_id * num_per_worker ... dataset.end = min(dataset.start + num_per_worker, end) ... >>> dataloader = DataLoader( ... dataset, ... num_workers=2, ... batch_size=1, ... drop_last=True, ... worker_init_fn=worker_init_fn) ... >>> for data in dataloader: ... print(data) # doctest: +SKIP("The output depends on the environment.") Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[2]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[3]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[4]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[5]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[6]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[7]]) Tensor(shape=[1, 1], dtype=int64, place=Place(cpu), stop_gradient=True, [[8]]) cCrrr r r r r r rzIterableDataset.__init__cCr)Nr__iter__rr r r r r rzIterableDataset.__iter__cCr)N/'{}' should not be called for IterableDataset{}r RuntimeErrorrrrrr r r rrzIterableDataset.__getitem__cCr)Nr!rr"r r r r rrzIterableDataset.__len__N)rrrrr r rrr r r r rQs! rc@r) TensorDataseta Dataset defined by a list of tensors. Each tensor should be in shape of [N, ...], while N is the sample number, and ecah tensor contains a field of sample, :code:`TensorDataset` retrieve each sample by indexing tensors in the 1st dimension. Args: tensors(list|tuple): A list/tuple of tensors with same shape in the 1st dimension. Returns: Dataset: a Dataset instance wrapping tensors. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> from paddle.io import TensorDataset >>> input_np = np.random.random([2, 3, 4]).astype('float32') >>> input = paddle.to_tensor(input_np) >>> label_np = np.random.random([2, 1]).astype('int32') >>> label = paddle.to_tensor(label_np) >>> dataset = TensorDataset([input, label]) >>> for i in range(len(dataset)): ... input, label = dataset[i] ... # do something cs8tstdtfddDsJd|_dS)Nz1TensorDataset con only be used in imperative modec3s(|]}|jddjdkVqdS)rN)shape.0Ztensortensorsr r 0s z)TensorDataset.__init__..z0tensors not have same shape of the 1st dimension)rZin_dynamic_moder#allr))r r)r r(r r +s   zTensorDataset.__init__cstfdd|jDS)Nc3s|]}|VqdSrr r&indexr r r*6sz,TensorDataset.__getitem__..)tupler))r r-r r,r r5szTensorDataset.__getitem__cCs|jdjdSNr)r)r%r r r r r8zTensorDataset.__len__Nrr r r r r$s " r$cCs(|dur|St|ttfrt|S|gSr) isinstancelistr.)valuer r r to_list<s r4c@r) ComposeDataseta A Dataset which composes fields of multiple datasets. This dataset is used for composing fileds of multiple map-style datasets of same length. Args: datasets(list of Dataset): List of datasets to be composed. Returns: Dataset: A Dataset which composes fields of multiple datasets. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> from paddle.io import Dataset, ComposeDataset >>> # define a random dataset >>> class RandomDataset(Dataset): ... def __init__(self, num_samples): ... self.num_samples = num_samples ... ... def __getitem__(self, idx): ... image = np.random.random([32]).astype('float32') ... label = np.random.randint(0, 9, (1, )).astype('int64') ... return image, label ... ... def __len__(self): ... return self.num_samples ... >>> dataset = ComposeDataset([RandomDataset(10), RandomDataset(10)]) >>> for i in range(len(dataset)): ... image1, label1, image2, label2 = dataset[i] ... # do something cCst||_t|jdksJdt|jD]+\}}t|ts"Jdt|tr+Jd|dkr@t|t|j|dks@JdqdS)Nr"input datasets shoule not be emptyz.each input dataset should be paddle.io.Datasetz'paddle.io.IterableDataset not supportedz"lengths of datasets should be same)r2datasetslen enumerater1rrr r8idatasetr r r r ms*  zComposeDataset.__init__cCst|jdSr/)r9r8r r r r r|szComposeDataset.__len__cCs*g}|jD] }|t||qt|Sr)r8extendr4r.)r rsampler=r r r rs zComposeDataset.__getitem__N)rrrrr rrr r r r r5Ds ( r5c@s eZdZdZddZddZdS) ChainDataseta A Dataset which chains multiple iterable-style datasets. This dataset is used for assembling multiple datasets which should be :ref:`api_paddle_io_IterableDataset`. Args: datasets(list of IterableDatasets): List of datasets to be chainned. Returns: paddle.io.IterableDataset: A Dataset which chains fields of multiple datasets. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> from paddle.io import IterableDataset, ChainDataset >>> # define a random dataset >>> class RandomDataset(IterableDataset): ... def __init__(self, num_samples): ... self.num_samples = num_samples ... ... def __iter__(self): ... for i in range(10): ... image = np.random.random([32]).astype('float32') ... label = np.random.randint(0, 9, (1, )).astype('int64') ... yield image, label ... >>> dataset = ChainDataset([RandomDataset(10), RandomDataset(10)]) >>> for image, label in iter(dataset): ... # do something ... ... cCsJt||_t|jdksJdt|jD] \}}t|ts"JdqdS)Nrr6z3ChainDataset only support paddle.io.IterableDataset)r2r8r9r:r1rr;r r r r s zChainDataset.__init__ccs|jD]}|EdHqdSr)r8)r r=r r r r s  zChainDataset.__iter__N)rrrrr r r r r r r@s' r@c@r) Subseta Subset of a dataset at specified indices. Args: dataset (Dataset): The whole Dataset. indices (sequence): Indices in the whole set selected for subset. Returns: List[Dataset]: A Dataset which is the subset of the original dataset. Examples: .. code-block:: python >>> import paddle >>> from paddle.io import Subset >>> # example 1: >>> a = paddle.io.Subset(dataset=range(1, 4), indices=[0, 2]) >>> print(list(a)) [1, 3] >>> # example 2: >>> b = paddle.io.Subset(dataset=range(1, 4), indices=[1, 1]) >>> print(list(b)) [2, 2] cCs||_||_dSrr=indices)r r=rCr r r r s zSubset.__init__cCs|j|j|SrrBrr r r rr0zSubset.__getitem__cCs t|jSr)r9rCr r r r r zSubset.__len__Nrr r r r rAs  rAcsHt|tkr tdtt|fddtt||DS)a$ Randomly split a dataset into non-overlapping new datasets of given lengths. Optionally fix the generator for reproducible results, e.g.: Args: dataset (Dataset): Dataset to be split lengths (sequence): lengths of splits to be produced generator (Generator, optional): Generator used for the random permutation. Default is None then the DefaultGenerator is used in manual_seed(). Returns: Datasets: A list of subset Datasets, which are the non-overlapping subsets of the original Dataset. Examples: .. code-block:: python >>> import paddle >>> paddle.seed(2023) >>> a_list = paddle.io.random_split(range(10), [3, 7]) >>> print(len(a_list)) 2 >>> # output of the first subset >>> for idx, v in enumerate(a_list[0]): ... print(idx, v) # doctest: +SKIP("The output depends on the environment.") 0 7 1 6 2 5 >>> # output of the second subset >>> for idx, v in enumerate(a_list[1]): ... print(idx, v) # doctest: +SKIP("The output depends on the environment.") 0 1 1 9 2 4 3 2 4 0 5 3 6 8 zDSum of input lengths does not equal the length of the input dataset!cs&g|]\}}t|||qSr )rA)r'offsetlengthrBr r sz random_split..)sumr9 ValueErrorpaddleZrandpermtolistzip _accumulate)r=lengths generatorr rBr random_splits+  rPcCs||Srr )xyr r r srSccsPt|}zt|}Wn tyYdSw|V|D] }|||}|VqdS)a Return running totals Args: iterable: any iterable object for example dataset. y (x): one element in the iterable object. fn (x, y): Defaults to lambdax. Yields: yields total from beginning iterator to current iterator. Example code: .. code-block:: python >>> list(_accumulate([1, 2, 3, 4, 5])) [1, 3, 6, 10, 15] >>> import operator >>> list(_accumulate([1, 2, 3, 4, 5], operator.mul)) [1, 2, 6, 24, 120] N)iternext StopIteration)iterablefnittotalelementr r r rMs   rMc@s>eZdZdZeddZdeefddZddZ d d Z d S) ConcatDataseta Dataset as a concatenation of multiple datasets. This class is useful to assemble different existing datasets. Args: datasets (sequence): List of datasets to be concatenated Returns: Dataset: A Dataset which concatenated by multiple datasets. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> from paddle.io import Dataset, ConcatDataset >>> # define a random dataset >>> class RandomDataset(Dataset): ... def __init__(self, num_samples): ... self.num_samples = num_samples ... ... def __getitem__(self, idx): ... image = np.random.random([32]).astype('float32') ... label = np.random.randint(0, 9, (1, )).astype('int64') ... return image, label ... ... def __len__(self): ... return self.num_samples ... >>> dataset = ConcatDataset([RandomDataset(10), RandomDataset(10)]) >>> for i in range(len(dataset)): ... image, label = dataset[i] ... # do something cCs6gd}}|D]}t|}|||||7}q|Sr/)r9append)sequencerselr r r cumsumfs  zConcatDataset.cumsumr8cCsPt||_t|jdksJd|jD] }t|trJdq||j|_dS)Nrz(datasets should not be an empty iterablez.ConcatDataset does not support IterableDataset)r2r8r9r1rrccumulative_sizes)r r8dr r r r os  zConcatDataset.__init__cCs |jdS)N)rdr r r r rzrDzConcatDataset.__len__cCsf|dkr| t|krtdt||}t|j|}|dkr#|}n ||j|d}|j||S)Nrz8absolute value of index should not exceed dataset lengthr7)r9rIbisect bisect_rightrdr8)r rZ dataset_idxZ sample_idxr r r r}s zConcatDataset.__getitem__N) rrrr staticmethodrcrrr rrr r r r r\>s'  r\r)rgtypingrrJrrrr$r4r5r@rArPrMr\r r r r s  :84B5 (8#