o "jM@sddlZddlZddlZddlZddlZddlZddlZddlmZm Z m Z ddl m Z m Z ddlmZmZmZddlmZddlmZmZmZd Zdad ad add d ZddZddZGdddZGdddZ dS)N)_current_expected_place_get_paddle_place_get_paddle_place_list)corein_dynamic_mode) BatchSamplerIterableDatasetSubset)_InfiniteIterableSampler)_DataLoaderIterMultiProcess_DataLoaderIterSingleProcess _DatasetKind<FcCs |a|adSN) USE_AUTOTUNE TUNING_STEPS)Z use_autotuneZ tuning_stepsrQ/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddle/io/reader.pyset_autotune_config0srcGs:t|dkrtSt|dkrt|dtsJ|dadS)Nrr)lenUSE_PINNED_MEMORY isinstancebool)argsrrruse_pinned_memory7s  rcCsRt|ttfs |g}g}|D]}t|tjs!t}|||}||q|Sr)rlisttuplerZPlaceZ set_placeappend)placesretptmprrr_convert_places@s   r%c@sDeZdZddZddZddZddZd d Zd d Zd dZ dS)AuToTunecCs||_td|_dS)Nr)loadermultiprocessing cpu_countmax_num_worker)selfr'rrr__init__PszAuToTune.__init__cCs,tr|s |jjS|}|dur|jjSt}tdtdt|jjd}t d}tdt|j d}||j kr|||_| |}|d|krT|}|}n| ||||j }||krbn|}tdt|dt||d 7}||j ksAt d t|td tt|d |S) Nz(========= DataLoader Auto Tune =========zUser config for DataLoader: rinfz"Tuning Range for num_workers: 0 ~ g?z num_workers: avg_cost: rz'auto_tune dataLoader best_num_workers: z AutoTuning Cost for DataLoader: z seconds)r need_autotuner' num_workersget_autotune_loadertimeloggingdebugstrfloatr*evaluate_reader_costis_bestinfo)r+Zauto_tune_loaderZauto_tune_startZbest_num_workersZmin_costr0avg_costZ update_numrrr__call__Tsj        zAuToTune.__call__cCstjdks tjdkr dSdS)Ndarwinwin32FT)sysplatformr+rrrr/szAuToTune.need_autotunecCs*t|tt|}t|tt|d}|S)N)indices)minrrr rrange)r+dataset batch_sizeZ num_samples sub_datasetrrrget_sub_datasetszAuToTune.get_sub_datasetcCst|j}|jjj}t|jjtjjr:|jjj}| ||}tjj|||jjj |jjj |jjj |jjj d|_|St|jjtjjr^|jjjj}| ||}tjj|||jjj d|_|Sd}|S)N)rDrEZ num_replicasrankshuffle drop_last)rDrErJ)copyr' batch_samplerrErpaddleioZDistributedBatchSamplerrDrGZnranksZ local_rankrIrJr ZsamplerZ data_source)r+r'rErDrFrrrr1s6       zAuToTune.get_autotune_loadercCsg}d}t}t|D]\}}|t|t}q t|dkr6t|ddt|dd}|St|ddt|dd}|S)Nrr)r2 enumerater rsum)r+readerZcostsr:startidatarrrr7s    zAuToTune.evaluate_reader_costc Csd}|d}d}||krF|dkrF||j_||}tdt|dt||d7}||d|kr6|S|d7}|d9}||krF|dks|S)Nrrzfor back num_workers: r.gffffff?g?)r'r0r7r3r4r5) r+rQZ best_workersZ best_timeZnum_work_boundarystepr0boundaryr2rrrr8s, zAuToTune.is_bestN) __name__ __module__ __qualname__r,r;r/rGr1r7r8rrrrr&Os: r&c@sPeZdZdZ               ddd Zd d Zd d ZddZdS) DataLoadera DataLoader prodives an iterator which iterates given dataset once by the batch_sampler. DataLoader supports single-process and multi-prcess data loading, multi-process workers will be used to load data asynchronously if :attr:`num_workers` is set as a positive number. DataLoader supports map-style dataset and iterable-style dataset. For map-style datast(can get a sample from dataset with a given index), please see :code:`paddle.io.Dataset`. For iterable-style datast(get samples from dataset iteratively, like a Python iterator), please see :code:`paddle.io.IterableDataset`. For :code:`batch_sampler` please see :code:`paddle.io.BatchSampler` Notes: GPU tensor operation is not supported in subprocess currently, please don't use GPU tensor operations in pipeline which will be performed in subprocess, such as dataset transforms, collte_fn, etc. Numpy array and CPU tensor operation is supported. **Disable automatic batching** In certain cases such as some NLP tasks, instead of automatic batching, handling batching manually in dataset is needed by users. For these cases, automatic batching is disabled if both :attr:`batch_size` and :attr:`batch_sampler` is set as None, each data got from :attr:`dataset` should be batched data and will be processed with function define by :attr:`collate_fn` or :attr:`default_collate_fn`. Notes: When automatic batching is disabled, :attr:`default_collate_fn` will do nothing to data from dataset. Args: dataset(Dataset): the dataset to load data from, should be an instance of subclass of :code:`paddle.io.Dataset` or :code:`paddle.io.IterableDataset`. feed_list (list(Tensor)|tuple(Tensor), optional): feed Tensor list. The Tensors should be created by :code:`paddle.static.data()`. :attr:`feed_list` must be set if :attr:`return_list` is False. Default None. places(list(Place)|tuple(Place)|list(str), optional): a list of Place, to put data onto, :attr:`places` can be None, if :attr:`places` is None, default place(CPUPlace or CUDAPlace(0)) will be used. Default None. If ``places`` is list of string, the string in the list can be ``cpu``, ``gpu:x`` and ``gpu_pinned``, where ``x`` is the index of the GPUs. return_list (bool, optional): whether the return value on each device is presented as a list. If :attr:`return_list=False`, the return value on each device would be a dict of str -> Tensor, where the key of the dict is the name of each fed Tensors. If :attr:`return_list=True`, the return value on each device would be a list(Tensor). :attr:`return_list` can only be True in dynamic graph mode. Default True. batch_sampler(BatchSampler, optional): an instance of `paddle.io.BatchSampler` to generate batch indices to draw samples from :attr:`dataset` and combine a batch. Default None. batch_size(int|None, optional): sample number in a mini-batch, a substitution parameter for :attr:`batch_sampler`, if :attr:`batch_sampler` is not set, a default `paddle.io.BatchSampler` will be used and initialize by :attr:`batch_size`, :attr:`shuffle` and :attr:`drop_last`. Default 1. shuffle(bool, optional): whther to shuffle indices order before genrate batch indices, a substitution parameter for :attr:`batch_sampler` see :attr:`batch_size`. Default False. drop_last(bool, optional): whether drop the last incomplete batch dataset size is not divisible by the batch size, a substitution parameter for :attr:`batch_sampler`, see :attr:`batch_size`. Default False collate_fn(callable, optional): function to generate mini-batch data by merging the sample list, None for only stack each fields of sample in axis 0(same as :attr::`np.stack(..., axis=0)`). Default None num_workers(int, optional): the number of subprocess to load data, 0 for no subprocess used and loading data in main process. Default 0 use_buffer_reader (bool, optional): whether to use bufferred reader. If use_buffer_reader=True, the DataLoader would prefetch batch data asynchronously, so it would speed up data feeding and occupies a little more CPU or GPU memory, i.e., the memory of one batch input data. Default True. prefetch_factor (int, optional): Number of batch data the DataLoader would prefetch if use_buffer_reader=True. Default 2. use_shared_memory (bool, optional): whether to use shared memory to speed up putting data into inter-process queue, set :attr:`use_shared_memory` as True only when the shared memory space on your machine(e.g. space of '/dev/shm' on Linux operating sysytem) is large enough. Shared memory will only be enabled in multi-process mode(num_workers > 0). Default True. timeout(int, optional): the timeout value for getting data form output queue of subprocesses. Default 0. worker_init_fn(callable, optional): init function which will be called with worker id on each subproces starting if not set as None. Default None. Returns: DataLoader: an iterable object for data iterating, each elemnet of the generated data is a Tensor. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> import paddle.nn as nn >>> import paddle.nn.functional as F >>> from paddle.io import Dataset, BatchSampler, DataLoader >>> BATCH_NUM = 20 >>> BATCH_SIZE = 16 >>> EPOCH_NUM = 4 >>> IMAGE_SIZE = 784 >>> CLASS_NUM = 10 >>> # define a random dataset >>> class RandomDataset(Dataset): ... def __init__(self, num_samples): ... self.num_samples = num_samples ... ... def __getitem__(self, idx): ... image = np.random.random([IMAGE_SIZE]).astype('float32') ... label = np.random.randint(0, CLASS_NUM - 1, (1, )).astype('int64') ... return image, label ... ... def __len__(self): ... return self.num_samples ... >>> dataset = RandomDataset(BATCH_NUM * BATCH_SIZE) >>> class SimpleNet(nn.Layer): ... def __init__(self): ... super().__init__() ... self.fc = nn.Linear(IMAGE_SIZE, CLASS_NUM) ... ... def forward(self, image, label=None): ... return self.fc(image) ... >>> simple_net = SimpleNet() >>> opt = paddle.optimizer.SGD(learning_rate=1e-3, ... parameters=simple_net.parameters()) ... >>> loader = DataLoader(dataset, ... batch_size=BATCH_SIZE, ... shuffle=True, ... drop_last=True, ... num_workers=2) ... >>> for e in range(EPOCH_NUM): ... for i, (image, label) in enumerate(loader()): ... out = simple_net(image) ... loss = F.cross_entropy(out, label) ... avg_loss = paddle.mean(loss) ... avg_loss.backward() ... opt.minimize(avg_loss) ... simple_net.clear_gradients() ... print("Epoch {} batch {}: loss = {}".format(e, i, np.mean(loss.numpy()))) Notes: For reading iterable dataset with multiprocess Dataloader, please see :code:`paddle.io.IterableDataset` NTrFrrcCs||_| |_| |_| |_||_||_|sts|dusJd||_|dur)t}t |t t fr5t |}nt |}t||_| dksFJd| dkr[tjdksTtjdkr[tdd} | |_| dksfJd| |_| rr| dkrrd|_|dkszJd ||_t |trtj|_|rtd ||durtd ntj|_|dur|d kr|s|rJd ||_d|_n+|durd|_d|_n |dksJd||_t |trt|||_n t ||||d|_||_!|jdu|_"d|_#trt$durdnt$|_#||_%d|_&t'|(|_dS)Nz.feed_list should be set when return_list=Falserz*num_workers should be a non-negative valuer<r=zDataLoader with multi-process mode is not supported on MacOs and Windows currently. Please use signle-process mode with num_workers = 0 insteadz*prefetch_factor should be a positive valueFz&timeout should be a non-negative valuez5IterableDataset not support shuffle, but got shuffle=z0IterableDataset expect unspecified batch_samplerrzJbatch_size/shuffle/drop_last should not be set when batch_sampler is givenzMbatch_size should be None or a positive value when batch_sampler is not given)rDrErIrJT)) return_list collate_fnuse_buffer_readerprefetch_factorworker_init_fnrDr feed_listrrrrrrr%r!r>r?warningswarnr0use_shared_memorytimeoutr rITER dataset_kind ValueErrorZMAPrLrEr r rJauto_collate_batchZ pin_memoryr_persistent_workers _iteratorr&r;)r+rDrar!r\rLrErIrJr]r0r^r_rdrer`Zpersistent_workersrrrr,s         zDataLoader.__init__cCs.|jtjkr td|jrt|jSt|jS)Nz'length of IterableDataset not supported)rgrrfrhrirrLrDr@rrr__len__s   zDataLoader.__len__cCsJ|jdkr t|S|jr!|jdurt||_|jS|j|jSt|S)Nr)r0rrjrkr _resetr@rrr__iter__s    zDataLoader.__iter__cCs|Sr)rnr@rrrr;szDataLoader.__call__)NNTNrFFNrTrTrNF)rXrYrZ__doc__r,rlrnr;rrrrr[s,+ s r[)r)!rKr3r(r>r2rbrMZbase.frameworkrrrZ frameworkrrZ dataloaderr r r Zdataloader.batch_samplerr Zdataloader.dataloader_iterr rrZQUEUE_GET_TIMEOUTrrrrrr%r&r[rrrrs,