o jlG@s"ddlZddlZddlmZddlmZddlmZddl m Z m Z m Z m Z mZmZgZejjjdejjjd ejjjd ejjjd ejjjd ejjjd ejjjdejjjdejjjdejjjdejjjdejjjdi Zejjjejjjejjjejjjejjjejjjejjjejjjejjjejjjejjjejjjd Z ej!jdej!j"d ej!j#d ej!j$d ej!j%d ej!jd ej!jdej!jdej!jdej!jdej!jdej!jdi Z&d,ddZ'ddZ(ddZ) d-ddZ*d-dd Z+ d-d!d"Z,e-e.e fe/e fd#fd$d%Z0Gd&d'd'Z1Gd(d)d)Z2Gd*d+d+Z3dS).N)Value) ParameterMeta)core)Variable_cpu_num _cuda_idsdefault_main_programin_dygraph_mode in_pir_modeboolfloat16uint16float32float64int8int16int32int64uint8 complex64 complex128 r rrrrrrrrrrrNCHWcCst|jdkr |tjS|dkrt|gd}tjddtjgd|j}t||j}|dkr8t|gd}|S)NrZNHWC)rrrcSstdtd|dd?S)NWz)convert_float_to_uint16..Zotypes)rrrr) sizeviewnprZ transpose vectorizeflatreshapeshape)dataZ data_formatnew_datar%r%r&convert_float_to_uint16Os  r3cCs*tjddtjgd|j}t||jS)NcSstdtd|d>dS)Nrrrrrr#r%r%r&r'cr(z)convert_uint16_to_float..r))r,r-rr.r/r0)r1r2r%r%r&convert_uint16_to_floatasr4c Cst|tjjr|tvrt|St|tjr|tvrt|Sn2t|trB|tt j t j t j t j t jt jt jt jt jt jt jf vrA|jSn|dvrJt|S|dvrPdStd|)Nr)Zbfloat16rzdtype must be any of [bool, float16, uint16, float32, float64, int8, int16, int32, int64, uint8, complex64, complex128, bfloat16], but received %s) isinstancerVarDescVarType_PADDLE_DTYPE_2_NUMPY_DTYPEDataType_PADDLE_PIR_DTYPE_2_NUMPY_DTYPEtyper r,rrrrrrrrrrr__name__str TypeErrordtyper%r%r& convert_dtypeisB  rAcCsBtrt||ttf||nt||t||t|j||||dSN)r check_typerrr check_dtyper@)input input_nameexpected_dtypeop_name extra_messager%r%r&check_variable_and_dtypes rKcCstrdSddlm}|rt|ts|f}|tjjf7}nt|tjjr0td|d|dt||sBtd |||t ||dS)Nr)in_to_static_modezPlease use `with base.dygraph.guard()` as context or `base.enable_dygraph()` to switch to imperative mode firstly. Because received 'z' in z is a imperative Variable.z6The type of '{}' in {} must be {}, but received {}. {}) r Z dygraph.baserLr5tuplereagerZTensorr>formatr;)rFrG expected_typerIrJrLr%r%r&rDs,   rDcCs4trdSt||vrtd|||t||dS)Nz;The data type of '{}' in {} must be {}, but received {}. {})r rAr>rO)Z input_dtyperGrHrIrJr%r%r&rEs rE)rrc CstrdSt|d|||dur9t|ts9|D]!}t|d|||dur8t|tr8t|jd||dd|q|durLt|trNt|jd||dSdSdS)Nr0zelement of shapez9If element of shape is Tensor, its data type should be {}z, )r rDr5rrEr@rOjoin)r0rIZexpected_shape_typeZexpected_element_typeZexpected_tensor_dtypeitemr%r%r& check_shapes( rSc@s<eZdZddZddZddZddZd d Zd d Zd S)DataToLoDTensorConvertercCsZ||_||_||_d}|jD]}|dkr|d7}|dkr!d|_nqt||_|dSNrr)place lod_levelr0rAr@_reset)selfrVrWr0r@Z negtive_countsr%r%r&__init__s   z!DataToLoDTensorConverter.__init__cCs g|_ddt|jD|_dS)NcSsg|]}gqSr%r%).0_r%r%r& sz3DataToLoDTensorConverter._reset..)r1rangerWlodrYr%r%r&rXszDataToLoDTensorConverter._resetcCs|||j|jdSrC) _feed_impl_r`rW)rYr1r%r%r&feedszDataToLoDTensorConverter.feedcCsR|dkr |j|dS|dt||D]}|||dd|dqdSrU)r1appendlenrb)rYr1r`rWZ each_datar%r%r&rbs z$DataToLoDTensorConverter._feed_impl_cCsDt|j|D]\}}||kr|dkr|dkrtd|j|qdS)NrzDShape not match. What is defined in data layer is {}, but receive {})zipr0 ValueErrorrO)rYr0s1s2r%r%r& _check_shapesz%DataToLoDTensorConverter._check_shapecCstj|j|jd}|jr0t|jt|jkr0z||j}Wnty/td|j|jwt }| ||j |j dkrF||j||S)Nr?zBReshape error. What is defined in data layer is {}, but receive {}r)r,arrayr1r@r0rer/rgrOrZ LoDTensorsetrVrWZset_recursive_sequence_lengthsr`rX)rYZarrtr%r%r&done&s$   zDataToLoDTensorConverter.doneN) r< __module__ __qualname__r[rXrcrbrjrnr%r%r%r&rTs rTc@s$eZdZddZddZddZdS)BatchedTensorProviderc Cs\||_||_||_g|_||_|D]}|jdksJd|jt|jd|j|j dqdS)Nrzlod_level must be 0rVrWr0r@) rV batch_size generator converters drop_lastrWrdrTr0r@)rY feed_listrVrsrtrvvarr%r%r&r[;s zBatchedTensorProvider.__init__cCsdd|jDS)NcSg|]}|qSr%)rnr\cr%r%r&r^Nz/BatchedTensorProvider._done..)rurar%r%r&_doneMszBatchedTensorProvider._doneccsd}|D]#}t||jD] \}}|j|q|d7}||jkr*d}|Vq|js9|dkr9|VdSdd|jDdS)NrrcSryr%)rXrzr%r%r&r^^r|z2BatchedTensorProvider.__call__..)rtrfrur1rdrsr}rv)rYidx each_sample each_sloteach_converterr%r%r&__call__Ps   zBatchedTensorProvider.__call__N)r<rorpr[r}rr%r%r%r&rq:s rqc@s*eZdZdZd ddZddZddZdS) DataFeedera :api_attr: Static Graph DataFeeder converts the data that returned by a reader into a data structure that can feed into Executor. The reader is usually a python generator that returns a list of mini-batch data entries. Parameters: feed_list (list): Variables or names of Variables that need to feed. place (:ref:`api_paddle_CPUPlace` | :ref:`api_paddle_CUDAPlace` ): place indicates the device (CPU | GPU) the data will be fed into, if you want to feed data into GPU, please using :code:`base.CUDAPlace(i)` (:code:`i` represents the GPU id), or if you want to feed data into CPU, please using :code:`base.CPUPlace()`. program (:ref:`api_paddle_static_Program` , optional): The Program that will feed data into, if program is None, it will use default_main_program(). Default None. Raises: :code:`ValueError` - If some Variables are not in this Program. Example: .. code-block:: python >>> import numpy as np >>> import paddle >>> from paddle import base >>> paddle.enable_static() >>> place = paddle.CPUPlace() >>> def reader(): ... for _ in range(4): ... yield np.random.random([4]).astype('float32'), np.random.random([3]).astype('float32'), ... >>> main_program = paddle.static.Program() >>> startup_program = paddle.static.Program() >>> with paddle.static.program_guard(main_program, startup_program): ... data_1 = paddle.static.data(name='data_1', shape=[None, 2, 2], dtype='float32') ... data_2 = paddle.static.data(name='data_2', shape=[None, 1, 3], dtype='float32') ... out = paddle.static.nn.fc(x=[data_1, data_2], size=2) ... # ... >>> feeder = base.DataFeeder([data_1, data_2], place) >>> exe = paddle.static.Executor(place) >>> exe.run(startup_program) >>> feed_data = feeder.feed(reader()) >>> # print feed_data to view feed results >>> # print(feed_data['data_1']) >>> # print(feed_data['data_2']) >>> outs = exe.run( ... program=main_program, ... feed=feed_data, ... fetch_list=[out] ... ) >>> print(outs) NcCsg|_g|_g|_g|_|durt}|D]4}t|tr$|d|}t|t s-t d|j |j |j |j |j |j|j |jq||_dS)Nrz+Feed list should contain a list of variable) feed_dtypes feed_names feed_shapesfeed_lod_levelr r5r=blockrxrr>rdr@namerWr0rV)rYrwrVprogramZeach_varr%r%r&r[s    zDataFeeder.__init__c Csg}t|j|j|jD]\}}}|t|j|||dq |D]%}t|t|ks5Jdt|t|ft||D] \}}||q:qi} t|j |D] \} }| | | <qM| S)au According to :code:`feed_list` of :code:`DataFeeder` and :code:`iterable` , converts the input into a data structure that can feed into Executor. Parameters: iterable (generator): user defined python generator to read the raw input data Returns: :code:`dict`: a :code:`dict` that contains (variable name - converted tensor) pairs Example: .. code-block:: python >>> # In this example, reader - generator will return a list of ndarray of 3 elements >>> # feed API will convert each ndarray input into a tensor >>> # the return result is a dict with keys: data_1, data_2, data_3 >>> # result['data_1'] a LoD-Tensor with shape of [5, 2, 1, 3]. 5 is batch size, and [2, 1, 3] is the real shape of data_1. >>> # result['data_2'], result['data_3'] are similar. >>> import numpy as np >>> import paddle >>> from paddle import base >>> paddle.enable_static() >>> def reader(limit=5): ... for i in range(1, limit + 1): ... yield np.ones([6]).astype('float32') * i , np.ones([1]).astype('int64') * i, np.random.random([9]).astype('float32') ... >>> data_1 = paddle.static.data(name='data_1', shape=[None, 2, 1, 3]) >>> data_2 = paddle.static.data(name='data_2', shape=[None, 1], dtype='int64') >>> data_3 = paddle.static.data(name='data_3', shape=[None, 3, 3], dtype='float32') >>> feeder = base.DataFeeder(['data_1','data_2', 'data_3'], paddle.CPUPlace()) >>> result = feeder.feed(reader()) >>> print(result['data_1']) >>> print(result['data_2']) >>> print(result['data_3']) rrzDThe number of fields in data (%d) does not match len(feed_list) (%d)) rfrrrrdrTrVrercrrn) rYiterable converterrWr0r@rrrZret_dictZ each_namer%r%r&rcs0(   zDataFeeder.feedcCs.|durt|St|jtjrttStSrC)intr5rVrZ CUDAPlacerer r)rYZ num_placesr%r%r&_get_number_of_places_s  z!DataFeeder._get_number_of_places_rC)r<rorp__doc__r[rcrr%r%r%r&ras  ? Ar)r)rB)4r numpyr,ZpirrZpir.corerrBrZ frameworkrrr r r r __all__r6r7ZBOOLZFP16ZBF16ZFP32ZFP64ZINT8ZINT16ZINT32ZINT64ZUINT8Z COMPLEX64Z COMPLEX128r8Z_NUMPY_DTYPE_2_PADDLE_DTYPEr9ZFLOAT16ZBFLOAT16ZFLOAT32ZFLOAT64r:r3r4rArKrDrElistrMrrSrTrqrr%r%r%r&sz                 : #  ;'