o j(@s.ddlmZddlmZgZGdddZdS)) text_format) data_feed_pb2c@s8eZdZdZddZddZddZdd Zd d Zd S) DataFeedDesca :api_attr: Static Graph Datafeed descriptor, describing input training data format. This class is currently only used for AsyncExecutor (See comments for class AsyncExecutor for a brief introduction) DataFeedDesc shall be initialized from a valid protobuf message from disk. See :code:`paddle/base/framework/data_feed.proto` for message definition. A typical message might look like: Examples: .. code-block:: python >>> import paddle.base as base >>> with open("data.proto", "w") as f: ... f.write('name: "MultiSlotDataFeed"\n') ... f.write('batch_size: 2\n') ... f.write('multi_slot_desc {\n') ... f.write(' slots {\n') ... f.write(' name: "words"\n') ... f.write(' type: "uint64"\n') ... f.write(' is_dense: false\n') ... f.write(' is_used: true\n') ... f.write(' }\n') ... f.write(' slots {\n') ... f.write(' name: "label"\n') ... f.write(' type: "uint64"\n') ... f.write(' is_dense: false\n') ... f.write(' is_used: true\n') ... f.write(' }\n') ... f.write('}') >>> data_feed = base.DataFeedDesc('data.proto') However, users usually shouldn't care about the message format; instead, they are encouraged to use :code:`Data Generator` as a tool to generate a valid data description, in the process of converting their raw log files to training files acceptable to AsyncExecutor. DataFeedDesc can also be changed during runtime. Once you got familiar with what each field mean, you can modify it to better suit your need. E.g.: .. code-block:: python >>> import paddle.base as base >>> data_feed = base.DataFeedDesc('data.proto') >>> data_feed.set_batch_size(128) >>> data_feed.set_dense_slots(['words']) # The slot named 'words' will be dense >>> data_feed.set_use_slots(['words']) # The slot named 'words' will be used >>> # Finally, the content can be dumped out for debugging purpose: >>> print(data_feed.desc()) Args: proto_file(string): Disk file containing a data feed description. cCs|t|_d|j_t|d}t||jWdn1s"wY|jjdkrYsz)DataFeedDesc.__init__..) rr proto_descZ pipe_commandopenrParsereadr enumeratemulti_slot_descslots_DataFeedDesc__name_to_index)selfZ proto_filefrrr __init__Ss     zDataFeedDesc.__init__cCs ||j_dS)a\ Set :attr:`batch_size` in ``paddle.base.DataFeedDesc`` . :attr:`batch_size` can be changed during training. Examples: .. code-block:: python >>> import paddle.base as base >>> with open("data.proto", "w") as f: ... f.write('name: "MultiSlotDataFeed"\n') ... f.write('batch_size: 2\n') ... f.write('multi_slot_desc {\n') ... f.write(' slots {\n') ... f.write(' name: "words"\n') ... f.write(' type: "uint64"\n') ... f.write(' is_dense: false\n') ... f.write(' is_used: true\n') ... f.write(' }\n') ... f.write(' slots {\n') ... f.write(' name: "label"\n') ... f.write(' type: "uint64"\n') ... f.write(' is_dense: false\n') ... f.write(' is_used: true\n') ... f.write(' }\n') ... f.write('}') >>> data_feed = base.DataFeedDesc('data.proto') >>> data_feed.set_batch_size(128) Args: batch_size (int): The number of batch size. Returns: None. N)r batch_size)rrrrr set_batch_size^s #zDataFeedDesc.set_batch_sizecC8|jjdkr td|D] }d|jjj|j|_q dS)a Set slots in :attr:`dense_slots_name` as dense slots. **Note: In default, all slots are sparse slots.** Features for a dense slot will be fed into a Tensor, while those for a sparse slot will be fed into a LoDTensor. Examples: .. code-block:: python >>> import paddle.base as base >>> with open("data.proto", "w") as f: ... f.write('name: "MultiSlotDataFeed"\n') ... f.write('batch_size: 2\n') ... f.write('multi_slot_desc {\n') ... f.write(' slots {\n') ... f.write(' name: "words"\n') ... f.write(' type: "uint64"\n') ... f.write(' is_dense: false\n') ... f.write(' is_used: true\n') ... f.write(' }\n') ... f.write(' slots {\n') ... f.write(' name: "label"\n') ... f.write(' type: "uint64"\n') ... f.write(' is_dense: false\n') ... f.write(' is_used: true\n') ... f.write(' }\n') ... f.write('}') >>> data_feed = base.DataFeedDesc('data.proto') >>> data_feed.set_dense_slots(['words']) Args: dense_slots_name (list(str)): a list of slot names which will be set dense. Returns: None. rzNOnly MultiSlotDataFeed needs set_dense_slots, please check your datafeed.protoTN)rr ValueErrorrrrZis_dense)rZdense_slots_namer rrr set_dense_slotss &zDataFeedDesc.set_dense_slotscCr)a Set if a specific slot will be used for training. A dataset shall contain a lot of features, through this function one can select which ones will be used for a specific model. Examples: .. code-block:: python >>> import paddle.base as base >>> with open("data.proto", "w") as f: ... f.write('name: "MultiSlotDataFeed"\n') ... f.write('batch_size: 2\n') ... f.write('multi_slot_desc {\n') ... f.write(' slots {\n') ... f.write(' name: "words"\n') ... f.write(' type: "uint64"\n') ... f.write(' is_dense: false\n') ... f.write(' is_used: true\n') ... f.write(' }\n') ... f.write(' slots {\n') ... f.write(' name: "label"\n') ... f.write(' type: "uint64"\n') ... f.write(' is_dense: false\n') ... f.write(' is_used: true\n') ... f.write(' }\n') ... f.write('}') >>> data_feed = base.DataFeedDesc('data.proto') >>> data_feed.set_use_slots(['words']) Args: use_slots_name: a list of slot names which will be used in training Note: Default is not used for all slots rzLOnly MultiSlotDataFeed needs set_use_slots, please check your datafeed.protoTN)rr rrrrZis_used)rZuse_slots_namer rrr set_use_slotss $zDataFeedDesc.set_use_slotscCs t|jS)a Returns a protobuf message for this DataFeedDesc Examples: .. code-block:: python >>> import paddle.base as base >>> with open("data.proto", "w") as f: ... f.write('name: "MultiSlotDataFeed"\n') ... f.write('batch_size: 2\n') ... f.write('multi_slot_desc {\n') ... f.write(' slots {\n') ... f.write(' name: "words"\n') ... f.write(' type: "uint64"\n') ... f.write(' is_dense: false\n') ... f.write(' is_used: true\n') ... f.write(' }\n') ... f.write(' slots {\n') ... f.write(' name: "label"\n') ... f.write(' type: "uint64"\n') ... f.write(' is_dense: false\n') ... f.write(' is_used: true\n') ... f.write(' }\n') ... f.write('}') >>> data_feed = base.DataFeedDesc('data.proto') >>> print(data_feed.desc()) Returns: A string message )rZMessageToStringr)rrrr descs zDataFeedDesc.descN) __name__ __module__ __qualname____doc__rrrrrrrrr rs< %/ -rN)Zgoogle.protobufrZpaddle.base.protor__all__rrrrr s