o
#j @ s` d dl Z d dlZd dlZd dlZd dlZd dlmZ d dlm Z g Z
dZ dZ G dd de Z
dS ) N)_check_exists_and_download)Datasetz6https://dataset.bj.bcebos.com/imdb%2FaclImdb_v1.tar.gzZ 7c2ac02c03563afcf9b574c7e56c153ac @ sB e Zd ZdZdddZdd Zd
d Zd d
Zdd Zdd Z dS )Imdba
Implementation of `IMDB `_ dataset.
Args:
data_file(str): path to data tar file, can be set None if
:attr:`download` is True. Default None
mode(str): 'train' 'test' mode. Default 'train'.
cutoff(int): cutoff number for building word dictionary. Default 150.
download(bool): whether to download dataset automatically if
:attr:`data_file` is not set. Default True
Returns:
Dataset: instance of IMDB dataset
Examples:
.. code-block:: python
>>> # doctest: +TIMEOUT(75)
>>> import paddle
>>> from paddle.text.datasets import Imdb
>>> class SimpleNet(paddle.nn.Layer):
... def __init__(self):
... super().__init__()
...
... def forward(self, doc, label):
... return paddle.sum(doc), label
>>> imdb = Imdb(mode='train')
>>> for i in range(10):
... doc, label = imdb[i]
... doc = paddle.to_tensor(doc)
... label = paddle.to_tensor(label)
...
... model = SimpleNet()
... image, label = model(doc, label)
... print(doc.shape, label.shape)
[121] [1]
[115] [1]
[386] [1]
[471] [1]
[585] [1]
[206] [1]
[221] [1]
[324] [1]
[166] [1]
[598] [1]
Ntrain Tc C sj | dv s
J d| | | _|| _| jd u r)|s J dt|ttd|| _| || _| d S )N)r testz(mode should be 'train', 'test', but got z>data_file is not set and downloading automatically is disabledZimdb) lowermode data_filer URLMD5_build_work_dictword_idx
_load_anno)selfr
r cutoffdownload r Z/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddle/text/datasets/imdb.py__init__U s
z
Imdb.__init__c
s t t}td}| |D ]}|D ]
}|| d7 < qq fdd| D }t|dd d}tt | \}}t
tt |t t |} t || d< | S ) Nz/aclImdb/((train)|(test))/((pos)|(neg))/.*\.txt$ c s g | ]
}|d kr|qS )r r ).0xr r r
s s z)Imdb._build_work_dict..c S s | d | d fS )Nr r r )r r r r u s z'Imdb._build_work_dict..)key)
collections defaultdictintrecompile _tokenizeitemssortedlistzipdictrangelen)
r r Z word_freqpatterndocword
dictionarywords_r r r r r
k s
zImdb._build_work_dictc C s g }t | jB}| }|d urBt||jr2|||
d d t j
d | }|d usW d |S W d |S 1 sMw Y |S )Ns
zlatin-1)tarfileopenr
nextboolmatchnameappend extractfilereadrstrip translatestring punctuationencoder split)r r+ dataZtarftfr r r r# { s,
zImdb._tokenizec s t dj d}t dj d}jd g _g _|D ]}j fdd|D jd q$|D ]}j fdd|D jd q@d S )
NzaclImdb/z
/pos/.*\.txt$z
/neg/.*\.txt$r c g | ] }j | qS r r getr wZUNKr r r r z#Imdb._load_anno..r c rB r rC rE rG r r r rH r )r! r" r r docslabelsr# r7 )r Z pos_patternZ neg_patternr, r rG r r s
zImdb._load_annoc C s" t | j| t | j| gfS N)nparrayrI rJ )r idxr r r __getitem__ s "zImdb.__getitem__c C s
t | jS rK )r* rI )r r r r __len__ s
z Imdb.__len__)Nr r T)
__name__
__module__ __qualname____doc__r r
r# r rO rP r r r r r s
5 r )r r! r< r1 numpyrL Zpaddle.dataset.commonr Z paddle.ior __all__r r r r r r r s