o
)j7 @ s d dl Z d dlZd dlZd dlZd dlZd dlmZmZ d dlm Z d dl
m Z m Z d dl
mZ d dlmZ e Zee d dkZ G d d
d
eZG d d d eZdS )
N)movermtree)Dict) FILE_HASH)MODELSCOPE_ENABLE_DEFAULT_HASH_VALIDATION) compute_hash)
get_loggerFalsetruec @ s| e Zd ZdZdZdZdZ defddZdd Z d
d Z
d d
Z dd Z dd Z
dd Zdd Zdd Zdd Zdd Zd S ) FileSystemCachez.mscz.mdlidz.mvcache_root_locationc K s* t | _tj|dd || _| dS )zBase file system cache interface.
Args:
cache_root_location (str): The root location to store files.
kwargs(dict): The keyword arguments.
Texist_okN) threadingRLock _cache_lockosmakedirsr
load_cache)selfr
kwargs r ]/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/modelscope/hub/utils/caching.py__init__ s
zFileSystemCache.__init__c C s | j S N)r
r r r r get_root_location1 s z!FileSystemCache.get_root_locationc C s g | _ tj| jtj}tj|rzz2t|d
}t
|}W d n1 s(w Y t |t r8|| _ W d S t
d|t|j W d S tyy } z)t
d|| z
t||d W n tyf Y nw W Y d }~d S W Y d }~d S d }~ww d S )Nrbz1Cache index %s has unexpected type %s, resetting.uk Failed to load cache index %s: %s. Resetting — already-downloaded files will be re-validated on next run.
.corrupted) cached_filesr pathjoinr
r
KEY_FILE_NAMEexistsopenpickleload
isinstancelistloggerwarningtype__name__ ExceptionreplaceOSError)r cache_keys_file_pathfdataer r r r 4 sB
zFileSystemCache.load_cachec C s t j| jtj}tjd| jd\}}z3t |d }t
t | j | |
t | W d n1 s9w Y t || W dS tyj zt | W n ty] Y nw t j|rit | w )z8Write .msc atomically. Caller must hold ``_cache_lock``.z.tmp)suffixdirwbN)r r! r" r
r r# tempfilemkstempfdopenr& dumpr) r flushfsyncfilenor/ r. closer0 r$ unlink)r r1 fdZ
temp_filenamer2 r r r _save_cached_files_unlockedN s.
z+FileSystemCache._save_cached_files_unlockedc C s4 | j
| W d dS 1 sw Y dS )a
Save cache metadata in order to verify that the cached content is consistent with the remote content.
Example of the cached content:
[{'Path': 'configuration.json', 'Revision': 'f01dxxx'}, {'Path': 'model.bin', 'Revision': '1159xxx'}, ...]
N)r rB r r r r save_cached_filesc s
"z!FileSystemCache.save_cached_filesc C dS )zCheck the key is in the cache, if exist, return the file, otherwise return None.
Args:
key(str): The cache key.
Raises:
None
Nr r keyr r r get_filem s zFileSystemCache.get_filec C rD )zPut file to the cache.
Args:
key (str): The cache key
location (str): Location of the file, we will move the file to cache.
Raises:
None
Nr )r rF locationr r r put_filex s
zFileSystemCache.put_filec C s& || j v r| j | | dS dS )zvRemove cache key in index, The file is removed manually
Args:
key (dict): The cache key.
N)r removerC rE r r r
remove_key s
zFileSystemCache.remove_keyc C s | j D ] }||kr dS qdS )NTF)r )r rF Z
cache_filer r r r$ s
zFileSystemCache.existsc C s t | j | dS )zRemove all files and metadata from the cache
In the case of multiple cache locations, this clears only the last one,
which is assumed to be the read/write one.
N)r r
r r r r r clear_cache s
zFileSystemCache.clear_cachec C s t | S r )hashlibsha256encode hexdigestrE r r r hash_name s zFileSystemCache.hash_nameN)r-
__module__ __qualname__r# MODEL_META_FILE_NAMEMODEL_META_MODEL_IDMODEL_VERSION_FILE_NAMEstrr r r rB rC rG rI rK r$ rL rQ r r r r r s&
r c s e Zd ZdZd fdd Zdd Zdd Zd efd
d Zd d
Z dd Z
dd Z dd Z dd Z
dd Zdd Zdd Zd d Z ZS ) ModelFileSystemCachezLocal cache file layout
cache_root/owner/model_name/individual cached files and cache index file '.mcs'
Save only one version for each file.
Nc sp |du s|du rt tj| | nt tj||| tjd||f i| _| |
| _ dS )a\ Put file to the cache
Args:
cache_root(`str`): The modelscope local cache root(default: ~/.cache/modelscope/hub)
owner(`str`): The model owner.
name('str'): The name of the model
Returns:
Raises:
None
model_id = {owner}/{name}
Nz%s/%s) superr r r! r" load_model_metar rU
model_metasave_model_metaload_model_versionZcached_model_revision)r Z
cache_rootownername __class__r r r s
z ModelFileSystemCache.__init__c C s t j| jtj}t j|rjz-t|d
}t |}W d n1 s%w Y t
|t r5|| _ W d S t
d| W n- tyi } z!t
d|| z
t ||d W n ty^ Y nw W Y d }~nd }~ww tjdi| _ d S )Nr z-Model meta %s has unexpected type, resetting.z,Failed to load model meta %s: %s. Resetting.r unknown)r r! r" r
r rT r$ r% r& r' r( dictr[ r* r+ r. r/ r0 rU )r meta_file_pathr2 r3 r4 r r r rZ s4
z$ModelFileSystemCache.load_model_metac C s\ t j| jtj}t j|r,t|d}| W d S 1 s%w Y d S d S )Nr)
r r! r" r
r rV r$ r% readstrip)r model_version_file_pathr2 r r r r] s
$z'ModelFileSystemCache.load_model_version
revision_infoc C s t j| jtj}t|d+}t|tr$d|d |d f }| | n
| | W d d S W d d S 1 s s"