o
0j/ @ s d Z ddlZddlZddlZddlZddlmZmZ ddlm Z ddl
m Z m Z m
Z
ddlmZ ddlmZ ee d d
kZ G d d d eZG d
d deZdS )z
caching tools
N)movermtree)Dict) FILE_HASH'AISTUDIO_ENABLE_DEFAULT_HASH_VALIDATION CACHE_KEY ) compute_hash)logFalsetruec @ sx e Zd ZdZdZdZdZdZ defddZ d d
Z
d d Z d
d Z dd Z
dd Zdd Zdd Zdd Zdd ZdS ) FileSystemCacheinfoz.mscz.mdlidz.mvcache_root_locationc C s t j|dd || _| dS )zBase file system cache interface.
Args:
cache_root_location (str): The root location to store files.
kwargs(dict): The keyword arguments.
Texist_okN)osmakedirsr
load_cache)selfr r [/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/aistudio_sdk/utils/caching.py__init__ s
zFileSystemCache.__init__c C s | j S )location)r r r r r get_root_location- s z!FileSystemCache.get_root_locationc C sd g | _ tj| jtj}tj|r0t|d}t
|| _ W d dS 1 s)w Y dS dS )loadrbN) cached_filesr pathjoinr r
KEY_FILE_NAMEexistsopenpickler )r cache_keys_file_pathfr r r r 1 s
"zFileSystemCache.load_cachec C sd t j| jtj}t \}}t|d}t
| j | W d n1 s&w Y t || dS )zSave cache metadata.wbN)
r r r! r r
r" tempfilemkstempr$ r% dumpr r )r r& fdfnr' r r r save_cached_files: s
z!FileSystemCache.save_cached_filesc C dS )zCheck the key is in the cache, if exists, return the file, otherwise return None.
Args:
key(str): The cache key.
Raises:
None
Nr r keyr r r get_fileD s zFileSystemCache.get_filec C r/ )zPut file to the cache.
Args:
key (str): The cache key
location (str): Location of the file, we will move the file to cache.
Raises:
None
Nr )r r1 r r r r put_fileO s
zFileSystemCache.put_filec C s& || j v r| j | | dS dS )zvRemove cache key in index, The file is removed manually
Args:
key (dict): The cache key.
N)r remover. r0 r r r
remove_key[ s
zFileSystemCache.remove_keyc C s | j D ] }||kr dS qdS )checkTF)r )r r1 Z
cache_filer r r r# e s
zFileSystemCache.existsc C s t | j | dS )zRemove all files and metadata from the cache
In the case of multiple cache locations, this clears only the last one,
which is assumed to be the read/write one.
N)r r r r r r r clear_cachem s
zFileSystemCache.clear_cachec C s t | S )name)hashlibsha256encode hexdigestr0 r r r hash_nameu s zFileSystemCache.hash_nameN)__name__
__module__ __qualname____doc__r" MODEL_META_FILE_NAMEMODEL_META_MODEL_IDMODEL_VERSION_FILE_NAMEstrr r r r. r2 r3 r5 r# r7 r= r r r r r
s&
r
c s e Zd ZdZd fdd Zdd Zdd Zd efd
d Zd d
Z dd Z
dd Z dd Z dd Z
dd Zdd Zdd Zd d Z ZS ) ModelFileSystemCachezLocal cache file layout
cache_root/owner/model_name/individual cached files and cache index file '.mcs'
Save only one version for each file.
Nc sp |du s|du rt tj| | nt tj||| tjd||f i| _| |
| _ dS )aU Put file to the cache
Args:
cache_root(`str`): The aistudio local cache root(default: ~/.cache/aistudio/)
owner(`str`): The model owner.
name('str'): The name of the model
Returns:
Raises:
None
model_id = {owner}/{name}
Nz%s/%s) superr r r r! load_model_metar
rC
model_metasave_model_metaload_model_versionZcached_model_revision)r Z
cache_rootownerr8 __class__r r r s
z ModelFileSystemCache.__init__c C sj t j| jtj}t j|r-t|d}t || _
W d dS 1 s&w Y dS tj di| _
dS )zget metar Nunknown) r r r! r r
rB r# r$ r% r rI rC r Zmeta_file_pathr' r r r rH s
"z$ModelFileSystemCache.load_model_metac C s\ t j| jtj}t j|r,t|d}| W d S 1 s%w Y dS dS )zuse version inforN)
r r r! r r
rD r# r$ readstrip)r model_version_file_pathr' r r r rK s
$z'ModelFileSystemCache.load_model_version
revision_infoc C s t j| jtj}t|d.}t|tr'd|d | dp df }|
| n
|
| W d dS W d dS 1 s?w Y dS )zsave file infowzRevision:%s,CreatedAt:%sRevisionZ CreatedAtrO N) r r r! r r
rD r$
isinstancedictgetwrite)r rU rT r' Zversion_info_strr r r save_model_version s
"z'ModelFileSystemCache.save_model_versionc C s | j tj S )rZ )rI r
rC r r r r get_model_id s z!ModelFileSystemCache.get_model_idc C sP t j| jtj}t|d}t| j | W d dS 1 s!w Y dS )saver( N)
r r r! r r
rB r$ r% r+ rI rP r r r rJ s
"z$ModelFileSystemCache.save_model_metac C sN | j D ]!}||d kr$tj| j|d }tj|r | S | | qdS )zRetrieve the cache if there is file match the path.
Args:
file_path (str): The file path in the model.
Returns:
path: the full path of the file.
PathN)r r r r! r r# r5 )r file_path cached_filecached_file_pathr r r get_file_by_path s
z%ModelFileSystemCache.get_file_by_pathc C sj | j D ]/}||d kr2|d |s||d r2tj| j|d }tj|r-| S | | qdS )zRetrieve the cache if there is file match the path.
Args:
file_path (str): The file path in the model.
commit_id (str): The commit id of the file
Returns:
path: the full path of the file.
r_ rW N)r
startswithr r r! r r# r5 )r r` commit_idra rb r r r get_file_by_path_and_commit_id s
z3ModelFileSystemCache.get_file_by_path_and_commit_idc C sZ | |}| jD ]"}||kr*tj| j|d }tj|r"| S | | dS qdS )zCheck if exist cache file.
Args:
model_file_info (ModelFileInfo): The file information of the file.
Returns:
str: The file path.
r_ N)$_ModelFileSystemCache__get_cache_keyr r r r! r r# r5 )r model_file_info cache_keyra orig_pathr r r get_file_by_info s
z%ModelFileSystemCache.get_file_by_infoc C s |d |d d}|S )Nr sha)r_ rW r )r rh ri r r r Z__get_cache_key s z$ModelFileSystemCache.__get_cache_keyc C s | |}d}|d }tj| j|d }| jD ]B}|d |krZ|d |d s2|d |d rZ|t }|durZtj|rZt rGt
|}n|}||krQd} n
t d| d q|rjtj|redS |
| dS ) a Check the file is cached or not. Note existence check will also cover digest check
Args:
model_file_info (CachedFileInfo): The cached file info
Returns:
bool: If exists and has the same hash, return True otherwise False
Fr_ r rW NTzFile [z?] exists in cache but with a mismatched hash, will re-download.)rg r r r! r r rd r r# enable_default_hash_validationr r
r r5 ) r rh r1 Z is_existsr` Zcache_file_pathZ
cached_keyZ
expected_hashZcache_file_sha256r r r r# sB
zModelFileSystemCache.existsc C sZ | j D ]'}|d |d kr*| | tj| j|d }tj|r't| dS qdS )zWe in cache, remove it.
Args:
model_file_info (ModelFileInfo): The model file information from server.
r_ r N)r r5 r r r! r r# r4 )r rh ra r` r r r remove_if_exists4 s
z%ModelFileSystemCache.remove_if_existsc C sp | | | |}tj| j|d }tj|}tj|s'tj|dd t || | j
| | |S )a[ Put model on model_file_location to cache, the model first download to /tmp, and move to cache.
Args:
model_file_info (str): The file description returned by get_model_files.
model_file_location (str): The location of the temporary file.
Returns:
str: The location of the cached file.
r_ Tr )
rn rg r r r! r dirnamer# r r r appendr. )r rh Zmodel_file_locationri Zcache_full_pathZcache_file_dirr r r r3 C s
z ModelFileSystemCache.put_file)NN)r> r? r@ rA r rH rK r r\ r] rJ rc rf rk rg r# rn r3
__classcell__r r rM r rF z s
)rF )rA r9 r r% r) shutilr r typingr Zaistudio_sdk.configr r r utilr Z aistudio_sdkr
getenvrS lowerrm objectr
rF r r r r s d