o )j7@sddlZddlZddlZddlZddlZddlmZmZddlm Z m Z m Z ddl m Z ddlmZmZddlmZmZmZmZddlmZddlmZdd lmZdd lmZdd lmZm Z eZ!gd Z"ej#d dZ$GdddZ%Gddde Z&dS)N)FutureThreadPoolExecutor)SEEK_ENDSEEK_SETBytesIO)Path)LockThread)DictListOptionalUnion)HubApi) Visibility)DEFAULT_REPOSITORY_REVISION) get_logger) CommitInfo RepoUtils)z.gitz.git/*z*/.gitz **/.git/**c#sj}|j}  d dtttttttfdtdtttttfdtttttfdttttff fdd }zt |||_dVW||_dS||_w) z'Patch upload_folder for CommitSchedulerNfolder_path_or_files path_in_repoallow_patternsignore_patternsreturnc s>jt|tr tdtj|rtdt| t dfddt dD}|r<|ddnd}g}i}tj|||d D].} || } | } j| d usfj| | jkrzt| | j} ||| | f| j|| <qL|_|st d Wd |SWd |S1swY|S) zX Patched version that supports incremental updates for CommitScheduler. zJUploading multiple files or folders is not supported for scheduled commit.z5Uploading file is not supported for scheduled commit.z-Listing files to upload for scheduled commit.cs$i|]}|r||qS)is_file relative_toas_posix).0path folder_pathr`/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/modelscope/hub/commit_scheduler.py 5s z\patch_upload_folder_for_scheduler..patched_prepare_upload_folder..z**/*/)rrN0No changed files to upload for scheduled commit.)lock isinstancelist ValueErrorosrisfiler expanduserresolveloggerdebugsortedglobstriprZfilter_repo_objectskeysstat last_uploadedgetst_mtime PartialFileIOst_sizeappend_pending_tracker_updates) Zapi_selfrrrrZrelpath_to_abspathprefixZprepared_repo_objectsZfiles_to_trackrelpathZ local_pathr4Z partial_filescheduler_instancerr!patched_prepare_upload_foldersh          * **zHpatch_upload_folder_for_scheduler..patched_prepare_upload_folder)NN) apiZ_prepare_upload_folderr strrr r tupler(types MethodType)r?rAZoriginal_preparer@rr>r!!patch_upload_folder_for_schedulers,6rFc@seZdZdZdddeejddddd dedeee fdee e fde ed e ed e ed e ed e ed e ee eefde ee eefde eddfddZd!ddZd"ddZd!ddZd!ddZdefddZde efddZde efdd ZdS)#CommitScheduleraF A scheduler that automatically uploads a local folder to ModelScope Hub at specified intervals (e.g., every 5 minutes). It's recommended to use the scheduler as a context manager to ensure proper cleanup and final commit execution when your script completes. Alternatively, you can manually stop the scheduler using the `stop` method. Args: repo_id (`str`): The id of the repo to commit to. folder_path (`str` or `Path`): Local folder path that will be monitored and uploaded periodically. interval (`int` or `float`, *optional*): Time interval in minutes between each upload operation. Defaults to 5 minutes. path_in_repo (`str`, *optional*): Target directory path within the repository, such as `"models/"`. If not specified, files are uploaded to the repository root. repo_type (`str`, *optional*): Repository type for the target repo. Defaults to `model`. revision (`str`, *optional*): Target branch or revision for commits. Defaults to `master`. visibility (`str`, *optional*): The visibility of the repo, could be `public`, `private`, `internal`, default to `public`. token (`str`, *optional*): The token to use to commit to the repo. Defaults to the token saved on the machine. allow_patterns (`List[str]` or `str`, *optional*): File patterns to include in uploads. Only files matching these patterns will be uploaded. ignore_patterns (`List[str]` or `str`, *optional*): File patterns to exclude from uploads. Files matching these patterns will be skipped. hub_api (`HubApi`, *optional*): Custom [`HubApi`] instance for Hub operations. Allows for customized configurations like user agent or token settings. Example: ```py >>> from pathlib import Path >>> from modelscope.hub import CommitScheduler # Create scheduler with 10-minute intervals >>> data_file = Path("workspace/experiment.log") >>> scheduler = CommitScheduler( ... repo_id="my_experiments", ... repo_type="dataset", ... folder_path=data_file.parent, ... interval=10 ... ) >>> with data_file.open("a") as f: ... f.write("experiment started") # Later in the workflow... >>> with data_file.open("a") as f: ... f.write("experiment completed") ``` Context manager usage: ```py >>> from pathlib import Path >>> from modelscope.hub import CommitScheduler >>> with CommitScheduler( ... repo_id="my_experiments", ... repo_type="dataset", ... folder_path="workspace", ... interval=10 ... ) as scheduler: ... log_file = Path("workspace/progress.log") ... with log_file.open("a") as f: ... f.write("starting process") ... # ... perform work ... ... with log_file.open("a") as f: ... f.write("process finished") # Scheduler automatically stops and performs final upload ``` N) intervalr repo_typerevision visibilitytokenrrhub_apirepo_idr rIrrJrKrLrMrrrNrc Cs,| pt|_t||_|jstd||pd|_| |_ | dur*g} nt | t r2| g} | t |_ |jj||||ddd|_||_||_||_||_i|_|dkr_td|dt|_||_d|_td |jd |jd |jd td d|_t|jdd|_|j t!"|j#dS)NzFolder path does not exist: r$TF)rOrMrJrLexist_okZcreate_default_configrz,"interval" must be a positive integer, not "z".zScheduled job to push z to z at an interval of z minutes.) max_workers)targetdaemon)$rrArr,r-r existsr)rrr'rBIGNORE_GIT_FOLDER_PATTERNSrZ create_reporepo_urlrOrJrKrMr5rr&rI_CommitScheduler__stoppedr.inforexecutorr _run_schedulerZ_scheduler_threadstartatexitregistercommit_scheduled_changes) selfrOr rIrrJrKrLrMrrrNrrr!__init__sP        zCommitScheduler.__init__cCs d|_dS)zStop the scheduler.TN)rXr`rrr!stops zCommitScheduler.stopcCs|SNrrbrrr! __enter__szCommitScheduler.__enter__cCs||dSrd)triggerresultrc)r`exc_type exc_value tracebackrrr!__exit__s zCommitScheduler.__exit__cCs.|js||_t|jd|jrdSdS)N<)rXrfZ last_futuretimesleeprIrbrrr!r[s zCommitScheduler._run_schedulercCs|j|jS)z0Trigger a background commit and return a future.)rZZsubmit_commit_scheduled_changesrbrrr!rfszCommitScheduler.triggerc CsN|jrdStdz|}|WSty&}z td|d}~ww)Nz((Background) scheduled commit triggered.zError while pushing to Hub: )rXr.rYr_ Exceptionerror)r`valueerrr!ros z)CommitScheduler._commit_scheduled_changesc CsHzui|_t||jj|j|j|jd|j|j|j |j |j d }Wdn1s*wY|dur;t dWdS|j-t|drb|j|jt dt|jd|`Wd|WSWd|WS1snwY|WSty}z"dt|vrt dWYd}~dSt|dr|`t d |d}~ww) zCPush folder to the Hub and return commit info if changes are found.zScheduled Commit) rOr rZcommit_messagerMrJrrrKNr%r;z!Updated modification tracker for z files.zNo files to uploadzError during scheduled commit: )r;rFrAZ upload_folderrOr rrMrJrrrKr.r/r&hasattrr5updatelenrprBrq)r`Z commit_inforsrrr!r_ s\       z(CommitScheduler.commit_scheduled_changesrN)rrG)__name__ __module__ __qualname____doc__rrZPUBLICrBr rintfloatr r rrarcrerkr[rrfrror_rrrr!rG]sVT        >    rGcseZdZdZdeeefdeddfddZddd Z dd d Z dfd d Z defddZ defddZ deffdd ZdefddZefdededefddZd deedefddZZS)!r8z)rrrbrrr!__repr__YszPartialFileIO.__repr__cCs|jSrd)rrbrrr!__len__\szPartialFileIO.__len__namecs.|ds |dvrt|Std|d)N_>seekreadrtellrz PartialFileIO does not support 'z'.) startswithr__getattribute__NotImplementedError)r`rrrr!r_s zPartialFileIO.__getattribute__cCs |jSrd)rrrbrrr!rfs zPartialFileIO.tell_PartialFileIO__offset_PartialFileIO__whencecCsB|tkr t||}t}|j||}||jkr|j|jS|S)zs,      C]