32 Commits

Author	SHA1	Message	Date
oobabooga	37754164eb	Move argparse	2023-03-29 20:47:36 -03:00
oobabooga	6403e72062	Merge branch 'main' into nikita-skakun-optimize-download-model	2023-03-29 20:45:33 -03:00
oobabooga	1445ea86f7	Add --output and better metadata for downloading models	2023-03-29 20:26:44 -03:00
Nikita Skakun	aaa218a102	Remove unused import.	2023-03-28 18:32:49 -07:00
Nikita Skakun	ff515ec2fe	Improve progress bar visual style ... This commit reverts the performance improvements of the previous commit for for improved visual style of multithreaded progress bars. The style of the progress bar has been modified to take up the same amount of size to align them.	2023-03-28 18:29:20 -07:00
Nikita Skakun	4d8e101006	Refactor download process to use multiprocessing ... The previous implementation used threads to download files in parallel, which could lead to performance issues due to the Global Interpreter Lock (GIL). This commit refactors the download process to use multiprocessing instead, which allows for true parallelism across multiple CPUs. This results in significantly faster downloads, particularly for large models.	2023-03-28 14:24:23 -07:00
oobabooga	91aa5b460e	If both .pt and .safetensors are present, download only safetensors	2023-03-28 13:08:38 -03:00
Florian Kusche	19174842b8	Also download Markdown files	2023-03-26 19:41:14 +02:00
oobabooga	bb4cb22453	Download .pt files using download-model.py (for 4-bit models)	2023-03-24 00:49:04 -03:00
oobabooga	164e05daad	Download .py files using download-model.py	2023-03-19 20:34:52 -03:00
oobabooga	104293f411	Add LoRA support	2023-03-16 21:31:39 -03:00
oobabooga	1d7e893fa1	Merge pull request #211 from zoidbb/add-tokenizer-to-hf-downloads ... download tokenizer when present	2023-03-10 00:46:21 -03:00
oobabooga	875847bf88	Consider tokenizer a type of text	2023-03-10 00:45:28 -03:00
oobabooga	249c268176	Fix the download script for long lists of files on HF	2023-03-10 00:41:10 -03:00
Ber Zoidberg	ec3de0495c	download tokenizer when present	2023-03-09 19:08:09 -08:00
oobabooga	7c70e0e2a6	Fix the download script (sort of)	2023-03-02 14:05:21 -03:00
oobabooga	fe1771157f	Properly scrape huggingface for download links (for #122)	2023-02-24 14:06:42 -03:00
oobabooga	bb1dac2f76	Convert the download option (A-Z) to upper case	2023-02-20 15:50:48 -03:00
oobabooga	fd8070b960	Give some default options in the download script	2023-02-16 23:04:13 -03:00
oobabooga	bf9dd8f8ee	Add --text-only option to the download script	2023-02-12 00:42:56 -03:00
oobabooga	66862203fc	Only download safetensors if both pytorch and safetensors are present	2023-02-12 00:06:22 -03:00
oobabooga	219366342b	Sort imports according to PEP8 (based on #67)	2023-02-10 15:40:03 -03:00
oobabooga	9215e281ba	Add --threads option to the download script	2023-02-03 18:57:12 -03:00
oobabooga	03f084f311	Add safetensors support	2023-02-03 18:36:32 -03:00
oobabooga	1e541d4882	Update download-model.py	2023-01-21 00:43:00 -03:00
oobabooga	18ef72d7c0	Update download-model.py	2023-01-21 00:38:23 -03:00
81300	fffd49e64e	Add --branch option to the model download script	2023-01-20 22:51:56 +02:00
oobabooga	6456777b09	Clean things up	2023-01-16 16:35:45 -03:00
oobabooga	fcda5d7107	Fix the download script on windows (#6)	2023-01-13 09:05:21 -03:00
oobabooga	dd1bed2d8b	Fix the download script	2023-01-07 16:49:21 -03:00
oobabooga	5345685ead	Make paths cross-platform (should work on Windows now)	2023-01-07 16:33:43 -03:00
oobabooga	fed7233ff4	Add script to download models	2023-01-06 19:57:31 -03:00