Mirrors/text-generation-webui
1
0
Fork 0
mirror of https://github.com/oobabooga/text-generation-webui.git synced 2024-11-24 00:46:53 +01:00
Code Issues Packages Projects Releases Wiki Activity
c54d1daaaa
text-generation-webui/training/datasets/put-trainer-datasets-here.txt

2 lines
78 B
Plaintext
Raw Normal View History

Add ability to load all text files from a subdirectory for training (#1997) * Update utils.py returns individual txt files and subdirectories to getdatasets to allow for training from a directory of text files * Update training.py minor tweak to training on raw datasets to detect if a directory is selected, and if so, to load in all the txt files in that directory for training * Update put-trainer-datasets-here.txt document * Minor change * Use pathlib, sort by natural keys * Space --------- Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
2023-07-12 16:44:30 +02:00
to load multiple raw text files create a subdirectory and put them all there
Reference in New Issue Copy Permalink