Mirrors/text-generation-webui - text-generation-webui - Gitea: Git with a cup of tea

Mirrors/text-generation-webui

mirror of https://github.com/oobabooga/text-generation-webui.git synced 2024-12-27 06:39:33 +01:00

2 lines

78 B

Plaintext

Raw Normal View History

Add ability to load all text files from a subdirectory for training (#1997) * Update utils.py returns individual txt files and subdirectories to getdatasets to allow for training from a directory of text files * Update training.py minor tweak to training on raw datasets to detect if a directory is selected, and if so, to load in all the txt files in that directory for training * Update put-trainer-datasets-here.txt document * Minor change * Use pathlib, sort by natural keys * Space --------- Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>

2023-07-12 16:44:30 +02:00

to load multiple raw text files create a subdirectory and put them all there