增加仅字符分割参数，如果开启，仅采用字符分割，不开启，在分割完以后如果chunk过大，会继续根据token size分割，更新测试文件

Browse Source

...

This commit is contained in:

童石渊

2025-01-09 11:55:49 +08:00

parent b8f7a0a1e7

commit dd213c95be

4 changed files with 1328 additions and 758 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

1296

examples/test_split_by_character.ipynb Normal file

File diff suppressed because it is too large Load Diff