gardel/lightrag
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Enhance robustness of CVS processing ,Fix potential CSV parsing issues

Browse Source 
- Add CSV quoting for all fields
- Remove null characters from CSV input
- Improve CSV data integrity
- Ensure consistent CSV formatting
This commit is contained in:
yangdx
2025-01-27 15:32:27 +08:00
parent 2cd4fc0258
commit 66720409bd
1 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
lightrag/utils.py
View File

@@ -237,13 +237,13 @@ def truncate_list_by_token_size(list_data: list, key: callable, max_token_size:
def list_of_list_to_csv(data: List[List[str]]) -> str: def list_of_list_to_csv(data: List[List[str]]) -> str:
output = io.StringIO() output = io.StringIO()
writer = csv.writer(output) writer = csv.writer(output, quoting=csv.QUOTE_ALL)
writer.writerows(data) writer.writerows(data)
return output.getvalue() return output.getvalue()
def csv_string_to_list(csv_string: str) -> List[List[str]]: def csv_string_to_list(csv_string: str) -> List[List[str]]:
output = io.StringIO(csv_string) output = io.StringIO(csv_string.replace('\x00', ''))
reader = csv.reader(output) reader = csv.reader(output)
return [row for row in reader] return [row for row in reader]