Huỳnh Triệu Vĩ
2f7fe5e4b6
feat: fix delete by document id
2025-02-27 23:34:57 +07:00
yangdx
db2a902fcb
Rename get_scan_lock to get_storage_lock
2025-02-28 00:34:33 +08:00
yangdx
cf2f6b726c
Add newline after Gunicorn initialization message for better readability
2025-02-27 20:26:12 +08:00
yangdx
05d03638ec
Clean up logging output and remove redundant log messages
2025-02-27 20:17:28 +08:00
yangdx
05cf029bcc
fix: convert multiprocessing managed dict to normal dict before JSON dump
2025-02-27 20:16:53 +08:00
yangdx
27500191b4
Standarize scan progress namespace initialization
2025-02-27 19:08:36 +08:00
yangdx
64f22966a3
Fix linting
2025-02-27 19:05:51 +08:00
yangdx
946095ef80
Fix multiprocess dict creation logic, add process safety locks for namespace creation.
2025-02-27 19:03:53 +08:00
yangdx
92ecb0da97
Refactor document scanning progress share variable initialization
2025-02-27 16:07:00 +08:00
yangdx
e881bc0709
simplify process state management by removing redundant multiprocess flag
2025-02-27 15:36:12 +08:00
yangdx
1699b10a25
Refactor direct client/graph access to reduce redundant get calls in vector/graph ops
2025-02-27 15:14:54 +08:00
yangdx
438e4780a8
Refactor Faiss index access with helper method to improve code organization
2025-02-27 15:09:19 +08:00
yangdx
f007ebf006
Refactor initialization logic for vector, KV and graph storage implementations
...
• Add try_initialize_namespace check
• Move init code out of storage locks
• Reduce redundant init conditions
• Simplify initialization flow
• Make init thread-safer
2025-02-27 14:55:07 +08:00
yangdx
03d05b094d
Improve Gunicorn support and cleanup shared storage initialization
...
• Move Gunicorn check before other startup
• Improve startup flow organization
2025-02-27 14:13:42 +08:00
yangdx
7aec78833c
Implement Gunicorn+Uvicorn integration for shared data preloading
...
- Create run_with_gunicorn.py script to properly initialize shared data in the
main process before forking worker processes
- Revert unvicorn to single process mode only, and let gunicorn do all the multi-process jobs
2025-02-27 13:25:22 +08:00
yangdx
7c237920b1
Refactor shared storage to support both single and multi-process modes
...
• Initialize storage based on worker count
• Remove redundant global variable checks
• Add explicit mutex initialization
• Centralize shared storage initialization
• Fix process/thread lock selection logic
2025-02-27 08:48:33 +08:00
zrguo
781ce7dfe2
Merge pull request #953 from zhicwu/main
...
Use consistent format as in examples
2025-02-26 23:23:44 +08:00
Zhichun Wu
d79a9d7acc
consistent format
2025-02-26 23:04:21 +08:00
Yannick Stephan
c785b4ba7d
Merge pull request #952 from YanSte/version-2
...
Version 1.2.2
2025-02-26 12:13:52 +01:00
Yannick Stephan
8fa45556d0
updated version to 1.2.2
2025-02-26 12:12:08 +01:00
Yannick Stephan
3c9908b94a
fixed lint
2025-02-26 12:11:28 +01:00
Yannick Stephan
2ba3447a56
Merge branch 'HKUDS:main' into main
2025-02-26 12:10:43 +01:00
Yannick Stephan
4963305dc3
Merge pull request #950 from cnjack/feat/custom_doc_ids
...
add support for the single document and custom chunks method
2025-02-26 12:08:02 +01:00
Yannick Stephan
3dca265b64
Merge pull request #940 from FeHuynhVI/FEHuynhVI-fix-run-async-safely
...
fix this event loop is already running
2025-02-26 12:06:58 +01:00
yangdx
7436c06f6c
Fix linting
2025-02-26 18:11:16 +08:00
yangdx
7d12715f09
Refactor shared storage to safely handle multi-process initialization and data sharing
...
• Add namespace initialization check
• Use atomic operations for shared data
2025-02-26 18:11:02 +08:00
yangdx
4eb069d1d6
Initialize scan_progress with default values if not already set
2025-02-26 17:42:49 +08:00
yangdx
145bacc773
Add empty graph creation logging in NetworkXStorage
2025-02-26 17:42:30 +08:00
jack
fee90ddd9d
add support for the single document and custom chunks method
2025-02-26 14:41:10 +08:00
yangdx
41f5d208a9
fix: shared data intitialization failed for multi-worker
2025-02-26 13:32:15 +08:00
yangdx
2c019dbc7b
Refactor storage initialization to avoid redundant intitial data loads across processes, show init logs to first load only
2025-02-26 12:28:49 +08:00
yangdx
15a6a9cf7c
fix: log filtering void when uvicorn wokers is greater than 1
...
- Centralize logging setup
- Fix logger propagation issues
2025-02-26 12:23:35 +08:00
yangdx
2752a764ae
Refactor storage implementations to support both single and multi-process modes
...
• Add shared storage management module
• Support process/thread lock based on mode
2025-02-26 05:38:38 +08:00
Yannick Stephan
32b498b71c
Merge pull request #946 from ArnoChenFx/dev-webui
...
Enhance Graph Visualization with Configurable Depth and Layout Iterations
2025-02-25 19:40:17 +01:00
ArnoChen
798c37d4fa
build webui
2025-02-25 18:36:34 +08:00
ArnoChen
40a1a94a31
add graph depth and layout iteration settings
2025-02-25 18:32:53 +08:00
yangdx
8050b0f91b
feat: automatically initialize API manager in single process mode
...
- Add manager init check in __post_init__
- Call initialize_manager if needed
- Add info log message for init
- Ensure API manager is ready for use
2025-02-25 12:09:30 +08:00
yangdx
a642bb3190
refactor: use shared manager from main process for storage implementations.
2025-02-25 12:08:49 +08:00
yangdx
e22e014f22
feat(storage): Add shared memory support for FAISS
2025-02-25 11:25:06 +08:00
yangdx
362321204f
Merge branch 'main' into add-multi-worker-support
2025-02-25 11:15:12 +08:00
yangdx
087d5770b0
feat(storage): Add shared memory support for file-based storage implementations
...
This commit adds multiprocessing shared memory support to file-based storage implementations:
- JsonDocStatusStorage
- JsonKVStorage
- NanoVectorDBStorage
- NetworkXStorage
Each storage module now uses module-level global variables with multiprocessing.Manager() to ensure data consistency across multiple uvicorn workers. All processes will see
updates immediately when data is modified through ainsert function.
2025-02-25 11:10:13 +08:00
yangdx
7262f61b0e
add redis configuration and update workers default value
2025-02-25 10:47:27 +08:00
yangdx
04fc5ce604
Remove unspported endpoint from splash mesages
2025-02-25 09:45:14 +08:00
yangdx
ddc366b672
Optimize display_splash_screen function
...
- Merge System Configuration into Server Configuration section
- Add Workers parameter display after Port parameter
2025-02-25 09:44:17 +08:00
yangdx
d74a23d2cc
Add multiple workers support for API Server
2025-02-25 09:37:00 +08:00
Huỳnh Triệu Vĩ
4a3c6de4ba
remove character ticks
2025-02-25 04:18:52 +07:00
Huỳnh Triệu Vĩ
a4d88b8cd4
fix this event loop is already running
2025-02-25 04:16:22 +07:00
Yannick Stephan
7f96ab654a
Merge pull request #933 from ArnoChenFx/dev-webui
...
Add retrieval message code highlighting and copy button
2025-02-24 12:11:06 +01:00
ArnoChen
e54aa276c8
Merge remote-tracking branch 'origin/main' into dev-webui
2025-02-24 19:00:06 +08:00
ArnoChen
69b36d855b
build webui
2025-02-24 18:59:31 +08:00
