yangdx
1699b10a25
Refactor direct client/graph access to reduce redundant get calls in vector/graph ops
2025-02-27 15:14:54 +08:00
yangdx
438e4780a8
Refactor Faiss index access with helper method to improve code organization
2025-02-27 15:09:19 +08:00
yangdx
f007ebf006
Refactor initialization logic for vector, KV and graph storage implementations
...
• Add try_initialize_namespace check
• Move init code out of storage locks
• Reduce redundant init conditions
• Simplify initialization flow
• Make init thread-safer
2025-02-27 14:55:07 +08:00
yangdx
03d05b094d
Improve Gunicorn support and cleanup shared storage initialization
...
• Move Gunicorn check before other startup
• Improve startup flow organization
2025-02-27 14:13:42 +08:00
yangdx
7aec78833c
Implement Gunicorn+Uvicorn integration for shared data preloading
...
- Create run_with_gunicorn.py script to properly initialize shared data in the
main process before forking worker processes
- Revert unvicorn to single process mode only, and let gunicorn do all the multi-process jobs
2025-02-27 13:25:22 +08:00
yangdx
7c237920b1
Refactor shared storage to support both single and multi-process modes
...
• Initialize storage based on worker count
• Remove redundant global variable checks
• Add explicit mutex initialization
• Centralize shared storage initialization
• Fix process/thread lock selection logic
2025-02-27 08:48:33 +08:00
yangdx
7436c06f6c
Fix linting
2025-02-26 18:11:16 +08:00
yangdx
7d12715f09
Refactor shared storage to safely handle multi-process initialization and data sharing
...
• Add namespace initialization check
• Use atomic operations for shared data
2025-02-26 18:11:02 +08:00
yangdx
145bacc773
Add empty graph creation logging in NetworkXStorage
2025-02-26 17:42:30 +08:00
yangdx
2c019dbc7b
Refactor storage initialization to avoid redundant intitial data loads across processes, show init logs to first load only
2025-02-26 12:28:49 +08:00
yangdx
2752a764ae
Refactor storage implementations to support both single and multi-process modes
...
• Add shared storage management module
• Support process/thread lock based on mode
2025-02-26 05:38:38 +08:00
yangdx
a642bb3190
refactor: use shared manager from main process for storage implementations.
2025-02-25 12:08:49 +08:00
yangdx
e22e014f22
feat(storage): Add shared memory support for FAISS
2025-02-25 11:25:06 +08:00
yangdx
362321204f
Merge branch 'main' into add-multi-worker-support
2025-02-25 11:15:12 +08:00
yangdx
087d5770b0
feat(storage): Add shared memory support for file-based storage implementations
...
This commit adds multiprocessing shared memory support to file-based storage implementations:
- JsonDocStatusStorage
- JsonKVStorage
- NanoVectorDBStorage
- NetworkXStorage
Each storage module now uses module-level global variables with multiprocessing.Manager() to ensure data consistency across multiple uvicorn workers. All processes will see
updates immediately when data is modified through ainsert function.
2025-02-25 11:10:13 +08:00
yangdx
9fd0ab185f
Removed unnecessary comment
2025-02-24 16:28:18 +08:00
yangdx
b8543b8701
Remove unnecessary comment in networkx_impl.py
2025-02-24 09:55:03 +08:00
yangdx
9dd18ccd89
Simplify log message
2025-02-24 03:52:43 +08:00
yangdx
6cf555ebd5
Fix linting
2025-02-24 03:33:22 +08:00
yangdx
b4543561f6
Limit subgraph size to 500 nodes in NetworkXStorage
...
- Add max_graph_nodes check
- Reduce subgraph by degree
- Log graph size reduction
2025-02-24 03:32:33 +08:00
yangdx
b13c947360
Implement get_all_labels and get_knowledge_graph methods in NetworkXStorage
2025-02-24 02:37:42 +08:00
zrguo
ff921b1274
Merge pull request #916 from ShanGor/main
...
Add support to namespace_prefix for postgres_impl
2025-02-21 22:53:06 +08:00
Samuel Chan
60ad8ccf12
Add support to namespace_prefix
2025-02-21 22:48:23 +08:00
yangdx
043dbad683
Merge remote-tracking branch 'origin/main' into refactor-api-server
2025-02-21 11:24:16 +08:00
Yannick Stephan
678e0f9aea
Revert "Cleanup of code"
2025-02-20 15:09:43 +01:00
Yannick Stephan
439685e69c
Revert "removed get_knowledge_graph"
2025-02-20 14:29:36 +01:00
Yannick Stephan
c4562f71b9
cleanup extraction
2025-02-20 14:17:26 +01:00
Yannick Stephan
214e3e8ad5
fixed last update
2025-02-20 14:12:19 +01:00
Yannick Stephan
8413537ad2
cleanup
2025-02-20 13:44:17 +01:00
Yannick Stephan
4923f5c1d7
cleanup kg
2025-02-20 13:39:46 +01:00
Yannick Stephan
c7bc2c63cf
cleanup storages
2025-02-20 13:21:41 +01:00
Yannick Stephan
32d0f1acb0
added docs and fields
2025-02-20 13:09:33 +01:00
Yannick Stephan
9f2c659d9c
remove unused log
2025-02-20 12:54:14 +01:00
Yannick Stephan
cedb5f3975
Merge pull request #886 from YanSte/clean-2
...
removed get_knowledge_graph
2025-02-19 22:59:10 +01:00
Yannick Stephan
89c35c82c7
removed get_knowledge_graph
2025-02-19 22:57:46 +01:00
Yannick Stephan
4978379b9c
Merge pull request #885 from YanSte/remove-unsused
...
Cleanup of code
2025-02-19 22:54:58 +01:00
Yannick Stephan
eb3306f34c
cleanup
2025-02-19 22:52:49 +01:00
Yannick Stephan
48a1ad9b3b
Merge pull request #883 from YanSte/fix-return-none
...
Optimised returns
2025-02-19 22:24:50 +01:00
Yannick Stephan
9277fe8c29
fixed return
2025-02-19 22:22:41 +01:00
yangdx
f776db0779
Improved document status retrieval with content fallback.
...
- Added content fallback to content_summary
- Handled missing fields gracefully
- Made data copy to avoid modification
- Added error logging for missing fields
- Improved code readability and robustness
2025-02-20 04:09:02 +08:00
Saifeddine ALOUI
45ee4dd08c
fixed linting
2025-02-19 20:50:39 +01:00
Saifeddine ALOUI
b44d582ebf
Update tidb_impl.py
2025-02-19 19:52:10 +01:00
Saifeddine ALOUI
473e52a095
Update qdrant_impl.py
2025-02-19 19:51:39 +01:00
Saifeddine ALOUI
521dbf2be1
Update postgres_impl.py
2025-02-19 19:50:46 +01:00
Saifeddine ALOUI
9ec46b8c5c
Update oracle_impl.py
2025-02-19 19:50:24 +01:00
Saifeddine ALOUI
d3c443529c
Update nano_vector_db_impl.py
2025-02-19 19:49:41 +01:00
Saifeddine ALOUI
b9d4ea5f0d
Update mongo_impl.py
2025-02-19 19:49:23 +01:00
Saifeddine ALOUI
ea5d8207a2
Update milvus_impl.py
2025-02-19 19:47:20 +01:00
Saifeddine ALOUI
d25a523175
Update gremlin_impl.py
2025-02-19 19:46:43 +01:00
Saifeddine ALOUI
8a0dbc97eb
Update faiss_impl.py
2025-02-19 19:44:27 +01:00
