gardel/lightrag
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,185 Commits 2 Branches 0 Tags
c0b22a8ae2a11ec03a57b366185403af31d1b180
Commit Graph

2185 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
yangdx
c0b22a8ae2 Merge branch 'main' into add-multi-worker-support 2025-03-02 02:54:57 +08:00
yangdx
e20aeada92 docs: add gunicorn deployment guide and update server 2025-03-02 02:43:11 +08:00
yangdx
8fb3670ac7 Add additional log-related patterns to gitignore 2025-03-02 02:26:21 +08:00
yangdx
7cd25fe5ab Improve shared storage cleanup and clarify initialization in multi-worker setup 2025-03-02 01:00:27 +08:00
yangdx
8d6960f280 Fix linting 2025-03-02 00:13:11 +08:00
yangdx
f76cf98dbd Add automatic dependency checking and installation for server startup 
• Added check_and_install_dependencies()
• Install missing dependencies automatically
 2025-03-02 00:04:59 +08:00
yangdx
e8d0d065f3 fix: Improve async handling and FAISS storage reliability 
- Add async context manager support
- Fix embedding data type conversion
- Improve error handling in FAISS ops
- Add multiprocess storage sync
 2025-03-01 23:35:09 +08:00
yangdx
9aef112d51 Fix incorrect comment about update flag behavior in FAISS implementation 2025-03-01 22:27:12 +08:00
zrguo
9aa438cf79 fix linting 2025-03-01 18:35:12 +08:00
zrguo
a8f4385c05 Add clear_cache 2025-03-01 18:30:58 +08:00
zrguo
a923ee89b0 Update __init__.py 2025-03-01 17:47:53 +08:00
zrguo
4219454fab fix format 2025-03-01 17:45:06 +08:00
yangdx
e3a40c2fdb Fix linting 2025-03-01 16:23:34 +08:00
yangdx
3507e894d9 Merge branch 'main' into add-multi-worker-support 2025-03-01 15:55:37 +08:00
yangdx
d18eb52ccc Add type ignore comments for asyncpg imports to suppress mypy errors 2025-03-01 15:38:39 +08:00
yangdx
3511b9805c Add auto-installation of gunicorn if not present in environment 2025-03-01 15:34:01 +08:00
yangdx
ab704aae47 Config log setting for very woker properly for Gunicorn mode. 2025-03-01 15:13:21 +08:00
yangdx
40e9e26edb feat: add update flags status to API health endpoint 2025-03-01 14:58:26 +08:00
yangdx
41eff2ca2f Fix data persistence issue in NanoVectorDBStorage 2025-03-01 13:35:00 +08:00
yangdx
48d9800573 Enhance gunicorn config handling with env vars and command line arg priority 
• Add env var support for timeout/keepalive
• Prioritize CLI args over env vars
• Standardize default timeout to 150s
 2025-03-01 12:58:52 +08:00
yangdx
35bcfca28f feat: add multi-process support for FAISS vector storage 
• Add storage update flag and locks
• Support cross-process index reload
• Add async initialize method
 2025-03-01 12:42:30 +08:00
yangdx
d4f6dcfd54 Improve multi-process data synchronization and persistence in storage implementations 
• Remove _get_client() or _get_graph() from index_done_callback
• Add return value for index_done_callback
 2025-03-01 12:41:30 +08:00
yangdx
c07a5039b7 Refactor shared storage locks to separate pipeline, storage and internal locks for deadlock preventing 2025-03-01 10:48:55 +08:00
yangdx
d3de57c1e4 Add multi-process support for vector database and graph storage with lock flags 
• Implement storage lock mechanism
• Add update flag handling
• Add cross-process reload detection
 2025-03-01 10:37:05 +08:00
yangdx
d704512139 Refactor shared storage module to improve async handling and naming consistency 
• Add async support for get_namespace_data
• Rename get_update_flags to get_update_flag
• Rename set_update_flag to set_all_update_flags
• Update docstrings for clarity
• Fix typos in log messages
 2025-03-01 05:01:26 +08:00
yangdx
fd76e00c6a Refactor storage initialization to separate object creation from data loading 
• Split __post_init__ and initialize()
• Move data loading to initialize()
• Add FastAPI lifespan integration
 2025-03-01 03:48:19 +08:00
Yannick Stephan
5bbe61a02d Merge pull request #960 from FeHuynhVI/fix-delete-by-doc-id 
feat: fix delete by document id
 2025-02-28 19:44:05 +01:00
yangdx
b3328542c7 refactor: migrate synchronous locks to async locks for improved concurrency 
• Add UnifiedLock wrapper class
• Convert with blocks to async with
 2025-03-01 02:22:35 +08:00
yangdx
a721421bd8 Add async support and update flag mechanism for shared storage 
• Use asyncio.Lock  instead of thread lock for single process mode
• Add storage update notification system
 2025-03-01 01:49:26 +08:00
yangdx
aac1bdd9e6 feat: add configurable log rotation settings via environment variables 
• Add LOG_DIR env var for log file location
• Add LOG_MAX_BYTES for max log file size
• Add LOG_BACKUP_COUNT for backup count
 2025-02-28 23:21:14 +08:00
yangdx
731d820bcc Remove redundancy set_logger function and related calls 2025-02-28 21:46:45 +08:00
yangdx
c973498c34 Fix linting 2025-02-28 21:35:04 +08:00
yangdx
c37b1e8aa7 Align Gunicorn configuration with Uvicorn 
- centralize config in gunicorn_config.py
- fix log level handling in Gunicorn
 2025-02-28 20:41:11 +08:00
yangdx
ff549a3a9c Update Gunicorn config with logging filters and worker-specific configurations 2025-02-28 17:45:40 +08:00
yangdx
f588cdc5df Optimize logging config & worker handling for different server modes 
• Separate logging config for uvicorn/gunicorn
• Force workers=1 in uvicorn mode
• Add warning for worker count in uvicorn
 2025-02-28 16:50:54 +08:00
yangdx
81f6f6e343 Fix lightrag logger initailization problem, fix gunicorn acccess log missing 2025-02-28 16:07:33 +08:00
yangdx
157ec862ae Enhance logging system with file rotation and unified configuration 
• Unify logging across Gunicorn and Uvicorn
• Add rotating file handlers
 2025-02-28 14:57:25 +08:00
yangdx
8cd45161f2 feat: add history_messages to track pipeline processing progress 
• Add shared history_messages list
• Track pipeline progress with messages
 2025-02-28 13:53:40 +08:00
yangdx
b090a22be7 Add concurrency check for auto scan task to prevent duplicate scans 
• Add pipeline status check before scan
• Add storage lock protection
• Add latest_message to status tracking
• Add helpful log message at startup
 2025-02-28 12:22:20 +08:00
yangdx
04bd5413c9 Add API endpoint to retrieve document indexing pipeline status 
• GET /pipeline_status endpoint added
• Returns current pipeline processing state
 2025-02-28 12:21:50 +08:00
yangdx
b2da69b7f1 Add pipeline status control for concurrent document indexing processes 
• Add shared pipeline status namespace
• Implement concurrent process control
• Add request queuing for pending jobs
 2025-02-28 11:52:42 +08:00
yangdx
feaa7ce69d Remove auto-scaling of workers based on CPU count in gunicorn config 2025-02-28 10:54:43 +08:00
yangdx
b4bcd76599 Remove useless scan progress tracking functionality and related code 2025-02-28 10:53:36 +08:00
yangdx
cd7648791a Fix linting 2025-02-28 01:25:59 +08:00
yangdx
3dcfa561d7 Remove debug logging 2025-02-28 01:15:12 +08:00
yangdx
291e0c1b14 revert vector and graph use local data(single process) 2025-02-28 01:14:25 +08:00
Huỳnh Triệu Vĩ
2f7fe5e4b6 feat: fix delete by document id 2025-02-27 23:34:57 +07:00
yangdx
db2a902fcb Rename get_scan_lock to get_storage_lock 2025-02-28 00:34:33 +08:00
yangdx
cf2f6b726c Add newline after Gunicorn initialization message for better readability 2025-02-27 20:26:12 +08:00
yangdx
05d03638ec Clean up logging output and remove redundant log messages 2025-02-27 20:17:28 +08:00