| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
batching (#7697)
* mount: add singleflight to deduplicate concurrent EnsureVisited calls
When multiple goroutines access the same uncached directory simultaneously,
they would all make redundant network requests to the filer. This change
uses singleflight.Group to ensure only one goroutine fetches the directory
entries while others wait for the result.
This fixes a race condition where concurrent lookups or readdir operations
on the same uncached directory would:
1. Make duplicate network requests to the filer
2. Insert duplicate entries into LevelDB cache
3. Waste CPU and network bandwidth
* mount: fetch parent directories in parallel during EnsureVisited
Previously, when accessing a deep path like /a/b/c/d, the parent directories
were fetched serially from target to root. This change:
1. Collects all uncached directories from target to root first
2. Fetches them all in parallel using errgroup
3. Relies on singleflight (from previous commit) for deduplication
This reduces latency when accessing deep uncached paths, especially in
high-latency network environments where parallel requests can significantly
improve performance.
* mount: add batch inserts for LevelDB meta cache
When populating the meta cache from filer, entries were inserted one-by-one
into LevelDB. This change:
1. Adds BatchInsertEntries method to LevelDBStore that uses LevelDB's
native batch write API
2. Updates MetaCache to keep a direct reference to the LevelDB store
for batch operations
3. Modifies doEnsureVisited to collect entries and insert them in
batches of 100 entries
Batch writes are more efficient because:
- Reduces number of individual write operations
- Reduces disk syncs
- Improves throughput for large directories
* mount: fix potential nil dereference in MarkChildrenCached
Add missing check for inode existence in inode2path map before accessing
the InodeEntry. This prevents a potential nil pointer dereference if the
inode exists in path2inode but not in inode2path (which could happen due
to race conditions or bugs).
This follows the same pattern used in IsChildrenCached which properly
checks for existence before accessing the entry.
* mount: fix batch flush when last entry is hidden
The previous batch insert implementation relied on the isLast flag to flush
remaining entries. However, if the last entry is a hidden system entry
(like 'topics' or 'etc' in root), the callback returns early and the
remaining entries in the batch are never flushed.
Fix by:
1. Only flush when batch reaches threshold inside the callback
2. Flush any remaining entries after ReadDirAllEntries completes
3. Use error wrapping instead of logging+returning to avoid duplicate logs
4. Create new slice after flush to allow GC of flushed entries
5. Add documentation for batchInsertSize constant
This ensures all entries are properly inserted regardless of whether
the last entry is hidden, and prevents memory retention issues.
* mount: add context support for cancellation in EnsureVisited
Thread context.Context through the batch insert call chain to enable
proper cancellation and timeout support:
1. Use errgroup.WithContext() so if one fetch fails, others are cancelled
2. Add context parameter to BatchInsertEntries for consistency with InsertEntry
3. Pass context to ReadDirAllEntries for cancellation during network calls
4. Check context cancellation before starting work in doEnsureVisited
5. Use %w for error wrapping to preserve error types for inspection
This prevents unnecessary work when one directory fetch fails and makes
the batch operations consistent with the existing context-aware APIs.
|
|
* added error return in type ListEachEntryFunc
* return error if errClose
* fix fmt.Errorf
* fix return errClose
* use %w fmt.Errorf
* added entry in messege error
* add callbackErr in ListDirectoryEntries
* fix error
* add log
* clear err when the scanner stops on io.EOF, so returning err doesn’t surface EOF as a failure.
* more info in error
* add ctx to logs, error handling
* fix return eachEntryFunc
* fix
* fix log
* fix return
* fix foundationdb test s
* fix eachEntryFunc
* fix return resEachEntryFuncErr
* Update weed/filer/filer.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update weed/filer/elastic/v7/elastic_store.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update weed/filer/hbase/hbase_store.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update weed/filer/foundationdb/foundationdb_store.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update weed/filer/ydb/ydb_store.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fix
* add scanErr
---------
Co-authored-by: Roman Tamarov <r.tamarov@kryptonite.ru>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
|
|
|
|
|
|
fix https://github.com/seaweedfs/seaweedfs/issues/5527
|
|
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
|
|
|
|
Signed-off-by: wusong <wangwusong@virtaitech.com>
Signed-off-by: wusong <wangwusong@virtaitech.com>
Co-authored-by: wusong <wangwusong@virtaitech.com>
|
|
This reverts commit 03466f955e7907f5e7442dd3e60c45a3ab261ea3.
fix https://github.com/seaweedfs/seaweedfs/issues/3386
|
|
|
|
|
|
|
|
|
