aboutsummaryrefslogtreecommitdiff
path: root/weed/mount/meta_cache
AgeCommit message (Collapse)AuthorFilesLines
2025-12-09mount: improve EnsureVisited performance with dedup, parallelism, and ↵Chris Lu2-33/+92
batching (#7697) * mount: add singleflight to deduplicate concurrent EnsureVisited calls When multiple goroutines access the same uncached directory simultaneously, they would all make redundant network requests to the filer. This change uses singleflight.Group to ensure only one goroutine fetches the directory entries while others wait for the result. This fixes a race condition where concurrent lookups or readdir operations on the same uncached directory would: 1. Make duplicate network requests to the filer 2. Insert duplicate entries into LevelDB cache 3. Waste CPU and network bandwidth * mount: fetch parent directories in parallel during EnsureVisited Previously, when accessing a deep path like /a/b/c/d, the parent directories were fetched serially from target to root. This change: 1. Collects all uncached directories from target to root first 2. Fetches them all in parallel using errgroup 3. Relies on singleflight (from previous commit) for deduplication This reduces latency when accessing deep uncached paths, especially in high-latency network environments where parallel requests can significantly improve performance. * mount: add batch inserts for LevelDB meta cache When populating the meta cache from filer, entries were inserted one-by-one into LevelDB. This change: 1. Adds BatchInsertEntries method to LevelDBStore that uses LevelDB's native batch write API 2. Updates MetaCache to keep a direct reference to the LevelDB store for batch operations 3. Modifies doEnsureVisited to collect entries and insert them in batches of 100 entries Batch writes are more efficient because: - Reduces number of individual write operations - Reduces disk syncs - Improves throughput for large directories * mount: fix potential nil dereference in MarkChildrenCached Add missing check for inode existence in inode2path map before accessing the InodeEntry. This prevents a potential nil pointer dereference if the inode exists in path2inode but not in inode2path (which could happen due to race conditions or bugs). This follows the same pattern used in IsChildrenCached which properly checks for existence before accessing the entry. * mount: fix batch flush when last entry is hidden The previous batch insert implementation relied on the isLast flag to flush remaining entries. However, if the last entry is a hidden system entry (like 'topics' or 'etc' in root), the callback returns early and the remaining entries in the batch are never flushed. Fix by: 1. Only flush when batch reaches threshold inside the callback 2. Flush any remaining entries after ReadDirAllEntries completes 3. Use error wrapping instead of logging+returning to avoid duplicate logs 4. Create new slice after flush to allow GC of flushed entries 5. Add documentation for batchInsertSize constant This ensures all entries are properly inserted regardless of whether the last entry is hidden, and prevents memory retention issues. * mount: add context support for cancellation in EnsureVisited Thread context.Context through the batch insert call chain to enable proper cancellation and timeout support: 1. Use errgroup.WithContext() so if one fetch fails, others are cancelled 2. Add context parameter to BatchInsertEntries for consistency with InsertEntry 3. Pass context to ReadDirAllEntries for cancellation during network calls 4. Check context cancellation before starting work in doEnsureVisited 5. Use %w for error wrapping to preserve error types for inspection This prevents unnecessary work when one directory fetch fails and makes the batch operations consistent with the existing context-aware APIs.
2025-11-25Add error list each entry func (#7485)tam-i131-2/+2
* added error return in type ListEachEntryFunc * return error if errClose * fix fmt.Errorf * fix return errClose * use %w fmt.Errorf * added entry in messege error * add callbackErr in ListDirectoryEntries * fix error * add log * clear err when the scanner stops on io.EOF, so returning err doesn’t surface EOF as a failure. * more info in error * add ctx to logs, error handling * fix return eachEntryFunc * fix * fix log * fix return * fix foundationdb test s * fix eachEntryFunc * fix return resEachEntryFuncErr * Update weed/filer/filer.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update weed/filer/elastic/v7/elastic_store.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update weed/filer/hbase/hbase_store.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update weed/filer/foundationdb/foundationdb_store.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update weed/filer/ydb/ydb_store.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * fix * add scanErr --------- Co-authored-by: Roman Tamarov <r.tamarov@kryptonite.ru> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: chrislu <chris.lu@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-05-22added context to filer_client method calls (#6808)Aleksey Kosov1-1/+1
Co-authored-by: akosov <a.kosov@kryptonite.ru>
2025-03-19Fixes files with TTL can not be read in a mounted folder (#6646)jang1lee1-1/+1
2025-03-12Fixes files with TTL are not listed in a mounted folder. (#6621)jang1lee1-1/+1
2024-10-09avoid nilchrislu1-11/+13
I1010 03:10:25.220345 mount_std.go:281 This is SeaweedFS version 30GB 3.76 82ed61c6552e5095c682131012c91594ad2643e6 linux amd64 ^[OR^[OR2024/10/10 03:12:56 Unimplemented opcode POLL panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x3983b41] goroutine 112 [running]: github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents.func2.mergeProceesors.1(0xc000513540) /github/workspace/weed/mount/meta_cache/meta_cache_subscribe.go:30 +0x41 github.com/seaweedfs/seaweedfs/weed/pb.WithFilerClientFollowMetadata.makeSubscribeMetadataFunc.func1({0x514a560, 0xc0009b06d0}) /github/workspace/weed/pb/filer_pb_tail.go:87 +0x293 github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).WithFilerClient.func1.1(0xc000d9e008) /github/workspace/weed/mount/wfs_filer_client.go:27 +0x68 github.com/seaweedfs/seaweedfs/weed/pb.WithGrpcClient(0xa9?, 0x9d1f90fb?, 0xc00098fbe0, {0xc000e1c078, 0x15}, 0x0?, {0xc000a76bd0?, 0x1?, 0x1? }) /github/workspace/weed/pb/grpc_client_server.go:155 +0x33a github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).WithFilerClient.func1() /github/workspace/weed/mount/wfs_filer_client.go:25 +0x14c github.com/seaweedfs/seaweedfs/weed/util.Retry({0x489a10c, 0xa}, 0xc00098fcf8) /github/workspace/weed/util/retry.go:16 +0xa7 github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).WithFilerClient(0x0?, 0x30?, 0xc00056a568?) /github/workspace/weed/mount/wfs_filer_client.go:18 +0x65 github.com/seaweedfs/seaweedfs/weed/pb.WithFilerClientFollowMetadata({0x50f5898, 0xc000e2c180}, 0xc00101e000, 0xc000d22030) /github/workspace/weed/pb/filer_pb_tail.go:51 +0x87 github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents.func2() /github/workspace/weed/mount/meta_cache/meta_cache_subscribe.go:115 +0xcb github.com/seaweedfs/seaweedfs/weed/util.RetryUntil({0x48b8b35, 0x11}, 0xc00098fef0, 0x4af0048) /github/workspace/weed/util/retry.go:64 +0x4e github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents(0xc000d1d980, 0x9d1f90fb, {0x50f5898, 0xc000e2c180}, {0x7ffdf6f98dca, 0x4}, 0x17fcf8b7c7ecf8cf, {0xc0000ac208, 0x1, 0x1}) /github/workspace/weed/mount/meta_cache/meta_cache_subscribe.go:113 +0x39b created by github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).StartBackgroundTasks in goroutine 1 /github/workspace/weed/mount/weedfs.go:150 +0x146
2024-10-06use only one metadata follow processchrislu1-3/+38
2024-08-08fuse mount handles ttl entrieschrislu1-0/+7
fix https://github.com/seaweedfs/seaweedfs/issues/5527
2024-07-12ensure metadata follow a specific folderchrislu1-1/+7
fix https://github.com/seaweedfs/seaweedfs/issues/5774
2023-10-13disconnect from old subscriberschrislu1-3/+2
2023-10-01RetryForever => RetryUntilchrislu1-1/+1
2023-07-11Fix data loss: add lock for metacache (#4664)wusong1-18/+19
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-03-21refactoringchrislu1-1/+13
2023-01-29[mount] fix metacache update (#4161)wusong1-2/+6
2022-12-08Fix hardlink counting (#4042)wusong1-1/+3
Signed-off-by: wusong <wangwusong@virtaitech.com> Signed-off-by: wusong <wangwusong@virtaitech.com> Co-authored-by: wusong <wangwusong@virtaitech.com>
2022-09-20s3 to watch specific directorieschrislu1-1/+1
2022-09-15docs(meta_cache): readability fixes (#3691)Ryan Russell2-2/+2
Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-07-31Revert "rename: delete source entry metadata only, skipping hard links"chrislu2-16/+4
This reverts commit 03466f955e7907f5e7442dd3e60c45a3ab261ea3. fix https://github.com/seaweedfs/seaweedfs/issues/3386
2022-07-29move to https://github.com/seaweedfs/seaweedfschrislu4-15/+15
2022-07-23metadata subscription uses client epochchrislu1-1/+3
2022-06-02fix filer.sync missing source srv uploaded files to target when target downcreeew1-1/+1
2022-05-30subscribe metadata between a rangechrislu1-1/+1
2022-05-24remove unused parameterchrislu1-13/+4
2022-05-24mount: do not mark meta_cache when there is an errRobert Coelho1-1/+2
2022-02-28mount: add back support for filer.pathchrislu2-2/+5
2022-02-25rename: delete source entry metadata only, skipping hard linkschrislu2-4/+16
2022-02-25refactoringchrislu1-2/+2
2022-02-16list entries while reading from remotechrislu1-8/+19
2022-02-14listen for metadata updateschrislu5-0/+428
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-25 08:48:08 +0000