aboutsummaryrefslogtreecommitdiff
path: root/weed/mount
AgeCommit message (Collapse)AuthorFilesLines
2024-09-04support write once read manychrislu3-2/+12
fix https://github.com/seaweedfs/seaweedfs/issues/5954
2024-09-04Revert "weed mount, weed dav add option to force cache"chrislu1-2/+1
This reverts commit 7367b976b05bfa69158a60f205dec970c48f50f0.
2024-09-04weed mount, weed dav add option to force cachechrislu1-1/+2
2024-08-08fuse mount handles ttl entrieschrislu1-0/+7
fix https://github.com/seaweedfs/seaweedfs/issues/5527
2024-08-07simplifychrislu2-8/+2
2024-08-07randomizing next file handle idchrislu2-3/+5
2024-07-24Fix dead lock (#5815)Chris Lu9-37/+15
* reduce locks to avoid dead lock Flush->FlushData->uplloadPipeline.FluahAll uploaderCount>0 goroutine 1 [sync.Cond.Wait, 71 minutes]: sync.runtime_notifyListWait(0xc0007ae4d0, 0x0) /usr/local/go/src/runtime/sema.go:569 +0x159 sync.(*Cond).Wait(0xc001a59290?) /usr/local/go/src/sync/cond.go:70 +0x85 github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).waitForCurrentWritersToComplete(0xc0002ee4d0) /github/workspace/weed/mount/page_writer/upload_pipeline_lock.go:58 +0x32 github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).FlushAll(0xc0002ee4d0) /github/workspace/weed/mount/page_writer/upload_pipeline.go:151 +0x25 github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).FlushData(0xc00087e840) /github/workspace/weed/mount/dirty_pages_chunked.go:54 +0x29 github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).FlushData(...) /github/workspace/weed/mount/page_writer.go:50 github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).doFlush(0xc0006ad600, 0xc00030d380, 0x0, 0x0) /github/workspace/weed/mount/weedfs_file_sync.go:101 +0x169 github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).Flush(0xc0006ad600, 0xc001a594a8?, 0xc0004c1ca0) /github/workspace/weed/mount/weedfs_file_sync.go:59 +0x48 github.com/hanwen/go-fuse/v2/fuse.doFlush(0xc0000da870?, 0xc0004c1b08) SaveContent -> MemChunk.RLock -> ChunkedDirtyPages.saveChunkedFileIntervalToStorage pages.fh.AddChunks([]*filer_pb.FileChunk{chunk}) fh.entryLock.Lock() sync.(*RWMutex).Lock(0x0?) /usr/local/go/src/sync/rwmutex.go:146 +0x31 github.com/seaweedfs/seaweedfs/weed/mount.(*FileHandle).AddChunks(0xc00030d380, {0xc00028bdc8, 0x1, 0x1}) /github/workspace/weed/mount/filehandle.go:93 +0x45 github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).saveChunkedFileIntervalToStorage(0xc00087e840, {0x2be7ac0, 0xc00018d9e0}, 0x0, 0x121, 0x17e3c624565ace45, 0x1?) /github/workspace/weed/mount/dirty_pages_chunked.go:80 +0x2d4 github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*MemChunk).SaveContent(0xc0008d9130, 0xc0008093e0) /github/workspace/weed/mount/page_writer/page_chunk_mem.go:115 +0x112 github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).moveToSealed.func1() /github/workspace/weed/mount/page_writer/upload_pipeline.go:187 +0x55 github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute.func1() /github/workspace/weed/util/limited_executor.go:38 +0x62 created by github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute in goroutine 1 /github/workspace/weed/util/limited_executor.go:33 +0x97 On metadata update fh.entryLock.Lock() fh.dirtyPages.Destroy() up.chunksLock.Lock => each sealed chunk.FreeReference => MemChunk.Lock goroutine 134 [sync.RWMutex.Lock, 71 minutes]: sync.runtime_SemacquireRWMutex(0xc0007c3558?, 0xea?, 0x3fb0800?) /usr/local/go/src/runtime/sema.go:87 +0x25 sync.(*RWMutex).Lock(0xc0007c35a8?) /usr/local/go/src/sync/rwmutex.go:151 +0x6a github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*MemChunk).FreeResource(0xc0008d9130) /github/workspace/weed/mount/page_writer/page_chunk_mem.go:38 +0x2a github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*SealedChunk).FreeReference(0xc00071cdb0, {0xc0006ba1a0, 0x20}) /github/workspace/weed/mount/page_writer/upload_pipeline.go:38 +0xb7 github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).Shutdown(0xc0002ee4d0) /github/workspace/weed/mount/page_writer/upload_pipeline.go:220 +0x185 github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).Destroy(0xc0008cea40?) /github/workspace/weed/mount/dirty_pages_chunked.go:87 +0x17 github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).Destroy(...) /github/workspace/weed/mount/page_writer.go:78 github.com/seaweedfs/seaweedfs/weed/mount.NewSeaweedFileSystem.func3({0xc00069a6c0, 0x30}, 0x6?) /github/workspace/weed/mount/weedfs.go:119 +0x17a github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.NewMetaCache.func1({0xc00069a6c0?, 0xc00069a480?}, 0x4015b40?) /github/workspace/weed/mount/meta_cache/meta_cache.go:37 +0x1c github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents.func1(0xc000661810) /github/workspace/weed/mount/meta_cache/meta_cache_subscribe.go:43 +0x570 * use locked entry everywhere * modifiable remote entry * skip locking after getting lock from fhLockTable
2024-07-16Added tls for http clients (#5766)vadimartynov1-1/+5
* Added global http client * Added Do func for global http client * Changed the code to use the global http client * Fix http client in volume uploader * Fixed pkg name * Fixed http util funcs * Fixed http client for bench_filer_upload * Fixed http client for stress_filer_upload * Fixed http client for filer_server_handlers_proxy * Fixed http client for command_fs_merge_volumes * Fixed http client for command_fs_merge_volumes and command_volume_fsck * Fixed http client for s3api_server * Added init global client for main funcs * Rename global_client to client * Changed: - fixed NewHttpClient; - added CheckIsHttpsClientEnabled func - updated security.toml in scaffold * Reduce the visibility of some functions in the util/http/client pkg * Added the loadSecurityConfig function * Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-07-12ensure metadata follow a specific folderchrislu1-1/+7
fix https://github.com/seaweedfs/seaweedfs/issues/5774
2024-06-06Fix Issue #5649 (#5652)Gaspare Iengo3-6/+29
2024-05-20go fmtchrislu2-2/+2
2024-04-11log possible racing conditionchrislu2-12/+12
fix https://github.com/seaweedfs/seaweedfs/issues/5412
2024-03-27lock instead of RLock, to prevent racing conditionchrislu1-2/+2
https://github.com/seaweedfs/seaweedfs/discussions/5432#discussioncomment-8933608
2024-03-22[mount] fix GetAttr blocks count (#5409)wusong1-1/+1
2023-11-17Waiting for writes to complete with acquired locks may lead to deadlock, ↵Viktor Kuzmin1-2/+5
fixes #4952
2023-11-01skip if path changedchrislu1-1/+4
mitigate https://github.com/seaweedfs/seaweedfs/issues/4968
2023-10-13disconnect from old subscriberschrislu1-3/+2
2023-10-01RetryForever => RetryUntilchrislu1-1/+1
2023-09-23read source uses shared lockchrislu1-1/+1
2023-09-21mount switch to ordered lock requestschrislu7-15/+23
2023-09-18Revert "Revert "Merge branch 'master' into sub""chrislu1-3/+4
This reverts commit 0bb97709d41b1be4c74f01dcc65aac6d5f88bd16.
2023-09-18Revert "Merge branch 'master' into sub"chrislu1-4/+3
This reverts commit 4d414f54a224142f3f4d934f4af3b5dceb6fec6b, reversing changes made to 482742514656e9b5a652acf7406740fbc55db13d.
2023-09-14avoid passe lock by valuewang wusong1-3/+4
2023-08-16re-orderingchrislu1-1/+1
2023-08-16renamingchrislu2-19/+19
2023-08-16add separate cache directory for write bufferschrislu1-1/+4
2023-07-17Fix large file chunks deleted by mistake (#4678)wusong1-2/+8
2023-07-11Fix data loss: add lock for metacache (#4664)wusong1-18/+19
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-07-03Fix buff use (#4628)wusong1-2/+2
2023-04-23[mount]Fix copy file range (#4407)zemul1-6/+2
* fix:mount deadlock * fix --------- Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-04-15update fuse client log which should output the fh of FileHandle. (#4404)Chengyu Liu2-3/+2
2023-04-13Mount concurrent read (#4400)zemul5-12/+15
* fix:mount deadlock * feat: concurrent read * fix * Remove useless code * fix --------- Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-03-21refactoringchrislu1-1/+13
2023-03-15Skip parent directory creation in mount (#4310)Patrick Schmidt3-14/+23
A POSIX filesystem does not implicitly create parent directories when they do not exist. Directories must be explicitly created and permissions be set. This also fixes a bug where asynchronous operations would create a file in the filer before the parent directory was created. If the file was a symlink or another special type of file the directory would inherit that type and become unusable in the mounted FS.
2023-02-15[mount] Fix read file-data in entry.content (#4207)wusong1-1/+1
2023-01-31memchunk save content twice (#4174)wusong1-7/+3
2023-01-31[mount]: delete redundant code (#4173)wusong1-3/+3
2023-01-29[mount] fix metacache update (#4161)wusong1-2/+6
2023-01-20grpc connection to filer add sw-client-id headerchrislu1-1/+1
2023-01-16use one readerCache for the whole filechrislu1-1/+4
2023-01-15use memory chunks only for sequential writeschrislu1-1/+2
2023-01-13allow random writes for mem chunkchrislu1-2/+1
2023-01-12merge chunks during upload (#4130)Chris Lu2-3/+26
* merge chunks during upload * fix test
2023-01-10no need to set chunks during flushchrislu1-2/+0
2023-01-10avoid repeatedly adding manifest chunkschrislu1-1/+2
2023-01-10avoid possible nil entrychrislu1-5/+5
2023-01-03add back previous chunk upload selection algochrislu4-5/+35
2023-01-02more solid weed mount (#4089)Chris Lu21-314/+413
* compare chunks by timestamp * fix slab clearing error * fix test compilation * move oldest chunk to sealed, instead of by fullness * lock on fh.entryViewCache * remove verbose logs * revert slat clearing * less logs * less logs * track write and read by timestamp * remove useless logic * add entry lock on file handle release * use mem chunk only, swap file chunk has problems * comment out code that maybe used later * add debug mode to compare data read and write * more efficient readResolvedChunks with linked list * small optimization * fix test compilation * minor fix on writer * add SeparateGarbageChunks * group chunks into sections * turn off debug mode * fix tests * fix tests * tmp enable swap file chunk * Revert "tmp enable swap file chunk" This reverts commit 985137ec472924e4815f258189f6ca9f2168a0a7. * simple refactoring * simple refactoring * do not re-use swap file chunk. Sealed chunks should not be re-used. * comment out debugging facilities * either mem chunk or swap file chunk is fine now * remove orderedMutex as *semaphore.Weighted not found impactful * optimize size calculation for changing large files * optimize performance to avoid going through the long list of chunks * still problems with swap file chunk * rename * tiny optimization * swap file chunk save only successfully read data * fix * enable both mem and swap file chunk * resolve chunks with range * rename * fix chunk interval list * also change file handle chunk group when adding chunks * pick in-active chunk with time-decayed counter * fix compilation * avoid nil with empty fh.entry * refactoring * rename * rename * refactor visible intervals to *list.List * refactor chunkViews to *list.List * add IntervalList for generic interval list * change visible interval to use IntervalList in generics * cahnge chunkViews to *IntervalList[*ChunkView] * use NewFileChunkSection to create * rename variables * refactor * fix renaming leftover * renaming * renaming * add insert interval * interval list adds lock * incrementally add chunks to readers Fixes: 1. set start and stop offset for the value object 2. clone the value object 3. use pointer instead of copy-by-value when passing to interval.Value 4. use insert interval since adding chunk could be out of order * fix tests compilation * fix tests compilation
2022-12-21lock and unlock chunks for readchrislu1-0/+4
2022-12-21remove duplicated logic of removing from writableChunkschrislu1-3/+1
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-25 13:09:17 +0000