aboutsummaryrefslogtreecommitdiff
path: root/weed/storage/backend
AgeCommit message (Collapse)AuthorFilesLines
2025-07-10[Enhancement] support fix for remote files with command fix (#6961)chalet1-0/+5
2025-06-26update s3 session cache key (#6923)chalet1-1/+2
2025-06-03change version directorychrislu1-4/+3
2025-01-22cloud tier: support for Alibaba Cloud OSS (#6466)ludwigxia2-5/+10
2024-11-21ReadAt may return io.EOF t end of filechrislu1-1/+6
related to https://github.com/seaweedfs/seaweedfs/issues/6219
2024-04-02fix: always close volume file (#4530) (#5459)Henco Appel1-4/+11
If sync fails then close is never called. We should always be calling close on the file. Co-authored-by: Hendrik Appel <happel@europe.altair.com>
2024-03-19fix compilationchrislu1-1/+1
2024-03-18chore(deps): bump github.com/rclone/rclone from 1.65.2 to 1.66.0 (#5387)dependabot[bot]1-1/+1
* chore(deps): bump github.com/rclone/rclone from 1.65.2 to 1.66.0 Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone) from 1.65.2 to 1.66.0. - [Release notes](https://github.com/rclone/rclone/releases) - [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md) - [Commits](https://github.com/rclone/rclone/compare/v1.65.2...v1.66.0) --- updated-dependencies: - dependency-name: github.com/rclone/rclone dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Fix compilation error with rclone update (#5395) Fix compilation error --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Damiano Albani <damiano.albani@gmail.com>
2024-02-14Fix Broken Links (#5287)Justin Dhillon1-2/+2
* https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-setprocessworkingsetsize * https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-getprocessworkingsetsize * remove https://github.com/h5bp/html5-boilerplate/blob/master/src/css/main.css * https://github.com/AShiou/hof
2023-11-23reduce binary size by skipping rclonechrislu3-0/+7
2023-06-24Support templating name of files stored via Rclone backend (#4606)Damiano Albani1-3/+33
2023-06-02set df.File to nil after it is closedchrislu1-3/+19
possibly fix https://github.com/seaweedfs/seaweedfs/issues/4530
2023-04-18Rclone storage backend (#4402)Damiano Albani2-0/+289
* Add Rclone storage backend * Support templating the name of files stored via Rclone * Enable Rclone accounting * Remove redundant type conversion * Provide progress information for Rclone download/upload operations * Log error when Rclone can't instantiate filesystem * Remove filename templating functionality for Rclone storage To (maybe) be later reintroduced as a generic functionality for all storage backends. * Remove S3 specific check * Move Rclone config initialisation to init() method
2022-10-13vacuum metrics and force sync dst files (#3832)Konstantin Lebedev1-0/+3
2022-10-12cloud tier: add retry when copying data filechrislu1-1/+5
fix https://github.com/seaweedfs/seaweedfs/issues/3828
2022-10-10revert disabling FSync for non Mac (#3814)Konstantin Lebedev1-2/+10
2022-09-16refactor(storage): readability improvements (#3703)Ryan Russell1-1/+1
Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14go fmtchrislu1-20/+25
2022-09-04cloud tier: s3 consume all read response bodychrislu1-7/+8
fix https://github.com/seaweedfs/seaweedfs/issues/3584
2022-08-15Revert "Refactor for Sync method (#3426)"chrislu4-69/+7
This reverts commit 670cb759f82815461bc854569542582da51a6199. with the pr weed/storage () - (master) > go test -count=1 ./... ok github.com/seaweedfs/seaweedfs/weed/storage 18.486s ? github.com/seaweedfs/seaweedfs/weed/storage/backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map 0.025s ? github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding 0.864s ? github.com/seaweedfs/seaweedfs/weed/storage/idx [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/needle 0.110s ok github.com/seaweedfs/seaweedfs/weed/storage/needle_map 24.414s ok github.com/seaweedfs/seaweedfs/weed/storage/super_block 0.203s ? github.com/seaweedfs/seaweedfs/weed/storage/types [no test files] ? github.com/seaweedfs/seaweedfs/weed/storage/volume_info [no test files] weed/storage () - (master) > weed/storage () - (master) > without the pr weed/storage () - (master) > weed/storage () - (master) > go test -count=1 ./... ok github.com/seaweedfs/seaweedfs/weed/storage 1.617s ? github.com/seaweedfs/seaweedfs/weed/storage/backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map 0.026s ? github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding 0.906s ? github.com/seaweedfs/seaweedfs/weed/storage/idx [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/needle 0.202s ok github.com/seaweedfs/seaweedfs/weed/storage/needle_map 24.533s ok github.com/seaweedfs/seaweedfs/weed/storage/super_block 0.280s ? github.com/seaweedfs/seaweedfs/weed/storage/types [no test files] ? github.com/seaweedfs/seaweedfs/weed/storage/volume_info [no test files]
2022-08-10Refactor for Sync method (#3426)Rain Li4-7/+69
2022-07-29move to https://github.com/seaweedfs/seaweedfschrislu9-17/+17
2022-06-30stuck with file.Sync()chrislu1-1/+2
starting to get weed/storage () - (master) > go test ./... I0630 12:19:54 65819 needle_map_metric_test.go:26] FileCount expected 10000 actual 10000 I0630 12:19:54 65819 needle_map_metric_test.go:27] DeletedSize expected 1648 actual 1648 I0630 12:19:54 65819 needle_map_metric_test.go:28] ContentSize expected 10000 actual 10000 I0630 12:19:54 65819 needle_map_metric_test.go:29] DeletedCount expected 1648 actual 1959 I0630 12:19:54 65819 needle_map_metric_test.go:30] MaxFileKey expected 10000 actual 10000 I0630 12:19:54 65819 volume_loading.go:136] loading index /var/folders/jv/0rlhdck17jzgt7b3hcnq63mc0000gp/T/TestCompaction909350722/001/1.idx to memory I0630 12:20:36 65819 volume_vacuum.go:98] Committing volume 1 vacuuming... panic: test timed out after 10m0s goroutine 61 [running]: testing.(*M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2029 +0x8e created by time.goFunc /usr/local/go/src/time/sleep.go:176 +0x32 goroutine 1 [chan receive, 10 minutes]: testing.(*T).Run(0xc0000cd520, {0x1c09bde?, 0x5ab798ea5c102?}, 0x1c67bd8) /usr/local/go/src/testing/testing.go:1487 +0x37a testing.runTests.func1(0xc0000cd520?) /usr/local/go/src/testing/testing.go:1839 +0x6e testing.tRunner(0xc0000cd520, 0xc00039fcd8) /usr/local/go/src/testing/testing.go:1439 +0x102 testing.runTests(0xc0000dadc0?, {0x25b4460, 0x6, 0x6}, {0x2855108?, 0x40?, 0x25c4b80?}) /usr/local/go/src/testing/testing.go:1837 +0x457 testing.(*M).Run(0xc0000dadc0) /usr/local/go/src/testing/testing.go:1719 +0x5d9 main.main() _testmain.go:57 +0x1aa goroutine 21 [chan receive]: github.com/chrislusf/seaweedfs/weed/glog.(*loggingT).flushDaemon(0x0?) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:883 +0x6a created by github.com/chrislusf/seaweedfs/weed/glog.init.0 /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:410 +0x1bf goroutine 55 [syscall]: syscall.syscall(0x44ac0f96?, 0xc00007ad80?, 0xc00098b480?, 0x10588df?) /usr/local/go/src/runtime/sys_darwin.go:22 +0x4e syscall.fcntl(0x100000001?, 0xc00098b4b0?, 0x1069a7c?) /usr/local/go/src/syscall/zsyscall_darwin_amd64.go:319 +0x30 internal/poll.(*FD).Fsync.func1(...) /usr/local/go/src/internal/poll/fd_fsync_darwin.go:18 internal/poll.ignoringEINTR(...) /usr/local/go/src/internal/poll/fd_posix.go:74 internal/poll.(*FD).Fsync(0xc00038f1e0?) /usr/local/go/src/internal/poll/fd_fsync_darwin.go:17 +0xfc os.(*File).Sync(0xc00012a030) /usr/local/go/src/os/file_posix.go:168 +0x4e github.com/chrislusf/seaweedfs/weed/storage/backend.(*DiskFile).Sync(...) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/backend/disk_file.go:84 github.com/chrislusf/seaweedfs/weed/storage.(*Volume).makeupDiff(0xc0000bb440, {0xc000633a40, 0x52}, {0xc000633aa0, 0x52}, {0xc000633b00, 0x52}, {0xc000633b60, 0x52}) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:295 +0x12fa github.com/chrislusf/seaweedfs/weed/storage.(*Volume).CommitCompact(0xc0000bb440) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:119 +0x3e8 github.com/chrislusf/seaweedfs/weed/storage.TestCompaction(0xc00025a000) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum_test.go:89 +0x305 testing.tRunner(0xc00025a000, 0x1c67bd8) /usr/local/go/src/testing/testing.go:1439 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1486 +0x35f goroutine 56 [chan receive, 10 minutes]: github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker.func1() /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:244 +0x41a created by github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:234 +0x56 FAIL github.com/chrislusf/seaweedfs/weed/storage 600.194s ? github.com/chrislusf/seaweedfs/weed/storage/backend [no test files] ok github.com/chrislusf/seaweedfs/weed/storage/backend/memory_map (cached) ? github.com/chrislusf/seaweedfs/weed/storage/backend/s3_backend [no test files] ok github.com/chrislusf/seaweedfs/weed/storage/erasure_coding (cached) ? github.com/chrislusf/seaweedfs/weed/storage/idx [no test files] ok github.com/chrislusf/seaweedfs/weed/storage/needle (cached) ok github.com/chrislusf/seaweedfs/weed/storage/needle_map (cached) ok github.com/chrislusf/seaweedfs/weed/storage/super_block (cached) ? github.com/chrislusf/seaweedfs/weed/storage/types [no test files] ? github.com/chrislusf/seaweedfs/weed/storage/volume_info [no test files] FAIL
2022-05-11fix atomic add int64chrislu1-1/+1
fix https://github.com/chrislusf/seaweedfs/issues/3038#issuecomment-1123269831
2022-05-02use storage_class for backend configurationchrislu1-1/+1
fix https://github.com/chrislusf/seaweedfs/issues/3008
2022-04-30volume.tier.upload progress starts negative #2992chrislu1-13/+21
fix https://github.com/chrislusf/seaweedfs/issues/2992
2022-04-30s3 backend support customizing storage classchrislu2-3/+9
2022-03-10checks disk file existKonstantin Lebedev1-1/+4
2021-11-02go fmtChris Lu1-4/+4
2021-10-29cloud tier: remove tagging since not all s3 vendors support thisChris Lu3-18/+4
2021-10-29s3 header add user agentChris Lu1-0/+5
2021-10-01add source name to error messageChris Lu1-1/+1
2021-09-01go fmtChris Lu6-3/+8
2021-08-25cloud drive: add support for WasabiChris Lu1-0/+1
* disable md5, sha256 checking to avoid reading one chunk twice * single threaded upload to avoid chunk swapping (to be enhanced later)
2021-08-23cloud drive: s3 configurable force path styleChris Lu1-0/+1
2021-08-23do not force path style for better compatibilityChris Lu1-1/+0
2021-03-05volume server: support tcp direct put/get/deleteChris Lu1-1/+1
2021-03-01skip already loaded backendsChris Lu1-0/+6
2021-02-28cloud tier to non-AWS s3 gatewaysChris Lu1-2/+3
2021-02-20disk file avoid file.Stat()Chris Lu1-1/+10
2021-02-20Revert "Revert "avoid file.Stat()""Chris Lu1-7/+26
This reverts commit 7ef4c24f186f97d9a531849ed626ac1732437b77.
2021-02-20Revert "avoid file.Stat()"Chris Lu1-26/+7
This reverts commit 98c93ca46587795fa03a0c52bea5be31a886be87.
2021-02-20avoid file.Stat()Chris Lu1-7/+26
avoid one Syscall, but did not help on performance though
2021-02-19Revert "volume: avoid file.stat(), file.seek() if possible during writes"Chris Lu1-21/+2
This reverts commit c78409a5983145784d128c86541c6715dad3a937.
2021-02-18volume: avoid file.stat(), file.seek() if possible during writesChris Lu1-2/+21
2021-01-12avoid concurrent map updates to viperChris Lu1-2/+2
2020-10-04adjust log levelChris Lu1-1/+1
2020-08-16adjust logsChris Lu1-1/+1
2020-04-12refactoringChris Lu1-1/+1
2020-04-11refactoringChris Lu4-3/+75
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-20 23:58:05 +0000