aboutsummaryrefslogtreecommitdiff
path: root/weed/storage/disk_location.go
AgeCommit message (Collapse)AuthorFilesLines
2025-10-26fix commentchrislu1-1/+1
2025-10-26Volume Server: handle incomplete ec encoding (#7384)Chris Lu1-10/+19
* handle incomplete ec encoding * unit tests * simplify, and better logs * Update disk_location_ec.go When loadEcShards() fails partway through, some EC shards may already be loaded into the l.ecVolumes map in memory. The previous code only cleaned up filesystem files but left orphaned in-memory state, which could cause memory leaks and inconsistent state. * address comments * Performance: Avoid Double os.Stat() Call * Platform Compatibility: Use filepath.Join * in memory cleanup * Update disk_location_ec.go * refactor * Added Shard Size Validation * check ec shard sizes * validate shard size * calculate expected shard size * refactoring * minor * fix shard directory * 10GB sparse files can be slow or fail on non-sparse FS. Use 10MB to hit SmallBlockSize math (1MB shards) deterministically. * grouping logic should be updated to use both collection and volumeId to ensure correctness * unexpected error * handle exceptions in tests; use constants * The check for orphaned shards should be performed for the previous volume before resetting sameVolumeShards for the new volume. * address comments * Eliminated Redundant Parsing in checkOrphanedShards * minor * Avoid misclassifying local EC as distributed when .dat stat errors occur; also standardize unload-before-remove. * fmt * refactor * refactor * adjust to warning
2025-10-20create new volumes on less occupied disk locations (#7349)Chris Lu1-0/+13
* create new volumes on less occupied disk locations * add unit tests * address comments * fixes
2025-07-30Admin: misc improvements on admin server and workers. EC now works. (#7055)Chris Lu1-10/+15
* initial design * added simulation as tests * reorganized the codebase to move the simulation framework and tests into their own dedicated package * integration test. ec worker task * remove "enhanced" reference * start master, volume servers, filer Current Status ✅ Master: Healthy and running (port 9333) ✅ Filer: Healthy and running (port 8888) ✅ Volume Servers: All 6 servers running (ports 8080-8085) 🔄 Admin/Workers: Will start when dependencies are ready * generate write load * tasks are assigned * admin start wtih grpc port. worker has its own working directory * Update .gitignore * working worker and admin. Task detection is not working yet. * compiles, detection uses volumeSizeLimitMB from master * compiles * worker retries connecting to admin * build and restart * rendering pending tasks * skip task ID column * sticky worker id * test canScheduleTaskNow * worker reconnect to admin * clean up logs * worker register itself first * worker can run ec work and report status but: 1. one volume should not be repeatedly worked on. 2. ec shards needs to be distributed and source data should be deleted. * move ec task logic * listing ec shards * local copy, ec. Need to distribute. * ec is mostly working now * distribution of ec shards needs improvement * need configuration to enable ec * show ec volumes * interval field UI component * rename * integration test with vauuming * garbage percentage threshold * fix warning * display ec shard sizes * fix ec volumes list * Update ui.go * show default values * ensure correct default value * MaintenanceConfig use ConfigField * use schema defined defaults * config * reduce duplication * refactor to use BaseUIProvider * each task register its schema * checkECEncodingCandidate use ecDetector * use vacuumDetector * use volumeSizeLimitMB * remove remove * remove unused * refactor * use new framework * remove v2 reference * refactor * left menu can scroll now * The maintenance manager was not being initialized when no data directory was configured for persistent storage. * saving config * Update task_config_schema_templ.go * enable/disable tasks * protobuf encoded task configurations * fix system settings * use ui component * remove logs * interface{} Reduction * reduce interface{} * reduce interface{} * avoid from/to map * reduce interface{} * refactor * keep it DRY * added logging * debug messages * debug level * debug * show the log caller line * use configured task policy * log level * handle admin heartbeat response * Update worker.go * fix EC rack and dc count * Report task status to admin server * fix task logging, simplify interface checking, use erasure_coding constants * factor in empty volume server during task planning * volume.list adds disk id * track disk id also * fix locking scheduled and manual scanning * add active topology * simplify task detector * ec task completed, but shards are not showing up * implement ec in ec_typed.go * adjust log level * dedup * implementing ec copying shards and only ecx files * use disk id when distributing ec shards 🎯 Planning: ActiveTopology creates DestinationPlan with specific TargetDisk 📦 Task Creation: maintenance_integration.go creates ECDestination with DiskId 🚀 Task Execution: EC task passes DiskId in VolumeEcShardsCopyRequest 💾 Volume Server: Receives disk_id and stores shards on specific disk (vs.store.Locations[req.DiskId]) 📂 File System: EC shards and metadata land in the exact disk directory planned * Delete original volume from all locations * clean up existing shard locations * local encoding and distributing * Update docker/admin_integration/EC-TESTING-README.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * check volume id range * simplify * fix tests * fix types * clean up logs and tests --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-06-19pass volume version when creating a volumechrislu1-1/+1
2025-03-31chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 (#6679)dependabot[bot]1-1/+1
* chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 Bumps [gocloud.dev](https://github.com/google/go-cloud) from 0.40.0 to 0.41.0. - [Release notes](https://github.com/google/go-cloud/releases) - [Commits](https://github.com/google/go-cloud/compare/v0.40.0...v0.41.0) --- updated-dependencies: - dependency-name: gocloud.dev dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * fix error * fix printing errors * Update go.mod --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: chrislu <chris.lu@gmail.com>
2025-02-24added re-generating and writing the Volume UUID if it is empty (#6568)Aleksey Kosov1-8/+15
2024-07-10Detect underflow when calculating unused space (#5758)Dan1-2/+6
* Detect underflow when calculating unused space * Detect underflow when calculating unused space
2024-06-28fixed fail to initialize existing ec volume when volume server has separate ↵Taehyung Lim1-1/+1
index dictory (#5723)
2023-10-03Revert "fix compilation"chrislu1-1/+0
This reverts commit 451ec6504d07812b074555a1fbb2a831bfa8e5e3.
2023-10-03fix compilationchrislu1-0/+1
2023-08-10Save disk space metrics immediately (#4740)Nikita Mochalov1-0/+1
2023-08-09Fix resource leaks (#4737)Nikita Mochalov1-20/+28
* Fix division by zero * Fix file handle leak * Fix file handle leak * Fix file handle leak * Fix goroutine leak
2023-06-14avoid data race read volume.IsEmpty (#4574)柏杰1-5/+5
* avoid data race read volume.IsEmpty - avoid phantom read isEmpty for onlyEmpty - use `v.DataBackend.GetStat()` in v.dataFileAccessLock scope * add Destroy(onlyEmpty: true) test * add Destroy(onlyEmpty: false) test * remove unused `IsEmpty()` * change literal `8` to `SuperBlockSize`
2022-11-14Lazy loading (#3958)Guo Lei1-7/+7
* types packages is imported more than onece * lazy-loading * fix bugs * fix bugs * fix unit tests * fix test error * rename function * unload ldb after initial startup * Don't load ldb when starting volume server if ldbtimeout is set. * remove uncessary unloadldb * Update weed/command/server.go Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> * Update weed/command/volume.go Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: guol-fnst <goul-fnst@fujitsu.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-10-13vacuum metrics and force sync dst files (#3832)Konstantin Lebedev1-1/+1
2022-08-26avoid race conditions for diskLocation.MaxVolumeCount (#3526)Konstantin Lebedev1-3/+3
2022-07-29move to https://github.com/seaweedfs/seaweedfschrislu1-6/+6
2022-07-14use 10 or numCPU workers if env is not foundguol-fnst1-0/+4
2022-07-14optimiz concurrencyguol-fnst1-2/+9
user can customize number of workers via env "GOMAXPROCS"
2022-07-12speeding up loading volumesguol-fnst1-1/+6
2022-06-27fix 3238: handle errors for GenerateDirUuid methodgarenchan1-3/+4
2022-05-17fix naming conventionguol-fnst1-15/+15
notify volume server of duplicate directoris improve searching efficiency
2022-05-17rename UUID fileguol-fnst1-1/+4
fix typo move locationUUID into DiskLocation
2022-05-16avoid duplicated volume directoryguol-fnst1-0/+24
2022-04-26use two flags: v.isCompacting and v.isCommitCompactingchrislu1-1/+1
2022-02-16sync data first before stoppingchrislu1-0/+10
2021-12-16remove debug messageschrislu1-3/+3
fix https://github.com/chrislusf/seaweedfs/issues/2514
2021-12-05volume: load volume can optionally be skipped, if ec volume existschrislu1-6/+11
fix https://github.com/chrislusf/seaweedfs/issues/2489
2021-10-14refactor: move from io/ioutil to io and os packageEng Zer Jun1-16/+15
The io/ioutil package has been deprecated as of Go 1.16, see https://golang.org/doc/go1.16#ioutil. This commit replaces the existing io/ioutil functions with their new definitions in io and os packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-05skip ec volumes when loading normal volumesChris Lu1-0/+5
2021-09-11add volume not found error type, to reduce error logChris Lu1-2/+4
2021-04-27minFreeSpace refactoredbingoohuang1-8/+6
2021-04-26minFreeSpace argument allows size like 10GiBbingoohuang1-9/+15
2021-04-15adjust textChris Lu1-1/+1
2021-02-16this can compile now!!!Chris Lu1-2/+3
2021-02-13support customizable disk typeChris Lu1-7/+0
2021-02-09Merge branch 'master' into support_ssd_volumeChris Lu1-4/+4
2021-02-07renaming NeedleMapType to NeedleMapKindbingoohuang1-4/+4
2020-12-17Merge branch 'master' into support_ssd_volumeChris Lu1-4/+23
2020-12-17volume: avoid reprocessing the same volumeChris Lu1-4/+23
fix https://github.com/chrislusf/seaweedfs/issues/1682
2020-12-17Revert "Merge pull request #1683 from qieqieplus/master"Chris Lu1-7/+5
This reverts commit 8cb67952db822ef232dd2701ce78a5486d54b4ea, reversing changes made to 200e56215a4fdc4ec424f05fb975cdc876d7a2b4.
2020-12-17Revert "Merge pull request #1683 from qieqieplus/master"Chris Lu1-6/+4
This reverts commit 8cb67952db822ef232dd2701ce78a5486d54b4ea, reversing changes made to 200e56215a4fdc4ec424f05fb975cdc876d7a2b4.
2020-12-16go fmtChris Lu1-4/+4
2020-12-16Merge branch 'master' into support_ssd_volumeChris Lu1-5/+7
2020-12-16smaller critical sectionchenqieqie1-5/+5
2020-12-16fix race condition when loading volumes concurrentlychenqieqie1-5/+7
2020-12-14adjust volume server UIChris Lu1-0/+7
2020-12-13rename from volumeType to diskTypeChris Lu1-3/+3
2020-12-13volume server set volume type and heartbeat to the masterChris Lu1-1/+3
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-20 22:54:01 +0000