aboutsummaryrefslogtreecommitdiff
path: root/weed/storage/volume.go
AgeCommit message (Collapse)AuthorFilesLines
2025-07-30Admin: misc improvements on admin server and workers. EC now works. (#7055)Chris Lu1-0/+2
* initial design * added simulation as tests * reorganized the codebase to move the simulation framework and tests into their own dedicated package * integration test. ec worker task * remove "enhanced" reference * start master, volume servers, filer Current Status āœ… Master: Healthy and running (port 9333) āœ… Filer: Healthy and running (port 8888) āœ… Volume Servers: All 6 servers running (ports 8080-8085) šŸ”„ Admin/Workers: Will start when dependencies are ready * generate write load * tasks are assigned * admin start wtih grpc port. worker has its own working directory * Update .gitignore * working worker and admin. Task detection is not working yet. * compiles, detection uses volumeSizeLimitMB from master * compiles * worker retries connecting to admin * build and restart * rendering pending tasks * skip task ID column * sticky worker id * test canScheduleTaskNow * worker reconnect to admin * clean up logs * worker register itself first * worker can run ec work and report status but: 1. one volume should not be repeatedly worked on. 2. ec shards needs to be distributed and source data should be deleted. * move ec task logic * listing ec shards * local copy, ec. Need to distribute. * ec is mostly working now * distribution of ec shards needs improvement * need configuration to enable ec * show ec volumes * interval field UI component * rename * integration test with vauuming * garbage percentage threshold * fix warning * display ec shard sizes * fix ec volumes list * Update ui.go * show default values * ensure correct default value * MaintenanceConfig use ConfigField * use schema defined defaults * config * reduce duplication * refactor to use BaseUIProvider * each task register its schema * checkECEncodingCandidate use ecDetector * use vacuumDetector * use volumeSizeLimitMB * remove remove * remove unused * refactor * use new framework * remove v2 reference * refactor * left menu can scroll now * The maintenance manager was not being initialized when no data directory was configured for persistent storage. * saving config * Update task_config_schema_templ.go * enable/disable tasks * protobuf encoded task configurations * fix system settings * use ui component * remove logs * interface{} Reduction * reduce interface{} * reduce interface{} * avoid from/to map * reduce interface{} * refactor * keep it DRY * added logging * debug messages * debug level * debug * show the log caller line * use configured task policy * log level * handle admin heartbeat response * Update worker.go * fix EC rack and dc count * Report task status to admin server * fix task logging, simplify interface checking, use erasure_coding constants * factor in empty volume server during task planning * volume.list adds disk id * track disk id also * fix locking scheduled and manual scanning * add active topology * simplify task detector * ec task completed, but shards are not showing up * implement ec in ec_typed.go * adjust log level * dedup * implementing ec copying shards and only ecx files * use disk id when distributing ec shards šŸŽÆ Planning: ActiveTopology creates DestinationPlan with specific TargetDisk šŸ“¦ Task Creation: maintenance_integration.go creates ECDestination with DiskId šŸš€ Task Execution: EC task passes DiskId in VolumeEcShardsCopyRequest šŸ’¾ Volume Server: Receives disk_id and stores shards on specific disk (vs.store.Locations[req.DiskId]) šŸ“‚ File System: EC shards and metadata land in the exact disk directory planned * Delete original volume from all locations * clean up existing shard locations * local encoding and distributing * Update docker/admin_integration/EC-TESTING-README.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * check volume id range * simplify * fix tests * fix types * clean up logs and tests --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-06-19pass volume version when creating a volumechrislu1-2/+2
2024-09-05persist readonly state to volume info (#5977)Bruce1-2/+10
2024-04-17rename VolumeServerVolumeGauge (#5504)steve.wei1-1/+1
2023-10-03Revert "Revert "Revert "Add disk type to prometheus metrics" (#4777)""chrislu1-1/+1
This reverts commit 567d78892860f87c207beb56a3323b4b9e90dfc9.
2023-10-02Revert "Revert "Add disk type to prometheus metrics" (#4777)"chrislu1-1/+1
This reverts commit 9215ba24be0aec0c5804927cc73613560e57cbc0.
2023-07-04fix copying level db files during commichrislu1-1/+1
fix https://github.com/seaweedfs/seaweedfs/issues/4635
2023-06-18remove duplicate `fileCount` query (#4588)Bai Jie1-1/+0
2023-06-18doIsEmpty() return error if v.DataBackend is nil (#4587)Bai Jie1-2/+6
2023-06-14avoid data race read volume.IsEmpty (#4574)꟏ꝰ1-4/+23
* avoid data race read volume.IsEmpty - avoid phantom read isEmpty for onlyEmpty - use `v.DataBackend.GetStat()` in v.dataFileAccessLock scope * add Destroy(onlyEmpty: true) test * add Destroy(onlyEmpty: false) test * remove unused `IsEmpty()` * change literal `8` to `SuperBlockSize`
2023-06-14fix get file stats for IsEmpty (#4576)Konstantin Lebedev1-2/+2
2023-06-12fix from comment delete volume is empty (#4573)Konstantin Lebedev1-1/+2
* fix from coments https://github.com/seaweedfs/seaweedfs/pull/4561 * fix tests --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-12Delete volume is empty (#4561)Konstantin Lebedev1-0/+3
* use onlyEmpty for deleteVolume https://github.com/seaweedfs/seaweedfs/issues/4559 * fix IsEmpty * fix test --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2022-11-14Lazy loading (#3958)Guo Lei1-1/+3
* types packages is imported more than onece * lazy-loading * fix bugs * fix bugs * fix unit tests * fix test error * rename function * unload ldb after initial startup * Don't load ldb when starting volume server if ldbtimeout is set. * remove uncessary unloadldb * Update weed/command/server.go Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> * Update weed/command/volume.go Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: guol-fnst <goul-fnst@fujitsu.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-10-13vacuum metrics and force sync dst files (#3832)Konstantin Lebedev1-17/+1
2022-09-04adjust log levelchrislu1-1/+1
2022-08-30avoid race conditions access to SuperBlock.Version (#3539)Konstantin Lebedev1-0/+3
* avoid race conditions access to SuperBlock.Version https://github.com/seaweedfs/seaweedfs/issues/3515 * superBlockAccessLock replace to sync.Mutex
2022-08-23optimiz commitig compact (#3388)Guo Lei1-0/+1
* optimiz vacuuming volume * fix bugx * rename parameters * fix conflict * change copyDataBasedOnIndexFile to an instance method * close needlemap * optimiz commiting Vacuum volume for leveldb index * fix bugs * fix leveldb loading bugs * refactor * fix leveldb loading bug * add leveldb recovery * add test case for levelDB * modify test case to cover all the new branches * use one tmpNm instead of two instances * refactor * refactor * move setWatermark to the end * add test for watermark and updating leveldb * fix error logic * refactor, add test * check nil before close needlemapeer add test case fix metric bug * add tests, fix bugs * adjust log level remove wrong test case refactor * avoid duplicate updating metric for leveldb index
2022-07-29move to https://github.com/seaweedfs/seaweedfschrislu1-9/+9
2022-06-10Revert "Merge pull request #3159 from shichanglin5/_duplicateUUID"chrislu1-0/+1
This reverts commit 37da68931941b1f084b7f4312722e4241e9ab0b0, reversing changes made to 00d53c34c44dd6442ec276f184344faa20c16db0.
2022-06-09perf: Optimized volume handling duplicateUUID logic to avoid quitting when ā†µshichanglin51-1/+0
volume is actualy normal Under normal circumstances, there will be no problems, but when the master is debugged in the local environment, the volume client cannot communicate with the master normally, so the sendHeartBeat logic is restarted, and a new connection is created to report the heartbeat. If the master has not cleared the uuid of the volume at this time, then The master will respond to volume duplicateUUIDS, and the volume service will exit, but in fact the uuid of the volume is not duplicated
2022-04-26volume close should wait for committing compactionchrislu1-4/+6
2022-04-26Merge pull request #2974 from kmlebedev/wait_volume_closed_compressionChris Lu1-0/+4
waite volume being closed during compression idx
2022-04-26use two flags: v.isCompacting and v.isCommitCompactingchrislu1-1/+2
2022-04-26volume: sync to disk before copying volume fileschrislu1-0/+15
address https://github.com/chrislusf/seaweedfs/issues/2976
2022-04-26waite volume being closed during compression idxKonstantin Lebedev1-0/+4
2022-04-18avoid invalid memory address or nil pointer dereferenceKonstantin Lebedev1-1/+1
2022-02-16sync data first before stoppingchrislu1-0/+15
2021-06-02avoid data loss after restarting a container with a volum serverKonstantin Lebedev1-0/+6
2021-03-13prevent nil volume nmChris Lu1-3/+13
2021-02-16this can compile now!!!Chris Lu1-1/+1
2021-02-09Merge branch 'master' into support_ssd_volumeChris Lu1-2/+2
2021-02-07renaming NeedleMapType to NeedleMapKindbingoohuang1-2/+2
2020-12-22go fmtChris Lu1-1/+1
2020-12-13allocate volume by disk typeChris Lu1-1/+5
2020-12-13rename from volumeType to diskTypeChris Lu1-1/+1
2020-12-13volume server set volume type and heartbeat to the masterChris Lu1-1/+2
2020-12-11adjust loggingChris Lu1-2/+2
2020-11-28volume: detect and drop volumes with disk IO errorChris Lu1-2/+4
from Jethro in slack: is it possible to make the assign request a bit smarter? Currently Iā€™m in the state that a disk failed but all assign request are being send to this volume. It would be cool if the master sees this and stopped using this volume. e=HTTP(http://x:8089/913,045a782b63176edf) not 200 but 500 Internal Server Error Body={"size":740167,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"ee4381e202212ff3aee647704c036689"} e=HTTP(http://x:8089/913,045a782c90240077) not 200 but 500 Internal Server Error Body={"size":792779,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"c43463ccc11eb6eb2fc306f407a6a953"} e=HTTP(http://x:8089/913,045a782e6b7901ea) not 200 but 500 Internal Server Error Body={"size":3962392,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"04c91198e9b276c81f11dbf189af5d28"}
2020-11-27volume: add "-dir.idx" option for separate index storageChris Lu1-4/+18
fix https://github.com/chrislusf/seaweedfs/issues/1265
2020-10-24reduce locksChris Lu1-11/+27
2020-08-19Added VolumeMarkWritable and VolumeStatus grpc methodsJames Hartig1-0/+5
This is necessary for copy to mark as read-only and then restore the original state afterwards.
2020-07-03refactoringChris Lu1-10/+5
2020-06-05Set volumes ReadOnly if low free disk spaceEvgenii Kozlov1-1/+7
2020-05-06Revert "Revert "Merge pull request #1299 from song-zhang/master""Chris Lu1-1/+6
This reverts commit afb6a1dbb46a2a8793c8faa0450a2609f8e01af3.
2020-05-04Revert "Merge pull request #1299 from song-zhang/master"Chris Lu1-6/+1
This reverts commit 9016fa19ba4507bdf9ac0addc6d79f147c00302a, reversing changes made to 47234760f40e4d2cea87b4a83d2178b8181598f5.
2020-05-04use async write to persistent file to disk - part1zhangsong1-1/+6
2020-03-17volume: compaction can cause readonly volumesChris Lu1-1/+5
address https://github.com/chrislusf/seaweedfs/issues/1233
2020-03-09filer: support TTL for all filer storesChris Lu1-2/+2
2020-03-01volume: fix readonly status reportingChris Lu1-1/+1
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-21 02:48:32 +0000