aboutsummaryrefslogtreecommitdiff
path: root/weed/shell/command_ec_common.go
AgeCommit message (Collapse)AuthorFilesLines
2025-07-16convert error fromating to %w everywhere (#6995)Chris Lu1-1/+1
2025-05-12Move `shell.ErrorWaitGroup` into a common file, to cleanly reuse across ↵Lisandro Pin1-56/+0
`weed shell` commands. (#6780) Move `shell.ErrorWaitGroup` into a dedicated common file, to cleanly reuse across `weed shell` commands.
2025-05-09Improve safety for weed shell's `ec.encode`. (#6773)Lisandro Pin1-0/+8
Improve safety for weed shells `ec.encode`. The current process for `ec.encode` is: 1. EC shards for a volume are generated and added to a single server 2. The original volume is deleted 3. EC shards get re-balanced across the entire topology It is then possible to lose data between #2 and #3, if the underlying volume storage/server/rack/DC happens to fail, for whatever reason. As a fix, this MR reworks `ec.encode` so: * Newly created EC shards are spread across all locations for the source volume. * Source volumes are deleted only after EC shards are converted and balanced.
2025-02-28`ec.encode`: Fix resolution of target collections. (#6585)Lisandro Pin1-2/+2
* Don't ignore empty (`""`) collection names when computing collections for a given volume ID. * `ec.encode`: Fix resolution of target collections. When no `volumeId` parameter is provided, compute volumes based on the provided collection name, even if it's empty (`""`). This restores behavior to before recent EC rebalancing rework. See also https://github.com/seaweedfs/seaweedfs/blob/ec30a504bae6cad75f859964e14c60d39cc43709/weed/shell/command_ec_encode.go#L99 .
2025-02-28Fix calculation of node's free EC shard slots. (#6584)Lisandro Pin1-1/+7
2025-02-07Nit: remove missing newlines on `weed shell` commands output. (#6524)Lisandro Pin1-1/+1
Nit: remove missing newlines on `weed` commands output.
2025-02-06Remove warning on EC balancing if no replica placement settings are found. ↵Lisandro Pin1-3/+0
(#6516) Effectively undoes c9399a68; with ff8bd862, a replica placement type `000` will no longer break shards re-balancing. Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2025-02-04Nit: fix missing newline on EC balancing warnings regarding replica settings ↵Lisandro Pin1-1/+1
(#6509) Nit: fix missing newline on EC balancing warnings regarding replica settings. See 79136812.
2025-01-30Improve EC shards balancing logic regarding replica placement settings. (#6491)Lisandro Pin1-4/+4
The replica placement type specifies numebr of _replicas_ on the same/different rack; that means we can have one EC shard copy on each, even if the replica setting is zero. This PR reworks replica placement parsing for EC rebalancing, so we check allow (replica placement + 1) when selecting racks and nodes to balance EC shards into.
2025-01-29`ec.balance`: Allow EC balancing without collections. (#6488)Lisandro Pin1-4/+4
2025-01-29`ec.encode`: Display a warning on EC balancing if no replica placement ↵Lisandro Pin1-10/+18
settings are found. (#6487)
2024-12-19"golang.org/x/exp/slices" => "slices" and go fmtchrislu1-1/+1
2024-12-18Allow configuring the maximum number of concurrent tasks for EC ↵Lisandro Pin1-45/+48
parallelization. (#6376) Follow-up to b0210df0.
2024-12-17Rework `shell.EcBalance()`'s waitgroup code into a standalone type. (#6373)Lisandro Pin1-54/+61
Rework `shell.EcBalance()`'s waitgroup with errors code into a standalone type. We'll re-use this for other EC jobs - for example, volume creation. Also fixes potential concurrency issues when collecting error results.
2024-12-15Parallelize EC shards balancing within racks (#6354)Lisandro Pin1-5/+5
Parallelize EC shards balancing within racks.
2024-12-13Parallelize EC shards balancing across racks. (#6352)Lisandro Pin1-5/+6
2024-12-13Parallelize EC balancing for racks. (#6351)Lisandro Pin1-15/+15
2024-12-12Begin implementing EC balancing parallelization support. (#6342)Lisandro Pin1-7/+54
* Begin implementing EC balancing parallelization support. Impacts both `ec.encode` and `ec.balance`, * Nit: improve type naming. * Make the goroutine workgroup handler for `EcBalance()` a bit smarter/error-proof. * Nit: unify naming for `ecBalancer` wait group methods with the rest of the module. * Fix concurrency bug. * Fix whitespace after Gitlab automerge. * Delete stray TODO.
2024-12-12Limit EC re-balancing for `ec.encode` to relevant collections when a volume ↵Lisandro Pin1-0/+41
ID argument is provided. (#6347) Limit EC re-balancing for `ec.encode` to relevant collections when a volume ID is provided.
2024-12-10Unify the re-balancing logic for `ec.encode` with `ec.balance`. (#6339)Lisandro Pin1-18/+108
Among others, this enables recent changes related to topology aware re-balancing at EC encoding time.
2024-12-06Remove average constraints when selecting nodes/racks to balance EC shards ↵Lisandro Pin1-37/+8
into. (#6325)
2024-12-05Share common parameters for EC re-balancing functions under a single struct. ↵Lisandro Pin1-55/+63
(#6319) TODO cleanup for https://github.com/seaweedfs/seaweedfs/discussions/6179.
2024-12-04Account for replication placement settings when balancing EC shards within ↵Lisandro Pin1-10/+8
the same rack. (#6317) * Account for replication placement settings when balancing EC shards within racks. * Update help contents for `ec.balance`. * Add a few more representative test cases for `pickEcNodeToBalanceShardsInto()`.
2024-12-04Account for replication placement settings when balancing EC shards across ↵Lisandro Pin1-7/+9
racks. (#6316)
2024-12-02Resolve replica placement for EC volumes from master server defaults. (#6303)Lisandro Pin1-21/+46
2024-11-28Display details upon failures to re-balance EC shards racks. (#6299)Lisandro Pin1-11/+12
2024-11-27Improve EC shards rebalancing logic across nodes (#6297)Lisandro Pin1-21/+59
* Improve EC shards rebalancing logic across nodes. - Favor target nodes with less preexisting shards, to ensure a fair distribution. - Randomize selection when multiple possible target nodes are available. - Add logic to account for replication settings when selecting target nodes (currently disabled). * Fix minor test typo. * Clarify internal error messages for `pickEcNodeToBalanceShardsInto()`.
2024-11-21use math rand v2chrislu1-2/+2
2024-11-21Improve EC shards rebalancing logic across racks (#6270)Lisandro Pin1-20/+63
Improve EC shards rebalancing logic across racks. - Favor target shards with less preexisting shards, to ensure a fair distribution. - Randomize selection when multiple possible target shards are available. - Add logic to account for replication settings when selecting target shards (currently disabled).
2024-11-19Unify usage of shell.EcNode.dc as DataCenterId. (#6258)Lisandro Pin1-6/+5
2024-11-18Introduce logic to resolve volume replica placement within EC rebalancing. ↵Lisandro Pin1-26/+44
(#6254) * Rename `command_ec_encode_test.go` to `command_ec_common_test.go`. All tests defined in this file are now for `command_ec_common.go`. * Minor code cleanups. - Fix broken `ec.balance` test. - Rework integer ceiling division to not use floats, which can introduce precision errors. * Introduce logic to resolve volume replica placement within EC rebalancing. This will be used to make rebalancing logic topology-aware. * Give shell.EcNode.dc a dedicated DataCenterId type.
2024-11-04Refactor `ec.balance` logic into a `weeed/shell/command_ec_common.go`… (#6195)Lisandro Pin1-1/+423
* Refactor `ec.balance` logic into a `weeed/shell/command_ec_common.go` standalone function. This is a prerequisite to unify the balance logic for `ec.balance` and `ec.encode'. * s/Balance()/EcBalance()/g
2023-09-25Revert "Revert "Merge branch 'master' of ↵chrislu1-4/+4
https://github.com/seaweedfs/seaweedfs"" This reverts commit 8cb42c39
2023-09-18Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs"chrislu1-4/+4
This reverts commit 2e5aa06026750c99ea283181974d2ccfe5eb0468, reversing changes made to 4d414f54a224142f3f4d934f4af3b5dceb6fec6b.
2023-09-18Bump github.com/rclone/rclone from 1.63.1 to 1.64.0 (#4850)dependabot[bot]1-4/+4
* Bump github.com/rclone/rclone from 1.63.1 to 1.64.0 Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone) from 1.63.1 to 1.64.0. - [Release notes](https://github.com/rclone/rclone/releases) - [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md) - [Commits](https://github.com/rclone/rclone/compare/v1.63.1...v1.64.0) --- updated-dependencies: - dependency-name: github.com/rclone/rclone dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * API changes * go mod --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: chrislu <chris.lu@gmail.com>
2022-10-09refactor to change capacity data typechrislu1-4/+0
2022-09-14refactor(shell): `Decending` -> `Descending` (#3675)Ryan Russell1-2/+2
Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-08-22shell: stop long running jobs if lock is lostchrislu1-0/+4
2022-07-29move to https://github.com/seaweedfs/seaweedfschrislu1-8/+8
2022-04-18enhancement: replace sort.Slice with slices.SortFunc to reduce reflectionjustin1-9/+8
2022-02-08ec.encode: calculate free ec slots based on (maxVolumeCount-volumeCount)chrislu1-1/+1
fix https://github.com/chrislusf/seaweedfs/issues/2642
2022-02-08volume.balance: add delay during tight loopchrislu1-1/+1
fix https://github.com/chrislusf/seaweedfs/issues/2637
2021-12-26use streaming mode for long poll grpc callschrislu1-4/+4
streaming mode would create separate grpc connections for each call. this is to ensure the long poll connections are properly closed.
2021-09-12change server address from string to a typeChris Lu1-11/+15
2021-02-22refactoringChris Lu1-7/+4
2021-02-16avoid nilChris Lu1-1/+7
2021-02-16avoid possible nil disk infoChris Lu1-24/+27
2021-02-16this can compile now!!!Chris Lu1-11/+25
2020-09-14shell: add volumeServer.evacuate commandChris Lu1-4/+19
2020-09-12shell: volume.balance plan by ratio of fullnessChris Lu1-0/+4
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-25 16:45:25 +0000