aboutsummaryrefslogtreecommitdiff
path: root/weed/shell/command_ec_encode.go
AgeCommit message (Collapse)AuthorFilesLines
12 daysNit: have `ec.encode` exit immediately if no volumes are processed. (#7654)Lisandro Pin1-0/+4
* Nit: have `ec.encode` exit immediately if no volumes are processed. * Update weed/shell/command_ec_encode.go Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-08-23Shell: support regular expression for collection selection (#7158)Chris Lu1-10/+32
* support regular expression for collection selection * refactor * ordering * fix exact match * Update command_volume_balance_test.go * simplify * Update command_volume_balance.go * comment
2025-08-07Shell: add verbose ec encoding mode (#7105)Chris Lu1-20/+127
* add verbose ec encoding mode * address comments
2025-08-03mount ec shards correctly (#7079)Chris Lu1-2/+2
2025-07-16convert error fromating to %w everywhere (#6995)Chris Lu1-5/+5
2025-07-14Collecting volume locations for volumes before EC encodingchrislu1-11/+26
fix https://github.com/seaweedfs/seaweedfs/issues/6963
2025-06-15Support filtering source disk type in volume.tier.upload (#6868)NyaMisty1-3/+5
2025-05-13Nit: unify the default `--maxParallelization` value for `weed shell` ↵Lisandro Pin1-1/+1
commands supporting this option (#6788)
2025-05-09Improve safety for weed shell's `ec.encode`. (#6773)Lisandro Pin1-35/+55
Improve safety for weed shells `ec.encode`. The current process for `ec.encode` is: 1. EC shards for a volume are generated and added to a single server 2. The original volume is deleted 3. EC shards get re-balanced across the entire topology It is then possible to lose data between #2 and #3, if the underlying volume storage/server/rack/DC happens to fail, for whatever reason. As a fix, this MR reworks `ec.encode` so: * Newly created EC shards are spread across all locations for the source volume. * Source volumes are deleted only after EC shards are converted and balanced.
2025-05-08Improve parallelization for `ec.encode` (#6769)Lisandro Pin1-34/+61
Improve parallelization for `ec.encode`. Instead of processing one volume at at time, perform all EC conversion steps (mark readonly -> generate EC shards -> delete volume -> remount) in parallel for all of them. This should substantially improve performance when EC encoding entire collections.
2025-02-28`ec.encode`: Fix resolution of target collections. (#6585)Lisandro Pin1-8/+4
* Don't ignore empty (`""`) collection names when computing collections for a given volume ID. * `ec.encode`: Fix resolution of target collections. When no `volumeId` parameter is provided, compute volumes based on the provided collection name, even if it's empty (`""`). This restores behavior to before recent EC rebalancing rework. See also https://github.com/seaweedfs/seaweedfs/blob/ec30a504bae6cad75f859964e14c60d39cc43709/weed/shell/command_ec_encode.go#L99 .
2025-02-10`ec.encode`: Explictly mount EC shards after volume conversion. (#6528)Lisandro Pin1-2/+12
This guarantees EC shards are immediately available after encoding, even if not affected by subsequent re-balancing.
2025-01-17`ec.encode`: Fix bug causing source volumes not being deleted after EC ↵Lisandro Pin1-1/+18
conversion. (#6447) This logic was originally part of `spreadEcShards()`, which got removed during the unification effort with `ec.balance` (https://github.com/seaweedfs/seaweedfs/pull/6344), accidentally breaking functionality in the process. The commit restores the deletion code for EC'd volumes - with parallelization support.
2024-12-19Fix volume replica parallelization within `ec.encode`. (#6377)Lisandro Pin1-5/+3
See 826edd5d.
2024-12-18Allow configuring the maximum number of concurrent tasks for EC ↵Lisandro Pin1-2/+2
parallelization. (#6376) Follow-up to b0210df0.
2024-12-18Parallelize volume replica operations within `ec.encode`. (#6374)Lisandro Pin1-6/+15
2024-12-12Begin implementing EC balancing parallelization support. (#6342)Lisandro Pin1-3/+2
* Begin implementing EC balancing parallelization support. Impacts both `ec.encode` and `ec.balance`, * Nit: improve type naming. * Make the goroutine workgroup handler for `EcBalance()` a bit smarter/error-proof. * Nit: unify naming for `ecBalancer` wait group methods with the rest of the module. * Fix concurrency bug. * Fix whitespace after Gitlab automerge. * Delete stray TODO.
2024-12-12Limit EC re-balancing for `ec.encode` to relevant collections when a volume ↵Lisandro Pin1-5/+1
ID argument is provided. (#6347) Limit EC re-balancing for `ec.encode` to relevant collections when a volume ID is provided.
2024-12-12Delete legacy balancing code for `ec.encode`. (#6344)Lisandro Pin1-133/+0
2024-12-10Unify the re-balancing logic for `ec.encode` with `ec.balance`. (#6339)Lisandro Pin1-29/+44
Among others, this enables recent changes related to topology aware re-balancing at EC encoding time.
2024-11-19Unify usage of shell.EcNode.dc as DataCenterId. (#6258)Lisandro Pin1-2/+2
2024-11-09delete aborted ec shards from both source and target servers (#6221)Chris Lu1-1/+4
fix https://github.com/seaweedfs/seaweedfs/issues/6205#issuecomment-2465004586
2024-10-31fix format (#6185)wyang1-1/+1
unitest weed/shell fail
2024-10-28fix formatchrislu1-2/+2
2024-10-24ensure 2 volume space since actual need 1.4x volume size empty spacechrislu1-2/+2
2024-10-24correcting free volume count, factor it during ec encoding to ensure enough ↵chrislu1-3/+19
disk space available fix https://github.com/seaweedfs/seaweedfs/issues/6163
2024-09-29refactorchrislu1-1/+1
2024-09-28add IsResourceHeavy() to command interfacechrislu1-0/+4
2024-09-24fix(volume): don't persist RO state in specific cases (#6058)Max Denushev1-3/+4
* fix(volume): don't persist RO state in specific cases * fix(volume): writable always persist
2024-06-02Ignore remote volume when selecting volumes in operation ↵NyaMisty1-0/+4
(ec.encode/volume.tier.upload) (#5635)
2023-10-22log full percentagechrislu1-1/+1
2023-10-05default to skip if less than 4 nodeschrislu1-0/+19
2023-07-06clone volume locations in case they are changedchrislu1-1/+1
fix https://github.com/seaweedfs/seaweedfs/issues/4642
2023-06-12Delete volume is empty (#4561)Konstantin Lebedev1-1/+1
* use onlyEmpty for deleteVolume https://github.com/seaweedfs/seaweedfs/issues/4559 * fix IsEmpty * fix test --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-02-09fix bug when vid not foundchrislu1-1/+1
fix https://github.com/seaweedfs/seaweedfs/issues/4193
2022-08-22shell: stop long running jobs if lock is lostchrislu1-0/+4
2022-07-29move to https://github.com/seaweedfs/seaweedfschrislu1-7/+7
2022-04-05erasure coding: tracking encoded/decoded volumeschrislu1-1/+1
If an EC shard is created but not spread to other servers, the masterclient would think this shard is not located here.
2022-02-08volume.balance: add delay during tight loopchrislu1-1/+1
fix https://github.com/chrislusf/seaweedfs/issues/2637
2021-12-26use streaming mode for long poll grpc callschrislu1-1/+1
streaming mode would create separate grpc connections for each call. this is to ensure the long poll connections are properly closed.
2021-12-10add lock messageschrislu1-1/+1
2021-11-04randomize a bit for ec shards distributionChris Lu1-1/+2
2021-11-01adjust help message since both fullPercent and quietFor are needed.Chris Lu1-1/+1
2021-09-13shell: do not need to lock to see volume -hChris Lu1-4/+4
2021-09-13erasure coding: add cleanup step if anything goes wrongChris Lu1-0/+14
2021-09-12change server address from string to a typeChris Lu1-6/+7
2021-08-13shell: volume.tier.move makes up changes if volume move failedChris Lu1-23/+1
2021-05-06optional parallel copy ec shardsChris Lu1-20/+27
fix https://github.com/chrislusf/seaweedfs/issues/2048
2021-02-28adjust textChris Lu1-1/+1
2021-02-22refactoringChris Lu1-7/+4
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-21 05:21:37 +0000