aboutsummaryrefslogtreecommitdiff
path: root/weed/server/volume_grpc_copy.go
AgeCommit message (Collapse)AuthorFilesLines
11 daysfix: prevent empty .vif files from ec.decode causing parse errors (#7686)Chris Lu1-0/+9
* fix: prevent empty .vif files from ec.decode causing parse errors When ec.decode copies .vif files from EC shard nodes, if a source node doesn't have the .vif file, an empty .vif file was created on the target node. This caused volume.configure.replication to fail with 'proto: syntax error' when trying to parse the empty file. This fix: 1. In writeToFile: Remove empty files when no data was written (source file was not found) to avoid leaving corrupted empty files 2. In MaybeLoadVolumeInfo: Handle empty .vif files gracefully by treating them as non-existent, allowing the system to create a proper one Fixes #7666 * refactor: remove redundant dst.Close() and add error logging Address review feedback: - Remove redundant dst.Close() call since defer already handles it - Add error logging for os.Remove() failure
2025-12-05Separate vacuum speed from replication speed (#7632)msementsov1-2/+2
2025-12-04fix: properly handle errors in writeToFile to prevent 0-byte EC shards (#7620)Chris Lu1-3/+5
Fixes #7619 The writeToFile function had two critical bugs that could cause data loss during EC shard evacuation when the destination disk is full: Bug 1: When os.OpenFile fails (e.g., disk full), the error was silently ignored and nil was returned. This caused the caller to think the copy succeeded. Bug 2: When dst.Write fails (e.g., 'no space left on device'), the error was completely ignored because the return value was not checked. When evacuating EC shards to a full volume server (especially on BTRFS): 1. OpenFile may succeed (creates 0-byte file inode) 2. Write fails with 'no space left on device' 3. Errors were ignored, function returned nil 4. Caller thinks copy succeeded and deletes source shard 5. Result: 0-byte shard on destination, data loss! This fix ensures both errors are properly returned, preventing data loss. Added unit tests to verify the fix.
2025-07-30Admin: misc improvements on admin server and workers. EC now works. (#7055)Chris Lu1-0/+117
* initial design * added simulation as tests * reorganized the codebase to move the simulation framework and tests into their own dedicated package * integration test. ec worker task * remove "enhanced" reference * start master, volume servers, filer Current Status ✅ Master: Healthy and running (port 9333) ✅ Filer: Healthy and running (port 8888) ✅ Volume Servers: All 6 servers running (ports 8080-8085) 🔄 Admin/Workers: Will start when dependencies are ready * generate write load * tasks are assigned * admin start wtih grpc port. worker has its own working directory * Update .gitignore * working worker and admin. Task detection is not working yet. * compiles, detection uses volumeSizeLimitMB from master * compiles * worker retries connecting to admin * build and restart * rendering pending tasks * skip task ID column * sticky worker id * test canScheduleTaskNow * worker reconnect to admin * clean up logs * worker register itself first * worker can run ec work and report status but: 1. one volume should not be repeatedly worked on. 2. ec shards needs to be distributed and source data should be deleted. * move ec task logic * listing ec shards * local copy, ec. Need to distribute. * ec is mostly working now * distribution of ec shards needs improvement * need configuration to enable ec * show ec volumes * interval field UI component * rename * integration test with vauuming * garbage percentage threshold * fix warning * display ec shard sizes * fix ec volumes list * Update ui.go * show default values * ensure correct default value * MaintenanceConfig use ConfigField * use schema defined defaults * config * reduce duplication * refactor to use BaseUIProvider * each task register its schema * checkECEncodingCandidate use ecDetector * use vacuumDetector * use volumeSizeLimitMB * remove remove * remove unused * refactor * use new framework * remove v2 reference * refactor * left menu can scroll now * The maintenance manager was not being initialized when no data directory was configured for persistent storage. * saving config * Update task_config_schema_templ.go * enable/disable tasks * protobuf encoded task configurations * fix system settings * use ui component * remove logs * interface{} Reduction * reduce interface{} * reduce interface{} * avoid from/to map * reduce interface{} * refactor * keep it DRY * added logging * debug messages * debug level * debug * show the log caller line * use configured task policy * log level * handle admin heartbeat response * Update worker.go * fix EC rack and dc count * Report task status to admin server * fix task logging, simplify interface checking, use erasure_coding constants * factor in empty volume server during task planning * volume.list adds disk id * track disk id also * fix locking scheduled and manual scanning * add active topology * simplify task detector * ec task completed, but shards are not showing up * implement ec in ec_typed.go * adjust log level * dedup * implementing ec copying shards and only ecx files * use disk id when distributing ec shards 🎯 Planning: ActiveTopology creates DestinationPlan with specific TargetDisk 📦 Task Creation: maintenance_integration.go creates ECDestination with DiskId 🚀 Task Execution: EC task passes DiskId in VolumeEcShardsCopyRequest 💾 Volume Server: Receives disk_id and stores shards on specific disk (vs.store.Locations[req.DiskId]) 📂 File System: EC shards and metadata land in the exact disk directory planned * Delete original volume from all locations * clean up existing shard locations * local encoding and distributing * Update docker/admin_integration/EC-TESTING-README.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * check volume id range * simplify * fix tests * fix types * clean up logs and tests --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-07-16convert error fromating to %w everywhere (#6995)Chris Lu1-2/+2
2025-06-16add version to volume protochrislu1-0/+1
2024-09-03copy ec shards to disks already having ec volumeschrislu1-1/+3
fix https://github.com/seaweedfs/seaweedfs/issues/5615
2024-06-14Added context for the MasterClient's methods to avoid endless loops (#5628)vadimartynov1-3/+3
* Added context for the MasterClient's methods to avoid endless loops * Returned WithClient function. Added WithClientCustomGetMaster function * Hid unused ctx arguments * Using a common context for the KeepConnectedToMaster and WaitUntilConnected functions * Changed the context termination check in the tryConnectToMaster function * Added a child context to the tryConnectToMaster function * Added a common context for KeepConnectedToMaster and WaitUntilConnected functions in benchmark
2024-03-09remove repetitive words (#5364)rustrover1-1/+1
2023-10-24fix copying .vif file in VolumeCopy (#4943)zehweh1-1/+1
closes #4934 fixes #2633 might fix #3528
2023-06-12Delete volume is empty (#4561)Konstantin Lebedev1-1/+1
* use onlyEmpty for deleteVolume https://github.com/seaweedfs/seaweedfs/issues/4559 * fix IsEmpty * fix test --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2022-10-16shell: volume.move handles volume moved to cloud tierchrislu1-18/+28
fix https://github.com/seaweedfs/seaweedfs/issues/3803
2022-08-23remove old raft servers if they don't answer to pings for too long (#3398)askeipx1-3/+4
* remove old raft servers if they don't answer to pings for too long add ping durations as options rename ping fields fix some todos get masters through masterclient raft remove server from leader use raft servers to ping them CheckMastersAlive for hashicorp raft only * prepare blocking ping * pass waitForReady as param * pass waitForReady through all functions * waitForReady works * refactor * remove unneeded params * rollback unneeded changes * fix
2022-08-21feat(weed.move): add a speed limit parameter of moving files (#3478)qzh1-6/+18
* feat(weed.move): add a speed limit parameter of moving files * fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
2022-07-29move to https://github.com/seaweedfs/seaweedfschrislu1-11/+11
2022-06-20Determine whether to preallocate according to the master configuration ↵石昌林1-0/+24
before executing copy volume
2022-04-26volume: sync to disk before copying volume fileschrislu1-0/+1
address https://github.com/chrislusf/seaweedfs/issues/2976
2022-02-04change to util.WriteFilechrislu1-1/+1
2021-12-26use streaming mode for long poll grpc callschrislu1-2/+2
streaming mode would create separate grpc connections for each call. this is to ensure the long poll connections are properly closed.
2021-11-15prevent nil responseChris Lu1-1/+1
fix https://github.com/chrislusf/seaweedfs/issues/2452
2021-11-02go fmtChris Lu1-1/+1
2021-10-24volume copy: stream out copying progress and avoid grpc request timeoutChris Lu1-14/+41
fix https://github.com/chrislusf/seaweedfs/issues/2386
2021-10-14refactor: move from io/ioutil to io and os packageEng Zer Jun1-4/+3
The io/ioutil package has been deprecated as of Go 1.16, see https://golang.org/doc/go1.16#ioutil. This commit replaces the existing io/ioutil functions with their new definitions in io and os packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-09-12change server address from string to a typeChris Lu1-1/+2
2021-09-01go fmtChris Lu1-4/+4
2021-09-01volume: copy file also copies modification timeChris Lu1-14/+35
to ensure ttl can work well
2021-08-13volume copying: clean up stale volume data filesChris Lu1-3/+12
fix https://github.com/chrislusf/seaweedfs/issues/2250
2021-05-07minorChris Lu1-1/+1
2021-02-22volume.tier.move: passing non-empty disk typeChris Lu1-1/+1
2021-02-16use hdd instead of empty stringChris Lu1-1/+1
2021-02-16this can compile now!!!Chris Lu1-1/+2
2021-02-09volume: add capability to change disk type when moving a volumeChris Lu1-1/+5
2020-12-13allocate volume by disk typeChris Lu1-5/+6
2020-11-27volume: add "-dir.idx" option for separate index storageChris Lu1-13/+18
fix https://github.com/chrislusf/seaweedfs/issues/1265
2020-10-31go fmtChris Lu1-1/+1
2020-10-27volume: add a note file to avoid incomplete volume filesChris Lu1-0/+5
fix https://github.com/chrislusf/seaweedfs/issues/1567
2020-10-27copy large file firstChris Lu1-3/+2
2020-09-01volume: Don't unmount before deleting volume in copyJames Hartig1-7/+2
If we unmount first and then delete, the delete fails because the volume was unmounted. Delete ends up doing the same thing as the unmount anyways.
2020-05-17add error to avoid copying not found volumeChris Lu1-4/+7
fix https://github.com/chrislusf/seaweedfs/issues/1317
2020-04-05refactoringChris Lu1-1/+1
2020-04-05add util.PathJoinChris Lu1-2/+1
2020-04-02volume copying adds cleaning up on errorChris Lu1-4/+22
fix https://github.com/chrislusf/seaweedfs/issues/1253
2020-02-25remove ctx if possibleChris Lu1-6/+5
2020-02-25avoid reusing context objectChris Lu1-2/+2
fix https://github.com/chrislusf/seaweedfs/issues/1182
2020-01-26use the same context object in order to retryChris Lu1-1/+1
2020-01-04volume: copy volumes also include .vif fileChris Lu1-7/+8
2019-12-28fix edge casesChris Lu1-0/+3
2019-12-23shell: add ec.decode commandChris Lu1-9/+13
2019-10-06scaffold for volume server query featureChris Lu1-4/+0
2019-06-20go fmtChris Lu1-2/+2
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-20 22:44:43 +0000