aboutsummaryrefslogtreecommitdiff
path: root/weed/command/filer_copy.go
AgeCommit message (Collapse)AuthorFilesLines
2025-07-16convert error fromating to %w everywhere (#6995)Chris Lu1-8/+8
2025-05-22added context to filer_client method calls (#6808)Aleksey Kosov1-2/+2
Co-authored-by: akosov <a.kosov@kryptonite.ru>
2024-10-13remove unusedchrislu1-2/+0
2024-08-01rename functionschrislu1-1/+1
2024-07-16Added tls for http clients (#5766)vadimartynov1-3/+18
* Added global http client * Added Do func for global http client * Changed the code to use the global http client * Fix http client in volume uploader * Fixed pkg name * Fixed http util funcs * Fixed http client for bench_filer_upload * Fixed http client for stress_filer_upload * Fixed http client for filer_server_handlers_proxy * Fixed http client for command_fs_merge_volumes * Fixed http client for command_fs_merge_volumes and command_volume_fsck * Fixed http client for s3api_server * Added init global client for main funcs * Rename global_client to client * Changed: - fixed NewHttpClient; - added CheckIsHttpsClientEnabled func - updated security.toml in scaffold * Reduce the visibility of some functions in the util/http/client pkg * Added the loadSecurityConfig function * Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-07-16Added loadSecurityConfigOnce (#5792)vadimartynov1-1/+1
2024-06-14Added context for the MasterClient's methods to avoid endless loops (#5628)vadimartynov1-1/+1
* Added context for the MasterClient's methods to avoid endless loops * Returned WithClient function. Added WithClientCustomGetMaster function * Hid unused ctx arguments * Using a common context for the KeepConnectedToMaster and WaitUntilConnected functions * Changed the context termination check in the tryConnectToMaster function * Added a child context to the tryConnectToMaster function * Added a common context for KeepConnectedToMaster and WaitUntilConnected functions in benchmark
2023-12-22adjust optionschrislu1-1/+1
2023-12-22fixchrislu1-1/+1
2023-12-22go fmtchrislu1-15/+15
2023-12-22option to filer.copy with public volume server urlchrislu1-0/+5
fix https://github.com/seaweedfs/seaweedfs/issues/5126
2023-01-20grpc connection to filer add sw-client-id headerchrislu1-5/+7
2023-01-02more solid weed mount (#4089)Chris Lu1-4/+4
* compare chunks by timestamp * fix slab clearing error * fix test compilation * move oldest chunk to sealed, instead of by fullness * lock on fh.entryViewCache * remove verbose logs * revert slat clearing * less logs * less logs * track write and read by timestamp * remove useless logic * add entry lock on file handle release * use mem chunk only, swap file chunk has problems * comment out code that maybe used later * add debug mode to compare data read and write * more efficient readResolvedChunks with linked list * small optimization * fix test compilation * minor fix on writer * add SeparateGarbageChunks * group chunks into sections * turn off debug mode * fix tests * fix tests * tmp enable swap file chunk * Revert "tmp enable swap file chunk" This reverts commit 985137ec472924e4815f258189f6ca9f2168a0a7. * simple refactoring * simple refactoring * do not re-use swap file chunk. Sealed chunks should not be re-used. * comment out debugging facilities * either mem chunk or swap file chunk is fine now * remove orderedMutex as *semaphore.Weighted not found impactful * optimize size calculation for changing large files * optimize performance to avoid going through the long list of chunks * still problems with swap file chunk * rename * tiny optimization * swap file chunk save only successfully read data * fix * enable both mem and swap file chunk * resolve chunks with range * rename * fix chunk interval list * also change file handle chunk group when adding chunks * pick in-active chunk with time-decayed counter * fix compilation * avoid nil with empty fh.entry * refactoring * rename * rename * refactor visible intervals to *list.List * refactor chunkViews to *list.List * add IntervalList for generic interval list * change visible interval to use IntervalList in generics * cahnge chunkViews to *IntervalList[*ChunkView] * use NewFileChunkSection to create * rename variables * refactor * fix renaming leftover * renaming * renaming * add insert interval * interval list adds lock * incrementally add chunks to readers Fixes: 1. set start and stop offset for the value object 2. clone the value object 3. use pointer instead of copy-by-value when passing to interval.Value 4. use insert interval since adding chunk could be out of order * fix tests compilation * fix tests compilation
2022-09-14go fmtchrislu1-13/+13
2022-09-14refactor: `concurrentFiles` and `concurrentChunks` var name fix (#3655)Ryan Russell1-7/+7
2022-08-23remove old raft servers if they don't answer to pings for too long (#3398)askeipx1-1/+1
* remove old raft servers if they don't answer to pings for too long add ping durations as options rename ping fields fix some todos get masters through masterclient raft remove server from leader use raft servers to ping them CheckMastersAlive for hashicorp raft only * prepare blocking ping * pass waitForReady as param * pass waitForReady through all functions * waitForReady works * refactor * remove unneeded params * rollback unneeded changes * fix
2022-08-20refactoringchrislu1-56/+20
2022-08-20filer.copy: retryable file part uploadchrislu1-43/+26
2022-08-20add fileId to UploadWithRetry return resultchrislu1-3/+1
2022-08-20filer.copy: retryable uploadchrislu1-48/+46
2022-08-20remove unused collection and replication from upload resultchrislu1-6/+5
2022-08-20remove unused codechrislu1-7/+0
2022-07-29move to https://github.com/seaweedfs/seaweedfschrislu1-9/+9
2022-06-06filer: remove replication, collection, disk_type info from entry metadatachrislu1-20/+16
these metadata can change and are not used
2022-04-21fix typochrislu1-1/+1
2022-01-19url should be always using forward slashchrislu1-1/+1
2022-01-11filer.copy: don't crash when volume creation failsKyle Sanderson1-1/+2
panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247] goroutine 7482 [running]: github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x2) /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7 created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225 panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247] goroutine 7480 [running]: github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x0) /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7 created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
2021-12-26use streaming mode for long poll grpc callschrislu1-7/+7
streaming mode would create separate grpc connections for each call. this is to ensure the long poll connections are properly closed.
2021-10-14refactor: move from io/ioutil to io and os packageEng Zer Jun1-6/+4
The io/ioutil package has been deprecated as of Go 1.16, see https://golang.org/doc/go1.16#ioutil. This commit replaces the existing io/ioutil functions with their new definitions in io and os packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-01typoChris Lu1-1/+1
2021-10-01retry both assign volume and uploading dataChris Lu1-28/+34
fix https://github.com/chrislusf/seaweedfs/issues/2351
2021-09-12change server address from string to a typeChris Lu1-42/+25
2021-09-07handle ipv6 addressesChris Lu1-1/+1
2021-09-06refactoringChris Lu1-5/+30
2021-08-18filer.copy: large files pack file ids into chunksChris Lu1-1/+58
2021-07-23filer.copy: fail early if assign request failedChris Lu1-0/+3
fix https://github.com/chrislusf/seaweedfs/issues/2216
2021-07-23weed -h subcommand list is not sorted, makes discovery difficultChris Lu1-14/+14
fix https://github.com/chrislusf/seaweedfs/issues/2214
2021-06-13filer.copy: zero fileSize for directoriesChris Lu1-1/+5
2021-05-12filer.copy: "check.size" before copying filesChris Lu1-5/+57
fix https://github.com/chrislusf/seaweedfs/issues/2067
2021-05-07add retry to assign volumeChris Lu1-36/+40
fix https://github.com/chrislusf/seaweedfs/issues/2056
2021-05-07remove duplicated codeChris Lu1-3/+0
2021-05-06go fmtChris Lu1-1/+1
2021-04-21filer.copy copy empty foldersChris Lu1-17/+19
fix https://github.com/chrislusf/seaweedfs/issues/2016
2021-04-01smaller file chunks are fasterChris Lu1-1/+1
2021-02-22disk type support custom tagsChris Lu1-1/+1
2021-02-17refactoring to get master function, instead of passing master values directlyChris Lu1-1/+3
this will enable retrying later
2021-02-14disk type only supports hdd and ssd, not ready for random tags yetChris Lu1-1/+1
2021-02-13adjust help messageChris Lu1-1/+1
2021-02-09Merge branch 'master' into support_ssd_volumeChris Lu1-1/+1
2021-01-06add "weed filer.cat" to read files directly from volume serversChris Lu1-1/+1
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-21 06:51:30 +0000