aboutsummaryrefslogtreecommitdiff
path: root/weed/operation/upload_content.go
AgeCommit message (Collapse)AuthorFilesLines
2024-07-16Added tls for http clients (#5766)vadimartynov1-33/+51
* Added global http client * Added Do func for global http client * Changed the code to use the global http client * Fix http client in volume uploader * Fixed pkg name * Fixed http util funcs * Fixed http client for bench_filer_upload * Fixed http client for stress_filer_upload * Fixed http client for filer_server_handlers_proxy * Fixed http client for command_fs_merge_volumes * Fixed http client for command_fs_merge_volumes and command_volume_fsck * Fixed http client for s3api_server * Added init global client for main funcs * Rename global_client to client * Changed: - fixed NewHttpClient; - added CheckIsHttpsClientEnabled func - updated security.toml in scaffold * Reduce the visibility of some functions in the util/http/client pkg * Added the loadSecurityConfig function * Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-07-01refactor all methods strings to const (#5726)Konstantin Lebedev1-1/+1
2024-04-11[volume] Reduce the number of buffers for uploading one chunk (#5458)Konstantin Lebedev1-5/+20
2024-01-10chore: add status code for request_total metrics (#5188)Konstantin Lebedev1-1/+1
2023-10-03weed/operation: fix dropped error (#4888)Lars Lehtonen1-0/+3
2023-10-01RetryForever => RetryUntilchrislu1-1/+1
2023-04-21mount: add retry for read only case (#4416)wusong1-6/+8
* mount: add retry for read only case Signed-off-by: Wusong Wang <wangwusong@virtaitech.com> * add new util retry function for mount Signed-off-by: Wusong Wang <wangwusong@virtaitech.com> * change error list param Signed-off-by: Wusong Wang <wangwusong@virtaitech.com> --------- Signed-off-by: Wusong Wang <wangwusong@virtaitech.com> Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>
2023-01-02more solid weed mount (#4089)Chris Lu1-2/+2
* compare chunks by timestamp * fix slab clearing error * fix test compilation * move oldest chunk to sealed, instead of by fullness * lock on fh.entryViewCache * remove verbose logs * revert slat clearing * less logs * less logs * track write and read by timestamp * remove useless logic * add entry lock on file handle release * use mem chunk only, swap file chunk has problems * comment out code that maybe used later * add debug mode to compare data read and write * more efficient readResolvedChunks with linked list * small optimization * fix test compilation * minor fix on writer * add SeparateGarbageChunks * group chunks into sections * turn off debug mode * fix tests * fix tests * tmp enable swap file chunk * Revert "tmp enable swap file chunk" This reverts commit 985137ec472924e4815f258189f6ca9f2168a0a7. * simple refactoring * simple refactoring * do not re-use swap file chunk. Sealed chunks should not be re-used. * comment out debugging facilities * either mem chunk or swap file chunk is fine now * remove orderedMutex as *semaphore.Weighted not found impactful * optimize size calculation for changing large files * optimize performance to avoid going through the long list of chunks * still problems with swap file chunk * rename * tiny optimization * swap file chunk save only successfully read data * fix * enable both mem and swap file chunk * resolve chunks with range * rename * fix chunk interval list * also change file handle chunk group when adding chunks * pick in-active chunk with time-decayed counter * fix compilation * avoid nil with empty fh.entry * refactoring * rename * rename * refactor visible intervals to *list.List * refactor chunkViews to *list.List * add IntervalList for generic interval list * change visible interval to use IntervalList in generics * cahnge chunkViews to *IntervalList[*ChunkView] * use NewFileChunkSection to create * rename variables * refactor * fix renaming leftover * renaming * renaming * add insert interval * interval list adds lock * incrementally add chunks to readers Fixes: 1. set start and stop offset for the value object 2. clone the value object 3. use pointer instead of copy-by-value when passing to interval.Value 4. use insert interval since adding chunk could be out of order * fix tests compilation * fix tests compilation
2022-10-28refactor filer proto chunk variable from mtime to modified_ts_nschrislu1-1/+1
2022-10-19add md5 header when UploadData to replication in ReplicatedWrite (#3881)liubaojiang1-0/+5
2022-09-10refactorchrislu1-3/+4
2022-08-31upload_content: upload close response.Bodychrislu1-1/+2
2022-08-20adjust logchrislu1-1/+1
2022-08-20add fileId to UploadWithRetry return resultchrislu1-2/+2
2022-08-20filer.copy: retryable uploadchrislu1-0/+49
2022-08-04refactor FilerRequest metrics (#3402)Konstantin Lebedev1-0/+2
* refactor FilerRequest metrics * avoid double count proxy * defer to
2022-07-29move to https://github.com/seaweedfs/seaweedfschrislu1-4/+4
2022-03-15reduced the timeout for creating a connection from 30 to 10 secondsKonstantin Lebedev1-0/+6
2022-01-05mount: skip special character in the filenameschrislu1-7/+7
fix https://github.com/chrislusf/seaweedfs/issues/2559
2021-10-14refactor: move from io/ioutil to io and os packageEng Zer Jun1-3/+2
The io/ioutil package has been deprecated as of Go 1.16, see https://golang.org/doc/go1.16#ioutil. This commit replaces the existing io/ioutil functions with their new definitions in io and os packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-09-06refactorChris Lu1-23/+39
2021-09-06refactoringChris Lu1-28/+38
2021-04-28fix aws style Etag for chunksKonstantin Lebedev1-1/+1
2021-04-18resend the http request if connection is staleChris Lu1-4/+5
2021-04-10go fmtChris Lu1-1/+1
2021-04-07skip connection reset errorChris Lu1-2/+4
fix https://github.com/chrislusf/seaweedfs/issues/1971 this is because the connections are pooled but the volume server has reset the connection
2021-04-01refactor buffer poolChris Lu1-3/+2
2021-03-07fix nilChris Lu1-1/+1
2021-03-07stats collect chunk upload retry countChris Lu1-0/+2
2021-02-22filer: avoid encryption and compression at the same timeChris Lu1-16/+17
fix https://github.com/chrislusf/seaweedfs/issues/1828
2021-02-18Merge remote-tracking branch 'origin/master'bingoohuang1-0/+2
2021-02-18use backticks instead of double quotes to avoid escaped additionally in regexbingoohuang1-1/+1
2021-02-17go fmtChris Lu1-1/+1
2021-02-17add sleep between upload retriesChris Lu1-0/+1
2021-02-12adjust http max idle connections per hostChris Lu1-0/+1
related to https://github.com/chrislusf/seaweedfs/issues/1802
2020-11-11add a {X-,}Idempotency-Key header for http POSTChris Lu1-0/+1
fix https://github.com/chrislusf/seaweedfs/issues/1592 according to https://github.com/golang/go/issues/19943
2020-11-11print out data sizeChris Lu1-1/+1
2020-11-02avoid []byte reuseChris Lu1-4/+1
2020-11-02Revert "revert bytebufferpool, seems problematic"Chris Lu1-4/+9
This reverts commit e0fcab47c00bbebf2352000bc8f8a9a2e30c076f.
2020-11-02revert bytebufferpool, seems problematicChris Lu1-9/+4
2020-10-30add back upload retriesChris Lu1-1/+1
2020-10-24print for debuggingChris Lu1-0/+2
2020-10-23adjust compression thresholdChris Lu1-1/+1
2020-10-23byte buffer for uploadingChris Lu1-3/+4
2020-10-23fix bug found by testsChris Lu1-0/+1
2020-10-23memory pool to reduce memory allocationChris Lu1-1/+4
2020-10-20mount: shortcut when there is only one chunkChris Lu1-4/+9
2020-10-20Revert "a better byte buffer pool"Chris Lu1-6/+2
This reverts commit 59c6422777d8a6703865c8654eb215140cebd29b.
2020-10-20a better byte buffer poolChris Lu1-2/+6
2020-09-09filer: cross cluster synchronizationChris Lu1-3/+5
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-26 09:01:59 +0000