aboutsummaryrefslogtreecommitdiff
path: root/weed/util/http_util.go
AgeCommit message (Collapse)AuthorFilesLines
2024-07-16Added tls for http clients (#5766)vadimartynov1-497/+0
* Added global http client * Added Do func for global http client * Changed the code to use the global http client * Fix http client in volume uploader * Fixed pkg name * Fixed http util funcs * Fixed http client for bench_filer_upload * Fixed http client for stress_filer_upload * Fixed http client for filer_server_handlers_proxy * Fixed http client for command_fs_merge_volumes * Fixed http client for command_fs_merge_volumes and command_volume_fsck * Fixed http client for s3api_server * Added init global client for main funcs * Rename global_client to client * Changed: - fixed NewHttpClient; - added CheckIsHttpsClientEnabled func - updated security.toml in scaffold * Reduce the visibility of some functions in the util/http/client pkg * Added the loadSecurityConfig function * Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-07-01refactor all methods strings to const (#5726)Konstantin Lebedev1-7/+7
2024-04-24filer read chunk retry if status code 499 (#5528)Konstantin Lebedev1-1/+1
2024-04-08fix: filer authenticate with with volume server (#5480)Henco Appel1-15/+22
2024-01-13refactoringchrislu1-0/+38
2023-09-01weed/util: fix dropped errors (#4803)Lars Lehtonen1-0/+12
2022-08-31just a bit saferchrislu1-0/+3
2022-08-31minorchrislu1-2/+2
2022-08-31s3: close response body with copying objectchrislu1-6/+6
2022-08-19filer: retryable when error is not foundchrislu1-1/+1
2022-08-18minorchrislu1-1/+3
2022-08-04filer prefer volume server in same data center (#3405)Konstantin Lebedev1-3/+2
* initial prefer same data center https://github.com/seaweedfs/seaweedfs/issues/3404 * GetDataCenter * prefer same data center for ReplicationSource * GetDataCenterId * remove glog
2022-07-29move to https://github.com/seaweedfs/seaweedfschrislu1-3/+3
2022-02-26use memory poolchrislu1-1/+3
2022-01-01Merge pull request #2543 from skurfuerst/seaweedfs-158Chris Lu1-2/+15
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
2021-12-31BUGFIX: ensure Authorization header is only added onceSebastian Kurfuerst1-2/+2
2021-12-30FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 ClientSebastian Kurfuerst1-2/+15
- one JWT for reading and one for writing, analogous to how the JWT between Master and Volume Server works - I did not implement IP `whiteList` parameter on the filer Additionally, because http_util.DownloadFile now sets the JWT, the `download` command should now work when `jwt.signing.read` is configured. By looking at the code, I think this case did not work before. ## Docs to be adjusted after a release Page `Amazon-S3-API`: ``` # Authentication with Filer You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as explained in [Security-Configuration](Security-Configuration) - controlled by the `grpc.*` configuration in `security.toml`. Starting with version XX, it is also possible to authenticate the HTTP operations between the S3-API-Proxy and the Filer (especially uploading new files). This is configured by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. With both configurations (gRPC and JWT), it is possible to have Filer and S3 communicate in fully authenticated fashion; so Filer will reject any unauthenticated communication. ``` Page `Security Overview`: ``` The following items are not covered, yet: - master server http REST services Starting with version XX, the Filer HTTP REST services can be secured with a JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. ... Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer. Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).** ... # Securing Filer HTTP with JWT To enable JWT-based access control for the Filer, 1. generate `security.toml` file by `weed scaffold -config=security` 2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string 3. copy the same `security.toml` file to the filers and all S3 proxies. If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`. If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`. The S3 API Gateway reads the above JWT keys and sends authenticated HTTP requests to the filer. ``` Page `Security Configuration`: ``` (update scaffold file) ... [filer_jwt.signing] key = "blahblahblahblah" [filer_jwt.signing.read] key = "blahblahblahblah" ``` Resolves: #158
2021-12-29s3: increase timeout limitchrislu1-2/+21
https://github.com/chrislusf/seaweedfs/issues/2541
2021-12-03return ' shouldRetry=true' so that filer can retry the failed chunkTanmoy Majumdar1-1/+1
2021-10-14refactor: move from io/ioutil to io and os packageEng Zer Jun1-8/+7
The io/ioutil package has been deprecated as of Go 1.16, see https://golang.org/doc/go1.16#ioutil. This commit replaces the existing io/ioutil functions with their new definitions in io and os packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-08-12use grpc and jwtChris Lu1-1/+1
2021-04-10go fmtChris Lu1-1/+0
2021-04-10add gateway for easier POST and DELETE blobsChris Lu1-0/+22
2021-03-16revert fasthttp changesChris Lu1-1/+1
related to https://github.com/chrislusf/seaweedfs/issues/1907
2021-02-12use fasthttp lib to readChris Lu1-1/+1
2021-02-12adjust http max idle connections per hostChris Lu1-0/+1
related to https://github.com/chrislusf/seaweedfs/issues/1802
2020-11-30close http request bodyChris Lu1-0/+5
2020-11-15remove unused functionChris Lu1-17/+0
2020-11-11s3: do not close reader too earlyChris Lu1-1/+0
fix https://github.com/chrislusf/seaweedfs/issues/1609
2020-10-13Only wait on retryable requestsChris Lu1-18/+20
2020-09-09return http response directlyChris Lu1-2/+2
2020-08-24close http responseChris Lu1-0/+1
2020-08-01non-fatal errorChris Lu1-1/+1
2020-08-01http request use gzip if possibleChris Lu1-8/+46
2020-08-01in case when content is not compressedChris Lu1-1/+1
2020-06-20rename UnCompressData to DecompressDataChris Lu1-1/+1
2020-06-19renamingChris Lu1-5/+5
2020-04-17fix "call of Unmarshal passes non-pointer as second argument"Chris Lu1-1/+1
2020-03-29return part of the chunk if chunkview is not the full chunkChris Lu1-4/+8
2020-03-08filer: remember content is gzipped or notChris Lu1-5/+11
2020-03-06filer: option to encrypt data on volume serverChris Lu1-11/+40
2020-02-14refactoring the close http responseChris Lu1-5/+5
2020-02-14properly close http responseChris Lu1-7/+12
2020-02-10Revert "HEAD operation changes to fasthttp"Chris Lu1-17/+8
This reverts commit 58f126fd27bb2f366f76f42223b93ba3b31a0bd8.
2020-02-10HEAD operation changes to fasthttpChris Lu1-8/+17
2020-01-31s3: add s3 copyChris Lu1-0/+21
fix https://github.com/chrislusf/seaweedfs/issues/1190
2020-01-22fix http range requestsChris Lu1-2/+2
2020-01-08skip error when draining readerChris Lu1-2/+5
fix https://github.com/chrislusf/seaweedfs/issues/1179
2019-12-25ignore draining errorChris Lu1-1/+1
2019-12-26feat: drains http body if buffer is too smalldivinerapier1-1/+9
Signed-off-by: divinerapier <poriter.coco@gmail.com>
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-21 06:50:27 +0000