aboutsummaryrefslogtreecommitdiff
path: root/weed/server/filer_server.go
AgeCommit message (Collapse)AuthorFilesLines
2022-03-06Set default leveldb2 enabledKonstantin Lebedev1-2/+2
avoid Filer store is enabled for both leveldb2 and mysql
2022-01-02Merge branch 'master' into metadata_follow_with_client_idChris Lu1-1/+12
2022-01-01Merge pull request #2543 from skurfuerst/seaweedfs-158Chris Lu1-1/+12
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
2021-12-30FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 ClientSebastian Kurfuerst1-1/+12
- one JWT for reading and one for writing, analogous to how the JWT between Master and Volume Server works - I did not implement IP `whiteList` parameter on the filer Additionally, because http_util.DownloadFile now sets the JWT, the `download` command should now work when `jwt.signing.read` is configured. By looking at the code, I think this case did not work before. ## Docs to be adjusted after a release Page `Amazon-S3-API`: ``` # Authentication with Filer You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as explained in [Security-Configuration](Security-Configuration) - controlled by the `grpc.*` configuration in `security.toml`. Starting with version XX, it is also possible to authenticate the HTTP operations between the S3-API-Proxy and the Filer (especially uploading new files). This is configured by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. With both configurations (gRPC and JWT), it is possible to have Filer and S3 communicate in fully authenticated fashion; so Filer will reject any unauthenticated communication. ``` Page `Security Overview`: ``` The following items are not covered, yet: - master server http REST services Starting with version XX, the Filer HTTP REST services can be secured with a JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. ... Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer. Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).** ... # Securing Filer HTTP with JWT To enable JWT-based access control for the Filer, 1. generate `security.toml` file by `weed scaffold -config=security` 2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string 3. copy the same `security.toml` file to the filers and all S3 proxies. If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`. If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`. The S3 API Gateway reads the above JWT keys and sends authenticated HTTP requests to the filer. ``` Page `Security Configuration`: ``` (update scaffold file) ... [filer_jwt.signing] key = "blahblahblahblah" [filer_jwt.signing.read] key = "blahblahblahblah" ``` Resolves: #158
2021-12-30add client id for all metadata listening clientschrislu1-0/+5
2021-12-26use streaming mode for long poll grpc callschrislu1-1/+1
streaming mode would create separate grpc connections for each call. this is to ensure the long poll connections are properly closed.
2021-12-22filer server: allow upload file to specific dataNodebanjiaojuhao1-0/+1
2021-12-05upgrade protoc to 3.17.3chrislu1-0/+2
$brew install protobuf $ protoc --version libprotoc 3.17.3 $ go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.26 $ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.1
2021-11-06filer: deprecate "-peers" optionChris Lu1-1/+0
2021-11-06auto updated filer peer listChris Lu1-2/+2
2021-11-05Revert "master: rename grpc function KeepConnected() to ↵Chris Lu1-1/+1
SubscribeVolumeLocationUpdates()" This reverts commit af71ae11aa29350a60ed7d3b9a16276a06ba9dcc.
2021-11-03master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()Chris Lu1-1/+1
2021-10-04add redis3Chris Lu1-0/+1
2021-09-12change server address from string to a typeChris Lu1-14/+6
2021-09-07handle ipv6 addressesChris Lu1-1/+1
2021-07-28add ReadRemote(), add read remote setup when filer startsChris Lu1-0/+2
2021-07-01go fmtChris Lu1-1/+1
2021-05-23filer: Support sqlite as filer meta storeChris Lu1-0/+1
2021-05-22s3: deprecating filer.options.buckets_fsyncChris Lu1-1/+1
2021-04-01dedicated upload processorChris Lu1-1/+1
avoid thundering effect of overloading volume servers
2021-03-30filer, volume: add concurrent upload size limit to avoid OOMChris Lu1-19/+24
add some back pressure when writes are slow
2021-03-02support IPv6Chris Lu1-1/+1
2021-01-19filer store: add postgres2Chris Lu1-0/+1
2021-01-19filer store: add mysql2Chris Lu1-0/+1
2021-01-12go fmtChris Lu1-12/+12
2021-01-12filer: add leveldb3Chris Lu1-0/+1
support dedicated leveldb instance for each bucket
2021-01-10filer: change to saveToFilerLimit from cacheToFilerLimitChris Lu1-12/+12
short circuit saving small files to volume server
2020-12-23hook upChris Lu1-0/+1
2020-12-22filer: add -defaultStoreDir so that filer.toml can be skippedChris Lu1-0/+2
fix https://github.com/chrislusf/seaweedfs/issues/1659
2020-12-04filer: add option to cache small files to filer storeChris Lu1-0/+1
2020-11-27filer: readonly handle static resourcesChris Lu1-0/+1
fix https://github.com/chrislusf/seaweedfs/issues/1641
2020-11-16add commentChris Lu1-0/+1
2020-11-16deprecating "filer.options.buckets_fsync" in filer.tomlChris Lu1-0/+1
2020-11-15filer: load filer conf when startingChris Lu1-0/+2
2020-11-12fiil serverUrls sorted by data centerKonstantin Lebedev1-1/+1
2020-10-20filer: support default rackChris Lu1-1/+3
fix https://github.com/chrislusf/seaweedfs/issues/1546
2020-09-30filer: replication follows master config if not specifiedChris Lu1-18/+15
2020-09-24adjust for metrics portChris Lu1-3/+1
2020-09-24add start metrics serverKonstantin Lebedev1-0/+2
2020-09-20adjust loggingChris Lu1-5/+0
2020-09-19volume: get metrics configuration from masterChris Lu1-0/+3
fix https://github.com/chrislusf/seaweedfs/issues/1354
2020-09-17refactoringChris Lu1-9/+11
2020-09-16simplify metrics configuration logicChris Lu1-4/+1
2020-09-03change filer2 to filer.ruitao.liu1-1/+1
2020-09-03new filer option to es v7.ruitao.liu1-0/+1
2020-09-01rename filer2 to filerChris Lu1-12/+12
2020-07-12refactoringChris Lu1-11/+1
2020-07-07filer: default filer store directoryChris Lu1-10/+11
1. set default filer store directory 2. set peers, avoiding empty string counted as 1.
2020-07-06fix compilationChris Lu1-1/+1
2020-07-06filer: support cross filer meta data sync if sharing the same storeChris Lu1-1/+10
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-21 02:29:03 +0000