aboutsummaryrefslogtreecommitdiff
path: root/weed/command/scaffold
AgeCommit message (Collapse)AuthorFilesLines
2024-08-08fix invalid tomlchrislu1-1/+1
2024-07-16Added tls for http clients (#5766)vadimartynov1-2/+6
* Added global http client * Added Do func for global http client * Changed the code to use the global http client * Fix http client in volume uploader * Fixed pkg name * Fixed http util funcs * Fixed http client for bench_filer_upload * Fixed http client for stress_filer_upload * Fixed http client for filer_server_handlers_proxy * Fixed http client for command_fs_merge_volumes * Fixed http client for command_fs_merge_volumes and command_volume_fsck * Fixed http client for s3api_server * Added init global client for main funcs * Rename global_client to client * Changed: - fixed NewHttpClient; - added CheckIsHttpsClientEnabled func - updated security.toml in scaffold * Reduce the visibility of some functions in the util/http/client pkg * Added the loadSecurityConfig function * Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-07-11Added tls for http handlers (#5764)vadimartynov1-2/+9
* Added https handler for filer * Added example for security.toml
2024-05-18Feature/mongodb security (#5602)sb1-0/+1
2024-05-18add filer configuration options for username/password and tls (#5601)sb1-0/+6
2024-03-24Add TLS support, initial etcd connection check, fix key_prefix handling for ↵Nikita Borzykh1-0/+6
etcd backend meta storage (#5403)
2024-02-27Disable filer UI in configuration (#5297)jerebear121-0/+7
* Add filer.ui.enabled configuration property * Add filer.expose_directory_metadata to config * Ammend commit * Remove ShowUI reference * Update all routes that allow directory metadata * Add cmd flag to server.go
2023-12-20Set allowed origins in config (#5109)jerebear121-0/+5
* Add a way to use a JWT in an HTTP only cookie If a JWT is not included in the Authorization header or a query string, attempt to get a JWT from an HTTP only cookie. * Added a way to specify allowed origins header from config * Removed unecessary log * Check list of domains from config or command flag * Handle default wildcard and change name of config value to cors
2023-12-20chore: add dsn for connection to mysql (#5060)Konstantin Lebedev1-0/+3
* chore: add dsn for connection to mysql * add comment * new comment * fix: validate dsn and adapt password
2023-10-12 fix: disallow file name too long when writing a file (#4881)Konstantin Lebedev1-0/+1
* fix: disallow file name too long when writing a file * bool LongerName to MaxFilenameLength --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-05-18Feat: etcd filer store keys should have customizable prefix (#4484)Mesar Hameed1-0/+1
An etcd cluster is not necessarily only dedicated to seaweedfs. This security enhancement adds a customizable key_prefix option to the etcd filer store. This will allow an etcd cluster administrator to limit the seaweedfs etcd user to only read/write a subset of keys under the key_prefix, instead of all keys on the etcd cluster.
2023-05-15Feat: support username/password authentication for etcd filer store s… (#4477)Mesar Hameed1-0/+2
Feat: support username/password authentication for etcd filer store seaweedfs/seaweedfs#4262 Co-authored-by: Mesar Hameed <mesar.hameed@gmail.com>
2023-04-04use UPSERT for postgres style databaseschrislu1-3/+3
2023-02-05adding support for B2 region (#4177)Muhammad Hallaj bin Subery1-0/+1
Co-authored-by: Muhammad Hallaj bin Subery <hallaj@tuta.io>
2023-01-01filer.store.mysql: Use utf8mb4 instead of 3 byte UTF8 (#4094)CommanderRoot1-13/+13
2023-01-01filer.store.mysql: Replace deprecated upsert syntax (#4096)CommanderRoot1-2/+2
2022-09-15docs(command): readability fixes (#3686)Ryan Russell2-2/+2
Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-07-27exclude directories to sync on filerKonstantin Lebedev1-0/+2
2022-07-08filer tikv support tlsduanhongyi1-0/+8
2022-07-07shell: remove unused filer parameter from shell.tomlchrislu1-2/+0
2022-06-26Update tikv client version and add one PC supportyulai.li3-15/+150
2022-05-15s3: add grpc server to accept configuration changeschrislu1-0/+5
2022-05-03add options to scaffoldKonstantin Lebedev1-5/+7
2022-05-03Merge branch 'new_master' into ydbKonstantin Lebedev1-0/+1
# Conflicts: # go.mod # go.sum
2022-05-02ydb-go-sdk move to v3Konstantin Lebedev1-0/+11
2022-04-30s3 backend support customizing storage classchrislu1-0/+1
2022-04-01change user and pass to username and passworda1-2/+2
2022-03-17prefix search, bucket implementedelee1-3/+7
2022-03-17arangodb adapterelee1-0/+6
2022-03-16Add mTLS support for both master and volume http server.Berck Nash1-0/+6
2022-02-15FilerStore: add redis_luabanjiaojuhao1-0/+34
2022-01-18add gocql timeout settingguol-fnst1-0/+2
2022-01-12master: remove hard coded filer settings in master.tomlchrislu1-3/+0
fix https://github.com/chrislusf/seaweedfs/issues/2529
2021-12-31BUGFIX: security.toml contained wrong keysSebastian Kurfuerst1-2/+2
2021-12-30FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 ClientSebastian Kurfuerst1-4/+26
- one JWT for reading and one for writing, analogous to how the JWT between Master and Volume Server works - I did not implement IP `whiteList` parameter on the filer Additionally, because http_util.DownloadFile now sets the JWT, the `download` command should now work when `jwt.signing.read` is configured. By looking at the code, I think this case did not work before. ## Docs to be adjusted after a release Page `Amazon-S3-API`: ``` # Authentication with Filer You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as explained in [Security-Configuration](Security-Configuration) - controlled by the `grpc.*` configuration in `security.toml`. Starting with version XX, it is also possible to authenticate the HTTP operations between the S3-API-Proxy and the Filer (especially uploading new files). This is configured by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. With both configurations (gRPC and JWT), it is possible to have Filer and S3 communicate in fully authenticated fashion; so Filer will reject any unauthenticated communication. ``` Page `Security Overview`: ``` The following items are not covered, yet: - master server http REST services Starting with version XX, the Filer HTTP REST services can be secured with a JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. ... Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer. Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).** ... # Securing Filer HTTP with JWT To enable JWT-based access control for the Filer, 1. generate `security.toml` file by `weed scaffold -config=security` 2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string 3. copy the same `security.toml` file to the filers and all S3 proxies. If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`. If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`. The S3 API Gateway reads the above JWT keys and sends authenticated HTTP requests to the filer. ``` Page `Security Configuration`: ``` (update scaffold file) ... [filer_jwt.signing] key = "blahblahblahblah" [filer_jwt.signing.read] key = "blahblahblahblah" ``` Resolves: #158
2021-12-24BUGFIX: add access.ui setting to scaffolded security.tomlSebastian Kurfürst1-0/+7
... The property is read here: https://github.com/chrislusf/seaweedfs/blob/b70cb3e0b2a75543f0410d4c09f42aa95fcf2ee2/weed/server/volume_server.go#L69
2021-11-29redis3 supports sentinelChris Lu1-0/+8
2021-11-29add s3.clean.uploads -timeAgo=24hChris Lu1-0/+1
2021-11-25fix redis2 sentinel config examplelimd1-1/+1
2021-11-25Merge remote-tracking branch 'origin/master'limd1-3/+3
2021-11-25support redis sentinellimd1-0/+8
2021-11-11filer: fix mysql2 SQL templateChris Lu1-1/+1
2021-11-11filer: fix mysql command to upsertChris Lu1-2/+2
2021-10-10removing tikv to resolve "go mod tidy" problemChris Lu1-8/+0
tikv is causing "go mod tidy" problem. Need to resolve this before adding tikv back. go mod tidy go: finding module for package github.com/coreos/etcd/clientv3/balancer/picker go: finding module for package cloud.google.com/go/kms/apiv1 go: finding module for package github.com/coreos/etcd/clientv3/balancer/resolver/endpoint go: finding module for package google.golang.org/grpc/naming go: finding module for package github.com/coreos/etcd/clientv3/credentials go: finding module for package github.com/coreos/etcd/clientv3/balancer go: finding module for package github.com/d4l3k/messagediff go: found github.com/coreos/etcd/clientv3/balancer in github.com/coreos/etcd v3.3.26+incompatible go: found github.com/coreos/etcd/clientv3/balancer/picker in github.com/coreos/etcd v3.3.26+incompatible go: found github.com/coreos/etcd/clientv3/balancer/resolver/endpoint in github.com/coreos/etcd v3.3.26+incompatible go: found github.com/coreos/etcd/clientv3/credentials in github.com/coreos/etcd v3.3.26+incompatible go: found cloud.google.com/go/kms/apiv1 in cloud.google.com/go/kms v1.0.0 go: found github.com/d4l3k/messagediff in github.com/d4l3k/messagediff v1.2.1 go: finding module for package google.golang.org/grpc/naming github.com/chrislusf/seaweedfs/weed/filer/tikv imports github.com/tikv/client-go/v2/tikv imports go.etcd.io/etcd/clientv3 tested by go.etcd.io/etcd/clientv3.test imports github.com/coreos/etcd/integration imports github.com/coreos/etcd/proxy/grpcproxy imports google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
2021-10-10removing etcd sequencerChris Lu1-4/+1
causing go mod tidy problem. If anyone wants this, please help to resolve this first. github.com/chrislusf/seaweedfs/weed/sequence imports go.etcd.io/etcd/client tested by go.etcd.io/etcd/client.test imports github.com/coreos/etcd/integration imports github.com/coreos/etcd/proxy/grpcproxy imports google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
2021-10-04add redis3Chris Lu1-0/+22
2021-09-08mysql table DDL: minor, not necessary, but a little more strictChris Lu1-2/+2
the directory is already hashed into dirhash. So this change is not strictly required.
2021-08-26Add deleterange_concurrency to filer configuration fileyulai.li1-0/+2
2021-08-26Update filer.toml scaffold to support tikv storeyulai.li1-0/+6
2021-08-01add -force option following #2228Chris Lu1-1/+1
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-21 19:28:29 +0000