aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)AuthorFilesLines
2025-11-04Nit: use `time.Duration`s instead of constants in seconds. (#7438)Lisandro Pin7-14/+20
Nit: use `time.Durations` instead of constants in seconds. Makes for slightly more readable code.
2025-11-04test s3 put multipartKonstantin Lebedev1-1/+5
2025-11-04test s3 putKonstantin Lebedev3-8/+21
2025-11-04del unusing func removeExpiredObjectKonstantin Lebedev1-8/+0
2025-11-04filer delete meta and dataKonstantin Lebedev2-2/+8
2025-11-04move s3 delete expired entry to filerKonstantin Lebedev4-12/+9
2025-11-04Merge branch 'master' into allow_delete_objects_by_TTLKonstantin Lebedev19-85/+514
2025-11-04Merge remote-tracking branch 'fork/allow_delete_objects_by_TTL' into ↵Konstantin Lebedev2-39/+40
allow_delete_objects_by_TTL
2025-11-04clear TtlSeconds for volumeKonstantin Lebedev1-2/+5
2025-11-03go modchrislu2-39/+40
2025-11-04GetS3ExpireTime on filerKonstantin Lebedev2-6/+22
2025-11-03adjust testchrislu1-7/+17
2025-11-03chore(deps): bump go.mongodb.org/mongo-driver from 1.17.4 to 1.17.6 (#7430)dependabot[bot]2-3/+3
Bumps [go.mongodb.org/mongo-driver](https://github.com/mongodb/mongo-go-driver) from 1.17.4 to 1.17.6. - [Release notes](https://github.com/mongodb/mongo-go-driver/releases) - [Commits](https://github.com/mongodb/mongo-go-driver/compare/v1.17.4...v1.17.6) --- updated-dependencies: - dependency-name: go.mongodb.org/mongo-driver dependency-version: 1.17.6 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2025-11-03chore(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.18.19 to ↵dependabot[bot]2-15/+15
1.18.20 (#7432) chore(deps): bump github.com/aws/aws-sdk-go-v2/credentials Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.18.19 to 1.18.20. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/config/v1.18.20/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.18.19...config/v1.18.20) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/credentials dependency-version: 1.18.20 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-03S3: prevent deleting buckets with object locking (#7434)Chris Lu14-33/+452
* prevent deleting buckets with object locking * addressing comments * Update s3api_bucket_handlers.go * address comments * early return * refactor * simplify * constant * go fmt
2025-11-034.004.00chrislu1-2/+2
2025-11-04resolv coderabbitaiKonstantin Lebedev4-10/+2
2025-11-03chore(deps): bump helm/kind-action from 1.12.0 to 1.13.0 (#7428)dependabot[bot]1-1/+1
Bumps [helm/kind-action](https://github.com/helm/kind-action) from 1.12.0 to 1.13.0. - [Release notes](https://github.com/helm/kind-action/releases) - [Commits](https://github.com/helm/kind-action/compare/v1.12.0...v1.13.0) --- updated-dependencies: - dependency-name: helm/kind-action dependency-version: 1.13.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-03chore(deps): bump cloud.google.com/go/storage from 1.57.0 to 1.57.1 (#7431)dependabot[bot]2-3/+3
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.57.0 to 1.57.1. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/spanner/v1.57.0...storage/v1.57.1) --- updated-dependencies: - dependency-name: cloud.google.com/go/storage dependency-version: 1.57.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-03chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.88.3 to ↵dependabot[bot]2-27/+27
1.89.1 (#7433) chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.88.3 to 1.89.1. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.88.3...service/s3/v1.89.1) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3 dependency-version: 1.89.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-04fix s3testsKonstantin Lebedev6-7/+34
2025-11-034.00chrislu1-1/+1
2025-11-03fix locationPrefix for updateEntriesTTLKonstantin Lebedev1-3/+2
2025-11-03fix IsExpiredKonstantin Lebedev1-1/+1
2025-11-03fix test lifecycle expirationKonstantin Lebedev3-1/+38
2025-11-03fix opt allowDeleteObjectsByTTL for serverKonstantin Lebedev1-1/+1
2025-11-03add lifecycle expiration s3 testsKonstantin Lebedev1-4/+10
2025-11-03delete on get and headKonstantin Lebedev3-7/+29
2025-11-03pass opt allowDeleteObjectsByTTL to all serversKonstantin Lebedev3-0/+3
2025-11-03disable delete expires s3 entry in filerKonstantin Lebedev3-3/+13
2025-11-03do delete expired entries on s3 list requestKonstantin Lebedev3-0/+11
https://github.com/seaweedfs/seaweedfs/issues/6837
2025-11-024.00chrislu1-2/+2
2025-11-02 Rework collection resultion for `ec.rebuild`, in preparation for ↵Lisandro Pin1-9/+9
parallelization. (#7420) * Rework collection resultion for `ec.rebuild`, in preparation for parallelization. See https://github.com/seaweedfs/seaweedfs/issues/7416 . * simplify * Update weed/shell/command_ec_rebuild.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: chrislu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-11-02fix go install (#7425)Chris Lu1-3/+0
fix https://github.com/seaweedfs/seaweedfs/issues/7424
2025-11-01Filer: separate context for streaming (#7423)Chris Lu1-1/+9
* separate context for streaming * Update weed/server/filer_server_handlers_read.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-11-01fix volume utilization icon renderingchrislu2-35/+31
2025-11-01fix: handle 'default' collection filter in cluster volumes pagechrislu2-9/+3
- Update matchesCollection to recognize 'default' as filter for empty collection - Remove incorrect conversion of 'default' to empty string in handlers - Fixes issue where ?collection=default would show all collections instead of just default collection
2025-11-01adjust volume server linkchrislu2-254/+494
2025-10-31S3: load bucket object locking configuration if not found in cache (#7422)Chris Lu4-34/+72
* load bucket object locking configuration if not found in cache * fix cache building, more specific error, add back metrics
2025-10-31S3: fix TestSignedStreamingUploadInvalidSignature test (#7421)Chris Lu2-2/+10
* Added continue statements after all state transitions in the state machine to ensure immediate state processing * simplify * remove redundant continue clause * ensure wrong signature
2025-10-31weed master add peers=none option for faster startup (#7419)Chris Lu2-9/+47
* weed master -peers=none * single master mode only when peers is none * refactoring * revert duplicated code * revert * Update weed/command/master.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * preventing "none" passed to other components if master is not started --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-10-31Adjust cli option (#7418)Chris Lu7-34/+91
* adjust "weed benchmark" CLI to use readOnly/writeOnly * consistently use "-master" CLI option * If both -readOnly and -writeOnly are specified, the current logic silently allows it with -writeOnly taking precedence. This is confusing and could lead to unexpected behavior.
2025-10-31avoid unnecessary fail fastchrislu1-3/+7
fix https://github.com/seaweedfs/seaweedfs/issues/7417
2025-10-31detect ipv6chrislu1-1/+1
2025-10-30Fix masterclient vidmap race condition (#7412)Chris Lu3-34/+133
* fallback to check master * clean up * parsing * refactor * handle parse error * return error * avoid dup lookup * use batch key * dedup lookup logic * address comments * errors.Join(lookupErrors...) * add a comment * Fix: Critical data race in MasterClient vidMap Fixes a critical data race where resetVidMap() was writing to the vidMap pointer while other methods were reading it concurrently without synchronization. Changes: - Removed embedded *vidMap from MasterClient - Added vidMapLock (sync.RWMutex) to protect vidMap pointer access - Created safe accessor methods (GetLocations, GetDataCenter, etc.) - Updated all direct vidMap accesses to use thread-safe methods - Updated resetVidMap() to acquire write lock during pointer swap The vidMap already has internal locking for its operations, but this fix protects the vidMap pointer itself from concurrent read/write races. Verified with: go test -race ./weed/wdclient/... Impact: - Prevents potential panics from concurrent pointer access - No performance impact - uses RWMutex for read-heavy workloads - Maintains backward compatibility through wrapper methods * fmt * Fix: Critical data race in MasterClient vidMap Fixes a critical data race where resetVidMap() was writing to the vidMap pointer while other methods were reading it concurrently without synchronization. Changes: - Removed embedded *vidMap from MasterClient struct - Added vidMapLock (sync.RWMutex) to protect vidMap pointer access - Created minimal public accessor methods for external packages: * GetLocations, GetLocationsClone, GetVidLocations * LookupFileId, LookupVolumeServerUrl * GetDataCenter - Internal code directly locks and accesses vidMap (no extra indirection) - Updated resetVidMap() to acquire write lock during pointer swap - Updated shell/commands.go to use GetDataCenter() method Design Philosophy: - vidMap already has internal locking for its map operations - This fix specifically protects the vidMap *pointer* from concurrent access - Public methods for external callers, direct locking for internal use - Minimizes wrapper overhead while maintaining thread safety Verified with: go test -race ./weed/wdclient/... (passes) Impact: - Prevents potential panics/crashes from data races - Minimal performance impact (RWMutex for read-heavy workload) - Maintains full backward compatibility * fix more concurrent access * reduce lock scope * Optimize vidMap locking for better concurrency Improved locking strategy based on the understanding that: - vidMapLock protects the vidMap pointer from concurrent swaps - vidMap has internal locks that protect its data structures Changes: 1. Read operations: Grab pointer with RLock, release immediately, then operate - Reduces lock hold time - Allows resetVidMap to proceed sooner - Methods: GetLocations, GetLocationsClone, GetVidLocations, LookupVolumeServerUrl, GetDataCenter 2. Write operations: Changed from Lock() to RLock() - RLock prevents pointer swap during operation - Allows concurrent readers and other writers (serialized by vidMap's lock) - Methods: addLocation, deleteLocation, addEcLocation, deleteEcLocation Benefits: - Significantly reduced lock contention - Better concurrent performance under load - Still prevents all race conditions Verified with: go test -race ./weed/wdclient/... (passes) * Further reduce lock contention in LookupVolumeIdsWithFallback Optimized two loops that were holding RLock for extended periods: Before: - Held RLock during entire loop iteration - Included string parsing and cache lookups - Could block resetVidMap for significant time with large batches After: - Grab vidMap pointer with brief RLock - Release lock immediately - Perform all loop operations on local pointer Impact: - First loop: Cache check on initial volumeIds - Second loop: Double-check after singleflight wait Benefits: - Minimal lock hold time (just pointer copy) - resetVidMap no longer blocked by long loops - Better concurrent performance with large volume ID lists - Still thread-safe (vidMap methods have internal locks) Verified with: go test -race ./weed/wdclient/... (passes) * Add clarifying comments to vidMap helper functions Added inline documentation to each helper function (addLocation, deleteLocation, addEcLocation, deleteEcLocation) explaining the two-level locking strategy: - RLock on vidMapLock prevents resetVidMap from swapping the pointer - vidMap has internal locks that protect the actual map mutations - This design provides optimal concurrency The comments make it clear why RLock (not Lock) is correct and intentional, preventing future confusion about the locking strategy. * Improve encapsulation: Add shallowClone() method to vidMap Added a shallowClone() method to vidMap to improve encapsulation and prevent MasterClient from directly accessing vidMap's internal fields. Changes: 1. Added vidMap.shallowClone() in vid_map.go - Encapsulates the shallow copy logic within vidMap - Makes vidMap responsible for its own state representation - Documented that caller is responsible for thread safety 2. Simplified resetVidMap() in masterclient.go - Uses tail := mc.vidMap.shallowClone() instead of manual field access - Cleaner, more maintainable code - Better adherence to encapsulation principles Benefits: - Improved code organization and maintainability - vidMap internals are now properly encapsulated - Easier to modify vidMap structure in the future - More self-documenting code Verified with: go test -race ./weed/wdclient/... (passes) * Optimize locking: Reduce lock acquisitions and use helper methods Two optimizations to further reduce lock contention and improve code consistency: 1. LookupFileIdWithFallback: Eliminated redundant lock acquisition - Before: Two separate locks to get vidMap and dataCenter - After: Single lock gets both values together - Benefit: 50% reduction in lock/unlock overhead for this hot path 2. KeepConnected: Use GetDataCenter() helper for consistency - Before: Manual lock/unlock to access DataCenter field - After: Use existing GetDataCenter() helper method - Benefit: Better encapsulation and code consistency Impact: - Reduced lock contention in high-traffic lookup path - More consistent use of accessor methods throughout codebase - Cleaner, more maintainable code Verified with: go test -race ./weed/wdclient/... (passes) * Refactor: Extract common locking patterns into helper methods Eliminated code duplication by introducing two helper methods that encapsulate the common locking patterns used throughout MasterClient: 1. getStableVidMap() - For read operations - Acquires lock, gets pointer, releases immediately - Returns stable snapshot for thread-safe reads - Used by: GetLocations, GetLocationsClone, GetVidLocations, LookupFileId, LookupVolumeServerUrl, GetDataCenter 2. withCurrentVidMap(f func(vm *vidMap)) - For write operations - Holds RLock during callback execution - Prevents pointer swap while allowing concurrent operations - Used by: addLocation, deleteLocation, addEcLocation, deleteEcLocation Benefits: - Reduced code duplication (eliminated 48 lines of repetitive locking code) - Centralized locking logic makes it easier to understand and maintain - Self-documenting pattern through named helper methods - Easier to modify locking strategy in the future (single point of change) - Improved readability - accessor methods are now one-liners Code size reduction: ~40% fewer lines for accessor/helper methods Verified with: go test -race ./weed/wdclient/... (passes) * consistent * Fix cache pointer race condition with atomic.Pointer Use atomic.Pointer for vidMap cache field to prevent data races during cache trimming in resetVidMap. This addresses the race condition where concurrent GetLocations calls could read the cache pointer while resetVidMap is modifying it during cache chain trimming. Changes: - Changed cache field from *vidMap to atomic.Pointer[vidMap] - Updated all cache access to use Load() and Store() atomic operations - Updated shallowClone, GetLocations, deleteLocation, deleteEcLocation - Updated resetVidMap to use atomic operations for cache trimming * Merge: Resolve conflict in deleteEcLocation - keep atomic.Pointer and fix bug Resolved merge conflict by combining: 1. Atomic pointer access pattern (from HEAD): cache.Load() 2. Correct method call (from fix): deleteEcLocation (not deleteLocation) Resolution: - Before (HEAD): cachedMap.deleteLocation() - WRONG, reintroduced bug - Before (fix): vc.cache.deleteEcLocation() - RIGHT method, old pattern - After (merged): cachedMap.deleteEcLocation() - RIGHT method, new pattern This preserves both improvements: ✓ Thread-safe atomic.Pointer access pattern ✓ Correct recursive call to deleteEcLocation Verified with: go test -race ./weed/wdclient/... (passes) * Update vid_map.go * remove shallow clone * simplify
2025-10-30Fix IPv6 host header formatting to match AWS SDK behavior (#7414)Chris Lu3-15/+20
* Add nginx reverse proxy documentation for S3 API Fixes #7407 Add comprehensive documentation and example configuration for using nginx as a reverse proxy with SeaweedFS S3 API while maintaining AWS Signature V4 authentication compatibility. Changes: - Add docker/nginx/README.md with detailed setup guide - Add docker/nginx/s3-example.conf with working configuration - Update docker/nginx/proxy.conf with important S3 notes The documentation covers: - Critical requirements for AWS Signature V4 authentication - Common mistakes and why they break S3 authentication - Complete working nginx configurations - Debugging tips and troubleshooting - Performance tuning recommendations * Fix IPv6 host header formatting to match AWS SDK behavior Follow-up to PR #7403 When a default port (80 for HTTP, 443 for HTTPS) is stripped from an IPv6 address, the square brackets should also be removed to match AWS SDK behavior for S3 signature calculation. Reference: https://github.com/aws/aws-sdk-go-v2/blob/main/aws/signer/internal/v4/host.go The AWS SDK's stripPort function explicitly removes brackets when returning an IPv6 address without a port. Changes: - Update extractHostHeader to strip brackets from IPv6 addresses when no port or default port is used - Update test expectations to match AWS SDK behavior - Add detailed comments explaining the AWS SDK compatibility requirement This ensures S3 signature validation works correctly with IPv6 addresses behind reverse proxies, matching AWS S3 canonical request format. Fixes the issue raised in PR #7403 comment: https://github.com/seaweedfs/seaweedfs/pull/7403#issuecomment-3471105438 * Update docker/nginx/README.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Add nginx reverse proxy documentation for S3 API Fixes #7407 Add comprehensive documentation and example configuration for using nginx as a reverse proxy with SeaweedFS S3 API while maintaining AWS Signature V4 authentication compatibility. Changes: - Add docker/nginx/README.md with detailed setup guide - Add docker/nginx/s3-example.conf with working configuration - Update docker/nginx/proxy.conf with important S3 notes The documentation covers: - Critical requirements for AWS Signature V4 authentication - Common mistakes and why they break S3 authentication - Complete working nginx configurations - Debugging tips and troubleshooting - Performance tuning recommendations Fix IPv6 host header formatting to match AWS SDK behavior Follow-up to PR #7403 When a default port (80 for HTTP, 443 for HTTPS) is stripped from an IPv6 address, the square brackets should also be removed to match AWS SDK behavior for S3 signature calculation. Reference: https://github.com/aws/aws-sdk-go-v2/blob/main/aws/signer/internal/v4/host.go The AWS SDK's stripPort function explicitly removes brackets when returning an IPv6 address without a port. Changes: - Update extractHostHeader to strip brackets from IPv6 addresses when no port or default port is used - Update test expectations to match AWS SDK behavior - Add detailed comments explaining the AWS SDK compatibility requirement This ensures S3 signature validation works correctly with IPv6 addresses behind reverse proxies, matching AWS S3 canonical request format. Fixes the issue raised in PR #7403 comment: https://github.com/seaweedfs/seaweedfs/pull/7403#issuecomment-3471105438 * Revert "Merge branch 'fix-ipv6-brackets-default-port' of https://github.com/seaweedfs/seaweedfs into fix-ipv6-brackets-default-port" This reverts commit cca3f3985ff5263698d4be27a919cf52bbc5739f, reversing changes made to 2b8f9de78ebaa285f43f38eec5e0be88a4e56715. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-10-30Filer: fallback to check master (#7411)Chris Lu2-43/+184
* fallback to check master * clean up * parsing * refactor * handle parse error * return error * avoid dup lookup * use batch key * dedup lookup logic * address comments * errors.Join(lookupErrors...) * add a comment
2025-10-30network: Adaptive timeout (#7410)Chris Lu1-4/+67
* server can start when no network for local dev * fixed superfluous response.WriteHeader call" warning * adaptive based on last write time * more doc * refactoring
2025-10-30fixed superfluous response.WriteHeader call" warningchrislu1-3/+2
2025-10-30server can start when no network for local devchrislu1-1/+5
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-21 08:45:22 +0000