aboutsummaryrefslogtreecommitdiff
path: root/weed/s3api
AgeCommit message (Collapse)AuthorFilesLines
2025-07-15rewrite, simplify, avoid unused functions (#6989)Chris Lu4-923/+501
* adding cors support * address some comments * optimize matchesWildcard * address comments * fix for tests * address comments * address comments * address comments * path building * refactor * Update weed/s3api/s3api_bucket_config.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * address comment Service-level responses need both Access-Control-Allow-Methods and Access-Control-Allow-Headers. After setting Access-Control-Allow-Origin and Access-Control-Expose-Headers, also set Access-Control-Allow-Methods: * and Access-Control-Allow-Headers: * so service endpoints satisfy CORS preflight requirements. * Update weed/s3api/s3api_bucket_config.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix * refactor * Update weed/s3api/s3api_bucket_config.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_server.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * simplify * add cors tests * fix tests * fix tests * remove unused functions * fix tests * simplify * address comments * fix * Update weed/s3api/auth_signature_v4.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * rename variable * Revert "Apply suggestion from @Copilot" This reverts commit fce2d4e57e6f712672e62e8c63468c6b89878c6c. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-15adding cors support (#6987)Chris Lu9-51/+1701
* adding cors support * address some comments * optimize matchesWildcard * address comments * fix for tests * address comments * address comments * address comments * path building * refactor * Update weed/s3api/s3api_bucket_config.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * address comment Service-level responses need both Access-Control-Allow-Methods and Access-Control-Allow-Headers. After setting Access-Control-Allow-Origin and Access-Control-Expose-Headers, also set Access-Control-Allow-Methods: * and Access-Control-Allow-Headers: * so service endpoints satisfy CORS preflight requirements. * Update weed/s3api/s3api_bucket_config.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix * refactor * Update weed/s3api/s3api_bucket_config.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_server.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * simplify * add cors tests * fix tests * fix tests --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-14ensure bucket existschrislu1-0/+36
related to https://github.com/seaweedfs/seaweedfs/issues/6971
2025-07-13admin ui adds object lock permissionschrislu2-9/+77
2025-07-13Add policy engine (#6970)Chris Lu21-44/+5449
2025-07-12implement PubObjectRetention and WORM (#6969)Chris Lu9-41/+1786
* implement PubObjectRetention and WORM * Update s3_worm_integration_test.go * avoid previous buckets * Update s3-versioning-tests.yml * address comments * address comments * rename to ExtObjectLockModeKey * only checkObjectLockPermissions if versioningEnabled * address comments * comments * Revert "comments" This reverts commit 6736434176f86c6e222b867777324b17c2de716f. * Update s3api_object_handlers_skip.go * Update s3api_object_retention_test.go * add version id to ObjectIdentifier * address comments * add comments * Add proper error logging for timestamp parsing failures * address comments * add version id to the error * Update weed/s3api/s3api_object_retention_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_retention.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * constants * fix comments * address comments * address comment * refactor out handleObjectLockAvailabilityCheck * errors.Is ErrBucketNotFound * better error checking * address comments --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-11More efficient copy object (#6665)Chris Lu3-39/+601
* it compiles * refactored * reduce to 4 concurrent chunk upload * CopyObjectPartHandler * copy a range of the chunk data, fix offset size in copied chunks * Update s3api_object_handlers_copy.go What the PR Accomplishes: CopyObjectHandler - Now copies entire objects by copying chunks individually instead of downloading/uploading the entire file CopyObjectPartHandler - Handles copying parts of objects for multipart uploads by copying only the relevant chunk portions Efficient Chunk Copying - Uses direct chunk-to-chunk copying with proper volume assignment and concurrent processing (limited to 4 concurrent operations) Range Support - Properly handles range-based copying for partial object copies * fix compilation * fix part destination * handling small objects * use mkFile * copy to existing file or part * add testing tools * adjust tests * fix chunk lookup * refactoring * fix TestObjectCopyRetainingMetadata * ensure bucket name not conflicting * fix conditional copying tests * remove debug messages * add custom s3 copy tests
2025-07-10refactor: Performance and readability improvement on isDefaultPort (#6960)Joon Young Baik1-3/+8
2025-07-09S3: add object versioning (#6945)Chris Lu10-78/+1171
* add object versioning * add missing file * Update weed/s3api/s3api_object_versioning.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_versioning.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_versioning.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * ListObjectVersionsResult is better to show multiple version entries * fix test * Update weed/s3api/s3api_object_handlers_put.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_versioning.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * multiple improvements * move PutBucketVersioningHandler into weed/s3api/s3api_bucket_handlers.go file * duplicated code for reading bucket config, versioningEnabled, etc. try to use functions * opportunity to cache bucket config * error handling if bucket is not found * in case bucket is not found * fix build * add object versioning tests * remove non-existent tests * add tests * add versioning tests * skip a new test * ensure .versions directory exists before saving info into it * fix creating version entry * logging on creating version directory * Update s3api_object_versioning_test.go * retry and wait for directory creation * revert add more logging * Update s3api_object_versioning.go * more debug messages * clean up logs, and touch directory correctly * log the .versions creation and then parent directory listing * use mkFile instead of touch touch is for update * clean up data * add versioning test in go * change location * if modified, latest version is moved to .versions directory, and create a new latest version Core versioning functionality: WORKING TestVersioningBasicWorkflow - PASS TestVersioningDeleteMarkers - PASS TestVersioningMultipleVersionsSameObject - PASS TestVersioningDeleteAndRecreate - PASS TestVersioningListWithPagination - PASS ❌ Some advanced features still failing: ETag calculation issues (using mtime instead of proper MD5) Specific version retrieval (EOF error) Version deletion (internal errors) Concurrent operations (race conditions) * calculate multi chunk md5 Test Results - All Passing: ✅ TestBucketListReturnDataVersioning - PASS ✅ TestVersioningCreateObjectsInOrder - PASS ✅ TestVersioningBasicWorkflow - PASS ✅ TestVersioningMultipleVersionsSameObject - PASS ✅ TestVersioningDeleteMarkers - PASS * dedupe * fix TestVersioningErrorCases * fix eof error of reading old versions * get specific version also check current version * enable integration tests for versioning * trigger action to work for now * Fix GitHub Actions S3 versioning tests workflow - Fix syntax error (incorrect indentation) - Update directory paths from weed/s3api/versioning_tests/ to test/s3/versioning/ - Add push trigger for add-object-versioning branch to enable CI during development - Update artifact paths to match correct directory structure * Improve CI robustness for S3 versioning tests Makefile improvements: - Increase server startup timeout from 30s to 90s for CI environments - Add progressive timeout reporting (logs at 30s, full logs at 90s) - Better error handling with server logs on failure - Add server PID tracking for debugging - Improved test failure reporting GitHub Actions workflow improvements: - Increase job timeouts to account for CI environment delays - Add system information logging (memory, disk space) - Add detailed failure reporting with server logs - Add process and network diagnostics on failure - Better error messaging and log collection These changes should resolve the 'Server failed to start within 30 seconds' issue that was causing the CI tests to fail. * adjust testing volume size * Update Makefile * Update Makefile * Update Makefile * Update Makefile * Update s3-versioning-tests.yml * Update s3api_object_versioning.go * Update Makefile * do not clean up * log received version id * more logs * printout response * print out list version response * use tmp files when put versioned object * change to versions folder layout * Delete weed-test.log * test with mixed versioned and unversioned objects * remove versionDirCache * remove unused functions * remove unused function * remove fallback checking * minor --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-06fix: s3 return BadDigest (#6714)Konstantin Lebedev3-4/+13
* fix: s3 return BadDigest * adjust error message checking --------- Co-authored-by: chrislu <chris.lu@gmail.com>
2025-07-02Add credential storage (#6938)Chris Lu3-30/+80
* add credential store interface * load credential.toml * lint * create credentialManager with explicit store type * add type name * InitializeCredentialManager * remove unused functions * fix missing import * fix import * fix nil configuration
2025-06-19Fix chunked data reading if iam not enabled (#6898)Chris Lu2-1/+194
* fix chunked data reading if iam not enabled * add unit test
2025-06-16Fix url scheme using for forwarded request with changed proto (#6884)Alexey1-1/+7
2025-06-16sort lifecycleschrislu1-1/+10
2025-06-03change version directorychrislu1-2/+2
2025-05-27unlimit the list entries in completeMultipartUpload (#6822)Bruce Zou1-3/+4
2025-05-26Unable to upload empty files to seaweed via. multipart when length is ↵royatwp1-1/+5
unknown (#6729)
2025-05-22added context to filer_client method calls (#6808)Aleksey Kosov5-16/+17
Co-authored-by: akosov <a.kosov@kryptonite.ru>
2025-05-01[s3] refactor duplicate put-object code paths (#6727)Taehyung Lim3-42/+38
2025-04-23[s3] use chunkedReader in PutObjectPartHandler to handle aws-chunked ↵Taehyung Lim1-1/+1
encoding (#6725)
2025-04-23[s3] fix checksum algorithm CRC64NVMe (#6722)Konstantin Lebedev1-2/+2
2025-04-08* Fix s3 auth failed with X-Forwarded-Host and X-Forwarded-Port (#6698)zuzuviewer1-3/+3
2025-03-31chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 (#6679)dependabot[bot]1-1/+1
* chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 Bumps [gocloud.dev](https://github.com/google/go-cloud) from 0.40.0 to 0.41.0. - [Release notes](https://github.com/google/go-cloud/releases) - [Commits](https://github.com/google/go-cloud/compare/v0.40.0...v0.41.0) --- updated-dependencies: - dependency-name: gocloud.dev dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * fix error * fix printing errors * Update go.mod --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: chrislu <chris.lu@gmail.com>
2025-03-21add more logging for s3 signature (#6652)Tom Crasset1-2/+7
2025-03-20Fix s3 auth failed with X-Forwarded-Host and X-Forwarded-Port (#6650)zuzuviewer1-2/+31
2025-03-18skip headers when signing a requestchrislu1-7/+16
fix https://github.com/seaweedfs/seaweedfs/issues/6576#issuecomment-2724577279
2025-03-18fix: restore deletion audit of individual objects (#6644)SmoothDenis2-3/+15
2025-03-07chore: make function comment match function name (#6607)NinaLua1-1/+1
2025-02-12implement s3 streaming-unsigned-payload-trailer (#6539)Tom Crasset4-47/+392
* implement s3 streaming-unsigned-payload-trailer * chore: remove print
2025-02-07omit http and https ports when using X-Forwarded-Port (#6527)Tom Crasset1-1/+1
2025-02-07add s3 signature tests and prepare implementation of ↵Tom Crasset7-8/+208
STREAMING-UNSIGNED-PAYLOAD-TRAILER (#6525) * add tests for s3 signature * add test for newSignV4ChunkedReader.Read() * add glog import
2025-02-07fix: record and delete bucket metrics after inactive (#6523)zouyixiong4-3/+10
Co-authored-by: XYZ <XYZ>
2025-02-06use X-Forwarded-Host and X-Forwarded-Port to determine correct host for AWS ↵Tom Crasset1-6/+21
signature (#6514)
2025-02-03Revert "Add content length to filer's requests (#6489)"chrislu1-1/+0
This reverts commit 9f42ce3f01468317870c0c37f3a0ebf986a504f9.
2025-02-01adjust fixchrislu1-0/+1
fix https://github.com/seaweedfs/seaweedfs/issues/6497
2025-01-29verify if bucket exists in GetBucketLocationHandler (#6485)Tom Crasset1-0/+7
2025-01-29Add content length to filer's requests (#6489)Hadi Zamani1-0/+1
2025-01-25Add metrics for uploaded and deleted s3 objects (#6475)Hadi Zamani3-1/+6
2025-01-17fix S3 per-user-directory Policy (#6443)Tom Crasset2-2/+32
* fix S3 per-user-directory Policy * Delete docker/config.json * add tests * remove logs * undo modifications of weed/shell/command_volume_balance.go * remove modifications of docker-compose * fix failing test --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2025-01-16Add bucket's traffic metrics (#6444)Hadi Zamani3-9/+18
* Add bucket's traffic metrics * Add bucket traffic to dashboards * Fix bucket metrics help messages * Fix variable names
2025-01-16improve iam error handling (#6446)Tom Crasset1-0/+2
* improve iam error handling * Delete docker/test.py
2025-01-15Bugfix s3 audit missing requester for PUT requests (#6434)ftong20201-3/+1
fix s3 audit missing requster for PUT
2024-12-19fix compilationchrislu1-3/+0
2024-12-19"golang.org/x/exp/slices" => "slices" and go fmtchrislu3-2/+4
2024-12-19Fix for DeleteMultipleObjectsHandler wrongly deleting parent folders (#6380)Warren Hodgkinson1-2/+7
What problem are we solving? Fix: #6379 How are we solving the problem? We check for the AllowEmptyFolders option prior to cascade deleting parent folders in S3 DeleteMultipleObjectsHandler. How is the PR tested? We ran SeaweedFS in a Kubernetes Cluster with a joint Filer and S3 server in one container, with leveldb2 as the filer storage, and AllowEmptyFolders set to true. When using the Distribution Registry as the S3 client, it calls the DeleteMultipleObjectsHandler as part of the artifact upload process (uploads to a temp location, then performs a copy and delete). Without this fix, the deletion cascade deleted parent folder until the entire contents of the bucket were gone. With this fix, the existing content of the bucket remained, and the newly uploaded content was added. Checks [ ] I have added unit tests if possible. [ ] I will add related wiki document changes and link to this PR after merging. Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2024-12-12fix: record and delete bucket metrics after inactive (#6349)zouyixiong1-3/+6
2024-11-26change comment for expect header (#6289)steve.wei1-5/+4
* remove case of expect * Set the default value of the Expect header for compatibility.
2024-11-20consistent error message if auth is not setupchrislu1-1/+6
fix https://github.com/seaweedfs/seaweedfs/issues/6262
2024-11-01Update filer_multipart.go (#6188)han1-0/+1
fix the #6177 bug
2024-10-23remove X-Forwarded-Forchrislu1-2/+0
fix https://github.com/seaweedfs/seaweedfs/issues/6136
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-25 05:57:09 +0000