aboutsummaryrefslogtreecommitdiff
path: root/weed/s3api
AgeCommit message (Collapse)AuthorFilesLines
2025-07-18address test errorschrislu3-24/+28
With these fixes, the s3-tests should now: ✅ Return InvalidBucketState (409 Conflict) for object lock operations on invalid buckets ✅ Return MalformedXML for invalid retention configurations ✅ Include VersionId in response headers when available ✅ Return proper HTTP status codes (403 Forbidden for retention mode changes) ✅ Handle all object lock validation errors consistently
2025-07-18address test errorschrislu4-5/+23
2025-07-18read entry oncechrislu1-26/+58
2025-07-18errorfchrislu3-3/+3
2025-07-18change error stylechrislu2-3/+9
2025-07-18Update weed/s3api/object_lock_utils.goChris Lu1-3/+3
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-18Update weed/s3api/s3api_object_handlers_put.goChris Lu1-1/+1
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-18address commentschrislu5-33/+10
2025-07-18use fields instead of xmlchrislu4-57/+72
2025-07-18admin ui and api handler are consistent nowchrislu1-0/+228
2025-07-18fix routingchrislu1-3/+3
2025-07-18Update s3api_object_handlers_put.gochrislu1-0/+1
2025-07-18rename variablechrislu1-2/+5
2025-07-18avoid duplidated workchrislu1-15/+4
2025-07-18Update weed/s3api/s3api_server.goChris Lu1-1/+1
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-07-18refactorchrislu3-235/+258
2025-07-18increase bucket config cache TTLchrislu2-3/+7
2025-07-18subscribe to bucket configuration changeschrislu1-3/+73
2025-07-18cache and use bucket object lock configchrislu4-62/+160
2025-07-17fix GetObjectLockConfigurationHandlerchrislu2-6/+109
2025-07-16convert error fromating to %w everywhere (#6995)Chris Lu12-45/+45
2025-07-16Object locking need to persist the tags and set the headers (#6994)Chris Lu5-21/+874
* fix object locking read and write No logic to include object lock metadata in HEAD/GET response headers No logic to extract object lock metadata from PUT request headers * add tests for object locking * Update weed/s3api/s3api_object_handlers_put.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * refactor * add unit tests * sync versions * Update s3_worm_integration_test.go * fix legal hold values * lint * fix tests * racing condition when enable versioning * fix tests * validate put object lock header * allow check lock permissions for PUT * default to OFF legal hold * only set object lock headers for objects that are actually from object lock-enabled buckets fix --- FAIL: TestAddObjectLockHeadersToResponse/Handle_entry_with_no_object_lock_metadata (0.00s) * address comments * fix tests * purge * fix * refactoring * address comment * address comment * Update weed/s3api/s3api_object_handlers_put.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers_put.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * avoid nil * ensure locked objects cannot be overwritten --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-16unusedchrislu1-23/+0
2025-07-15S3 Object Lock: ensure x-amz-bucket-object-lock-enabled header (#6990)Chris Lu3-0/+49
* ensure x-amz-bucket-object-lock-enabled header * fix tests * combine 2 metadata changes into one * address comments * Update s3api_bucket_handlers.go * Update weed/s3api/s3api_bucket_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update test/s3/retention/object_lock_reproduce_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update test/s3/retention/object_lock_validation_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update test/s3/retention/s3_bucket_object_lock_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_bucket_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_bucket_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update test/s3/retention/s3_bucket_object_lock_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_bucket_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * package name --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-15rewrite, simplify, avoid unused functions (#6989)Chris Lu4-923/+501
* adding cors support * address some comments * optimize matchesWildcard * address comments * fix for tests * address comments * address comments * address comments * path building * refactor * Update weed/s3api/s3api_bucket_config.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * address comment Service-level responses need both Access-Control-Allow-Methods and Access-Control-Allow-Headers. After setting Access-Control-Allow-Origin and Access-Control-Expose-Headers, also set Access-Control-Allow-Methods: * and Access-Control-Allow-Headers: * so service endpoints satisfy CORS preflight requirements. * Update weed/s3api/s3api_bucket_config.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix * refactor * Update weed/s3api/s3api_bucket_config.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_server.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * simplify * add cors tests * fix tests * fix tests * remove unused functions * fix tests * simplify * address comments * fix * Update weed/s3api/auth_signature_v4.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * rename variable * Revert "Apply suggestion from @Copilot" This reverts commit fce2d4e57e6f712672e62e8c63468c6b89878c6c. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-15adding cors support (#6987)Chris Lu9-51/+1701
* adding cors support * address some comments * optimize matchesWildcard * address comments * fix for tests * address comments * address comments * address comments * path building * refactor * Update weed/s3api/s3api_bucket_config.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * address comment Service-level responses need both Access-Control-Allow-Methods and Access-Control-Allow-Headers. After setting Access-Control-Allow-Origin and Access-Control-Expose-Headers, also set Access-Control-Allow-Methods: * and Access-Control-Allow-Headers: * so service endpoints satisfy CORS preflight requirements. * Update weed/s3api/s3api_bucket_config.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix * refactor * Update weed/s3api/s3api_bucket_config.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_server.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * simplify * add cors tests * fix tests * fix tests --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-14ensure bucket existschrislu1-0/+36
related to https://github.com/seaweedfs/seaweedfs/issues/6971
2025-07-13admin ui adds object lock permissionschrislu2-9/+77
2025-07-13Add policy engine (#6970)Chris Lu21-44/+5449
2025-07-12implement PubObjectRetention and WORM (#6969)Chris Lu9-41/+1786
* implement PubObjectRetention and WORM * Update s3_worm_integration_test.go * avoid previous buckets * Update s3-versioning-tests.yml * address comments * address comments * rename to ExtObjectLockModeKey * only checkObjectLockPermissions if versioningEnabled * address comments * comments * Revert "comments" This reverts commit 6736434176f86c6e222b867777324b17c2de716f. * Update s3api_object_handlers_skip.go * Update s3api_object_retention_test.go * add version id to ObjectIdentifier * address comments * add comments * Add proper error logging for timestamp parsing failures * address comments * add version id to the error * Update weed/s3api/s3api_object_retention_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_retention.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * constants * fix comments * address comments * address comment * refactor out handleObjectLockAvailabilityCheck * errors.Is ErrBucketNotFound * better error checking * address comments --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-11More efficient copy object (#6665)Chris Lu3-39/+601
* it compiles * refactored * reduce to 4 concurrent chunk upload * CopyObjectPartHandler * copy a range of the chunk data, fix offset size in copied chunks * Update s3api_object_handlers_copy.go What the PR Accomplishes: CopyObjectHandler - Now copies entire objects by copying chunks individually instead of downloading/uploading the entire file CopyObjectPartHandler - Handles copying parts of objects for multipart uploads by copying only the relevant chunk portions Efficient Chunk Copying - Uses direct chunk-to-chunk copying with proper volume assignment and concurrent processing (limited to 4 concurrent operations) Range Support - Properly handles range-based copying for partial object copies * fix compilation * fix part destination * handling small objects * use mkFile * copy to existing file or part * add testing tools * adjust tests * fix chunk lookup * refactoring * fix TestObjectCopyRetainingMetadata * ensure bucket name not conflicting * fix conditional copying tests * remove debug messages * add custom s3 copy tests
2025-07-10refactor: Performance and readability improvement on isDefaultPort (#6960)Joon Young Baik1-3/+8
2025-07-09S3: add object versioning (#6945)Chris Lu10-78/+1171
* add object versioning * add missing file * Update weed/s3api/s3api_object_versioning.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_versioning.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_versioning.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * ListObjectVersionsResult is better to show multiple version entries * fix test * Update weed/s3api/s3api_object_handlers_put.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_versioning.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * multiple improvements * move PutBucketVersioningHandler into weed/s3api/s3api_bucket_handlers.go file * duplicated code for reading bucket config, versioningEnabled, etc. try to use functions * opportunity to cache bucket config * error handling if bucket is not found * in case bucket is not found * fix build * add object versioning tests * remove non-existent tests * add tests * add versioning tests * skip a new test * ensure .versions directory exists before saving info into it * fix creating version entry * logging on creating version directory * Update s3api_object_versioning_test.go * retry and wait for directory creation * revert add more logging * Update s3api_object_versioning.go * more debug messages * clean up logs, and touch directory correctly * log the .versions creation and then parent directory listing * use mkFile instead of touch touch is for update * clean up data * add versioning test in go * change location * if modified, latest version is moved to .versions directory, and create a new latest version Core versioning functionality: WORKING TestVersioningBasicWorkflow - PASS TestVersioningDeleteMarkers - PASS TestVersioningMultipleVersionsSameObject - PASS TestVersioningDeleteAndRecreate - PASS TestVersioningListWithPagination - PASS ❌ Some advanced features still failing: ETag calculation issues (using mtime instead of proper MD5) Specific version retrieval (EOF error) Version deletion (internal errors) Concurrent operations (race conditions) * calculate multi chunk md5 Test Results - All Passing: ✅ TestBucketListReturnDataVersioning - PASS ✅ TestVersioningCreateObjectsInOrder - PASS ✅ TestVersioningBasicWorkflow - PASS ✅ TestVersioningMultipleVersionsSameObject - PASS ✅ TestVersioningDeleteMarkers - PASS * dedupe * fix TestVersioningErrorCases * fix eof error of reading old versions * get specific version also check current version * enable integration tests for versioning * trigger action to work for now * Fix GitHub Actions S3 versioning tests workflow - Fix syntax error (incorrect indentation) - Update directory paths from weed/s3api/versioning_tests/ to test/s3/versioning/ - Add push trigger for add-object-versioning branch to enable CI during development - Update artifact paths to match correct directory structure * Improve CI robustness for S3 versioning tests Makefile improvements: - Increase server startup timeout from 30s to 90s for CI environments - Add progressive timeout reporting (logs at 30s, full logs at 90s) - Better error handling with server logs on failure - Add server PID tracking for debugging - Improved test failure reporting GitHub Actions workflow improvements: - Increase job timeouts to account for CI environment delays - Add system information logging (memory, disk space) - Add detailed failure reporting with server logs - Add process and network diagnostics on failure - Better error messaging and log collection These changes should resolve the 'Server failed to start within 30 seconds' issue that was causing the CI tests to fail. * adjust testing volume size * Update Makefile * Update Makefile * Update Makefile * Update Makefile * Update s3-versioning-tests.yml * Update s3api_object_versioning.go * Update Makefile * do not clean up * log received version id * more logs * printout response * print out list version response * use tmp files when put versioned object * change to versions folder layout * Delete weed-test.log * test with mixed versioned and unversioned objects * remove versionDirCache * remove unused functions * remove unused function * remove fallback checking * minor --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-06fix: s3 return BadDigest (#6714)Konstantin Lebedev3-4/+13
* fix: s3 return BadDigest * adjust error message checking --------- Co-authored-by: chrislu <chris.lu@gmail.com>
2025-07-02Add credential storage (#6938)Chris Lu3-30/+80
* add credential store interface * load credential.toml * lint * create credentialManager with explicit store type * add type name * InitializeCredentialManager * remove unused functions * fix missing import * fix import * fix nil configuration
2025-06-19Fix chunked data reading if iam not enabled (#6898)Chris Lu2-1/+194
* fix chunked data reading if iam not enabled * add unit test
2025-06-16Fix url scheme using for forwarded request with changed proto (#6884)Alexey1-1/+7
2025-06-16sort lifecycleschrislu1-1/+10
2025-06-03change version directorychrislu1-2/+2
2025-05-27unlimit the list entries in completeMultipartUpload (#6822)Bruce Zou1-3/+4
2025-05-26Unable to upload empty files to seaweed via. multipart when length is ↵royatwp1-1/+5
unknown (#6729)
2025-05-22added context to filer_client method calls (#6808)Aleksey Kosov5-16/+17
Co-authored-by: akosov <a.kosov@kryptonite.ru>
2025-05-01[s3] refactor duplicate put-object code paths (#6727)Taehyung Lim3-42/+38
2025-04-23[s3] use chunkedReader in PutObjectPartHandler to handle aws-chunked ↵Taehyung Lim1-1/+1
encoding (#6725)
2025-04-23[s3] fix checksum algorithm CRC64NVMe (#6722)Konstantin Lebedev1-2/+2
2025-04-08* Fix s3 auth failed with X-Forwarded-Host and X-Forwarded-Port (#6698)zuzuviewer1-3/+3
2025-03-31chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 (#6679)dependabot[bot]1-1/+1
* chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 Bumps [gocloud.dev](https://github.com/google/go-cloud) from 0.40.0 to 0.41.0. - [Release notes](https://github.com/google/go-cloud/releases) - [Commits](https://github.com/google/go-cloud/compare/v0.40.0...v0.41.0) --- updated-dependencies: - dependency-name: gocloud.dev dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * fix error * fix printing errors * Update go.mod --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: chrislu <chris.lu@gmail.com>
2025-03-21add more logging for s3 signature (#6652)Tom Crasset1-2/+7
2025-03-20Fix s3 auth failed with X-Forwarded-Host and X-Forwarded-Port (#6650)zuzuviewer1-2/+31
2025-03-18skip headers when signing a requestchrislu1-7/+16
fix https://github.com/seaweedfs/seaweedfs/issues/6576#issuecomment-2724577279
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-21 09:50:45 +0000