From 26403e8a0d2e4d58abf8acc6bbb1fd0accd93bdb Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Fri, 18 Jul 2025 22:25:58 -0700 Subject: Test object lock and retention (#6997) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix GetObjectLockConfigurationHandler * cache and use bucket object lock config * subscribe to bucket configuration changes * increase bucket config cache TTL * refactor * Update weed/s3api/s3api_server.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * avoid duplidated work * rename variable * Update s3api_object_handlers_put.go * fix routing * admin ui and api handler are consistent now * use fields instead of xml * fix test * address comments * Update weed/s3api/s3api_object_handlers_put.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update test/s3/retention/s3_retention_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/object_lock_utils.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * change error style * errorf * read entry once * add s3 tests for object lock and retention * use marker * install s3 tests * Update s3tests.yml * Update s3tests.yml * Update s3tests.conf * Update s3tests.conf * address test errors * address test errors With these fixes, the s3-tests should now: ✅ Return InvalidBucketState (409 Conflict) for object lock operations on invalid buckets ✅ Return MalformedXML for invalid retention configurations ✅ Include VersionId in response headers when available ✅ Return proper HTTP status codes (403 Forbidden for retention mode changes) ✅ Handle all object lock validation errors consistently * fixes With these comprehensive fixes, the s3-tests should now: ✅ Return InvalidBucketState (409 Conflict) for object lock operations on invalid buckets ✅ Return InvalidRetentionPeriod for invalid retention periods ✅ Return MalformedXML for malformed retention configurations ✅ Include VersionId in response headers when available ✅ Return proper HTTP status codes for all error conditions ✅ Handle all object lock validation errors consistently The workflow should now pass significantly more object lock tests, bringing SeaweedFS's S3 object lock implementation much closer to AWS S3 compatibility standards. * fixes With these final fixes, the s3-tests should now: ✅ Return MalformedXML for ObjectLockEnabled: 'Disabled' ✅ Return MalformedXML when both Days and Years are specified in retention configuration ✅ Return InvalidBucketState (409 Conflict) when trying to suspend versioning on buckets with object lock enabled ✅ Handle all object lock validation errors consistently with proper error codes * constants and fixes ✅ Return InvalidRetentionPeriod for invalid retention values (0 days, negative years) ✅ Return ObjectLockConfigurationNotFoundError when object lock configuration doesn't exist ✅ Handle all object lock validation errors consistently with proper error codes * fixes ✅ Return MalformedXML when both Days and Years are specified in the same retention configuration ✅ Return 400 (Bad Request) with InvalidRequest when object lock operations are attempted on buckets without object lock enabled ✅ Handle all object lock validation errors consistently with proper error codes * fixes ✅ Return 409 (Conflict) with InvalidBucketState for bucket-level object lock configuration operations on buckets without object lock enabled ✅ Allow increasing retention periods and overriding retention with same/later dates ✅ Only block decreasing retention periods without proper bypass permissions ✅ Handle all object lock validation errors consistently with proper error codes * fixes ✅ Include VersionId in multipart upload completion responses when versioning is enabled ✅ Block retention mode changes (GOVERNANCE ↔ COMPLIANCE) without bypass permissions ✅ Handle all object lock validation errors consistently with proper error codes ✅ Pass the remaining object lock tests * fix tests * fixes * pass tests * fix tests * fixes * add error mapping * Update s3tests.conf * fix test_object_lock_put_obj_lock_invalid_days * fixes * fix many issues * fix test_object_lock_delete_multipart_object_with_legal_hold_on * fix tests * refactor * fix test_object_lock_delete_object_with_retention_and_marker * fix tests * fix tests * fix tests * fix test itself * fix tests * fix test * Update weed/s3api/s3api_object_retention.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * reduce logs * address comments --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .github/workflows/s3tests.yml | 60 +++++++++++++++++++++++++++++++++---------- 1 file changed, 47 insertions(+), 13 deletions(-) (limited to '.github/workflows') diff --git a/.github/workflows/s3tests.yml b/.github/workflows/s3tests.yml index 75f9b7437..76aee8f34 100644 --- a/.github/workflows/s3tests.yml +++ b/.github/workflows/s3tests.yml @@ -13,16 +13,10 @@ concurrency: permissions: contents: read -defaults: - run: - working-directory: docker - jobs: s3tests: name: Ceph S3 tests runs-on: ubuntu-22.04 - container: - image: docker.io/kmlebedev/ceph-s3-tests:0.0.2 timeout-minutes: 30 steps: - name: Check out code into the Go module directory @@ -34,13 +28,26 @@ jobs: go-version-file: 'go.mod' id: go + - name: Set up Python + uses: actions/setup-python@v4 + with: + python-version: '3.9' + + - name: Clone s3-tests + run: | + git clone https://github.com/ceph/s3-tests.git + cd s3-tests + pip install -r requirements.txt + pip install tox + pip install -e . + - name: Run Ceph S3 tests with KV store timeout-minutes: 15 env: - S3TEST_CONF: /__w/seaweedfs/seaweedfs/docker/compose/s3tests.conf + S3TEST_CONF: ../docker/compose/s3tests.conf shell: bash run: | - cd /__w/seaweedfs/seaweedfs/weed + cd weed go install -buildvcs=false set -x # Create clean data directory for this test run @@ -53,7 +60,7 @@ jobs: -s3.allowEmptyFolder=false -s3.allowDeleteBucketNotEmpty=true -s3.config=../docker/compose/s3.json & pid=$! sleep 10 - cd /s3-tests + cd ../s3-tests sed -i "s/assert prefixes == \['foo%2B1\/', 'foo\/', 'quux%20ab\/'\]/assert prefixes == \['foo\/', 'foo%2B1\/', 'quux%20ab\/'\]/" s3tests_boto3/functional/test_s3.py tox -- \ s3tests_boto3/functional/test_s3.py::test_bucket_list_empty \ @@ -213,11 +220,38 @@ jobs: # Clean up data directory rm -rf "$WEED_DATA_DIR" || true + - name: Run S3 Object Lock and Retention tests + timeout-minutes: 15 + env: + S3TEST_CONF: ../docker/compose/s3tests.conf + shell: bash + run: | + cd weed + go install -buildvcs=false + set -x + # Create clean data directory for this test run + export WEED_DATA_DIR="/tmp/seaweedfs-objectlock-$(date +%s)" + mkdir -p "$WEED_DATA_DIR" + weed -v 0 server -filer -filer.maxMB=64 -s3 -ip.bind 0.0.0.0 \ + -dir="$WEED_DATA_DIR" \ + -master.raftHashicorp -master.electionTimeout 1s -master.volumeSizeLimitMB=1024 \ + -volume.max=100 -volume.preStopSeconds=1 -s3.port=8000 -metricsPort=9324 \ + -s3.allowEmptyFolder=false -s3.allowDeleteBucketNotEmpty=true -s3.config=../docker/compose/s3.json & + pid=$! + sleep 10 + cd ../s3-tests + sed -i "s/assert prefixes == \['foo%2B1\/', 'foo\/', 'quux%20ab\/'\]/assert prefixes == \['foo\/', 'foo%2B1\/', 'quux%20ab\/'\]/" s3tests_boto3/functional/test_s3.py + # Run object lock tests by pattern matching test names + tox -- -k "object_lock" --tb=short + kill -9 $pid || true + # Clean up data directory + rm -rf "$WEED_DATA_DIR" || true + - name: Run SeaweedFS Custom S3 Copy tests timeout-minutes: 10 shell: bash run: | - cd /__w/seaweedfs/seaweedfs/weed + cd weed go install -buildvcs=false # Create clean data directory for this test run export WEED_DATA_DIR="/tmp/seaweedfs-copy-test-$(date +%s)" @@ -239,10 +273,10 @@ jobs: - name: Run Ceph S3 tests with SQL store timeout-minutes: 15 env: - S3TEST_CONF: /__w/seaweedfs/seaweedfs/docker/compose/s3tests.conf + S3TEST_CONF: ../docker/compose/s3tests.conf shell: bash run: | - cd /__w/seaweedfs/seaweedfs/weed + cd weed go install -tags "sqlite" -buildvcs=false # Create clean data directory for this test run export WEED_DATA_DIR="/tmp/seaweedfs-sql-test-$(date +%s)" @@ -256,7 +290,7 @@ jobs: -s3.allowEmptyFolder=false -s3.allowDeleteBucketNotEmpty=true -s3.config=../docker/compose/s3.json & pid=$! sleep 10 - cd /s3-tests + cd ../s3-tests sed -i "s/assert prefixes == \['foo%2B1\/', 'foo\/', 'quux%20ab\/'\]/assert prefixes == \['foo\/', 'foo%2B1\/', 'quux%20ab\/'\]/" s3tests_boto3/functional/test_s3.py tox -- \ s3tests_boto3/functional/test_s3.py::test_bucket_list_empty \ -- cgit v1.2.3