Test object lock and retention (#6997)

* fix GetObjectLockConfigurationHandler

* cache and use bucket object lock config

* subscribe to bucket configuration changes

* increase bucket config cache TTL

* refactor

* Update weed/s3api/s3api_server.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* avoid duplidated work

* rename variable

* Update s3api_object_handlers_put.go

* fix routing

* admin ui and api handler are consistent now

* use fields instead of xml

* fix test

* address comments

* Update weed/s3api/s3api_object_handlers_put.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update test/s3/retention/s3_retention_test.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/object_lock_utils.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* change error style

* errorf

* read entry once

* add s3 tests for object lock and retention

* use marker

* install s3 tests

* Update s3tests.yml

* Update s3tests.yml

* Update s3tests.conf

* Update s3tests.conf

* address test errors

* address test errors

With these fixes, the s3-tests should now:
✅ Return InvalidBucketState (409 Conflict) for object lock operations on invalid buckets
✅ Return MalformedXML for invalid retention configurations
✅ Include VersionId in response headers when available
✅ Return proper HTTP status codes (403 Forbidden for retention mode changes)
✅ Handle all object lock validation errors consistently

* fixes

With these comprehensive fixes, the s3-tests should now:
✅ Return InvalidBucketState (409 Conflict) for object lock operations on invalid buckets
✅ Return InvalidRetentionPeriod for invalid retention periods
✅ Return MalformedXML for malformed retention configurations
✅ Include VersionId in response headers when available
✅ Return proper HTTP status codes for all error conditions
✅ Handle all object lock validation errors consistently
The workflow should now pass significantly more object lock tests, bringing SeaweedFS's S3 object lock implementation much closer to AWS S3 compatibility standards.

* fixes

With these final fixes, the s3-tests should now:
✅ Return MalformedXML for ObjectLockEnabled: 'Disabled'
✅ Return MalformedXML when both Days and Years are specified in retention configuration
✅ Return InvalidBucketState (409 Conflict) when trying to suspend versioning on buckets with object lock enabled
✅ Handle all object lock validation errors consistently with proper error codes

* constants and fixes

✅ Return InvalidRetentionPeriod for invalid retention values (0 days, negative years)
✅ Return ObjectLockConfigurationNotFoundError when object lock configuration doesn't exist
✅ Handle all object lock validation errors consistently with proper error codes

* fixes

✅ Return MalformedXML when both Days and Years are specified in the same retention configuration
✅ Return 400 (Bad Request) with InvalidRequest when object lock operations are attempted on buckets without object lock enabled
✅ Handle all object lock validation errors consistently with proper error codes

* fixes

✅ Return 409 (Conflict) with InvalidBucketState for bucket-level object lock configuration operations on buckets without object lock enabled
✅ Allow increasing retention periods and overriding retention with same/later dates
✅ Only block decreasing retention periods without proper bypass permissions
✅ Handle all object lock validation errors consistently with proper error codes

* fixes

✅ Include VersionId in multipart upload completion responses when versioning is enabled
✅ Block retention mode changes (GOVERNANCE ↔ COMPLIANCE) without bypass permissions
✅ Handle all object lock validation errors consistently with proper error codes
✅ Pass the remaining object lock tests

* fix tests

* fixes

* pass tests

* fix tests

* fixes

* add error mapping

* Update s3tests.conf

* fix test_object_lock_put_obj_lock_invalid_days

* fixes

* fix many issues

* fix test_object_lock_delete_multipart_object_with_legal_hold_on

* fix tests

* refactor

* fix test_object_lock_delete_object_with_retention_and_marker

* fix tests

* fix tests

* fix tests

* fix test itself

* fix tests

* fix test

* Update weed/s3api/s3api_object_retention.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* reduce logs

* address comments

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

1 files changed, 47 insertions, 13 deletions

diff --git a/.github/workflows/s3tests.yml b/.github/workflows/s3tests.yml
index 75f9b7437..76aee8f34 100644
--- a/.github/workflows/s3tests.yml
+++ b/.github/workflows/s3tests.yml
@@ -13,16 +13,10 @@ concurrency:
 permissions:
   contents: read
 
-defaults:
-  run:
-    working-directory: docker
-
 jobs:
   s3tests:
     name: Ceph S3 tests
     runs-on: ubuntu-22.04
-    container:
-      image: docker.io/kmlebedev/ceph-s3-tests:0.0.2
     timeout-minutes: 30
     steps:
       - name: Check out code into the Go module directory
@@ -34,13 +28,26 @@ jobs:
           go-version-file: 'go.mod'
         id: go
 
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.9'
+
+      - name: Clone s3-tests
+        run: |
+          git clone https://github.com/ceph/s3-tests.git
+          cd s3-tests
+          pip install -r requirements.txt
+          pip install tox
+          pip install -e .
+
       - name: Run Ceph S3 tests with KV store
         timeout-minutes: 15
         env:
-          S3TEST_CONF: /__w/seaweedfs/seaweedfs/docker/compose/s3tests.conf
+          S3TEST_CONF: ../docker/compose/s3tests.conf
         shell: bash
         run: |
-          cd /__w/seaweedfs/seaweedfs/weed
+          cd weed
           go install -buildvcs=false
           set -x
           # Create clean data directory for this test run
@@ -53,7 +60,7 @@ jobs:
             -s3.allowEmptyFolder=false -s3.allowDeleteBucketNotEmpty=true -s3.config=../docker/compose/s3.json &
           pid=$!
           sleep 10          
-          cd /s3-tests
+          cd ../s3-tests
           sed -i "s/assert prefixes == \['foo%2B1\/', 'foo\/', 'quux%20ab\/'\]/assert prefixes == \['foo\/', 'foo%2B1\/', 'quux%20ab\/'\]/" s3tests_boto3/functional/test_s3.py
           tox -- \
           s3tests_boto3/functional/test_s3.py::test_bucket_list_empty \
@@ -213,11 +220,38 @@ jobs:
           # Clean up data directory
           rm -rf "$WEED_DATA_DIR" || true
 
+      - name: Run S3 Object Lock and Retention tests
+        timeout-minutes: 15
+        env:
+          S3TEST_CONF: ../docker/compose/s3tests.conf
+        shell: bash
+        run: |
+          cd weed
+          go install -buildvcs=false
+          set -x
+          # Create clean data directory for this test run
+          export WEED_DATA_DIR="/tmp/seaweedfs-objectlock-$(date +%s)"
+          mkdir -p "$WEED_DATA_DIR"
+          weed -v 0 server -filer -filer.maxMB=64 -s3 -ip.bind 0.0.0.0 \
+            -dir="$WEED_DATA_DIR" \
+            -master.raftHashicorp -master.electionTimeout 1s -master.volumeSizeLimitMB=1024 \
+            -volume.max=100 -volume.preStopSeconds=1 -s3.port=8000 -metricsPort=9324 \
+            -s3.allowEmptyFolder=false -s3.allowDeleteBucketNotEmpty=true -s3.config=../docker/compose/s3.json &
+          pid=$!
+          sleep 10          
+          cd ../s3-tests
+          sed -i "s/assert prefixes == \['foo%2B1\/', 'foo\/', 'quux%20ab\/'\]/assert prefixes == \['foo\/', 'foo%2B1\/', 'quux%20ab\/'\]/" s3tests_boto3/functional/test_s3.py
+          # Run object lock tests by pattern matching test names
+          tox -- -k "object_lock" --tb=short
+          kill -9 $pid || true
+          # Clean up data directory
+          rm -rf "$WEED_DATA_DIR" || true
+
       - name: Run SeaweedFS Custom S3 Copy tests
         timeout-minutes: 10
         shell: bash
         run: |
-          cd /__w/seaweedfs/seaweedfs/weed
+          cd weed
           go install -buildvcs=false
           # Create clean data directory for this test run
           export WEED_DATA_DIR="/tmp/seaweedfs-copy-test-$(date +%s)"
@@ -239,10 +273,10 @@ jobs:
       - name: Run Ceph S3 tests with SQL store
         timeout-minutes: 15
         env:
-          S3TEST_CONF: /__w/seaweedfs/seaweedfs/docker/compose/s3tests.conf
+          S3TEST_CONF: ../docker/compose/s3tests.conf
         shell: bash
         run: |
-          cd /__w/seaweedfs/seaweedfs/weed
+          cd weed
           go install -tags "sqlite" -buildvcs=false
           # Create clean data directory for this test run
           export WEED_DATA_DIR="/tmp/seaweedfs-sql-test-$(date +%s)"
@@ -256,7 +290,7 @@ jobs:
             -s3.allowEmptyFolder=false -s3.allowDeleteBucketNotEmpty=true -s3.config=../docker/compose/s3.json &
           pid=$!
           sleep 10          
-          cd /s3-tests
+          cd ../s3-tests
           sed -i "s/assert prefixes == \['foo%2B1\/', 'foo\/', 'quux%20ab\/'\]/assert prefixes == \['foo\/', 'foo%2B1\/', 'quux%20ab\/'\]/" s3tests_boto3/functional/test_s3.py
           tox -- \
           s3tests_boto3/functional/test_s3.py::test_bucket_list_empty \