Test object lock and retention (#6997)

* fix GetObjectLockConfigurationHandler

* cache and use bucket object lock config

* subscribe to bucket configuration changes

* increase bucket config cache TTL

* refactor

* Update weed/s3api/s3api_server.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* avoid duplidated work

* rename variable

* Update s3api_object_handlers_put.go

* fix routing

* admin ui and api handler are consistent now

* use fields instead of xml

* fix test

* address comments

* Update weed/s3api/s3api_object_handlers_put.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update test/s3/retention/s3_retention_test.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/object_lock_utils.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* change error style

* errorf

* read entry once

* add s3 tests for object lock and retention

* use marker

* install s3 tests

* Update s3tests.yml

* Update s3tests.yml

* Update s3tests.conf

* Update s3tests.conf

* address test errors

* address test errors

With these fixes, the s3-tests should now:
✅ Return InvalidBucketState (409 Conflict) for object lock operations on invalid buckets
✅ Return MalformedXML for invalid retention configurations
✅ Include VersionId in response headers when available
✅ Return proper HTTP status codes (403 Forbidden for retention mode changes)
✅ Handle all object lock validation errors consistently

* fixes

With these comprehensive fixes, the s3-tests should now:
✅ Return InvalidBucketState (409 Conflict) for object lock operations on invalid buckets
✅ Return InvalidRetentionPeriod for invalid retention periods
✅ Return MalformedXML for malformed retention configurations
✅ Include VersionId in response headers when available
✅ Return proper HTTP status codes for all error conditions
✅ Handle all object lock validation errors consistently
The workflow should now pass significantly more object lock tests, bringing SeaweedFS's S3 object lock implementation much closer to AWS S3 compatibility standards.

* fixes

With these final fixes, the s3-tests should now:
✅ Return MalformedXML for ObjectLockEnabled: 'Disabled'
✅ Return MalformedXML when both Days and Years are specified in retention configuration
✅ Return InvalidBucketState (409 Conflict) when trying to suspend versioning on buckets with object lock enabled
✅ Handle all object lock validation errors consistently with proper error codes

* constants and fixes

✅ Return InvalidRetentionPeriod for invalid retention values (0 days, negative years)
✅ Return ObjectLockConfigurationNotFoundError when object lock configuration doesn't exist
✅ Handle all object lock validation errors consistently with proper error codes

* fixes

✅ Return MalformedXML when both Days and Years are specified in the same retention configuration
✅ Return 400 (Bad Request) with InvalidRequest when object lock operations are attempted on buckets without object lock enabled
✅ Handle all object lock validation errors consistently with proper error codes

* fixes

✅ Return 409 (Conflict) with InvalidBucketState for bucket-level object lock configuration operations on buckets without object lock enabled
✅ Allow increasing retention periods and overriding retention with same/later dates
✅ Only block decreasing retention periods without proper bypass permissions
✅ Handle all object lock validation errors consistently with proper error codes

* fixes

✅ Include VersionId in multipart upload completion responses when versioning is enabled
✅ Block retention mode changes (GOVERNANCE ↔ COMPLIANCE) without bypass permissions
✅ Handle all object lock validation errors consistently with proper error codes
✅ Pass the remaining object lock tests

* fix tests

* fixes

* pass tests

* fix tests

* fixes

* add error mapping

* Update s3tests.conf

* fix test_object_lock_put_obj_lock_invalid_days

* fixes

* fix many issues

* fix test_object_lock_delete_multipart_object_with_legal_hold_on

* fix tests

* refactor

* fix test_object_lock_delete_object_with_retention_and_marker

* fix tests

* fix tests

* fix tests

* fix test itself

* fix tests

* fix test

* Update weed/s3api/s3api_object_retention.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* reduce logs

* address comments

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

1 files changed, 100 insertions, 23 deletions

diff --git a/weed/s3api/s3api_object_handlers_put.go b/weed/s3api/s3api_object_handlers_put.go
index 50d308566..011a039d3 100644
--- a/weed/s3api/s3api_object_handlers_put.go
+++ b/weed/s3api/s3api_object_handlers_put.go
@@ -23,16 +23,25 @@ import (
 
 // Object lock validation errors
 var (
-	ErrObjectLockVersioningRequired       = errors.New("object lock headers can only be used on versioned buckets")
-	ErrInvalidObjectLockMode              = errors.New("invalid object lock mode")
-	ErrInvalidLegalHoldStatus             = errors.New("invalid legal hold status")
-	ErrInvalidRetentionDateFormat         = errors.New("invalid retention until date format")
-	ErrRetentionDateMustBeFuture          = errors.New("retention until date must be in the future")
-	ErrObjectLockModeRequiresDate         = errors.New("object lock mode requires retention until date")
-	ErrRetentionDateRequiresMode          = errors.New("retention until date requires object lock mode")
-	ErrGovernanceBypassVersioningRequired = errors.New("governance bypass header can only be used on versioned buckets")
-	ErrInvalidObjectLockDuration          = errors.New("object lock duration must be greater than 0 days")
-	ErrObjectLockDurationExceeded         = errors.New("object lock duration exceeds maximum allowed days")
+	ErrObjectLockVersioningRequired          = errors.New("object lock headers can only be used on versioned buckets")
+	ErrInvalidObjectLockMode                 = errors.New("invalid object lock mode")
+	ErrInvalidLegalHoldStatus                = errors.New("invalid legal hold status")
+	ErrInvalidRetentionDateFormat            = errors.New("invalid retention until date format")
+	ErrRetentionDateMustBeFuture             = errors.New("retain until date must be in the future")
+	ErrObjectLockModeRequiresDate            = errors.New("object lock mode requires retention until date")
+	ErrRetentionDateRequiresMode             = errors.New("retention until date requires object lock mode")
+	ErrGovernanceBypassVersioningRequired    = errors.New("governance bypass header can only be used on versioned buckets")
+	ErrInvalidObjectLockDuration             = errors.New("object lock duration must be greater than 0 days")
+	ErrObjectLockDurationExceeded            = errors.New("object lock duration exceeds maximum allowed days")
+	ErrObjectLockConfigurationMissingEnabled = errors.New("object lock configuration must specify ObjectLockEnabled")
+	ErrInvalidObjectLockEnabledValue         = errors.New("invalid object lock enabled value")
+	ErrRuleMissingDefaultRetention           = errors.New("rule configuration must specify DefaultRetention")
+	ErrDefaultRetentionMissingMode           = errors.New("default retention must specify Mode")
+	ErrInvalidDefaultRetentionMode           = errors.New("invalid default retention mode")
+	ErrDefaultRetentionMissingPeriod         = errors.New("default retention must specify either Days or Years")
+	ErrDefaultRetentionBothDaysAndYears      = errors.New("default retention cannot specify both Days and Years")
+	ErrDefaultRetentionDaysOutOfRange        = errors.New("default retention days must be between 0 and 36500")
+	ErrDefaultRetentionYearsOutOfRange       = errors.New("default retention years must be between 0 and 100")
 )
 
 func (s3a *S3ApiServer) PutObjectHandler(w http.ResponseWriter, r *http.Request) {
@@ -110,8 +119,8 @@ func (s3a *S3ApiServer) PutObjectHandler(w http.ResponseWriter, r *http.Request)
 		// For non-versioned buckets, check if existing object has object lock protections
 		// that would prevent overwrite (PUT operations overwrite existing objects in non-versioned buckets)
 		if !versioningEnabled {
-			bypassGovernance := r.Header.Get("x-amz-bypass-governance-retention") == "true"
-			if err := s3a.checkObjectLockPermissions(r, bucket, object, "", bypassGovernance); err != nil {
+			governanceBypassAllowed := s3a.evaluateGovernanceBypassRequest(r, bucket, object)
+			if err := s3a.enforceObjectLockProtections(r, bucket, object, "", governanceBypassAllowed); err != nil {
 				glog.V(2).Infof("PutObjectHandler: object lock permissions check failed for %s/%s: %v", bucket, object, err)
 				s3err.WriteErrorResponse(w, r, s3err.ErrAccessDenied)
 				return
@@ -460,7 +469,7 @@ func (s3a *S3ApiServer) applyBucketDefaultRetention(bucket string, entry *filer_
 		return fmt.Errorf("default retention missing mode")
 	}
 
-	if defaultRetention.Days == 0 && defaultRetention.Years == 0 {
+	if !defaultRetention.DaysSet && !defaultRetention.YearsSet {
 		return fmt.Errorf("default retention missing period")
 	}
 
@@ -468,9 +477,9 @@ func (s3a *S3ApiServer) applyBucketDefaultRetention(bucket string, entry *filer_
 	var retainUntilDate time.Time
 	now := time.Now()
 
-	if defaultRetention.Days > 0 {
+	if defaultRetention.DaysSet && defaultRetention.Days > 0 {
 		retainUntilDate = now.AddDate(0, 0, defaultRetention.Days)
-	} else if defaultRetention.Years > 0 {
+	} else if defaultRetention.YearsSet && defaultRetention.Years > 0 {
 		retainUntilDate = now.AddDate(defaultRetention.Years, 0, 0)
 	}
 
@@ -553,26 +562,94 @@ func (s3a *S3ApiServer) validateObjectLockHeaders(r *http.Request, versioningEna
 
 // mapValidationErrorToS3Error maps object lock validation errors to appropriate S3 error codes
 func mapValidationErrorToS3Error(err error) s3err.ErrorCode {
+	// Check for sentinel errors first
 	switch {
 	case errors.Is(err, ErrObjectLockVersioningRequired):
+		// For object lock operations on non-versioned buckets, return InvalidRequest
+		// This matches the test expectations
 		return s3err.ErrInvalidRequest
 	case errors.Is(err, ErrInvalidObjectLockMode):
+		// For invalid object lock mode, return InvalidRequest
+		// This matches the test expectations
 		return s3err.ErrInvalidRequest
 	case errors.Is(err, ErrInvalidLegalHoldStatus):
-		return s3err.ErrInvalidRequest
+		// For invalid legal hold status in XML body, return MalformedXML
+		// AWS S3 treats invalid status values in XML as malformed content
+		return s3err.ErrMalformedXML
 	case errors.Is(err, ErrInvalidRetentionDateFormat):
+		// For malformed retention date format, return MalformedDate
+		// This matches the test expectations
 		return s3err.ErrMalformedDate
-	case errors.Is(err, ErrRetentionDateMustBeFuture),
-		errors.Is(err, ErrObjectLockModeRequiresDate),
-		errors.Is(err, ErrRetentionDateRequiresMode):
+	case errors.Is(err, ErrRetentionDateMustBeFuture):
+		// For retention dates in the past, return InvalidRequest
+		// This matches the test expectations
 		return s3err.ErrInvalidRequest
-	case errors.Is(err, ErrGovernanceBypassVersioningRequired):
+	case errors.Is(err, ErrObjectLockModeRequiresDate):
+		// For mode without retention date, return InvalidRequest
+		// This matches the test expectations
 		return s3err.ErrInvalidRequest
-	case errors.Is(err, ErrInvalidObjectLockDuration):
+	case errors.Is(err, ErrRetentionDateRequiresMode):
+		// For retention date without mode, return InvalidRequest
+		// This matches the test expectations
 		return s3err.ErrInvalidRequest
-	case errors.Is(err, ErrObjectLockDurationExceeded):
+	case errors.Is(err, ErrGovernanceBypassVersioningRequired):
+		// For governance bypass on non-versioned bucket, return InvalidRequest
+		// This matches the test expectations
 		return s3err.ErrInvalidRequest
-	default:
+	case errors.Is(err, ErrMalformedXML):
+		// For malformed XML in request body, return MalformedXML
+		// This matches the test expectations for invalid retention mode and legal hold status
+		return s3err.ErrMalformedXML
+	case errors.Is(err, ErrInvalidRetentionPeriod):
+		// For invalid retention period (e.g., Days <= 0), return InvalidRetentionPeriod
+		// This matches the test expectations
+		return s3err.ErrInvalidRetentionPeriod
+	case errors.Is(err, ErrComplianceModeActive):
+		// For compliance mode retention violations, return AccessDenied
+		// This matches the test expectations
+		return s3err.ErrAccessDenied
+	case errors.Is(err, ErrGovernanceModeActive):
+		// For governance mode retention violations, return AccessDenied
+		// This matches the test expectations
+		return s3err.ErrAccessDenied
+	case errors.Is(err, ErrObjectUnderLegalHold):
+		// For legal hold violations, return AccessDenied
+		// This matches the test expectations
+		return s3err.ErrAccessDenied
+	case errors.Is(err, ErrGovernanceBypassNotPermitted):
+		// For governance bypass permission violations, return AccessDenied
+		// This matches the test expectations
+		return s3err.ErrAccessDenied
+	// Validation error constants
+	case errors.Is(err, ErrObjectLockConfigurationMissingEnabled):
+		return s3err.ErrMalformedXML
+	case errors.Is(err, ErrInvalidObjectLockEnabledValue):
+		return s3err.ErrMalformedXML
+	case errors.Is(err, ErrRuleMissingDefaultRetention):
+		return s3err.ErrMalformedXML
+	case errors.Is(err, ErrDefaultRetentionMissingMode):
+		return s3err.ErrMalformedXML
+	case errors.Is(err, ErrInvalidDefaultRetentionMode):
+		return s3err.ErrMalformedXML
+	case errors.Is(err, ErrDefaultRetentionMissingPeriod):
+		return s3err.ErrMalformedXML
+	case errors.Is(err, ErrDefaultRetentionBothDaysAndYears):
+		return s3err.ErrMalformedXML
+	case errors.Is(err, ErrDefaultRetentionDaysOutOfRange):
+		return s3err.ErrInvalidRetentionPeriod
+	case errors.Is(err, ErrDefaultRetentionYearsOutOfRange):
+		return s3err.ErrInvalidRetentionPeriod
+	}
+
+	// Check for error constants from the updated validation functions
+	switch {
+	case errors.Is(err, ErrRetentionMissingMode):
+		return s3err.ErrInvalidRequest
+	case errors.Is(err, ErrRetentionMissingRetainUntilDate):
 		return s3err.ErrInvalidRequest
+	case errors.Is(err, ErrInvalidRetentionModeValue):
+		return s3err.ErrMalformedXML
 	}
+
+	return s3err.ErrInvalidRequest
 }