S3: add object versioning (#6945)

* add object versioning

* add missing file

* Update weed/s3api/s3api_object_versioning.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_object_versioning.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_object_versioning.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* ListObjectVersionsResult is better to show multiple version entries

* fix test

* Update weed/s3api/s3api_object_handlers_put.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_object_versioning.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* multiple improvements

* move PutBucketVersioningHandler into weed/s3api/s3api_bucket_handlers.go file
* duplicated code for reading bucket config, versioningEnabled, etc. try to use functions
* opportunity to cache bucket config

* error handling if bucket is not found

* in case bucket is not found

* fix build

* add object versioning tests

* remove non-existent tests

* add tests

* add versioning tests

* skip a new test

* ensure .versions directory exists before saving info into it

* fix creating version entry

* logging on creating version directory

* Update s3api_object_versioning_test.go

* retry and wait for directory creation

* revert add more logging

* Update s3api_object_versioning.go

* more debug messages

* clean up logs, and touch directory correctly

* log the .versions creation and then parent directory listing

* use mkFile instead of touch

touch is for update

* clean up data

* add versioning test in go

* change location

* if modified, latest version is moved to .versions directory, and create a new latest version

 Core versioning functionality: WORKING
TestVersioningBasicWorkflow - PASS
TestVersioningDeleteMarkers - PASS
TestVersioningMultipleVersionsSameObject - PASS
TestVersioningDeleteAndRecreate - PASS
TestVersioningListWithPagination - PASS
❌ Some advanced features still failing:
ETag calculation issues (using mtime instead of proper MD5)
Specific version retrieval (EOF error)
Version deletion (internal errors)
Concurrent operations (race conditions)

* calculate multi chunk md5

Test Results - All Passing:
✅ TestBucketListReturnDataVersioning - PASS
✅ TestVersioningCreateObjectsInOrder - PASS
✅ TestVersioningBasicWorkflow - PASS
✅ TestVersioningMultipleVersionsSameObject - PASS
✅ TestVersioningDeleteMarkers - PASS

* dedupe

* fix TestVersioningErrorCases

* fix eof error of reading old versions

* get specific version also check current version

* enable integration tests for versioning

* trigger action to work for now

* Fix GitHub Actions S3 versioning tests workflow

- Fix syntax error (incorrect indentation)
- Update directory paths from weed/s3api/versioning_tests/ to test/s3/versioning/
- Add push trigger for add-object-versioning branch to enable CI during development
- Update artifact paths to match correct directory structure

* Improve CI robustness for S3 versioning tests

Makefile improvements:
- Increase server startup timeout from 30s to 90s for CI environments
- Add progressive timeout reporting (logs at 30s, full logs at 90s)
- Better error handling with server logs on failure
- Add server PID tracking for debugging
- Improved test failure reporting

GitHub Actions workflow improvements:
- Increase job timeouts to account for CI environment delays
- Add system information logging (memory, disk space)
- Add detailed failure reporting with server logs
- Add process and network diagnostics on failure
- Better error messaging and log collection

These changes should resolve the 'Server failed to start within 30 seconds' issue
that was causing the CI tests to fail.

* adjust testing volume size

* Update Makefile

* Update Makefile

* Update Makefile

* Update Makefile

* Update s3-versioning-tests.yml

* Update s3api_object_versioning.go

* Update Makefile

* do not clean up

* log received version id

* more logs

* printout response

* print out list version response

* use tmp files when put versioned object

* change to versions folder layout

* Delete weed-test.log

* test with mixed versioned and unversioned objects

* remove versionDirCache

* remove unused functions

* remove unused function

* remove fallback checking

* minor

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

1 files changed, 246 insertions, 0 deletions

diff --git a/weed/s3api/s3api_bucket_config.go b/weed/s3api/s3api_bucket_config.go
new file mode 100644
index 000000000..273eb6fbd
--- /dev/null
+++ b/weed/s3api/s3api_bucket_config.go
@@ -0,0 +1,246 @@
+package s3api
+
+import (
+	"fmt"
+	"sync"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/glog"
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
+)
+
+// BucketConfig represents cached bucket configuration
+type BucketConfig struct {
+	Name         string
+	Versioning   string // "Enabled", "Suspended", or ""
+	Ownership    string
+	ACL          []byte
+	Owner        string
+	LastModified time.Time
+	Entry        *filer_pb.Entry
+}
+
+// BucketConfigCache provides caching for bucket configurations
+type BucketConfigCache struct {
+	cache map[string]*BucketConfig
+	mutex sync.RWMutex
+	ttl   time.Duration
+}
+
+// NewBucketConfigCache creates a new bucket configuration cache
+func NewBucketConfigCache(ttl time.Duration) *BucketConfigCache {
+	return &BucketConfigCache{
+		cache: make(map[string]*BucketConfig),
+		ttl:   ttl,
+	}
+}
+
+// Get retrieves bucket configuration from cache
+func (bcc *BucketConfigCache) Get(bucket string) (*BucketConfig, bool) {
+	bcc.mutex.RLock()
+	defer bcc.mutex.RUnlock()
+
+	config, exists := bcc.cache[bucket]
+	if !exists {
+		return nil, false
+	}
+
+	// Check if cache entry is expired
+	if time.Since(config.LastModified) > bcc.ttl {
+		return nil, false
+	}
+
+	return config, true
+}
+
+// Set stores bucket configuration in cache
+func (bcc *BucketConfigCache) Set(bucket string, config *BucketConfig) {
+	bcc.mutex.Lock()
+	defer bcc.mutex.Unlock()
+
+	config.LastModified = time.Now()
+	bcc.cache[bucket] = config
+}
+
+// Remove removes bucket configuration from cache
+func (bcc *BucketConfigCache) Remove(bucket string) {
+	bcc.mutex.Lock()
+	defer bcc.mutex.Unlock()
+
+	delete(bcc.cache, bucket)
+}
+
+// Clear clears all cached configurations
+func (bcc *BucketConfigCache) Clear() {
+	bcc.mutex.Lock()
+	defer bcc.mutex.Unlock()
+
+	bcc.cache = make(map[string]*BucketConfig)
+}
+
+// getBucketConfig retrieves bucket configuration with caching
+func (s3a *S3ApiServer) getBucketConfig(bucket string) (*BucketConfig, s3err.ErrorCode) {
+	// Try cache first
+	if config, found := s3a.bucketConfigCache.Get(bucket); found {
+		return config, s3err.ErrNone
+	}
+
+	// Load from filer
+	bucketEntry, err := s3a.getEntry(s3a.option.BucketsPath, bucket)
+	if err != nil {
+		if err == filer_pb.ErrNotFound {
+			return nil, s3err.ErrNoSuchBucket
+		}
+		glog.Errorf("getBucketConfig: failed to get bucket entry for %s: %v", bucket, err)
+		return nil, s3err.ErrInternalError
+	}
+
+	config := &BucketConfig{
+		Name:  bucket,
+		Entry: bucketEntry,
+	}
+
+	// Extract configuration from extended attributes
+	if bucketEntry.Extended != nil {
+		if versioning, exists := bucketEntry.Extended[s3_constants.ExtVersioningKey]; exists {
+			config.Versioning = string(versioning)
+		}
+		if ownership, exists := bucketEntry.Extended[s3_constants.ExtOwnershipKey]; exists {
+			config.Ownership = string(ownership)
+		}
+		if acl, exists := bucketEntry.Extended[s3_constants.ExtAmzAclKey]; exists {
+			config.ACL = acl
+		}
+		if owner, exists := bucketEntry.Extended[s3_constants.ExtAmzOwnerKey]; exists {
+			config.Owner = string(owner)
+		}
+	}
+
+	// Cache the result
+	s3a.bucketConfigCache.Set(bucket, config)
+
+	return config, s3err.ErrNone
+}
+
+// updateBucketConfig updates bucket configuration and invalidates cache
+func (s3a *S3ApiServer) updateBucketConfig(bucket string, updateFn func(*BucketConfig) error) s3err.ErrorCode {
+	config, errCode := s3a.getBucketConfig(bucket)
+	if errCode != s3err.ErrNone {
+		return errCode
+	}
+
+	// Apply update function
+	if err := updateFn(config); err != nil {
+		glog.Errorf("updateBucketConfig: update function failed for bucket %s: %v", bucket, err)
+		return s3err.ErrInternalError
+	}
+
+	// Prepare extended attributes
+	if config.Entry.Extended == nil {
+		config.Entry.Extended = make(map[string][]byte)
+	}
+
+	// Update extended attributes
+	if config.Versioning != "" {
+		config.Entry.Extended[s3_constants.ExtVersioningKey] = []byte(config.Versioning)
+	}
+	if config.Ownership != "" {
+		config.Entry.Extended[s3_constants.ExtOwnershipKey] = []byte(config.Ownership)
+	}
+	if config.ACL != nil {
+		config.Entry.Extended[s3_constants.ExtAmzAclKey] = config.ACL
+	}
+	if config.Owner != "" {
+		config.Entry.Extended[s3_constants.ExtAmzOwnerKey] = []byte(config.Owner)
+	}
+
+	// Save to filer
+	err := s3a.updateEntry(s3a.option.BucketsPath, config.Entry)
+	if err != nil {
+		glog.Errorf("updateBucketConfig: failed to update bucket entry for %s: %v", bucket, err)
+		return s3err.ErrInternalError
+	}
+
+	// Update cache
+	s3a.bucketConfigCache.Set(bucket, config)
+
+	return s3err.ErrNone
+}
+
+// isVersioningEnabled checks if versioning is enabled for a bucket (with caching)
+func (s3a *S3ApiServer) isVersioningEnabled(bucket string) (bool, error) {
+	config, errCode := s3a.getBucketConfig(bucket)
+	if errCode != s3err.ErrNone {
+		if errCode == s3err.ErrNoSuchBucket {
+			return false, filer_pb.ErrNotFound
+		}
+		return false, fmt.Errorf("failed to get bucket config: %v", errCode)
+	}
+
+	return config.Versioning == "Enabled", nil
+}
+
+// getBucketVersioningStatus returns the versioning status for a bucket
+func (s3a *S3ApiServer) getBucketVersioningStatus(bucket string) (string, s3err.ErrorCode) {
+	config, errCode := s3a.getBucketConfig(bucket)
+	if errCode != s3err.ErrNone {
+		return "", errCode
+	}
+
+	if config.Versioning == "" {
+		return "Suspended", s3err.ErrNone
+	}
+
+	return config.Versioning, s3err.ErrNone
+}
+
+// setBucketVersioningStatus sets the versioning status for a bucket
+func (s3a *S3ApiServer) setBucketVersioningStatus(bucket, status string) s3err.ErrorCode {
+	return s3a.updateBucketConfig(bucket, func(config *BucketConfig) error {
+		config.Versioning = status
+		return nil
+	})
+}
+
+// getBucketOwnership returns the ownership setting for a bucket
+func (s3a *S3ApiServer) getBucketOwnership(bucket string) (string, s3err.ErrorCode) {
+	config, errCode := s3a.getBucketConfig(bucket)
+	if errCode != s3err.ErrNone {
+		return "", errCode
+	}
+
+	return config.Ownership, s3err.ErrNone
+}
+
+// setBucketOwnership sets the ownership setting for a bucket
+func (s3a *S3ApiServer) setBucketOwnership(bucket, ownership string) s3err.ErrorCode {
+	return s3a.updateBucketConfig(bucket, func(config *BucketConfig) error {
+		config.Ownership = ownership
+		return nil
+	})
+}
+
+// removeBucketConfigKey removes a specific configuration key from bucket
+func (s3a *S3ApiServer) removeBucketConfigKey(bucket, key string) s3err.ErrorCode {
+	return s3a.updateBucketConfig(bucket, func(config *BucketConfig) error {
+		if config.Entry.Extended != nil {
+			delete(config.Entry.Extended, key)
+		}
+
+		// Update our local config too
+		switch key {
+		case s3_constants.ExtVersioningKey:
+			config.Versioning = ""
+		case s3_constants.ExtOwnershipKey:
+			config.Ownership = ""
+		case s3_constants.ExtAmzAclKey:
+			config.ACL = nil
+		case s3_constants.ExtAmzOwnerKey:
+			config.Owner = ""
+		}
+
+		return nil
+	})
+}