diff options
Diffstat (limited to 'test/foundationdb/README.md')
| -rw-r--r-- | test/foundationdb/README.md | 372 |
1 files changed, 372 insertions, 0 deletions
diff --git a/test/foundationdb/README.md b/test/foundationdb/README.md new file mode 100644 index 000000000..ba1e7627a --- /dev/null +++ b/test/foundationdb/README.md @@ -0,0 +1,372 @@ +# FoundationDB Integration Testing + +This directory contains integration tests and setup scripts for the FoundationDB filer store in SeaweedFS. + +## Quick Start + +```bash +# ✅ GUARANTEED TO WORK - Run reliable tests (no FoundationDB dependencies) +make test-reliable # Validation + Mock tests + +# Run individual test types +make test-mock # Mock FoundationDB tests (always work) +go test -v ./validation_test.go # Package structure validation + +# 🐳 FULL INTEGRATION (requires Docker + FoundationDB dependencies) +make setup # Start FoundationDB cluster + SeaweedFS +make test # Run all integration tests +make test-simple # Simple containerized test environment + +# Clean up +make clean # Clean main environment +make clean-simple # Clean simple test environment + +# 🍎 ARM64 / APPLE SILICON SUPPORT +make setup-arm64 # Native ARM64 FoundationDB (builds from source) +make setup-emulated # x86 emulation (faster setup) +make test-arm64 # Test with ARM64 native +make test-emulated # Test with x86 emulation +``` + +### Test Levels + +1. **✅ Validation Tests** (`validation_test.go`) - Always work, no dependencies +2. **✅ Mock Tests** (`mock_integration_test.go`) - Test FoundationDB store logic with mocks +3. **⚠️ Real Integration Tests** (`foundationdb_*_test.go`) - Require actual FoundationDB cluster + +### ARM64 / Apple Silicon Support + +**🍎 For M1/M2/M3 Mac users:** FoundationDB's official Docker images are AMD64-only. We provide three solutions: + +- **Native ARM64** (`make setup-arm64`) - Downloads official FoundationDB ARM64 packages and builds SeaweedFS natively (≈2-3 min setup, best performance) +- **x86 Emulation** (`make setup-emulated`) - Uses Docker emulation (fast setup, slower runtime) +- **Mock Testing** (`make test-mock`) - No FoundationDB needed (instant, tests logic only) + +The ARM64 setup automatically builds both FoundationDB and SeaweedFS from source using `docker-compose.arm64.yml` and dedicated ARM64 Dockerfiles. No pre-built images required! + +📖 **Detailed Guide:** See [README.ARM64.md](README.ARM64.md) for complete ARM64 documentation. + +## Test Environment + +The test environment includes: + +- **3-node FoundationDB cluster** (fdb1, fdb2, fdb3) for realistic distributed testing +- **Database initialization service** (fdb-init) that configures the cluster +- **SeaweedFS service** configured to use the FoundationDB filer store +- **Automatic service orchestration** with proper startup dependencies + +## Test Structure + +### Integration Tests + +#### `foundationdb_integration_test.go` +- Basic CRUD operations (Create, Read, Update, Delete) +- Directory operations and listing: + - `ListDirectoryEntries` - List all entries in a directory + - `ListDirectoryPrefixedEntries` - List entries matching a prefix + - `DeleteFolderChildren` - Bulk deletion of directory contents +- Transaction handling (begin, commit, rollback) +- Key-Value operations +- Large entry handling with compression +- Error scenarios and edge cases + +**Note:** These tests operate at the filer store level, testing the metadata index operations that underpin S3 bucket listing and directory tree operations. + +#### `foundationdb_concurrent_test.go` +- Concurrent insert operations across multiple goroutines +- Concurrent read/write operations on shared files +- Concurrent transaction handling with conflict resolution +- Concurrent directory operations +- Concurrent key-value operations +- Stress testing under load + +#### `test_fdb_s3.sh` - End-to-End S3 Integration Tests +- **S3 bucket creation** - Create buckets via S3 API +- **S3 file upload** - Upload files to buckets +- **S3 bucket listing** (`aws s3 ls`) - **Validates listing operations work correctly** +- **S3 file download** - Retrieve and verify file contents +- **S3 file deletion** - Delete objects and verify removal +- **FoundationDB backend verification** - Confirms data is stored in FDB +- **Filer directory operations** - Direct filer API calls for directory creation/listing + +**This test validates the complete S3 workflow including the listing operations that were problematic in earlier versions.** + +#### Unit Tests (`weed/filer/foundationdb/foundationdb_store_test.go`) +- Store initialization and configuration +- Key generation and directory prefixes +- Error handling and validation +- Performance benchmarks +- Configuration validation + +## Configuration + +### Environment Variables + +The tests can be configured using environment variables: + +```bash +export FDB_CLUSTER_FILE=/var/fdb/config/fdb.cluster +export WEED_FOUNDATIONDB_ENABLED=true +export WEED_FOUNDATIONDB_API_VERSION=740 +export WEED_FOUNDATIONDB_TIMEOUT=10s +``` + +#### Docker Compose Environment Variables + +The `docker-compose.yml` file supports the following optional environment variables with sensible defaults: + +```bash +# FoundationDB image (default: foundationdb/foundationdb:7.1.61) +export FOUNDATIONDB_IMAGE=foundationdb/foundationdb:7.1.61 + +# FoundationDB port (default: 4500) +export FDB_PORT=4500 + +# FoundationDB cluster file contents (default: docker:docker@fdb1:4500,fdb2:4500,fdb3:4500) +export FDB_CLUSTER_FILE_CONTENTS="docker:docker@fdb1:4500,fdb2:4500,fdb3:4500" + +# SeaweedFS image (default: chrislusf/seaweedfs:latest) +export SEAWEEDFS_IMAGE=chrislusf/seaweedfs:latest +``` + +**Note:** These variables are optional. If not set, the docker-compose will use the default values shown above, allowing `docker-compose up` to work out-of-the-box without any `.env` file or manual configuration. + +### Docker Compose Configuration + +The `docker-compose.yml` sets up: + +1. **FoundationDB Cluster**: 3 coordinating nodes with data distribution +2. **Database Configuration**: Single SSD storage class for testing +3. **SeaweedFS Integration**: Automatic filer store configuration +4. **Volume Persistence**: Data persists between container restarts + +### Test Configuration Files + +- `filer.toml`: FoundationDB filer store configuration +- `s3.json`: S3 API credentials for end-to-end testing +- `Makefile`: Test automation and environment management + +## Test Commands + +### Setup Commands + +```bash +make setup # Full environment setup +make dev-fdb # Just FoundationDB cluster +make install-deps # Check dependencies +make check-env # Validate configuration +``` + +### Test Commands + +```bash +make test # All tests +make test-unit # Go unit tests +make test-integration # Integration tests +make test-e2e # End-to-end S3 tests (includes S3 bucket listing) +make test-crud # Basic CRUD operations +make test-concurrent # Concurrency tests +make test-benchmark # Performance benchmarks +``` + +#### S3 and Listing Operation Coverage + +**✅ Currently Tested:** +- **S3 bucket listing** (`aws s3 ls`) - Validated in `test_fdb_s3.sh` +- **Directory metadata listing** (`ListDirectoryEntries`) - Tested in `foundationdb_integration_test.go` +- **Prefix-based listing** (`ListDirectoryPrefixedEntries`) - Tested in `foundationdb_integration_test.go` +- **Filer directory operations** - Basic filer API calls in `test_fdb_s3.sh` +- **Metadata index operations** - All CRUD operations on directory entries + +**⚠️ Limited/Future Coverage:** +- **Recursive tree operations** - Not explicitly tested (e.g., `weed filer.tree` command) +- **Large directory stress tests** - Listings with thousands of entries not currently benchmarked +- **Concurrent listing operations** - Multiple simultaneous directory listings under load +- **S3 ListObjectsV2 pagination** - Large bucket listing with continuation tokens + +**Recommendation:** If experiencing issues with S3 listing operations in production, add stress tests for large directories and concurrent listing scenarios to validate FoundationDB's range scan performance at scale. + +### Debug Commands + +```bash +make status # Show service status +make logs # Show all logs +make logs-fdb # FoundationDB logs only +make logs-seaweedfs # SeaweedFS logs only +make debug # Debug information +``` + +### Cleanup Commands + +```bash +make clean # Stop services and cleanup +``` + +## Test Data + +Tests use isolated directory prefixes to avoid conflicts: + +- **Unit tests**: `seaweedfs_test` +- **Integration tests**: `seaweedfs_test` +- **Concurrent tests**: `seaweedfs_concurrent_test_<timestamp>` +- **E2E tests**: `seaweedfs` (default) + +## Expected Test Results + +### Performance Expectations + +Based on FoundationDB characteristics: +- **Single operations**: < 10ms latency +- **Batch operations**: High throughput with transactions +- **Concurrent operations**: Linear scaling with multiple clients +- **Directory listings**: Efficient range scans + +### Reliability Expectations + +- **ACID compliance**: All operations are atomic and consistent +- **Fault tolerance**: Automatic recovery from node failures +- **Concurrency**: No data corruption under concurrent load +- **Durability**: Data persists across restarts + +## Troubleshooting + +### Common Issues + +1. **FoundationDB Connection Errors** + ```bash + # Check cluster status + make status + + # Verify cluster file + docker-compose exec fdb-init cat /var/fdb/config/fdb.cluster + ``` + +2. **Test Failures** + ```bash + # Check service logs + make logs-fdb + make logs-seaweedfs + + # Run with verbose output + go test -v -tags foundationdb ./... + ``` + +3. **Performance Issues** + ```bash + # Check cluster health + docker-compose exec fdb-init fdbcli --exec 'status details' + + # Monitor resource usage + docker stats + ``` + +4. **Docker Issues** + ```bash + # Clean Docker state + make clean + docker system prune -f + + # Restart from scratch + make setup + ``` + +### Debug Mode + +Enable verbose logging for detailed troubleshooting: + +```bash +# SeaweedFS debug logs +WEED_FILER_OPTIONS_V=2 make test + +# FoundationDB debug logs (in fdbcli) +configure new single ssd; status details +``` + +### Manual Testing + +For manual verification: + +```bash +# Start environment +make dev-fdb + +# Connect to FoundationDB +docker-compose exec fdb-init fdbcli + +# FDB commands: +# status - Show cluster status +# getrange "" \xFF - Show all keys +# getrange seaweedfs seaweedfs\xFF - Show SeaweedFS keys +``` + +### Listing Operations Return Empty Results + +**Symptoms:** Uploads succeed, direct file reads work, but listing operations (`aws s3 ls`, `s3.bucket.list`, `weed filer.ls/tree`) return no results. + +**Test Coverage:** The `test_fdb_s3.sh` script explicitly tests S3 bucket listing (`aws s3 ls`) to catch this class of issue. Integration tests cover the underlying `ListDirectoryEntries` operations. + +**Diagnostic steps:** + +```bash +# 1. Verify writes reached FoundationDB +docker-compose exec fdb-init fdbcli +> getrange seaweedfs seaweedfs\xFF +# If no keys appear, writes aren't reaching the store + +# 2. Check SeaweedFS volume assignment +curl http://localhost:9333/cluster/status +# Look for "AssignVolume" errors in logs: +make logs-seaweedfs | grep -i "assignvolume\|writable" + +# 3. Verify filer health and configuration +curl http://localhost:8888/statistics/health +make logs-seaweedfs | grep -i "store\|foundationdb" +``` + +**Interpretation:** +- No SeaweedFS keys in FDB: Directory index writes failing; check filer logs for write errors +- AssignVolume errors: Volume assignment blocked; check master status and disk space +- Filer health errors: Configuration or connectivity issue; restart services and verify filer.toml + +**Recovery:** +- If fresh data: restart services (`make clean && make setup`) +- If production data: ensure volume assignment works, check disk space on data nodes + +## CI Integration + +For continuous integration: + +```bash +# CI test suite +make ci-test # Unit + integration tests +make ci-e2e # Full end-to-end test suite +``` + +The tests are designed to be reliable in CI environments with: +- Automatic service startup and health checking +- Timeout handling for slow CI systems +- Proper cleanup and resource management +- Detailed error reporting and logs + +## Performance Benchmarks + +Run performance benchmarks: + +```bash +make test-benchmark + +# Sample expected results: +# BenchmarkFoundationDBStore_InsertEntry-8 1000 1.2ms per op +# BenchmarkFoundationDBStore_FindEntry-8 5000 0.5ms per op +# BenchmarkFoundationDBStore_KvOperations-8 2000 0.8ms per op +``` + +## Contributing + +When adding new tests: + +1. Use the `//go:build foundationdb` build tag +2. Follow the existing test structure and naming +3. Include both success and error scenarios +4. Add appropriate cleanup and resource management +5. Update this README with new test descriptions |