diff options
Diffstat (limited to 'test/postgres/README.md')
| -rw-r--r-- | test/postgres/README.md | 320 |
1 files changed, 320 insertions, 0 deletions
diff --git a/test/postgres/README.md b/test/postgres/README.md new file mode 100644 index 000000000..2466c6069 --- /dev/null +++ b/test/postgres/README.md @@ -0,0 +1,320 @@ +# SeaweedFS PostgreSQL Protocol Test Suite + +This directory contains a comprehensive Docker Compose test setup for the SeaweedFS PostgreSQL wire protocol implementation. + +## Overview + +The test suite includes: +- **SeaweedFS Cluster**: Full SeaweedFS server with MQ broker and agent +- **PostgreSQL Server**: SeaweedFS PostgreSQL wire protocol server +- **MQ Data Producer**: Creates realistic test data across multiple topics and namespaces +- **PostgreSQL Test Client**: Comprehensive Go client testing all functionality +- **Interactive Tools**: psql CLI access for manual testing + +## Quick Start + +### 1. Run Complete Test Suite (Automated) +```bash +./run-tests.sh all +``` + +This will automatically: +1. Start SeaweedFS and PostgreSQL servers +2. Create test data in multiple MQ topics +3. Run comprehensive PostgreSQL client tests +4. Show results + +### 2. Manual Step-by-Step Testing +```bash +# Start the services +./run-tests.sh start + +# Create test data +./run-tests.sh produce + +# Run automated tests +./run-tests.sh test + +# Connect with psql for interactive testing +./run-tests.sh psql +``` + +### 3. Interactive PostgreSQL Testing +```bash +# Connect with psql +./run-tests.sh psql + +# Inside psql session: +postgres=> SHOW DATABASES; +postgres=> \c analytics; +postgres=> SHOW TABLES; +postgres=> SELECT COUNT(*) FROM user_events; +postgres=> SELECT COUNT(*) FROM user_events; +postgres=> \q +``` + +## Test Data Structure + +The producer creates realistic test data across multiple namespaces: + +### Analytics Namespace +- **`user_events`** (1000 records): User interaction events + - Fields: id, user_id, user_type, action, status, amount, timestamp, metadata + - User types: premium, standard, trial, enterprise + - Actions: login, logout, purchase, view, search, click, download + +- **`system_logs`** (500 records): System operation logs + - Fields: id, level, service, message, error_code, timestamp + - Levels: debug, info, warning, error, critical + - Services: auth-service, payment-service, user-service, etc. + +- **`metrics`** (800 records): System metrics + - Fields: id, name, value, tags, timestamp + - Metrics: cpu_usage, memory_usage, disk_usage, request_latency, etc. + +### E-commerce Namespace +- **`product_views`** (1200 records): Product interaction data + - Fields: id, product_id, user_id, category, price, view_count, timestamp + - Categories: electronics, books, clothing, home, sports, automotive + +- **`user_events`** (600 records): E-commerce specific user events + +### Logs Namespace +- **`application_logs`** (2000 records): Application logs +- **`error_logs`** (300 records): Error-specific logs with 4xx/5xx error codes + +## Architecture + +``` +┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ +│ PostgreSQL │ │ PostgreSQL │ │ SeaweedFS │ +│ Clients │◄──►│ Wire Protocol │◄──►│ SQL Engine │ +│ (psql, Go) │ │ Server │ │ │ +└─────────────────┘ └──────────────────┘ └─────────────────┘ + │ │ + ▼ ▼ + ┌──────────────────┐ ┌─────────────────┐ + │ Session │ │ MQ Broker │ + │ Management │ │ & Topics │ + └──────────────────┘ └─────────────────┘ +``` + +## Services + +### SeaweedFS Server +- **Ports**: 9333 (master), 8888 (filer), 8333 (S3), 8085 (volume), 9533 (metrics), 26777→16777 (MQ agent), 27777→17777 (MQ broker) +- **Features**: Full MQ broker, S3 API, filer, volume server +- **Data**: Persistent storage in Docker volume +- **Health Check**: Cluster status endpoint + +### PostgreSQL Server +- **Port**: 5432 (standard PostgreSQL port) +- **Protocol**: Full PostgreSQL 3.0 wire protocol +- **Authentication**: Trust mode (no password for testing) +- **Features**: Real-time MQ topic discovery, database context switching + +### MQ Producer +- **Purpose**: Creates realistic test data +- **Topics**: 7 topics across 3 namespaces +- **Data Types**: JSON messages with varied schemas +- **Volume**: ~4,400 total records with realistic distributions + +### Test Client +- **Language**: Go with standard `lib/pq` PostgreSQL driver +- **Tests**: 8 comprehensive test categories +- **Coverage**: System info, discovery, queries, aggregations, context switching + +## Available Commands + +```bash +./run-tests.sh start # Start services +./run-tests.sh produce # Create test data +./run-tests.sh test # Run client tests +./run-tests.sh psql # Interactive psql +./run-tests.sh logs # Show service logs +./run-tests.sh status # Service status +./run-tests.sh stop # Stop services +./run-tests.sh clean # Complete cleanup +./run-tests.sh all # Full automated test +``` + +## Test Categories + +### 1. System Information +- PostgreSQL version compatibility +- Current user and database +- Server settings and encoding + +### 2. Database Discovery +- `SHOW DATABASES` - List MQ namespaces +- Dynamic namespace discovery from filer + +### 3. Table Discovery +- `SHOW TABLES` - List topics in current namespace +- Real-time topic discovery + +### 4. Data Queries +- Basic `SELECT * FROM table` queries +- Sample data retrieval and display +- Column information + +### 5. Aggregation Queries +- `COUNT(*)`, `SUM()`, `AVG()`, `MIN()`, `MAX()` +- Aggregation operations +- Statistical analysis + +### 6. Database Context Switching +- `USE database` commands +- Session isolation testing +- Cross-namespace queries + +### 7. System Columns +- `_timestamp_ns`, `_key`, `_source` access +- MQ metadata exposure + +### 8. Complex Queries +- `WHERE` clauses with comparisons +- `LIMIT` +- Multi-condition filtering + +## Expected Results + +After running the complete test suite, you should see: + +``` +=== Test Results === +✅ Test PASSED: System Information +✅ Test PASSED: Database Discovery +✅ Test PASSED: Table Discovery +✅ Test PASSED: Data Queries +✅ Test PASSED: Aggregation Queries +✅ Test PASSED: Database Context Switching +✅ Test PASSED: System Columns +✅ Test PASSED: Complex Queries + +Test Results: 8/8 tests passed +🎉 All tests passed! +``` + +## Manual Testing Examples + +### Connect with psql +```bash +./run-tests.sh psql +``` + +### Basic Exploration +```sql +-- Check system information +SELECT version(); +SELECT current_user, current_database(); + +-- Discover data structure +SHOW DATABASES; +\c analytics; +SHOW TABLES; +DESCRIBE user_events; +``` + +### Data Analysis +```sql +-- Basic queries +SELECT COUNT(*) FROM user_events; +SELECT * FROM user_events LIMIT 5; + +-- Aggregations +SELECT + COUNT(*) as events, + AVG(amount) as avg_amount +FROM user_events +WHERE amount IS NOT NULL; + +-- Time-based analysis +SELECT + COUNT(*) as count +FROM user_events +WHERE status = 'active'; +``` + +### Cross-Namespace Analysis +```sql +-- Switch between namespaces +USE ecommerce; +SELECT COUNT(*) FROM product_views; + +USE logs; +SELECT COUNT(*) FROM application_logs; +``` + +## Troubleshooting + +### Services Not Starting +```bash +# Check service status +./run-tests.sh status + +# View logs +./run-tests.sh logs seaweedfs +./run-tests.sh logs postgres-server +``` + +### No Test Data +```bash +# Recreate test data +./run-tests.sh produce + +# Check producer logs +./run-tests.sh logs mq-producer +``` + +### Connection Issues +```bash +# Test PostgreSQL server health +docker-compose exec postgres-server nc -z localhost 5432 + +# Test SeaweedFS health +curl http://localhost:9333/cluster/status +``` + +### Clean Restart +```bash +# Complete cleanup and restart +./run-tests.sh clean +./run-tests.sh all +``` + +## Development + +### Modifying Test Data +Edit `producer.go` to change: +- Data schemas and volume +- Topic names and namespaces +- Record generation logic + +### Adding Tests +Edit `client.go` to add new test functions: +```go +func testNewFeature(db *sql.DB) error { + // Your test implementation + return nil +} + +// Add to tests slice in main() +{"New Feature", testNewFeature}, +``` + +### Custom Queries +Use the interactive psql session: +```bash +./run-tests.sh psql +``` + +## Production Considerations + +This test setup demonstrates: +- **Real MQ Integration**: Actual topic discovery and data access +- **Universal PostgreSQL Compatibility**: Works with any PostgreSQL client +- **Production-Ready Features**: Authentication, session management, error handling +- **Scalable Architecture**: Direct SQL engine integration, no translation overhead + +The test validates that SeaweedFS can serve as a drop-in PostgreSQL replacement for read-only analytics workloads on MQ data. |