diff options
| author | dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> | 2025-11-19 21:22:18 -0800 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2025-11-19 21:22:18 -0800 |
| commit | c14e513964ff708b7ace352b7e86198b3ebe6827 (patch) | |
| tree | fc64d74d241faf292d2b37177e266614e15d9d18 /other/java/hdfs2/README.md | |
| parent | 7dce429e61037529694727134f828d273704f8ec (diff) | |
| download | seaweedfs-c14e513964ff708b7ace352b7e86198b3ebe6827.tar.xz seaweedfs-c14e513964ff708b7ace352b7e86198b3ebe6827.zip | |
chore(deps): bump org.apache.hadoop:hadoop-common from 3.2.4 to 3.4.0 in /other/java/hdfs3 (#7512)
* chore(deps): bump org.apache.hadoop:hadoop-common in /other/java/hdfs3
Bumps org.apache.hadoop:hadoop-common from 3.2.4 to 3.4.0.
---
updated-dependencies:
- dependency-name: org.apache.hadoop:hadoop-common
dependency-version: 3.4.0
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
* add java client unit tests
* Update dependency-reduced-pom.xml
* add java integration tests
* fix
* fix buffer
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
Diffstat (limited to 'other/java/hdfs2/README.md')
| -rw-r--r-- | other/java/hdfs2/README.md | 190 |
1 files changed, 190 insertions, 0 deletions
diff --git a/other/java/hdfs2/README.md b/other/java/hdfs2/README.md new file mode 100644 index 000000000..e98b06506 --- /dev/null +++ b/other/java/hdfs2/README.md @@ -0,0 +1,190 @@ +# SeaweedFS Hadoop2 Client + +Hadoop FileSystem implementation for SeaweedFS, compatible with Hadoop 2.x/3.x. + +## Building + +```bash +mvn clean install +``` + +## Testing + +This project includes two types of tests: + +### 1. Configuration Tests (No SeaweedFS Required) + +These tests verify configuration handling and initialization logic without requiring a running SeaweedFS instance: + +```bash +mvn test -Dtest=SeaweedFileSystemConfigTest +``` + +### 2. Integration Tests (Requires SeaweedFS) + +These tests verify actual FileSystem operations against a running SeaweedFS instance. + +#### Prerequisites + +1. Start SeaweedFS with default ports: + ```bash + # Terminal 1: Start master + weed master + + # Terminal 2: Start volume server + weed volume -mserver=localhost:9333 + + # Terminal 3: Start filer + weed filer -master=localhost:9333 + ``` + +2. Verify services are running: + - Master: http://localhost:9333 + - Filer HTTP: http://localhost:8888 + - Filer gRPC: localhost:18888 + +#### Running Integration Tests + +```bash +# Enable integration tests +export SEAWEEDFS_TEST_ENABLED=true + +# Run all tests +mvn test + +# Run specific test +mvn test -Dtest=SeaweedFileSystemTest +``` + +### Test Configuration + +Integration tests can be configured via environment variables or system properties: + +- `SEAWEEDFS_TEST_ENABLED`: Set to `true` to enable integration tests (default: false) +- Tests use these default connection settings: + - Filer Host: localhost + - Filer HTTP Port: 8888 + - Filer gRPC Port: 18888 + +### Running Tests with Custom Configuration + +To test against a different SeaweedFS instance, modify the test code or use Hadoop configuration: + +```java +conf.set("fs.seaweed.filer.host", "your-host"); +conf.setInt("fs.seaweed.filer.port", 8888); +conf.setInt("fs.seaweed.filer.port.grpc", 18888); +``` + +## Test Coverage + +The test suite covers: + +- **Configuration & Initialization** + - URI parsing and configuration + - Default values + - Configuration overrides + - Working directory management + +- **File Operations** + - Create files + - Read files + - Write files + - Append to files + - Delete files + +- **Directory Operations** + - Create directories + - List directory contents + - Delete directories (recursive and non-recursive) + +- **Metadata Operations** + - Get file status + - Set permissions + - Set owner/group + - Rename files and directories + +## Usage in Hadoop + +1. Copy the built JAR to your Hadoop classpath: + ```bash + cp target/seaweedfs-hadoop2-client-*.jar $HADOOP_HOME/share/hadoop/common/lib/ + ``` + +2. Configure `core-site.xml`: + ```xml + <configuration> + <property> + <name>fs.seaweedfs.impl</name> + <value>seaweed.hdfs.SeaweedFileSystem</value> + </property> + <property> + <name>fs.seaweed.filer.host</name> + <value>localhost</value> + </property> + <property> + <name>fs.seaweed.filer.port</name> + <value>8888</value> + </property> + <property> + <name>fs.seaweed.filer.port.grpc</name> + <value>18888</value> + </property> + </configuration> + ``` + +3. Use SeaweedFS with Hadoop commands: + ```bash + hadoop fs -ls seaweedfs://localhost:8888/ + hadoop fs -mkdir seaweedfs://localhost:8888/test + hadoop fs -put local.txt seaweedfs://localhost:8888/test/ + ``` + +## Continuous Integration + +For CI environments, tests can be run in two modes: + +1. **Configuration Tests Only** (default, no SeaweedFS required): + ```bash + mvn test -Dtest=SeaweedFileSystemConfigTest + ``` + +2. **Full Integration Tests** (requires SeaweedFS): + ```bash + # Start SeaweedFS in CI environment + # Then run: + export SEAWEEDFS_TEST_ENABLED=true + mvn test + ``` + +## Troubleshooting + +### Tests are skipped + +If you see "Skipping test - SEAWEEDFS_TEST_ENABLED not set": +```bash +export SEAWEEDFS_TEST_ENABLED=true +``` + +### Connection refused errors + +Ensure SeaweedFS is running and accessible: +```bash +curl http://localhost:8888/ +``` + +### gRPC errors + +Verify the gRPC port is accessible: +```bash +# Should show the port is listening +netstat -an | grep 18888 +``` + +## Contributing + +When adding new features, please include: +1. Configuration tests (no SeaweedFS required) +2. Integration tests (with SEAWEEDFS_TEST_ENABLED guard) +3. Documentation updates + |
