diff options
| author | Chris Lu <chrislusf@users.noreply.github.com> | 2025-12-10 12:42:09 -0800 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2025-12-10 12:42:09 -0800 |
| commit | c1534200223d99cc193ce229439e4f8e15f1d9b2 (patch) | |
| tree | 3d5e5df5a295efa924d30d344ada067c059cef32 /go.sum | |
| parent | ae7333d28e77d907a9e7d50055e352c15678e723 (diff) | |
| download | seaweedfs-c1534200223d99cc193ce229439e4f8e15f1d9b2.tar.xz seaweedfs-c1534200223d99cc193ce229439e4f8e15f1d9b2.zip | |
filer: add write batching for FoundationDB store to improve throughput (#7708)
This addresses issue #7699 where FoundationDB filer store had low throughput
(~400-500 obj/s) due to each write operation creating a separate transaction.
Changes:
- Add writeBatcher that collects multiple writes into batched transactions
- New config options: batch_size (default: 100), batch_interval (default: 5ms)
- Batching provides ~5.7x throughput improvement (from ~456 to ~2600 obj/s)
Benchmark results with different batch sizes:
- batch_size=1: ~456 obj/s (baseline, no batching)
- batch_size=10: ~2621 obj/s (5.7x improvement)
- batch_size=16: ~2514 obj/s (5.5x improvement)
- batch_size=100: ~2617 obj/s (5.7x improvement)
- batch_size=1000: ~2593 obj/s (5.7x improvement)
The batch_interval timer (5ms) ensures writes are flushed promptly even
when batch is not full, providing good latency characteristics.
Addressed review feedback:
- Changed wait=false to wait=true in UpdateEntry/DeleteEntry to properly
propagate errors to callers
- Fixed timer reset race condition by stopping and draining before reset
Fixes #7699
Diffstat (limited to 'go.sum')
| -rw-r--r-- | go.sum | 2 |
1 files changed, 2 insertions, 0 deletions
@@ -653,6 +653,8 @@ github.com/apache/arrow/go/v10 v10.0.1/go.mod h1:YvhnlEePVnBS4+0z3fhPfUy7W1Ikj0I github.com/apache/thrift v0.16.0/go.mod h1:PHK3hniurgQaNMZYaCLEqXKsYK8upmhPbmdP2FXSqgU= github.com/apple/foundationdb/bindings/go v0.0.0-20240515141816-262c6fe778ad h1:fQBkhYv86zyW95PWhzBlkgz3NoY1ue0L+8oYBaoCMbg= github.com/apple/foundationdb/bindings/go v0.0.0-20240515141816-262c6fe778ad/go.mod h1:OMVSB21p9+xQUIqlGizHPZfjK+SHws1ht+ZytVDoz9U= +github.com/apple/foundationdb/bindings/go v0.0.0-20250911184653-27f7192f47c3 h1:WZaTKNHCfcw7fWSR6/RKnCldVzvYZC+Y20Su4lffEIg= +github.com/apple/foundationdb/bindings/go v0.0.0-20250911184653-27f7192f47c3/go.mod h1:OMVSB21p9+xQUIqlGizHPZfjK+SHws1ht+ZytVDoz9U= github.com/appscode/go-querystring v0.0.0-20170504095604-0126cfb3f1dc h1:LoL75er+LKDHDUfU5tRvFwxH0LjPpZN8OoG8Ll+liGU= github.com/appscode/go-querystring v0.0.0-20170504095604-0126cfb3f1dc/go.mod h1:w648aMHEgFYS6xb0KVMMtZ2uMeemhiKCuD2vj6gY52A= github.com/arangodb/go-driver v1.6.9 h1:zckB+xuA16NmHUuYOX7INCJTIyIkoBQjAGqNpiyf2HQ= |