aboutsummaryrefslogtreecommitdiff
path: root/weed/mq
AgeCommit message (Collapse)AuthorFilesLines
2025-07-16convert error fromating to %w everywhere (#6995)Chris Lu17-49/+49
2025-07-11Admin UI: Add message queue to admin UI (#6958)Chris Lu7-28/+445
* add a menu item "Message Queue" * add a menu item "Message Queue" * move the "brokers" link under it. * add "topics", "subscribers". Add pages for them. * refactor * show topic details * admin display publisher and subscriber info * remove publisher and subscribers from the topic row pull down * collecting more stats from publishers and subscribers * fix layout * fix publisher name * add local listeners for mq broker and agent * render consumer group offsets * remove subscribers from left menu * topic with retention * support editing topic retention * show retention when listing topics * create bucket * Update s3_buckets_templ.go * embed the static assets into the binary fix https://github.com/seaweedfs/seaweedfs/issues/6964
2025-06-23chore(deps): bump github.com/parquet-go/parquet-go from 0.24.0 to 0.25.1 (#6851)dependabot[bot]2-18/+41
* chore(deps): bump github.com/parquet-go/parquet-go from 0.24.0 to 0.25.1 Bumps [github.com/parquet-go/parquet-go](https://github.com/parquet-go/parquet-go) from 0.24.0 to 0.25.1. - [Release notes](https://github.com/parquet-go/parquet-go/releases) - [Changelog](https://github.com/parquet-go/parquet-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/parquet-go/parquet-go/compare/v0.24.0...v0.25.1) --- updated-dependencies: - dependency-name: github.com/parquet-go/parquet-go dependency-version: 0.25.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * adjust to updated API Fixed Reader Construction: Updated to use parquet.OpenFile() instead of passing io.Reader directly to NewReader() Fixed EOF Handling: Changed the order of operations to process rows before checking for EOF Added Zero Row Count Check: Added explicit check for rowCount == 0 as an additional termination condition --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: chrislu <chris.lu@gmail.com>
2025-05-28Add context with request (#6824)Aleksey Kosov3-4/+4
2025-05-22added context to filer_client method calls (#6808)Aleksey Kosov4-9/+13
Co-authored-by: akosov <a.kosov@kryptonite.ru>
2025-03-31chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 (#6679)dependabot[bot]2-2/+4
* chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 Bumps [gocloud.dev](https://github.com/google/go-cloud) from 0.40.0 to 0.41.0. - [Release notes](https://github.com/google/go-cloud/releases) - [Commits](https://github.com/google/go-cloud/compare/v0.40.0...v0.41.0) --- updated-dependencies: - dependency-name: gocloud.dev dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * fix error * fix printing errors * Update go.mod --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: chrislu <chris.lu@gmail.com>
2025-03-09Accumulated changes for message queue (#6600)Chris Lu25-718/+257
* rename * set agent address * refactor * add agent sub * pub messages * grpc new client * can publish records via agent * send init message with session id * fmt * check cancelled request while waiting * use sessionId * handle possible nil stream * subscriber process messages * separate debug port * use atomic int64 * less logs * minor * skip io.EOF * rename * remove unused * use saved offsets * do not reuse session, since always session id is new after restart remove last active ts from SessionEntry * simplify printing * purge unused * just proxy the subscription, skipping the session step * adjust offset types * subscribe offset type and possible value * start after the known tsns * avoid wrongly set startPosition * move * remove * refactor * typo * fix * fix changed path
2025-03-04adjust logschrislu1-3/+3
2025-02-25minorchrislu1-1/+2
2025-02-18Delete broker.gochrislu1-12/+0
2025-01-20Add message queue agent (#6463)Chris Lu31-120/+729
* scaffold message queue agent * adjust proto, add mq_agent * add agent client implementation * remove unused function * agent publish server implementation * adding agent
2024-12-19"golang.org/x/exp/slices" => "slices" and go fmtchrislu7-22/+22
2024-11-04merge current message queue code changes (#6201)Chris Lu23-189/+916
* listing files to convert to parquet * write parquet files * save logs into parquet files * pass by value * compact logs into parquet format * can skip existing files * refactor * refactor * fix compilation * when no partition found * refactor * add untested parquet file read * rename package * refactor * rename files * remove unused * add merged log read func * parquet wants to know the file size * rewind by time * pass in stop ts * add stop ts * adjust log * minor * adjust log * skip .parquet files when reading message logs * skip non message files * Update subscriber_record.go * send messages * skip message data with only ts * skip non log files * update parquet-go package * ensure a valid record type * add new field to a record type * Update read_parquet_to_log.go * fix parquet file name generation * separating reading parquet and logs * add key field * add skipped logs * use in memory cache * refactor * refactor * refactor * refactor, and change compact log * refactor * rename * refactor * fix format * prefix v to version directory
2024-10-03add helper functionschrislu2-1/+39
2024-08-12fix buildchrislu1-1/+1
2024-08-12fix testschrislu1-52/+6
2024-08-11adds lockingchrislu3-7/+15
2024-08-10refactorchrislu2-4/+7
2024-08-10Merge branch 'master' into mqchrislu6-2/+17
2024-07-16Added tls for http clients (#5766)vadimartynov6-2/+17
* Added global http client * Added Do func for global http client * Changed the code to use the global http client * Fix http client in volume uploader * Fixed pkg name * Fixed http util funcs * Fixed http client for bench_filer_upload * Fixed http client for stress_filer_upload * Fixed http client for filer_server_handlers_proxy * Fixed http client for command_fs_merge_volumes * Fixed http client for command_fs_merge_volumes and command_volume_fsck * Fixed http client for s3api_server * Added init global client for main funcs * Rename global_client to client * Changed: - fixed NewHttpClient; - added CheckIsHttpsClientEnabled func - updated security.toml in scaffold * Reduce the visibility of some functions in the util/http/client pkg * Added the loadSecurityConfig function * Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-06-14sort importschrislu1-1/+1
2024-06-14Merge branch 'master' into mqchrislu1-2/+3
2024-06-14Added context for the MasterClient's methods to avoid endless loops (#5628)vadimartynov1-2/+3
* Added context for the MasterClient's methods to avoid endless loops * Returned WithClient function. Added WithClientCustomGetMaster function * Hid unused ctx arguments * Using a common context for the KeepConnectedToMaster and WaitUntilConnected functions * Changed the context termination check in the tryConnectToMaster function * Added a child context to the tryConnectToMaster function * Added a common context for KeepConnectedToMaster and WaitUntilConnected functions in benchmark
2024-06-06refactor out FilerClientAccessorchrislu4-70/+8
2024-05-30remove per-message debug logschrislu3-6/+5
2024-05-30fix size calculationchrislu2-1/+15
2024-05-30debugchrislu2-2/+6
2024-05-30fix passing sub follower offsetchrislu1-1/+1
2024-05-30fix cases when buffer wraps aroundchrislu2-5/+31
2024-05-30fixchrislu1-1/+0
2024-05-30stop partitionOffsetChan if closedchrislu1-0/+5
2024-05-29skip control messageschrislu2-2/+8
2024-05-29fix inflight message trackerchrislu3-87/+124
2024-05-29passing along publisher namechrislu1-0/+1
2024-05-29avoid nil stopChanchrislu1-2/+3
2024-05-29less logschrislu1-10/+9
2024-05-27balance subscriberschrislu11-146/+141
need to ensure the offsets are persisted
2024-05-26add marketchrislu2-0/+449
2024-05-26use type ConsumerGroupInstanceIdchrislu4-13/+13
2024-05-23refactorchrislu2-19/+26
2024-05-23go fmtchrislu5-19/+19
2024-05-23coordinator receives unassignment ackchrislu5-16/+45
2024-05-21renamechrislu10-29/+29
2024-05-21rename filechrislu1-0/+0
2024-05-21rename Coordinator to SubCoordinatorchrislu2-12/+12
2024-05-21rename filechrislu1-0/+0
2024-05-21rename Balancer to PubBalancerchrislu8-21/+21
2024-05-21client side stop partition subscribing if unassignedchrislu2-12/+27
2024-05-21fix compilationchrislu1-7/+2
2024-05-21refactorchrislu3-50/+52
ToDo 1. handle unassignment on client side 2. get notified of completed unassignment 3. in consumer_group.go, balance and send unassignment
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-24 18:22:30 +0000