aboutsummaryrefslogtreecommitdiff
path: root/weed/mq/broker
AgeCommit message (Collapse)AuthorFilesLines
2025-07-16convert error fromating to %w everywhere (#6995)Chris Lu6-11/+11
2025-07-11Admin UI: Add message queue to admin UI (#6958)Chris Lu4-23/+275
* add a menu item "Message Queue" * add a menu item "Message Queue" * move the "brokers" link under it. * add "topics", "subscribers". Add pages for them. * refactor * show topic details * admin display publisher and subscriber info * remove publisher and subscribers from the topic row pull down * collecting more stats from publishers and subscribers * fix layout * fix publisher name * add local listeners for mq broker and agent * render consumer group offsets * remove subscribers from left menu * topic with retention * support editing topic retention * show retention when listing topics * create bucket * Update s3_buckets_templ.go * embed the static assets into the binary fix https://github.com/seaweedfs/seaweedfs/issues/6964
2025-05-22added context to filer_client method calls (#6808)Aleksey Kosov1-2/+3
Co-authored-by: akosov <a.kosov@kryptonite.ru>
2025-03-31chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 (#6679)dependabot[bot]2-2/+4
* chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 Bumps [gocloud.dev](https://github.com/google/go-cloud) from 0.40.0 to 0.41.0. - [Release notes](https://github.com/google/go-cloud/releases) - [Commits](https://github.com/google/go-cloud/compare/v0.40.0...v0.41.0) --- updated-dependencies: - dependency-name: gocloud.dev dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * fix error * fix printing errors * Update go.mod --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: chrislu <chris.lu@gmail.com>
2025-03-09Accumulated changes for message queue (#6600)Chris Lu1-7/+39
* rename * set agent address * refactor * add agent sub * pub messages * grpc new client * can publish records via agent * send init message with session id * fmt * check cancelled request while waiting * use sessionId * handle possible nil stream * subscriber process messages * separate debug port * use atomic int64 * less logs * minor * skip io.EOF * rename * remove unused * use saved offsets * do not reuse session, since always session id is new after restart remove last active ts from SessionEntry * simplify printing * purge unused * just proxy the subscription, skipping the session step * adjust offset types * subscribe offset type and possible value * start after the known tsns * avoid wrongly set startPosition * move * remove * refactor * typo * fix * fix changed path
2025-02-25minorchrislu1-1/+2
2025-01-20Add message queue agent (#6463)Chris Lu4-9/+8
* scaffold message queue agent * adjust proto, add mq_agent * add agent client implementation * remove unused function * agent publish server implementation * adding agent
2024-11-04merge current message queue code changes (#6201)Chris Lu6-150/+11
* listing files to convert to parquet * write parquet files * save logs into parquet files * pass by value * compact logs into parquet format * can skip existing files * refactor * refactor * fix compilation * when no partition found * refactor * add untested parquet file read * rename package * refactor * rename files * remove unused * add merged log read func * parquet wants to know the file size * rewind by time * pass in stop ts * add stop ts * adjust log * minor * adjust log * skip .parquet files when reading message logs * skip non message files * Update subscriber_record.go * send messages * skip message data with only ts * skip non log files * update parquet-go package * ensure a valid record type * add new field to a record type * Update read_parquet_to_log.go * fix parquet file name generation * separating reading parquet and logs * add key field * add skipped logs * use in memory cache * refactor * refactor * refactor * refactor, and change compact log * refactor * rename * refactor * fix format * prefix v to version directory
2024-08-11adds lockingchrislu1-3/+0
2024-08-10Merge branch 'master' into mqchrislu2-2/+9
2024-07-16Added tls for http clients (#5766)vadimartynov2-2/+9
* Added global http client * Added Do func for global http client * Changed the code to use the global http client * Fix http client in volume uploader * Fixed pkg name * Fixed http util funcs * Fixed http client for bench_filer_upload * Fixed http client for stress_filer_upload * Fixed http client for filer_server_handlers_proxy * Fixed http client for command_fs_merge_volumes * Fixed http client for command_fs_merge_volumes and command_volume_fsck * Fixed http client for s3api_server * Added init global client for main funcs * Rename global_client to client * Changed: - fixed NewHttpClient; - added CheckIsHttpsClientEnabled func - updated security.toml in scaffold * Reduce the visibility of some functions in the util/http/client pkg * Added the loadSecurityConfig function * Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-06-14sort importschrislu1-1/+1
2024-06-14Merge branch 'master' into mqchrislu1-2/+3
2024-06-14Added context for the MasterClient's methods to avoid endless loops (#5628)vadimartynov1-2/+3
* Added context for the MasterClient's methods to avoid endless loops * Returned WithClient function. Added WithClientCustomGetMaster function * Hid unused ctx arguments * Using a common context for the KeepConnectedToMaster and WaitUntilConnected functions * Changed the context termination check in the tryConnectToMaster function * Added a child context to the tryConnectToMaster function * Added a common context for KeepConnectedToMaster and WaitUntilConnected functions in benchmark
2024-06-06refactor out FilerClientAccessorchrislu1-2/+3
2024-05-30remove per-message debug logschrislu2-3/+3
2024-05-30debugchrislu2-2/+6
2024-05-30fix passing sub follower offsetchrislu1-1/+1
2024-05-29skip control messageschrislu1-2/+4
2024-05-29fix inflight message trackerchrislu1-13/+18
2024-05-27balance subscriberschrislu3-5/+16
need to ensure the offsets are persisted
2024-05-23go fmtchrislu1-3/+3
2024-05-23coordinator receives unassignment ackchrislu1-1/+6
2024-05-21renamechrislu8-27/+27
2024-05-21rename Coordinator to SubCoordinatorchrislu1-1/+1
2024-05-21rename Balancer to PubBalancerchrislu1-1/+1
2024-05-20avoid concurrent processing for the same keychrislu1-0/+4
2024-05-20fixchrislu1-0/+3
2024-05-20go fmtchrislu3-8/+6
2024-05-20Merge branch 'master' into mqchrislu4-10/+7
2024-05-20go fmtchrislu4-10/+7
2024-05-20adjust logschrislu1-1/+1
2024-05-20track offsetchrislu1-2/+6
2024-05-19less logschrislu1-1/+1
2024-05-19logschrislu1-1/+1
2024-05-19persist consumer group offsetchrislu5-31/+113
1. use one follower 2. read write consumer group offset
2024-05-17add subscribe followerchrislu2-0/+86
2024-05-16consumer acks received messageschrislu2-8/+25
2024-05-16adjust errorschrislu2-4/+5
2024-05-14move initial assignment to rebalance logicchrislu2-15/+7
2024-05-14move read write topic config into filer client accessorchrislu4-53/+13
2024-05-13refactorchrislu1-2/+2
2024-04-12save schema when configuring topicchrislu1-0/+10
2024-04-02refactorchrislu1-1/+1
2024-04-02comment out printlnchrislu2-7/+4
2024-04-01shutdown follower when leader stopschrislu1-6/+107
2024-04-01send flush message to follower before shutting down logBufferchrislu2-0/+14
2024-04-01renamingchrislu4-7/+7
2024-03-31add control message to data messagechrislu1-0/+3
2024-03-31add publisher name for debuggingchrislu1-4/+4