aboutsummaryrefslogtreecommitdiff
path: root/weed/pb/mq.proto
AgeCommit message (Collapse)AuthorFilesLines
2025-01-20Add message queue agent (#6463)Chris Lu1-316/+0
* scaffold message queue agent * adjust proto, add mq_agent * add agent client implementation * remove unused function * agent publish server implementation * adding agent
2024-05-29passing along publisher namechrislu1-1/+1
2024-05-27balance subscriberschrislu1-1/+5
need to ensure the offsets are persisted
2024-05-23coordinator receives unassignment ackchrislu1-0/+4
2024-05-21refactorchrislu1-2/+5
ToDo 1. handle unassignment on client side 2. get notified of completed unassignment 3. in consumer_group.go, balance and send unassignment
2024-05-20unusedchrislu1-5/+0
2024-05-20track offsetchrislu1-0/+2
2024-05-19start consuming ASAPchrislu1-0/+9
2024-05-19persist consumer group offsetchrislu1-3/+5
1. use one follower 2. read write consumer group offset
2024-05-17add subscribe followerchrislu1-1/+22
2024-05-13consumer instance passing MaxPartitionCount to coordinatorchrislu1-0/+1
2024-04-12add proto for valuechrislu1-0/+4
2024-03-31add control message to data messagechrislu1-3/+6
2024-03-31send is_close message to brokerchrislu1-0/+1
2024-03-31add publisher name for debuggingchrislu1-0/+1
2024-03-27simplifychrislu1-2/+2
2024-03-24setup follower by publisherchrislu1-6/+2
* the subscriber would getOrGen a local partition and wait * the publisher would getOrGen a local partition. If localPartition follower is not setup, and init message has follower info, it would create followers based on init.Messages.
2024-03-20publisher is able to fully send data and receive ackschrislu1-5/+17
still needs to close the pipes cleanly
2024-03-17remove FollowInMemoryMessageschrislu1-28/+0
2024-03-10ask follower to followchrislu1-0/+39
2024-02-29publish init add followerschrislu1-0/+1
2024-01-16remove isForPublish from LookupTopicBrokerschrislu1-1/+0
also adds a return parameter: whether the topic exists or not
2024-01-08add batch index for each memory bufferchrislu1-1/+9
2024-01-05subscribe with partition offsetchrislu1-6/+2
2024-01-05rename functionschrislu1-6/+6
2024-01-05add offsetchrislu1-0/+10
2023-12-28subscriber can get assignmentschrislu1-0/+1
2023-12-28subscriber keep connected to the balancerchrislu1-1/+1
2023-12-11Merge accumulated changes related to message queue (#5098)Chris Lu1-82/+66
* balance partitions on brokers * prepare topic partition first and then publish, move partition * purge unused APIs * clean up * adjust logs * add BalanceTopics() grpc API * configure topic * configure topic command * refactor * repair missing partitions * sequence of operations to ensure ordering * proto to close publishers and consumers * rename file * topic partition versioned by unixTimeNs * create local topic partition * close publishers * randomize the client name * wait until no publishers * logs * close stop publisher channel * send last ack * comments * comment * comments * support list of brokers * add cli options * Update .gitignore * logs * return io.eof directly * refactor * optionally create topic * refactoring * detect consumer disconnection * sub client wait for more messages * subscribe by time stamp * rename * rename to sub_balancer * rename * adjust comments * rename * fix compilation * rename * rename * SubscriberToSubCoordinator * sticky rebalance * go fmt * add tests * balance partitions on brokers * prepare topic partition first and then publish, move partition * purge unused APIs * clean up * adjust logs * add BalanceTopics() grpc API * configure topic * configure topic command * refactor * repair missing partitions * sequence of operations to ensure ordering * proto to close publishers and consumers * rename file * topic partition versioned by unixTimeNs * create local topic partition * close publishers * randomize the client name * wait until no publishers * logs * close stop publisher channel * send last ack * comments * comment * comments * support list of brokers * add cli options * Update .gitignore * logs * return io.eof directly * refactor * optionally create topic * refactoring * detect consumer disconnection * sub client wait for more messages * subscribe by time stamp * rename * rename to sub_balancer * rename * adjust comments * rename * fix compilation * rename * rename * SubscriberToSubCoordinator * sticky rebalance * go fmt * add tests * tracking topic=>broker * merge * comment
2023-10-12Revert "rename"chrislu1-3/+3
This reverts commit 35b5264ab77fcc0e2e11ad72e92aabe82a567a8b.
2023-10-11renamechrislu1-3/+3
2023-09-30adjust mq.protochrislu1-9/+6
2023-09-30adjust mq protochrislu1-10/+17
2023-09-26rename functionchrislu1-6/+6
2023-09-24create topic and report topicorigin/subchrislu1-0/+7
2023-09-24do create topicchrislu1-0/+8
2023-09-24add CreateTopic APIchrislu1-0/+9
2023-09-19wipchrislu1-3/+10
2023-09-16balancer workschrislu1-4/+2
2023-09-14receive broker statschrislu1-0/+23
2023-09-06ack intervalchrislu1-0/+1
2023-09-04api for subchrislu1-0/+2
2023-09-01can pub and subchrislu1-3/+11
2023-08-27rename to lookupchrislu1-17/+12
2023-08-27pub sub initial testschrislu1-1/+1
2023-08-27subchrislu1-6/+3
2023-08-26adjust protochrislu1-5/+29
2023-08-20Squashed commit of the following:chrislu1-3/+71
commit 32f4b1a13057d56b6de487cdb80ff7c205af01a6 Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 20 22:52:19 2023 -0700 fix compilation commit e77ad33b7ca0423138fbae26a4433b60923a9588 Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 20 22:46:44 2023 -0700 pub commit f431f30cc7ca277ca299e3cd118c05537fb9f5c3 Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 20 13:27:39 2023 -0700 fix generic type commit 4e9dcb18293fd1e3e306e2dceb995dfd67a35e1d Merge: 30f942580 16e3f2d52 Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 20 12:47:14 2023 -0700 Merge branch 'master' into pubsub commit 30f942580ad1bb32ae94aade2e3a21ec3ab63e21 Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 20 11:10:58 2023 -0700 wip commit f8b00980bc2f3879bb43decffd9a08d842f196f2 Author: chrislu <chris.lu@gmail.com> Date: Tue Jul 25 09:14:35 2023 -0700 add design document commit 08d2bebe42a26ebc39f1542f54d99e73620727dd Author: chrislu <chris.lu@gmail.com> Date: Tue Jul 25 09:14:06 2023 -0700 minor commit bcfa7982b262a40fcdce6fc6613fad2ce07c13da Author: chrislu <chris.lu@gmail.com> Date: Tue Jul 25 09:13:49 2023 -0700 rename
2022-07-31scaffoldchrislu1-0/+17
2022-07-29move to https://github.com/seaweedfs/seaweedfschrislu1-1/+1
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-25 12:58:58 +0000