aboutsummaryrefslogtreecommitdiff
path: root/weed/mq/pub_balancer
AgeCommit message (Collapse)AuthorFilesLines
2025-01-20Add message queue agent (#6463)Chris Lu6-15/+20
* scaffold message queue agent * adjust proto, add mq_agent * add agent client implementation * remove unused function * agent publish server implementation * adding agent
2024-12-19"golang.org/x/exp/slices" => "slices" and go fmtchrislu1-12/+12
2024-11-04merge current message queue code changes (#6201)Chris Lu1-2/+2
* listing files to convert to parquet * write parquet files * save logs into parquet files * pass by value * compact logs into parquet format * can skip existing files * refactor * refactor * fix compilation * when no partition found * refactor * add untested parquet file read * rename package * refactor * rename files * remove unused * add merged log read func * parquet wants to know the file size * rewind by time * pass in stop ts * add stop ts * adjust log * minor * adjust log * skip .parquet files when reading message logs * skip non message files * Update subscriber_record.go * send messages * skip message data with only ts * skip non log files * update parquet-go package * ensure a valid record type * add new field to a record type * Update read_parquet_to_log.go * fix parquet file name generation * separating reading parquet and logs * add key field * add skipped logs * use in memory cache * refactor * refactor * refactor * refactor, and change compact log * refactor * rename * refactor * fix format * prefix v to version directory
2024-08-12fix testschrislu1-52/+6
2024-05-27balance subscriberschrislu1-4/+0
need to ensure the offsets are persisted
2024-05-21renamechrislu1-1/+1
2024-05-21rename filechrislu1-0/+0
2024-05-21rename Balancer to PubBalancerchrislu5-16/+16
2024-05-20Merge branch 'master' into mqchrislu1-3/+3
2024-05-20go fmtchrislu1-3/+3
2024-05-19persist consumer group offsetchrislu3-56/+24
1. use one follower 2. read write consumer group offset
2024-04-28fix testchrislu2-3/+0
2024-03-27simplifychrislu3-26/+17
2024-03-16fix testchrislu1-1/+0
2024-03-16go fmtchrislu2-23/+23
2024-03-07edge cases for broker assignmentchrislu2-9/+23
2024-03-01handle single broker casechrislu2-2/+37
2024-02-29test low active brokerschrislu2-1/+24
2024-02-29assign followerschrislu2-22/+211
2024-02-29go fmtchrislu2-9/+9
2024-02-04fix divide by zeroChris Lu1-7/+11
2024-01-22updated and added assignmentschrislu1-6/+11
2024-01-22add/remove assigned partitionschrislu1-2/+6
2024-01-21fix assignments if brokers changedchrislu1-0/+20
2024-01-21refactorchrislu2-3/+12
2024-01-20refactorchrislu1-0/+10
2024-01-20refactorchrislu1-2/+6
2024-01-16simplify to LookupTopicPartitions(topic)chrislu1-25/+1
2024-01-16simplify apichrislu1-2/+5
2024-01-16remove isForPublish from LookupTopicBrokerschrislu1-3/+3
also adds a return parameter: whether the topic exists or not
2024-01-16renamechrislu1-1/+1
2024-01-16renamechrislu2-2/+2
2024-01-10passing partition generation timestampchrislu4-3/+20
2024-01-03subscriber can be notified of the assignment change when topic is just ↵chrislu2-2/+5
configured Next: Subscriber needs to read by the timestamp offset.
2024-01-02lookup existing topic partitionschrislu1-3/+2
2023-12-28subscriber can get assignmentschrislu1-4/+5
2023-12-22go fmtchrislu2-6/+7
2023-12-11fix testchrislu1-3/+9
2023-12-11Merge accumulated changes related to message queue (#5098)Chris Lu13-0/+898
* balance partitions on brokers * prepare topic partition first and then publish, move partition * purge unused APIs * clean up * adjust logs * add BalanceTopics() grpc API * configure topic * configure topic command * refactor * repair missing partitions * sequence of operations to ensure ordering * proto to close publishers and consumers * rename file * topic partition versioned by unixTimeNs * create local topic partition * close publishers * randomize the client name * wait until no publishers * logs * close stop publisher channel * send last ack * comments * comment * comments * support list of brokers * add cli options * Update .gitignore * logs * return io.eof directly * refactor * optionally create topic * refactoring * detect consumer disconnection * sub client wait for more messages * subscribe by time stamp * rename * rename to sub_balancer * rename * adjust comments * rename * fix compilation * rename * rename * SubscriberToSubCoordinator * sticky rebalance * go fmt * add tests * balance partitions on brokers * prepare topic partition first and then publish, move partition * purge unused APIs * clean up * adjust logs * add BalanceTopics() grpc API * configure topic * configure topic command * refactor * repair missing partitions * sequence of operations to ensure ordering * proto to close publishers and consumers * rename file * topic partition versioned by unixTimeNs * create local topic partition * close publishers * randomize the client name * wait until no publishers * logs * close stop publisher channel * send last ack * comments * comment * comments * support list of brokers * add cli options * Update .gitignore * logs * return io.eof directly * refactor * optionally create topic * refactoring * detect consumer disconnection * sub client wait for more messages * subscribe by time stamp * rename * rename to sub_balancer * rename * adjust comments * rename * fix compilation * rename * rename * SubscriberToSubCoordinator * sticky rebalance * go fmt * add tests * tracking topic=>broker * merge * comment
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-25 09:50:35 +0000