aboutsummaryrefslogtreecommitdiff
path: root/weed/mq/client
AgeCommit message (Collapse)AuthorFilesLines
2025-07-16convert error fromating to %w everywhere (#6995)Chris Lu5-13/+13
2025-03-09Accumulated changes for message queue (#6600)Chris Lu15-593/+127
* rename * set agent address * refactor * add agent sub * pub messages * grpc new client * can publish records via agent * send init message with session id * fmt * check cancelled request while waiting * use sessionId * handle possible nil stream * subscriber process messages * separate debug port * use atomic int64 * less logs * minor * skip io.EOF * rename * remove unused * use saved offsets * do not reuse session, since always session id is new after restart remove last active ts from SessionEntry * simplify printing * purge unused * just proxy the subscription, skipping the session step * adjust offset types * subscribe offset type and possible value * start after the known tsns * avoid wrongly set startPosition * move * remove * refactor * typo * fix * fix changed path
2025-01-20Add message queue agent (#6463)Chris Lu10-81/+389
* scaffold message queue agent * adjust proto, add mq_agent * add agent client implementation * remove unused function * agent publish server implementation * adding agent
2024-12-19"golang.org/x/exp/slices" => "slices" and go fmtchrislu3-3/+3
2024-11-04merge current message queue code changes (#6201)Chris Lu4-8/+33
* listing files to convert to parquet * write parquet files * save logs into parquet files * pass by value * compact logs into parquet format * can skip existing files * refactor * refactor * fix compilation * when no partition found * refactor * add untested parquet file read * rename package * refactor * rename files * remove unused * add merged log read func * parquet wants to know the file size * rewind by time * pass in stop ts * add stop ts * adjust log * minor * adjust log * skip .parquet files when reading message logs * skip non message files * Update subscriber_record.go * send messages * skip message data with only ts * skip non log files * update parquet-go package * ensure a valid record type * add new field to a record type * Update read_parquet_to_log.go * fix parquet file name generation * separating reading parquet and logs * add key field * add skipped logs * use in memory cache * refactor * refactor * refactor * refactor, and change compact log * refactor * rename * refactor * fix format * prefix v to version directory
2024-08-12fix buildchrislu1-1/+1
2024-08-10Merge branch 'master' into mqchrislu4-0/+8
2024-07-16Added tls for http clients (#5766)vadimartynov4-0/+8
* Added global http client * Added Do func for global http client * Changed the code to use the global http client * Fix http client in volume uploader * Fixed pkg name * Fixed http util funcs * Fixed http client for bench_filer_upload * Fixed http client for stress_filer_upload * Fixed http client for filer_server_handlers_proxy * Fixed http client for command_fs_merge_volumes * Fixed http client for command_fs_merge_volumes and command_volume_fsck * Fixed http client for s3api_server * Added init global client for main funcs * Rename global_client to client * Changed: - fixed NewHttpClient; - added CheckIsHttpsClientEnabled func - updated security.toml in scaffold * Reduce the visibility of some functions in the util/http/client pkg * Added the loadSecurityConfig function * Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-05-30fixchrislu1-1/+0
2024-05-30stop partitionOffsetChan if closedchrislu1-0/+5
2024-05-29skip control messageschrislu1-0/+4
2024-05-29passing along publisher namechrislu1-0/+1
2024-05-27balance subscriberschrislu3-2/+10
need to ensure the offsets are persisted
2024-05-23go fmtchrislu4-16/+16
2024-05-23coordinator receives unassignment ackchrislu3-15/+33
2024-05-21client side stop partition subscribing if unassignedchrislu2-12/+27
2024-05-21fix compilationchrislu1-7/+2
2024-05-21refactorchrislu2-32/+37
ToDo 1. handle unassignment on client side 2. get notified of completed unassignment 3. in consumer_group.go, balance and send unassignment
2024-05-20refactorchrislu2-104/+112
2024-05-20remove ProcessorConfigurationchrislu5-22/+17
2024-05-20sending keyed offsetchrislu1-3/+12
2024-05-20minorchrislu1-7/+7
2024-05-20go fmtchrislu6-27/+26
2024-05-20Merge branch 'master' into mqchrislu3-3/+3
2024-05-20go fmtchrislu4-4/+4
2024-05-20track offsetchrislu5-6/+9
2024-05-19fixchrislu1-1/+1
2024-05-19ConcurrentPartitionLimitchrislu3-15/+23
2024-05-19start consuming ASAPchrislu1-1/+1
2024-05-19persist consumer group offsetchrislu3-6/+2
1. use one follower 2. read write consumer group offset
2024-05-16consumer acks received messageschrislu2-13/+32
2024-05-14subscriber receives partitions and dispatch to processorschrislu3-36/+94
2024-05-13consumer instance passing MaxPartitionCount to coordinatorchrislu2-1/+2
2024-05-10clean up unused variableschrislu3-5/+0
2024-05-05add delaychrislu1-2/+5
2024-05-05reset last errorchrislu1-0/+1
2024-05-02renamechrislu1-4/+4
2024-05-02subscriber can unmarshal the recordchrislu2-4/+94
2024-05-02renamechrislu1-0/+0
2024-05-02rename field typeschrislu1-1/+1
2024-05-02consistent with protobuf data typeschrislu1-8/+8
2024-05-02add example to publish a recordchrislu2-1/+134
2024-05-02renamechrislu1-0/+1
2024-04-15Merge branch 'master' into mq-subscribechrislu1-1/+1
2024-04-14chore: fix some typos in comments (#5497)yudrywet1-1/+1
2024-04-12publisher send schemachrislu3-0/+15
2024-04-12rename parameterchrislu3-11/+9
2024-03-31add control message to data messagechrislu2-2/+4
2024-03-31send is_close message to brokerchrislu3-1/+20
2024-03-31add publisher name for debuggingchrislu3-5/+11