阅读 19 SEO

kafka重平衡

kafka

rebalance

重新给消费者分配任务的过程

重平衡的是对业务侵入特别严重的,如果消费者很多的话,那么可能会导致集群一定时间内不可用。

发生时机

  1. 主题分区发生变化
  2. 消费端 新增或者减少

怎么去做重平衡

协调着强制要求 消费者重新加入组的过程。 其实就是发 joinGroup请求,把需要的信息带来,并规定这次重平衡的leader,让leader负责把信息制定好,然后分发下去。

Adding servers to a Kafka cluster is easy, just assign them a unique broker id and start up Kafka on your new servers. However these new servers will not automatically be assigned any data partitions, so unless partitions are moved to them they won't be doing any work until new topics are created. So usually when you add machines to your cluster you will want to migrate some existing data to these machines.

The process of migrating data is manually initiated but fully automated. Under the covers what happens is that Kafka will add the new server as a follower of the partition it is migrating and allow it to fully replicate the existing data in that partition. When the new server has fully replicated the contents of this partition and joined the in-sync replica one of the existing replicas will delete their partition's data.

The partition reassignment tool can be used to move partitions across brokers. An ideal partition distribution would ensure even data load and partition sizes across all brokers. The partition reassignment tool does not have the capability to automatically study the data distribution in a Kafka cluster and move partitions around to attain an even load distribution. As such, the admin has to figure out which topics or partitions should be moved around.

参考

  1. https://community.cloudera.com/t5/Support-Questions/Balancing-kafka-topics-when-new-broker-is-added/td-p/175783

为什么快

  1. zero copy
    1. 得益于broker,我们写进去之后,不进入用户态 直接做copy
  2. page cache
  3. batch write
    1. 批量写
  4. 顺序写

作者:lucasgao

原文链接:https://www.jianshu.com/p/90da97e42dda

文章分类
后端
文章标签
版权声明:本文内容由互联网用户自发贡献,该文观点仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌抄袭侵权/违法违规的内容, 请发送邮件至 gxwowoo@163.com 举报,一经查实,本站将立刻删除。
相关推荐