HA Provisioning For Backend Instances For Maxwell




In my previous post I have explain how we are running maxwell as service in Ubuntu. Now in this post I will be explaining how we achieved HA for Backend Instances (MySQL Hosts), Maxwell was using to read binlog.

So the stack we have introduced to achieve high availability contains HAProxy (http://www.haproxy.org/) and GTID enabled slaves. HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications.

We have installed HAProxy on the local machine where we are running Maxwell and configured maxwell to connect to localhost where we have used HAProxy primary-secondary backup strategy using below config (/etc/haproxy/haproxy.cfg):

defaults
       mode tcp
       timeout connect 5000ms
       timeout client 86400s
       timeout server 86400s

listen  MySQL
        bind 127.0.0.1:3306
        mode tcp
        option mysql-check user haproxy  # mysql-check enables a database server check, to determine whether the node is currently operational
server Primary host1:3306 check
server Secondary host2:3306 check backup

Now stack becomes:


This stack has solved our problem incase MySQL slave goes down, Our whole pipeline will not be impacted.

Next Post: We will be publishing on How we achieved HA for Maxwell service. 

Comments

  1. can you please elaborate why were GTID slaves needed in the first place?

    ReplyDelete
  2. Hey Thanks for going through the blog. Maxwell read binlogs using either coordinates (binlog name and position) or GTID. So there might be data mismatch or coordinate not found issue if we do not use GTID enabled slaves and do switchover to secondary slave.

    ReplyDelete

Post a Comment