Orchestrator RAFT Leader Check with Proxy pass with Basic Auth Using Nginx



Recently we have setup Orchestrator in High Availability mode using RAFT. We are running a 3 node setup in which there used to be a leader and rest 2 are Healthy raft member.


So To access orchestrator service we may only speak to the leader node using /api/leader-check as HTTP health check for our proxy. This url returns http 200 on leader and 404 on members. So using below code in open nginx we have setup http health check with basic auth.
Prerequisite: Lua support should be enabled in nginx.
Below code is to define upstreams with healthcheck:
upstream orchestrator { server 10.xx.xx.35:3000 max_fails=2; server 10.xx.xx.37:3000 max_fails=2; server 10.xx.xx.40:3000 max_fails=2; } lua_shared_dict myhealthcheck 1m; lua_socket_log_errors off; include /etc/nginx/lua/active_health_checks.lua;
Lua Script for health check: 
Before creating script we will need a hash with base64 encoding below is the command to create it:
echo -n 'user:password' | base64
Lua script defined in HealthCheck: /etc/nginx/lua/active_health_checks.lua
init_worker_by_lua_block { local hc = require "resty.upstream.healthcheck" local ok, err = hc.spawn_checker{ shm = "myhealthcheck", -- defined by "lua_shared_dict" upstream = "orchestrator", -- defined by "upstream" type = "http", http_req = "GET /api/leader-check HTTP/1.0\r\nHost: orchestrator.domain.com\r\nAuthorization: Basic {output of command}\r\n\r\n", interval = 3000, -- run the check cycle every 3 sec timeout = 3000, -- 3 sec is the timeout for network operations fall = 3, -- # of successive failures before turning a peer down rise = 2, -- # of successive successes before turning a peer up valid_statuses = {200, 302, 301}, -- a list valid HTTP status code concurrency = 10, -- concurrency level for test requests } if not ok then ngx.log(ngx.ERR, "failed to spawn health checker: ", err) return end }
Note: I am quite new to Nginx So I would love to see comments in improving it.

Comments

  1. Looks good! (Though I'm not very knowledgeable about Nginx and Lua).
    It's worth noting that in a Raft setup, even if you do happen to make a request to a non-leader node, the node you communicate with will forward the request to the leader. This works when the node you communicate with can _see_ the leader. It's best to communicate to the leader directly.

    ReplyDelete
    Replies
    1. Thanks much for the clarification! Please keep guiding us.

      Delete

Post a Comment

Popular posts from this blog

Shell/Bash Commands Execution from MySQL Client or Stored Procedure/function

How to remove/deregister an instance from PMM?

MySQL: How to monitor MySQL Replication Lag in MilliSeconds With PMM and pt-heartbeat