HA centos 7 configuration

Physical node OS

  • CentOS Linux release 7.3

Create linux HA ,3 x physical server

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.xx.xx.130 node01
192.xx.xx.131 node02
192.xx.xx.132 node03

Share strorages

NFS/Lustre

Setup HA cluster

1
2
3
# yum -y install pacemaker corosync fence-agents-all fence-agents-virsh \
fence-virt pacemaker-remote pcs fence-virtd resource-agents \
fence-virtd-libvirt fence-virtd-multicast

3 nodes glusterfs config

1
2
3
4
$ zpool create tank -O canmount=on -o ashift=9 -o cachefile=none -O xattr=sa -O compression=lz4 -O acltype=posixacl raidz2 /dev/sd{c..h}
$ zpool add tank cache /dev/sda4 /dev/sdb4
$ zpool add tank log mirror /dev/sda3 /dev/sdb3
$ gluster volume create ec-test disperse-data 2 redundancy 1 transport tcp node1:/tank/glusterfs node2:/tank/glusterfs node3:/tank/glusterfs force

Setup physical server

1
2
3
4
5
6
7
8
9
10
$ echo hacluster | passwd --stdin hacluster
$ systemctl enable pcsd.service
$ systemctl disable corosync.service pcaemaker.service
$ systemctl stop corosync.service pcaemaker.service
$ systemctl start pcsd.service
$ systemctl is-active pcsd.service
$ pcs cluster auth node01 node02 node03 ##input hacluster password for auth
$ pcs cluster setup --name ha-cluster node01,172.29.xx.xx node02,172.92.xx.xx node03,172.29.xx.xx --transport udpu --rrpmode passive --token 17000 --addr0 172.29.xx.0 --addr1 192.168.101.0
$ pcs cluster setup --start --name ha-cluster node01 node02 node03
## pcs property set no-quorum-policy=stop #don't need to stop

4.1.2 Option no-quorum-policy

This global option defines what to do when a cluster partition does not have quorum (no majority of nodes is part of the partition).

Allowed values are:

ignore
The quorum state does not influence the cluster behavior; resource management is continued.

This setting is useful for the following scenarios:

Two-node clusters: Since a single node failure would always result in a loss of majority, usually you want the cluster to carry on regardless. Resource integrity is ensured using fencing, which also prevents split brain scenarios.

Resource-driven clusters: For local clusters with redundant communication channels, a split brain scenario only has a certain probability. Thus, a loss of communication with a node most likely indicates that the node has crashed, and that the surviving nodes should recover and start serving the resources again.

If no-quorum-policy is set to ignore, a 4-node cluster can sustain concurrent failure of three nodes before service is lost. With the other settings, it would lose quorum after concurrent failure of two nodes.

freeze
If quorum is lost, the cluster partition freezes. Resource management is continued: running resources are not stopped (but possibly restarted in response to monitor events), but no further resources are started within the affected partition.

This setting is recommended for clusters where certain resources depend on communication with other nodes (for example, OCFS2 mounts). In this case, the default setting no-quorum-policy=stop is not useful, as it would lead to the following scenario: Stopping those resources would not be possible while the peer nodes are unreachable. Instead, an attempt to stop them would eventually time out and cause a stop failure, triggering escalated recovery and fencing.

stop (default value)
If quorum is lost, all resources in the affected cluster partition are stopped in an orderly fashion.

suicide
If quorum is lost, all nodes in the affected cluster partition are fenced.

Setting /etc/corosync/corosync.conf for every node

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
totem {
version: 2
secauth: off
cluster_name: homer-test
transport: udpu
rrp_mode: passive
token: 17000

interface {
ringnumber: 0
bindnetaddr: 192.168.xx.0
mcastaddr: 239.255.1.1
mcastport: 5405
}

interface {
ringnumber: 1
bindnetaddr: 172.29.xx.0
mcastaddr: 239.255.2.1
mcastport: 5405
}
}

nodelist {
node {
ring0_addr: bgi-sz1-homer-test73
ring1_addr: 172.29.xx.xx
nodeid: 1
}

node {
ring0_addr: bgi-sz1-homer-test74
ring1_addr: 172.29.xx.xx
nodeid: 2
}

node {
ring0_addr: bgi-sz1-homer-test76
ring1_addr: 172.29.xx.xx
nodeid: 3
}
}

quorum {
provider: corosync_votequorum
}

logging {
to_logfile: yes
logfile: /var/log/cluster/corosync.log
to_syslog: yes
}

Start cluster

1
2
3
4
5
6
7
8
9
10
$ pcs cluster start --all
$ corosync-cfgtool -s
Printing ring status.
Local node ID 1
RING ID 0
id = 192.168.xx.130
status = ring 0 active with no faults
RING ID 1
id = 172.29.xx.73
status = ring 1 active with no faults

Create stonith for physical server

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# pcs stonith create node01-fencing fence_ipmilan pcmk_host_list="nod01" \
ipaddr="node1_ipmi_ipaddr" login=ADMIN passwd=ADMIN lanplus=true power_wait=4 \
op monitor interval=60s
# pcs stonith create node02-fencing fence_ipmilan pcmk_host_list="node02" \
ipaddr="node2_ipmi_ipaddr" login=ADMIN passwd=ADMIN lanplus=true power_wait=4 \
op monitor interval=60s
# pcs stonith create node03-fencing fence_ipmilan pcmk_host_list="node03" \
ipaddr="node3_ipmi_ipaddr" login=ADMIN passwd=ADMIN lanplus=true power_wait=4 op \
monitor interval=60s

# pcs property set stonith-enabled=true

# pcs status
Cluster name: ha-cluster
...
Online: [ node02 node03 node01 ]

Full list of resources:

node01-fencing (stonith:fence_ipmilan): Started node02
node03-fencing (stonith:fence_ipmilan): Started node01
node02-fencing (stonith:fence_ipmilan): Started node03

PCSD Status:
node03: Online
node01: Online
node02: Online

Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled

Get quorum info

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
$ corosync-quorumtool
Quorum information
------------------
Date: Mon Mar 13 22:08:01 2017
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000001
Ring ID: 1/568
Quorate: Yes

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 192.168.xx.130 (local)
0x00000002 1 192.168.xx.131
0x00000003 1 192.168.xx.132

$

$ corosync-cmapctl|grep members
runtime.totem.pg.mrp.srp.members.1.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.1.ip (str) = r(0) ip(192.168.xx.130) r(1) ip(172.29.xx.73)
runtime.totem.pg.mrp.srp.members.1.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.1.status (str) = joined
runtime.totem.pg.mrp.srp.members.2.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.2.ip (str) = r(0) ip(192.168.xx.131) r(1) ip(172.29.xx.74)
runtime.totem.pg.mrp.srp.members.2.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.2.status (str) = joined
runtime.totem.pg.mrp.srp.members.3.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.3.ip (str) = r(0) ip(192.168.xx.132) r(1) ip(172.29.xx.76)
runtime.totem.pg.mrp.srp.members.3.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.3.status (str) = joined

backup and restore configuration

1
2
3
4
$ pcs cluster cib > xx.xml
$ pcs cluster cib-push xx.xml

$ pcs config

Create resources

1
2
3
4
5
6
7
8
9
10
$ pcs resource create VirtualIP1 ocf:heartbeat:IPaddr2 ip=10.0.0.77 cidr_netmask=8 nic=br0 op monitor interval=60s
$ pcs constraint location VirtualIP prefers bgi-sz1-homer-test73=0

$ pcs resource create ansible-kvm VirtualDomain hypervisor="qemu:///system" config="/export/glusterfs/vms/kvm/xml/ansible-kvm.xml" migration_transport=ssh op start timeout="60s" op stop timeout="60s" op monitor timeout="30" interval="20" meta allow-migrate="true" op migrate_from interval="0" timeout="120s" op migrate_to interval="0" timeout="120"

# if you want add to group, add --group xxx
$ pcs resource create ansible-kvm VirtualDomain hypervisor="qemu:///system" config="/export/glusterfs/vms/kvm/xml/ansible-kvm.xml" migration_transport=ssh op start timeout="60s" op stop timeout="60s" op monitor timeout="30" interval="20" meta allow-migrate="true" op migrate_from interval="0" timeout="120s" op migrate_to interval="0" timeout="120" --group GROUP-A

# http
# pcs resource create webres apache configfile="/etc/httpd/conf/httpd.conf" statusurl="http://127.0.0.1/server-status"

Create the resource group

1
pcs resource group add GROUP-A resource-1 resource-2 resource-3

monitor

1
2
crm_mon
pcs status nodes

Standby and maintance

1
2
3
4
5
6
7
8
9
10
11
pcs cluster standby
pcs cluster unstandby
pcs cluster stop
pcs cluster stop --all

#BAN" the resource group in which you would like to stop the cluster services
pcs resource ban resource-name node-name

#maintenance
pcs property set maintenance-mode=true
pcs property set maintenance-mode=flase

Resource constraint

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ pcs constraint
Location Constraints:
Resource: VirtualIP
Enabled on: bgi-sz1-homer-test73 (score:0)
Resource: VirtualIP2
Enabled on: bgi-sz1-homer-test74 (score:0)
Resource: VirtualIP3
Enabled on: bgi-sz1-homer-test76 (score:0)
Ordering Constraints:
Colocation Constraints:
Ticket Constraints:

$ pcs constraint location resource-name prefers nodename=50

# Using location constraint, you can also avoid specific node to run a particular resource
$ pcs constraint location resource-name avoids nodename=50

Resource order

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# create
$ pcs constraint order resource1 then resource2
$ pcs constraint order resource2 then resource3
$ pcs constraint

# remove
$ pcs constraint order --full
$ pcs constraint order remove resource1 id_output

# example

$ pcs constraint order VirtualIP1 then manager-kvm
Adding VirtualIP1 manager-kvm (kind: Mandatory) (Options: first-action=start then-action=start)
$ pcs constraint
Location Constraints:
Resource: VirtualIP
Enabled on: bgi-sz1-homer-test73 (score:0)
Resource: VirtualIP2
Enabled on: bgi-sz1-homer-test74 (score:0)
Resource: VirtualIP3
Enabled on: bgi-sz1-homer-test76 (score:0)
Ordering Constraints:
start VirtualIP1 then start manager-kvm (kind:Mandatory)
Colocation Constraints:
Ticket Constraints:

$ pcs constraint order --full
Ordering Constraints:
start VirtualIP1 then start manager-kvm (kind:Mandatory) (id:order-VirtualIP1-manager-kvm-mandatory)

#input id
$ pcs constraint order remove VirtualIP1 order-VirtualIP1-manager-kvm-mandatory
$ pcs constraint order --full
Ordering Constraints:

Resource migration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ pcs resource update resource-name meta migration-threshold="2"
or
$ pcs resource defaults migration-threshold=2

$ pcs resource update resource-name meta failure-timeout=60s
$ pcs resource show resource-name
# show fail-count
$ pcs resource failcount show
or
$ crm_failcount -r resource-name

# reset the fail-counts
$ pcs resource failcount reset resource-name node-name
or
$ pcs resource cleanup resource-name
$ pcs resource failcount show resource-name

Not move resource back

1
2
$ pcs resource defaults resource-stickiness=100
$ pcs resource defaults

Colocation constraint

A colocation constraint determines that the location of one resource depends on the location of another resource

1
2
3
4
5
6
$ pcs constraint colocation add resource1 resource2 INFINITY
$ pcs constraint colocation add resource1 with resource2 INFINITY
#remove
$ pcs constraint --full

$ pcs constraint colocation remove resource1

Resource location

1
2
3
4
5
6
7
8
9
10
$ pcs constraint location resource-name prefers node1
$ pcs constraint location resource-name avoids node1
$ pcs constraint --full
$ pcs constraint location remove location-resource-name
$ pcs constraint location resource-name prefers node1=INFINITY
$ crm_simulate -sL # get resource score
# Resource pefer node1,node2,node3
pcs constraint location resource-name prefers node1=200
pcs constraint location resource-name prefers node2=100
pcs constraint location resource-name prefers node3=0

Update resource config

1
$ pcs resource update ansible-kvm config=/opt/lustresz/DAS_NFS/vms/HA-kvm/homerl/config/ansible-kvm.xml

Manage resource

1
2
3
4
5
$ pcs resource disable resource-name    
$ pcs resource enable resource-name
$ pcs resource failcount show resource-name
$ pcs resource failcount reset resource-name
$ pcs resource cleanup resource-name

Clear error output

1
pcs resource cleanup

Check cluster status

1
$ crm_verify -L -V

Destroy cluster

1
2
3
4
5
pcs cluster destroy --all

#or
systemctl stop pacemaker corosync
rm -f /var/lib/pacemaker/cib/*

Reference
RHEL 7 – Pacemaker
High_Availability_Add-On_Reference