Skip to content

Instantly share code, notes, and snippets.

@jupierce
Forked from sdodson/etcd.conf
Created August 1, 2017 14:01
Show Gist options
  • Select an option

  • Save jupierce/84f53dd293f06a75803d2b686633230f to your computer and use it in GitHub Desktop.

Select an option

Save jupierce/84f53dd293f06a75803d2b686633230f to your computer and use it in GitHub Desktop.
rollback-procedure
# cat /etc/etcd/etcd.conf
ETCD_NAME=ose3-master.example.com
ETCD_LISTEN_PEER_URLS=https://192.168.122.159:2380
ETCD_DATA_DIR=/var/lib/etcd/
#ETCD_SNAPSHOT_COUNTER=10000
ETCD_HEARTBEAT_INTERVAL=500
ETCD_ELECTION_TIMEOUT=2500
ETCD_LISTEN_CLIENT_URLS=https://192.168.122.159:2379
#ETCD_MAX_SNAPSHOTS=5
#ETCD_MAX_WALS=5
#ETCD_CORS=
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.122.159:2380
ETCD_INITIAL_CLUSTER=ose3-master.example.com=https://192.168.122.159:2380,ose3-node1.example.com=https://192.168.122.232:2380,ose3-node2.example.com=https://192.168.122.63:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1
#ETCD_DISCOVERY=
#ETCD_DISCOVERY_SRV=
#ETCD_DISCOVERY_FALLBACK=proxy
#ETCD_DISCOVERY_PROXY=
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.122.159:2379
#[proxy]
#ETCD_PROXY=off
#[security]
ETCD_CA_FILE=/etc/etcd/ca.crt
ETCD_CERT_FILE=/etc/etcd/server.crt
ETCD_KEY_FILE=/etc/etcd/server.key
ETCD_PEER_CA_FILE=/etc/etcd/ca.crt
ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt
ETCD_PEER_KEY_FILE=/etc/etcd/peer.key
#[logging]
ETCD_DEBUG="False"

On all masters

  1. Remove the following keys from /etc/origin/master/master-config.yaml kubernetesMasterconfig.apiServerArguments.storage-backend kubernetesMasterconfig.apiServerArguments.storage-media-type
  2. systemctl stop atomic-openshift-master atomic-openshift-master-api atomic-openshift-master-controllers atomic-openshift-node

On all etcd hosts

  1. systemctl stop etcd
  2. mv /var/lib/etcd/member ~/bad-etcd

On first etcd host

  1. cp -a /var/lib/etcd/openshift-pre-migrate-#######/member /var/lib/etcd/
  2. chown -R etcd:etcd /var/lib/etcd/member
  3. restorecon -RF /var/lib/etcd/member
  4. echo "ETCD_FORCE_NEW_CLUSTER=true" >> /etc/etcd/etcd.conf
  5. systemctl start etcd
  6. etcdctl2 cluster-health
  7. systemctl stop etcd
  8. sed -i /ETCD_FORCE.*/g /etc/etcd/etcd.conf
  9. systemctl start etcd
  10. etcdctl2 member list
  11. etcdctl2 member update ID####### https://ip.ip.ip.ip:2380
  12. etcdctl2 member add ETCD_NAME_from_2nd_etcd_host https://ip2.ip2.ip2.ip2:2380 copy output
  13. on 2nd etcd host, append to end of /etc/etcd/etcd.conf
  14. on 2nd etcd host, syststemctl start etcd
  15. etcdctl2 cluster-health # verify two healthy members
  16. etcdctl2 member add ETCD_NAME_from_3rd_etcd_host https://ip3.ip3.ip3.ip3:2380
  17. on 3rd etcd host, append onto /etc/etcd/etcd.conf
  18. on 3rd etcd host, systemctl start etcd
  19. etcdctl2 cluster-health # verify three healthy members

On all masters, one at a time

  1. systemctl start atomic-openshift-master-api atomic-openshift-master-controllers atomic-openshift-node

Verify openshift integrity

On 2nd and 3rd etcd host

  1. remove appended lines from /etc/etcd/etcd.conf
  2. systemctl restart etcd
  3. etcdctl2 cluster-health # verify three healthy members
@jupierce
Copy link
Author

jupierce commented Aug 1, 2017

1. cp -a /var/lib/etcd/openshift-pre-migrate-#######/member /var/lib/etcd/
Should be openshift-pre-migration-####

14. on 2nd etcd host, syststemctl start etcd
systemctl misspelled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment