Skip to content

Instantly share code, notes, and snippets.

@sdodson

sdodson/etcd.conf

Last active August 1, 2017 14:01
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save sdodson/de8464ceda838cede41ce54342d96f32 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save sdodson/de8464ceda838cede41ce54342d96f32 to your computer and use it in GitHub Desktop.
Download ZIP
rollback-procedure
Raw
etcd.conf
# cat /etc/etcd/etcd.conf
ETCD_NAME=ose3-master.example.com
ETCD_LISTEN_PEER_URLS=https://192.168.122.159:2380
ETCD_DATA_DIR=/var/lib/etcd/
#ETCD_SNAPSHOT_COUNTER=10000
ETCD_HEARTBEAT_INTERVAL=500
ETCD_ELECTION_TIMEOUT=2500
ETCD_LISTEN_CLIENT_URLS=https://192.168.122.159:2379
#ETCD_MAX_SNAPSHOTS=5
#ETCD_MAX_WALS=5
#ETCD_CORS=
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.122.159:2380
ETCD_INITIAL_CLUSTER=ose3-master.example.com=https://192.168.122.159:2380,ose3-node1.example.com=https://192.168.122.232:2380,ose3-node2.example.com=https://192.168.122.63:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1
#ETCD_DISCOVERY=
#ETCD_DISCOVERY_SRV=
#ETCD_DISCOVERY_FALLBACK=proxy
#ETCD_DISCOVERY_PROXY=
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.122.159:2379
#[proxy]
#ETCD_PROXY=off
#[security]
ETCD_CA_FILE=/etc/etcd/ca.crt
ETCD_CERT_FILE=/etc/etcd/server.crt
ETCD_KEY_FILE=/etc/etcd/server.key
ETCD_PEER_CA_FILE=/etc/etcd/ca.crt
ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt
ETCD_PEER_KEY_FILE=/etc/etcd/peer.key
#[logging]
ETCD_DEBUG="False"
Raw
rollback.md

On all masters

  1. Remove the following keys from /etc/origin/master/master-config.yaml kubernetesMasterconfig.apiServerArguments.storage-backend kubernetesMasterconfig.apiServerArguments.storage-media-type
  2. systemctl stop atomic-openshift-master atomic-openshift-master-api atomic-openshift-master-controllers atomic-openshift-node

On all etcd hosts

  1. systemctl stop etcd
  2. mv /var/lib/etcd/member ~/bad-etcd

On first etcd host

  1. cp -a /var/lib/etcd/openshift-pre-migrate-#######/member /var/lib/etcd/
  2. chown -R etcd:etcd /var/lib/etcd/member
  3. restorecon -RF /var/lib/etcd/member
  4. echo "ETCD_FORCE_NEW_CLUSTER=true" >> /etc/etcd/etcd.conf
  5. systemctl start etcd
  6. etcdctl2 cluster-health
  7. systemctl stop etcd
  8. sed -i /ETCD_FORCE.*/g /etc/etcd/etcd.conf
  9. systemctl start etcd
  10. etcdctl2 member list
  11. etcdctl2 member update ID####### https://ip.ip.ip.ip:2380
  12. etcdctl2 member add ETCD_NAME_from_2nd_etcd_host https://ip2.ip2.ip2.ip2:2380 copy output
  13. on 2nd etcd host, append to end of /etc/etcd/etcd.conf
  14. on 2nd etcd host, syststemctl start etcd
  15. etcdctl2 cluster-health # verify two healthy members
  16. etcdctl2 member add ETCD_NAME_from_3rd_etcd_host https://ip3.ip3.ip3.ip3:2380
  17. on 3rd etcd host, append onto /etc/etcd/etcd.conf
  18. on 3rd etcd host, systemctl start etcd
  19. etcdctl2 cluster-health # verify three healthy members

On all masters, one at a time

  1. systemctl start atomic-openshift-master-api atomic-openshift-master-controllers atomic-openshift-node

Verify openshift integrity

On 2nd and 3rd etcd host

  1. remove appended lines from /etc/etcd/etcd.conf
  2. systemctl restart etcd
  3. etcdctl2 cluster-health # verify three healthy members
@sdodson
Copy link
Author

sdodson commented Jul 25, 2017

@ingvagabund we need to look at distilling this down to something that we can put into the docs. I was really hoping that the process would be simpler than this. Do you have any tricks?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment