Skip to content

Instantly share code, notes, and snippets.

@anoldguy

anoldguy/etcd2.log

Last active August 23, 2016 15:28
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save anoldguy/12350030c8f2348bbd2e46e760b04076 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save anoldguy/12350030c8f2348bbd2e46e760b04076 to your computer and use it in GitHub Desktop.
Download ZIP
etcd auto join!
Raw
etcd2.log
-- Logs begin at Tue 2016-08-23 14:13:04 UTC, end at Tue 2016-08-23 15:05:50 UTC. --
Aug 23 14:13:42 ip-10-111-13-245.ec2.internal systemd[1]: Stopped etcd2.
Aug 23 14:14:09 ip-10-111-13-245.ec2.internal systemd[1]: Starting etcd2...
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://10.111.13.245:2379
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd2
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: recognized and used environment variable ETCD_ELECTION_TIMEOUT=1200
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://10.111.13.245:2380
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: recognized and used environment variable ETCD_INITIAL_CLUSTER=i-066bec7aa6a963d89=http://10.111.13.245:2380,i-0a83d3f217a063659=http://10.111.13.246:2380,i-0870117c4eaad8b68=http://10.111.13.244:2380
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://10.111.13.245:2380
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: recognized and used environment variable ETCD_NAME=i-066bec7aa6a963d89
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: etcd Version: 2.3.2
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: Git SHA: ce63f10
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: Go Version: go1.5.4
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: Go OS/Arch: linux/amd64
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: setting maximum number of CPUs to 1, total number of available CPUs is 1
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: listening for peers on http://10.111.13.245:2380
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: listening for client requests on http://0.0.0.0:2379
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: name = i-066bec7aa6a963d89
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: data dir = /var/lib/etcd2
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: member dir = /var/lib/etcd2/member
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: heartbeat = 100ms
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: election = 1200ms
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: snapshot count = 10000
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: advertise client URLs = http://10.111.13.245:2379
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: initial advertise peer URLs = http://10.111.13.245:2380
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: initial cluster = i-066bec7aa6a963d89=http://10.111.13.245:2380,i-0870117c4eaad8b68=http://10.111.13.244:2380,i-0a83d3f217a063659=http://10.111.13.246:2380
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: starting member bfe995873720b445 in cluster fbc5684f035dc6a0
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: bfe995873720b445 became follower at term 0
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: newRaft bfe995873720b445 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: bfe995873720b445 became follower at term 1
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: starting server... [version: 2.3.2, cluster version: to_be_decided]
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal systemd[1]: Started etcd2.
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: bfe995873720b445 [term: 1] received a MsgApp message with higher term from 71ef72e21299a380 [term: 4]
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: bfe995873720b445 became follower at term 4
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: raft.node: bfe995873720b445 elected leader 71ef72e21299a380 at term 4
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: added member 71ef72e21299a380 [http://10.111.13.246:2380] to cluster fbc5684f035dc6a0
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: added member b099b1b4cf08f858 [http://10.111.13.244:2380] to cluster fbc5684f035dc6a0
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: added local member bfe995873720b445 [http://10.111.13.245:2380] to cluster fbc5684f035dc6a0
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: set the initial cluster version to 2.2
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: the connection with 71ef72e21299a380 became active
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: the connection with b099b1b4cf08f858 became active
Aug 23 14:14:12 ip-10-111-13-245.ec2.internal etcd2[1259]: published {Name:i-066bec7aa6a963d89 ClientURLs:[http://10.111.13.245:2379]} to cluster fbc5684f035dc6a0
Aug 23 14:14:14 ip-10-111-13-245.ec2.internal etcd2[1259]: updated the cluster version from 2.2 to 2.3
Aug 23 14:56:45 ip-10-111-13-245.ec2.internal etcd2[1259]: failed to dial 71ef72e21299a380 on stream MsgApp v2 (dial tcp 10.111.13.246:2380: getsockopt: connection refused)
Aug 23 14:56:45 ip-10-111-13-245.ec2.internal etcd2[1259]: the connection with 71ef72e21299a380 became inactive
Aug 23 14:56:46 ip-10-111-13-245.ec2.internal etcd2[1259]: bfe995873720b445 is starting a new election at term 4
Aug 23 14:56:46 ip-10-111-13-245.ec2.internal etcd2[1259]: bfe995873720b445 became candidate at term 5
Aug 23 14:56:46 ip-10-111-13-245.ec2.internal etcd2[1259]: bfe995873720b445 received vote from bfe995873720b445 at term 5
Aug 23 14:56:46 ip-10-111-13-245.ec2.internal etcd2[1259]: bfe995873720b445 [logterm: 4, index: 6929] sent vote request to 71ef72e21299a380 at term 5
Aug 23 14:56:46 ip-10-111-13-245.ec2.internal etcd2[1259]: bfe995873720b445 [logterm: 4, index: 6929] sent vote request to b099b1b4cf08f858 at term 5
Aug 23 14:56:46 ip-10-111-13-245.ec2.internal etcd2[1259]: raft.node: bfe995873720b445 lost leader 71ef72e21299a380 at term 5
Aug 23 14:56:46 ip-10-111-13-245.ec2.internal etcd2[1259]: bfe995873720b445 received vote from b099b1b4cf08f858 at term 5
Aug 23 14:56:46 ip-10-111-13-245.ec2.internal etcd2[1259]: bfe995873720b445 [quorum:2] has received 2 votes and 0 vote rejections
Aug 23 14:56:46 ip-10-111-13-245.ec2.internal etcd2[1259]: bfe995873720b445 became leader at term 5
Aug 23 14:56:46 ip-10-111-13-245.ec2.internal etcd2[1259]: raft.node: bfe995873720b445 elected leader bfe995873720b445 at term 5
Aug 23 14:56:48 ip-10-111-13-245.ec2.internal etcd2[1259]: failed to reach the peerURL(http://10.111.13.246:2380) of member 71ef72e21299a380 (Get http://10.111.13.246:2380/version: dial tcp 10.111.13.246:2380: i/o timeout)
Aug 23 14:56:48 ip-10-111-13-245.ec2.internal etcd2[1259]: cannot get the version of member 71ef72e21299a380 (Get http://10.111.13.246:2380/version: dial tcp 10.111.13.246:2380: i/o timeout)
Aug 23 14:56:53 ip-10-111-13-245.ec2.internal etcd2[1259]: failed to reach the peerURL(http://10.111.13.246:2380) of member 71ef72e21299a380 (Get http://10.111.13.246:2380/version: dial tcp 10.111.13.246:2380: i/o timeout)
Aug 23 14:56:53 ip-10-111-13-245.ec2.internal etcd2[1259]: cannot get the version of member 71ef72e21299a380 (Get http://10.111.13.246:2380/version: dial tcp 10.111.13.246:2380: i/o timeout)
Aug 23 14:56:58 ip-10-111-13-245.ec2.internal etcd2[1259]: failed to reach the peerURL(http://10.111.13.246:2380) of member 71ef72e21299a380 (Get http://10.111.13.246:2380/version: dial tcp 10.111.13.246:2380: i/o timeout)
Aug 23 14:56:58 ip-10-111-13-245.ec2.internal etcd2[1259]: cannot get the version of member 71ef72e21299a380 (Get http://10.111.13.246:2380/version: dial tcp 10.111.13.246:2380: i/o timeout)
Aug 23 14:57:03 ip-10-111-13-245.ec2.internal etcd2[1259]: failed to reach the peerURL(http://10.111.13.246:2380) of member 71ef72e21299a380 (Get http://10.111.13.246:2380/version: dial tcp 10.111.13.246:2380: i/o timeout)
Aug 23 14:57:03 ip-10-111-13-245.ec2.internal etcd2[1259]: cannot get the version of member 71ef72e21299a380 (Get http://10.111.13.246:2380/version: dial tcp 10.111.13.246:2380: i/o timeout)
Aug 23 14:57:09 ip-10-111-13-245.ec2.internal etcd2[1259]: failed to reach the peerURL(http://10.111.13.246:2380) of member 71ef72e21299a380 (Get http://10.111.13.246:2380/version: dial tcp 10.111.13.246:2380: i/o timeout)
Aug 23 14:57:09 ip-10-111-13-245.ec2.internal etcd2[1259]: cannot get the version of member 71ef72e21299a380 (Get http://10.111.13.246:2380/version: dial tcp 10.111.13.246:2380: i/o timeout)
Aug 23 14:57:12 ip-10-111-13-245.ec2.internal etcd2[1259]: the connection to peer 71ef72e21299a380 is unhealthy
*snip repeat messages for the 3 min while the ASG realizes the instance is gone and starts a new instance*
Aug 23 15:01:04 ip-10-111-13-245.ec2.internal etcd2[1259]: removed member 71ef72e21299a380 from cluster fbc5684f035dc6a0
Aug 23 15:01:05 ip-10-111-13-245.ec2.internal etcd2[1259]: failed to reach the peerURL(http://10.111.13.246:2380) of member 71ef72e21299a380 (Get http://10.111.13.246:2380/version: dial tcp 10.111.13.246:2380: i/o timeout)
Aug 23 15:01:05 ip-10-111-13-245.ec2.internal etcd2[1259]: cannot get the version of member 71ef72e21299a380 (Get http://10.111.13.246:2380/version: dial tcp 10.111.13.246:2380: i/o timeout)
Aug 23 15:01:07 ip-10-111-13-245.ec2.internal etcd2[1259]: added member 77da4ca4248bede6 [http://10.111.13.249:2380] to cluster fbc5684f035dc6a0
Aug 23 15:01:09 ip-10-111-13-245.ec2.internal etcd2[1259]: failed to reach the peerURL(http://10.111.13.249:2380) of member 77da4ca4248bede6 (Get http://10.111.13.249:2380/version: dial tcp 10.111.13.249:2380: getsockopt: connection refused)
Aug 23 15:01:09 ip-10-111-13-245.ec2.internal etcd2[1259]: cannot get the version of member 77da4ca4248bede6 (Get http://10.111.13.249:2380/version: dial tcp 10.111.13.249:2380: getsockopt: connection refused)
Aug 23 15:01:13 ip-10-111-13-245.ec2.internal etcd2[1259]: the connection with 77da4ca4248bede6 became active
Raw
main.tf
resource "aws_launch_configuration" "k8s_control_plane" {
name_prefix = "${var.name}_control_"
instance_type = "${var.default_instance_size}"
security_groups = ["${aws_security_group.sg_kubernetes_control_plane.id}"]
key_name = "${aws_key_pair.control_plane_key.key_name}"
image_id = "${var.ami}"
iam_instance_profile = "${aws_iam_instance_profile.k8s_control_profile.id}"
associate_public_ip_address = true
user_data = <<EOF
#cloud-config
coreos:
etcd2:
advertise-client-urls: http://$private_ipv4:2379
initial-advertise-peer-urls: http://$private_ipv4:2380
listen-client-urls: http://0.0.0.0:2379
listen-peer-urls: http://$private_ipv4:2380
units:
- name: etcd2.service
command: stop
- name: etcd-peers.service
command: start
content: |
[Unit]
Description=Write a file with the etcd peers that we should bootstrap to
[Service]
Restart=on-failure
RestartSec=10
ExecStartPre=/usr/bin/docker pull monsantoco/etcd-aws-cluster:latest
ExecStartPre=/usr/bin/docker run --rm=true -v /etc/sysconfig/:/etc/sysconfig/ monsantoco/etcd-aws-cluster:latest
ExecStart=/usr/bin/systemctl start etcd2
- name: fleet.service
command: start
write_files:
- path: /etc/systemd/system/etcd2.service.d/30-etcd_peers.conf
permissions: 0644
content: |
[Service]
# Load the other hosts in the etcd leader autoscaling group from file
EnvironmentFile=/etc/sysconfig/etcd-peers
EOF
lifecycle {
create_before_destroy = true
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment