Ivan Leichtling ivanlei

I hereby claim:

I am ivanlei on github.
I am ivanlei (https://keybase.io/ivanlei) on keybase.
I have a public key whose fingerprint is ED33 D65D 38F7 4B7B 53B6 34D3 3C08 7CBB 189C C81C

To claim this, I am signing this object:

S3 support SNS notifications for new objects. These SNS notifications can fan out to SQS queues.

Logstash can read from inputs including:

but neither of these are sufficient:

S3 input has very low performance in attempting to read for a bucket with a high number of writes. The S3 reader lands up spending most of its time listing the bucket contents vs. reading objects.
SQS input works well. It works well with a single logstash process or a cluster of multiple processes. However, the SQS input doesn't understand the format of an SNS notification object for S3 changes.

	2017/03/05 10:12:56 [INFO] Terraform version: 0.8.8 403a86dc557fae52f8e39676b11e1e4356b7d1a2
	2017/03/05 10:12:56 [INFO] CLI args: []string{"/Volumes/dat/Users/ivanlei/bin/terraform", "plan"}
	2017/03/05 10:12:56 [DEBUG] Detected home directory from env var: /Volumes/dat/Users/ivanlei
	2017/03/05 10:12:56 [DEBUG] Detected home directory from env var: /Volumes/dat/Users/ivanlei
	2017/03/05 10:12:56 [DEBUG] Attempting to open CLI config file: /Volumes/dat/Users/ivanlei/.terraformrc
	2017/03/05 10:12:56 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
	2017/03/05 10:12:56 [DEBUG] Detected home directory from env var: /Volumes/dat/Users/ivanlei
	2017/03/05 10:12:56 [DEBUG] plugin: waiting for all plugin processes to complete...
	panic: runtime error: index out of range

	$ terraform plan
	2017/03/05 10:12:56 [INFO] Terraform version: 0.8.8 403a86dc557fae52f8e39676b11e1e4356b7d1a2
	2017/03/05 10:12:56 [INFO] CLI args: []string{"/Volumes/dat/Users/ivanlei/bin/terraform", "plan"}
	2017/03/05 10:12:56 [DEBUG] Detected home directory from env var: /Volumes/dat/Users/ivanlei
	2017/03/05 10:12:56 [DEBUG] Detected home directory from env var: /Volumes/dat/Users/ivanlei
	2017/03/05 10:12:56 [DEBUG] Attempting to open CLI config file: /Volumes/dat/Users/ivanlei/.terraformrc
	2017/03/05 10:12:56 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
	2017/03/05 10:12:56 [DEBUG] Detected home directory from env var: /Volumes/dat/Users/ivanlei
	2017/03/05 10:12:56 [DEBUG] plugin: waiting for all plugin processes to complete...
	panic: runtime error: index out of range

	```shell
	$ terraform plan
	2017/03/05 10:11:05 [INFO] Terraform version: 0.8.8 403a86dc557fae52f8e39676b11e1e4356b7d1a2
	2017/03/05 10:11:05 [INFO] CLI args: []string{"/Volumes/dat/Users/ivanlei/bin/terraform", "plan"}
	2017/03/05 10:11:05 [DEBUG] Detected home directory from env var: /Volumes/dat/Users/ivanlei
	2017/03/05 10:11:05 [DEBUG] Detected home directory from env var: /Volumes/dat/Users/ivanlei
	2017/03/05 10:11:05 [DEBUG] Attempting to open CLI config file: /Volumes/dat/Users/ivanlei/.terraformrc
	2017/03/05 10:11:05 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
	2017/03/05 10:11:05 [DEBUG] Detected home directory from env var: /Volumes/dat/Users/ivanlei
	2017/03/05 10:11:05 [DEBUG] plugin: waiting for all plugin processes to complete...

	$ export TF_LOG=TRACE
	$ terraform plan
	2017/03/05 10:01:09 [INFO] Terraform version: 0.8.8 403a86dc557fae52f8e39676b11e1e4356b7d1a2
	2017/03/05 10:01:09 [INFO] CLI args: []string{"/Volumes/dat/Users/ivanlei/bin/terraform", "plan"}
	2017/03/05 10:01:09 [DEBUG] Detected home directory from env var: /Volumes/dat/Users/ivanlei
	2017/03/05 10:01:09 [DEBUG] Detected home directory from env var: /Volumes/dat/Users/ivanlei
	2017/03/05 10:01:09 [DEBUG] Attempting to open CLI config file: /Volumes/dat/Users/ivanlei/.terraformrc
	2017/03/05 10:01:09 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
	2017/03/05 10:01:09 [DEBUG] Detected home directory from env var: /Volumes/dat/Users/ivanlei
	2017/03/05 10:01:09 [DEBUG] plugin: waiting for all plugin processes to complete...

	2017/03/05 09:59:20 [INFO] Terraform version: 0.8.8 403a86dc557fae52f8e39676b11e1e4356b7d1a2
	2017/03/05 09:59:20 [INFO] CLI args: []string{"/Volumes/dat/Users/ivanlei/bin/terraform", "plan"}
	2017/03/05 09:59:20 [DEBUG] Detected home directory from env var: /Volumes/dat/Users/ivanlei
	2017/03/05 09:59:20 [DEBUG] Detected home directory from env var: /Volumes/dat/Users/ivanlei
	2017/03/05 09:59:20 [DEBUG] Attempting to open CLI config file: /Volumes/dat/Users/ivanlei/.terraformrc
	2017/03/05 09:59:20 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
	2017/03/05 09:59:20 [DEBUG] Detected home directory from env var: /Volumes/dat/Users/ivanlei
	2017/03/05 09:59:20 [DEBUG] plugin: waiting for all plugin processes to complete...
	panic: runtime error: index out of range

	2017/03/05 09:48:24 [INFO] Terraform version: 0.8.8 403a86dc557fae52f8e39676b11e1e4356b7d1a2
	2017/03/05 09:48:24 [INFO] CLI args: []string{"/Volumes/dat/Users/ivanlei/bin/terraform", "plan"}
	2017/03/05 09:48:24 [DEBUG] Detected home directory from env var: /Volumes/dat/Users/ivanlei
	2017/03/05 09:48:24 [DEBUG] Detected home directory from env var: /Volumes/dat/Users/ivanlei
	2017/03/05 09:48:24 [DEBUG] Attempting to open CLI config file: /Volumes/dat/Users/ivanlei/.terraformrc
	2017/03/05 09:48:24 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
	2017/03/05 09:48:24 [DEBUG] Detected home directory from env var: /Volumes/dat/Users/ivanlei
	2017/03/05 09:48:24 [DEBUG] plugin: waiting for all plugin processes to complete...
	panic: runtime error: index out of range

	### Keybase proof

	I hereby claim:

	* I am ivanlei on github.
	* I am ivanlei (https://keybase.io/ivanlei) on keybase.
	* I have a public key whose fingerprint is A809 2A4A FC1D 5E1D 346F 4A5C 0827 0A86 9915 D0B8

	To claim this, I am signing this object:

	# -- coding: utf-8 --

	#
	# This is a script takes AWS credentials csv file containing an AWS keypair and:
	# * Uses keyring package to store the Secret Access Key
	# * Outputs the proper config lines for .boto file to use the keyring
	#
	import sys
	import keyring
	import optparse