jravetch

The point of this is to use cheap machines with small/slow storage to coordinate client requests while dedicating the machines with the big and fast storage to doing what they do best. I found that request coordination was contributing to about half the CPU usage on our Cassandra nodes, on average. Solid state storage is quite expensive, nearly doubling the cost of typical hardware. It also means that if people have control over hardware placement within the network, they can place proxy nodes closer to the client without impacting their storage footprint or fault tolerance characteristics.

This is accomplished in Cassandra by passing the -Dcassandra.join_ring=false option when the process is started. These nodes will connect to the seeds, cache the gossip data, load the schema, and begin listening for client requests. Messages like "/x.x.x.x is now UP!" will appear on the other nodes.

There are also some more practical benefits to this. Handling client requests caused us to push the NewSize of the heap up

Adding 2 nodes to an existing 3-node ZooKeeper ensemble without losing the Quorum

Since many deployments may start out with 3 nodes and so little is known about how to grow a cluster from 3 memebrs to 5 members without losing the existing Quorum, here is an example of how this might be achieved.

In this example, all 5 nodes will be running on the same Vagrant host for the purpose of illustration, running on distinct configurations (ports and data directories) without the actual load of clients.

YMMV. Caveat usufructuarius.

Step 1: Have a healthy 3-node ensemble

	---
	###
	# Elasticsearch Rolling restart using Ansible
	###

	##
	## Why is this needed?
	##
	#
	# Even if you use a serial setting to limit the number of nodes processed at one

	#!/usr/bin/python -tt
	#
	import datetime
	import json
	import logging
	import logging.handlers
	import optparse
	import os
	import re
	import shutil

	#! env python

	from boto import ec2

	conn = ec2.connect_to_region('us-west-2')

	vols = conn.get_all_volumes(filters={'status': 'available'})
	for vol in vols:
	print 'checking vol:', vol.id, 'status:', vol.status, 'attachment_id:', vol.attach_data.status
	conn.delete_volume(vol.id)

	file{ [ '/backups', '/backups/redis' ]:
	ensure => 'directory',
	owner => 'root'
	}
	cron{ 'redis-bgsave':
	command => 'redis-cli bgsave',
	user => 'root',
	hour => '0-23/2',
	minute => 3,
	}

	#!/bin/bash
	#
	# this script will attempt to detect any ephemeral drives on an EC2 node and create a RAID-0 stripe
	# mounted at /mnt. It should be run early on the first boot of the system.
	#
	# Beware, This script is NOT fully idempotent.
	#

	METADATA_URL_BASE="http://169.254.169.254/2012-01-12"

	#!/usr/bin/env python
	import os
	import sys
	import argparse

	try:
	from boto.ec2.connection import EC2Connection
	except ImportError:
	sys.stderr.write('Please install boto ( http://docs.pythonboto.org/en/latest/getting_started.html )\n')
	sys.exit(1)

	#!/usr/bin/python

	### BEGIN INIT INFO
	# Provides: update53
	# Required-Start: $network $local_fs
	# Required-Stop:
	# Should-Start:
	# Should-Stop:
	# X-Start-Before: couchbase-server
	# X-Stop-After: couchbase-server

	#!/usr/bin/php
	<?php
	/**
	* .git/hooks/pre-commit
	*
	* This pre-commit hooks will check for PHP error (lint), and make sure the code
	* is PSR compliant.
	*
	* Dependecy: PHP-CS-Fixer (https://github.com/fabpot/PHP-CS-Fixer)
	*