bkay brandonkrull

These are the Kickstarter Engineering and Data role definitions for both teams.

Technical	Data	People
Junior Software Engineer
Software Engineer	Data Analyst
Senior Software Engineer	Data Scientist	Engineering Manager
Staff Engineer	VP of Data	Engineering Director
Principal Engineer		CTO

Rails naming conventions

General Ruby conventions

Class names are CamelCase.

Methods and variables are snake_case.

Methods with a ? suffix will return a boolean.

Turning PostgreSQL into a queue serving 10,000 jobs per second

RDBMS-based job queues have been criticized recently for being unable to handle heavy loads. And they deserve it, to some extent, because the queries used to safely lock a job have been pretty hairy. SELECT FOR UPDATE followed by an UPDATE works fine at first, but then you add more workers, and each is trying to SELECT FOR UPDATE the same row (and maybe throwing NOWAIT in there, then catching the errors and retrying), and things slow down.

On top of that, they have to actually update the row to mark it as locked, so the rest of your workers are sitting there waiting while one of them propagates its lock to disk (and the disks of however many servers you're replicating to). QueueClassic got some mileage out of the novel idea of randomly picking a row near the front of the queue to lock, but I can't still seem to get more than an an extra few hundred jobs per second out of it under heavy load.

So, many developers have started going straight t

In Rails 3

NOTE: This post now lives (and kept up to date) on my blog: http://hakunin.com/rails3-load-paths

If you add a dir directly under app/

Do nothing. All files in this dir are eager loaded in production and lazy loaded in development by default.

	Works with other engineers and Product to formulate project goals and feature specifications.
	Strong user knowledge of infrastructure tooling (CI, deployment, monitoring).
	Asks for help at appropriate points when stuck.
	Active participation in PR reviews.
	Create well-written, comprehensive tests.
	Domain expert with respect to applications / systems owned.
	Play key role in developing product roadmaps.
	Has deep understanding of customer & business needs.
	Make considered decisions when and how to integrate third party libraries and systems.
	Proactively communicates with stakeholders about issues affecting personal delivery of project.

	# Example of using an InitContainer in place of a GitRepo volume.
	# Unilke GitRepo volumes, this approach runs the git command in a container,
	# with the associated hardening.
	apiVersion: v1
	kind: Pod
	metadata:
	name: git-repo-demo
	annotations:
	seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
	spec:

	import os
	import unittest

	from airflow.models import DagBag

	class TestDags(unittest.TestCase):
	"""
	Generic tests that all DAGs in the repository should be able to pass.
	"""

	-- show running queries (pre 9.2)
	SELECT procpid, age(clock_timestamp(), query_start), usename, current_query
	FROM pg_stat_activity
	WHERE current_query != '<IDLE>' AND current_query NOT ILIKE '%pg_stat_activity%'
	ORDER BY query_start desc;

	-- show running queries (9.2)
	SELECT pid, age(clock_timestamp(), query_start), usename, query
	FROM pg_stat_activity
	WHERE query != '<IDLE>' AND query NOT ILIKE '%pg_stat_activity%'