Skip to content

Instantly share code, notes, and snippets.

@mikesparr

mikesparr/gcp-redis-twemproxy-benchmark.sh

Last active January 24, 2025 17:17
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save mikesparr/62631b855e0854d6c47602fed9dc34ba to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save mikesparr/62631b855e0854d6c47602fed9dc34ba to your computer and use it in GitHub Desktop.
Download ZIP
Experiment with Google Cloud Memorystore Redis instances fronted by internal load balancer and Twemproxy
Raw
gcp-redis-twemproxy-benchmark.sh
#!/usr/bin/env bash
#####################################################################
# REFERENCES
# - https://cloud.google.com/blog/products/databases/running-redis-on-gcp-four-deployment-scenarios
# - https://cloud.google.com/memorystore/docs/redis/high-availability
# - https://redis.com/redis-enterprise/technology/highly-available-redis/
#####################################################################
export PROJECT_ID=$(gcloud config get-value project)
export PROJECT_USER=$(gcloud config get-value core/account) # set current user
export PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)")
export IDNS=${PROJECT_ID}.svc.id.goog # workload identity domain
export GCP_REGION="us-central1" # CHANGEME (OPT)
export GCP_ZONE="us-central1-a" # CHANGEME (OPT)
export NETWORK_NAME="default"
# enable apis
gcloud services enable compute.googleapis.com \
redis.googleapis.com \
artifactregistry.googleapis.com \
cloudbuild.googleapis.com \
storage.googleapis.com
# configure gcloud sdk
gcloud config set compute/region $GCP_REGION
gcloud config set compute/zone $GCP_ZONE
#####################################################################
# ARTIFACT REGISTRY SETUP
#####################################################################
# create repository
export REPO_NAME="pod3" # CHANGEME (OPT)
gcloud artifacts repositories create $REPO_NAME \
--repository-format=docker \
--location=$GCP_REGION \
--description="Docker repository"
# configure auth
gcloud auth configure-docker ${GCP_REGION}-docker.pkg.dev
#####################################################################
# REDIS INSTANCES
#
# TIP: this can take a long time, but CTRL+C will cycle through loop and launch them in parallel
#
#####################################################################
# create redis memorystore instances
for i in {1..9}; do gcloud redis instances create redis${i} --size=1 --region=$GCP_REGION --tier=STANDARD; done
#####################################################################
# TWEMPROXY DOCKER IMAGE
#####################################################################
# prepare Twemproxy container
export PROXY_DIR="twemproxy"
export PROXY_CONFIG_FILE="nutcracker.yml"
export IMAGE_NAME="twemproxy"
export TAG_NAME="test1"
export PROXY_PORT=26379
mkdir $PROXY_DIR
cd $PROXY_DIR
cat > $PROXY_CONFIG_FILE << EOF
alpha:
listen: 0.0.0.0:$PROXY_PORT
hash: fnv1a_64
distribution: ketama
timeout: 1000
backlog: 512
preconnect: true
redis: true
auto_eject_hosts: true
server_retry_timeout: 2000
server_failure_limit: 2
servers:
EOF
# append redis servers to config file
gcloud redis instances list \
--region=$GCP_REGION | awk '{ printf " - %s:%s:1\n", $6, $7 }' | tail -n +2 >> $PROXY_CONFIG_FILE
# create dockerfile
cat > Dockerfile << EOF
FROM alpine:3.16.2
RUN echo "http://dl-cdn.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories
RUN apk add --update twemproxy
EXPOSE $PROXY_PORT
COPY $PROXY_CONFIG_FILE /etc/nutcracker/
ENTRYPOINT ["/usr/sbin/nutcracker"]
CMD ["-c", "/etc/nutcracker/$PROXY_CONFIG_FILE"]
EOF
# build / push image to artifact registry (using local Dockerfile)
gcloud builds submit --tag ${GCP_REGION}-docker.pkg.dev/${PROJECT_ID}/${REPO_NAME}/${IMAGE_NAME}:${TAG_NAME}
#####################################################################
# TWEMPROXY CLUSTER
#####################################################################
export INSTANCE_TEMPLATE_NAME="twemproxy"
export INSTANCE_GROUP_NAME="ig-twemproxy"
# create instance template
gcloud compute instance-templates create-with-container $INSTANCE_TEMPLATE_NAME \
--machine-type=n1-standard-8 \
--tags=twemproxy-$PROXY_PORT,allow-health-checks-tcp \
--container-image ${GCP_REGION}-docker.pkg.dev/${PROJECT_ID}/${REPO_NAME}/${IMAGE_NAME}:${TAG_NAME}
# create managed instance group
gcloud compute instance-groups managed create $INSTANCE_GROUP_NAME \
--base-instance-name $INSTANCE_GROUP_NAME \
--size 3 \
--template twemproxy \
--region $GCP_REGION
gcloud compute instance-groups managed set-autoscaling $INSTANCE_GROUP_NAME \
--max-num-replicas 10 \
--min-num-replicas 3 \
--target-cpu-utilization 0.6 \
--cool-down-period 60 \
--region $GCP_REGION
#####################################################################
# LOAD BALANCING
#####################################################################
export HEALTH_CHECK_NAME="hc-twemproxy"
export LOAD_BALANCER_NAME="ilb-twemproxy"
export FWD_RULE_NAME="fr-ilb-twemproxy"
# create health check
gcloud compute health-checks create tcp $HEALTH_CHECK_NAME \
--port $PROXY_PORT --check-interval 5 --healthy-threshold 2
# create backend service
gcloud compute backend-services create $LOAD_BALANCER_NAME \
--load-balancing-scheme internal \
--session-affinity client_ip_port_proto \
--region $GCP_REGION \
--health-checks $HEALTH_CHECK_NAME \
--protocol tcp
# add instance group to backend
gcloud compute backend-services add-backend $LOAD_BALANCER_NAME \
--instance-group $INSTANCE_GROUP_NAME \
--instance-group-region $GCP_REGION \
--region $GCP_REGION
# add forwarding rule
gcloud compute forwarding-rules create $FWD_RULE_NAME \
--load-balancing-scheme internal \
--ip-protocol tcp \
--ports $PROXY_PORT \
--backend-service $LOAD_BALANCER_NAME \
--region $GCP_REGION
# add firewall rules
gcloud compute firewall-rules create allow-twemproxy \
--action allow --direction INGRESS \
--source-ranges 10.128.0.0/20 \
--target-tags twemproxy-$PROXY_PORT \
--rules tcp:$PROXY_PORT
gcloud compute firewall-rules create allow-health-check \
--action allow --direction INGRESS \
--source-ranges 130.211.0.0/22,35.191.0.0/16 \
--target-tags allow-health-checks-tcp \
--rules tcp,udp,icmp
# set load balancer IP
export ILB_ADDRESS=$(gcloud compute forwarding-rules describe $FWD_RULE_NAME --format="value(IPAddress)")
#####################################################################
# BENCHMARKING (TEMPORARY INSTANCE)
#####################################################################
export TEST_VM_NAME="redis-benchmark"
export SCRIPT_NAME="startup.sh"
# create startup script to install redis tools
cat > $SCRIPT_NAME << EOF
#! /bin/bash
apt update
apt -y install redis-tools
EOF
# create instance with startup script (local)
gcloud compute instances create $TEST_VM_NAME \
--image-project=debian-cloud \
--image-family=debian-10 \
--metadata-from-file=startup-script="./$SCRIPT_NAME"
# test redis proxy connection (ping => PONG)
gcloud compute ssh \
--zone $GCP_ZONE $TEST_VM_NAME \
-- redis-cli -h $ILB_ADDRESS -p $PROXY_PORT ping
# run redis benchmark
gcloud compute ssh \
--zone $GCP_ZONE $TEST_VM_NAME \
-- redis-benchmark -q -h $ILB_ADDRESS -p $PROXY_PORT \
-c 100 -n 1000000 -k 1 -t set,get,mset,incr,lpush,lpop,sadd,spop,lpush,lrange
#####################################################################
# CLEAN UP
#####################################################################
# change to parent dir
cd ..
# remove working dir
rm -rf $PROXY_DIR="twemproxy"
# delete benchmark instance
gcloud compute instances delete $TEST_VM_NAME
# delete project to avoid unnecessary billing
gcloud projects delete $PROJECT_ID
@mikesparr
Copy link
Author

mikesparr commented Oct 20, 2022 •
edited
Loading

Overview

An article shared by a customer was very dated so I wanted to update and simplify the instructions to try out Twemproxy and ILB fronting a pool of Redis servers run by Google Cloud Memorystore.

Changes

  • The original article just used Container Registry, so this version uses the new Artifact Registry
  • Updated Alpine base image and version
  • Fixed awk script to capture correct columns (IP, PORT)
  • Added additional protocols to health check (icmp, udp)
  • Used variables so more configurable

Result

Screen Shot 2022-10-20 at 2 12 50 PM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment