Skip to content

Instantly share code, notes, and snippets.

View dslaw's full-sized avatar

dslaw

6 followers · 0 following
View GitHub Profile
@dslaw
dslaw / check-pg-activity.sql
Last active September 4, 2020 06:18
Query Postgres activity
select
datname,
state,
usename || ':' || application_name as user_app,
to_char(query_start, 'YYYY-MM-DD HH24:MI') as query_start,
to_char(age(now(), query_start), 'HH24:MI') as "age",
wait_event_type || ':' || wait_event as wait_event,
left(query, 50) as "query",
pid
from pg_stat_activity
@dslaw
dslaw / README.md
Last active August 9, 2019 21:25
Example of CDC using a trigger.

Create a dev database:

docker run \
    -d \
    -p 8011:5432 \
    -e POSTGRES_USER=dev \
    -e POSTGRES_DB=dev \
    -e POSTGRES_PASSWORD=dev \
 --name=pubsub-example \
@dslaw
dslaw / README.md
Last active January 14, 2019 01:08
Setup PIA

After installing PIA provided ovpn files into /etc/openvpn, create a file under /etc/openvpn/pia.txt with your credentials in it. Then install network-manager-openvpn, which provides the update-resolv-conf script. Update the ovpn files to use the script, which resolves DNS issues:

IFS=$(echo -en "\n\b")
for f in $(ls /etc/openvpn/*.ovpn); do
    echo "script-security 2" >> ${f}
    echo "up /etc/openvpn/update-resolv-conf" >> ${f}
 echo "down /etc/openvpn/update-resolv-conf" >> ${f}
@dslaw
dslaw / spark-misc.md
Last active June 24, 2019 16:45
Miscellaneous tidbits on working with (Py)Spark

Add packages to the Spark session from within python. They'll be automatically downloaded if they're not already available.

# Using `JsonSerDe` as an example.
os.environ["PYSPARK_SUBMIT_ARGS"] = (
   '--packages "org.apache.hive.hcatalog.data.JsonSerDe" pyspark-shell'
)
spark = SparkSession.builder.config().getOrCreate()
@dslaw
dslaw / example.py
Last active April 16, 2019 20:23
Example demonstrating how to inspect canonical HTTP requests to S3
import boto3
s3_client = boto3.client("s3")
# List available operations.
s3_client._service_model.operation_names
operation_name = "ListObjectsV2"
params = {
@dslaw
dslaw / data_element_numbers.py
Last active October 23, 2018 22:22
Extract data element numbers for Univ of Oregon solar data
"""Parse data element numbers as trees."""
# XXX: Second spectral digit is broken - both values are 0!
from dataclasses import dataclass
from lxml import html
from typing import Any, Dict, List
DELIMITER = "\x97" # —
MIN_COLSPAN = 1
@dslaw
dslaw / .skhdrc
Last active October 22, 2021 14:47
Setup for macOS
# open terminal
cmd - return : osascript -e 'tell application "iTerm" to create window with default profile'
# enter fullscreen mode for the focused container
# cmd + f clobbers text search in firefox.
shift + cmd - f : yabai -m window --toggle zoom-fullscreen
# kill focused window
shift + cmd - q : yabai -m window --close
@dslaw
dslaw / plot_mixture.py
Last active February 26, 2018 00:26
Plot mixture of univariate distributions
from scipy.stats import norm
import matplotlib.pyplot as plt
import numpy as np
plt.style.use("ggplot")
# Simulate from a Univariate Gaussian Mixture Model.
@dslaw
dslaw / plot_state_seq.py
Last active February 26, 2018 00:29
Plot state sequence with generative distributions
from scipy.stats import norm
import numpy as np
import matplotlib.pyplot as plt
plt.style.use("ggplot")
def save_and_close(fig, filename):
@dslaw
dslaw / lru_cache.py
Created January 31, 2018 05:44
Example of LRU Cache in Python3 using OrderedDict.
from collections import OrderedDict
class LRUCache(object):
def __init__(self, max_size=4):
if max_size <= 0:
raise ValueError
self.max_size = max_size
self._items = OrderedDict()