Neal Magee nmagee

Lab: JSON Serialization and Deserialization with REST APIs

Estimated Time: 30-45 minutes
Difficulty: Intermediate
Prerequisites: Basic Python, HTTP requests, JSON concepts

Learning Objectives

By the end of this lab, you will be able to:

Fetch data from REST APIs and deserialize JSON responses

Data Windowing

With endless streaming sources, how do we generate analytics? Windowing is a meaningful way of aggregating data into useful chunks. A consumer operation, not a producer operation. Think of it as a group_by operation for streaming data.

`GROUP BY` - for data at rest

Note the two attached python scripts that use the NYC Taxi Data and group results by either day or hour (of a single day).

`prefect`

Install

In a new virtual environment:

pip install prefect requests sympy

Server

Kubernetes Cron Jobs

To process multiple jobs at scale in an organized and manageable way, organizations might choose from a variety of tools.

One powerful tool to run and scheduled jobs is Kubernetes, an open source platform first created by Google. Kubernetes (or K8S) orchestrates dozens or even hundreds (or thousands) of containerized jobs using a "declared state" model - which is to say that developers describe the state of their application and the cluster makes it happen.

In this lab you will create and submit a CronJob to run in the UVA Kubernetes cluster.

Instructions

DS2002 Lab 3 - Data Cleaning Scripts

Sample Data

Sample data - https://s3.amazonaws.com/ds2002-resources/labs/lab3-bundle.tar.gz - tar-zipped TSV
Stock Data - https://s3.amazonaws.com/ds2002-resources/labs/stock_data.tsv - TSV
Flight Log - https://s3.amazonaws.com/ds2002-resources/labs/flights.csv - CSV

1. Write a `bash` script to fetch and decompress a remote bundle

`bash` Scripting Exercises

Download the files below. Then try running the textstats.sh script with each of them.
Play wordle using the script below. Can you figure out
- How to print the secret word before you start guessing? and
- Where bash is getting the list of 5-letter words?
Using the fileinfo.sh script below, complete the script to display basic information about any file passed as an argument to the script. You will have to complete line 8 to begin, as well as use echo to display values to the user. BONUS: add another variable that calculates how many lines the file contains. Display the 4 file attributes.

S3 Sample Data

Fetch dummy data folders and files

curl https://s3.amazonaws.com/ds2002-resources/data/bundle.tar.gz > \
  bundle.tar.gz && \
  tar -xzvf bundle.tar.gz && \
  rm bundle.tar.gz

	networks:
	redpanda_network:
	driver: bridge
	volumes:
	redpanda-0: null
	redpanda-1: null
	redpanda-2: null
	services:
	redpanda-0:
	command:

	from airflow import Dataset
	from airflow.decorators import (
	dag,
	task,
	)
	from airflow.models import Variable
	from pendulum import datetime, duration
	import requests
	from pymongo import MongoClient, errors
	from bson.json_util import dumps

	import boto3
	from botocore.exceptions import ClientError

	def get_secret(secret_name,region_name):

	# Create a Secrets Manager client
	session = boto3.session.Session()
	client = session.client(
	service_name='secretsmanager',
	region_name=region_name