ilyas ahsan ilyasahsan123
ilyasahsan123
/ top_10_most_film_by_director.sql
Created
April 3, 2019 13:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| CREATE OR REPLACE VIEW `tulisanmedium.digitalviews.top_10_most_film_by_director` | |
| OPTIONS( | |
| description="Top 5 most film by director for digital content purpose. requested by digital team" | |
| ) | |
| AS | |
| select | |
| count(distinct title) number_of_films | |
| ,director | |
| ,string_agg(distinct title order by title limit 5) as films | |
| from |
ilyasahsan123
/ dg.yaml
Created
March 25, 2019 13:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| AWSTemplateFormatVersion: '2010-09-09' | |
| Transform: 'AWS::Serverless-2016-10-31' | |
| Description: An AWS Serverless Specification template describing your function. | |
| Resources: | |
| dogreat: | |
| Type: 'AWS::Serverless::Function' | |
| Properties: | |
| Handler: lambda_function.lambda_handler | |
| Runtime: python3.6 | |
| CodeUri: s3://do-great-bucket/source.zip |
ilyasahsan123
/ buildspec.yaml
Created
March 25, 2019 13:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| version: 0.2 | |
| env: | |
| variables: | |
| INPUT_TEMPLATE: "appspec.yaml" | |
| S3_BUCKET: "do-great-bucket" | |
| phases: | |
| install: | |
| commands: | |
| - wget https://bootstrap.pypa.io/get-pip.py | |
| - python get-pip.py |
ilyasahsan123
/ mapping_series.json
Created
March 25, 2019 13:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| PUT series/_mappings/movie | |
| { | |
| "properties": { | |
| "film_to_franchise": { | |
| "type": "join", | |
| "eager_global_ordinals": true, | |
| "relations": { | |
| "franchise": "film" | |
| } | |
| }, |
ilyasahsan123
/ insert_data_to_index_series.json
Created
March 25, 2019 13:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| PUT series/movie/1?routing=1 | |
| { | |
| "id": "1", | |
| "film_to_franchise": { | |
| "name": "franchise" | |
| }, | |
| "title": "Star Wars" | |
| } | |
| PUT series/movie/1196?routing=1 | |
| { |
ilyasahsan123
/ kafka-connectors-medium
Created
January 28, 2019 15:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| curl -XPOST "http://localhost:8083/connectors" -H 'Content-Type: application/json' -d' | |
| { | |
| "name": "debezium", | |
| "config": { | |
| "connector.class": "io.debezium.connector.mysql.MySqlConnector", | |
| "database.hostname": "localhost", | |
| "database.port": "3306", | |
| "database.user": "root", | |
| "database.password": "", | |
| "database.server.id": "184054", |
ilyasahsan123
/ consumer.py
Last active
January 28, 2019 15:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import json | |
| from kafka import KafkaConsumer | |
| from google.cloud import bigquery | |
| if __name__ == "__main__": | |
| # bigwuery configuration | |
| bigquery_client = bigquery.Client() | |
| dataset_ref = bigquery_client.dataset('DATASET_NAME') | |
| table_ref = dataset_ref.table('TABLE_NAME') |
ilyasahsan123
/ consumer.py
Created
January 28, 2019 14:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import json | |
| from kafka import KafkaConsumer | |
| from google.cloud import bigquery | |
| if __name__ == "__main__": | |
| # bigwuery configuration | |
| bigquery_client = bigquery.Client() | |
| dataset_ref = bigquery_client.dataset('DATASET_NAME') | |
| table_ref = dataset_ref.table('TABLE_NAME') |
ilyasahsan123
/ beam_streaming_sample.py
Last active
January 3, 2019 16:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import json | |
| import apache_beam as beam | |
| import apache_beam.transforms.window as window | |
| from apache_beam.options.pipeline_options import PipelineOptions | |
| from apache_beam.options.pipeline_options import StandardOptions | |
| from apache_beam.io.gcp.internal.clients import bigquery | |
| if __name__ == "__main__": | |
| # create pipeline |
ilyasahsan123
/ subscriber.py
Created
November 11, 2018 08:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import argparse | |
| def list_subscriptions_in_topic(project_id, topic_name): | |
| from google.cloud import pubsub_v1 | |
| publisher = pubsub_v1.PublisherClient() | |
| topic_path = publisher.topic_path(project_id, topic_name) | |
| for subscription in publisher.list_topic_subscriptions(topic_path): | |
| print(subscription) | |
NewerOlder