Skip to content

Instantly share code, notes, and snippets.

@Kiollpt

Kiollpt/flow.md

Last active July 14, 2021 02:25
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save Kiollpt/67fc697defcb0cd384bbc61e334b7145 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save Kiollpt/67fc697defcb0cd384bbc61e334b7145 to your computer and use it in GitHub Desktop.
Download ZIP
#airflow
Raw
flow.md

DD-DataDictionary

  • description: monitor the AWS RDBMS schema change

  • create new column to Summary or Detail:

    1. add COL into google sheet (fill default value it can decrease amount of updating time)
    2. orginzie sql in Summary or Detail func.(fetch table schema from inforamtion_schema table)
    3. col = a.columns[:NUM] in check_cell func -> NUM+=1

  • optimation: CDC replacement?

    • too slow to write?
    • can not handle API usuage limt well

  • Bottleneck: sleep time update_cell API

  • solution

  • rate limit

  • tenacity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment