You have created a VM, deployed your services in Docker on it, services (for example, Kafka + Kafka Connect) started writing data.
Suddenly you find that the disk volume is running out.
df -h
Artemiy Kzr kzzzr
kzzzr
/ _file_formats.md
Last active
August 17, 2025 21:11
File formats comparison: CSV, JSON, Parquet, ORC
Whenever you need to store your data on S3 / Data Lake / External table choose file format wisely:
- Parquet / ORC are the best options due to efficient data layout, compression, indexing capabilities
- Columnar formats allow for column projection and partition pruning (reading only relevant data!)
- Binary formats enable schema evolution which is very applicable for constantly changing business environment
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ## Enable debugging messages and exit on error | |
| # set -ex | |
| ## Assign variables directly or use ENV variables | |
| TS=`date +"%Y-%m-%d-%H-%M-%S-%Z"` | |
| # OXR_TOKEN='' | |
| # BASE_CURRENCY='RUB' | |
| # SYMBOLS='AED,EUR,GBP,RUB,USD' | |
| ## Chain commands |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {{ | |
| config( | |
| materialized='ephemeral' | |
| ) | |
| }} | |
| WITH accepted AS ( | |
| SELECT DISTINCT | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| XN Merge Left Join DS_DIST_NONE (cost=6768461698952.58..64203387414945.72 rows=11491303 width=4361) | |
| -> XN Hash Left Join DS_BCAST_INNER (cost=5768461175265.66..63203382323457.28 rows=11491303 width=3845) | |
| -> XN Merge Left Join DS_DIST_NONE (cost=4765778722374.23..62200689246924.30 rows=11491303 width=3321) | |
| -> XN Hash Left Join DS_DIST_INNER (cost=4765778722374.23..62200689217006.26 rows=11491303 width=3285) | |
| -> XN Merge Left Join DS_DIST_NONE (cost=3765767462785.01..36310404425364.76 rows=11491303 width=3284) | |
| -> XN Hash Left Join DS_DIST_NONE (cost=3765767462785.01..36310403604540.56 rows=11491303 width=3276) | |
| -> XN Hash Left Join DS_DIST_INNER (cost=3765766240852.13..36310400055617.20 rows=11491303 width=3251) | |
| -> XN Hash Left Join DS_DIST_NONE (cost=1765764690131.17..34310372401253.97 rows=11491303 width=3250) | |
| -> XN Hash Left Join DS_DIST_NONE (cost=1765764584059.78..34308739121313.73 rows=11491303 width=2734) | |
| -> XN Hash Left Join DS_DIST_INNER (co |
- Переиспользование кода в проектах: макросы, модели, тесты
- Адаптация макросов и моделей под разные СУБД
- Примеры модулей: utils, redshift, external_tables, facebook
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| select | |
| _id as audit_id | |
| , resource_id | |
| , driver_id | |
| , "user_id" | |
| , "event" | |
| , created_at | |
| , resource_type |
kzzzr
/ _assignment.md
Last active
October 14, 2023 17:23
Metabase + Clickhouse tutorial on Yandex.Cloud
Data Warehouse Analyst – Analytics Engineer @ OTUS.ru
Follow instructions in metabase_clickhouse.md below step by step.
kzzzr
/ assignment.md
Last active
December 8, 2025 17:10
Data Vault 2.0 + Greenplum + dbtVault assignment
Data Warehouse Analyst – Analytics Engineer @ OTUS.ru
0. Fork demo repo: https://github.com/kzzzr/dbtvault_greenplum_demo
Follow instructions in data_vault_tutorial.md step by step.
Add README.md with answers to the following questions:
kzzzr
/ de_bi_deployment.md
Last active
September 21, 2021 16:34
Data Engineer – BI Deployment Assignment
NewerOlder