Skip to content

Instantly share code, notes, and snippets.

@schwesig

schwesig/s3_storage_observability.txt

Last active October 28, 2025 14:56
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save schwesig/0f25d77f25a73abd547f28688525a347 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save schwesig/0f25d77f25a73abd547f28688525a347 to your computer and use it in GitHub Desktop.
Download ZIP
S3_Storage_Observability
Raw
s3_storage_observability.txt
# S3 on OpenStack NERC, migrated from minio on 2025-06-13 (minio started Jan 15 2025)
## 2025-09-23 03:00pm
### Bucket: acm-metrics
Total Objects: 2512
Total Size: 477.3 GiB
### Bucket: acm-metrics-hypershift2
Total Objects: 1339
Total Size: 14.8 GiB
### Bucket: acm-metrics-test
Total Objects: 1057
Total Size: 20.0 GiB
### Bucket: loki-logs-test
Total Objects: 0
Total Size: 0 Bytes
### Bucket: open-telemetry
Total Objects: 0
Total Size: 0 Bytes
### Bucket: open-telemetry-test
Total Objects: 0
Total Size: 0 Bytes
@schwesig
Copy link
Author

schwesig commented Oct 8, 2025 •
edited
Loading

2025-10-08 01-49 pm
data collection started Jan 15 2025

Bucket: acm-metrics

Total Objects: 2774
Total Size: 522.1 GiB

Kennzahl Wert
Zeitraum 277 Tage
Gesamtdatenmenge 522,1 GiB
Durchschnitt pro Tag ≈ 1,89 GiB/Tag

Bucket: acm-metrics-hypershift2

Total Objects: 1546
Total Size: 21.2 GiB

Bucket: acm-metrics-test

Total Objects: 1057
Total Size: 20.0 GiB

aws s3api list-objects-v2 \
  --bucket acm-metrics \
  --endpoint-url https://stack.nerc.mghpcc.org:13808 \
  --output json --query "sum(Contents[].Size)"
aws s3api list-objects-v2 \
  --bucket acm-metrics \
  --endpoint-url https://stack.nerc.mghpcc.org:13808 \
  --output json --query "sum(Contents[].Size)"
for b in acm-metrics acm-metrics-hypershift2 acm-metrics-test \
         loki-logs loki-logs-test \
         open-telemetry open-telemetry-test; do
  echo "### Bucket: $b"
  aws s3 ls s3://$b/ --recursive \
    --endpoint-url https://stack.nerc.mghpcc.org:13808 \
    --human-readable --summarize | tail -n 2
  echo
done

@schwesig
Copy link
Author

schwesig commented Oct 28, 2025

Bucket: acm-metrics

Total Objects: 3088
Total Size: 591.7 GiB

Bucket: acm-metrics-hypershift2

Total Objects: 1761
Total Size: 29.2 GiB

Bucket: acm-metrics-test

Total Objects: 1057
Total Size: 20.0 GiB

Bucket: loki-logs-test

Total Objects: 0
Total Size: 0 Bytes

Bucket: open-telemetry

Total Objects: 0
Total Size: 0 Bytes

Bucket: open-telemetry-test

Total Objects: 0
Total Size: 0 Bytes

@schwesig
Copy link
Author

schwesig commented Oct 28, 2025 •
edited
Loading 

echo "| Date       | Bytes     | MiB     | GiB     |"
echo "|------------|-----------|---------|---------|"

for i in {0..30}; do
  DAY=$(date -d "-$i day" +%Y-%m-%d)
  aws s3 ls s3://acm-metrics/ --recursive \
    --endpoint-url https://stack.nerc.mghpcc.org:13808 \
  | grep "$DAY" \
  | awk -v d="$DAY" 'BEGIN{sum=0}
       {sum += $3}
       END {
         mib = sum/1024/1024;
         gib = sum/1024/1024/1024;
         printf "| %s | %d | %.2f | %.2f |\n", d, sum, mib, gib
       }'
done
Date Bytes MiB GiB
2025-10-28 7857077874 7493.09 7.32
2025-10-27 32079892518 30593.77 29.88
2025-10-26 11677347734 11136.39 10.88
2025-10-25 10311231925 9833.56 9.60
2025-10-24 0 0.00 0.00
2025-10-23 9644908685 9198.10 8.98
2025-10-22 0 0.00 0.00
2025-10-21 8084740273 7710.21 7.53
2025-10-20 0 0.00 0.00
2025-10-19 8007399505 7636.45 7.46
2025-10-18 0 0.00 0.00
2025-10-17 8184415091 7805.27 7.62
2025-10-16 0 0.00 0.00
2025-10-15 8205295384 7825.18 7.64
2025-10-14 0 0.00 0.00
2025-10-13 8090056668 7715.28 7.53
2025-10-12 0 0.00 0.00
2025-10-11 8154914832 7777.13 7.59
2025-10-10 0 0.00 0.00
2025-10-09 7911879338 7545.36 7.37
2025-10-08 0 0.00 0.00
2025-10-07 7748094652 7389.16 7.22
2025-10-06 0 0.00 0.00
2025-10-05 7765105884 7405.38 7.23
2025-10-04 0 0.00 0.00
2025-10-03 7916092555 7549.37 7.37
2025-10-02 0 0.00 0.00
2025-10-01 8197106200 7817.37 7.63
2025-09-30 0 0.00 0.00
2025-09-29 8092382951 7717.50 7.54
2025-09-28 0 0.00 0.00

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment