Skip to content

Instantly share code, notes, and snippets.

@akhanf

akhanf/write_to_openstack.py

Last active November 27, 2025 15:02
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save akhanf/d943f10ec12f68e15463d979fe7cc5c6 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save akhanf/d943f10ec12f68e15463d979fe7cc5c6 to your computer and use it in GitHub Desktop.
Download ZIP
ome-zarr-py writing to openstack example
Raw
write_to_openstack.py
import numpy as np
import zarr
import s3fs
from zarr.storage import FsspecStore
from ome_zarr.writer import write_image
import os
# Create data
size_xy = 128
size_z = 10
rng = np.random.default_rng(0)
data = rng.poisson(lam=10, size=(size_z, size_xy, size_xy)).astype(np.uint8)
# Build S3 filesystem
fs = s3fs.S3FileSystem(
client_kwargs={"endpoint_url": "https://object-arbutus.cloud.computecanada.ca"},
asynchronous=True,
config_kwargs={'request_checksum_calculation':'WHEN_REQUIRED','response_checksum_validation':'WHEN_REQUIRED'}
)
# Build synchronous store
store = FsspecStore(
fs=fs,
path="khanlab-lightsheet/test_ngff_image2.zarr",
)
# Open root
root = zarr.group(store=store)
write_image(
image=data,
group=root,
axes="zyx",
storage_options=dict(chunks=(1, size_xy, size_xy)),
)
@akhanf
Copy link
Author

akhanf commented Nov 27, 2025

Corresponding environment variables to set to make use of the aws s3 cli:

export AWS_REQUEST_CHECKSUM_CALCULATION=when_required
export AWS_RESPONSE_CHECKSUM_VALIDATION=when_required

@akhanf
Copy link
Author

akhanf commented Nov 27, 2025

assumes you have these:

~/.aws/config:

[default]
endpoint_url = https://object-arbutus.cloud.computecanada.ca
signature_version = s3v4

[plugins]
endpoint = awscli_plugin_endpoint

~/.aws/credentials:

[default]
aws_access_key_id = XXXXXX
aws_secret_access_key = XXXXXX

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment