Harshavardhana harshavardhana

Storj S3 vs MinIO AIStor: Technical Comparison

The Elephant in the Room: Storj's S3 Gateway IS MinIO

storj/gateway-st/miniogw implements MinIO's ObjectLayer interface, routing S3 API calls to the Storj network instead of local disks. Storj's entire S3-compatible surface area is a MinIO fork — they are not building an S3 implementation; they are building a storage backend adapter for MinIO.

gateway-mt (the hosted, multi-tenant variant) is the same pattern wrapped in a MultiTenancyLayer. When you connect to Storj's S3 endpoint, you are talking to a MinIO process.

This is not incidental. It means Storj's S3 compatibility ceiling is bounded by whatever version of MinIO they have forked and are maintaining — and their own issue tracker (edge#27: "Replace Minio fork with most recent Apache2 version") shows they are perpetually behind upstream.

PySpark Parquet Overwrite — Partition Prefix Visibility Test

Tests that after Spark overwrites partitioned Parquet files on S3/MinIO, the date-level partition prefixes remain visible in delimited ListObjectsV2 so that Spark's partition discovery still works correctly.

What the test does

Generates two sample CSV files (batch1.csv, batch2.csv) with the same schema and same date partitions but different values.

MinIO AIStor vs MinIO OSS - Complete Technical Comparison (13,061 commits analyzed)

MinIO AIStor vs MinIO OSS - Complete Technical Comparison

Analysis based on full commit history review of 13,061 commits

MinIO AIStor vs MinIO OSS - Comprehensive Technical Comparison

Analysis based on full commit history review of 2800+ commits

MinIO: Synchronous vs Asynchronous Replication

Synchronous Replication

Client waits for replication to complete before receiving response.

┌────────┐         ┌─────────────────────────────┐         ┌────────────┐
│        │  PUT    │      SOURCE CLUSTER         │         │   REMOTE   │
│ Client ├────────►│                             │         │   TARGET   │

    Outbound
    ┌───────────────────┐             ┌────────────────┐               ┌───────────────┐                      ┌────────────────┐
    │                   │   Parr.     │                │  (http body)  │               │                      │                │
    │ Bitrot Hash       │     Write   │      Pipe      │      Read     │  HTTP buffer  │    Write (syscall)   │  TCP Buffer    │
    │ Erasure Shard     │ ──────────► │  (unbuffered)  │ ────────────► │   (64K Max)   │ ───────────────────► │    (4MB)       │
    │                   │             │                │               │  (io.Copy)    │                      │                │

	## template:jinja
	{#
	This file (/etc/cloud/templates/hosts.debian.tmpl) is only utilized
	if enabled in cloud-config. Specifically, in order to enable it
	you need to add the following to config:
	manage_etc_hosts: True
	-#}
	# Your system has configured 'manage_etc_hosts' as True.
	# As a result, if you wish for changes to this file to persist
	# then you will need to either

	version: '3.7'

	# Settings and configurations that are common for all containers
	x-minio-common: &minio-common
	image: quay.io/minio/minio:${RELEASE}
	command: server http://site1-minio{1...4}/data{1...2}
	environment:
	- MINIO_PROMETHEUS_AUTH_TYPE=public
	- CI=true

	"""
	FTP benchmark.

	Usage:
	ftpbench --help
	ftpbench -h <host> -u <user> -p <password> [options] login
	ftpbench -h <host> -u <user> -p <password> [options] upload <workdir> [-s <size>]
	ftpbench -h <host> -u <user> -p <password> [options] download <workdir> [-s <size>] [--files <count>]

	Connection options: