Skip to content

Instantly share code, notes, and snippets.

@rafwell
Last active September 8, 2025 18:25
Show Gist options
  • Select an option

  • Save rafwell/ca2a474bf42dd638d3a32b62ecbf8a81 to your computer and use it in GitHub Desktop.

Select an option

Save rafwell/ca2a474bf42dd638d3a32b62ecbf8a81 to your computer and use it in GitHub Desktop.
Migrate from local to s3 on chatwoot
Este script pretende copiar seu armazenamento local para o s3 da amazon.
Primeiro você deve incluir as variaáveis do armazenamento da amazon no seu env ou docker-compose.yaml, mas deixando o armazenamento como local.
Quando concluir o script, teste e altere o armazenamento de local para amazon
@rafwell
Copy link
Author

rafwell commented Sep 2, 2025

Realize backup do seu banco de dados e do container antes de começar!

  1. Crie o arquivo de migração dentro do container. Troque 'chatwoot' pelo nome do seu serviço rodando no docker
docker compose exec chatwoot sh -lc 'cat > /tmp/s3_migrate.rb <<'"'"'RUBY'"'"'
$stdout.sync = true
$stderr.sync = true

require "aws-sdk-s3"
require "json"
require "pathname"

STATE_FILE = "/app/storage/_s3_migrate.state.json"
BATCH_SIZE = (ENV["BATCH_SIZE"] || "2000").to_i

# --- S3 (lê do .env) ---
bucket   = ENV["S3_BUCKET_NAME"] || ENV["STORAGE_BUCKET_NAME"]
raise "Defina S3_BUCKET_NAME ou STORAGE_BUCKET_NAME" unless bucket
region   = ENV["AWS_REGION"] || ENV["STORAGE_REGION"] || "us-east-1"
endpoint = ENV["STORAGE_ENDPOINT"]
pathsty  = ENV["STORAGE_FORCE_PATH_STYLE"].to_s.downcase == "true"

creds = {
  region: region,
  access_key_id:     ENV["AWS_ACCESS_KEY_ID"]     || ENV["STORAGE_ACCESS_KEY_ID"],
  secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"] || ENV["STORAGE_SECRET_ACCESS_KEY"]
}
if endpoint && !endpoint.empty?
  creds[:endpoint] = endpoint
  creds[:force_path_style] = pathsty
end
client = Aws::S3::Client.new(**creds)

# --- Origem no disco ---
storage_root = Pathname.new("/app/storage")
def disk_path_for(root, key)
  root.join(key[0,2], key[2,2], key)
end

# --- Estado (checkpoint) ---
state = if File.exist?(STATE_FILE)
  JSON.parse(File.read(STATE_FILE)) rescue { "last_id" => 0, "copied" => 0, "skipped" => 0, "errors" => 0 }
else
  { "last_id" => 0, "copied" => 0, "skipped" => 0, "errors" => 0 }
end

def save_state(h)
  File.write(STATE_FILE, JSON.pretty_generate(h))
end

total = ActiveStorage::Blob.count
puts "Total de blobs: #{total}"
puts "Retomando de last_id=#{state["last_id"]}, progresso: copied=#{state["copied"]} skipped=#{state["skipped"]} errors=#{state["errors"]}"

loop do
  batch = ActiveStorage::Blob.where("id > ?", state["last_id"]).order(:id).limit(BATCH_SIZE).to_a
  break if batch.empty?

  batch.each do |blob|
    key = blob.key
    begin
      # já existe no bucket?
      exists = false
      begin
        head = client.head_object(bucket: bucket, key: key)
        exists = head.content_length.to_i > 0
      rescue Aws::S3::Errors::NotFound
        exists = false
      end
      if exists
        state["skipped"] += 1
        next
      end

      src_path = disk_path_for(storage_root, key)
      unless src_path.exist?
        state["errors"] += 1
        warn "NAO_ENCONTRADO_DISCO -> #{key} #{src_path}"
        next
      end

      # upload (SDK decide multipart)
      File.open(src_path, "rb") do |f|
        client.put_object(
          bucket: bucket,
          key: key,
          body: f,
          content_type: blob.content_type
        )
      end

      # valida por tamanho
      head = client.head_object(bucket: bucket, key: key)
      if head.content_length.to_i == blob.byte_size
        state["copied"] += 1
        puts "OK -> #{key} | id=#{blob.id} | copied=#{state["copied"]}"
      else
        state["errors"] += 1
        warn "ERRO_TAMANHO -> #{key} esperado=#{blob.byte_size} obtido=#{head.content_length}"
      end

    rescue => e
      state["errors"] += 1
      warn "ERRO -> #{key}: #{e.class}: #{e.message}"
    ensure
      state["last_id"] = blob.id
    end
  end

  save_state(state)
  puts "Checkpoint salvo: last_id=#{state["last_id"]} copied=#{state["copied"]} skipped=#{state["skipped"]} errors=#{state["errors"]}"
end

save_state(state)
puts "\nFIM: copied=#{state["copied"]} skipped=#{state["skipped"]} errors=#{state["errors"]}"
RUBY'

  1. Rode em background com nohup e log no host
BATCH_SIZE=2000
LOG="s3_migrate_$(date +%Y%m%d_%H%M%S).log"

nohup docker compose exec -T \
  -e BATCH_SIZE="$BATCH_SIZE" \
  chatwoot sh -lc 'RAILS_ENV=production bundle exec rails runner /tmp/s3_migrate.rb' \
  > "$LOG" 2>&1 &

echo "PID: $! | Log: $LOG"

  1. Acompanhe em tempo real
tail -f "$LOG"
  1. Se precisar, encerre o processo:
kill -TERM <PID_que_apareceu>

  1. Após executar toda copia, altere no seu env para amazon e reinicie o container
  2. Execute mais uma vez o comando de copia se quiser ter certeza de que nenhum arquivo foi esquecido
  3. Atualize a tabela active_storage_blobs mudando o armazenamento do arquivo
UPDATE active_storage_blobs
SET service_name = 'amazon'
WHERE service_name = 'local';

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment