Skip to content

Instantly share code, notes, and snippets.

@yashau
Last active February 5, 2026 22:39
Show Gist options
  • Select an option

  • Save yashau/f4c20a8b9be2448e073d07a621082080 to your computer and use it in GitHub Desktop.

Select an option

Save yashau/f4c20a8b9be2448e073d07a621082080 to your computer and use it in GitHub Desktop.
Viberglass — Self-hosted Viber message archive (project plan)

Viberglass — Comprehensive Project Plan

Self-hosted Viber message archive. Lightweight agents on Windows PCs extract deltas from encrypted Viber databases, sync them to a central server backed by PostgreSQL + SeaweedFS. Multi-number, content-deduplicated, browsable via SvelteKit. Export Viberator-compatible backups with date range selection.


Table of Contents

  1. Architecture
  2. Sync Agent (Go)
  3. Server (SvelteKit)
  4. Database Schema
  5. SeaweedFS Integration
  6. Deduplication Strategy
  7. Ingest API
  8. Web UI
  9. Viberator-Compatible Export
  10. Multi-Number Support
  11. Authentication & Security
  12. Deployment
  13. Project Structure
  14. Implementation Phases
  15. Viber DB Reference

1. Architecture

┌──────────────────────┐    ┌──────────────────────┐
│  Windows PC          │    │  Windows PC           │
│  ┌────────────────┐  │    │  ┌────────────────┐   │
│  │ Viber Desktop   │  │    │  │ Viber Desktop   │   │
│  │ viber.db (AES)  │  │    │  │ viber.db (AES)  │   │
│  └───────┬────────┘  │    │  └───────┬────────┘   │
│  ┌───────▼────────┐  │    │  ┌───────▼────────┐   │
│  │ viberglass-agent│  │    │  │ viberglass-agent│   │
│  │ (Go binary)    │  │    │  │ (Go binary)    │   │
│  └───────┬────────┘  │    │  └───────┬────────┘   │
└──────────┼───────────┘    └──────────┼────────────┘
           │ HTTPS multipart           │
           └───────────┬───────────────┘
                       ▼
          ┌─────────────────────────┐
          │   Viberglass Server     │
          │   (SvelteKit)           │
          │                         │
          │  ┌───────┐  ┌────────┐  │
          │  │ Ingest │  │  Web   │  │
          │  │  API   │  │  UI    │  │
          │  └───┬────┘  └────┬──┘  │
          │      │            │     │
          │  ┌───▼────────────▼──┐  │
          │  │   PostgreSQL      │  │
          │  │   (structured)    │  │
          │  └───────────────────┘  │
          │  ┌───────────────────┐  │
          │  │   SeaweedFS       │  │
          │  │   (media blobs)   │  │
          │  └───────────────────┘  │
          └─────────────────────────┘

Data flow:

  1. Agent decrypts local Viber DB (AES-128-OFB, key derived from Windows SID)
  2. Extracts events newer than last sync watermark
  3. Collects referenced media files from ViberDownloads/
  4. POSTs multipart payload (JSON metadata + media files) to server
  5. Server deduplicates, stores in PostgreSQL + SeaweedFS
  6. Server ACKs with sync receipt
  7. Agent optionally cleans up old local data (Viberator-style)

2. Sync Agent

Go binary for Windows. Derived from Viberator — reuses its proven AES-128-OFB crypto and Viber DB knowledge.

2.1 Commands

viberglass-agent sync [flags]        # One-shot: extract + upload delta
viberglass-agent sync --watch        # Polling loop: sync every interval
viberglass-agent register            # Register with server, get auth token
viberglass-agent status              # Show last sync info
viberglass-agent config              # Print current config

2.2 Configuration

viberglass-agent.toml (stored alongside the binary or in %APPDATA%/Viberglass/):

[server]
url = "https://viberglass.example.com"
token = ""                    # Set by 'register' command

[viber]
phone = ""                    # Auto-detected if single account
retention_days = 30           # Keep locally after sync (0 = no cleanup)

[sync]
interval = "6h"               # For --watch mode
kill_viber_for_cleanup = true  # Only kill Viber for cleanup ops (not needed for sync)

2.3 Crypto (from Viberator)

Reuse verbatim from Viberator — proven, tested:

  • Key derivation: Reverse Windows SID string → first 16 ASCII chars = AES-128 key
  • Page layout: 4096-byte pages, 12-byte nonce trailer, AES-128-OFB per page
  • Page 1 exception: Bytes 16–27 unencrypted (SQLite header fields)
  • WAL handling: Same page encryption, frame headers unencrypted, checkpoint before reading

2.4 Delta Extraction

State file: %APPDATA%/Viberglass/sync_state_<phone>.json
{
  "lastSyncedEventId": 50000,
  "lastSyncTimestamp": "2026-02-06T02:00:00Z",
  "serverSyncId": "uuid-of-last-confirmed-sync"
}

Extraction algorithm:

  1. Read state file → get lastSyncedEventId
  2. Copy viber.db + viber.db-wal to temp location (works while Viber is running)
  3. Decrypt temp copy + WAL → temp plaintext
  4. Query delta:
    SELECT * FROM Events WHERE EventID > ? ORDER BY EventID
  5. For each event, collect:
    • Messages row (by EventID)
    • Calls row (by EventID)
    • DownloadFile row (by EventID) → actual file from ViberDownloads/
    • UploadFile row (by EventID)
    • EventsMetaData rows (by EventID)
    • LikeRelation rows (by LikeEventID)
  6. Collect referenced contacts (Contact by ContactID) — include all, server deduplicates
  7. Collect referenced chats (ChatInfo by ChatID, ChatRelation)
  8. Package as JSON + media files
  9. POST to server
  10. On success: update state file, optionally run cleanup (cleanup DOES kill Viber)
  11. Clean up temp files

2.5 Live DB Access

Verified: Viber's DB files (viber.db, viber.db-wal, viber.db-shm) can be copied while Viber is running. Viber opens them with shared read access on Windows.

This means:

  • No need to kill Viber for sync operations
  • No exclusive file locking needed — just copy the files
  • The agent copies viber.db + viber.db-wal to a temp location, decrypts the copy, extracts delta
  • Viber continues running uninterrupted
  • kill_viber config option retained only for cleanup operations (which modify the DB)

Consistency: SQLite WAL mode guarantees that the main DB file + WAL together represent a consistent state at the last checkpoint boundary. Copying both files atomically (copy DB first, then WAL immediately after) gives a consistent snapshot.

2.6 Post-Sync Cleanup

Optional (controlled by retention_days). Reuses Viberator's cleanup logic:

  1. Delete events older than retention_days from local DB
  2. Delete child records: Messages, Calls, DownloadFile, UploadFile, EventsMetaData, LikeRelation, Reminders
  3. Delete orphaned ChatRelation, ChatInfo, ChatsMetaData, ContactMetaData
  4. VACUUM the database
  5. Delete orphaned media files from ViberDownloads/ directory
  6. Re-encrypt and write back

2.7 Sync Payload Format

Multipart HTTP POST:

POST /api/v1/sync
Authorization: Bearer <agent-token>
Content-Type: multipart/form-data

Part "payload" (application/json):
{
  "agentId": "machine-hostname-sha256-short",
  "phone": "9607771234",
  "syncBatch": {
    "fromEventId": 50001,
    "toEventId": 51234,
    "eventCount": 1234,
    "extractedAt": "2026-02-06T02:00:00Z"
  },
  "contacts": [
    {
      "contactId": 123,
      "name": "John",
      "number": "+9607771234",
      "mid": "viber-member-id",
      "encryptedMid": "...",
      "clientName": "John D",
      "downloadId": "...",
      "contactFlags": 0,
      "sortName": "john",
      "timestamp": 1700000000000,
      "dateOfBirth": null,
      "abContact": 0
    }
  ],
  "chats": [
    {
      "chatId": 456,
      "token": "chat-token-string",
      "columns": { ... }  // All ChatInfo columns as key-value
    }
  ],
  "chatRelations": [
    { "chatId": 456, "contactId": 123, "pgRole": 0 }
  ],
  "events": [
    {
      "eventId": 50001,
      "columns": { ... }  // All Events columns as key-value
    }
  ],
  "messages": [
    {
      "eventId": 50001,
      "body": "Hello!",
      "subject": null,
      "columns": { ... }
    }
  ],
  "calls": [...],
  "downloads": [
    {
      "eventId": 50001,
      "tempFileName": "photo_001.jpg",
      "columns": { ... }
    }
  ],
  "uploads": [...],
  "eventsMetadata": [...],
  "likeRelations": [...]
}

Part "media_50001_photo_001.jpg" (image/jpeg): <binary>
Part "media_50055_video_002.mp4" (video/mp4): <binary>
...

2.8 Payload Compression

The entire multipart request body is gzip-compressed before sending:

req.Header.Set("Content-Encoding", "gzip")
// Wrap multipart writer in gzip.NewWriter()
  • Text payloads (JSON metadata, message bodies) compress 70-80%
  • Media files (JPEG, MP4) are already compressed — gzip won't shrink them but the overhead is negligible
  • A typical 10MB sync payload drops to ~2-3MB on the wire
  • Server decompresses transparently via middleware

2.9 Sync Mode: Watch (Near-Realtime)

viberglass-agent sync --watch runs a polling loop:

  1. Every N seconds (configurable, default 60s), check viber.db last-modified time + file size
  2. If changed since last check → run delta extraction + sync
  3. Copies viber.db + WAL while Viber is running (verified: shared read access works)
  4. Decrypts the copy, extracts new events, uploads to server
  5. Does NOT clean up local DB in watch mode (cleanup is a separate explicit action)

Why polling, not filesystem watching: Windows ReadDirectoryChangesW (underlying API for all FS watchers) is unreliable — drops events under load, sends duplicates, no guaranteed ordering. A simple poll every 30-60s is 100% reliable, zero CPU, and "near-realtime" is more than good enough for an archive.

2.10 Windows Service Installation

The agent can install/uninstall itself as a Windows service:

viberglass-agent service install     # Install as Windows service (requires admin)
viberglass-agent service uninstall   # Remove service
viberglass-agent service start       # Start the service
viberglass-agent service stop        # Stop the service
viberglass-agent service status      # Check if running

Implementation: Use golang.org/x/sys/windows/svc — Go's official Windows service package. The agent binary acts as both the CLI and the service:

  • When run from CLI: normal command execution
  • When run by SCM (Service Control Manager): enters service loop, runs watch-mode sync

Service details:

  • Service name: ViberglasSyncAgent
  • Display name: Viberglass Sync Agent
  • Startup type: Automatic (starts on boot)
  • Runs as LocalSystem (has access to all user SIDs for Viber DB decryption)
  • Logs to Windows Event Log + optional log file
  • Respects sync.interval from config for polling frequency
  • Handles Stop, Shutdown, and Interrogate service control signals gracefully

Install flow:

  1. viberglass-agent service install — copies config path into service registry, registers with SCM
  2. viberglass-agent service start — starts the polling loop
  3. Agent detects all Viber accounts under %APPDATA% for all user profiles (since it runs as LocalSystem)

Alternative: Scheduled Task For users who prefer not to run a service, the scheduled task approach still works:

$action = New-ScheduledTaskAction -Execute "C:\path\to\viberglass-agent.exe" -Argument "sync"
$trigger = New-ScheduledTaskTrigger -Once -At (Get-Date) -RepetitionInterval (New-TimeSpan -Minutes 5)
$settings = New-ScheduledTaskSettingsSet -StartWhenAvailable -DontStopIfGoingOnBatteries
Register-ScheduledTask -TaskName "Viberglass Sync" -Action $action -Trigger $trigger -Settings $settings

2.11 Error Handling & Retry

  • HTTP POST with exponential backoff (3 retries, 5s/15s/45s)
  • If server returns 409 (partial conflict), agent logs duplicates and advances watermark
  • If server is unreachable, agent exits cleanly — next run picks up from same watermark
  • State file only updated after confirmed server ACK
  • Temp files always cleaned up (deferred in Go)

3. Server

SvelteKit application serving both the REST API (for agents) and the web UI (for browsing).

3.1 Tech Stack

Component Technology
Framework SvelteKit (latest, TypeScript)
ORM Drizzle ORM
Database PostgreSQL 17
Blob storage SeaweedFS (S3-compatible API via filer)
UI components shadcn-svelte (npm installed)
Auth Lucia or custom session-based
Styling Tailwind CSS (comes with shadcn-svelte)

3.2 API Routes

# Agent sync
POST   /api/v1/sync                    # Ingest delta from agent
GET    /api/v1/sync/status              # Agent checks its sync status

# Agent management
POST   /api/v1/agents/register          # Register new agent (returns token)
GET    /api/v1/agents                   # List all agents
DELETE /api/v1/agents/:id               # Revoke agent

# Data access (for web UI)
GET    /api/v1/accounts                 # List phone numbers/accounts
GET    /api/v1/chats?accountId=&q=      # List chats with search
GET    /api/v1/chats/:id/messages       # Paginated messages for a chat
GET    /api/v1/contacts?accountId=&q=   # List contacts with search
GET    /api/v1/search?q=&accountId=     # Full-text search across messages
GET    /api/v1/media/:id                # Proxy media file from SeaweedFS
GET    /api/v1/stats                    # Dashboard statistics

# Export
GET    /api/v1/export                   # Generate Viberator-compatible backup
       ?phone=...&from=...&to=...&chatIds=...

4. Database Schema

4.1 PostgreSQL

-- =====================================================
-- ACCOUNTS & AGENTS
-- =====================================================

-- Phone numbers / Viber accounts being archived
CREATE TABLE accounts (
  id            SERIAL PRIMARY KEY,
  phone         TEXT NOT NULL UNIQUE,        -- "9607771234" (digits only, with country code)
  display_name  TEXT,                        -- friendly name for UI
  created_at    TIMESTAMPTZ DEFAULT now(),
  updated_at    TIMESTAMPTZ DEFAULT now()
);

-- Sync agents (machines sending data)
CREATE TABLE agents (
  id            SERIAL PRIMARY KEY,
  agent_id      TEXT NOT NULL UNIQUE,        -- machine identifier (hostname hash)
  name          TEXT,                        -- friendly name ("Office PC", "Laptop")
  auth_token    TEXT NOT NULL UNIQUE,        -- bearer token (bcrypt hash stored)
  account_id    INTEGER NOT NULL REFERENCES accounts(id),
  last_sync_at  TIMESTAMPTZ,
  last_event_id BIGINT DEFAULT 0,           -- high-water mark
  ip_address    TEXT,                        -- last seen IP
  user_agent    TEXT,                        -- agent version
  created_at    TIMESTAMPTZ DEFAULT now()
);

-- =====================================================
-- CONTACTS
-- =====================================================

CREATE TABLE contacts (
  id                SERIAL PRIMARY KEY,
  account_id        INTEGER NOT NULL REFERENCES accounts(id),
  viber_contact_id  BIGINT,                  -- original ContactID
  viber_mid         TEXT,                    -- Viber member ID (dedup key)
  encrypted_mid     TEXT,                    -- encrypted MID
  name              TEXT,
  phone_number      TEXT,
  client_name       TEXT,
  sort_name         TEXT,
  download_id       TEXT,
  contact_flags     BIGINT,
  viber_timestamp   BIGINT,                  -- Viber's timestamp (ms epoch)
  date_of_birth     TEXT,
  ab_contact        BIGINT DEFAULT 0,
  created_at        TIMESTAMPTZ DEFAULT now(),
  updated_at        TIMESTAMPTZ DEFAULT now(),
  UNIQUE(account_id, viber_mid)
);

CREATE INDEX idx_contacts_phone ON contacts(phone_number);
CREATE INDEX idx_contacts_name ON contacts(account_id, name);

-- =====================================================
-- CHATS
-- =====================================================

CREATE TABLE chats (
  id              SERIAL PRIMARY KEY,
  account_id      INTEGER NOT NULL REFERENCES accounts(id),
  viber_chat_id   BIGINT NOT NULL,           -- original ChatID
  chat_token      TEXT,                       -- from ChatInfo.Token (dedup key)
  chat_name       TEXT,                       -- group name or derived from contact
  is_group        BOOLEAN DEFAULT false,
  last_event_at   TIMESTAMPTZ,               -- denormalized for sort
  message_count   INTEGER DEFAULT 0,          -- denormalized for display
  viber_columns   JSONB,                      -- all other ChatInfo columns
  created_at      TIMESTAMPTZ DEFAULT now(),
  updated_at      TIMESTAMPTZ DEFAULT now(),
  UNIQUE(account_id, viber_chat_id)
);

CREATE INDEX idx_chats_last_event ON chats(account_id, last_event_at DESC);

-- Chat-Contact membership (from ChatRelation)
CREATE TABLE chat_members (
  id              SERIAL PRIMARY KEY,
  chat_id         INTEGER NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
  contact_id      INTEGER NOT NULL REFERENCES contacts(id),
  pg_role         INTEGER,                    -- Viber's PGRole field
  UNIQUE(chat_id, contact_id)
);

-- =====================================================
-- EVENTS (core — every Viber interaction)
-- =====================================================

CREATE TABLE events (
  id              SERIAL PRIMARY KEY,
  account_id      INTEGER NOT NULL REFERENCES accounts(id),
  viber_event_id  BIGINT NOT NULL,           -- original EventID
  viber_token     BIGINT NOT NULL,           -- dedup key (Viber assigns unique tokens)
  chat_id         INTEGER REFERENCES chats(id),
  contact_id      INTEGER REFERENCES contacts(id),
  event_type      INTEGER,                   -- Viber's Type field
  timestamp       TIMESTAMPTZ NOT NULL,      -- derived from Viber's ms epoch
  viber_timestamp BIGINT NOT NULL,           -- original ms epoch (for export)
  direction       SMALLINT,                  -- 0=incoming, 1=outgoing (from Direction column)
  flags           BIGINT,
  viber_columns   JSONB,                     -- all other Events columns (for lossless export)
  synced_at       TIMESTAMPTZ DEFAULT now(),
  synced_from     INTEGER REFERENCES agents(id),
  UNIQUE(account_id, viber_token)
);

CREATE INDEX idx_events_chat_ts ON events(chat_id, timestamp DESC);
CREATE INDEX idx_events_account_ts ON events(account_id, timestamp DESC);
CREATE INDEX idx_events_viber_eid ON events(account_id, viber_event_id);

-- =====================================================
-- MESSAGES (text content)
-- =====================================================

CREATE TABLE messages (
  id              SERIAL PRIMARY KEY,
  event_id        INTEGER NOT NULL UNIQUE REFERENCES events(id) ON DELETE CASCADE,
  body            TEXT,
  subject         TEXT,
  thumbnail       BYTEA,
  viber_columns   JSONB,                     -- all other Messages columns
  body_hash       TEXT,                      -- SHA-256 of body (secondary dedup)
  fts             TSVECTOR GENERATED ALWAYS AS (
    to_tsvector('simple', coalesce(body, '') || ' ' || coalesce(subject, ''))
  ) STORED
);

CREATE INDEX idx_messages_fts ON messages USING GIN(fts);

-- =====================================================
-- CALLS
-- =====================================================

CREATE TABLE calls (
  id              SERIAL PRIMARY KEY,
  event_id        INTEGER NOT NULL UNIQUE REFERENCES events(id) ON DELETE CASCADE,
  duration        INTEGER,
  call_type       INTEGER,
  viber_call_id   BIGINT,
  viber_columns   JSONB
);

-- =====================================================
-- MEDIA (deduplicated blobs in SeaweedFS)
-- =====================================================

-- One row per unique file (by content hash)
CREATE TABLE media (
  id              SERIAL PRIMARY KEY,
  sha256          TEXT NOT NULL UNIQUE,       -- content hash = global dedup key
  file_name       TEXT,                       -- representative filename
  content_type    TEXT,                       -- MIME type (image/jpeg, video/mp4, etc.)
  file_size       BIGINT,                     -- bytes
  seaweed_fid     TEXT NOT NULL,              -- SeaweedFS file ID
  created_at      TIMESTAMPTZ DEFAULT now()
);

-- Junction: events ↔ media (many-to-many)
CREATE TABLE event_media (
  id              SERIAL PRIMARY KEY,
  event_id        INTEGER NOT NULL REFERENCES events(id) ON DELETE CASCADE,
  media_id        INTEGER NOT NULL REFERENCES media(id),
  direction       TEXT NOT NULL,              -- 'download' or 'upload'
  viber_file_name TEXT,                       -- original TempFileName from Viber
  viber_columns   JSONB,                      -- all other DownloadFile/UploadFile columns
  UNIQUE(event_id, media_id)
);

CREATE INDEX idx_event_media_event ON event_media(event_id);
CREATE INDEX idx_event_media_media ON event_media(media_id);

-- =====================================================
-- METADATA & REACTIONS
-- =====================================================

CREATE TABLE event_metadata (
  id              SERIAL PRIMARY KEY,
  event_id        INTEGER NOT NULL REFERENCES events(id) ON DELETE CASCADE,
  viber_columns   JSONB                       -- full EventsMetaData row
);

CREATE TABLE like_relations (
  id              SERIAL PRIMARY KEY,
  event_id        INTEGER NOT NULL REFERENCES events(id) ON DELETE CASCADE,
  viber_columns   JSONB                       -- full LikeRelation row
);

-- =====================================================
-- SYNC LOG (audit trail)
-- =====================================================

CREATE TABLE sync_log (
  id              SERIAL PRIMARY KEY,
  agent_id        INTEGER NOT NULL REFERENCES agents(id),
  sync_id         UUID NOT NULL DEFAULT gen_random_uuid(),
  from_event_id   BIGINT,
  to_event_id     BIGINT,
  events_received INTEGER DEFAULT 0,
  events_stored   INTEGER DEFAULT 0,
  duplicates      INTEGER DEFAULT 0,
  media_files     INTEGER DEFAULT 0,
  media_bytes     BIGINT DEFAULT 0,
  started_at      TIMESTAMPTZ DEFAULT now(),
  completed_at    TIMESTAMPTZ,
  status          TEXT DEFAULT 'processing',  -- processing | complete | failed
  error           TEXT
);

-- =====================================================
-- WEB AUTH (for UI login)
-- =====================================================

CREATE TABLE users (
  id              SERIAL PRIMARY KEY,
  username        TEXT NOT NULL UNIQUE,
  password_hash   TEXT NOT NULL,
  created_at      TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE sessions (
  id              TEXT PRIMARY KEY,            -- session token
  user_id         INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  expires_at      TIMESTAMPTZ NOT NULL
);

4.2 Key Schema Decisions

  1. viber_columns JSONB on most tables — stores ALL original Viber columns as-is. This ensures lossless round-trip: we can reconstruct the exact Viber DB schema for export without losing columns we didn't explicitly model. Structured fields (body, timestamp, etc.) are extracted for indexing/querying, but the full row is always preserved.

  2. viber_event_id + viber_token — we store both. viber_token is the dedup key (globally unique per account). viber_event_id is needed for export (Viberator restore uses EventID).

  3. viber_timestamp BIGINT — original millisecond epoch preserved alongside the PostgreSQL TIMESTAMPTZ. Export needs the exact original value.


5. SeaweedFS Integration

5.1 Why SeaweedFS

  • Efficient for many small-to-medium files (typical Viber media: photos 100KB–5MB, videos up to 100MB+)
  • S3-compatible API via the filer (standard tooling works)
  • Content-addressable storage pairs naturally with SHA-256 dedup
  • Self-hosted, no cloud dependency
  • Handles millions of files without filesystem inode exhaustion

5.2 Storage Flow

Agent sends media file
        │
        ▼
Server computes SHA-256
        │
        ├── EXISTS in media table?
        │     YES → create event_media link → skip upload
        │
        └── NEW → upload to SeaweedFS
                   │
                   ▼
              SeaweedFS returns fid
                   │
                   ▼
              INSERT media row (sha256, fid)
              INSERT event_media link

5.3 Upload to SeaweedFS

Via filer HTTP API:

POST http://seaweed-filer:8888/viber/<phone>/<YYYY-MM>/<sha256_prefix>_<filename>
Content-Type: image/jpeg
Body: <binary>

Response: { "name": "...", "size": 12345 }

The seaweed_fid stored in the media table is the full filer path, used to proxy back to users.

5.4 Serving Media

GET /api/v1/media/:id
→ Look up media.seaweed_fid
→ Proxy from SeaweedFS with appropriate Content-Type header
→ Cache-Control: immutable (content-addressed = never changes)

6. Deduplication Strategy

Dedup is per-record at every level, not batch-level.

6.1 Events

Key: (account_id, viber_token)

Viber assigns a unique Token (int64) to every event. Combined with the account, this is globally unique. If two agents send the same event, the second insert is skipped (UNIQUE constraint → ON CONFLICT DO NOTHING).

6.2 Messages

Key: Follows event dedup (1:1 relationship via event_id)

If the event is a duplicate, the message is never inserted. The body_hash field provides a secondary check for edge cases where Viber reuses event IDs across DB rebuilds.

6.3 Media Files

Key: SHA-256 content hash

Before uploading to SeaweedFS, the server hashes the file content:

  • If media.sha256 exists → reuse the existing seaweed_fid, only create a new event_media link
  • If new → upload to SeaweedFS → create media row + event_media link

This deduplicates across events, chats, agents, and even accounts (same photo shared by two different people = stored once).

6.4 Contacts

Key: (account_id, viber_mid)

Upsert on conflict — update name, phone, etc. if they've changed.

6.5 Chats

Key: (account_id, viber_chat_id)

Upsert on conflict — update chat name, metadata.

6.6 Ingest Transaction Flow

BEGIN TRANSACTION

For each contact in payload:
  INSERT INTO contacts ... ON CONFLICT (account_id, viber_mid) DO UPDATE SET ...

For each chat in payload:
  INSERT INTO chats ... ON CONFLICT (account_id, viber_chat_id) DO UPDATE SET ...
  Upsert chat_members

For each event in payload:
  INSERT INTO events ... ON CONFLICT (account_id, viber_token) DO NOTHING
  → If inserted (not duplicate):
    INSERT message (if present)
    INSERT call (if present)
    For each media file:
      Compute SHA-256
      SELECT id FROM media WHERE sha256 = ?
      → If exists: INSERT event_media link only
      → If new: Upload to SeaweedFS → INSERT media → INSERT event_media
    INSERT event_metadata (if present)
    INSERT like_relations (if present)

Update agent.last_sync_at, agent.last_event_id
INSERT sync_log

COMMIT

7. Ingest API

7.1 Endpoint

POST /api/v1/sync
Authorization: Bearer <agent-token>
Content-Type: multipart/form-data

7.2 Processing Steps

  1. Authenticate — verify bearer token against agents.auth_token
  2. Parse payload — extract JSON part, validate schema
  3. Create sync_log entry (status: processing)
  4. Map contacts — upsert, build viber_contact_id → server_contact_id map
  5. Map chats — upsert, build viber_chat_id → server_chat_id map
  6. Insert events — with remapped chat_id and contact_id, skip duplicates
  7. Insert child records — messages, calls, metadata, reactions (only for newly inserted events)
  8. Process media — for each media part:
    • Compute SHA-256
    • Check existing → link or upload+link
  9. Update denormalized fieldschats.last_event_at, chats.message_count
  10. Update agent watermarkagents.last_event_id, agents.last_sync_at
  11. Complete sync_log — counts, status: complete
  12. Return response

7.3 Response

{
  "syncId": "uuid",
  "status": "complete",
  "accepted": 1234,
  "duplicates": 56,
  "mediaStored": 23,
  "mediaDeduped": 5,
  "errors": 0
}

7.4 Error Cases

Scenario HTTP Status Behavior
Invalid/expired token 401 Reject
Malformed payload 400 Reject with validation errors
Partial failure (some events fail) 207 Accept good ones, report failures
Server error mid-transaction 500 Rollback, agent retries
All events are duplicates 200 Accept (idempotent), report duplicate count

8. Web UI

8.1 Layout

Sidebar navigation + main content area. Dark/light mode toggle.

┌──────────┬────────────────────────────────┐
│          │                                │
│  📊 Dash │   [Main Content Area]          │
│  💬 Chats│                                │
│  👤 Cont.│                                │
│  🔍 Search                                │
│  🤖 Agents                                │
│  ⚙️ Settings                              │
│          │                                │
│  [Account│                                │
│  Switcher]│                               │
└──────────┴────────────────────────────────┘

8.2 Pages

Dashboard

  • Total messages archived (per account and total)
  • Total media size (from SeaweedFS)
  • Number of active agents
  • Last sync time per agent
  • Messages per day chart (simple line/bar)
  • Storage breakdown

Conversations

  • List of chats sorted by last message timestamp
  • Avatar (first letter / contact photo if available), chat name, last message preview, timestamp
  • Search/filter bar
  • Badge for unread count since last viewed (optional)
  • Click → opens chat view

Chat View

  • Viber/WhatsApp-style message bubbles
  • Incoming (left, gray) / outgoing (right, colored)
  • Timestamps, grouped by date
  • Media rendered inline: images as thumbnails (click to expand), video with player, audio with player
  • Call events shown as system messages ("Voice call — 2:34")
  • Infinite scroll: load older messages on scroll up
  • Jump to date picker

Contacts

  • Searchable list: name, phone number, Viber MID
  • Click → see all chats with this contact

Search

  • Full-text search input
  • Results show: message snippet with highlighted match, chat name, timestamp
  • Click → jumps to that message in chat view
  • Filters: account, date range, chat

Agents

  • List of registered agents with status
  • Last sync time, events synced, IP address
  • Revoke/rename actions
  • Register new agent (generates token to paste into agent config)

Settings

  • Account management (rename, delete)
  • Create web UI user accounts (if multi-user)
  • Retention policies
  • SeaweedFS stats

8.3 shadcn-svelte Components

All installed via npm (npx shadcn-svelte@latest add <component>):

Component Usage
Button Actions everywhere
Card Dashboard stats, agent cards
Table Agent list, contacts
Input Search bars, forms
Dialog Confirmations, agent registration
Sheet Mobile sidebar
Avatar Chat list, contacts
Badge Status indicators
Separator Section dividers
ScrollArea Chat message container
Tooltip Timestamps, metadata
Command Search palette (Cmd+K)
DropdownMenu Context menus
Tabs Dashboard sections
Calendar Date range picker for export
Popover Date picker container
Select Account switcher, filters
Skeleton Loading states
Sonner (toast) Sync notifications, errors

9. Viberator-Compatible Export

The server generates .db files that viberator restore can import directly.

9.1 UI Flow

  1. Navigate to export page (or use export button on any chat/dashboard)
  2. Select account (phone number)
  3. Pick date range using shadcn-svelte Calendar component
  4. Optionally filter by specific chats
  5. Click "Export"
  6. Server generates file → browser downloads it

9.2 API

GET /api/v1/export?phone=9607771234&from=2025-01-01&to=2026-01-01&chatIds=1,2,3
Accept: application/octet-stream
Content-Disposition: attachment; filename="viber_backup_9607771234_20250101_20260101.db"

For exports with media:

GET /api/v1/export?phone=9607771234&from=2025-01-01&to=2026-01-01&includeMedia=true
→ Returns ZIP:
   viber_export_9607771234/
   ├── viber_backup.db
   └── media/
       ├── photo_001.jpg
       ├── video_002.mp4
       └── ...

9.3 Generation Process

  1. Create temporary SQLite database (in memory or tmpdir)
  2. Recreate Viber's original schema — same table names, same column names, same types:
    -- Must match Viber's actual schema exactly
    CREATE TABLE Events (EventID INTEGER PRIMARY KEY, ...)
    CREATE TABLE Messages (EventID INTEGER, Body TEXT, ...)
    CREATE TABLE Contact (ContactID INTEGER PRIMARY KEY, Name TEXT, Number TEXT, MID TEXT, ...)
    CREATE TABLE ChatInfo (ChatID INTEGER PRIMARY KEY, ...)
    CREATE TABLE ChatRelation (ChatID INTEGER, ContactID INTEGER, PGRole INTEGER)
    CREATE TABLE DownloadFile (EventID INTEGER, TempFileName TEXT, ...)
    CREATE TABLE UploadFile (EventID INTEGER, ...)
    CREATE TABLE Calls (EventID INTEGER, ...)
    CREATE TABLE EventsMetaData (EventID INTEGER, ...)
    CREATE TABLE LikeRelation (LikeEventID INTEGER, ...)
  3. Query PostgreSQL for events in date range → map back to Viber column format:
    • Use viber_columns JSONB to reconstruct full rows (lossless)
    • events.viber_event_idEvents.EventID
    • events.viber_tokenEvents.Token
    • events.viber_timestampEvents.TimeStamp (ms epoch)
    • Remap chat_id and contact_id back to sequential IDs for the export
  4. Insert all data into the temp SQLite
  5. Add _backup_info metadata table (Viberator format):
    CREATE TABLE _backup_info (key TEXT PRIMARY KEY, value TEXT);
    INSERT INTO _backup_info VALUES ('checksum', '<sha256>');
    INSERT INTO _backup_info VALUES ('created', '2026-02-06T03:00:00Z');
    INSERT INTO _backup_info VALUES ('phone', '9607771234');
    INSERT INTO _backup_info VALUES ('events', '12345');
    INSERT INTO _backup_info VALUES ('messages', '11000');
    INSERT INTO _backup_info VALUES ('source', 'viberglass');
  6. Compute SHA-256 checksum (same algorithm as Viberator):
    For each Event ordered by EventID: hash "E:{EventID}:{TimeStamp}:{Token}|"
    For each Message ordered by EventID: hash "M:{EventID}:{Body}|"
    Final: hex(sha256(all_hashed_data))
    
  7. Stream the .db file as download response

9.4 Media in Exports

When includeMedia=true:

  • Query event_media + media for all events in range
  • Download each file from SeaweedFS
  • DownloadFile.TempFileName in the SQLite points to media/<filename>
  • Bundle as ZIP with the .db file

9.5 Compatibility Contract

The export MUST work with viberator restore --input exported.db. This means:

  • Same table names and column names as Viber's DB
  • Same data types (INTEGER for IDs, TEXT for strings, etc.)
  • _backup_info table with checksum key
  • Checksum computed identically to Viberator
  • EventID auto-increment (not our server IDs)
  • ContactID, ChatID remapped to sequential integers

10. Multi-Number Support

Each Viber phone number is an account. The system is designed for multiple accounts from day one.

10.1 Data Isolation

All data tables have account_id foreign key. Queries always filter by account. No cross-account data leakage.

10.2 Agent-Account Binding

Each agent is bound to one account (one Viber installation per machine). Multiple agents can feed the same account (e.g., same number on desktop + laptop). Dedup handles overlap.

10.3 UI

  • Account switcher in sidebar (dropdown with phone numbers + display names)
  • "All accounts" view option for search and dashboard
  • Export is always per-account (Viberator backup is phone-specific)

11. Authentication & Security

11.1 Agent Auth

  • Agents authenticate via bearer token in Authorization header
  • Token generated during registration (POST /api/v1/agents/register)
  • Server stores bcrypt hash of token
  • Tokens can be revoked via web UI

11.2 Web UI Auth

  • Session-based authentication (HTTP-only cookie)
  • Username + password login (bcrypt hashed)
  • Single-user by default; multi-user optional
  • CSRF protection via SvelteKit's built-in handling

11.3 Transport Security

  • HTTPS required for production (agents send sensitive message data)
  • Self-signed certs acceptable for LAN deployment (agent config can skip TLS verify)

11.4 Data at Rest

  • PostgreSQL: standard disk encryption (OS-level or PG TDE if desired)
  • SeaweedFS: stored unencrypted (self-hosted, trusted environment)
  • No application-level encryption of stored messages (trade-off: enables full-text search)

12. Deployment

12.1 Docker Compose

services:
  app:
    build: ./server
    ports: ["3000:3000"]
    environment:
      DATABASE_URL: postgres://viberglass:viberglass@db:5432/viberglass
      SEAWEED_FILER_URL: http://seaweed-filer:8888
      AUTH_SECRET: ${AUTH_SECRET}
      ORIGIN: ${ORIGIN:-http://localhost:3000}
    depends_on:
      db:
        condition: service_healthy
      seaweed-filer:
        condition: service_started
    restart: unless-stopped

  db:
    image: postgres:17-alpine
    volumes: ["pgdata:/var/lib/postgresql/data"]
    environment:
      POSTGRES_DB: viberglass
      POSTGRES_USER: viberglass
      POSTGRES_PASSWORD: viberglass
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U viberglass"]
      interval: 5s
      timeout: 3s
      retries: 5
    restart: unless-stopped

  seaweed-master:
    image: chrislusf/seaweedfs
    command: master -ip=seaweed-master -defaultReplication=000
    ports: ["9333:9333"]
    volumes: ["swmaster:/data"]
    restart: unless-stopped

  seaweed-volume:
    image: chrislusf/seaweedfs
    command: volume -mserver=seaweed-master:9333 -port=8080 -max=100
    volumes: ["swvolume:/data"]
    depends_on: [seaweed-master]
    restart: unless-stopped

  seaweed-filer:
    image: chrislusf/seaweedfs
    command: filer -master=seaweed-master:9333
    ports: ["8888:8888"]
    depends_on: [seaweed-master, seaweed-volume]
    restart: unless-stopped

volumes:
  pgdata:
  swmaster:
  swvolume:

12.2 Environment Variables

DATABASE_URL=postgres://viberglass:viberglass@db:5432/viberglass
SEAWEED_FILER_URL=http://seaweed-filer:8888
AUTH_SECRET=<random-32-char-string>
ORIGIN=https://viberglass.example.com

12.3 Reverse Proxy

Put behind nginx/Caddy for HTTPS:

viberglass.example.com → localhost:3000

13. Project Structure

13.1 Agent (viberglass/agent/ — Go)

agent/
├── main.go                     # CLI entry point (cobra or flag)
├── go.mod
├── go.sum
├── internal/
│   ├── crypto/
│   │   ├── aes.go              # AES-128-OFB encrypt/decrypt (from Viberator)
│   │   ├── key.go              # SID-based key derivation
│   │   └── wal.go              # WAL handling
│   ├── viber/
│   │   ├── db.go               # Open, read, query Viber DB
│   │   ├── delta.go            # Delta extraction logic
│   │   ├── schema.go           # Viber table/column definitions
│   │   └── path.go             # Auto-detect Viber DB path
│   ├── sync/
│   │   ├── client.go           # HTTP client for server API
│   │   ├── payload.go          # Build multipart payload
│   │   └── state.go            # Read/write sync state file
│   ├── cleanup/
│   │   └── cleanup.go          # Post-sync local DB cleanup
│   └── config/
│       └── config.go           # TOML config parsing
├── .goreleaser.yml             # Build releases
└── README.md

13.2 Server (viberglass/server/ — SvelteKit)

server/
├── src/
│   ├── lib/
│   │   ├── server/
│   │   │   ├── db/
│   │   │   │   ├── schema.ts           # Drizzle schema (all tables)
│   │   │   │   ├── client.ts           # DB connection
│   │   │   │   └── migrate.ts          # Migration runner
│   │   │   ├── seaweed/
│   │   │   │   └── client.ts           # SeaweedFS upload/download
│   │   │   ├── auth/
│   │   │   │   ├── session.ts          # Web session management
│   │   │   │   └── agent-token.ts      # Agent token verification
│   │   │   ├── ingest/
│   │   │   │   ├── processor.ts        # Main ingest orchestrator
│   │   │   │   ├── dedup.ts            # Dedup logic
│   │   │   │   └── media.ts            # Media processing + SeaweedFS
│   │   │   └── export/
│   │   │       ├── generator.ts        # SQLite backup generator
│   │   │       └── checksum.ts         # Viberator-compatible checksum
│   │   ├── components/
│   │   │   └── chat/
│   │   │       ├── message-bubble.svelte
│   │   │       ├── chat-list-item.svelte
│   │   │       └── date-separator.svelte
│   │   └── utils/
│   │       └── format.ts               # Date formatting, file sizes, etc.
│   ├── routes/
│   │   ├── (app)/                       # Authenticated layout group
│   │   │   ├── +layout.svelte          # Sidebar + account switcher
│   │   │   ├── +layout.server.ts       # Auth guard
│   │   │   ├── dashboard/
│   │   │   │   └── +page.svelte
│   │   │   ├── chats/
│   │   │   │   ├── +page.svelte        # Chat list
│   │   │   │   └── [id]/
│   │   │   │       └── +page.svelte    # Chat view
│   │   │   ├── contacts/
│   │   │   │   └── +page.svelte
│   │   │   ├── search/
│   │   │   │   └── +page.svelte
│   │   │   ├── agents/
│   │   │   │   └── +page.svelte
│   │   │   ├── export/
│   │   │   │   └── +page.svelte        # Export with date range picker
│   │   │   └── settings/
│   │   │       └── +page.svelte
│   │   ├── api/v1/
│   │   │   ├── sync/
│   │   │   │   └── +server.ts
│   │   │   ├── agents/
│   │   │   │   ├── +server.ts
│   │   │   │   ├── register/
│   │   │   │   │   └── +server.ts
│   │   │   │   └── [id]/
│   │   │   │       └── +server.ts
│   │   │   ├── chats/
│   │   │   │   ├── +server.ts
│   │   │   │   └── [id]/
│   │   │   │       └── messages/+server.ts
│   │   │   ├── contacts/
│   │   │   │   └── +server.ts
│   │   │   ├── search/
│   │   │   │   └── +server.ts
│   │   │   ├── media/
│   │   │   │   └── [id]/
│   │   │   │       └── +server.ts
│   │   │   ├── export/
│   │   │   │   └── +server.ts
│   │   │   └── stats/
│   │   │       └── +server.ts
│   │   └── login/
│   │       ├── +page.svelte
│   │       └── +page.server.ts
│   └── app.html
├── drizzle/                            # Generated migrations
├── drizzle.config.ts
├── static/
├── svelte.config.js
├── tailwind.config.js
├── components.json                     # shadcn-svelte config
├── package.json
├── tsconfig.json
├── Dockerfile
└── README.md

14. Implementation Phases

Phase 1: Server Foundation (3-4 days)

  • Init SvelteKit project + TypeScript
  • Set up Drizzle ORM with PostgreSQL
  • Create all migrations (full schema from Section 4)
  • Install shadcn-svelte + Tailwind + core components
  • Basic layout: sidebar, dark/light mode, responsive
  • Login page + session auth
  • Docker Compose with PG + SeaweedFS
  • SeaweedFS client (upload, download, delete)
  • Health check endpoint

Phase 2: Ingest API (2-3 days)

  • POST /api/v1/sync — full multipart ingest pipeline
  • Contact/chat upsert with dedup
  • Event insert with (account_id, viber_token) dedup
  • Message/call/metadata insert
  • Media processing: SHA-256 dedup + SeaweedFS upload
  • Sync log tracking
  • Agent auth middleware (bearer token)
  • POST /api/v1/agents/register
  • GET /api/v1/agents
  • Integration tests

Phase 3: Sync Agent (3-4 days)

  • Extract Viberator crypto into internal/crypto/ package
  • Viber DB reader: open, decrypt, query
  • Delta extraction with watermark tracking
  • Multipart POST builder with media files
  • HTTP client with retry logic
  • Post-sync cleanup (from Viberator)
  • Exclusive DB lock during sync
  • TOML config parsing
  • State file persistence
  • sync command (one-shot)
  • sync --watch (polling loop)
  • register command
  • status command
  • Windows scheduled task helper docs
  • Build for Windows amd64

Phase 4: Web UI — Core (3-4 days)

  • Dashboard: stats cards, last sync times
  • Conversations list: sorted by last message, search, account filter
  • Chat view: message bubbles, timestamps, date separators
  • Chat view: inline media (images, video, audio)
  • Chat view: call events as system messages
  • Chat view: infinite scroll (load older on scroll up)
  • Contact list with search
  • Account switcher in sidebar
  • Media proxy endpoint (GET /api/v1/media/:id)

Phase 5: Search, Export & Polish (2-3 days)

  • Full-text search (PostgreSQL tsvector)
  • Search results with context + chat navigation
  • Export page: date range picker (shadcn Calendar)
  • Export API: generate Viberator-compatible SQLite backup
  • Export: checksum computation (matching Viberator algorithm)
  • Export: optional media ZIP bundle
  • Agent management page (rename, revoke)
  • Settings page
  • Media gallery view per chat
  • Cmd+K search palette
  • Loading states, error handling, empty states
  • Mobile responsive pass

15. Viber DB Reference

For export compatibility, the exact Viber SQLite schema must be understood. Key tables and their columns:

Events

The core table. Every interaction (message, call, system event) is an event.

Column Type Notes
EventID INTEGER PK Auto-increment
Type INTEGER Event type (message, call, etc.)
TimeStamp INTEGER Millisecond epoch
Token INTEGER Unique per event (dedup key)
ChatID INTEGER FK to ChatInfo
ContactID INTEGER FK to Contact
Direction INTEGER 0=incoming, 1=outgoing
Body TEXT (in older schemas, also in Messages)
... ... Many more columns

Messages

Text content for message events.

Column Type Notes
EventID INTEGER FK to Events
Body TEXT Message text
Subject TEXT Subject line (rare)
... ... Thumbnails, formatting, etc.

Contact

Viber contacts.

Column Type Notes
ContactID INTEGER PK
Name TEXT Display name
Number TEXT Phone number
MID TEXT Viber member ID
EncryptedMID TEXT
ClientName TEXT
SortName TEXT
ContactFlags INTEGER
Timestamp INTEGER
DateOfBirth TEXT
ABContact INTEGER

ChatInfo

Chat threads (1:1 and groups).

Column Type Notes
ChatID INTEGER PK
Token TEXT Chat token (dedup key)
... ... Name, flags, metadata

ChatRelation

Chat ↔ Contact membership.

Column Type Notes
ChatID INTEGER FK to ChatInfo
ContactID INTEGER FK to Contact
PGRole INTEGER Role in group

DownloadFile

Media files received.

Column Type Notes
EventID INTEGER FK to Events
TempFileName TEXT Filename on disk
... ... Size, type, status

UploadFile

Media files sent.

Calls

Call events.

EventsMetaData

Additional event metadata.

LikeRelation

Reactions to messages.


Plan created: 2026-02-06 Project: Viberglass Repos: viberglass/agent (Go), viberglass/server (SvelteKit)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment