WhatsApp Conversations Feature

Overview

The WhatsApp Conversations feature enables automated, personalized WhatsApp messaging flows for customers who interact with places through Forkast. When a customer taps the WhatsApp button in a PasosModule step, a conversation is automatically created and tracked in the database, enabling contextual, fork-specific replies without requiring the customer to send any codes or identifiers.

Implementation Status: ✅ Production Ready

• ✅ conversations collection with 9 indexes (including TTL)
• ✅ Multi-channel form_templates architecture (pasos-form, whatsapp, email)
• ✅ Default WhatsApp templates created
• ✅ Conversation creation API (fully implemented)
• ✅ Webhook conversation flow processing (fully implemented)
• ✅ Phone lookup and message routing (fully implemented)
• ✅ Two-tree architecture (bot-initiated + user-initiated flows)
• ✅ Variable replacement engine (fully implemented)
• ✅ Message history tracking (fully implemented)
• 🚧 Place.forks.forms database migration (script ready, not executed)
• ✅ Visual conversation flow builder UI (implemented)
• ✅ Place edit form integration (implemented)

Architecture

Core Concept

Instead of parsing codes from customer messages, we use phone number lookup to match incoming WhatsApp messages to existing conversations. This provides a seamless user experience while maintaining full conversation tracking and history.

Key Components

1. Conversations Collection - New MongoDB collection storing all WhatsApp interactions
2. Conversation Document - Created when customer taps WhatsApp button in PasosModule
3. Phone Number Lookup - Incoming messages matched to conversations by customer phone
4. 72-Hour Expiration - Conversations remain active for 72 hours, then new ones are created
5. Message History - All received and sent messages tracked in arrays

---

Database Schema

Conversation Document

interface Conversation {
  _id: ObjectId;
  shortId: string; // Unique ID with "co_" prefix (e.g., "co_a1b2c3")
  type: 'whatsapp' | 'email' | 'sms' | 'voice'; // Communication channel type

  // Customer Information
  customer: {
    phone: string; // E.164 format: "+1234567890" - for quick lookup
    id: ObjectId; // Reference to customers._id (source of truth for name, email, etc.)
  };

  // Place Information
  place: {
    shortId: string; // "restaurantabc" - reference to places.shortId
    id?: ObjectId; // Optional: reference to places._id
  };

  // Visit Information
  visit: {
    id: ObjectId; // Reference to visits._id (source of truth for visit metadata)
    fork: 'instagram' | 'google'; // Which fork they came from (denormalized for quick filtering)
  };

  // Message History
  receivedMessages: Array<{
    messageId: string; // Twilio MessageSid
    body: string; // Message content
    receivedAt: Date; // When received
    from: string; // Sender (whatsapp:+1234567890)
    profileName?: string; // WhatsApp profile name
  }>;

  sentMessages: Array<{
    messageId?: string; // Twilio MessageSid (if available)
    body: string; // Message content
    sentAt: Date; // When sent
    to: string; // Recipient (whatsapp:+1234567890)
    status?: 'sent' | 'delivered' | 'read' | 'failed'; // Delivery status
  }>;

  // Lifecycle
  status: 'active' | 'expired'; // Active within 72 hours
  createdAt: Date; // When conversation was created
  lastActivityAt: Date; // Last message sent or received
  expiresAt: Date; // createdAt + 72 hours

  // Metadata
  metadata?: {
    source?: string; // How conversation was initiated
    deviceType?: string;
    userAgent?: string;
  };
}

✅ Indexes (CREATED)

Database: forkast.conversations

Created by: scripts/create-conversation-indexes-correct-db.js

// ✅ 9 indexes created and active:
1. idx_customer_phone_type_expires - { 'customer.phone': 1, type: 1, expiresAt: 1 }
2. idx_place_created - { placeId: 1, createdAt: -1 }
3. idx_status_updated - { status: 1, updatedAt: -1 }
4. idx_current_step - { 'flow.currentStepId': 1 }
5. idx_short_id - { shortId: 1 } (unique)
6. idx_visit_id - { visitId: 1 }
7. idx_active_conversations - { status: 1, expiresAt: 1 }
8. idx_expiresAt_ttl - { expiresAt: 1 } (TTL index, expireAfterSeconds: 0)
9. _id - Default MongoDB index

Note: Actual schema uses placeId (ObjectId) instead of place.shortId for better performance

---

User Flow

1. Customer Arrives at Place Page

Customer scans QR code → /ig/restaurantabc

2. Customer Completes PasosModule Steps

Step 1: Enter email → Creates/updates customer record
Step 2: Enter phone → Updates customer with phone number
Step 3: Tap WhatsApp button

3. WhatsApp Button Tap (Frontend)

Location: src/components/place/InputTypes/WhatsAppButton.tsx

Process:

1. Customer has already entered phone number in previous step
2. Create conversation document via API call (links phone → customer → visit → place)
3. Replace {{place.shortId}} and {{place.name}} in pre-filled message
4. Open WhatsApp with processed message

Pre-filled Message Example:

"Hola! Me gustaría participar en el sorteo semanal en {{place.name}}."

Important Notes:

• ✅ No code/ID needed in message - phone number lookup handles everything!
• ✅ Conversation already created - when they send the message, we match by phone
• ✅ Seamless UX - customer doesn't need to remember or type any codes
• ⚠️ Don't include {{conversation.shortId}} - not needed for lookup (phone is the key)

4. Conversation Creation (Backend)

✅ Endpoint: POST /api/conversations/create (FULLY IMPLEMENTED)

Location: src/app/api/conversations/create/route.ts

Request:

{
  "type": "whatsapp",
  "customer": {
    "phone": "+16463345377",
    "id": "507f1f77bcf86cd799439011"
  },
  "place": {
    "shortId": "restaurantabc"
  },
  "visit": {
    "id": "507f1f77bcf86cd799439012",
    "fork": "instagram"
  }
}

Response:

{
  "success": true,
  "conversation": {
    "shortId": "co_a1b2c3",
    "expiresAt": "2025-10-29T12:00:00Z"
  }
}

Logic:

// Check if active conversation exists for this phone and type
const existingConversation = await conversations.findOne({
  'customer.phone': phone,
  type: 'whatsapp', // Only match same communication channel
  expiresAt: { $gte: new Date() }
});

if (existingConversation) {
  // Update existing conversation
  return existingConversation;
} else {
  // Create new conversation
  const newConversation = {
    shortId: generateShortId('co_'),
    type: 'whatsapp',
    customer: {
      phone: phone,
      id: new ObjectId(customerId)
    },
    place: {
      shortId: placeShortId
    },
    visit: {
      id: new ObjectId(visitId),
      fork: fork
    },
    receivedMessages: [],
    sentMessages: [],
    status: 'active',
    createdAt: new Date(),
    lastActivityAt: new Date(),
    expiresAt: new Date(Date.now() + 72 * 60 * 60 * 1000) // 72 hours
  };

  await conversations.insertOne(newConversation);
  return newConversation;
}

5. Customer Sends WhatsApp Message

Customer taps "Send" in WhatsApp → Message sent to Twilio number

6. Twilio Webhook Receives Message

✅ Endpoint: POST /api/twilio/whatsapp (FULLY IMPLEMENTED)

Location: src/app/api/twilio/whatsapp/route.ts

Current Status:

• ✅ Webhook endpoint receives and validates Twilio messages
• ✅ Phone lookup and customer creation fully operational
• ✅ Conversation lookup by phone + place implemented
• ✅ Template-based reply messages with variable replacement
• ✅ Message history tracking (receivedMessages + sentMessages)
• ✅ Two-tree architecture (bot-initiated + user-initiated flows)
• ✅ Priority-based keyword/regex matching
• ✅ Step progression and state management

Incoming Data:

{
  MessageSid: "SM1234567890",
  From: "whatsapp:+16463345377",
  To: "whatsapp:+14155238886",
  Body: "Hola! Me gustaría participar en el sorteo semanal.",
  ProfileName: "Joe Saavedra"
}

✅ ACTUAL Processing Logic (FULLY IMPLEMENTED):

The webhook implements a sophisticated two-tree architecture for handling conversations:

Two-Tree Architecture

1. Main Conversation Tree (conversationTree):

• Sequential, bot-initiated steps
• Automatically progresses through predefined steps
• Tracks currentStepIndex in conversation
• Example: Welcome → Ask Question → Confirmation → End

2. User-Initiated Tree (userInitiatedTree):

• Triggered AFTER main tree completes
• Responds to customer-initiated messages
• Three trigger types:
  • keyword_match: Match specific keywords
  • regex_match: Match regex patterns
  • sequential: Cycle through responses in order
• Priority-based matching (higher priority checked first)

Processing Flow:

// 1. Extract phone number and find most recent visit
const customerPhone = extractPhoneNumber(webhook.From); // "+16463345377"
const visit = await findMostRecentVisitByPhone(customerPhone);

if (!visit) {
  return genericResponse();
}

// 2. Find active conversation for this phone + place
const conversation = await conversations.findOne({
  'customer.phone': customerPhone,
  'place.shortId': visit.placeShortId,
  type: 'whatsapp',
  status: 'active',
  expiresAt: { $gte: new Date() }
});

// 3. Create new conversation if none exists or expired
if (!conversation) {
  conversation = await createOrRenewConversation(customerPhone, visit);
}

// 4. Load place and WhatsApp template
const place = await places.findOne({ shortId: visit.placeShortId });
const whatsappTemplateId = place.forks?.[visit.fork]?.forms?.whatsapp?.formId || 'default-whatsapp';
const template = await getFormTemplates(whatsappTemplateId);

// 5. Determine which message to send
let responseMessage;

if (conversation.currentStepIndex < template.conversationTree.length) {
  // Still in main conversation tree
  const currentStep = template.conversationTree[conversation.currentStepIndex];
  responseMessage = currentStep.message.text;

  // Advance to next step
  await conversations.updateOne(
    { _id: conversation._id },
    { $inc: { currentStepIndex: 1 } }
  );
} else if (template.userInitiatedTree) {
  // Main tree complete, check user-initiated responses
  const matchedStep = findMatchingUserInitiatedStep(template.userInitiatedTree, webhook.Body);
  responseMessage = matchedStep?.message.text || defaultResponse;
} else {
  // No user-initiated tree, use default response
  responseMessage = "Thanks for your message!";
}

// 6. Replace variables in message
const context = {
  customer: await getCustomer(conversation.customer.id),
  place: place,
  conversation: conversation,
  currentFork: visit.fork
};
const finalMessage = replaceVariables(responseMessage, context);

// 7. Track message history
await conversations.updateOne(
  { _id: conversation._id },
  {
    $push: {
      receivedMessages: {
        messageId: webhook.MessageSid,
        body: webhook.Body,
        receivedAt: new Date(),
        from: webhook.From,
        profileName: webhook.ProfileName
      },
      sentMessages: {
        body: finalMessage,
        sentAt: new Date(),
        to: webhook.From,
        status: 'sent'
      }
    },
    $set: { lastActivityAt: new Date() }
  }
);

// 8. Send TwiML reply
const twiml = new MessagingResponse();
twiml.message(finalMessage);
return twiml;

Priority-Based User-Initiated Matching

When multiple user-initiated steps exist, the webhook:

1. Sorts by priority (higher first)
2. Checks keyword_match steps
3. Checks regex_match steps
4. Falls back to sequential or default

Example:

userInitiatedTree: [
  {
    stepId: "urgent_help",
    trigger: "keyword_match",
    keywords: ["urgent", "help", "ayuda"],
    priority: 100,
    message: "I'll connect you with support right away!"
  },
  {
    stepId: "general_followup",
    trigger: "sequential",
    priority: 0,
    message: "Still here if you need anything!"
  }
]

---

Template Variables

Available Variables

Admin and business owners can use these variables in WhatsApp reply messages:

Variable	Description	Example
{{place.name}}	Place name	Restaurant La Casa
{{place.shortId}}	Unique place identifier	restaurantabc
{{place.forks.instagram.url}}	Instagram profile URL	https://instagram.com/restaurant
{{place.forks.instagram.username}}	Instagram username	@restaurant
{{place.forks.google.writeReview}}	Google review URL	https://g.page/r/...
{{customer.firstName}}	Customer first name	Jose
{{customer.email}}	Customer email	[email protected]
{{customer.phone}}	Customer phone	+16463345377
{{conversation.shortId}}	Conversation ID	co_a1b2c3
{{currentFork}}	Current fork context	instagram or google

Usage in Pre-filled Messages

PasosModule WhatsApp Step Configuration:

{
  inputType: 'whatsapp',
  phoneNumber: '+51999999999',
  prefilledMessage: 'Hola! Estoy en {{place.name}} ({{place.shortId}}) y me gustaría participar.',
  buttonText: 'Enviar WhatsApp'
}

Processed Message:

"Hola! Estoy en Restaurant La Casa (restaurantabc) y me gustaría participar."

Usage in Reply Messages

Place.forks[fork].forms.whatsapp Architecture

⚠️ MIGRATION STATUS: The Place.forks.forms structure is fully defined in TypeScript types but the database migration has NOT been executed yet. Places in production currently do not have the forms field. The migration script exists at scripts/migrate-places-add-forms.js but needs to be run.

Designed Structure:

Each Place will define fork-specific WhatsApp configuration:

// Place document structure
{
  forks: {
    instagram: {
      forms: {
        whatsapp: {
          formId: "default-whatsapp", // References form_templates.id
          action: "¡Síguenos en Instagram para ganar S./100 esta semana! {{place.forks.instagram.url}}"
        }
      }
    },
    google: {
      forms: {
        whatsapp: {
          formId: "raffle-with-review", // Custom multi-step template
          action: "Déjanos una reseña en Google y gana 10% de descuento! {{place.forks.google.writeReview}}"
        }
      }
    }
  }
}

Instagram Fork Reply Template:

{{forkType.forms.whatsapp.action}}

Thanks for signing up to win S./100 this week at {{place.name}}!

We will message you here if you are this week's winner!

Rendered for Instagram:

¡Síguenos en Instagram para ganar S./100 esta semana! https://instagram.com/restaurant

Thanks for signing up to win S./100 this week at Restaurant La Casa!

We will message you here if you are this week's winner!

Google Fork Reply Template:

{{forkType.forms.whatsapp.action}}

Thanks for signing up at {{place.name}}!

We appreciate your feedback!

Rendered for Google:

Déjanos una reseña en Google y gana 10% de descuento! https://g.page/r/...

Thanks for signing up at Restaurant La Casa!

We appreciate your feedback!

Advanced Example with Multi-Step Conversation:

// Template in form_templates (type: "whatsapp")
{
  id: "favorite_dish",
  type: "whatsapp",
  conversationTree: [
    {
      stepId: "welcome",
      message: {
        text: "Thanks {{customer.firstName}} for signing up at {{place.name}}! What's your favorite dish?"
      },
      expectedResponse: { type: "text", required: true },
      nextStep: { default: "confirm" }
    },
    {
      stepId: "confirm",
      message: {
        text: "Great choice! We'll let the chef know. Good luck in this week's raffle!"
      },
      completeConversation: true
    }
  ]
}

---

API Endpoints

1. Create Conversation

✅ Endpoint: POST /api/conversations/create (FULLY IMPLEMENTED)

Location: src/app/api/conversations/create/route.ts

Purpose: Create or retrieve active conversation when customer taps WhatsApp button

Features:

• Checks for existing active conversations (72-hour window)
• Returns existing conversation if found
• Creates new conversation with TTL expiration if none exists
• Supports preview mode for testing without database writes

Request:

{
  type: 'whatsapp' | 'email' | 'sms' | 'voice'; // Required: Communication channel
  customer: {
    phone: string; // Required: E.164 format (for quick lookup)
    id: string; // Required: Customer ObjectId as string (customers._id)
  };
  place: {
    shortId: string; // Required: Place shortId (places.shortId)
    id?: string; // Optional: Place ObjectId as string (places._id)
  };
  visit: {
    id: string; // Required: Visit ObjectId as string (visits._id)
  };
  fork: 'instagram' | 'google'; // Required: Fork type (denormalized for filtering)
  metadata?: {
    source?: string;
    deviceType?: string;
    userAgent?: string;
  };
}

Response:

{
  success: boolean;
  conversation: {
    shortId: string;
    expiresAt: string; // ISO date
    isNew: boolean; // true if newly created, false if existing
  };
  error?: string;
}

2. Get Conversation

🚧 Endpoint: GET /api/conversations/[shortId] (NOT YET IMPLEMENTED)

Purpose: Retrieve conversation details (admin only)

Response:

{
  success: boolean;
  conversation: Conversation;
}

3. List Conversations

🚧 Endpoint: GET /api/conversations?placeShortId=abc&status=active (NOT YET IMPLEMENTED)

Purpose: List conversations with filters (admin only)

Query Parameters:

• placeShortId - Filter by place
• customerId - Filter by customer
• status - Filter by status (active/expired)
• limit - Pagination limit
• offset - Pagination offset

---

Conversation Lifecycle

States

Created → Active (0-72 hours) → Expired (>72 hours)

Expiration Logic

Automatic Expiration:

• Conversations expire 72 hours after createdAt
• Expired conversations have status: 'expired'
• New messages from same phone create new conversation after expiration

Why 72 Hours?

• WhatsApp business messaging best practice
• Keeps conversations contextually relevant
• Prevents stale conversation threads
• Balances between continuity and freshness

Cleanup Strategy

Option 1: TTL Index (Recommended)

db.conversations.createIndex(
  { expiresAt: 1 },
  { expireAfterSeconds: 0 }
);

Option 2: Status Update

// Cron job or scheduled function
await conversations.updateMany(
  { expiresAt: { $lt: new Date() }, status: 'active' },
  { $set: { status: 'expired' } }
);

---

Integration Points

1. Visit Document

Add reference to conversation:

interface Visit {
  // ... existing fields
  conversationId?: ObjectId; // Reference to conversation
  conversationShortId?: string; // Denormalized for quick lookup
}

2. Customer Document

Track conversation history:

interface Customer {
  // ... existing fields
  conversations?: Array<{
    conversationId: ObjectId;
    shortId: string;
    placeShortId: string;
    createdAt: Date;
    status: 'active' | 'expired';
  }>;
}

---

Admin Panel Features

Conversation Management

View Conversations:

• List all conversations for a place
• Filter by status, date range, customer
• Search by phone number or conversation ID

Conversation Details:

• Full message history (received & sent)
• Customer information
• Associated visit and place data
• Conversation timeline

Analytics:

• Total conversations per place
• Response rate
• Average conversation length
• Popular message times

---

Security & Privacy

Data Protection

1. Phone Number Encryption - Consider encrypting phone numbers at rest
2. Message Retention - Define retention policy (e.g., 90 days)
3. GDPR Compliance - Allow customers to request conversation deletion
4. Access Control - Only admins and business owners can view conversations

Rate Limiting

Prevent abuse:

• Max 10 messages per phone per hour
• Max 3 new conversations per phone per day
• Cooldown period after spam detection

---

Testing Scenarios

Test Case 1: First-Time Customer

1. Customer scans QR code → creates visit
2. Enters email/phone → creates customer
3. Taps WhatsApp button → creates conversation
4. Sends message → receives fork-specific reply
5. Verify conversation document created correctly

Test Case 2: Returning Customer (Within 72 Hours)

1. Customer who messaged 24 hours ago
2. Sends new message
3. Should use existing conversation
4. Verify message added to receivedMessages array

Test Case 3: Returning Customer (After 72 Hours)

1. Customer who messaged 80 hours ago
2. Old conversation expired
3. Sends new message
4. Should create new conversation
5. Verify new conversation document created

Test Case 4: Unknown Phone Number

1. Random phone sends message to Twilio number
2. No conversation found
3. Should send generic reply
4. Optional: Create orphaned conversation for tracking

---

Testing Strategy

Unit Tests

• Variable replacement engine
• Step router logic with conditional routing
• Condition evaluation (keyword_match, regex_match, equals)
• Phone number normalization
• Short ID generation

Integration Tests

• Full conversation flow from start to end
• Timeout handling and reminder steps
• Conditional routing based on user responses
• Template variable injection and replacement
• Message history tracking
• Conversation expiration and renewal

Manual Testing

• Test conversations via ngrok + real WhatsApp
• Verify each branch of conversation tree
• Test edge cases:
  • Unexpected responses
  • Timeouts (24+ hours)
  • Multiple customers messaging simultaneously
  • Expired conversations
  • Unknown phone numbers
  • Invalid message formats

Testing Checklist

• Create conversation via WhatsAppButton
• Send initial message and receive first step
• Test keyword matching (yes/no responses)
• Test conditional routing
• Verify variable replacement
• Check message history in database
• Test conversation after 72 hours (should create new)
• Test user-initiated responses after main tree
• Verify priority-based matching
• Test with multiple places for same customer

---

Example Use Cases

Use Case 1: Instagram Raffle

Initial Message → Ask to Follow → Ask to Share Story → Confirmation

Flow Details:

1. Customer taps WhatsApp button
2. Receives welcome message with instructions
3. Asked to follow Instagram account
4. If yes: Asked to share story
5. If no: Encouraged to follow with link
6. Upon completion: Confirmation message
7. After main tree: User-initiated responses for follow-up questions

Use Case 2: Google Review Request

Initial Message → Ask for Rating (1-5) → If 5: Request Review → If <5: Gather Feedback

Flow Details:

1. Welcome message thanking customer for visit
2. Ask for rating 1-5
3. If 5 stars: Send Google review link
4. If <5 stars: Ask what could be improved
5. Collect feedback and thank customer
6. Update visit status based on completion

Use Case 3: Restaurant Reservation

Greeting → Ask Date → Ask Time → Ask Party Size → Confirm → Send Calendar Invite

Flow Details:

1. Welcome message
2. Ask preferred date
3. Ask preferred time
4. Ask party size
5. Confirm all details
6. Send confirmation with calendar invite link
7. Update internal reservation system

Use Case 4: Product Feedback Survey

Welcome → Ask Product Quality → Ask Service Quality → Ask Recommendation Likelihood → Thank You

Flow Details:

1. Thank customer for purchase
2. Rate product quality (1-10)
3. Rate service quality (1-10)
4. Net Promoter Score question
5. Optional: Open feedback text
6. Thank you and discount code for next visit

---

Future Enhancements

Phase 2: Multi-Step Flows

Enable conversational flows:

{
  flow: {
    step1: {
      trigger: 'initial',
      message: 'Gracias! ¿Prefieres mesa o barra?',
      options: ['mesa', 'barra']
    },
    step2: {
      trigger: 'response',
      message: 'Perfecto! ¿Cuántas personas?',
      validation: 'number'
    }
  }
}

Phase 3: AI-Powered Responses

• Natural language understanding
• Automated FAQ responses
• Smart routing to human agents

Phase 4: WhatsApp Business API Features

• Message templates
• Media messages (images, PDFs)
• Interactive buttons and lists
• Payment integration

---

Implementation Checklist

Database

• Create conversations collection
• Add required indexes (9 indexes including TTL)
• Update Visit schema with conversationId (optional enhancement)
• Update Customer schema with conversation tracking (optional enhancement)
• Set up TTL index for automatic cleanup

Backend

• Create TypeScript types for Conversation
• Implement POST /api/conversations/create endpoint
• Update Twilio webhook to use phone lookup
• Add conversation message tracking (received/sent)
• Implement conversation expiration logic
• Implement two-tree architecture
• Build variable replacement engine
• Add priority-based keyword/regex matching

Frontend

• Update WhatsAppButton component to create conversation
• Handle conversation creation errors gracefully
• Add conversation template variables
• Add forms.whatsapp configuration to Place edit form
• Build visual conversation flow builder UI

Testing

• Unit tests for conversation creation logic
• Integration tests for webhook processing
• Test 72-hour expiration
• Test phone number lookup edge cases
• Load testing for concurrent conversations

Documentation

• Create WHATSAPP_CONVERSATIONS.md
• Update API documentation
• Create admin user guide
• Add inline code comments

---

Migration Strategy

Existing Data

For places that already have WhatsApp integration:

1. No Migration Needed - Old conversations won't have references
2. New Conversations Only - New interactions create conversation documents
3. Graceful Fallback - Webhook handles messages without conversations

Deployment Steps

1. Deploy database schema (create collection + indexes)
2. Deploy backend API endpoints
3. Deploy updated webhook logic
4. Deploy frontend changes
5. Test with staging environment
6. Monitor logs for errors
7. Gradual rollout to production

---

Monitoring & Alerts

Key Metrics

• Conversation Creation Rate - Track new conversations per hour
• Message Processing Time - Webhook response time
• Failed Lookups - Messages without matching conversations
• Expiration Rate - How many conversations expire daily

Alerts

• High Error Rate - >5% webhook errors
• Slow Response - Webhook taking >2 seconds
• Database Issues - Connection failures
• Spam Detection - Unusual message patterns

---

Support & Troubleshooting

Common Issues

Issue: Customer not receiving reply

• Check conversation exists and is active
• Verify place has whatsappReplyMessage configured
• Check Twilio delivery status
• Verify phone number format

Issue: Duplicate conversations

• Check phone number normalization
• Verify TTL index is working
• Check for race conditions in creation

Issue: Wrong reply message

• Verify fork type in conversation document
• Check place configuration
• Verify template variable replacement

Debug Commands

// Find conversation by phone
db.conversations.findOne({ 'customer.phone': "+1234567890" })

// Check active conversations
db.conversations.find({
  status: "active",
  expiresAt: { $gte: new Date() }
})

// View conversation history
db.conversations.aggregate([
  { $match: { shortId: "co_abc123" } },
  { $project: {
    messageCount: {
      $add: [
        { $size: "$receivedMessages" },
        { $size: "$sentMessages" }
      ]
    },
    lastMessage: { $arrayElemAt: ["$receivedMessages", -1] }
  }}
])

---

Advanced: Conversation Templates (form_type: whatsapp)

Overview

Beyond simple auto-replies, the system supports automated conversation flows using templates. These templates define multi-step conversations with conditional logic, similar to web forms but via WhatsApp messaging.

Form Type System Extension

The form_templates collection now supports two types:

• pasos-form: Traditional web-based forms (PasosModule)
• whatsapp: WhatsApp conversation templates (new)

Conversation Template Schema

form_templates Collection (forkast_admin database)

The system stores all templates in a single collection with type-specific schemas:

// Pasos-form template (existing, now with type field)
{
  _id: ObjectId("..."),
  id: "default-pasos",
  type: "pasos-form", // REQUIRED field
  name: "Default (2-Step)",
  description: "Standard form for Instagram places",
  active: true,
  steps: [
    {
      fieldId: "email",
      inputType: "email",
      title: "Tu correo electrónico",
      cta: "Registra tu correo y participa:",
      buttonText: "Participar",
      required: true
    }
  ],
  createdAt: Date,
  updatedAt: Date
}

// WhatsApp conversation template (simple 1-step)
{
  _id: ObjectId("..."),
  id: "default-whatsapp",
  type: "whatsapp", // REQUIRED field
  name: "Default WhatsApp",
  description: "Simple auto-reply message",
  active: true,

  // Main conversation flow (bot-initiated)
  conversationTree: [
    {
      stepId: "welcome",
      stepNumber: 1,
      trigger: "initial",
      message: {
        text: "Thanks for signing up! {{place.name}} will be in touch soon.",
        attachments: []
      },
      expectedResponse: { type: "any", required: false },
      nextStep: { default: "end", conditional: [] },
      autoAdvanceDelay: null
    },
    {
      stepId: "end",
      stepNumber: 99,
      trigger: "auto",
      isTerminal: true,
      message: null
    }
  ],

  // Optional: User-initiated responses (after main tree completes)
  userInitiatedTree: [
    {
      stepId: "help_response",
      trigger: "keyword_match",
      keywords: ["help", "ayuda", "info"],
      priority: 10,
      message: {
        text: "How can I help you? Reply with your question."
      }
    },
    {
      stepId: "sequential_followup",
      trigger: "sequential",
      priority: 0,
      message: {
        text: "Thanks for your message! We'll get back to you soon."
      }
    }
  ],

  createdAt: Date,
  updatedAt: Date
}

// Multi-step WhatsApp template example (favorite_dish)
{
  _id: ObjectId("..."),
  id: "favorite_dish",
  type: "whatsapp",
  name: "Favorite Dish Survey",
  description: "4-step conversation to collect favorite dish",
  active: true,

  conversationTree: [
    {
      stepId: "welcome",
      stepNumber: 1,
      trigger: "initial",
      message: {
        text: "¡Hola {{customer.firstName}}! 👋\n\nGracias por registrarte en {{place.name}}.\n\nPara participar en el sorteo semanal de S./100, necesito que sigas estos pasos:",
        attachments: []
      },
      expectedResponse: {
        type: "any",
        required: false,
        keywords: [],
        validation: null
      },
      nextStep: {
        default: "ask_instagram_follow",
        conditional: []
      },
      autoAdvanceDelay: 2000
    },
    {
      stepId: "ask_instagram_follow",
      stepNumber: 2,
      trigger: "auto",
      message: {
        text: "Paso 1: Síguenos en Instagram\n{{place.forks.instagram.url}}\n\n¿Ya nos sigues? Responde Sí o No"
      },
      expectedResponse: {
        type: "keyword",
        required: true,
        keywords: ["si", "sí", "yes", "no"],
        caseSensitive: false
      },
      nextStep: {
        default: "thank_you",
        conditional: [
          {
            if: { type: "keyword_match", keywords: ["si", "sí", "yes"] },
            then: "ask_story_share"
          },
          {
            if: { type: "keyword_match", keywords: ["no"] },
            then: "encourage_follow"
          }
        ]
      },
      autoAdvanceDelay: null,
      maxWaitTime: 86400000, // 24 hours
      onTimeout: "remind_instagram_follow"
    },
    {
      stepId: "ask_story_share",
      stepNumber: 4,
      trigger: "auto",
      message: {
        text: "¡Perfecto! 🎉\n\nPaso 2: Comparte esta historia en tu Instagram\n\nCuando lo hayas compartido, responde 'Listo'"
      },
      expectedResponse: {
        type: "keyword",
        required: true,
        keywords: ["listo", "ready", "done", "ya"],
        caseSensitive: false
      },
      nextStep: { default: "confirmation" }
    },
    {
      stepId: "confirmation",
      stepNumber: 5,
      trigger: "auto",
      message: {
        text: "¡Excelente! ✅\n\nYa estás participando en el sorteo semanal de S./100.\n\n¡Mucha suerte! 🍀"
      },
      expectedResponse: { type: "any", required: false },
      nextStep: { default: "end" },
      completeConversation: true,
      updateVisitStatus: "completed"
    },
    {
      stepId: "end",
      stepNumber: 99,
      trigger: "auto",
      isTerminal: true,
      message: null
    }
  ],

  // Template variables available
  variables: {
    customer: ["firstName", "lastName", "email", "phone"],
    place: ["name", "shortId", "forks.instagram.url", "forks.instagram.username"],
    raffle: ["nextDrawDate", "nextDrawTime", "prizeAmount"]
  },

  createdAt: Date,
  updatedAt: Date
}

TypeScript Interface

interface WhatsAppTemplate {
  _id?: ObjectId;
  type: "whatsapp";
  id: string;
  name: string;
  description?: string;

  // Main bot-initiated conversation flow (sequential steps)
  conversationTree: ConversationStep[];

  // User-initiated responses (conditional/keyword-based)
  userInitiatedTree?: UserInitiatedStep[];

  active?: boolean;
  createdAt?: Date;
  updatedAt?: Date;
}

interface ConversationStep {
  stepId: string;
  stepNumber: number;
  trigger: "initial" | "auto" | "conditional" | "timeout";

  message: {
    text: string; // Supports {{variable}} replacement
    attachments?: string[];
  };

  expectedResponse: {
    type: "any" | "keyword" | "number" | "text" | "confirmation";
    required: boolean;
    keywords?: string[];
    validation?: string;
    caseSensitive?: boolean;
  };

  nextStep: {
    default: string;
    conditional?: Array<{
      if: {
        type: "keyword_match" | "regex_match" | "equals";
        keywords?: string[];
        pattern?: string;
        value?: any;
      };
      then: string;
    }>;
  };

  autoAdvanceDelay?: number;
  maxWaitTime?: number;
  onTimeout?: string;

  completeConversation?: boolean;
  updateVisitStatus?: string;
  isTerminal?: boolean;
}

interface UserInitiatedStep {
  stepId: string;
  trigger: "sequential" | "keyword_match" | "regex_match" | "sentiment";

  // Matching conditions
  keywords?: string[];
  pattern?: string;
  caseSensitive?: boolean;
  priority?: number; // Higher priority checked first

  message: {
    text: string;
    attachments?: string[];
  };

  nextStep?: string;
  completeConversation?: boolean;
  isTerminal?: boolean;
}

Example Template: Instagram Raffle

{
  type: "whatsapp",
  templateId: "instagram_raffle_flow",
  name: "Instagram Weekly Raffle Flow",
  description: "Automated conversation for Instagram raffle participation",

  config: {
    fork: "instagram",
    language: "es",
    defaultReplyDelay: 1000,
    maxInactivityHours: 24,
  },

  conversationTree: [
    {
      stepId: "welcome",
      stepNumber: 1,
      trigger: "initial",
      message: {
        text: "¡Hola {{customer.firstName}}! 👋\n\nGracias por registrarte en {{place.name}}.\n\nPara participar en el sorteo semanal de S./100, necesito que sigas estos pasos:",
      },
      expectedResponse: { type: "any", required: false },
      nextStep: { default: "ask_instagram_follow" },
      autoAdvanceDelay: 2000,
    },

    {
      stepId: "ask_instagram_follow",
      stepNumber: 2,
      trigger: "auto",
      message: {
        text: "Paso 1: Síguenos en Instagram\n{{place.forks.instagram.url}}\n\n¿Ya nos sigues? Responde Sí o No",
      },
      expectedResponse: {
        type: "keyword",
        required: true,
        keywords: ["si", "sí", "yes", "no"],
        caseSensitive: false,
      },
      nextStep: {
        default: "thank_you",
        conditional: [
          {
            if: { type: "keyword_match", keywords: ["si", "sí", "yes"] },
            then: "ask_story_share"
          },
          {
            if: { type: "keyword_match", keywords: ["no"] },
            then: "encourage_follow"
          }
        ]
      },
      maxWaitTime: 86400000, // 24 hours
      onTimeout: "remind_instagram_follow",
    },

    {
      stepId: "ask_story_share",
      stepNumber: 3,
      trigger: "auto",
      message: {
        text: "¡Perfecto! 🎉\n\nPaso 2: Comparte esta historia en tu Instagram\n\nCuando lo hayas compartido, responde 'Listo'",
      },
      expectedResponse: {
        type: "keyword",
        required: true,
        keywords: ["listo", "ready", "done", "ya"],
      },
      nextStep: { default: "confirmation" },
    },

    {
      stepId: "confirmation",
      stepNumber: 4,
      trigger: "auto",
      message: {
        text: "¡Excelente! ✅\n\nYa estás participando en el sorteo semanal de S./100.\n\n¡Mucha suerte! 🍀",
      },
      expectedResponse: { type: "any", required: false },
      nextStep: { default: "end" },
      completeConversation: true,
      updateVisitStatus: "completed",
    },

    {
      stepId: "end",
      stepNumber: 99,
      trigger: "auto",
      isTerminal: true,
      message: null,
    }
  ],

  variables: {
    customer: ["firstName", "lastName", "email", "phone"],
    place: ["name", "shortId", "forks.instagram.url", "forks.instagram.username"],
    raffle: ["nextDrawDate", "nextDrawTime", "prizeAmount"],
  },
}

Conversation State Tracking

When using templates, the Conversation document includes flow state:

interface Conversation {
  // ... existing fields

  flow?: {
    templateId: string; // Which template is being used
    currentStepId: string; // Current step in conversation tree
    stepHistory: Array<{
      stepId: string;
      enteredAt: Date;
      exitedAt?: Date;
      userResponse?: string;
    }>;
    variables: Record<string, any>; // Computed variables
    isComplete: boolean;
    completedAt?: Date;
  };
}

Variable Replacement

Templates support dynamic variable replacement:

Variable Category	Examples
Customer	{{customer.firstName}}, {{customer.email}}
Place	{{place.name}}, {{place.shortId}}
Place Forks	{{place.forks.instagram.url}}, {{place.forks.google.writeReview}}
Conversation	{{conversation.shortId}}, {{conversation.createdAt}}
Custom	{{raffle.nextDrawDate}}, {{raffle.prizeAmount}}

Backend Flow Logic

Message Processing with Templates:

async function handleIncomingMessage(webhook) {
  const conversation = await findActiveConversation(phone);

  if (conversation?.flow) {
    // Using conversation template
    const template = await loadTemplate(conversation.flow.templateId);
    const currentStep = findStep(template, conversation.flow.currentStepId);

    // Determine next step based on user response
    const nextStepId = determineNextStep(currentStep, webhook.Body, template);
    const nextStep = findStep(template, nextStepId);

    // Replace variables in message
    const message = replaceVariables(nextStep.message.text, {
      conversation,
      customer: await getCustomer(conversation.customer.id),
      place: await getPlace(conversation.place.shortId),
    });

    // Update conversation flow state
    await updateConversationFlow(conversation._id, {
      currentStepId: nextStepId,
      stepHistory: [...conversation.flow.stepHistory, {
        stepId: nextStepId,
        enteredAt: new Date(),
        userResponse: webhook.Body,
      }],
      isComplete: nextStep.completeConversation || false,
    });

    return sendTwiMLMessage(message);

  } else {
    // Fallback to simple auto-reply
    return sendSimpleReply(place.forks[fork].whatsappReplyMessage);
  }
}

✅ Place Configuration (NEW ARCHITECTURE)

ACTUAL IMPLEMENTATION:

Places use unified forms structure across all channels:

interface Place {
  // ... existing fields

  forks: {
    instagram?: {
      // ... existing fields (url, username, etc.)

      // ✅ NEW: Unified forms structure
      forms?: {
        whatsapp?: {
          formId: string; // References form_templates.id
          action: string; // Customizable CTA with {{variables}}
        };
        email?: {
          formId: string;
          action: string;
        };
        pasos?: {
          formId: string;
          action: string; // Used as form CTA button text
        };
      };

      // 🚧 DEPRECATED (still present, to be migrated):
      whatsappReplyMessage?: string; // Old simple auto-reply
    };
    google?: {
      // Same structure
    };
  };
}

Key Design:

• ✅ Fork-agnostic templates use {{forkType.forms.whatsapp.action}}
• ✅ Place-level customization via action text
• 🚧 Migration script ready: scripts/migrate-places-add-forms.js

Admin Panel Features

✅ Template Management (IMPLEMENTED):

• ✅ List all templates at /admin/form-types
• ✅ Filter by type (Pasos Web, WhatsApp, Email)
• ✅ Create new templates with type selector
• ✅ Edit existing templates
• ✅ Delete templates (with confirmation)
• ✅ Duplicate templates
• ✅ Preview templates (basic)

🚧 Template Editor UI (BASIC ONLY):

• ✅ Basic form fields (name, description, active)
• 🚧 Visual conversation flow builder (NOT YET BUILT)
• 🚧 Step-by-step conversation builder (NOT YET BUILT)
• 🚧 Conditional routing configuration (NOT YET BUILT)
• 🚧 Variable insertion helper (NOT YET BUILT)
• 🚧 Visual flow diagram (NOT YET BUILT)
• 🚧 Template testing interface (NOT YET BUILT)

🚧 Place Assignment (NOT YET BUILT):

• 🚧 Add forms.whatsapp configuration to Place edit form
• 🚧 Template selector dropdown
• 🚧 Action text editor with variable helper

API Endpoints

✅ Template CRUD (IMPLEMENTED):

✅ GET    /api/admin/form-templates          - List all templates (all types)
✅ GET    /api/admin/form-templates?id=[id]  - Get single template
✅ POST   /api/admin/form-templates          - Create template
✅ PUT    /api/admin/form-templates          - Update template
✅ DELETE /api/admin/form-templates?id=[id]  - Delete template
🚧 POST   /api/admin/form-templates/test     - Test template (NOT YET BUILT)

Note: The API path is /api/admin/form-templates (not /api/admin/form-types).

Migration Strategy

✅ Phase 1: Database & Type System (COMPLETED)

1. ✅ Added type field to all form_templates
2. ✅ Created conversations collection with 9 indexes
3. ✅ Created default WhatsApp templates (default-whatsapp, favorite_dish)
4. ✅ Extended TypeScript types for multi-channel support
5. ✅ Designed Place.forks.forms structure

🚧 Phase 2: Frontend & Place Migration (IN PROGRESS)

1. ✅ Built template list UI with type filters
2. ✅ Built type selector for creating templates
3. 🚧 Build visual conversation flow builder
4. 🚧 Add forms configuration to Place edit form
5. 🚧 Run Place migration script

✅ Phase 3: Backend Conversation Flow (COMPLETED)

1. ✅ Implemented conversation creation API
2. ✅ Updated webhook to support phone lookup and template loading
3. ✅ Implemented variable replacement engine
4. ✅ Built step router with two-tree architecture
5. ✅ Added message history tracking
6. ✅ Implemented priority-based keyword/regex matching

Phase 4: Advanced Features (FUTURE)

1. Media support (images, videos, PDFs)
2. Interactive buttons and lists
3. AI-powered responses
4. Analytics and A/B testing

Use Cases

Instagram Raffle:

Welcome → Ask to Follow → Ask to Share Story → Confirmation

Google Review Request:

Greeting → Ask Rating (1-5) → If 5: Request Review → If <5: Gather Feedback

Restaurant Reservation:

Greeting → Ask Date → Ask Time → Ask Party Size → Confirm → Calendar Invite

---

Conclusion

The WhatsApp Conversations feature provides a robust, scalable foundation for automated customer messaging. By using phone number lookup instead of code parsing, we deliver a seamless user experience while maintaining comprehensive conversation tracking and history.

Key Benefits:

• ✅ No codes needed - natural messaging flow
• ✅ Full conversation history and audit trail
• ✅ Contextual, personalized replies based on fork type
• ✅ 72-hour conversation windows for relevance
• ✅ Scalable architecture for future enhancements
• ✅ Multi-channel support: Unified system for WhatsApp, Email, Web Forms
• ✅ Reusable templates: Create once, use across multiple places
• ✅ Dynamic variables: Personalization with {{customer.firstName}}, etc.
• ✅ Fork-agnostic: Templates work across Instagram and Google forks

✅ Completed (January 2025):

1. ✅ Created conversations collection with 9 indexes (including TTL)
2. ✅ Extended form_templates to support multi-channel templates (pasos-form, whatsapp, email)
3. ✅ Created default WhatsApp templates (default-whatsapp, favorite_dish)
4. ✅ Built frontend UI for template management (list, filter, create, edit, delete)
5. ✅ Designed Place.forks.forms architecture (TypeScript types complete)
6. ✅ Implemented template CRUD APIs (/api/admin/form-templates)
7. ✅ Full TypeScript type system with discriminated unions
8. ✅ Implemented conversation creation API (/api/conversations/create)
9. ✅ Updated Twilio webhook with full conversation flow support
10. ✅ Implemented two-tree architecture (bot-initiated + user-initiated)
11. ✅ Implemented variable replacement engine with rich context
12. ✅ Built step router with priority-based matching
13. ✅ Added complete message history tracking
14. ✅ Phone lookup and conversation matching by place
15. ✅ 72-hour TTL expiration with automatic renewal

🚧 In Progress / Next Steps:

1. 🚧 Build visual conversation flow builder UI (templates must be created via DB currently)
2. 🚧 Add forms.whatsapp configuration to Place edit form
3. 🚧 Run Place migration script (migrate-places-add-forms.js) to update production data
4. 🚧 Create template testing/simulation interface
5. 🚧 Add variable autocomplete helper for template editors
6. 🚧 Build conversation analytics dashboard
7. 🚧 Monitor and iterate based on usage patterns

---

Files & Scripts Reference

Migration Scripts (/scripts)

Conversation & Template Setup:

• create-conversation-indexes-correct-db.js - Creates 9 indexes on conversations collection (EXECUTED)
• create-whatsapp-templates-correct-db.js - Creates default WhatsApp templates (EXECUTED)
• add-type-to-existing-form-types.js - Adds type field to existing templates (EXECUTED)
• fix-form-type-types.js - Fixes specific template types (EXECUTED)
• migrate-whatsapp-two-tree-architecture.js - Migrates to two-tree architecture (PREPARED)
• migrate-places-add-forms.js - Migrates Place schema to new forms structure (NOT YET RUN)
• run-multi-channel-migrations.js - Comprehensive migration runner

TypeScript Types (/src/lib/types/)

Core Type Files:

• formTemplates.ts - Form template types, conversation step types, user-initiated step types
  • FormTemplateType, ForkType, ChannelType
  • FormTemplateDocument, WhatsAppTemplate, PasosFormTemplate
  • ConversationStep, UserInitiatedStep
  • ChannelFormConfig, ForkForms
  • Type guards: isWhatsAppTemplate(), isPasosFormTemplate()
• conversation.ts - Conversation document types, message types
  • Conversation, ConversationStatus, ConversationType
  • ReceivedMessage, SentMessage
  • CreateConversationRequest, CreateConversationResponse
  • VariableReplacementContext
• database.ts - Place schema types
  • PlaceUpdateFields, GoogleForkData, InstagramForkData
• twilio.ts - Twilio webhook types
  • TwilioWhatsAppWebhook

Frontend Components (/src/app/admin/form-types/)

Admin UI:

• page.tsx - Form Templates list page with type filters
• components/FormTypeList.tsx - Template list with badges and action buttons
• components/FormTypeFormContainer.tsx - Type selector UI (card-based)
• components/FormTypeForm.tsx - Template editor (basic fields only)

Backend APIs (/src/app/api/)

Core Endpoints:

• admin/form-templates/route.ts - Template CRUD API (GET, POST, PUT, DELETE)
• conversations/create/route.ts - Conversation creation endpoint
• twilio/whatsapp/route.ts - Twilio webhook with two-tree architecture

Utilities:

• lib/utils/variableReplacement.ts - Variable replacement engine
• lib/utils/twilioHelpers.ts - Twilio helper functions
• lib/utils/customerHelpers.ts - Customer utilities (shortId generation)
• lib/settings/serverUtils.ts - Server-side settings utilities

Frontend Components (/src/components/)

Customer-Facing:

• place/InputTypes/WhatsAppButton.tsx - WhatsApp button in PasosModule
• place/PasosModule.tsx - Multi-step form component

Database Collections

forkast_admin database:

• form_templates - All templates (pasos-form, whatsapp, email)
  • Currently: 10 templates (6 pasos-form, 3 whatsapp, 1 email)
  • Indexes: _id, id (with unique constraint)

forkast database:

• conversations - WhatsApp conversation state
  • 9 indexes including TTL for auto-expiration
  • Currently: Ready for production use
• customers - Customer information
• visits - Visit tracking
• places - Place/business information

---

Related Documentation:

• TypeScript types in /src/lib/types/formTemplates.ts
• Migration scripts in /scripts directory
• Frontend components in /src/app/admin/form-types/
• Backend APIs in /src/app/api/

Last Updated: October 2025