Skip to content

Instantly share code, notes, and snippets.

@hoangsonww

hoangsonww/URL-Shortening-Service-Design.md

Created March 29, 2025 21:00
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save hoangsonww/e1e4a356316c3059cc3d8488d5b00e8a to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save hoangsonww/e1e4a356316c3059cc3d8488d5b00e8a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
URL-Shortening-Service-Design.md

URL Shortening Service

A URL shortening service transforms long URLs into concise, easy-to-share links. Think of services like TinyURL or Bitly. This project provides a RESTful API and a simple web interface for creating, managing, and tracking short URLs.

Table of Contents

  1. Introduction
  2. Features
  3. Architecture & Design
  4. Tech Stack
  5. Installation & Setup
  6. Database Schema
  7. API Endpoints
  8. Frontend (Optional)
  9. Testing
  10. Deployment
  11. Security Considerations
  12. Scalability & Future Enhancements
  13. Troubleshooting & FAQ
  14. Contributing
  15. License

Introduction

The URL Shortening Service is designed to help users create compact, memorable URLs that redirect to longer original addresses. This service is useful for social media sharing, marketing campaigns, and analytics. It not only reduces URL length but also collects click data, enabling users to monitor usage and engagement.

Why a URL Shortener?

  • Simplicity: Long URLs can be unwieldy and error-prone when sharing.
  • Analytics: Track how many times a link is clicked, from where, and by what device.
  • Customization: Allow users to customize their shortened URL (when not auto-generated).
  • Branding: Custom aliases can improve brand visibility and recall.

Features

  • Short URL Generation: Automatically generate a unique short code from a long URL.
  • Custom Aliases: Allow users to choose a custom alias instead of a random string.
  • Redirection: Seamless HTTP redirection from the short URL to the original URL.
  • Analytics: Log every click with timestamp, IP, user-agent, and (optionally) geolocation.
  • Expiration: Option to set expiration dates for short URLs.
  • Rate Limiting: Protect against abuse by limiting API calls per IP or user.
  • API & Web Interface: Accessible via a RESTful API and optionally a frontend dashboard.
  • Scalable Design: Ready to be deployed in a distributed environment with caching, load balancing, and database replication.

Architecture & Design

System Overview

The URL Shortening Service follows a microservices-inspired architecture, decoupling the core components for easier maintenance and scalability. The main components are:

  • API Gateway / Application Server: Exposes the REST API endpoints.
  • URL Mapping Service: Contains the business logic for encoding/decoding URLs.
  • Analytics Service: Captures and stores click events.
  • Database Layer: Stores URL mappings, user data (if applicable), and analytics.
  • Caching Layer: Uses a fast in-memory store (e.g., Redis) to quickly resolve frequently accessed short URLs.
  • Frontend: A simple web interface for users to interact with the service (optional).

Component Breakdown

  1. API Gateway:

    • Handles incoming requests.
    • Validates API inputs.
    • Implements rate limiting and authentication (if needed).

  2. URL Encoder/Decoder Service:

    • Generates a unique short code.
    • Checks for custom alias conflicts.
    • Ensures bi-directional mapping between the short code and long URL.
    • Uses base conversion (e.g., Base62) for encoding numeric IDs if needed.

  3. Analytics Logger:

    • Records every redirection event.
    • Captures metadata (timestamp, IP address, user-agent, etc.).
    • Provides endpoints for retrieving usage statistics.

  4. Database:

    • URL Mapping Table: Stores mappings, creation dates, expiration dates, and custom alias flags.
    • Analytics Table: Stores click logs for reporting and analysis.

  5. Caching:

    • Uses Redis or similar to cache frequently accessed URL mappings.
    • Reduces latency for redirection requests.

Data Flow

  1. Shortening Request:

    • User submits a long URL (with optional custom alias and expiration date).
    • The API validates the input, checks for alias availability, and generates a short code.
    • The mapping is saved in the database, and optionally cached.

  2. Redirection Request:

    • User accesses the short URL.
    • The system first checks the cache. If found, it redirects immediately.
    • If not in cache, it queries the database, updates the cache, and logs the click event.
    • The user is then redirected (HTTP 302) to the long URL.

  3. Analytics Request:

    • The client requests analytics data for a given short URL.
    • The API aggregates data from the Analytics Table and returns it in a structured format.

Tech Stack

  • Backend Framework:

    • Primary options: Spring Boot (Java), Express.js (Node.js), or Django (Python)
    • REST API design with JSON communication.

  • Database:

    • PostgreSQL / MySQL for relational data
    • Optional: MongoDB for flexible schema analytics storage.

  • Caching:

    • Redis for in-memory caching of URL mappings.

  • Frontend (optional):

    • React or Vue.js for a dynamic user interface.

  • Deployment:

    • Docker containers, Kubernetes orchestration
    • Cloud providers: AWS, GCP, or Azure.

  • Monitoring & Logging:

    • ELK Stack (Elasticsearch, Logstash, Kibana) or Prometheus + Grafana for monitoring.

Installation & Setup

Prerequisites

  • Development Environment:

    • JDK 11+ (if using Java) or Node.js (if using Express.js) or Python 3.x (if using Django)
    • Git for version control
    • Docker (for containerization)

  • Database Setup:

    • PostgreSQL/MySQL server installed locally or accessible remotely
    • Redis server installed locally or via cloud provider

Local Setup

  1. Clone the Repository

    git clone https://github.com/your-username/url-shortener.git
cd url-shortener

  2. Install Dependencies

    Depending on your chosen backend:

    • Java (Maven): 
      mvn clean install
    • Node.js: 
      npm install
    • Python: 
      pip install -r requirements.txt

  3. Environment Variables

    Create a .env file in the project root. Below is an example for a Java Spring Boot project:

    # Database settings
DATABASE_URL=jdbc:postgresql://localhost:5432/url_shortener
DATABASE_USER=your_db_username
DATABASE_PASSWORD=your_db_password

# Redis settings
REDIS_HOST=localhost
REDIS_PORT=6379

# Application settings
BASE_URL=http://short.ly/
SERVER_PORT=8080

# Optional: Security keys, API rate limits, etc.
API_KEY=your_api_key_here
RATE_LIMIT=100

  4. Database Migrations

    Use your framework’s migration tool (Flyway for Java, Sequelize for Node.js, Django migrations for Python) to initialize the schema:

    # Java example using Flyway
mvn flyway:migrate

  5. Running the Application

    • Java: 
      mvn spring-boot:run
    • Node.js: 
      npm start
    • Python: 
      python manage.py runserver

Database Schema

URL Mapping Table

Column Type Constraints Description
id BIGSERIAL Primary Key Unique identifier
code VARCHAR(10) Unique, not null Generated short code or custom alias
long_url TEXT Not null Original URL provided by the user
created_at TIMESTAMP Default current_timestamp Timestamp when the mapping was created
expire_at TIMESTAMP Nullable Expiration date for the short URL (if provided)
is_custom BOOLEAN Default false Indicates if the alias was custom provided

Analytics Table

Column Type Constraints Description
id BIGSERIAL Primary Key Unique identifier
url_id BIGINT Foreign key (URL Mapping) Reference to the corresponding short URL
click_time TIMESTAMP Default current_timestamp Timestamp of the click event
ip_address VARCHAR(45) Not null IP address of the user (IPv4/IPv6)
user_agent TEXT Nullable Browser or client details
location VARCHAR(255) Nullable Geolocation data (if available)

API Endpoints

1. Create Short URL

  • Endpoint: POST /api/shorten

  • Description: Accepts a long URL (plus optional custom alias and expiration) and returns the shortened URL.

  • Request Headers:

    • Content-Type: application/json
    • Authorization: Bearer <token> (if secured)

  • Request Body:

    {
  "longUrl": "https://www.example.com/some/very/long/url",
  "customAlias": "myCustom",   // optional
  "expireAt": "2025-12-31T23:59:59Z" // optional, ISO8601 format
}

  • Response:

    {
  "shortUrl": "http://short.ly/abc123",
  "longUrl": "https://www.example.com/some/very/long/url",
  "expireAt": "2025-12-31T23:59:59Z"
}

  • Error Responses:

    • 400 Bad Request: Invalid URL format or alias conflict.
    • 429 Too Many Requests: Rate limit exceeded.

2. Redirect to Long URL

  • Endpoint: GET /:code
  • Description: Redirects the user to the original URL.
  • Example:
    • Request: GET http://short.ly/abc123
    • Response: HTTP 302 redirect to https://www.example.com/some/very/long/url
  • Behavior:
    • If the code is not found or expired, return a 404 page or custom error message.
    • Log the click event (IP, user-agent, timestamp).

3. Retrieve URL Analytics

  • Endpoint: GET /api/analytics/:code

  • Description: Retrieves detailed analytics for the short URL.

  • Response:

    {
  "code": "abc123",
  "longUrl": "https://www.example.com/some/very/long/url",
  "clickCount": 150,
  "analytics": [
    {
      "clickTime": "2025-03-30T12:00:00Z",
      "ipAddress": "192.168.1.1",
      "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)...",
      "location": "New York, USA"
    },
    {
      "clickTime": "2025-03-30T12:05:00Z",
      "ipAddress": "192.168.1.2",
      "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)...",
      "location": "San Francisco, USA"
    }
  ]
}

4. Check Custom Alias Availability

  • Endpoint: GET /api/check-alias/:alias

  • Description: Checks if a custom alias is available.

  • Response:

    {
  "alias": "myCustom",
  "available": true
}

Frontend (Optional)

If you choose to build a web-based dashboard:

  • Framework: React or Vue.js
  • Pages/Components:
    • Home: Form to submit a long URL with options for custom alias and expiration.
    • Dashboard: Displays a list of shortened URLs, their analytics, and options to edit or delete.
    • Analytics View: Detailed charts and logs for each URL.

Testing

Unit Testing

  • Backend:
    • Java: Use JUnit and Mockito for service layer tests.
    • Node.js: Use Mocha/Chai for endpoint and business logic tests.
    • Python: Use PyTest and unittest.mock.
  • Areas to Cover:
    • URL encoding/decoding logic.
    • Custom alias validation.
    • Expiration logic.

Integration Testing

  • Use tools such as Postman, Insomnia, or automated test scripts (e.g., using Selenium for frontend) to verify:
    • End-to-end URL shortening and redirection.
    • Correct logging of analytics data.
    • Handling of invalid inputs, expired URLs, and rate limits.

Load Testing

  • Simulate high traffic to test cache performance and database scalability using tools like Apache JMeter or Locust.

Deployment

Docker Setup

Create a Dockerfile to containerize your application:

# Use an official base image
FROM openjdk:11-jre-slim

# Set working directory in the container
WORKDIR /app

# Copy the built application (jar file) into the container
COPY target/url-shortener.jar app.jar

# Expose the port the app runs on
EXPOSE 8080

# Run the jar file
CMD ["java", "-jar", "app.jar"]

Build and run:

docker build -t url-shortener .
docker run -d -p 8080:8080 --env-file .env url-shortener

Kubernetes Deployment

Prepare manifests for Kubernetes:

  • deployment.yaml: Define pods, replicas, and resource requests/limits.
  • service.yaml: Expose the deployment with a ClusterIP or LoadBalancer service.

Example snippet for deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: url-shortener-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: url-shortener
  template:
    metadata:
      labels:
        app: url-shortener
    spec:
      containers:
      - name: url-shortener
        image: your-docker-repo/url-shortener:latest
        ports:
        - containerPort: 8080
        envFrom:
        - secretRef:
            name: url-shortener-secrets

Cloud Deployment Considerations

  • Managed Kubernetes: Consider using AWS EKS, Google GKE, or Azure AKS.
  • CI/CD Pipelines: Automate build, test, and deployment using Jenkins, GitHub Actions, or GitLab CI.
  • Monitoring: Integrate Prometheus/Grafana and centralized logging (e.g., ELK Stack).

Security Considerations

  • Input Sanitization: Validate and sanitize all inputs to prevent XSS, SQL injection, and other attacks.
  • HTTPS: Use SSL/TLS to secure communications.
  • Authentication/Authorization: Secure APIs using API keys, OAuth, or JWT.
  • Rate Limiting: Implement to prevent abuse (e.g., DoS attacks).
  • Logging & Auditing: Log access and errors securely. Use centralized logging to monitor suspicious activities.
  • Database Security: Use parameterized queries or an ORM to prevent SQL injection. Secure database credentials and connections.
  • Data Privacy: Ensure any user data or analytics are stored and processed according to privacy regulations (e.g., GDPR).

Scalability & Future Enhancements

  • Distributed Database: Use sharding or replication to scale the URL mapping database.
  • Advanced Analytics: Integrate third-party analytics tools or build custom dashboards for real-time insights.
  • Custom Domain Support: Allow users to link their own domains for shortening.
  • URL Preview: Provide a preview page showing a summary of the destination URL before redirection.
  • Multi-region Deployment: Reduce latency by deploying in multiple geographic regions.
  • User Accounts: Allow users to register, manage their URLs, and view personalized analytics.
  • A/B Testing: Experiment with different algorithms or UIs to improve user experience.

Troubleshooting & FAQ

Common Issues

  • Short URL Not Redirecting:

    • Ensure that the code exists and hasn’t expired.
    • Check if the cache/database is properly populated.

  • Custom Alias Conflict:

    • Verify that the alias isn’t already taken by another URL.
    • Use the provided API endpoint to check alias availability.

  • Rate Limit Exceeded:

    • Review API usage policies and ensure your application handles HTTP 429 responses gracefully.

FAQ

Q: How are collisions in the short code prevented?
A: The service uses random generation (or sequential ID conversion with Base62) and checks the database/cache for existing codes. In case of a collision, a new code is generated.

Q: Can I change the expiration of a short URL?
A: Yes, the API allows setting an expiration time. Future enhancements may include editing an existing URL’s expiration via an authenticated endpoint.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment