To address the latency issue in the application, we'll need to first isolate the potential bottleneck component and investigate each component separately.

Potential Bottlenecks :

Frontend :-

• Network Analysis : Check Network logs for the excessive size or latency in the browser.

• Rendering Time : Some Complex UI component may also contribute to bigger rendering time.

• Javascript Execution : Identify JS heavy Scripts which may cause delays and optimize.

• Third Party Scripts/lib : Invesitgate the impact of third party scripts and libraries on page load time.

Backend :-

• Database Queries: Analyze slow database queries, tables.

• API Response Time: Measure time taken by each API endpoint, identify bottlenecks.

• Server Resource Utilization: Monitor CPU, memory, and disk I/O to ensure efficient resource usage.

• Service Dependencies: Investigate external service calls and their latency.

Infrastructure :-

• Server Performance: Check server CPU, memory, and disk usage under load.

• Load Balancing: Ensure proper load balancing configuration to distribute traffic evenly.

• Scaling Configuration: Evaluate if the infrastructure can handle the current load efficiently.

Collect Metrics, Events, Logs for Analysis :

Frontend :-

• Browser Developer Tools: Utilize browser developer tools to profile performance, analyze rendering, and network activity.

• Lighthouse: Use Lighthouse for auditing web page performance and getting actionable suggestions.

• Real User Monitoring (RUM): Implement RUM tools to collect performance data from actual users.

• Synthetic Monitoring: Set up synthetic tests to simulate user interactions and measure performance metrics.

Backend :-

• Profiling Tools: Deploy profiling tools like New Relic, AppDynamics, or Datadog to identify performance bottlenecks in code.

• Load Testing: Conduct load tests with tools like JMeter or Gatling to simulate heavy traffic and measure response times.

• Database Profiling: Use database profiling tools to analyze query execution plans bloated tables.

• A/B Testing: Test different code implementations or configurations in production to identify performance improvements.

Infrastructure :-

• Monitoring Tools: Use monitoring tools like Prometheus, Grafana, or Datadog to monitor server metrics in real-time.

• Latency Tracing: Implement distributed tracing or APM using tools like Jaeger or Zipkin to track requests across different services.

• Capacity Planning: Estimate future traffic growth and scale infrastructure accordingly.

• Private Endpoints: Check for EFS, DB Private endpoints connectivity.

Apply Fixes:

• Metrics: Collect performance metrics from frontend, backend, and infrastructure components. APM generally gives a fair overview of API calls taking more times.

• Identify Bottlenecks: Analyze the data to pinpoint areas with high latency or resource utilization.

• Prioritize Fixes: Prioritize fixes based on the impact on latency and ease of implementation.

• Implement Optimizations: Make necessary optimizations in code, configurations, or infrastructure.

Test Changes:

• Deploy changes to a staging environment and conduct performance tests to validate improvements.

• Monitor in Production: Monitor performance in the production environment and verify if the latency has decreased.

• Iterate: Continuously monitor and optimize performance, iterating on the process to maintain low latency.

By following this approach, we can systematically identify and address the latency issues across the application while ensuring adherence to the p99 latency commitment.