High overall system latency

Incident Report for Onfleet

Postmortem

We have just concluded a major effort to diagnose and address the periodic performance decreases that have recently affected the Onfleet system. At this time, we believe that bottlenecks have been identified and that the root causes have been addressed, and we are observing improvements on performance-related metrics across the board.

This was a major effort that included the following:

  • Database performance improvements, by means of node redistribution, query optimization, and adjustment of indices
  • Refactoring of the primary dashboard-oriented data loading endpoints
  • Improvements to our webhook message processing system
  • Development of a modern incremental fetching scheme to decrease dashboard lock time
  • Significant adjustments to our internal server architecture provisioning

In aggregate, these changes are not only allowing the system to cope with higher load levels, but are also delivering an improved performance baseline.

Our team will continue efforts to improve our instrumentation in order to proactively identify choke points in the future. Additionally, the team will maintain a focus on performance improvements with a particular focus on customers with high task usage.

Posted Jul 17, 2025 - 12:12 PDT

Resolved

System responsiveness is returning to normal.

Our engineering teams will continue to work on root causes to avoid latency spikes in the future.
Posted Jul 14, 2025 - 15:06 PDT

Monitoring

We are seeing system latency normalizing after putting some mitigations in place. Our infrastructure and database teams are continuing to monitor this situation.
Posted Jul 14, 2025 - 14:07 PDT

Investigating

We are experiencing an emergent issue affecting overall system performance. All system functions are available with decreased performance.

Our team is actively working to mitigate this issue.
Posted Jul 14, 2025 - 12:00 PDT
This incident affected: Dashboard, API, Maps, iOS, Android, Locations streaming, Locations storage, Analytics, Search, ETA, Incoming Voice Proxy, Incoming Text Proxy, Outgoing Voice Proxy, Outgoing Text Gateway and Proxy, and Telephony Charges Monitoring.