At 2023-01-27 9:45 UTC, a failover event occurred due to a transient networking issue for a database cluster related to location storage. This event lasted for around a minute. As a result, several processes related to persisting location data entered a state which prevented them from persisting locations. Due to a lack of effective monitoring, we were not immediately aware of this issue.

The root issue was resolved at 2023-01-30 22:11 UTC, soon after we became aware of it. However, as a result of this issue, complete location information for about 20% of tasks during this time period was not collected. We subsequently backfilled derived location information when appropriate, providing distance information for about 75% of these tasks by 2023-01-31 19:07 UTC.

We understand how crucial location information is and apologize for any impact this incident may have caused.

We have since improved our monitoring so a similar incident would have immediately paged an on-call engineer.