Requires current evidence from google.com to answer a time-sensitive status request.
web_fetch completeddispatch toolu_01FYfmMV6r2KJBtHG3c86jk4
Input
{
"brief": "Incident history JSON for past 14 days",
"max_age_seconds": 3600,
"url": "https://www.google.com/appsstatus/dashboard/incidents.json"
}
Tool Progress E0009
(no result)
Tool Result E0010
{"content":"[{\"id\":\"fJ4691Pe2g9KsTbP8g5F\",\"number\":\"11425782758836097339\",\"begin\":\"2026-09-11T13:00:00+00:00\",\"created\":\"2026-09-11T20:45:19+00:00\",\"end\":\"2026-09-11T20:22:00+00:00\",\"modified\":\"2026-09-11T20:45:42+00:00\",\"external_desc\":\"**Summary:**\\nMitigated intermittent elevated error rates with Gemini 3.1 Pro models on Gemini Web and Gemini App.\\n**Description:**\\nBetween 6:00 PDT and 13:22 PDT on 11 September 2026, a subset of Gemini users may have experienced elevated error rates with Gemini 3.1 Pro models on Gemini Web and Gemini App.\\nFrom preliminary analysis, the issue was triggered by a recent rollout. Our engineers identified the cause and mitigated it with a code change.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms:**\\nCustomers may have encountered intermittent errors when using Gemini 3.1 Pro models on Gemini Web and Gemini App.\",\"updates\":[{\"created\":\"2026-09-11T20:45:19+00:00\",\"modified\":\"2026-09-11T20:45:42+00:00\",\"when\":\"2026-09-11T20:45:19+00:00\",\"text\":\"**Summary:**\\nMitigated intermittent elevated error rates with Gemini 3.1 Pro models on Gemini Web and Gemini App.\\n**Description:**\\nBetween 6:00 PDT and 13:22 PDT on 11 September 2026, a subset of Gemini users may have experienced elevated error rates with Gemini 3.1 Pro models on Gemini Web and Gemini App.\\nFrom preliminary analysis, the issue was triggered by a recent rollout. Our engineers identified the cause and mitigated it with a code change.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms:**\\nCustomers may have encountered intermittent errors when using Gemini 3.1 Pro models on Gemini Web and Gemini App.\",\"status\":\"AVAILABLE\"}],\"most_recent_update\":{\"created\":\"2026-09-11T20:45:19+00:00\",\"modified\":\"2026-09-11T20:45:42+00:00\",\"when\":\"2026-09-11T20:45:19+00:00\",\"text\":\"**Summary:**\\nMitigated intermittent elevated error rates with Gemini 3.1 Pro models on Gemini Web and Gemini App.\\n**Description:**\\nBetween 6:00 PDT and 13:22 PDT on 11 September 2026, a subset of Gemini users may have experienced elevated error rates with Gemini 3.1 Pro models on Gemini Web and Gemini App.\\nFrom preliminary analysis, the issue was triggered by a recent rollout. Our engineers identified the cause and mitigated it with a code change.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms:**\\nCustomers may have encountered intermittent errors when using Gemini 3.1 Pro models on Gemini Web and Gemini App.\",\"status\":\"AVAILABLE\"},\"status_impact\":\"SERVICE_INFORMATION\",\"severity\":\"low\",\"service_key\":\"npdyhgECDJ6tB66MxXyo\",\"service_name\":\"Gemini\",\"affected_products\":[{\"title\":\"Gemini\",\"id\":\"npdyhgECDJ6tB66MxXyo\"}],\"uri\":\"incidents/fJ4691Pe2g9KsTbP8g5F\"},{\"id\":\"y2ufhCB886p7rveYNnko\",\"number\":\"1967299596760361422\",\"begin\":\"2026-08-28T17:50:00+00:00\",\"created\":\"2026-08-28T19:25:37+00:00\",\"end\":\"2026-08-28T20:34:00+00:00\",\"modified\":\"2026-09-04T03:43:27+00:00\",\"external_desc\":\"**Summary:**\\nGoogle Chat users are experiencing an issue where they receive notifications for new messages, but the actual messages fail to appear in their chat application unless the user refreshes their view.\\n**Description:**\\nWe are experiencing an intermittent issue with Google Chat Friday, 2026-08-28 10:25 PDT.\\nOur engineering team has investigated the issue and are currently working on mitigating the issue.\\nWe will provide more information by Friday, 2026-08-28 14:15 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Customer Symptoms:**\\nUsers are experiencing an issue where they receive notifications for new messages, but the actual messages fail to appear in their chat application unless the user refreshes their view\\n**Workaround:**\\nUsers can refresh to see new messages\",\"updates\":[{\"created\":\"2026-09-04T03:43:07+00:00\",\"modified\":\"2026-09-04T03:43:27+00:00\",\"when\":\"2026-09-04T03:43:07+00:00\",\"text\":\"## Incident Report\\n## Summary\\nOn Friday, August 28, 2026, between 10:50 US/Pacific and 13:34 US/Pacific for a duration of 2 hours and 44 minutes, Google Chat users experienced delays in receiving real-time incoming messages within active conversation streams across web, desktop, and mobile applications. While users continued to receive notifications for new messages, the message contents did not automatically render in active conversation streams until the user manually refreshed the application or sent a reply message.\\nTo our valued Google Workspace customers, we are deeply sorry for the service interruption you experienced. We are taking immediate and decisive action to address the issue and strengthen our systems against future incidents.\\n## Root Cause\\nThe incident was triggered during a global database schema rollout. An incompatible database schema change caused database writes required for message delivery to start failing.\\nSpecifically, a mechanism for schema synchronization—designed to ensure high consistency between primary storage and live message delivery—encountered a rare race condition bug. This caused backend message dispatchers to attempt writes against an unready schema state, leading to write failures.\\nBecause live message delivery had an active dependency on these writes completing successfully before pushing events to user screens, the failure caused incoming messages to experience synchronization delays. Separate push notification services and core message storage operated independently and were unaffected, which is why users still received alerts for new messages while active conversation streams experienced delays.\\n## Remediation and Prevention\\nThe issue was detected through automated backend alerting on dispatch error rates, alongside corresponding customer reports. Google Engineering initiated parallel workstreams to restore service.\\n* At 12:44 US/Pacific, Google Engineering directly mitigated the source of the issue by removing the dependency that live message delivery had on the failing database write. This allowed new incoming messages to bypass the failed write and immediately resume normal real-time delivery to user screens.\\n* At 13:00 US/Pacific, Engineering halted and reverted the database schema change to its previous verified stable version, restoring full database compatibility.\\n* By 13:34 US/Pacific, as queued messages were processed, real-time message synchronization delays returned to 0% across all customer domains. Engineering verified that full service health was restored with zero message loss.\\nTo prevent this from happening again, we are executing the following preventative actions:\\n* **Hardened internal detection:** Strengthen automated validation and verification to cover this specific failure scenario across pre-production and internal rollout stages. This aims to catch and mitigate schema incompatibilities and synchronization edge cases internally before any customer traffic can be impacted in the future.\\n* **Resilience Against Race Conditions:** Implement Architectural improvements to make the schema synchronization and message delivery architecture resilient to race conditions. This aims to ensure that background schema and config synchronization inconsistencies cannot disrupt real-time message delivery.\\n \\n## Detailed Description of Impact\\nThe incident impact window occurred between 10:50 US/Pacific and 13:34 US/Pacific on Friday, 28 August 2026\\\\. This issue applied globally to all Google Chat environments.\\n**Google Chat:**\\n* **Delayed message rendering:** New messages did not appear in active conversation streams automatically. Users were required to manually refresh their client or send a new message/react to force the stream to synchronize.\\n* **Stalled read states and inbox sorting:** Background actions, such as updating unread badges and moving active spaces to the top of the inbox, experienced heavy internal error rates and were significantly delayed for users.\\n* **Search indexing delays:** New messages sent during the outage window stalled in the indexing pipeline, experiencing delays in becoming searchable.\\n**Scope Exclusions:**\\n* **No Data Loss:** At no point was customer data lost or compromised. All messages transmitted during the incident window were securely captured in primary storage and successfully delivered to conversation streams once the backlog cleared.\",\"status\":\"AVAILABLE\"},{\"created\":\"2026-08-28T20:55:51+00:00\",\"modified\":\"2026-09-04T03:43:07+00:00\",\"when\":\"2026-08-28T20:55:51+00:00\",\"text\":\"**Summary**\\nGoogle Chat users experienced an issue where they received notifications for new messages, but the actual messages failed to appear in their chat application unless the user refreshed their view.\\n**Description**\\nThe issue with Google Chat has been resolved for all affected users as of Friday, 2026-08-28 13:45 PDT.\\nFrom preliminary analysis the issue was triggered due to a recent change which was rolled back to mitigate the impact.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Customer Symptoms**\\nUsers were experiencing an issue where they received notifications for new messages, but the actual messages failed to appear in their chat application unless the user refreshed their view.\\n**Workaround**\\nNone as the issue is resolved.\",\"status\":\"AVAILABLE\"},{\"created\":\"2026-08-28T19:25:37+00:00\",\"modified\":\"2026-08-28T20:55:51+00:00\",\"when\":\"2026-08-28T19:25:37+00:00\",\"text\":\"**Summary:**\\nGoogle Chat users are experiencing an issue where they receive notifications for new messages, but the actual messages fail to appear in their chat application unless the user refreshes their view.\\n**Description:**\\nWe are experiencing an intermittent issue with Google Chat Friday, 2026-08-28 10:25 PDT.\\nOur engineering team has investigated the issue and are currently working on mitigating the issue.\\nWe will provide more information by Friday, 2026-08-28 14:15 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Customer Symptoms:**\\nUsers are experiencing an issue where they receive notifications for new messages, but the actual messages fail to appear in their chat application unless the user refreshes their view\\n**Workaround:**\\nUsers can refresh to see new messages\",\"status\":\"SERVICE_INFORMATION\"}],\"most_recent_update\":{\"created\":\"2026-09-04T03:43:07+00:00\",\"modified\":\"2026-09-04T03:43:27+00:00\",\"when\":\"2026-09-04T03:43:07+00:00\",\"text\":\"## Incident Report\\n## Summary\\nOn Friday, August 28, 2026, between 10:50 US/Pacific and 13:34 US/Pacific for a duration of 2 hours and 44 minutes, Google Chat users experienced delays in receiving real-time incoming messages within active conversation streams across web, desktop, and mobile applications. While users continued to receive notifications for new messages, the message contents did not automatically render in active conversation streams until the user manually refreshed the application or sent a reply message.\\nTo our valued Google Workspace customers, we are deeply sorry for the service interruption you experienced. We are taking immediate and decisive action to address the issue and strengthen our systems against future incidents.\\n## Root Cause\\nThe incident was triggered during a global database schema rollout. An incompatible database schema change caused database writes required for message delivery to start failing.\\nSpecifically, a mechanism for schema synchronization—designed to ensure high consistency between primary storage and live message delivery—encountered a rare race condition bug. This caused backend message dispatchers to attempt writes against an unready schema state, leading to write failures.\\nBecause live message delivery had an active dependency on these writes completing successfully before pushing events to user screens, the failure caused incoming messages to experience synchronization delays. Separate push notification services and core message storage operated independently and were unaffected, which is why users still received alerts for new messages while active conversation streams experienced delays.\\n## Remediation and Prevention\\nThe issue was detected through automated backend alerting on dispatch error rates, alongside corresponding customer reports. Google Engineering initiated parallel workstreams to restore service.\\n* At 12:44 US/Pacific, Google Engineering directly mitigated the source of the issue by removing the dependency that live message delivery had on the failing database write. This allowed new incoming messages to bypass the failed write and immediately resume normal real-time delivery to user screens.\\n* At 13:00 US/Pacific, Engineering halted and reverted the database schema change to its previous verified stable version, restoring full database compatibility.\\n* By 13:34 US/Pacific, as queued messages were processed, real-time message synchronization delays returned to 0% across all customer domains. Engineering verified that full service health was restored with zero message loss.\\nTo prevent this from happening again, we are executing the following preventative actions:\\n* **Hardened internal detection:** Strengthen automated validation and verification to cover this specific failure scenario across pre-production and internal rollout stages. This aims to catch and mitigate schema incompatibilities and synchronization edge cases internally before any customer traffic can be impacted in the future.\\n* **Resilience Against Race Conditions:** Implement Architectural improvements to make the schema synchronization and message delivery architecture resilient to race conditions. This aims to ensure that background schema and config synchronization inconsistencies cannot disrupt real-time message delivery.\\n \\n## Detailed Description of Impact\\nThe incident impact window occurred between 10:50 US/Pacific and 13:34 US/Pacific on Friday, 28 August 2026\\\\. This issue applied globally to all Google Chat environments.\\n**Google Chat:**\\n* **Delayed message rendering:** New messages did not appear in active conversation streams automatically. Users were required to manually refresh their client or send a new message/react to force the stream to synchronize.\\n* **Stalled read states and inbox sorting:** Background actions, such as updating unread badges and moving active spaces to the top of the inbox, experienced heavy internal error rates and were significantly delayed for users.\\n* **Search indexing delays:** New messages sent during the outage window stalled in the indexing pipeline, experiencing delays in becoming searchable.\\n**Scope Exclusions:**\\n* **No Data Loss:** At no point was customer data lost or compromised. All messages transmitted during the incident window were securely captured in primary storage and successfully delivered to conversation streams once the backlog cleared.\",\"status\":\"AVAILABLE\"},\"status_impact\":\"SERVICE_INFORMATION\",\"severity\":\"low\",\"service_key\":\"Ht9Vx5PFzDPHYvrnrvSk\",\"service_name\":\"Google Chat\",\"affected_products\":[{\"title\":\"Google Chat\",\"id\":\"Ht9Vx5PFzDPHYvrnrvSk\"}],\"uri\":\"incidents/y2ufhCB886p7rveYNnko\"},{\"id\":\"sjDiph2NXGAfkYYLfK7T\",\"number\":\"9682992681334163277\",\"begin\":\"2026-08-18T23:58:00+00:00\",\"created\":\"2026-08-19T03:57:37+00:00\",\"end\":\"2026-08-19T12:40:00+00:00\",\"modified\":\"2026-08-24T16:31:00+00:00\",\"external_desc\":\"**Summary**\\nSome customers may experience slowness while using Google Drive.\\n**Description**\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\nOur engineering team is currently working on a potential fix while continuing to investigate the full root cause.\\nWe will provide an update by Tuesday, 2026-08-18 22:00 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Symptoms**\\nAffected customers may experience slowness while using Google Drive.\\n**Workaround**\\nNone at this time.\",\"updates\":[{\"created\":\"2026-08-24T16:31:00+00:00\",\"modified\":\"2026-08-24T16:31:00+00:00\",\"when\":\"2026-08-24T16:31:00+00:00\",\"text\":\"# Incident Report\\n## Summary\\nOn Tuesday, 18 August 2026 at 16:58 US/Pacific, an incident affecting Google Drive occurred for a duration of 12 hours and 42 minutes. Customers located in Australia, Philippines and New Zealand experienced severe latency, sporadic loading failures, and errors when accessing Google Drive. \\nTo our valued Google Workspace customers, we are deeply sorry for the service interruption you experienced. We are taking immediate and decisive action to address the issue and strengthen our systems against future incidents. Other Google Workspace services were not affected.\\n## Root Cause\\nGoogle utilizes a multi-tiered global load balancing and network routing architecture to serve traffic efficiently. To ensure fair use and prevent congestion, bandwidth allocation is managed by attributing traffic to specific downstream services.\\nA misconfiguration in the attribution of traffic caused the bandwidth usage to be incorrectly attributed to the frontend network load balancing tier itself, rather than to the Drive service.\\nThe sequence of events leading to customer impact was as follows:\\n* Traffic from the regional load balancing tier was incorrectly attributed to the load balancing infrastructure rather than the Drive service.\\n* Unrelated traffic caused network bandwidth usage attributed to the load balancing infrastructure in this region to exceed its quota.\\n* Network bandwidth management systems applied throttling to this traffic to enforce quotas, causing severe network congestion affecting Drive traffic in this region.\\n* This congestion led to increased latency, slowness, and error rates for users accessing Drive in the affected regions.\\n## Remediation and Prevention\\nGoogle engineers were alerted to the issue via automated monitoring and an influx of customer support cases, with initial incident response teams engaged by 17:21 US/Pacific on 18 August , 2026\\\\.\\nEngineers initially attempted to alleviate the congestion by draining regional frontend traffic to an alternate geographic location, but this proved ineffective and was reverted. The team then mitigated the issue by migrating regional traffic to an upgraded internal network routing architecture, which correctly set the traffic attribution for Drive.\\nNormal operations were fully restored at 05:40 once 100% of the regional traffic was migrated to the upgraded network architecture.\\nTo prevent a recurrence of this issue, our engineering teams are undertaking the following structural preventative actions:\\n* **Resilient Network Protocol Migration:** We are expediting the complete migration of all regional frontend traffic to the newer, resilient network routing protocol to ensure accurate bandwidth attribution under all congestion scenarios.\\n* **Delegation Root Cause Remediation:** We are addressing the underlying mechanism that caused the incorrect delegation and analyzing similar cases across the fleet to ensure it does not reoccur.\\n## Detailed Description of Impact\\nFrom 18 August 2026, 16:58 to 19 August 2026, 05:40 US/Pacific for a duration of 12 hours, 42 minutes, Google Drive users in Australia, Philippines and New Zealand experienced severe performance degradation, including high latency and errors accessing Drive or listing files via Editors. Service was fully restored as of 19 August, 05:40 US/Pacific.\\n**Google Drive:**\\n* Elevated Latency: Users experienced extreme slowness when loading the Drive interface or accessing files, with initial data retrieval times heavily delayed.\\n* Loading Failures and Errors: Many user requests timed out or returned HTTP 502 errors due to the aggressive network throttling dropping the connections.\\n**Scope Exclusion:**\\n* Customers accessing Google Drive from outside the Australia, Philippines and New Zealand regions were largely unaffected by this localized throttling event.\\n* Other Google Workspace applications not reliant on this specific routing path remained stable during the incident window.\",\"status\":\"AVAILABLE\"},{\"created\":\"2026-08-20T20:00:17+00:00\",\"modified\":\"2026-08-24T16:31:00+00:00\",\"when\":\"2026-08-20T20:00:17+00:00\",\"text\":\"# Preliminary Incident Report\\nWe sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Workspace, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future. \\nPlease note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. \\nIf you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article [**https://support.google.com/a/answer/1047213**](https://support.google.com/a/answer/1047213).\\n#[REDACTED]] Date/Time of the Issue (All time US/Pacific)\\nIncident Start: 18 August 2026 16:58\\nIncident End: 19 August 2026 05:40\\nDuration: 12 hours, 42 minutes \\n## Summary\\nOn August 18, 2026 from 16:58 to August 19, 2026 05:40 US/Pacific, Drive customers in Australia, Philippines and New Zealand experienced degraded performance and slowness. During this period users also experienced 502 errors. Other Google Workspace services were not affected. \\n## Preliminary Root Cause\\nGoogle utilizes a multi-tiered global load balancing and network routing architecture to serve traffic efficiently. To ensure fair use and prevent congestion, bandwidth allocation is managed by attributing traffic to specific downstream services.\\nA misconfiguration in the attribution of traffic caused the bandwidth usage to be incorrectly attributed to the frontend network load balancing tier itself, rather than to the Drive service.\\nThe sequence of events leading to customer impact was as follows:\\n* Traffic from the regional load balancing tier was incorrectly attributed to the load balancing infrastructure rather than the Drive service.\\n* Unrelated traffic caused network bandwidth usage attributed to the load balancing infrastructure in this region to exceed its quota.\\n* Network bandwidth management systems applied throttling to this traffic to enforce quotas, causing severe network congestion affecting Drive traffic in this region.\\n* This congestion led to increased latency, slowness, and error rates for users accessing Drive in the affected regions.\\n## Remediation \\nGoogle engineers were alerted to the issue via automated monitoring and an influx of customer support cases, with initial incident response teams engaged by 17:21 US/Pacific on August 18, 2026\\\\.\\nEngineers initially attempted to alleviate the congestion by draining regional frontend traffic to an alternate geographic location, but this proved ineffective and was reverted. The team then mitigated the issue by migrating regional traffic to an upgraded internal network routing architecture, which correctly set the traffic attribution for Drive.\\nNormal operations were fully restored once 100% of the regional traffic was migrated to the upgraded network architecture.\\n## Description of Impact\\nBetween August 18, 2026, at 16:58 US/Pacific and August 19, 2026, at 05:40 US/Pacific, Google Drive users in Australia, Philippines and New Zealand experienced severe latency, degraded performance and 502 errors.\\nDuring this roughly 12-hour and 42-minute window, affected customers may have observed significant slowness or errors when attempting to access Google Drive or listing files through Editors (e.g. [docs.google.com](http://docs.google.com)), with initial load times significantly delayed. The issue was fully mitigated as of August 19, 2026, at 05:40 US/Pacific, and normal service has been restored.\",\"status\":\"AVAILABLE\"},{\"created\":\"2026-08-19T13:05:05+00:00\",\"modified\":\"2026-08-20T20:00:17+00:00\",\"when\":\"2026-08-19T13:05:05+00:00\",\"text\":\"**Summary**\\nMultiple customers in Australia and New Zealand may have experienced slowness or errors while accessing Google Drive.\\n**Description**\\nThe issue with Google Drive has been resolved for all affected customers as of Wednesday, 2026-08-19 05:40 PDT.\\nOur engineering teams determined the slowness was the result of a localized capacity and routing issue within our regional infrastructure.\\nTo restore service, we implemented a fix to alleviate the network congestion and accelerated a planned infrastructure upgrade in the affected region to ensure normal performance levels.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms**\\nAffected customers in Australia and New Zealand may have experienced slowness or 502 errors, with some users briefly unable to access Drive.\\n**Workaround**\\nThe issue is now mitigated\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-08-19T11:26:14+00:00\",\"modified\":\"2026-08-19T13:05:05+00:00\",\"when\":\"2026-08-19T11:26:14+00:00\",\"text\":\"**Summary**\\nMultiple customers in Australia and New Zealand may experience slowness or errors while accessing Google Drive.\\n**Description**\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\nOur engineering team is actively working on a mitigation strategy. There is no ETA for full resolution.\\nWe will provide an update by Wednesday, 2026-08-19 06:00 PDT with current details.\\n**Symptoms**\\nAffected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-08-19T09:54:28+00:00\",\"modified\":\"2026-08-19T11:26:14+00:00\",\"when\":\"2026-08-19T09:54:28+00:00\",\"text\":\"**Summary**\\nMultiple customers in Australia and New Zealand may experience slowness or errors while accessing Google Drive.\\n**Description**\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\nOur engineering team is actively working on a mitigation strategy. There is no ETA for full resolution.\\nWe will provide an update by Wednesday, 2026-08-19 04:30 PDT with current details.\\n**Symptoms**\\nAffected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-08-19T08:54:05+00:00\",\"modified\":\"2026-08-19T09:54:28+00:00\",\"when\":\"2026-08-19T08:54:05+00:00\",\"text\":\"**Summary**\\nMultiple customers in Australia and New Zealand may experience slowness or errors while accessing Google Drive.\\n**Description**\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\nOur engineering team is actively working on a mitigation strategy. There is no ETA for full resolution.\\nWe will provide an update by Wednesday, 2026-08-19 03:00 PDT with current details.\\n**Symptoms**\\nAffected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-08-19T07:59:51+00:00\",\"modified\":\"2026-08-19T08:54:05+00:00\",\"when\":\"2026-08-19T07:59:51+00:00\",\"text\":\"**Summary**\\nMultiple customers in Australia and New Zealand may experience slowness or errors while accessing Google Drive.\\n**Description**\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\nOur engineering team has identified the root cause of the issue and is actively working on a mitigation strategy. There is no ETA for full resolution.\\nWe will provide an update by Wednesday, 2026-08-19 02:00 PDT with current details.\\n**Symptoms**\\nAffected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-08-19T06:49:41+00:00\",\"modified\":\"2026-08-19T07:59:51+00:00\",\"when\":\"2026-08-19T06:49:41+00:00\",\"text\":\"**Summary**\\nMultiple customers in Australia and New Zealand may experience slowness or errors while accessing Google Drive.\\n**Description**\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\nOur engineering team has identified the root cause of the issue and is actively working on a mitigation strategy. There is no ETA for full resolution.\\nWe will provide an update by Wednesday, 2026-08-19 01:30 PDT with current details.\\n**Symptoms**\\nAffected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-08-19T04:59:07+00:00\",\"modified\":\"2026-08-19T06:49:41+00:00\",\"when\":\"2026-08-19T04:59:07+00:00\",\"text\":\"**Summary**\\nMultiple customers in Australia and New Zealand may experience slowness or errors while using Google Drive.\\n**Description**\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\nOur engineering team is currently working on a potential fix while continuing to investigate the full root cause.\\nWe will provide an update by Wednesday, 2026-08-19 00:30 PDT with current details.\\n**Symptoms**\\nAffected customers in Australia and New Zealand may experience slowness or errors while using Google Drive. Some customers also may observe 502 errors with Drive inaccessible for a brief period.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_INFORMATION\"},{\"created\":\"2026-08-19T03:57:37+00:00\",\"modified\":\"2026-08-19T05:21:15+00:00\",\"when\":\"2026-08-19T03:57:37+00:00\",\"text\":\"**Summary**\\nSome customers may experience slowness while using Google Drive.\\n**Description**\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\nOur engineering team is currently working on a potential fix while continuing to investigate the full root cause.\\nWe will provide an update by Tuesday, 2026-08-18 22:00 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Symptoms**\\nAffected customers may experience slowness while using Google Drive.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_INFORMATION\"}],\"most_recent_update\":{\"created\":\"2026-08-24T16:31:00+00:00\",\"modified\":\"2026-08-24T16:31:00+00:00\",\"when\":\"2026-08-24T16:31:00+00:00\",\"text\":\"# Incident Report\\n## Summary\\nOn Tuesday, 18 August 2026 at 16:58 US/Pacific, an incident affecting Google Drive occurred for a duration of 12 hours and 42 minutes. Customers located in Australia, Philippines and New Zealand experienced severe latency, sporadic loading failures, and errors when accessing Google Drive. \\nTo our valued Google Workspace customers, we are deeply sorry for the service interruption you experienced. We are taking immediate and decisive action to address the issue and strengthen our systems against future incidents. Other Google Workspace services were not affected.\\n## Root Cause\\nGoogle utilizes a multi-tiered global load balancing and network routing architecture to serve traffic efficiently. To ensure fair use and prevent congestion, bandwidth allocation is managed by attributing traffic to specific downstream services.\\nA misconfiguration in the attribution of traffic caused the bandwidth usage to be incorrectly attributed to the frontend network load balancing tier itself, rather than to the Drive service.\\nThe sequence of events leading to customer impact was as follows:\\n* Traffic from the regional load balancing tier was incorrectly attributed to the load balancing infrastructure rather than the Drive service.\\n* Unrelated traffic caused network bandwidth usage attributed to the load balancing infrastructure in this region to exceed its quota.\\n* Network bandwidth management systems applied throttling to this traffic to enforce quotas, causing severe network congestion affecting Drive traffic in this region.\\n* This congestion led to increased latency, slowness, and error rates for users accessing Drive in the affected regions.\\n## Remediation and Prevention\\nGoogle engineers were alerted to the issue via automated monitoring and an influx of customer support cases, with initial incident response teams engaged by 17:21 US/Pacific on 18 August , 2026\\\\.\\nEngineers initially attempted to alleviate the congestion by draining regional frontend traffic to an alternate geographic location, but this proved ineffective and was reverted. The team then mitigated the issue by migrating regional traffic to an upgraded internal network routing architecture, which correctly set the traffic attribution for Drive.\\nNormal operations were fully restored at 05:40 once 100% of the regional traffic was migrated to the upgraded network architecture.\\nTo prevent a recurrence of this issue, our engineering teams are undertaking the following structural preventative actions:\\n* **Resilient Network Protocol Migration:** We are expediting the complete migration of all regional frontend traffic to the newer, resilient network routing protocol to ensure accurate bandwidth attribution under all congestion scenarios.\\n* **Delegation Root Cause Remediation:** We are addressing the underlying mechanism that caused the incorrect delegation and analyzing similar cases across the fleet to ensure it does not reoccur.\\n## Detailed Description of Impact\\nFrom 18 August 2026, 16:58 to 19 August 2026, 05:40 US/Pacific for a duration of 12 hours, 42 minutes, Google Drive users in Australia, Philippines and New Zealand experienced severe performance degradation, including high latency and errors accessing Drive or listing files via Editors. Service was fully restored as of 19 August, 05:40 US/Pacific.\\n**Google Drive:**\\n* Elevated Latency: Users experienced extreme slowness when loading the Drive interface or accessing files, with initial data retrieval times heavily delayed.\\n* Loading Failures and Errors: Many user requests timed out or returned HTTP 502 errors due to the aggressive network throttling dropping the connections.\\n**Scope Exclusion:**\\n* Customers accessing Google Drive from outside the Australia, Philippines and New Zealand regions were largely unaffected by this localized throttling event.\\n* Other Google Workspace applications not reliant on this specific routing path remained stable during the incident window.\",\"status\":\"AVAILABLE\"},\"status_impact\":\"SERVICE_DISRUPTION\",\"severity\":\"medium\",\"service_key\":\"VHNA7p3Z5p3iakj5sA8V\",\"service_name\":\"Google Drive\",\"affected_products\":[{\"title\":\"Google Drive\",\"id\":\"VHNA7p3Z5p3iakj5sA8V\"}],\"uri\":\"incidents/sjDiph2NXGAfkYYLfK7T\"},{\"id\":\"MpR24eKjibRq1YEEaTfp\",\"number\":\"17752500813029482969\",\"begin\":\"2026-08-18T19:03:00+00:00\",\"created\":\"2026-08-18T19:35:27+00:00\",\"end\":\"2026-08-18T22:30:00+00:00\",\"modified\":\"2026-08-18T23:02:54+00:00\",\"external_desc\":\"**Summary:**\\nInvestigating access issues with Classroom’s new web homepage.\\n**Description:**\\nWe are investigating reports of access issues with Classroom’s new web homepage beginning on Thursday, 2026-08-13.\\nOur engineering team is currently working on a potential fix while continuing to investigate the full root cause.\\nWe will provide an update by Tuesday, 2026-08-18 13:00 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Symptoms:**\\nAdmins and some Teachers may experience permission errors when accessing Classroom’s new web homepage.\\n**Workaround:**\\nUsers should still be able to navigate to other views from the left hand pane of the interface.\",\"updates\":[{\"created\":\"2026-08-18T23:02:54+00:00\",\"modified\":\"2026-08-18T23:02:54+00:00\",\"when\":\"2026-08-18T23:02:54+00:00\",\"text\":\"**Summary**\\n[Mitigated] Access issues for Workspace Admins with Classroom’s web homepage.\\n**Description**\\nThe issue with Classroom has been resolved for all affected users as of Tuesday, 2026-08-18 15:30 PDT.\\nFrom preliminary analysis, the issue was triggered due to a recent change which caused the permission mismatch, our Engineering team has rolled out the fix to mitigate the impact.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms**\\nWorkspace Admins may have experienced permission errors when accessing Classroom’s web homepage.\\nRemaining Classroom functionality remained unaffected.\\nThe Classroom Mobile App and the Classroom API were unaffected by the issue.\\n**Workaround**\\nNone as the issue is resolved\",\"status\":\"AVAILABLE\"},{\"created\":\"2026-08-18T20:27:49+00:00\",\"modified\":\"2026-08-18T23:02:54+00:00\",\"when\":\"2026-08-18T20:27:49+00:00\",\"text\":\"**Summary:**\\nMitigating access issues for Workspace Admins with Classroom’s web homepage.\\n**Description:**\\nOur engineering has identified the cause and mitigation work is currently underway.\\nThe mitigation rollout is expected to take several hours.\\nWe will provide more information by Wednesday, 2026-08-19 8:00 PDT.\\n**Symptoms:**\\nWorkspace Admins may experience permission errors when accessing Classroom’s web homepage.\\nRemaining Classroom functionality remains unaffected.\\nThe Classroom Mobile App and the Classroom API are unaffected by the issue.\\n**Workaround:**\\nAll other pages are unaffected and admins can navigate to other pages using the left hand pane on the interface.\\nWorkspace Admins can still create classes without any issue.\\nWorkspace Admins can also view the courses that they are in by going to https://classroom.google.com/h/tv\",\"status\":\"SERVICE_INFORMATION\"},{\"created\":\"2026-08-18T20:00:36+00:00\",\"modified\":\"2026-08-18T20:27:49+00:00\",\"when\":\"2026-08-18T20:00:36+00:00\",\"text\":\"**Summary:**\\nMitigating access issues with Classroom’s new web homepage.\\n**Description:**\\nMitigation work is currently underway by our engineering team.\\nThe mitigation is expected to complete by Tuesday, 2026-08-18 14:00 PDT.\\nWe will provide more information by Tuesday, 2026-08-18 14:15 PDT.\\n**Symptoms:**\\nAdmins and some Teachers may experience permission errors when accessing Classroom’s new web homepage.\\n**Workaround:**\\nUsers should still be able to navigate to other views from the left hand pane of the interface.\",\"status\":\"SERVICE_INFORMATION\"},{\"created\":\"2026-08-18T19:35:27+00:00\",\"modified\":\"2026-08-18T20:11:32+00:00\",\"when\":\"2026-08-18T19:35:27+00:00\",\"text\":\"**Summary:**\\nInvestigating access issues with Classroom’s new web homepage.\\n**Description:**\\nWe are investigating reports of access issues with Classroom’s new web homepage beginning on Thursday, 2026-08-13.\\nOur engineering team is currently working on a potential fix while continuing to investigate the full root cause.\\nWe will provide an update by Tuesday, 2026-08-18 13:00 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Symptoms:**\\nAdmins and some Teachers may experience permission errors when accessing Classroom’s new web homepage.\\n**Workaround:**\\nUsers should still be able to navigate to other views from the left hand pane of the interface.\",\"status\":\"SERVICE_INFORMATION\"}],\"most_recent_update\":{\"created\":\"2026-08-18T23:02:54+00:00\",\"modified\":\"2026-08-18T23:02:54+00:00\",\"when\":\"2026-08-18T23:02:54+00:00\",\"text\":\"**Summary**\\n[Mitigated] Access issues for Workspace Admins with Classroom’s web homepage.\\n**Description**\\nThe issue with Classroom has been resolved for all affected users as of Tuesday, 2026-08-18 15:30 PDT.\\nFrom preliminary analysis, the issue was triggered due to a recent change which caused the permission mismatch, our Engineering team has rolled out the fix to mitigate the impact.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms**\\nWorkspace Admins may have experienced permission errors when accessing Classroom’s web homepage.\\nRemaining Classroom functionality remained unaffected.\\nThe Classroom Mobile App and the Classroom API were unaffected by the issue.\\n**Workaround**\\nNone as the issue is resolved\",\"status\":\"AVAILABLE\"},\"status_impact\":\"SERVICE_INFORMATION\",\"severity\":\"low\",\"service_key\":\"3GoNVrVz48jeNZSyLJMV\",\"service_name\":\"Classroom\",\"affected_products\":[{\"title\":\"Classroom\",\"id\":\"3GoNVrVz48jeNZSyLJMV\"}],\"uri\":\"incidents/MpR24eKjibRq1YEEaTfp\"},{\"id\":\"AWyE1SfEzpe8Qed49Rog\",\"number\":\"12918412236754833550\",\"begin\":\"2026-06-30T09:03:00+00:00\",\"created\":\"2026-06-30T11:56:57+00:00\",\"end\":\"2026-07-01T03:45:00+00:00\",\"modified\":\"2026-07-01T04:58:32+00:00\",\"external_desc\":\"**Title**\\nInbound emails from customers to Google Support are experiencing issues.\\n**Description**\\nWe are experiencing an issue with Google Cloud Support beginning on Tuesday, 2026-06-30 02:03 PDT.\\nOur engineering team continues to investigate the issue.\\nWe will provide an update by Tuesday, 2026-06-30 05:30 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Symptoms**\\nInbound emails from customers to Google Support are experiencing issues.\\nCase creation continues to work.\\n**Workaround**\\nNone at this time.\",\"updates\":[{\"created\":\"2026-07-01T04:58:32+00:00\",\"modified\":\"2026-07-01T04:58:32+00:00\",\"when\":\"2026-07-01T04:58:32+00:00\",\"text\":\"**Title**\\n[Resolved] Inbound emails from customers to Google Support were experiencing issues.\\n**Description**\\nThe issue with inbound emails to Google Cloud Support has been resolved for all affected customers as of Tuesday, 2026-06-30 20:45 PDT.\\nFrom preliminary analysis, the engineering team identified a recent change which was preventing incoming emails from being processed. We have successfully rolled back the change to mitigate the impact.\\nBacklog processing of all the affected emails is now complete.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms**\\nInbound emails from customers to Google Support were experiencing issues.\\nCase creation and updates continued to work without any issues.\\n**Workaround**\\nNone as the issue is resolved.\",\"status\":\"AVAILABLE\"},{\"created\":\"2026-07-01T00:44:02+00:00\",\"modified\":\"2026-07-01T04:58:32+00:00\",\"when\":\"2026-07-01T00:44:02+00:00\",\"text\":\"**Title**\\nInbound emails from customers to Google Support are experiencing issues.\\n**Description**\\nThe mitigation rollout was completed as of 09:00 AM PDT on 30 June 2026, and all incoming emails are currently being processed without further disruption.\\nOur engineers are continuing to work replaying the inbound messages that were impacted between 25 June 2026 and 09:00 AM PDT on 30 June 2026. We expect the email replays to take several hours, please be advised that this backlog processing may cause case statuses to update dynamically or cause certain closed cases to temporarily reopen.\\nWe will provide more information by Tuesday, 2026-06-30 21:30 PDT.\\n**Symptoms**\\nInbound emails from customers to Google Support after 09:00 AM PDT on 30 June 2026 are working as expected. The affected emails from the time of the issue are being replayed.\\nCase creation and updates continues to work without any issues.\\n**Workaround**\\nCustomers can continue to use the portal to add comments / notes.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-06-30T19:03:19+00:00\",\"modified\":\"2026-07-01T00:44:02+00:00\",\"when\":\"2026-06-30T19:03:19+00:00\",\"text\":\"**Title**\\nInbound emails from customers to Google Support are experiencing issues.\\n**Description**\\nThe mitigation rollout was completed as of 09:00 AM PDT on 30 June 2026, and all incoming emails are currently being processed without further disruption.\\nOur engineers are continuing to work replaying the inbound messages that were impacted between 25 June 2026 and 09:00 AM PDT on 30 June 2026. We expect the email replays to take several hours, please be advised that this backlog processing may cause case statuses to update dynamically or cause certain closed cases to temporarily reopen.\\nWe will provide more information by Tuesday, 2026-06-30 17:00 PDT.\\n**Symptoms**\\nInbound emails from customers to Google Support after 09:00 AM PDT on 30 June 2026 are working as expected. The affected emails from the time of the issue are being replayed.\\nCase creation and updates continues to work without any issues.\\n**Workaround**\\nCustomers can continue to use the portal to add comments / notes.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-06-30T17:53:29+00:00\",\"modified\":\"2026-06-30T19:03:19+00:00\",\"when\":\"2026-06-30T17:53:29+00:00\",\"text\":\"**Title**\\nInbound emails from customers to Google Support are experiencing issues.\\n**Description**\\nThe mitigation rollout was completed as of 09:00 AM PDT on 30 June 2026, and all incoming emails are currently being processed without further disruption.\\nOur engineers are continuing to work replaying the inbound messages that were impacted between 25 June 2026 and 09:00 AM PDT on 30 June 2026. We expect the email replays to take several hours, please be advised that this backlog processing may cause case statuses to update dynamically or cause certain closed cases to temporarily reopen.\\nWe will provide more information by Tuesday, 2026-06-30 12:15 PDT.\\n**Symptoms**\\nInbound emails from customers to Google Support are experiencing issues.\\nCase creation and updates continues to work without any issues.\\n**Workaround**\\nCustomers can continue to use the portal to add comments / notes.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-06-30T16:36:46+00:00\",\"modified\":\"2026-06-30T17:53:29+00:00\",\"when\":\"2026-06-30T16:36:46+00:00\",\"text\":\"**Title**\\nInbound emails from customers to Google Support are experiencing issues.\\n**Description**\\nThe mitigation rollout was completed as of 09:00 AM PDT on 30 June 2026, and all incoming emails are currently being processed without further disruption.\\nOur engineering teams have shifted focus to replaying the inbound messages that were impacted between 25 June 2026 and 09:00 AM PDT on 30 June 2026.\\nWhile there is currently no specific ETA for the email replay to finish, please be advised that this backlog processing may cause case statuses to update dynamically or cause certain closed cases to temporarily reopen.\\nWe will provide more information by Tuesday, 2026-06-30 11:00 PDT.\\n**Symptoms**\\nInbound emails from customers to Google Support are experiencing issues.\\nCase creation and updates continues to work without any issues.\\n**Workaround**\\nCustomers can continue to use the portal to add comments / notes.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-06-30T15:48:35+00:00\",\"modified\":\"2026-06-30T16:36:46+00:00\",\"when\":\"2026-06-30T15:48:35+00:00\",\"text\":\"**Title**\\nInbound emails from customers to Google Support are experiencing issues.\\n**Description**\\nMitigation work is currently underway by our engineering team.\\nThe mitigation is expected to complete by Tuesday, 2026-06-30 09:45 PDT.\\nWe will provide more information by Tuesday, 2026-06-30 10:00 PDT.\\n**Symptoms**\\nInbound emails from customers to Google Support are experiencing issues.\\nCase creation and updates continues to work without any issues.\\n**Workaround**\\nCustomers can continue to use the portal to add comments / notes.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-06-30T14:51:10+00:00\",\"modified\":\"2026-06-30T15:48:35+00:00\",\"when\":\"2026-06-30T14:51:10+00:00\",\"text\":\"**Title**\\nInbound emails from customers to Google Support are experiencing issues.\\n**Description**\\nMitigation work is currently underway by our engineering team.\\nWe do not have an ETA for mitigation at this point.\\nWe will provide more information by Tuesday, 2026-06-30 08:45 PDT.\\n**Symptoms**\\nInbound emails from customers to Google Support are experiencing issues.\\nCase creation and updates continues to work without any issues.\\n**Workaround**\\nCustomers can continue to use the portal to add comments / notes.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-06-30T14:06:26+00:00\",\"modified\":\"2026-06-30T14:51:10+00:00\",\"when\":\"2026-06-30T14:06:26+00:00\",\"text\":\"**Title**\\nInbound emails from customers to Google Support are experiencing issues.\\n**Description**\\nWe are experiencing an issue with Google Cloud Support beginning on Tuesday, 2026-06-30 02:03 PDT.\\nOur engineering team continues to investigate the issue.\\nWe will provide an update by Tuesday, 2026-06-30 08:00 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Symptoms**\\nInbound emails from customers to Google Support are experiencing issues.\\nCase creation continues to work.\\n**Workaround**\\nCustomers can continue to use the portal to add comments / notes.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-06-30T13:28:46+00:00\",\"modified\":\"2026-06-30T14:06:26+00:00\",\"when\":\"2026-06-30T13:28:46+00:00\",\"text\":\"**Title**\\nInbound emails from customers to Google Support are experiencing issues.\\n**Description**\\nWe believe the issue with Google Support is mitigated.\\nOur engineering team continues to investigate the root cause.\\nCustomers are being advised if they have an urgent issue to resend previous communication to Google Support.\\nWe will provide an update by Tuesday, 2026-06-30 08:00 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Symptoms**\\nInbound emails from customers to Google Support are experiencing issues.\\nCase creation continues to work.\\n**Workaround**\\nCustomers can continue to use the portal to add comments / notes.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-06-30T12:28:59+00:00\",\"modified\":\"2026-06-30T13:28:46+00:00\",\"when\":\"2026-06-30T12:28:59+00:00\",\"text\":\"**Title**\\nInbound emails from customers to Google Support are experiencing issues.\\n**Description**\\nWe believe the issue with Google Support is mitigated.\\nOur engineering team continues to investigate the root cause.\\nCustomers are being advised if they have an urgent issue to resend previous communication to Google Support.\\nWe will provide an update by Tuesday, 2026-06-30 06:30 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Symptoms**\\nInbound emails from customers to Google Support are experiencing issues.\\nCase creation continues to work.\\n**Workaround**\\nCustomers can continue to use the portal to add comments / notes.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-06-30T12:17:19+00:00\",\"modified\":\"2026-06-30T12:28:59+00:00\",\"when\":\"2026-06-30T12:17:19+00:00\",\"text\":\"**Title**\\nInbound emails from customers to Google Support are experiencing issues.\\n**Description**\\nWe believe the issue with Google Support is mitigated.\\nOur engineering team continues to investigate the root cause.\\nCustomers are being advised if they have an urgent issue to resend previous communication to Google Support.\\nWe will provide an update by Tuesday, 2026-06-30 06:30 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Symptoms**\\nInbound emails from customers to Google Support are experiencing issues.\\nCase creation continues to work.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-06-30T11:56:57+00:00\",\"modified\":\"2026-06-30T12:17:19+00:00\",\"when\":\"2026-06-30T11:56:57+00:00\",\"text\":\"**Title**\\nInbound emails from customers to Google Support are experiencing issues.\\n**Description**\\nWe are experiencing an issue with Google Cloud Support beginning on Tuesday, 2026-06-30 02:03 PDT.\\nOur engineering team continues to investigate the issue.\\nWe will provide an update by Tuesday, 2026-06-30 05:30 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Symptoms**\\nInbound emails from customers to Google Support are experiencing issues.\\nCase creation continues to work.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_DISRUPTION\"}],\"most_recent_update\":{\"created\":\"2026-07-01T04:58:32+00:00\",\"modified\":\"2026-07-01T04:58:32+00:00\",\"when\":\"2026-07-01T04:58:32+00:00\",\"text\":\"**Title**\\n[Resolved] Inbound emails from customers to Google Support were experiencing issues.\\n**Description**\\nThe issue with inbound emails to Google Cloud Support has been resolved for all affected customers as of Tuesday, 2026-06-30 20:45 PDT.\\nFrom preliminary analysis, the engineering team identified a recent change which was preventing incoming emails from being processed. We have successfully rolled back the change to mitigate the impact.\\nBacklog processing of all the affected emails is now complete.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms**\\nInbound emails from customers to Google Support were experiencing issues.\\nCase creation and updates continued to work without any issues.\\n**Workaround**\\nNone as the issue is resolved.\",\"status\":\"AVAILABLE\"},\"status_impact\":\"SERVICE_DISRUPTION\",\"severity\":\"medium\",\"service_key\":\"dnz8HVUHoMMcW9wV91LY\",\"service_name\":\"Google Workspace Support\",\"affected_products\":[{\"title\":\"Google Workspace Support\",\"id\":\"dnz8HVUHoMMcW9wV91LY\"}],\"uri\":\"incidents/AWyE1SfEzpe8Qed49Rog\"},{\"id\":\"qqWhZSpoVLJWur1GfHfu\",\"number\":\"6760817193806533896\",\"begin\":\"2026-06-22T19:30:00+00:00\",\"created\":\"2026-06-22T22:28:55+00:00\",\"end\":\"2026-06-23T00:46:00+00:00\",\"modified\":\"2026-06-23T01:08:59+00:00\",\"external_desc\":\"**Title**\\nSome consumer Google Voice users are experiencing account suspensions.\\n**Description**\\nWe are experiencing an issue with Google Voice beginning at Monday, 2026-06-22 12:30 PDT.\\nOur engineering team has identified the root cause and is actively working to mitigate the issue.\\nWe will provide an update by Monday, 2026-06-22 17:30 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Customer Symptoms**\\nSome consumer Google Voice users are experiencing account suspensions\",\"updates\":[{\"created\":\"2026-06-23T01:08:59+00:00\",\"modified\":\"2026-06-23T01:08:59+00:00\",\"when\":\"2026-06-23T01:08:59+00:00\",\"text\":\"**Title**\\nSome consumer Google Voice users experienced account suspensions.\\n**Description**\\nThe issue with Google Voice has been resolved for all affected users as of Monday, 2026-06-22 17:46 PDT\\nOur engineers have mitigated the issue by rolling back to a previous version and reinstating suspended accounts.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms**\\nSome consumer Google Voice users experienced account suspensions\\n**Workaround**\\nThe issue is now mitigated.\",\"status\":\"AVAILABLE\"},{\"created\":\"2026-06-23T00:29:23+00:00\",\"modified\":\"2026-06-23T01:08:59+00:00\",\"when\":\"2026-06-23T00:29:23+00:00\",\"text\":\"**Title**\\nSome consumer Google Voice users are experiencing account suspensions.\\n**Description**\\nWe are experiencing an issue with Google Voice beginning at Monday, 2026-06-22 12:30 PDT.\\nOur engineering team has identified a mitigation strategy and is actively working on mitigation.\\nWe do not have an ETA for mitigation at this time.\\nWe will provide an update by Monday, 2026-06-22 22:30 PDT with current details.\\n**Symptoms**\\nSome consumer Google Voice users are experiencing account suspensions\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_INFORMATION\"},{\"created\":\"2026-06-22T22:28:55+00:00\",\"modified\":\"2026-06-23T00:29:23+00:00\",\"when\":\"2026-06-22T22:28:55+00:00\",\"text\":\"**Title**\\nSome consumer Google Voice users are experiencing account suspensions.\\n**Description**\\nWe are experiencing an issue with Google Voice beginning at Monday, 2026-06-22 12:30 PDT.\\nOur engineering team has identified the root cause and is actively working to mitigate the issue.\\nWe will provide an update by Monday, 2026-06-22 17:30 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Customer Symptoms**\\nSome consumer Google Voice users are experiencing account suspensions\",\"status\":\"SERVICE_INFORMATION\"}],\"most_recent_update\":{\"created\":\"2026-06-23T01:08:59+00:00\",\"modified\":\"2026-06-23T01:08:59+00:00\",\"when\":\"2026-06-23T01:08:59+00:00\",\"text\":\"**Title**\\nSome consumer Google Voice users experienced account suspensions.\\n**Description**\\nThe issue with Google Voice has been resolved for all affected users as of Monday, 2026-06-22 17:46 PDT\\nOur engineers have mitigated the issue by rolling back to a previous version and reinstating suspended accounts.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms**\\nSome consumer Google Voice users experienced account suspensions\\n**Workaround**\\nThe issue is now mitigated.\",\"status\":\"AVAILABLE\"},\"status_impact\":\"SERVICE_INFORMATION\",\"severity\":\"low\",\"service_key\":\"gh79SrQk6wqonWoxPdCm\",\"service_name\":\"Google Voice\",\"affected_products\":[{\"title\":\"Google Voice\",\"id\":\"gh79SrQk6wqonWoxPdCm\"}],\"uri\":\"incidents/qqWhZSpoVLJWur1GfHfu\"},{\"id\":\"CzZUn98mhTcEiCJo27Kv\",\"number\":\"11672950073358311792\",\"begin\":\"2026-06-10T10:20:00+00:00\",\"created\":\"2026-06-10T12:40:45+00:00\",\"end\":\"2026-06-10T17:30:00+00:00\",\"modified\":\"2026-06-16T17:41:16+00:00\",\"external_desc\":\"**Title**\\nGemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.\\n**Description**\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\nOur engineering team continues to investigate the issue.\\nWe will provide an update by Wednesday, 2026-06-10 06:30 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Symptoms**\\nGemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.\\n**Workaround**\\nNone at this time.\",\"updates\":[{\"created\":\"2026-06-16T17:41:16+00:00\",\"modified\":\"2026-06-16T17:41:17+00:00\",\"when\":\"2026-06-16T17:41:16+00:00\",\"text\":\"# Incident Report\\n## Summary\\nOn Wednesday, June 10, 2026, at 03:30 US/Pacific, an incident occurred affecting the availability of Gemini services across all surfaces, including web, mobile (iOS and Android), and integrations within Google Chrome. The incident resulted in an elevated error rate for approximately 6 hours and 55 minutes, with full service restoration confirmed after a total duration of 14 hours and 49 minutes.\\nWe sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Cloud, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future.\\n## Root Cause\\nThe disruption was caused by extreme read contention within the foundational database service that manages tool deployment metadata.\\nThe primary cause was an index design issue within the database where a column used for tracking deployment expirations contained a high volume of rows with similar values. This resulted in \\\"hotspotting,\\\" where traffic was concentrated on a very small number of database shards, overwhelming them. Additionally, the service utilized an in-memory cache with a very short Time To Live (TTL) of only one minute, requiring frequent database refreshes.\\nThe failure sequence was as follows:\\n1. **Technical Trigger:** The trigger was an upward spike in the Queries Per Second (QPS) from the frontend. The underlying system was already running close to maximum usage and the incoming QPS crossed a tipping threshold beyond which the underlying spanner started getting overloaded.\\n2. **Systemic Effect:** This short window, combined with an indexing anomaly where specific metadata fields containing empty values were clustered on a single database partition, led to a \\\"hot shard\\\" condition.\\n3. **Contention and Latency:** As request volume increased, the internal tool management service experienced a more than 10x surge in database calls because the cache hit rate dropped significantly.\\n4. **Failure State:** The underlying storage layer reached its capacity for handling incoming queries, and database failure rates skyrocketed to 60%. This led to cache hit rate dropping down to 50%.\\nAutomated validation failed to prevent this because the indexing strategy did not sufficiently account for the concentration of data rows when specific identifiers were absent.\\n## Remediation and Prevention\\nThe issue was detected by internal real-time monitoring alerts indicating high processing error rates. Engineering teams performed several targeted actions to restore the service:\\n* **Throttling:** Applied Rate Access Control Lists (RateACLs) to the most affected database shards to allow internal caches to recover.\\n* **Index Redistribution:** Gradually redistributed common values in the database index to random values in a specific target range, effectively spreading the load across multiple shards.\\n* **Cache Policy Update:** Deployed a configuration change to increase the in-memory cache Time To Live (TTL) from 1 minute to 20 minutes, significantly reducing database load.\\n* **Deadline Adjustment:** Shortened the Remote Procedure Call (RPC) deadlines for database queries to prevent request pile-ups during periods of high latency.\\n## Prevention\\nTo prevent a recurrence, Google engineering will be implementing the following actions:\\n* **Index Design Standards:** Restructure of database index to prevent \\\"hotspotting\\\".\\n* **Adaptive Cache Tuning:** Implement more robust cache policies that prevent load amplification during periods of backend stress.\\n* **Enhanced Hotspot Monitoring:** Deploy improved monitoring and alerting specifically designed to detect uneven database load distribution before it impacts user traffic.\\n* **Service Resiliency:** Implement advanced load protection mechanisms, such as smart task resizing and query coalescing for preventing spanner overloads.\\n## Detailed Description of Impact\\nThe impact window occurred between 03:30 US/Pacific and 10:25 US/Pacific on June 10, 2026\\\\.\\n**Gemini App (Web and Mobile):**\\n* High Failure Rates: Users experienced a 50% error rate when attempting to send prompts, receiving \\\"Something went wrong\\\" messages .\\n* Service Unavailability: The issue affected both consumer accounts and enterprise Google Workspace customers .\\n* Enterprise Google Workspace Users encountered disruptions when attempting to attach assets from Google Drive within the web application (gemini.google.com). Affected users observed the \\\"Add from Drive\\\" functionality to be visually disabled and unresponsive following interaction with the \\\"+\\\" prompt.\\n**Integrations:**\\n* **Google Chrome Integration:** The Gemini Side Panel and integrated conversational features encountered persistent timeouts.\",\"status\":\"AVAILABLE\"},{\"created\":\"2026-06-12T11:14:34+00:00\",\"modified\":\"2026-06-16T17:41:16+00:00\",\"when\":\"2026-06-12T11:14:34+00:00\",\"text\":\"# Preliminary Incident Report\\nWe apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article [https://support.google.com/a/answer/1047213](https://support.google.com/a/answer/1047213).\\n#[REDACTED]] Date/Time of the Issue (All time US/Pacific)\\nIncident Start: 10 June 2026 03:20\\nIncident End: 10 June 2026 10:25\\nDuration: 7 hours, 5 minutes\\n## Summary\\nOn Wednesday, 10 June 2026 03:20 US/Pacific, an incident occurred affecting the availability of Gemini services across all surfaces, including web, mobile (iOS and Android), and integrations within Google Chrome. The incident resulted in an elevated error rate for 7 hours and 5 minutes.\\n## Preliminary Root Cause\\nThe disruption was caused by extreme read contention within the foundational database service that manages tool deployment metadata.\\nThe primary cause was back-pressure on specific database replicas that experienced high loads. The issue was made worse because of an index design issue within the database where a column used for tracking deployment expirations contained a high volume of rows with a similar fixed value. This resulted in a condition where traffic was concentrated on a very small number of database instances, overwhelming them. Additionally, the service utilized an in-memory cache with a short Time To Live (TTL) of only one minute, requiring frequent database refreshes.\\nThe failure sequence was as follows:\\n1. **Technical Trigger:** The database service became overloaded, causing back-pressure on specific replicas.\\n2. **Systemic Effect:** An application level caching layer configuration is set to a short 1-minute expiration window for deployed tools. This short window, combined with an indexing anomaly where specific metadata fields containing empty values were clustered on a single database partition. The failed DB queries resulted in failure to populate application level cache.\\n3. **Contention and Latency:** As request volume increased, the internal tool management service experienced \\u003e10x surge in database calls because the cache hit rate dropped significantly.\\n4. **Failure State:** The underlying storage layer reached its transaction limits, with database response success rates dropping as low as 10% on the affected partitions.\\n## Remediation\\nThe issue was detected by internal real-time monitoring alerts indicating high processing error rates. Engineering teams performed several targeted actions to restore the service:\\n* **Throttling:** Applied rate limiting to the most affected database instances to allow internal caches to recover.\\n* **Index Redistribution:** Gradually redistributed fixed index values in the database to random values in a specific range, effectively spreading the load across multiple instances.\\n* **Cache Policy Update:** Deployed a configuration change to increase the in-memory cache Time To Live (TTL) from 1 minute to 20 minutes, significantly reducing database load.\\n* **Deadline Adjustment:** Shortened the Remote Procedure Call (RPC) deadlines for database queries to prevent request pile-ups during periods of high latency.\\n## Description of Impact\\nThe impact window occurred between 03:20 US/Pacific and 10:25 US/Pacific on 10 June 2026.\\n**Gemini App (Web and Mobile):**\\n* Users experienced a 50% error rate when attempting to send prompts, receiving \\\"Something went wrong\\\" messages. The issue affected both consumer accounts and enterprise Google Workspace customers .\\n**Integrations:**\\n* **Google Chrome Integration:** The Gemini Side Panel and integrated conversational features encountered persistent timeouts.\",\"status\":\"AVAILABLE\"},{\"created\":\"2026-06-11T01:07:25+00:00\",\"modified\":\"2026-06-12T11:14:34+00:00\",\"when\":\"2026-06-11T01:07:25+00:00\",\"text\":\"**Title**\\nGemini App in Workspace users were experiencing “Something Went Wrong” errors.\\n**Description**\\nThe issue with Gemini has been resolved for all the affected users as of Wednesday, 2026-06-10 10:30 US/Pacific.\\nFrom preliminary analysis, the issue was triggered by a performance issue in our backend database which impacted the retrieval of Gemini App tools catalog.\\nOur engineers mitigated the issue by optimizing the load distribution across our backend database fleet.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms**\\nAffected users may have seen \\\"Something went wrong\\\" errors (codes 1099, 1076) when prompting Gemini while using the following services.\\nGemini App in Workspace includes the following platforms: - Web, MacOS, iOS and Android - Gemini in Chrome\\n**Workaround**\\nNone as the issue is now resolved.\",\"status\":\"AVAILABLE\"},{\"created\":\"2026-06-10T20:19:31+00:00\",\"modified\":\"2026-06-11T01:07:25+00:00\",\"when\":\"2026-06-10T20:19:31+00:00\",\"text\":\"**Title**\\nGemini App in Workspace customers are experiencing “Something Went Wrong” errors.\\n**Description**\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\nOur engineering team has applied mitigations to reduce impact and confirmed the majority of users are no longer observing impact as of Wednesday, 2026-06-10 10:30 PDT. We will continue to monitor the service for stability.\\nWe will provide an update by Wednesday, 2026-06-10 20:30 PDT with current details.\\n**Symptoms**\\nThe following services are experiencing “Something Went Wrong 1099 or 1076” errors:\\nGemini app in Workspace includes the following platforms:\\n* Web, MacOS, iOS and Android\\n* Gemini in Chrome\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-06-10T18:37:53+00:00\",\"modified\":\"2026-06-10T20:19:31+00:00\",\"when\":\"2026-06-10T18:37:53+00:00\",\"text\":\"**Title**\\nGemini App in Workspace customers are experiencing “Something Went Wrong” errors.\\n**Description**\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\nOur engineering team has applied mitigations to reduce impact and continues to investigate the root cause. We are seeing signs of recovery and will continue to monitor progress.\\nWe will provide an update by Wednesday, 2026-06-10 14:30 PDT with current details.\\n**Symptoms**\\nThe following services are experiencing “Something Went Wrong 1099 or 1076” errors:\\nGemini app in Workspace includes the following platforms:\\n* Web, MacOS, iOS and Android\\n* Gemini in Chrome\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-06-10T16:24:54+00:00\",\"modified\":\"2026-06-10T18:37:53+00:00\",\"when\":\"2026-06-10T16:24:54+00:00\",\"text\":\"**Title**\\nGemini App in Workspace customers are experiencing “Something Went Wrong” errors.\\n**Description**\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\nOur engineering team has identified a mitigation which is currently in progress. We do not have an ETA for mitigation at this time.\\nWe will provide an update by Wednesday, 2026-06-10 12:30 PDT with current details.\\n**Symptoms**\\nThe following services are experiencing “Something Went Wrong 1099 or 1076” errors:\\nGemini app in Workspace includes the following platforms:\\n* Web, MacOS, iOS and Android\\n* Gemini in Chrome\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-06-10T15:10:52+00:00\",\"modified\":\"2026-06-10T16:24:54+00:00\",\"when\":\"2026-06-10T15:10:52+00:00\",\"text\":\"**Title**\\nGemini App in Workspace customers are experiencing “Something Went Wrong” errors.\\n**Description**\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\nOur engineering team continues to investigate the issue and identify a mitigation strategy.\\nWe will provide an update by Wednesday, 2026-06-10 09:30 PDT with current details.\\n**Symptoms**\\nThe following services are experiencing “Something Went Wrong” errors.:\\n* Gemini app in Workspace includes the following platforms;\\n* Web, MacOS, iOS and Android\\n* Gemini in Chrome\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-06-10T13:21:18+00:00\",\"modified\":\"2026-06-10T15:10:52+00:00\",\"when\":\"2026-06-10T13:21:18+00:00\",\"text\":\"**Title**\\nGemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.\\n**Description**\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\nOur engineering team continues to investigate the issue.\\nWe will provide an update by Wednesday, 2026-06-10 08:00 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Symptoms**\\nGemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-06-10T12:40:45+00:00\",\"modified\":\"2026-06-10T13:21:18+00:00\",\"when\":\"2026-06-10T12:40:45+00:00\",\"text\":\"**Title**\\nGemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.\\n**Description**\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\nOur engineering team continues to investigate the issue.\\nWe will provide an update by Wednesday, 2026-06-10 06:30 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Symptoms**\\nGemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_DISRUPTION\"}],\"most_recent_update\":{\"created\":\"2026-06-16T17:41:16+00:00\",\"modified\":\"2026-06-16T17:41:17+00:00\",\"when\":\"2026-06-16T17:41:16+00:00\",\"text\":\"# Incident Report\\n## Summary\\nOn Wednesday, June 10, 2026, at 03:30 US/Pacific, an incident occurred affecting the availability of Gemini services across all surfaces, including web, mobile (iOS and Android), and integrations within Google Chrome. The incident resulted in an elevated error rate for approximately 6 hours and 55 minutes, with full service restoration confirmed after a total duration of 14 hours and 49 minutes.\\nWe sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Cloud, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future.\\n## Root Cause\\nThe disruption was caused by extreme read contention within the foundational database service that manages tool deployment metadata.\\nThe primary cause was an index design issue within the database where a column used for tracking deployment expirations contained a high volume of rows with similar values. This resulted in \\\"hotspotting,\\\" where traffic was concentrated on a very small number of database shards, overwhelming them. Additionally, the service utilized an in-memory cache with a very short Time To Live (TTL) of only one minute, requiring frequent database refreshes.\\nThe failure sequence was as follows:\\n1. **Technical Trigger:** The trigger was an upward spike in the Queries Per Second (QPS) from the frontend. The underlying system was already running close to maximum usage and the incoming QPS crossed a tipping threshold beyond which the underlying spanner started getting overloaded.\\n2. **Systemic Effect:** This short window, combined with an indexing anomaly where specific metadata fields containing empty values were clustered on a single database partition, led to a \\\"hot shard\\\" condition.\\n3. **Contention and Latency:** As request volume increased, the internal tool management service experienced a more than 10x surge in database calls because the cache hit rate dropped significantly.\\n4. **Failure State:** The underlying storage layer reached its capacity for handling incoming queries, and database failure rates skyrocketed to 60%. This led to cache hit rate dropping down to 50%.\\nAutomated validation failed to prevent this because the indexing strategy did not sufficiently account for the concentration of data rows when specific identifiers were absent.\\n## Remediation and Prevention\\nThe issue was detected by internal real-time monitoring alerts indicating high processing error rates. Engineering teams performed several targeted actions to restore the service:\\n* **Throttling:** Applied Rate Access Control Lists (RateACLs) to the most affected database shards to allow internal caches to recover.\\n* **Index Redistribution:** Gradually redistributed common values in the database index to random values in a specific target range, effectively spreading the load across multiple shards.\\n* **Cache Policy Update:** Deployed a configuration change to increase the in-memory cache Time To Live (TTL) from 1 minute to 20 minutes, significantly reducing database load.\\n* **Deadline Adjustment:** Shortened the Remote Procedure Call (RPC) deadlines for database queries to prevent request pile-ups during periods of high latency.\\n## Prevention\\nTo prevent a recurrence, Google engineering will be implementing the following actions:\\n* **Index Design Standards:** Restructure of database index to prevent \\\"hotspotting\\\".\\n* **Adaptive Cache Tuning:** Implement more robust cache policies that prevent load amplification during periods of backend stress.\\n* **Enhanced Hotspot Monitoring:** Deploy improved monitoring and alerting specifically designed to detect uneven database load distribution before it impacts user traffic.\\n* **Service Resiliency:** Implement advanced load protection mechanisms, such as smart task resizing and query coalescing for preventing spanner overloads.\\n## Detailed Description of Impact\\nThe impact window occurred between 03:30 US/Pacific and 10:25 US/Pacific on June 10, 2026\\\\.\\n**Gemini App (Web and Mobile):**\\n* High Failure Rates: Users experienced a 50% error rate when attempting to send prompts, receiving \\\"Something went wrong\\\" messages .\\n* Service Unavailability: The issue affected both consumer accounts and enterprise Google Workspace customers .\\n* Enterprise Google Workspace Users encountered disruptions when attempting to attach assets from Google Drive within the web application (gemini.google.com). Affected users observed the \\\"Add from Drive\\\" functionality to be visually disabled and unresponsive following interaction with the \\\"+\\\" prompt.\\n**Integrations:**\\n* **Google Chrome Integration:** The Gemini Side Panel and integrated conversational features encountered persistent timeouts.\",\"status\":\"AVAILABLE\"},\"status_impact\":\"SERVICE_DISRUPTION\",\"severity\":\"medium\",\"service_key\":\"npdyhgECDJ6tB66MxXyo\",\"service_name\":\"Gemini\",\"affected_products\":[{\"title\":\"Gemini\",\"id\":\"npdyhgECDJ6tB66MxXyo\"}],\"uri\":\"incidents/CzZUn98mhTcEiCJo27Kv\"},{\"id\":\"vagmms1y6yRxwg6RMQZp\",\"number\":\"6811500350919047262\",\"begin\":\"2026-05-31T03:28:00+00:00\",\"created\":\"2026-05-31T04:59:04+00:00\",\"end\":\"2026-05-31T04:24:00+00:00\",\"modified\":\"2026-06-04T20:11:26+00:00\",\"external_desc\":\"**Title**\\nMultiple Workspace Products experiencing Issues.\\n**Description**\\nWe are experiencing an issue with Google Voice, Google Meet, Google Drive, Google Chat, Google Calendar, Gmail, Google Docs beginning on Saturday, 2026-05-30 20:28 PDT.\\nOur engineering team continues to investigate the issue.\\nWe will provide an update by Saturday, 2026-05-30 22:10 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Symptoms**\\nCustomers may experience issues loading the application/ intermittent high latency or errors while using the application.\\n**Workaround**\\nNone at this time.\",\"updates\":[{\"created\":\"2026-06-04T20:11:26+00:00\",\"modified\":\"2026-06-04T20:11:27+00:00\",\"when\":\"2026-06-04T20:11:26+00:00\",\"text\":\"# Incident Report\\n## Summary\\nStarting at 20:15 US/Pacific on Saturday, 30 May 2026, various Google Workspace services encountered service-level disruptions lasting approximately 75 minutes. The scope of impact spanned several products, specifically affecting Gmail, Google Docs, Drive, Calendar, Chat, Meet, and Google Voice.\\nThroughout the duration of this event, users faced obstacles while attempting to initialize applications, in addition to experiencing sporadic instances of elevated latency and request failures.\\nGoogle offers a sincere apology for the operational interference this issue introduced to your organization. Recognizing the critical nature of Google Cloud to your daily operations, we deeply regret any resulting loss in productivity. Engineering efforts are currently focused on remediating the underlying cause to ensure future stability.\\n## Root Cause\\nThe disruption originated from failures in a critical cluster infrastructure component responsible for Google's service discovery and configuration management systems. This system facilitates resource synchronization, manages shared data access, and coordinates background tasks for numerous Google services.\\nA high volume of rapid configuration changes triggered the regional failure in Workspace Services, preventing Workspace servers from reliably processing traffic. To mitigate the impact, Google Workspace traffic was redirected to alternative regions.\\n## Remediation and Prevention\\nGoogle engineers were alerted to the issue via automated monitoring on Saturday, 30 May 2026 at 20:28 US/Pacific and immediately began an investigation. The investigation identified elevated error rates and high latency affecting multiple Google Workspace backend services. Upon investigation our engineering team confirmed errors were primarily originating from some regional servers due to coordination service failures in the region.\\nTo mitigate the impact, Google engineers redirected traffic away from the affected servers.\\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\\n- Prevent recurrence of this incident by rolling out the fix for the latent encryption library issue, which improves the handling of internal log compaction compactions during active file writes, to prevent recurrence of the cluster instability.\\n- Investigating additional resilience in our database layer.\\n## Detailed Description of Impact\\nOn 30 May 2026 from 20:15 to 21:30 US/Pacific, affected customers encountered difficulties while loading applications or experienced failures when performing tasks within the following products:\\n- Gmail\\n- Google Calendar\\n- Google Chat\\n- Google Docs\\n- Google Drive\\n- Google Meet\\n- Google Voice\",\"status\":\"AVAILABLE\"},{\"created\":\"2026-06-02T03:50:13+00:00\",\"modified\":\"2026-06-04T20:11:26+00:00\",\"when\":\"2026-06-02T03:50:13+00:00\",\"text\":\"# Preliminary Incident Report\\nWe sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Workspace, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future.\\nPlease note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete.\\nIf you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using help article https://support.google.com/a/answer/1047213\\n#[REDACTED] Date/Time of the Issue (All time US/Pacific)\\nIncident Start: 30 May 2026 20:15\\nIncident End: 30 May 2026 21:30\\nDuration: 1 hour, 15 minutes\\n## Summary\\nStarting on Saturday, 30 May 2026 at 20:15 US/Pacific, multiple Google Workspace products experienced service disruptions for a duration of 1 hour and 15 minutes. The affected services included Google Voice, Google Meet, Google Drive, Google Chat, Google Calendar, Gmail, and Google Docs. During this incident, customers encountered difficulties loading applications, as well as intermittent high latency or errors.\\n## Preliminary Root Cause\\nThe disruption originated from failures in a regional infrastructure component responsible for Google's distributed lock service. This critical system facilitates resource synchronization, manages shared data access, and coordinates background tasks for numerous Google services.\\nA high volume of rapid configuration changes triggered the regional failure, preventing Workspace servers from reliably processing traffic. To mitigate the impact, Google Workspace traffic was redirected to alternative regions.\\nGoogle engineers have begun a full root cause analysis and will provide additional information once it is available.\\n## Remediation\\nGoogle engineers were alerted to the issue via automated monitoring at 20:28 US/Pacific on Saturday, 30 May 2026, and immediately began an investigation. The investigation identified elevated error rates and high latency affecting multiple Google Workspace backend services. Upon investigation our engineering team confirmed errors were primarily originating from some regional servers due to coordination service failures in the region.\\nTo mitigate the impact, Google engineers redirected traffic away from the affected servers.\\n## Description of Impact\\nAffected customers encountered difficulties loading applications or experienced failures when performing tasks within the following products:\\n* Gmail\\n* Google Calendar\\n* Google Chat\\n* Google Docs\\n* Google Drive\\n* Google Meet\\n* Google Voice\",\"status\":\"AVAILABLE\"},{\"created\":\"2026-05-31T05:11:14+00:00\",\"modified\":\"2026-06-02T03:50:13+00:00\",\"when\":\"2026-05-31T05:11:14+00:00\",\"text\":\"**Title**\\n[Mitigated] Multiple Workspace Products experienced Issues.\\n**Description**\\nThe issue with Google Voice, Google Meet, Google Drive, Google Chat, Google Calendar, Gmail, Google Docs has been resolved for all affected users as of Saturday, 2026-05-30 21:30 PDT.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms**\\nCustomers may have experienced issues loading the application/ intermittent high latency or errors while using the application.\\n**Workaround**\\nNone at this time as the issue stands mitigated\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-05-31T04:59:04+00:00\",\"modified\":\"2026-05-31T05:11:14+00:00\",\"when\":\"2026-05-31T04:59:04+00:00\",\"text\":\"**Title**\\nMultiple Workspace Products experiencing Issues.\\n**Description**\\nWe are experiencing an issue with Google Voice, Google Meet, Google Drive, Google Chat, Google Calendar, Gmail, Google Docs beginning on Saturday, 2026-05-30 20:28 PDT.\\nOur engineering team continues to investigate the issue.\\nWe will provide an update by Saturday, 2026-05-30 22:10 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Symptoms**\\nCustomers may experience issues loading the application/ intermittent high latency or errors while using the application.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_DISRUPTION\"}],\"most_recent_update\":{\"created\":\"2026-06-04T20:11:26+00:00\",\"modified\":\"2026-06-04T20:11:27+00:00\",\"when\":\"2026-06-04T20:11:26+00:00\",\"text\":\"# Incident Report\\n## Summary\\nStarting at 20:15 US/Pacific on Saturday, 30 May 2026, various Google Workspace services encountered service-level disruptions lasting approximately 75 minutes. The scope of impact spanned several products, specifically affecting Gmail, Google Docs, Drive, Calendar, Chat, Meet, and Google Voice.\\nThroughout the duration of this event, users faced obstacles while attempting to initialize applications, in addition to experiencing sporadic instances of elevated latency and request failures.\\nGoogle offers a sincere apology for the operational interference this issue introduced to your organization. Recognizing the critical nature of Google Cloud to your daily operations, we deeply regret any resulting loss in productivity. Engineering efforts are currently focused on remediating the underlying cause to ensure future stability.\\n## Root Cause\\nThe disruption originated from failures in a critical cluster infrastructure component responsible for Google's service discovery and configuration management systems. This system facilitates resource synchronization, manages shared data access, and coordinates background tasks for numerous Google services.\\nA high volume of rapid configuration changes triggered the regional failure in Workspace Services, preventing Workspace servers from reliably processing traffic. To mitigate the impact, Google Workspace traffic was redirected to alternative regions.\\n## Remediation and Prevention\\nGoogle engineers were alerted to the issue via automated monitoring on Saturday, 30 May 2026 at 20:28 US/Pacific and immediately began an investigation. The investigation identified elevated error rates and high latency affecting multiple Google Workspace backend services. Upon investigation our engineering team confirmed errors were primarily originating from some regional servers due to coordination service failures in the region.\\nTo mitigate the impact, Google engineers redirected traffic away from the affected servers.\\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\\n- Prevent recurrence of this incident by rolling out the fix for the latent encryption library issue, which improves the handling of internal log compaction compactions during active file writes, to prevent recurrence of the cluster instability.\\n- Investigating additional resilience in our database layer.\\n## Detailed Description of Impact\\nOn 30 May 2026 from 20:15 to 21:30 US/Pacific, affected customers encountered difficulties while loading applications or experienced failures when performing tasks within the following products:\\n- Gmail\\n- Google Calendar\\n- Google Chat\\n- Google Docs\\n- Google Drive\\n- Google Meet\\n- Google Voice\",\"status\":\"AVAILABLE\"},\"status_impact\":\"SERVICE_DISRUPTION\",\"severity\":\"medium\",\"service_key\":\"zall\",\"service_name\":\"Multiple Products\",\"affected_products\":[{\"title\":\"Gmail\",\"id\":\"prPt2Yra2CbGsbEm9cpC\"},{\"title\":\"Google Calendar\",\"id\":\"n5hVexuq1PM9onY9qrmo\"},{\"title\":\"Google Chat\",\"id\":\"Ht9Vx5PFzDPHYvrnrvSk\"},{\"title\":\"Google Docs\",\"id\":\"wNHuVFtZEWU5Mmj2eRCK\"},{\"title\":\"Google Drive\",\"id\":\"VHNA7p3Z5p3iakj5sA8V\"},{\"title\":\"Google Meet\",\"id\":\"sUH4BQXzYXma7NiS94Hi\"},{\"title\":\"Google Voice\",\"id\":\"gh79SrQk6wqonWoxPdCm\"}],\"uri\":\"incidents/vagmms1y6yRxwg6RMQZp\"},{\"id\":\"CzVZ5zk2Gf2ixPSt6RT3\",\"number\":\"17128406613861610004\",\"begin\":\"2026-05-21T07:00:00+00:00\",\"created\":\"2026-05-21T15:37:18+00:00\",\"end\":\"2026-05-21T15:47:00+00:00\",\"modified\":\"2026-05-21T16:10:17+00:00\",\"external_desc\":\"**Summary**\\nGmail users are experiencing delayed delivery of messages sent from the bank.in domains.\\n**Description**\\nWe are experiencing an issue with Gmail beginning on Thursday, 2026-05-21 at 00:00 PDT.\\nMitigation efforts are currently underway by our engineering team.\\nWe will provide the next update by Thursday, 2026-05-21 at 09:30 PDT.\\n**Diagnosis / Customer Symptoms**\\nGmail users are experiencing delayed delivery of messages sent from the bank.in domains.\\n**Workaround**\\nNone at this time.\",\"updates\":[{\"created\":\"2026-05-21T16:09:55+00:00\",\"modified\":\"2026-05-21T16:10:17+00:00\",\"when\":\"2026-05-21T16:09:55+00:00\",\"text\":\"**Summary**\\nGmail users experienced delayed delivery of messages sent from the bank.in domains.\\n**Description**\\nOur internal telemetry shows that the issue with Gmail has been resolved for all affected users as of Thursday, 2026-05-21 at 08:47 US/Pacific.\\nOur engineering team is actively monitoring the system to ensure continued stability.\\n**Diagnosis / Customer Symptoms**\\nCustomers should expect new messages to be delivered normally, older emails could take some time as the sender
[...truncated 13350 characters...]
ou for your patience while we worked on resolving the issue.\\n**Symptoms**\\nImpacted users might have experienced authentication and sync failures while using Microsoft Exchange Online accounts through Gmail App on Android devices.\\n**Workaround**\\nUpdate to the latest version of Gmail Android (2026.05.11) in Google Play Store.\",\"status\":\"AVAILABLE\"},{\"created\":\"2026-05-22T00:04:40+00:00\",\"modified\":\"2026-05-22T18:57:24+00:00\",\"when\":\"2026-05-22T00:04:40+00:00\",\"text\":\"**Summary**\\nGmail Android users using Microsoft Exchange Online accounts may fail to Log In and Sync due to authentication issues.\\n**Description**\\nOur engineering team identified the issue and released a new version of the Gmail Android application (version 2026.05.11) to the Google Play Store, which contains a fix to unblock affected users.\\nIf you are currently encountering this issue, please upgrade to the latest version of the Gmail Android app. Users operating on the new app version should be able to successfully log in and sync their accounts.\\nWe will provide more information by Friday, 2026-05-22 17:00 US/Pacific with current details.\\nWe thank you for your patience while we are working on resolving the issue.\\n**Symptoms**\\nImpacted users may experience authentication and sync failures while using Microsoft Exchange Online accounts through Gmail App on Android devices.\\n**Workaround**\\nAffected users could update to the latest version of Gmail Android (version 2026.05.11) in Google Play Store.\",\"status\":\"SERVICE_INFORMATION\"},{\"created\":\"2026-05-19T21:55:32+00:00\",\"modified\":\"2026-05-22T00:04:40+00:00\",\"when\":\"2026-05-19T21:55:32+00:00\",\"text\":\"**Summary**\\nGmail Android users using Microsoft Exchange Online accounts may fail to Log In and Sync due to authentication issues.\\n**Description**\\nWe are experiencing an issue with Gmail Android client connecting with Microsoft Exchange Online accounts beginning on Saturday 2026-05-06.\\nOur engineering team has identified the issue and started rolling out a new version of Gmail Android release that contains the fix to unblock users.\\nUsers already on the new app version will be able to Log In and Sync their accounts successfully.\\nWe will provide more information by Thursday, 2026-05-21 17:00 US/Pacific with current details.\\nWe apologize to all who are affected by the disruption.\\n**Customer Symptoms**\\nImpacted users may experience authentication and sync failures while using Microsoft Exchange Online accounts through Gmail App on Android devices.\\n**Workaround**\\nAffected users can use Microsoft Web App to access the Inbox.\",\"status\":\"SERVICE_INFORMATION\"},{\"created\":\"2026-05-18T21:45:35+00:00\",\"modified\":\"2026-05-19T21:55:32+00:00\",\"when\":\"2026-05-18T21:45:35+00:00\",\"text\":\"**Summary**\\nGmail Android users using Microsoft Exchange Online accounts may fail to Log In and Sync due to authentication issues.\\n**Description**\\nWe are experiencing an issue with Gmail Android client connecting with Microsoft Exchange Online accounts beginning on Saturday 2026-05-06.\\nOur engineering team has identified the issue and is actively working on rolling out a fix to unblock all the affected customers.\\nWe will provide more information by Tuesday, 2026-05-19 15:00 US/Pacific with current details.\\nWe apologize to all who are affected by the disruption.\\n**Customer Symptoms**\\nImpacted users may experience authentication and sync failures while using Microsoft Exchange Online accounts through Gmail App on Android devices.\\n**Workaround**\\nAffected users can use Microsoft Web App to access the Inbox.\",\"status\":\"SERVICE_INFORMATION\"},{\"created\":\"2026-05-16T02:05:03+00:00\",\"modified\":\"2026-05-18T21:45:35+00:00\",\"when\":\"2026-05-16T02:05:03+00:00\",\"text\":\"**Summary**\\nGmail Android users using Microsoft Exchange Online may fail to log in due to authentication issues.\\n**Description**\\nWe are experiencing an issue with Gmail beginning on Saturday 2026-05-06.\\nOur engineering team continues to investigate the issue.\\nWe will provide more information by Monday, 2026-05-18 15:00 US/Pacific with current details.\\nWe apologize to all who are affected by the disruption.\\n**Customer Symptoms**\\nImpacted users may experience authentication failures while logging in to Microsoft Exchange Online through\\nGmail App on Android devices.\\n**Workaround**\\nAffected users can use Microsoft web app to access the Inbox.\",\"status\":\"SERVICE_INFORMATION\"}],\"most_recent_update\":{\"created\":\"2026-05-22T18:57:24+00:00\",\"modified\":\"2026-05-22T18:57:41+00:00\",\"when\":\"2026-05-22T18:57:24+00:00\",\"text\":\"**Summary**\\n[Resolved] Gmail Android users utilizing Microsoft Exchange Online accounts experienced authentication and synchronization failures.\\n**Description**\\nThe issue with Gmail has been resolved in the new Gmail android release (version 2026.05.11) as of Thursday, 2026-05-21 15:00 PDT.\\nOur engineering team identified the issue and released a new version of the Gmail Android application to the Google Play Store, which contains a fix to unblock affected users.\\nIf you are currently encountering this issue, please upgrade to the latest version of the Gmail Android app. Users operating on the new app version will be able to successfully log in and sync their accounts.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms**\\nImpacted users might have experienced authentication and sync failures while using Microsoft Exchange Online accounts through Gmail App on Android devices.\\n**Workaround**\\nUpdate to the latest version of Gmail Android (2026.05.11) in Google Play Store.\",\"status\":\"AVAILABLE\"},\"status_impact\":\"SERVICE_INFORMATION\",\"severity\":\"low\",\"service_key\":\"prPt2Yra2CbGsbEm9cpC\",\"service_name\":\"Gmail\",\"affected_products\":[{\"title\":\"Gmail\",\"id\":\"prPt2Yra2CbGsbEm9cpC\"}],\"uri\":\"incidents/kWSuRqLdsXGzT7Qxjdmz\"},{\"id\":\"1hr418kjGyBZ9ugXy3W3\",\"number\":\"5513210214958819667\",\"begin\":\"2026-05-04T16:40:00+00:00\",\"created\":\"2026-05-05T23:35:49+00:00\",\"end\":\"2026-05-08T18:53:00+00:00\",\"modified\":\"2026-05-08T18:58:23+00:00\",\"external_desc\":\"**Summary**\\nInvestigating elevated error rates with Gemini conversations.\\n**Description**\\nMitigation work is currently underway by our engineering team.\\nWe do not have an ETA for mitigation at this point.\\nWe will provide more information by Tuesday, 2026-05-05 20:30 PDT.\\n**Symptoms**\\nAffected customers are experiencing \\\"Something went wrong(13)\\\" error message in their Gemini conversations.\\n**Workaround**\\nNone at this time.\",\"updates\":[{\"created\":\"2026-05-08T18:58:09+00:00\",\"modified\":\"2026-05-08T18:58:23+00:00\",\"when\":\"2026-05-08T18:58:09+00:00\",\"text\":\"**Summary**\\nGemini conversations may have experienced elevated error rates.\\n**Description**\\nThe issue with Gemini has been resolved for all affected users as of Friday, 2026-05-08 11:53 PDT.\\nThe issue was triggered due to a combination of insufficient database resources, and application bugs. The database capacity issues were resolved, and further code changes are being made to mitigate the application issue.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms**\\nAffected customers may have experienced \\\"Something went wrong(13)\\\" error message in their Gemini conversations.\\n**Workaround**\\nNone at this time.\",\"status\":\"AVAILABLE\"},{\"created\":\"2026-05-08T15:53:13+00:00\",\"modified\":\"2026-05-08T18:58:09+00:00\",\"when\":\"2026-05-08T15:53:13+00:00\",\"text\":\"**Summary**\\nGemini conversations may experience elevated error rates.\\n**Description**\\nOur engineering team has isolated the underlying issue and is in the process of reverting the specific code deployments that triggered the impact.\\nWe are actively monitoring the progress of the rollback, alongside system health indicators, to confirm that the restoration is proceeding effectively.\\nWe will provide more information by Friday, 2026-05-08 12:00 PDT.\\n**Symptoms**\\nAffected customers may encounter \\\"Something went wrong(13)\\\" error message in their Gemini conversations.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_INFORMATION\"},{\"created\":\"2026-05-08T10:23:42+00:00\",\"modified\":\"2026-05-08T15:53:13+00:00\",\"when\":\"2026-05-08T10:23:42+00:00\",\"text\":\"**Summary**\\nGemini conversations may experience elevated error rates.\\n**Description**\\nOur engineering team has isolated the underlying issue and is in the process of reverting the specific code deployments that triggered the impact.\\nWe are actively monitoring the progress of the rollback, alongside system health indicators, to confirm that the restoration is proceeding effectively.\\nWe will provide more information by Friday, 2026-05-08 09:00 PDT.\\n**Symptoms**\\nAffected customers may encounter \\\"Something went wrong(13)\\\" error message in their Gemini conversations.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_INFORMATION\"},{\"created\":\"2026-05-08T07:29:25+00:00\",\"modified\":\"2026-05-08T10:23:42+00:00\",\"when\":\"2026-05-08T07:29:25+00:00\",\"text\":\"**Summary**\\nGemini conversations may experience elevated error rates.\\n**Description**\\nOur engineering team has isolated the underlying issue and is in the process of reverting the specific code deployments that triggered the impact.\\nWe are actively monitoring the progress of the rollback, alongside system health indicators, to confirm that the restoration is proceeding effectively.\\nWe will provide more information by Friday, 2026-05-08 04:00 PDT.\\n**Symptoms**\\nAffected customers may encounter \\\"Something went wrong(13)\\\" error message in their Gemini conversations.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_INFORMATION\"},{\"created\":\"2026-05-08T05:16:23+00:00\",\"modified\":\"2026-05-08T07:29:25+00:00\",\"when\":\"2026-05-08T05:16:23+00:00\",\"text\":\"**Summary**\\nGemini conversations are experiencing elevated error rates.\\n**Description**\\nOur engineering team has determined that further investigation is required to mitigate the issue.\\nWe do not have an ETA for mitigation at this point.\\nWe will provide more information by Friday, 2026-05-08 01:00 PDT.\\n**Symptoms**\\nAffected customers may encounter \\\"Something went wrong(13)\\\" error message in their Gemini conversations.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_INFORMATION\"},{\"created\":\"2026-05-08T01:00:01+00:00\",\"modified\":\"2026-05-08T05:16:23+00:00\",\"when\":\"2026-05-08T01:00:01+00:00\",\"text\":\"**Summary**\\nGemini conversations may have experienced elevated error rates.\\n**Description**\\nThe issue with Gemini has been resolved for all affected users as of Thursday, 2026-05-07 17:00 PDT.\\nThe issue was triggered due to a combination of insufficient database resources, and application issues. The database capacity issues were resolved, and further code changes are being made to mitigate the application issues.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms**\\nAffected customers may have experienced the \\\"Something went wrong(13)\\\" error message in their Gemini conversations.\",\"status\":\"SERVICE_INFORMATION\"},{\"created\":\"2026-05-07T23:52:49+00:00\",\"modified\":\"2026-05-08T01:00:01+00:00\",\"when\":\"2026-05-07T23:52:49+00:00\",\"text\":\"**Summary**\\nGemini conversations are experiencing elevated error rates.\\n**Description**\\nOur engineering team has determined that further investigation is required to mitigate the issue.\\nWe do not have an ETA for mitigation at this point.\\nWe will provide more information by Thursday, 2026-05-07 20:30 PDT.\\n**Symptoms**\\nAffected customers are experiencing \\\"Something went wrong(13)\\\" error message in their Gemini conversations.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_INFORMATION\"},{\"created\":\"2026-05-06T19:11:02+00:00\",\"modified\":\"2026-05-07T23:52:49+00:00\",\"when\":\"2026-05-06T19:11:02+00:00\",\"text\":\"**Summary**\\nGemini conversations are experiencing elevated error rates.\\n**Description**\\nOur engineering team has determined that further investigation is required to mitigate the issue.\\nWe do not have an ETA for mitigation at this point.\\nWe will provide more information by Thursday, 2026-05-07 17:00 PDT.\\n**Symptoms**\\nAffected customers are experiencing \\\"Something went wrong(13)\\\" error message in their Gemini conversations.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_INFORMATION\"},{\"created\":\"2026-05-06T03:22:40+00:00\",\"modified\":\"2026-05-06T19:11:02+00:00\",\"when\":\"2026-05-06T03:22:40+00:00\",\"text\":\"**Summary**\\nInvestigating elevated error rates with Gemini conversations.\\n**Description**\\nMitigation work is currently underway by our engineering team.\\nWe do not have an ETA for mitigation at this point.\\nWe will provide more information by Wednesday, 2026-05-06 09:00 PDT.\\n**Symptoms**\\nAffected customers are experiencing \\\"Something went wrong(13)\\\" error message in their Gemini conversations.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_INFORMATION\"},{\"created\":\"2026-05-05T23:35:49+00:00\",\"modified\":\"2026-05-06T03:22:40+00:00\",\"when\":\"2026-05-05T23:35:49+00:00\",\"text\":\"**Summary**\\nInvestigating elevated error rates with Gemini conversations.\\n**Description**\\nMitigation work is currently underway by our engineering team.\\nWe do not have an ETA for mitigation at this point.\\nWe will provide more information by Tuesday, 2026-05-05 20:30 PDT.\\n**Symptoms**\\nAffected customers are experiencing \\\"Something went wrong(13)\\\" error message in their Gemini conversations.\\n**Workaround**\\nNone at this time.\",\"status\":\"SERVICE_INFORMATION\"}],\"most_recent_update\":{\"created\":\"2026-05-08T18:58:09+00:00\",\"modified\":\"2026-05-08T18:58:23+00:00\",\"when\":\"2026-05-08T18:58:09+00:00\",\"text\":\"**Summary**\\nGemini conversations may have experienced elevated error rates.\\n**Description**\\nThe issue with Gemini has been resolved for all affected users as of Friday, 2026-05-08 11:53 PDT.\\nThe issue was triggered due to a combination of insufficient database resources, and application bugs. The database capacity issues were resolved, and further code changes are being made to mitigate the application issue.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms**\\nAffected customers may have experienced \\\"Something went wrong(13)\\\" error message in their Gemini conversations.\\n**Workaround**\\nNone at this time.\",\"status\":\"AVAILABLE\"},\"status_impact\":\"SERVICE_INFORMATION\",\"severity\":\"low\",\"service_key\":\"npdyhgECDJ6tB66MxXyo\",\"service_name\":\"Gemini\",\"affected_products\":[{\"title\":\"Gemini\",\"id\":\"npdyhgECDJ6tB66MxXyo\"}],\"uri\":\"incidents/1hr418kjGyBZ9ugXy3W3\"},{\"id\":\"GDxbSoojBb9yyEhU2Cua\",\"number\":\"5147219210877965776\",\"begin\":\"2026-04-29T18:30:00+00:00\",\"created\":\"2026-04-29T19:17:28+00:00\",\"end\":\"2026-04-29T20:45:00+00:00\",\"modified\":\"2026-04-29T21:07:19+00:00\",\"external_desc\":\"**Title:**\\nInvestigating issues with Classroom.\\n**Description:**\\nWe are experiencing an issue with Classroom beginning at Wednesday, 2026-04-29 11:30 PDT.\\nOur engineering team continues to investigate the issue.\\nWe will provide more information by Wednesday, 2026-04-29 13:00 PDT.\\n**Symptoms:**\\nAffected users may experience issues accessing classroom.\\n**Workaround:**\\nNo workarounds available at this time.\",\"updates\":[{\"created\":\"2026-04-29T21:07:19+00:00\",\"modified\":\"2026-04-29T21:07:20+00:00\",\"when\":\"2026-04-29T21:07:19+00:00\",\"text\":\"**Title:**\\nAn issue with Classroom is resolved.\\n**Description:**\\nA fix has been deployed, and services have stabilized for all users.\\nThe issue with Google Classroom has been resolved for all affected users as of Wednesday, 2026-04-29 13:45 PDT.\\nThe issue was due to an internal software migration. Our engineers have mitigated the issue by rolling back the migration for impacted users.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms:**\\nAffected users may have experienced issues accessing the classroom.\\n**Workaround:**\\nThe issue is resolved.\",\"status\":\"AVAILABLE\"},{\"created\":\"2026-04-29T20:13:38+00:00\",\"modified\":\"2026-04-29T21:07:19+00:00\",\"when\":\"2026-04-29T20:13:38+00:00\",\"text\":\"**Title:**\\nInvestigating issues with Classroom.\\n**Description:**\\nA fix has been deployed, and services have stabilized for the majority of users.\\nWe are aware that a small subset of users may still encounter errors. Our engineering team is actively working on a secondary resolution to address these remaining users.\\nFull resolution for remaining users is expected to complete by Wednesday, 2026-04-29 14:00 PDT.\\nWe will provide an update by Wednesday, 2026-04-29 14:15 PDT with current details.\\n**Symptoms:**\\nAffected users may experience issues accessing classroom.\\n**Workaround:**\\nNo workarounds available at this time.\",\"status\":\"SERVICE_INFORMATION\"},{\"created\":\"2026-04-29T19:53:46+00:00\",\"modified\":\"2026-04-29T20:13:38+00:00\",\"when\":\"2026-04-29T19:53:46+00:00\",\"text\":\"**Title:**\\nInvestigating issues with Classroom.\\n**Description:**\\nOur engineering team has identified the cause and working on mitigation of the issue .\\nThe mitigation is expected to complete by Wednesday, 2026-04-29 15:00 PDT.\\nWe will provide more information by Wednesday, 2026-04-29 15:15 PDT.\\n**Symptoms:**\\nAffected users may experience issues accessing classroom.\\n**Workaround:**\\nNo workarounds available at this time.\",\"status\":\"SERVICE_INFORMATION\"},{\"created\":\"2026-04-29T19:17:28+00:00\",\"modified\":\"2026-04-29T19:53:46+00:00\",\"when\":\"2026-04-29T19:17:28+00:00\",\"text\":\"**Title:**\\nInvestigating issues with Classroom.\\n**Description:**\\nWe are experiencing an issue with Classroom beginning at Wednesday, 2026-04-29 11:30 PDT.\\nOur engineering team continues to investigate the issue.\\nWe will provide more information by Wednesday, 2026-04-29 13:00 PDT.\\n**Symptoms:**\\nAffected users may experience issues accessing classroom.\\n**Workaround:**\\nNo workarounds available at this time.\",\"status\":\"SERVICE_DISRUPTION\"}],\"most_recent_update\":{\"created\":\"2026-04-29T21:07:19+00:00\",\"modified\":\"2026-04-29T21:07:20+00:00\",\"when\":\"2026-04-29T21:07:19+00:00\",\"text\":\"**Title:**\\nAn issue with Classroom is resolved.\\n**Description:**\\nA fix has been deployed, and services have stabilized for all users.\\nThe issue with Google Classroom has been resolved for all affected users as of Wednesday, 2026-04-29 13:45 PDT.\\nThe issue was due to an internal software migration. Our engineers have mitigated the issue by rolling back the migration for impacted users.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms:**\\nAffected users may have experienced issues accessing the classroom.\\n**Workaround:**\\nThe issue is resolved.\",\"status\":\"AVAILABLE\"},\"status_impact\":\"SERVICE_INFORMATION\",\"severity\":\"low\",\"service_key\":\"3GoNVrVz48jeNZSyLJMV\",\"service_name\":\"Classroom\",\"affected_products\":[{\"title\":\"Classroom\",\"id\":\"3GoNVrVz48jeNZSyLJMV\"}],\"uri\":\"incidents/GDxbSoojBb9yyEhU2Cua\"},{\"id\":\"tVLvwJSkYhEtcLAMsekv\",\"number\":\"6658602565448188991\",\"begin\":\"2026-04-28T07:30:00+00:00\",\"created\":\"2026-04-28T15:20:36+00:00\",\"end\":\"2026-04-28T15:50:00+00:00\",\"modified\":\"2026-04-28T16:50:51+00:00\",\"external_desc\":\"**Title:**\\nInvestigating issues with slow loading of AppSheet\\n**Description:**\\nWe are experiencing an issue with AppSheet beginning at Tuesday, 2026-04-28 00:30 PDT.\\nOur engineering team continues to investigate the issue.\\nWe will provide an update by Tuesday, 2026-04-28 09:00 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Symptoms:**\\nAffected customers would experience slow loading times and internal server errors for AppSheet apps.\\n**Workaround:**\\nNone at this time.\",\"updates\":[{\"created\":\"2026-04-28T16:50:32+00:00\",\"modified\":\"2026-04-28T16:50:51+00:00\",\"when\":\"2026-04-28T16:50:32+00:00\",\"text\":\"**Title:**\\nMitigated issues with slow loading of AppSheet apps.\\n**Description:**\\nThe issue with AppSheet has been resolved for all affected users as of Tuesday, 2026-04-28 08:50 PDT.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms:**\\nAffected customers would have experienced slow loading times and internal server errors for AppSheet apps.\",\"status\":\"AVAILABLE\"},{\"created\":\"2026-04-28T16:00:44+00:00\",\"modified\":\"2026-04-28T16:50:32+00:00\",\"when\":\"2026-04-28T16:00:44+00:00\",\"text\":\"**Title:**\\nInvestigating issues with slow loading of AppSheet.\\n**Description:**\\nOur engineering team has implemented a fix and system telemetry indicates that latency and error rates have returned to normal levels. We are continuing to monitor the environment closely to ensure full stability.\\nWe will provide an update by Tuesday, 2026-04-28 10:30 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Symptoms:**\\nAffected customers would experience slow loading times and internal server errors for AppSheet apps.\\n**Workaround:**\\nNone at this time.\",\"status\":\"SERVICE_INFORMATION\"},{\"created\":\"2026-04-28T15:20:36+00:00\",\"modified\":\"2026-04-28T16:00:44+00:00\",\"when\":\"2026-04-28T15:20:36+00:00\",\"text\":\"**Title:**\\nInvestigating issues with slow loading of AppSheet\\n**Description:**\\nWe are experiencing an issue with AppSheet beginning at Tuesday, 2026-04-28 00:30 PDT.\\nOur engineering team continues to investigate the issue.\\nWe will provide an update by Tuesday, 2026-04-28 09:00 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Symptoms:**\\nAffected customers would experience slow loading times and internal server errors for AppSheet apps.\\n**Workaround:**\\nNone at this time.\",\"status\":\"SERVICE_INFORMATION\"}],\"most_recent_update\":{\"created\":\"2026-04-28T16:50:32+00:00\",\"modified\":\"2026-04-28T16:50:51+00:00\",\"when\":\"2026-04-28T16:50:32+00:00\",\"text\":\"**Title:**\\nMitigated issues with slow loading of AppSheet apps.\\n**Description:**\\nThe issue with AppSheet has been resolved for all affected users as of Tuesday, 2026-04-28 08:50 PDT.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Symptoms:**\\nAffected customers would have experienced slow loading times and internal server errors for AppSheet apps.\",\"status\":\"AVAILABLE\"},\"status_impact\":\"SERVICE_INFORMATION\",\"severity\":\"low\",\"service_key\":\"FWjKi5U7KX4FUUPThHAJ\",\"service_name\":\"AppSheet\",\"affected_products\":[{\"title\":\"AppSheet\",\"id\":\"FWjKi5U7KX4FUUPThHAJ\"}],\"uri\":\"incidents/tVLvwJSkYhEtcLAMsekv\"},{\"id\":\"JTgxXCtXTaw3aEVzUUfa\",\"number\":\"3709370212894413327\",\"begin\":\"2026-04-10T02:08:00+00:00\",\"created\":\"2026-04-10T10:07:12+00:00\",\"end\":\"2026-04-10T14:50:00+00:00\",\"modified\":\"2026-04-10T15:06:53+00:00\",\"external_desc\":\"**Summary**\\nGoogle Workspace users using Google IdP are unable to sign into their accounts using security keys and hardware based passkeys.\\n**Description**\\nWe are experiencing an issue with Workspace users beginning on Thursday, 2026-04-09 19:08 PDT.\\nOur engineering team continues to investigate the issue.\\nWe will provide an update by Friday, 2026-04-10 04:00 PDT with current details.\\nWe apologize to all who are affected by the disruption.\\n**Customer Symptoms**\\nGoogle Workspace users using Google IdP are unable to sign into their accounts using security keys and hardware based passkeys.\\n**Workaround**\\nBelow steps should be performed by Admin as the workaround:\\n- Admin should revoke the key.\\n- Admin to create the verification code to sign-in.\\n- The user should sign-in using Backup verification code and then re-enrolls the security key.\",\"updates\":[{\"created\":\"2026-04-10T15:06:51+00:00\",\"modified\":\"2026-04-10T15:06:53+00:00\",\"when\":\"2026-04-10T15:06:51+00:00\",\"text\":\"**Summary**\\nGoogle Workspace users using Google IdP were unable to sign into their accounts using security keys and hardware based passkeys.\\n**Description**\\nThe issue with Admin Console has been resolved for all affected users as of Friday, 2026-04-10 07:50 PDT.\\nSuspected CL have been rolled back and we are seeing in our graphs errors rate have gone down. Customers also confirmed that issues have been mitigated.\\nWe thank you for your patience while we worked on resolving the issue.\\n**Customer Symptoms**\\nGoogle Workspace users using Google IdP were unable to sign into their accounts using security keys and hardware based passkeys.\\n**Workaround**\\nBelow steps should be performed by Admin as the workaround:\\n* Admin should revoke the key.\\n* Admin to create the verification code to sign-in.\\n* The user should sign-in using Backup verification code and then re-enrolls the security key.\",\"status\":\"AVAILABLE\"},{\"created\":\"2026-04-10T11:27:56+00:00\",\"modified\":\"2026-04-10T15:06:51+00:00\",\"when\":\"2026-04-10T11:27:56+00:00\",\"text\":\"**Summary**\\nGoogle Workspace users using Google IdP are unable to sign into their accounts using security keys and hardware based passkeys.\\n**Description**\\nMitigation work is currently underway by our engineering team.\\nWe do not have an ETA for mitigation at this point.\\nWe will provide more information by Friday, 2026-04-10 09:00 PDT.\\n**Customer Symptoms**\\nGoogle Workspace users using Google IdP are unable to sign into their accounts using security keys and hardware based passkeys.\\n**Workaround**\\nBelow steps should be performed by Admin as the workaround:\\n- Admin should revoke the key.\\n- Admin to create the verification code to sign-in.\\n- The user should sign-in using Backup verification code and then re-enrolls the security key.\",\"status\":\"SERVICE_DISRUPTION\"},{\"created\":\"2026-04-10T10:55:50+00:00\",\"modified\":\"2026-04-10T11:27:56+00:00\",\"when\":\"2026-04-10T10:55:50+00:00\",\"text\":\"**Summary**\\nGoogle Workspace users using Goog
[...truncated]
Sub-agent trace (toolu_01FYfmMV6r2KJBtHG3c86jk4, 3 events)
tools_started web_fetch t=117195.205
Inner payload
{
"tool_name": "web_fetch",
"tool_input": {
"brief": "Incident history JSON for past 14 days",
"max_age_seconds": 3600,
"url": "https://www.google.com/appsstatus/dashboard/incidents.json"
},
"dispatch_id": "toolu_01FYfmMV6r2KJBtHG3c86jk4",
"parent_dispatch_id": "",
"handle": "",
"panel_kind": "web_fetch"
}tools_progress web_fetch t=117195.206
Inner payload
{
"tool_name": "web_fetch",
"dispatch_id": "toolu_01FYfmMV6r2KJBtHG3c86jk4",
"status": "running",
"result": null,
"error": "",
"elapsed": null,
"fields": {
"progress": {
"message": "Incident history JSON for past 14 days",
"metadata": {
"browser_chain": false,
"url": "https://www.google.com/appsstatus/dashboard/incidents.json"
}
},
"status": "running",
"updatedAt": 1789168571430
}
}tools_completed web_fetch t=117195.207
Inner payload
{
"tool_name": "web_fetch",
"dispatch_id": "toolu_01FYfmMV6r2KJBtHG3c86jk4",
"status": "completed",
"result": "{\"content\":\"[{\\\"id\\\":\\\"fJ4691Pe2g9KsTbP8g5F\\\",\\\"number\\\":\\\"11425782758836097339\\\",\\\"begin\\\":\\\"2026-09-11T13:00:00+00:00\\\",\\\"created\\\":\\\"2026-09-11T20:45:19+00:00\\\",\\\"end\\\":\\\"2026-09-11T20:22:00+00:00\\\",\\\"modified\\\":\\\"2026-09-11T20:45:42+00:00\\\",\\\"external_desc\\\":\\\"**Summary:**\\\\nMitigated intermittent elevated error rates with Gemini 3.1 Pro models on Gemini Web and Gemini App.\\\\n**Description:**\\\\nBetween 6:00 PDT and 13:22 PDT on 11 September 2026, a subset of Gemini users may have experienced elevated error rates with Gemini 3.1 Pro models on Gemini Web and Gemini App.\\\\nFrom preliminary analysis, the issue was triggered by a recent rollout. Our engineers identified the cause and mitigated it with a code change.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms:**\\\\nCustomers may have encountered intermittent errors when using Gemini 3.1 Pro models on Gemini Web and Gemini App.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-09-11T20:45:19+00:00\\\",\\\"modified\\\":\\\"2026-09-11T20:45:42+00:00\\\",\\\"when\\\":\\\"2026-09-11T20:45:19+00:00\\\",\\\"text\\\":\\\"**Summary:**\\\\nMitigated intermittent elevated error rates with Gemini 3.1 Pro models on Gemini Web and Gemini App.\\\\n**Description:**\\\\nBetween 6:00 PDT and 13:22 PDT on 11 September 2026, a subset of Gemini users may have experienced elevated error rates with Gemini 3.1 Pro models on Gemini Web and Gemini App.\\\\nFrom preliminary analysis, the issue was triggered by a recent rollout. Our engineers identified the cause and mitigated it with a code change.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms:**\\\\nCustomers may have encountered intermittent errors when using Gemini 3.1 Pro models on Gemini Web and Gemini App.\\\",\\\"status\\\":\\\"AVAILABLE\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-09-11T20:45:19+00:00\\\",\\\"modified\\\":\\\"2026-09-11T20:45:42+00:00\\\",\\\"when\\\":\\\"2026-09-11T20:45:19+00:00\\\",\\\"text\\\":\\\"**Summary:**\\\\nMitigated intermittent elevated error rates with Gemini 3.1 Pro models on Gemini Web and Gemini App.\\\\n**Description:**\\\\nBetween 6:00 PDT and 13:22 PDT on 11 September 2026, a subset of Gemini users may have experienced elevated error rates with Gemini 3.1 Pro models on Gemini Web and Gemini App.\\\\nFrom preliminary analysis, the issue was triggered by a recent rollout. Our engineers identified the cause and mitigated it with a code change.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms:**\\\\nCustomers may have encountered intermittent errors when using Gemini 3.1 Pro models on Gemini Web and Gemini App.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_INFORMATION\\\",\\\"severity\\\":\\\"low\\\",\\\"service_key\\\":\\\"npdyhgECDJ6tB66MxXyo\\\",\\\"service_name\\\":\\\"Gemini\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Gemini\\\",\\\"id\\\":\\\"npdyhgECDJ6tB66MxXyo\\\"}],\\\"uri\\\":\\\"incidents/fJ4691Pe2g9KsTbP8g5F\\\"},{\\\"id\\\":\\\"y2ufhCB886p7rveYNnko\\\",\\\"number\\\":\\\"1967299596760361422\\\",\\\"begin\\\":\\\"2026-08-28T17:50:00+00:00\\\",\\\"created\\\":\\\"2026-08-28T19:25:37+00:00\\\",\\\"end\\\":\\\"2026-08-28T20:34:00+00:00\\\",\\\"modified\\\":\\\"2026-09-04T03:43:27+00:00\\\",\\\"external_desc\\\":\\\"**Summary:**\\\\nGoogle Chat users are experiencing an issue where they receive notifications for new messages, but the actual messages fail to appear in their chat application unless the user refreshes their view.\\\\n**Description:**\\\\nWe are experiencing an intermittent issue with Google Chat Friday, 2026-08-28 10:25 PDT.\\\\nOur engineering team has investigated the issue and are currently working on mitigating the issue.\\\\nWe will provide more information by Friday, 2026-08-28 14:15 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Customer Symptoms:**\\\\nUsers are experiencing an issue where they receive notifications for new messages, but the actual messages fail to appear in their chat application unless the user refreshes their view\\\\n**Workaround:**\\\\nUsers can refresh to see new messages\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-09-04T03:43:07+00:00\\\",\\\"modified\\\":\\\"2026-09-04T03:43:27+00:00\\\",\\\"when\\\":\\\"2026-09-04T03:43:07+00:00\\\",\\\"text\\\":\\\"## Incident Report\\\\n## Summary\\\\nOn Friday, August 28, 2026, between 10:50 US/Pacific and 13:34 US/Pacific for a duration of 2 hours and 44 minutes, Google Chat users experienced delays in receiving real-time incoming messages within active conversation streams across web, desktop, and mobile applications. While users continued to receive notifications for new messages, the message contents did not automatically render in active conversation streams until the user manually refreshed the application or sent a reply message.\\\\nTo our valued Google Workspace customers, we are deeply sorry for the service interruption you experienced. We are taking immediate and decisive action to address the issue and strengthen our systems against future incidents.\\\\n## Root Cause\\\\nThe incident was triggered during a global database schema rollout. An incompatible database schema change caused database writes required for message delivery to start failing.\\\\nSpecifically, a mechanism for schema synchronization—designed to ensure high consistency between primary storage and live message delivery—encountered a rare race condition bug. This caused backend message dispatchers to attempt writes against an unready schema state, leading to write failures.\\\\nBecause live message delivery had an active dependency on these writes completing successfully before pushing events to user screens, the failure caused incoming messages to experience synchronization delays. Separate push notification services and core message storage operated independently and were unaffected, which is why users still received alerts for new messages while active conversation streams experienced delays.\\\\n## Remediation and Prevention\\\\nThe issue was detected through automated backend alerting on dispatch error rates, alongside corresponding customer reports. Google Engineering initiated parallel workstreams to restore service.\\\\n* At 12:44 US/Pacific, Google Engineering directly mitigated the source of the issue by removing the dependency that live message delivery had on the failing database write. This allowed new incoming messages to bypass the failed write and immediately resume normal real-time delivery to user screens.\\\\n* At 13:00 US/Pacific, Engineering halted and reverted the database schema change to its previous verified stable version, restoring full database compatibility.\\\\n* By 13:34 US/Pacific, as queued messages were processed, real-time message synchronization delays returned to 0% across all customer domains. Engineering verified that full service health was restored with zero message loss.\\\\nTo prevent this from happening again, we are executing the following preventative actions:\\\\n* **Hardened internal detection:** Strengthen automated validation and verification to cover this specific failure scenario across pre-production and internal rollout stages. This aims to catch and mitigate schema incompatibilities and synchronization edge cases internally before any customer traffic can be impacted in the future.\\\\n* **Resilience Against Race Conditions:** Implement Architectural improvements to make the schema synchronization and message delivery architecture resilient to race conditions. This aims to ensure that background schema and config synchronization inconsistencies cannot disrupt real-time message delivery.\\\\n \\\\n## Detailed Description of Impact\\\\nThe incident impact window occurred between 10:50 US/Pacific and 13:34 US/Pacific on Friday, 28 August 2026\\\\\\\\. This issue applied globally to all Google Chat environments.\\\\n**Google Chat:**\\\\n* **Delayed message rendering:** New messages did not appear in active conversation streams automatically. Users were required to manually refresh their client or send a new message/react to force the stream to synchronize.\\\\n* **Stalled read states and inbox sorting:** Background actions, such as updating unread badges and moving active spaces to the top of the inbox, experienced heavy internal error rates and were significantly delayed for users.\\\\n* **Search indexing delays:** New messages sent during the outage window stalled in the indexing pipeline, experiencing delays in becoming searchable.\\\\n**Scope Exclusions:**\\\\n* **No Data Loss:** At no point was customer data lost or compromised. All messages transmitted during the incident window were securely captured in primary storage and successfully delivered to conversation streams once the backlog cleared.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-08-28T20:55:51+00:00\\\",\\\"modified\\\":\\\"2026-09-04T03:43:07+00:00\\\",\\\"when\\\":\\\"2026-08-28T20:55:51+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGoogle Chat users experienced an issue where they received notifications for new messages, but the actual messages failed to appear in their chat application unless the user refreshed their view.\\\\n**Description**\\\\nThe issue with Google Chat has been resolved for all affected users as of Friday, 2026-08-28 13:45 PDT.\\\\nFrom preliminary analysis the issue was triggered due to a recent change which was rolled back to mitigate the impact.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Customer Symptoms**\\\\nUsers were experiencing an issue where they received notifications for new messages, but the actual messages failed to appear in their chat application unless the user refreshed their view.\\\\n**Workaround**\\\\nNone as the issue is resolved.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-08-28T19:25:37+00:00\\\",\\\"modified\\\":\\\"2026-08-28T20:55:51+00:00\\\",\\\"when\\\":\\\"2026-08-28T19:25:37+00:00\\\",\\\"text\\\":\\\"**Summary:**\\\\nGoogle Chat users are experiencing an issue where they receive notifications for new messages, but the actual messages fail to appear in their chat application unless the user refreshes their view.\\\\n**Description:**\\\\nWe are experiencing an intermittent issue with Google Chat Friday, 2026-08-28 10:25 PDT.\\\\nOur engineering team has investigated the issue and are currently working on mitigating the issue.\\\\nWe will provide more information by Friday, 2026-08-28 14:15 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Customer Symptoms:**\\\\nUsers are experiencing an issue where they receive notifications for new messages, but the actual messages fail to appear in their chat application unless the user refreshes their view\\\\n**Workaround:**\\\\nUsers can refresh to see new messages\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-09-04T03:43:07+00:00\\\",\\\"modified\\\":\\\"2026-09-04T03:43:27+00:00\\\",\\\"when\\\":\\\"2026-09-04T03:43:07+00:00\\\",\\\"text\\\":\\\"## Incident Report\\\\n## Summary\\\\nOn Friday, August 28, 2026, between 10:50 US/Pacific and 13:34 US/Pacific for a duration of 2 hours and 44 minutes, Google Chat users experienced delays in receiving real-time incoming messages within active conversation streams across web, desktop, and mobile applications. While users continued to receive notifications for new messages, the message contents did not automatically render in active conversation streams until the user manually refreshed the application or sent a reply message.\\\\nTo our valued Google Workspace customers, we are deeply sorry for the service interruption you experienced. We are taking immediate and decisive action to address the issue and strengthen our systems against future incidents.\\\\n## Root Cause\\\\nThe incident was triggered during a global database schema rollout. An incompatible database schema change caused database writes required for message delivery to start failing.\\\\nSpecifically, a mechanism for schema synchronization—designed to ensure high consistency between primary storage and live message delivery—encountered a rare race condition bug. This caused backend message dispatchers to attempt writes against an unready schema state, leading to write failures.\\\\nBecause live message delivery had an active dependency on these writes completing successfully before pushing events to user screens, the failure caused incoming messages to experience synchronization delays. Separate push notification services and core message storage operated independently and were unaffected, which is why users still received alerts for new messages while active conversation streams experienced delays.\\\\n## Remediation and Prevention\\\\nThe issue was detected through automated backend alerting on dispatch error rates, alongside corresponding customer reports. Google Engineering initiated parallel workstreams to restore service.\\\\n* At 12:44 US/Pacific, Google Engineering directly mitigated the source of the issue by removing the dependency that live message delivery had on the failing database write. This allowed new incoming messages to bypass the failed write and immediately resume normal real-time delivery to user screens.\\\\n* At 13:00 US/Pacific, Engineering halted and reverted the database schema change to its previous verified stable version, restoring full database compatibility.\\\\n* By 13:34 US/Pacific, as queued messages were processed, real-time message synchronization delays returned to 0% across all customer domains. Engineering verified that full service health was restored with zero message loss.\\\\nTo prevent this from happening again, we are executing the following preventative actions:\\\\n* **Hardened internal detection:** Strengthen automated validation and verification to cover this specific failure scenario across pre-production and internal rollout stages. This aims to catch and mitigate schema incompatibilities and synchronization edge cases internally before any customer traffic can be impacted in the future.\\\\n* **Resilience Against Race Conditions:** Implement Architectural improvements to make the schema synchronization and message delivery architecture resilient to race conditions. This aims to ensure that background schema and config synchronization inconsistencies cannot disrupt real-time message delivery.\\\\n \\\\n## Detailed Description of Impact\\\\nThe incident impact window occurred between 10:50 US/Pacific and 13:34 US/Pacific on Friday, 28 August 2026\\\\\\\\. This issue applied globally to all Google Chat environments.\\\\n**Google Chat:**\\\\n* **Delayed message rendering:** New messages did not appear in active conversation streams automatically. Users were required to manually refresh their client or send a new message/react to force the stream to synchronize.\\\\n* **Stalled read states and inbox sorting:** Background actions, such as updating unread badges and moving active spaces to the top of the inbox, experienced heavy internal error rates and were significantly delayed for users.\\\\n* **Search indexing delays:** New messages sent during the outage window stalled in the indexing pipeline, experiencing delays in becoming searchable.\\\\n**Scope Exclusions:**\\\\n* **No Data Loss:** At no point was customer data lost or compromised. All messages transmitted during the incident window were securely captured in primary storage and successfully delivered to conversation streams once the backlog cleared.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_INFORMATION\\\",\\\"severity\\\":\\\"low\\\",\\\"service_key\\\":\\\"Ht9Vx5PFzDPHYvrnrvSk\\\",\\\"service_name\\\":\\\"Google Chat\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Google Chat\\\",\\\"id\\\":\\\"Ht9Vx5PFzDPHYvrnrvSk\\\"}],\\\"uri\\\":\\\"incidents/y2ufhCB886p7rveYNnko\\\"},{\\\"id\\\":\\\"sjDiph2NXGAfkYYLfK7T\\\",\\\"number\\\":\\\"9682992681334163277\\\",\\\"begin\\\":\\\"2026-08-18T23:58:00+00:00\\\",\\\"created\\\":\\\"2026-08-19T03:57:37+00:00\\\",\\\"end\\\":\\\"2026-08-19T12:40:00+00:00\\\",\\\"modified\\\":\\\"2026-08-24T16:31:00+00:00\\\",\\\"external_desc\\\":\\\"**Summary**\\\\nSome customers may experience slowness while using Google Drive.\\\\n**Description**\\\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\\\nOur engineering team is currently working on a potential fix while continuing to investigate the full root cause.\\\\nWe will provide an update by Tuesday, 2026-08-18 22:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nAffected customers may experience slowness while using Google Drive.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-08-24T16:31:00+00:00\\\",\\\"modified\\\":\\\"2026-08-24T16:31:00+00:00\\\",\\\"when\\\":\\\"2026-08-24T16:31:00+00:00\\\",\\\"text\\\":\\\"# Incident Report\\\\n## Summary\\\\nOn Tuesday, 18 August 2026 at 16:58 US/Pacific, an incident affecting Google Drive occurred for a duration of 12 hours and 42 minutes. Customers located in Australia, Philippines and New Zealand experienced severe latency, sporadic loading failures, and errors when accessing Google Drive. \\\\nTo our valued Google Workspace customers, we are deeply sorry for the service interruption you experienced. We are taking immediate and decisive action to address the issue and strengthen our systems against future incidents. Other Google Workspace services were not affected.\\\\n## Root Cause\\\\nGoogle utilizes a multi-tiered global load balancing and network routing architecture to serve traffic efficiently. To ensure fair use and prevent congestion, bandwidth allocation is managed by attributing traffic to specific downstream services.\\\\nA misconfiguration in the attribution of traffic caused the bandwidth usage to be incorrectly attributed to the frontend network load balancing tier itself, rather than to the Drive service.\\\\nThe sequence of events leading to customer impact was as follows:\\\\n* Traffic from the regional load balancing tier was incorrectly attributed to the load balancing infrastructure rather than the Drive service.\\\\n* Unrelated traffic caused network bandwidth usage attributed to the load balancing infrastructure in this region to exceed its quota.\\\\n* Network bandwidth management systems applied throttling to this traffic to enforce quotas, causing severe network congestion affecting Drive traffic in this region.\\\\n* This congestion led to increased latency, slowness, and error rates for users accessing Drive in the affected regions.\\\\n## Remediation and Prevention\\\\nGoogle engineers were alerted to the issue via automated monitoring and an influx of customer support cases, with initial incident response teams engaged by 17:21 US/Pacific on 18 August , 2026\\\\\\\\.\\\\nEngineers initially attempted to alleviate the congestion by draining regional frontend traffic to an alternate geographic location, but this proved ineffective and was reverted. The team then mitigated the issue by migrating regional traffic to an upgraded internal network routing architecture, which correctly set the traffic attribution for Drive.\\\\nNormal operations were fully restored at 05:40 once 100% of the regional traffic was migrated to the upgraded network architecture.\\\\nTo prevent a recurrence of this issue, our engineering teams are undertaking the following structural preventative actions:\\\\n* **Resilient Network Protocol Migration:** We are expediting the complete migration of all regional frontend traffic to the newer, resilient network routing protocol to ensure accurate bandwidth attribution under all congestion scenarios.\\\\n* **Delegation Root Cause Remediation:** We are addressing the underlying mechanism that caused the incorrect delegation and analyzing similar cases across the fleet to ensure it does not reoccur.\\\\n## Detailed Description of Impact\\\\nFrom 18 August 2026, 16:58 to 19 August 2026, 05:40 US/Pacific for a duration of 12 hours, 42 minutes, Google Drive users in Australia, Philippines and New Zealand experienced severe performance degradation, including high latency and errors accessing Drive or listing files via Editors. Service was fully restored as of 19 August, 05:40 US/Pacific.\\\\n**Google Drive:**\\\\n* Elevated Latency: Users experienced extreme slowness when loading the Drive interface or accessing files, with initial data retrieval times heavily delayed.\\\\n* Loading Failures and Errors: Many user requests timed out or returned HTTP 502 errors due to the aggressive network throttling dropping the connections.\\\\n**Scope Exclusion:**\\\\n* Customers accessing Google Drive from outside the Australia, Philippines and New Zealand regions were largely unaffected by this localized throttling event.\\\\n* Other Google Workspace applications not reliant on this specific routing path remained stable during the incident window.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-08-20T20:00:17+00:00\\\",\\\"modified\\\":\\\"2026-08-24T16:31:00+00:00\\\",\\\"when\\\":\\\"2026-08-20T20:00:17+00:00\\\",\\\"text\\\":\\\"# Preliminary Incident Report\\\\nWe sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Workspace, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future. \\\\nPlease note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. \\\\nIf you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article [**https://support.google.com/a/answer/1047213**](https://support.google.com/a/answer/1047213).\\\\n#[REDACTED] Date/Time of the Issue (All time US/Pacific)\\\\nIncident Start: 18 August 2026 16:58\\\\nIncident End: 19 August 2026 05:40\\\\nDuration: 12 hours, 42 minutes \\\\n## Summary\\\\nOn August 18, 2026 from 16:58 to August 19, 2026 05:40 US/Pacific, Drive customers in Australia, Philippines and New Zealand experienced degraded performance and slowness. During this period users also experienced 502 errors. Other Google Workspace services were not affected. \\\\n## Preliminary Root Cause\\\\nGoogle utilizes a multi-tiered global load balancing and network routing architecture to serve traffic efficiently. To ensure fair use and prevent congestion, bandwidth allocation is managed by attributing traffic to specific downstream services.\\\\nA misconfiguration in the attribution of traffic caused the bandwidth usage to be incorrectly attributed to the frontend network load balancing tier itself, rather than to the Drive service.\\\\nThe sequence of events leading to customer impact was as follows:\\\\n* Traffic from the regional load balancing tier was incorrectly attributed to the load balancing infrastructure rather than the Drive service.\\\\n* Unrelated traffic caused network bandwidth usage attributed to the load balancing infrastructure in this region to exceed its quota.\\\\n* Network bandwidth management systems applied throttling to this traffic to enforce quotas, causing severe network congestion affecting Drive traffic in this region.\\\\n* This congestion led to increased latency, slowness, and error rates for users accessing Drive in the affected regions.\\\\n## Remediation \\\\nGoogle engineers were alerted to the issue via automated monitoring and an influx of customer support cases, with initial incident response teams engaged by 17:21 US/Pacific on August 18, 2026\\\\\\\\.\\\\nEngineers initially attempted to alleviate the congestion by draining regional frontend traffic to an alternate geographic location, but this proved ineffective and was reverted. The team then mitigated the issue by migrating regional traffic to an upgraded internal network routing architecture, which correctly set the traffic attribution for Drive.\\\\nNormal operations were fully restored once 100% of the regional traffic was migrated to the upgraded network architecture.\\\\n## Description of Impact\\\\nBetween August 18, 2026, at 16:58 US/Pacific and August 19, 2026, at 05:40 US/Pacific, Google Drive users in Australia, Philippines and New Zealand experienced severe latency, degraded performance and 502 errors.\\\\nDuring this roughly 12-hour and 42-minute window, affected customers may have observed significant slowness or errors when attempting to access Google Drive or listing files through Editors (e.g. [docs.google.com](http://docs.google.com)), with initial load times significantly delayed. The issue was fully mitigated as of August 19, 2026, at 05:40 US/Pacific, and normal service has been restored.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-08-19T13:05:05+00:00\\\",\\\"modified\\\":\\\"2026-08-20T20:00:17+00:00\\\",\\\"when\\\":\\\"2026-08-19T13:05:05+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nMultiple customers in Australia and New Zealand may have experienced slowness or errors while accessing Google Drive.\\\\n**Description**\\\\nThe issue with Google Drive has been resolved for all affected customers as of Wednesday, 2026-08-19 05:40 PDT.\\\\nOur engineering teams determined the slowness was the result of a localized capacity and routing issue within our regional infrastructure.\\\\nTo restore service, we implemented a fix to alleviate the network congestion and accelerated a planned infrastructure upgrade in the affected region to ensure normal performance levels.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nAffected customers in Australia and New Zealand may have experienced slowness or 502 errors, with some users briefly unable to access Drive.\\\\n**Workaround**\\\\nThe issue is now mitigated\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-08-19T11:26:14+00:00\\\",\\\"modified\\\":\\\"2026-08-19T13:05:05+00:00\\\",\\\"when\\\":\\\"2026-08-19T11:26:14+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nMultiple customers in Australia and New Zealand may experience slowness or errors while accessing Google Drive.\\\\n**Description**\\\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\\\nOur engineering team is actively working on a mitigation strategy. There is no ETA for full resolution.\\\\nWe will provide an update by Wednesday, 2026-08-19 06:00 PDT with current details.\\\\n**Symptoms**\\\\nAffected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-08-19T09:54:28+00:00\\\",\\\"modified\\\":\\\"2026-08-19T11:26:14+00:00\\\",\\\"when\\\":\\\"2026-08-19T09:54:28+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nMultiple customers in Australia and New Zealand may experience slowness or errors while accessing Google Drive.\\\\n**Description**\\\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\\\nOur engineering team is actively working on a mitigation strategy. There is no ETA for full resolution.\\\\nWe will provide an update by Wednesday, 2026-08-19 04:30 PDT with current details.\\\\n**Symptoms**\\\\nAffected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-08-19T08:54:05+00:00\\\",\\\"modified\\\":\\\"2026-08-19T09:54:28+00:00\\\",\\\"when\\\":\\\"2026-08-19T08:54:05+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nMultiple customers in Australia and New Zealand may experience slowness or errors while accessing Google Drive.\\\\n**Description**\\\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\\\nOur engineering team is actively working on a mitigation strategy. There is no ETA for full resolution.\\\\nWe will provide an update by Wednesday, 2026-08-19 03:00 PDT with current details.\\\\n**Symptoms**\\\\nAffected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-08-19T07:59:51+00:00\\\",\\\"modified\\\":\\\"2026-08-19T08:54:05+00:00\\\",\\\"when\\\":\\\"2026-08-19T07:59:51+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nMultiple customers in Australia and New Zealand may experience slowness or errors while accessing Google Drive.\\\\n**Description**\\\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\\\nOur engineering team has identified the root cause of the issue and is actively working on a mitigation strategy. There is no ETA for full resolution.\\\\nWe will provide an update by Wednesday, 2026-08-19 02:00 PDT with current details.\\\\n**Symptoms**\\\\nAffected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-08-19T06:49:41+00:00\\\",\\\"modified\\\":\\\"2026-08-19T07:59:51+00:00\\\",\\\"when\\\":\\\"2026-08-19T06:49:41+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nMultiple customers in Australia and New Zealand may experience slowness or errors while accessing Google Drive.\\\\n**Description**\\\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\\\nOur engineering team has identified the root cause of the issue and is actively working on a mitigation strategy. There is no ETA for full resolution.\\\\nWe will provide an update by Wednesday, 2026-08-19 01:30 PDT with current details.\\\\n**Symptoms**\\\\nAffected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-08-19T04:59:07+00:00\\\",\\\"modified\\\":\\\"2026-08-19T06:49:41+00:00\\\",\\\"when\\\":\\\"2026-08-19T04:59:07+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nMultiple customers in Australia and New Zealand may experience slowness or errors while using Google Drive.\\\\n**Description**\\\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\\\nOur engineering team is currently working on a potential fix while continuing to investigate the full root cause.\\\\nWe will provide an update by Wednesday, 2026-08-19 00:30 PDT with current details.\\\\n**Symptoms**\\\\nAffected customers in Australia and New Zealand may experience slowness or errors while using Google Drive. Some customers also may observe 502 errors with Drive inaccessible for a brief period.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-08-19T03:57:37+00:00\\\",\\\"modified\\\":\\\"2026-08-19T05:21:15+00:00\\\",\\\"when\\\":\\\"2026-08-19T03:57:37+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nSome customers may experience slowness while using Google Drive.\\\\n**Description**\\\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\\\nOur engineering team is currently working on a potential fix while continuing to investigate the full root cause.\\\\nWe will provide an update by Tuesday, 2026-08-18 22:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nAffected customers may experience slowness while using Google Drive.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-08-24T16:31:00+00:00\\\",\\\"modified\\\":\\\"2026-08-24T16:31:00+00:00\\\",\\\"when\\\":\\\"2026-08-24T16:31:00+00:00\\\",\\\"text\\\":\\\"# Incident Report\\\\n## Summary\\\\nOn Tuesday, 18 August 2026 at 16:58 US/Pacific, an incident affecting Google Drive occurred for a duration of 12 hours and 42 minutes. Customers located in Australia, Philippines and New Zealand experienced severe latency, sporadic loading failures, and errors when accessing Google Drive. \\\\nTo our valued Google Workspace customers, we are deeply sorry for the service interruption you experienced. We are taking immediate and decisive action to address the issue and strengthen our systems against future incidents. Other Google Workspace services were not affected.\\\\n## Root Cause\\\\nGoogle utilizes a multi-tiered global load balancing and network routing architecture to serve traffic efficiently. To ensure fair use and prevent congestion, bandwidth allocation is managed by attributing traffic to specific downstream services.\\\\nA misconfiguration in the attribution of traffic caused the bandwidth usage to be incorrectly attributed to the frontend network load balancing tier itself, rather than to the Drive service.\\\\nThe sequence of events leading to customer impact was as follows:\\\\n* Traffic from the regional load balancing tier was incorrectly attributed to the load balancing infrastructure rather than the Drive service.\\\\n* Unrelated traffic caused network bandwidth usage attributed to the load balancing infrastructure in this region to exceed its quota.\\\\n* Network bandwidth management systems applied throttling to this traffic to enforce quotas, causing severe network congestion affecting Drive traffic in this region.\\\\n* This congestion led to increased latency, slowness, and error rates for users accessing Drive in the affected regions.\\\\n## Remediation and Prevention\\\\nGoogle engineers were alerted to the issue via automated monitoring and an influx of customer support cases, with initial incident response teams engaged by 17:21 US/Pacific on 18 August , 2026\\\\\\\\.\\\\nEngineers initially attempted to alleviate the congestion by draining regional frontend traffic to an alternate geographic location, but this proved ineffective and was reverted. The team then mitigated the issue by migrating regional traffic to an upgraded internal network routing architecture, which correctly set the traffic attribution for Drive.\\\\nNormal operations were fully restored at 05:40 once 100% of the regional traffic was migrated to the upgraded network architecture.\\\\nTo prevent a recurrence of this issue, our engineering teams are undertaking the following structural preventative actions:\\\\n* **Resilient Network Protocol Migration:** We are expediting the complete migration of all regional frontend traffic to the newer, resilient network routing protocol to ensure accurate bandwidth attribution under all congestion scenarios.\\\\n* **Delegation Root Cause Remediation:** We are addressing the underlying mechanism that caused the incorrect delegation and analyzing similar cases across the fleet to ensure it does not reoccur.\\\\n## Detailed Description of Impact\\\\nFrom 18 August 2026, 16:58 to 19 August 2026, 05:40 US/Pacific for a duration of 12 hours, 42 minutes, Google Drive users in Australia, Philippines and New Zealand experienced severe performance degradation, including high latency and errors accessing Drive or listing files via Editors. Service was fully restored as of 19 August, 05:40 US/Pacific.\\\\n**Google Drive:**\\\\n* Elevated Latency: Users experienced extreme slowness when loading the Drive interface or accessing files, with initial data retrieval times heavily delayed.\\\\n* Loading Failures and Errors: Many user requests timed out or returned HTTP 502 errors due to the aggressive network throttling dropping the connections.\\\\n**Scope Exclusion:**\\\\n* Customers accessing Google Drive from outside the Australia, Philippines and New Zealand regions were largely unaffected by this localized throttling event.\\\\n* Other Google Workspace applications not reliant on this specific routing path remained stable during the incident window.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_DISRUPTION\\\",\\\"severity\\\":\\\"medium\\\",\\\"service_key\\\":\\\"VHNA7p3Z5p3iakj5sA8V\\\",\\\"service_name\\\":\\\"Google Drive\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Google Drive\\\",\\\"id\\\":\\\"VHNA7p3Z5p3iakj5sA8V\\\"}],\\\"uri\\\":\\\"incidents/sjDiph2NXGAfkYYLfK7T\\\"},{\\\"id\\\":\\\"MpR24eKjibRq1YEEaTfp\\\",\\\"number\\\":\\\"17752500813029482969\\\",\\\"begin\\\":\\\"2026-08-18T19:03:00+00:00\\\",\\\"created\\\":\\\"2026-08-18T19:35:27+00:00\\\",\\\"end\\\":\\\"2026-08-18T22:30:00+00:00\\\",\\\"modified\\\":\\\"2026-08-18T23:02:54+00:00\\\",\\\"external_desc\\\":\\\"**Summary:**\\\\nInvestigating access issues with Classroom’s new web homepage.\\\\n**Description:**\\\\nWe are investigating reports of access issues with Classroom’s new web homepage beginning on Thursday, 2026-08-13.\\\\nOur engineering team is currently working on a potential fix while continuing to investigate the full root cause.\\\\nWe will provide an update by Tuesday, 2026-08-18 13:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms:**\\\\nAdmins and some Teachers may experience permission errors when accessing Classroom’s new web homepage.\\\\n**Workaround:**\\\\nUsers should still be able to navigate to other views from the left hand pane of the interface.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-08-18T23:02:54+00:00\\\",\\\"modified\\\":\\\"2026-08-18T23:02:54+00:00\\\",\\\"when\\\":\\\"2026-08-18T23:02:54+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\n[Mitigated] Access issues for Workspace Admins with Classroom’s web homepage.\\\\n**Description**\\\\nThe issue with Classroom has been resolved for all affected users as of Tuesday, 2026-08-18 15:30 PDT.\\\\nFrom preliminary analysis, the issue was triggered due to a recent change which caused the permission mismatch, our Engineering team has rolled out the fix to mitigate the impact.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nWorkspace Admins may have experienced permission errors when accessing Classroom’s web homepage.\\\\nRemaining Classroom functionality remained unaffected.\\\\nThe Classroom Mobile App and the Classroom API were unaffected by the issue.\\\\n**Workaround**\\\\nNone as the issue is resolved\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-08-18T20:27:49+00:00\\\",\\\"modified\\\":\\\"2026-08-18T23:02:54+00:00\\\",\\\"when\\\":\\\"2026-08-18T20:27:49+00:00\\\",\\\"text\\\":\\\"**Summary:**\\\\nMitigating access issues for Workspace Admins with Classroom’s web homepage.\\\\n**Description:**\\\\nOur engineering has identified the cause and mitigation work is currently underway.\\\\nThe mitigation rollout is expected to take several hours.\\\\nWe will provide more information by Wednesday, 2026-08-19 8:00 PDT.\\\\n**Symptoms:**\\\\nWorkspace Admins may experience permission errors when accessing Classroom’s web homepage.\\\\nRemaining Classroom functionality remains unaffected.\\\\nThe Classroom Mobile App and the Classroom API are unaffected by the issue.\\\\n**Workaround:**\\\\nAll other pages are unaffected and admins can navigate to other pages using the left hand pane on the interface.\\\\nWorkspace Admins can still create classes without any issue.\\\\nWorkspace Admins can also view the courses that they are in by going to https://classroom.google.com/h/tv\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-08-18T20:00:36+00:00\\\",\\\"modified\\\":\\\"2026-08-18T20:27:49+00:00\\\",\\\"when\\\":\\\"2026-08-18T20:00:36+00:00\\\",\\\"text\\\":\\\"**Summary:**\\\\nMitigating access issues with Classroom’s new web homepage.\\\\n**Description:**\\\\nMitigation work is currently underway by our engineering team.\\\\nThe mitigation is expected to complete by Tuesday, 2026-08-18 14:00 PDT.\\\\nWe will provide more information by Tuesday, 2026-08-18 14:15 PDT.\\\\n**Symptoms:**\\\\nAdmins and some Teachers may experience permission errors when accessing Classroom’s new web homepage.\\\\n**Workaround:**\\\\nUsers should still be able to navigate to other views from the left hand pane of the interface.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-08-18T19:35:27+00:00\\\",\\\"modified\\\":\\\"2026-08-18T20:11:32+00:00\\\",\\\"when\\\":\\\"2026-08-18T19:35:27+00:00\\\",\\\"text\\\":\\\"**Summary:**\\\\nInvestigating access issues with Classroom’s new web homepage.\\\\n**Description:**\\\\nWe are investigating reports of access issues with Classroom’s new web homepage beginning on Thursday, 2026-08-13.\\\\nOur engineering team is currently working on a potential fix while continuing to investigate the full root cause.\\\\nWe will provide an update by Tuesday, 2026-08-18 13:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms:**\\\\nAdmins and some Teachers may experience permission errors when accessing Classroom’s new web homepage.\\\\n**Workaround:**\\\\nUsers should still be able to navigate to other views from the left hand pane of the interface.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-08-18T23:02:54+00:00\\\",\\\"modified\\\":\\\"2026-08-18T23:02:54+00:00\\\",\\\"when\\\":\\\"2026-08-18T23:02:54+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\n[Mitigated] Access issues for Workspace Admins with Classroom’s web homepage.\\\\n**Description**\\\\nThe issue with Classroom has been resolved for all affected users as of Tuesday, 2026-08-18 15:30 PDT.\\\\nFrom preliminary analysis, the issue was triggered due to a recent change which caused the permission mismatch, our Engineering team has rolled out the fix to mitigate the impact.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nWorkspace Admins may have experienced permission errors when accessing Classroom’s web homepage.\\\\nRemaining Classroom functionality remained unaffected.\\\\nThe Classroom Mobile App and the Classroom API were unaffected by the issue.\\\\n**Workaround**\\\\nNone as the issue is resolved\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_INFORMATION\\\",\\\"severity\\\":\\\"low\\\",\\\"service_key\\\":\\\"3GoNVrVz48jeNZSyLJMV\\\",\\\"service_name\\\":\\\"Classroom\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Classroom\\\",\\\"id\\\":\\\"3GoNVrVz48jeNZSyLJMV\\\"}],\\\"uri\\\":\\\"incidents/MpR24eKjibRq1YEEaTfp\\\"},{\\\"id\\\":\\\"AWyE1SfEzpe8Qed49Rog\\\",\\\"number\\\":\\\"12918412236754833550\\\",\\\"begin\\\":\\\"2026-06-30T09:03:00+00:00\\\",\\\"created\\\":\\\"2026-06-30T11:56:57+00:00\\\",\\\"end\\\":\\\"2026-07-01T03:45:00+00:00\\\",\\\"modified\\\":\\\"2026-07-01T04:58:32+00:00\\\",\\\"external_desc\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nWe are experiencing an issue with Google Cloud Support beginning on Tuesday, 2026-06-30 02:03 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Tuesday, 2026-06-30 05:30 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation continues to work.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-07-01T04:58:32+00:00\\\",\\\"modified\\\":\\\"2026-07-01T04:58:32+00:00\\\",\\\"when\\\":\\\"2026-07-01T04:58:32+00:00\\\",\\\"text\\\":\\\"**Title**\\\\n[Resolved] Inbound emails from customers to Google Support were experiencing issues.\\\\n**Description**\\\\nThe issue with inbound emails to Google Cloud Support has been resolved for all affected customers as of Tuesday, 2026-06-30 20:45 PDT.\\\\nFrom preliminary analysis, the engineering team identified a recent change which was preventing incoming emails from being processed. We have successfully rolled back the change to mitigate the impact.\\\\nBacklog processing of all the affected emails is now complete.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support were experiencing issues.\\\\nCase creation and updates continued to work without any issues.\\\\n**Workaround**\\\\nNone as the issue is resolved.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-07-01T00:44:02+00:00\\\",\\\"modified\\\":\\\"2026-07-01T04:58:32+00:00\\\",\\\"when\\\":\\\"2026-07-01T00:44:02+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nThe mitigation rollout was completed as of 09:00 AM PDT on 30 June 2026, and all incoming emails are currently being processed without further disruption.\\\\nOur engineers are continuing to work replaying the inbound messages that were impacted between 25 June 2026 and 09:00 AM PDT on 30 June 2026. We expect the email replays to take several hours, please be advised that this backlog processing may cause case statuses to update dynamically or cause certain closed cases to temporarily reopen.\\\\nWe will provide more information by Tuesday, 2026-06-30 21:30 PDT.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support after 09:00 AM PDT on 30 June 2026 are working as expected. The affected emails from the time of the issue are being replayed.\\\\nCase creation and updates continues to work without any issues.\\\\n**Workaround**\\\\nCustomers can continue to use the portal to add comments / notes.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T19:03:19+00:00\\\",\\\"modified\\\":\\\"2026-07-01T00:44:02+00:00\\\",\\\"when\\\":\\\"2026-06-30T19:03:19+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nThe mitigation rollout was completed as of 09:00 AM PDT on 30 June 2026, and all incoming emails are currently being processed without further disruption.\\\\nOur engineers are continuing to work replaying the inbound messages that were impacted between 25 June 2026 and 09:00 AM PDT on 30 June 2026. We expect the email replays to take several hours, please be advised that this backlog processing may cause case statuses to update dynamically or cause certain closed cases to temporarily reopen.\\\\nWe will provide more information by Tuesday, 2026-06-30 17:00 PDT.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support after 09:00 AM PDT on 30 June 2026 are working as expected. The affected emails from the time of the issue are being replayed.\\\\nCase creation and updates continues to work without any issues.\\\\n**Workaround**\\\\nCustomers can continue to use the portal to add comments / notes.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T17:53:29+00:00\\\",\\\"modified\\\":\\\"2026-06-30T19:03:19+00:00\\\",\\\"when\\\":\\\"2026-06-30T17:53:29+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nThe mitigation rollout was completed as of 09:00 AM PDT on 30 June 2026, and all incoming emails are currently being processed without further disruption.\\\\nOur engineers are continuing to work replaying the inbound messages that were impacted between 25 June 2026 and 09:00 AM PDT on 30 June 2026. We expect the email replays to take several hours, please be advised that this backlog processing may cause case statuses to update dynamically or cause certain closed cases to temporarily reopen.\\\\nWe will provide more information by Tuesday, 2026-06-30 12:15 PDT.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation and updates continues to work without any issues.\\\\n**Workaround**\\\\nCustomers can continue to use the portal to add comments / notes.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T16:36:46+00:00\\\",\\\"modified\\\":\\\"2026-06-30T17:53:29+00:00\\\",\\\"when\\\":\\\"2026-06-30T16:36:46+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nThe mitigation rollout was completed as of 09:00 AM PDT on 30 June 2026, and all incoming emails are currently being processed without further disruption.\\\\nOur engineering teams have shifted focus to replaying the inbound messages that were impacted between 25 June 2026 and 09:00 AM PDT on 30 June 2026.\\\\nWhile there is currently no specific ETA for the email replay to finish, please be advised that this backlog processing may cause case statuses to update dynamically or cause certain closed cases to temporarily reopen.\\\\nWe will provide more information by Tuesday, 2026-06-30 11:00 PDT.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation and updates continues to work without any issues.\\\\n**Workaround**\\\\nCustomers can continue to use the portal to add comments / notes.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T15:48:35+00:00\\\",\\\"modified\\\":\\\"2026-06-30T16:36:46+00:00\\\",\\\"when\\\":\\\"2026-06-30T15:48:35+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nMitigation work is currently underway by our engineering team.\\\\nThe mitigation is expected to complete by Tuesday, 2026-06-30 09:45 PDT.\\\\nWe will provide more information by Tuesday, 2026-06-30 10:00 PDT.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation and updates continues to work without any issues.\\\\n**Workaround**\\\\nCustomers can continue to use the portal to add comments / notes.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T14:51:10+00:00\\\",\\\"modified\\\":\\\"2026-06-30T15:48:35+00:00\\\",\\\"when\\\":\\\"2026-06-30T14:51:10+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nMitigation work is currently underway by our engineering team.\\\\nWe do not have an ETA for mitigation at this point.\\\\nWe will provide more information by Tuesday, 2026-06-30 08:45 PDT.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation and updates continues to work without any issues.\\\\n**Workaround**\\\\nCustomers can continue to use the portal to add comments / notes.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T14:06:26+00:00\\\",\\\"modified\\\":\\\"2026-06-30T14:51:10+00:00\\\",\\\"when\\\":\\\"2026-06-30T14:06:26+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nWe are experiencing an issue with Google Cloud Support beginning on Tuesday, 2026-06-30 02:03 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Tuesday, 2026-06-30 08:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation continues to work.\\\\n**Workaround**\\\\nCustomers can continue to use the portal to add comments / notes.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T13:28:46+00:00\\\",\\\"modified\\\":\\\"2026-06-30T14:06:26+00:00\\\",\\\"when\\\":\\\"2026-06-30T13:28:46+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nWe believe the issue with Google Support is mitigated.\\\\nOur engineering team continues to investigate the root cause.\\\\nCustomers are being advised if they have an urgent issue to resend previous communication to Google Support.\\\\nWe will provide an update by Tuesday, 2026-06-30 08:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation continues to work.\\\\n**Workaround**\\\\nCustomers can continue to use the portal to add comments / notes.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T12:28:59+00:00\\\",\\\"modified\\\":\\\"2026-06-30T13:28:46+00:00\\\",\\\"when\\\":\\\"2026-06-30T12:28:59+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nWe believe the issue with Google Support is mitigated.\\\\nOur engineering team continues to investigate the root cause.\\\\nCustomers are being advised if they have an urgent issue to resend previous communication to Google Support.\\\\nWe will provide an update by Tuesday, 2026-06-30 06:30 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation continues to work.\\\\n**Workaround**\\\\nCustomers can continue to use the portal to add comments / notes.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T12:17:19+00:00\\\",\\\"modified\\\":\\\"2026-06-30T12:28:59+00:00\\\",\\\"when\\\":\\\"2026-06-30T12:17:19+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nWe believe the issue with Google Support is mitigated.\\\\nOur engineering team continues to investigate the root cause.\\\\nCustomers are being advised if they have an urgent issue to resend previous communication to Google Support.\\\\nWe will provide an update by Tuesday, 2026-06-30 06:30 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation continues to work.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T11:56:57+00:00\\\",\\\"modified\\\":\\\"2026-06-30T12:17:19+00:00\\\",\\\"when\\\":\\\"2026-06-30T11:56:57+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nWe are experiencing an issue with Google Cloud Support beginning on Tuesday, 2026-06-30 02:03 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Tuesday, 2026-06-30 05:30 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation continues to work.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-07-01T04:58:32+00:00\\\",\\\"modified\\\":\\\"2026-07-01T04:58:32+00:00\\\",\\\"when\\\":\\\"2026-07-01T04:58:32+00:00\\\",\\\"text\\\":\\\"**Title**\\\\n[Resolved] Inbound emails from customers to Google Support were experiencing issues.\\\\n**Description**\\\\nThe issue with inbound emails to Google Cloud Support has been resolved for all affected customers as of Tuesday, 2026-06-30 20:45 PDT.\\\\nFrom preliminary analysis, the engineering team identified a recent change which was preventing incoming emails from being processed. We have successfully rolled back the change to mitigate the impact.\\\\nBacklog processing of all the affected emails is now complete.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support were experiencing issues.\\\\nCase creation and updates continued to work without any issues.\\\\n**Workaround**\\\\nNone as the issue is resolved.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_DISRUPTION\\\",\\\"severity\\\":\\\"medium\\\",\\\"service_key\\\":\\\"dnz8HVUHoMMcW9wV91LY\\\",\\\"service_name\\\":\\\"Google Workspace Support\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Google Workspace Support\\\",\\\"id\\\":\\\"dnz8HVUHoMMcW9wV91LY\\\"}],\\\"uri\\\":\\\"incidents/AWyE1SfEzpe8Qed49Rog\\\"},{\\\"id\\\":\\\"qqWhZSpoVLJWur1GfHfu\\\",\\\"number\\\":\\\"6760817193806533896\\\",\\\"begin\\\":\\\"2026-06-22T19:30:00+00:00\\\",\\\"created\\\":\\\"2026-06-22T22:28:55+00:00\\\",\\\"end\\\":\\\"2026-06-23T00:46:00+00:00\\\",\\\"modified\\\":\\\"2026-06-23T01:08:59+00:00\\\",\\\"external_desc\\\":\\\"**Title**\\\\nSome consumer Google Voice users are experiencing account suspensions.\\\\n**Description**\\\\nWe are experiencing an issue with Google Voice beginning at Monday, 2026-06-22 12:30 PDT.\\\\nOur engineering team has identified the root cause and is actively working to mitigate the issue.\\\\nWe will provide an update by Monday, 2026-06-22 17:30 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Customer Symptoms**\\\\nSome consumer Google Voice users are experiencing account suspensions\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-06-23T01:08:59+00:00\\\",\\\"modified\\\":\\\"2026-06-23T01:08:59+00:00\\\",\\\"when\\\":\\\"2026-06-23T01:08:59+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nSome consumer Google Voice users experienced account suspensions.\\\\n**Description**\\\\nThe issue with Google Voice has been resolved for all affected users as of Monday, 2026-06-22 17:46 PDT\\\\nOur engineers have mitigated the issue by rolling back to a previous version and reinstating suspended accounts.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nSome consumer Google Voice users experienced account suspensions\\\\n**Workaround**\\\\nThe issue is now mitigated.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-06-23T00:29:23+00:00\\\",\\\"modified\\\":\\\"2026-06-23T01:08:59+00:00\\\",\\\"when\\\":\\\"2026-06-23T00:29:23+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nSome consumer Google Voice users are experiencing account suspensions.\\\\n**Description**\\\\nWe are experiencing an issue with Google Voice beginning at Monday, 2026-06-22 12:30 PDT.\\\\nOur engineering team has identified a mitigation strategy and is actively working on mitigation.\\\\nWe do not have an ETA for mitigation at this time.\\\\nWe will provide an update by Monday, 2026-06-22 22:30 PDT with current details.\\\\n**Symptoms**\\\\nSome consumer Google Voice users are experiencing account suspensions\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-06-22T22:28:55+00:00\\\",\\\"modified\\\":\\\"2026-06-23T00:29:23+00:00\\\",\\\"when\\\":\\\"2026-06-22T22:28:55+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nSome consumer Google Voice users are experiencing account suspensions.\\\\n**Description**\\\\nWe are experiencing an issue with Google Voice beginning at Monday, 2026-06-22 12:30 PDT.\\\\nOur engineering team has identified the root cause and is actively working to mitigate the issue.\\\\nWe will provide an update by Monday, 2026-06-22 17:30 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Customer Symptoms**\\\\nSome consumer Google Voice users are experiencing account suspensions\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-06-23T01:08:59+00:00\\\",\\\"modified\\\":\\\"2026-06-23T01:08:59+00:00\\\",\\\"when\\\":\\\"2026-06-23T01:08:59+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nSome consumer Google Voice users experienced account suspensions.\\\\n**Description**\\\\nThe issue with Google Voice has been resolved for all affected users as of Monday, 2026-06-22 17:46 PDT\\\\nOur engineers have mitigated the issue by rolling back to a previous version and reinstating suspended accounts.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nSome consumer Google Voice users experienced account suspensions\\\\n**Workaround**\\\\nThe issue is now mitigated.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_INFORMATION\\\",\\\"severity\\\":\\\"low\\\",\\\"service_key\\\":\\\"gh79SrQk6wqonWoxPdCm\\\",\\\"service_name\\\":\\\"Google Voice\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Google Voice\\\",\\\"id\\\":\\\"gh79SrQk6wqonWoxPdCm\\\"}],\\\"uri\\\":\\\"incidents/qqWhZSpoVLJWur1GfHfu\\\"},{\\\"id\\\":\\\"CzZUn98mhTcEiCJo27Kv\\\",\\\"number\\\":\\\"11672950073358311792\\\",\\\"begin\\\":\\\"2026-06-10T10:20:00+00:00\\\",\\\"created\\\":\\\"2026-06-10T12:40:45+00:00\\\",\\\"end\\\":\\\"2026-06-10T17:30:00+00:00\\\",\\\"modified\\\":\\\"2026-06-16T17:41:16+00:00\\\",\\\"external_desc\\\":\\\"**Title**\\\\nGemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.\\\\n**Description**\\\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Wednesday, 2026-06-10 06:30 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nGemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-06-16T17:41:16+00:00\\\",\\\"modified\\\":\\\"2026-06-16T17:41:17+00:00\\\",\\\"when\\\":\\\"2026-06-16T17:41:16+00:00\\\",\\\"text\\\":\\\"# Incident Report\\\\n## Summary\\\\nOn Wednesday, June 10, 2026, at 03:30 US/Pacific, an incident occurred affecting the availability of Gemini services across all surfaces, including web, mobile (iOS and Android), and integrations within Google Chrome. The incident resulted in an elevated error rate for approximately 6 hours and 55 minutes, with full service restoration confirmed after a total duration of 14 hours and 49 minutes.\\\\nWe sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Cloud, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future.\\\\n## Root Cause\\\\nThe disruption was caused by extreme read contention within the foundational database service that manages tool deployment metadata.\\\\nThe primary cause was an index design issue within the database where a column used for tracking deployment expirations contained a high volume of rows with similar values. This resulted in \\\\\\\"hotspotting,\\\\\\\" where traffic was concentrated on a very small number of database shards, overwhelming them. Additionally, the service utilized an in-memory cache with a very short Time To Live (TTL) of only one minute, requiring frequent database refreshes.\\\\nThe failure sequence was as follows:\\\\n1. **Technical Trigger:** The trigger was an upward spike in the Queries Per Second (QPS) from the frontend. The underlying system was already running close to maximum usage and the incoming QPS crossed a tipping threshold beyond which the underlying spanner started getting overloaded.\\\\n2. **Systemic Effect:** This short window, combined with an indexing anomaly where specific metadata fields containing empty values were clustered on a single database partition, led to a \\\\\\\"hot shard\\\\\\\" condition.\\\\n3. **Contention and Latency:** As request volume increased, the internal tool management service experienced a more than 10x surge in database calls because the cache hit rate dropped significantly.\\\\n4. **Failure State:** The underlying storage layer reached its capacity for handling incoming queries, and database failure rates skyrocketed to 60%. This led to cache hit rate dropping down to 50%.\\\\nAutomated validation failed to prevent this because the indexing strategy did not sufficiently account for the concentration of data rows when specific identifiers were absent.\\\\n## Remediation and Prevention\\\\nThe issue was detected by internal real-time monitoring alerts indicating high processing error rates. Engineering teams performed several targeted actions to restore the service:\\\\n* **Throttling:** Applied Rate Access Control Lists (RateACLs) to the most affected database shards to allow internal caches to recover.\\\\n* **Index Redistribution:** Gradually redistributed common values in the database index to random values in a specific target range, effectively spreading the load across multiple shards.\\\\n* **Cache Policy Update:** Deployed a configuration change to increase the in-memory cache Time To Live (TTL) from 1 minute to 20 minutes, significantly reducing database load.\\\\n* **Deadline Adjustment:** Shortened the Remote Procedure Call (RPC) deadlines for database queries to prevent request pile-ups during periods of high latency.\\\\n## Prevention\\\\nTo prevent a recurrence, Google engineering will be implementing the following actions:\\\\n* **Index Design Standards:** Restructure of database index to prevent \\\\\\\"hotspotting\\\\\\\".\\\\n* **Adaptive Cache Tuning:** Implement more robust cache policies that prevent load amplification during periods of backend stress.\\\\n* **Enhanced Hotspot Monitoring:** Deploy improved monitoring and alerting specifically designed to detect uneven database load distribution before it impacts user traffic.\\\\n* **Service Resiliency:** Implement advanced load protection mechanisms, such as smart task resizing and query coalescing for preventing spanner overloads.\\\\n## Detailed Description of Impact\\\\nThe impact window occurred between 03:30 US/Pacific and 10:25 US/Pacific on June 10, 2026\\\\\\\\.\\\\n**Gemini App (Web and Mobile):**\\\\n* High Failure Rates: Users experienced a 50% error rate when attempting to send prompts, receiving \\\\\\\"Something went wrong\\\\\\\" messages .\\\\n* Service Unavailability: The issue affected both consumer accounts and enterprise Google Workspace customers .\\\\n* Enterprise Google Workspace Users encountered disruptions when attempting to attach assets from Google Drive within the web application (gemini.google.com). Affected users observed the \\\\\\\"Add from Drive\\\\\\\" functionality to be visually disabled and unresponsive following interaction with the \\\\\\\"+\\\\\\\" prompt.\\\\n**Integrations:**\\\\n* **Google Chrome Integration:** The Gemini Side Panel and integrated conversational features encountered persistent timeouts.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-06-12T11:14:34+00:00\\\",\\\"modified\\\":\\\"2026-06-16T17:41:16+00:00\\\",\\\"when\\\":\\\"2026-06-12T11:14:34+00:00\\\",\\\"text\\\":\\\"# Preliminary Incident Report\\\\nWe apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article [https://support.google.com/a/answer/1047213](https://support.google.com/a/answer/1047213).\\\\n#[REDACTED] Date/Time of the Issue (All time US/Pacific)\\\\nIncident Start: 10 June 2026 03:20\\\\nIncident End: 10 June 2026 10:25\\\\nDuration: 7 hours, 5 minutes\\\\n## Summary\\\\nOn Wednesday, 10 June 2026 03:20 US/Pacific, an incident occurred affecting the availability of Gemini services across all surfaces, including web, mobile (iOS and Android), and integrations within Google Chrome. The incident resulted in an elevated error rate for 7 hours and 5 minutes.\\\\n## Preliminary Root Cause\\\\nThe disruption was caused by extreme read contention within the foundational database service that manages tool deployment metadata.\\\\nThe primary cause was back-pressure on specific database replicas that experienced high loads. The issue was made worse because of an index design issue within the database where a column used for tracking deployment expirations contained a high volume of rows with a similar fixed value. This resulted in a condition where traffic was concentrated on a very small number of database instances, overwhelming them. Additionally, the service utilized an in-memory cache with a short Time To Live (TTL) of only one minute, requiring frequent database refreshes.\\\\nThe failure sequence was as follows:\\\\n1. **Technical Trigger:** The database service became overloaded, causing back-pressure on specific replicas.\\\\n2. **Systemic Effect:** An application level caching layer configuration is set to a short 1-minute expiration window for deployed tools. This short window, combined with an indexing anomaly where specific metadata fields containing empty values were clustered on a single database partition. The failed DB queries resulted in failure to populate application level cache.\\\\n3. **Contention and Latency:** As request volume increased, the internal tool management service experienced \\\\u003e10x surge in database calls because the cache hit rate dropped significantly.\\\\n4. **Failure State:** The underlying storage layer reached its transaction limits, with database response success rates dropping as low as 10% on the affected partitions.\\\\n## Remediation\\\\nThe issue was detected by internal real-time monitoring alerts indicating high processing error rates. Engineering teams performed several targeted actions to restore the service:\\\\n* **Throttling:** Applied rate limiting to the most affected database instances to allow internal caches to recover.\\\\n* **Index Redistribution:** Gradually redistributed fixed index values in the database to random values in a specific range, effectively spreading the load across multiple instances.\\\\n* **Cache Policy Update:** Deployed a configuration change to increase the in-memory cache Time To Live (TTL) from 1 minute to 20 minutes, significantly reducing database load.\\\\n* **Deadline Adjustment:** Shortened the Remote Procedure Call (RPC) deadlines for database queries to prevent request pile-ups during periods of high latency.\\\\n## Description of Impact\\\\nThe impact window occurred between 03:20 US/Pacific and 10:25 US/Pacific on 10 June 2026.\\\\n**Gemini App (Web and Mobile):**\\\\n* Users experienced a 50% error rate when attempting to send prompts, receiving \\\\\\\"Something went wrong\\\\\\\" messages. The issue affected both consumer accounts and enterprise Google Workspace customers .\\\\n**Integrations:**\\\\n* **Google Chrome Integration:** The Gemini Side Panel and integrated conversational features encountered persistent timeouts.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-06-11T01:07:25+00:00\\\",\\\"modified\\\":\\\"2026-06-12T11:14:34+00:00\\\",\\\"when\\\":\\\"2026-06-11T01:07:25+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nGemini App in Workspace users were experiencing “Something Went Wrong” errors.\\\\n**Description**\\\\nThe issue with Gemini has been resolved for all the affected users as of Wednesday, 2026-06-10 10:30 US/Pacific.\\\\nFrom preliminary analysis, the issue was triggered by a performance issue in our backend database which impacted the retrieval of Gemini App tools catalog.\\\\nOur engineers mitigated the issue by optimizing the load distribution across our backend database fleet.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nAffected users may have seen \\\\\\\"Something went wrong\\\\\\\" errors (codes 1099, 1076) when prompting Gemini while using the following services.\\\\nGemini App in Workspace includes the following platforms: - Web, MacOS, iOS and Android - Gemini in Chrome\\\\n**Workaround**\\\\nNone as the issue is now resolved.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-06-10T20:19:31+00:00\\\",\\\"modified\\\":\\\"2026-06-11T01:07:25+00:00\\\",\\\"when\\\":\\\"2026-06-10T20:19:31+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nGemini App in Workspace customers are experiencing “Something Went Wrong” errors.\\\\n**Description**\\\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\\\nOur engineering team has applied mitigations to reduce impact and confirmed the majority of users are no longer observing impact as of Wednesday, 2026-06-10 10:30 PDT. We will continue to monitor the service for stability.\\\\nWe will provide an update by Wednesday, 2026-06-10 20:30 PDT with current details.\\\\n**Symptoms**\\\\nThe following services are experiencing “Something Went Wrong 1099 or 1076” errors:\\\\nGemini app in Workspace includes the following platforms:\\\\n* Web, MacOS, iOS and Android\\\\n* Gemini in Chrome\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-10T18:37:53+00:00\\\",\\\"modified\\\":\\\"2026-06-10T20:19:31+00:00\\\",\\\"when\\\":\\\"2026-06-10T18:37:53+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nGemini App in Workspace customers are experiencing “Something Went Wrong” errors.\\\\n**Description**\\\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\\\nOur engineering team has applied mitigations to reduce impact and continues to investigate the root cause. We are seeing signs of recovery and will continue to monitor progress.\\\\nWe will provide an update by Wednesday, 2026-06-10 14:30 PDT with current details.\\\\n**Symptoms**\\\\nThe following services are experiencing “Something Went Wrong 1099 or 1076” errors:\\\\nGemini app in Workspace includes the following platforms:\\\\n* Web, MacOS, iOS and Android\\\\n* Gemini in Chrome\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-10T16:24:54+00:00\\\",\\\"modified\\\":\\\"2026-06-10T18:37:53+00:00\\\",\\\"when\\\":\\\"2026-06-10T16:24:54+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nGemini App in Workspace customers are experiencing “Something Went Wrong” errors.\\\\n**Description**\\\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\\\nOur engineering team has identified a mitigation which is currently in progress. We do not have an ETA for mitigation at this time.\\\\nWe will provide an update by Wednesday, 2026-06-10 12:30 PDT with current details.\\\\n**Symptoms**\\\\nThe following services are experiencing “Something Went Wrong 1099 or 1076” errors:\\\\nGemini app in Workspace includes the following platforms:\\\\n* Web, MacOS, iOS and Android\\\\n* Gemini in Chrome\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-10T15:10:52+00:00\\\",\\\"modified\\\":\\\"2026-06-10T16:24:54+00:00\\\",\\\"when\\\":\\\"2026-06-10T15:10:52+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nGemini App in Workspace customers are experiencing “Something Went Wrong” errors.\\\\n**Description**\\\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\\\nOur engineering team continues to investigate the issue and identify a mitigation strategy.\\\\nWe will provide an update by Wednesday, 2026-06-10 09:30 PDT with current details.\\\\n**Symptoms**\\\\nThe following services are experiencing “Something Went Wrong” errors.:\\\\n* Gemini app in Workspace includes the following platforms;\\\\n* Web, MacOS, iOS and Android\\\\n* Gemini in Chrome\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-10T13:21:18+00:00\\\",\\\"modified\\\":\\\"2026-06-10T15:10:52+00:00\\\",\\\"when\\\":\\\"2026-06-10T13:21:18+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nGemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.\\\\n**Description**\\\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Wednesday, 2026-06-10 08:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nGemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-10T12:40:45+00:00\\\",\\\"modified\\\":\\\"2026-06-10T13:21:18+00:00\\\",\\\"when\\\":\\\"2026-06-10T12:40:45+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nGemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.\\\\n**Description**\\\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Wednesday, 2026-06-10 06:30 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nGemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-06-16T17:41:16+00:00\\\",\\\"modified\\\":\\\"2026-06-16T17:41:17+00:00\\\",\\\"when\\\":\\\"2026-06-16T17:41:16+00:00\\\",\\\"text\\\":\\\"# Incident Report\\\\n## Summary\\\\nOn Wednesday, June 10, 2026, at 03:30 US/Pacific, an incident occurred affecting the availability of Gemini services across all surfaces, including web, mobile (iOS and Android), and integrations within Google Chrome. The incident resulted in an elevated error rate for approximately 6 hours and 55 minutes, with full service restoration confirmed after a total duration of 14 hours and 49 minutes.\\\\nWe sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Cloud, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future.\\\\n## Root Cause\\\\nThe disruption was caused by extreme read contention within the foundational database service that manages tool deployment metadata.\\\\nThe primary cause was an index design issue within the database where a column used for tracking deployment expirations contained a high volume of rows with similar values. This resulted in \\\\\\\"hotspotting,\\\\\\\" where traffic was concentrated on a very small number of database shards, overwhelming them. Additionally, the service utilized an in-memory cache with a very short Time To Live (TTL) of only one minute, requiring frequent database refreshes.\\\\nThe failure sequence was as follows:\\\\n1. **Technical Trigger:** The trigger was an upward spike in the Queries Per Second (QPS) from the frontend. The underlying system was already running close to maximum usage and the incoming QPS crossed a tipping threshold beyond which the underlying spanner started getting overloaded.\\\\n2. **Systemic Effect:** This short window, combined with an indexing anomaly where specific metadata fields containing empty values were clustered on a single database partition, led to a \\\\\\\"hot shard\\\\\\\" condition.\\\\n3. **Contention and Latency:** As request volume increased, the internal tool management service experienced a more than 10x surge in database calls because the cache hit rate dropped significantly.\\\\n4. **Failure State:** The underlying storage layer reached its capacity for handling incoming queries, and database failure rates skyrocketed to 60%. This led to cache hit rate dropping down to 50%.\\\\nAutomated validation failed to prevent this because the indexing strategy did not sufficiently account for the concentration of data rows when specific identifiers were absent.\\\\n## Remediation and Prevention\\\\nThe issue was detected by internal real-time monitoring alerts indicating high processing error rates. Engineering teams performed several targeted actions to restore the service:\\\\n* **Throttling:** Applied Rate Access Control Lists (RateACLs) to the most affected database shards to allow internal caches to recover.\\\\n* **Index Redistribution:** Gradually redistributed common values in the database index to random values in a specific target range, effectively spreading the load across multiple shards.\\\\n* **Cache Policy Update:** Deployed a configuration change to increase the in-memory cache Time To Live (TTL) from 1 minute to 20 minutes, significantly reducing database load.\\\\n* **Deadline Adjustment:** Shortened the Remote Procedure Call (RPC) deadlines for database queries to prevent request pile-ups during periods of high latency.\\\\n## Prevention\\\\nTo prevent a recurrence, Google engineering will be implementing the following actions:\\\\n* **Index Design Standards:** Restructure of database index to prevent \\\\\\\"hotspotting\\\\\\\".\\\\n* **Adaptive Cache Tuning:** Implement more robust cache policies that prevent load amplification during periods of backend stress.\\\\n* **Enhanced Hotspot Monitoring:** Deploy improved monitoring and alerting specifically designed to detect uneven database load distribution before it impacts user traffic.\\\\n* **Service Resiliency:** Implement advanced load protection mechanisms, such as smart task resizing and query coalescing for preventing spanner overloads.\\\\n## Detailed Description of Impact\\\\nThe impact window occurred between 03:30 US/Pacific and 10:25 US/Pacific on June 10, 2026\\\\\\\\.\\\\n**Gemini App (Web and Mobile):**\\\\n* High Failure Rates: Users experienced a 50% error rate when attempting to send prompts, receiving \\\\\\\"Something went wrong\\\\\\\" messages .\\\\n* Service Unavailability: The issue affected both consumer accounts and enterprise Google Workspace customers .\\\\n* Enterprise Google Workspace Users encountered disruptions when attempting to attach assets from Google Drive within the web application (gemini.google.com). Affected users observed the \\\\\\\"Add from Drive\\\\\\\" functionality to be visually disabled and unresponsive following interaction with the \\\\\\\"+\\\\\\\" prompt.\\\\n**Integrations:**\\\\n* **Google Chrome Integration:** The Gemini Side Panel and integrated conversational features encountered persistent timeouts.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_DISRUPTION\\\",\\\"severity\\\":\\\"medium\\\",\\\"service_key\\\":\\\"npdyhgECDJ6tB66MxXyo\\\",\\\"service_name\\\":\\\"Gemini\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Gemini\\\",\\\"id\\\":\\\"npdyhgECDJ6tB66MxXyo\\\"}],\\\"uri\\\":\\\"incidents/CzZUn98mhTcEiCJo27Kv\\\"},{\\\"id\\\":\\\"vagmms1y6yRxwg6RMQZp\\\",\\\"number\\\":\\\"6811500350919047262\\\",\\\"begin\\\":\\\"2026-05-31T03:28:00+00:00\\\",\\\"created\\\":\\\"2026-05-31T04:59:04+00:00\\\",\\\"end\\\":\\\"2026-05-31T04:24:00+00:00\\\",\\\"modified\\\":\\\"2026-06-04T20:11:26+00:00\\\",\\\"external_desc\\\":\\\"**Title**\\\\nMultiple Workspace Products experiencing Issues.\\\\n**Description**\\\\nWe are experiencing an issue with Google Voice, Google Meet, Google Drive, Google Chat, Google Calendar, Gmail, Google Docs beginning on Saturday, 2026-05-30 20:28 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Saturday, 2026-05-30 22:10 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nCustomers may experience issues loading the application/ intermittent high latency or errors while using the application.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-06-04T20:11:26+00:00\\\",\\\"modified\\\":\\\"2026-06-04T20:11:27+00:00\\\",\\\"when\\\":\\\"2026-06-04T20:11:26+00:00\\\",\\\"text\\\":\\\"# Incident Report\\\\n## Summary\\\\nStarting at 20:15 US/Pacific on Saturday, 30 May 2026, various Google Workspace services encountered service-level disruptions lasting approximately 75 minutes. The scope of impact spanned several products, specifically affecting Gmail, Google Docs, Drive, Calendar, Chat, Meet, and Google Voice.\\\\nThroughout the duration of this event, users faced obstacles while attempting to initialize applications, in addition to experiencing sporadic instances of elevated latency and request failures.\\\\nGoogle offers a sincere apology for the operational interference this issue introduced to your organization. Recognizing the critical nature of Google Cloud to your daily operations, we deeply regret any resulting loss in productivity. Engineering efforts are currently focused on remediating the underlying cause to ensure future stability.\\\\n## Root Cause\\\\nThe disruption originated from failures in a critical cluster infrastructure component responsible for Google's service discovery and configuration management systems. This system facilitates resource synchronization, manages shared data access, and coordinates background tasks for numerous Google services.\\\\nA high volume of rapid configuration changes triggered the regional failure in Workspace Services, preventing Workspace servers from reliably processing traffic. To mitigate the impact, Google Workspace traffic was redirected to alternative regions.\\\\n## Remediation and Prevention\\\\nGoogle engineers were alerted to the issue via automated monitoring on Saturday, 30 May 2026 at 20:28 US/Pacific and immediately began an investigation. The investigation identified elevated error rates and high latency affecting multiple Google Workspace backend services. Upon investigation our engineering team confirmed errors were primarily originating from some regional servers due to coordination service failures in the region.\\\\nTo mitigate the impact, Google engineers redirected traffic away from the affected servers.\\\\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\\\\n- Prevent recurrence of this incident by rolling out the fix for the latent encryption library issue, which improves the handling of internal log compaction compactions during active file writes, to prevent recurrence of the cluster instability.\\\\n- Investigating additional resilience in our database layer.\\\\n## Detailed Description of Impact\\\\nOn 30 May 2026 from 20:15 to 21:30 US/Pacific, affected customers encountered difficulties while loading applications or experienced failures when performing tasks within the following products:\\\\n- Gmail\\\\n- Google Calendar\\\\n- Google Chat\\\\n- Google Docs\\\\n- Google Drive\\\\n- Google Meet\\\\n- Google Voice\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-06-02T03:50:13+00:00\\\",\\\"modified\\\":\\\"2026-06-04T20:11:26+00:00\\\",\\\"when\\\":\\\"2026-06-02T03:50:13+00:00\\\",\\\"text\\\":\\\"# Preliminary Incident Report\\\\nWe sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Workspace, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future.\\\\nPlease note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete.\\\\nIf you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using help article https://support.google.com/a/answer/1047213\\\\n#[REDACTED] Date/Time of the Issue (All time US/Pacific)\\\\nIncident Start: 30 May 2026 20:15\\\\nIncident End: 30 May 2026 21:30\\\\nDuration: 1 hour, 15 minutes\\\\n## Summary\\\\nStarting on Saturday, 30 May 2026 at 20:15 US/Pacific, multiple Google Workspace products experienced service disruptions for a duration of 1 hour and 15 minutes. The affected services included Google Voice, Google Meet, Google Drive, Google Chat, Google Calendar, Gmail, and Google Docs. During this incident, customers encountered difficulties loading applications, as well as intermittent high latency or errors.\\\\n## Preliminary Root Cause\\\\nThe disruption originated from failures in a regional infrastructure component responsible for Google's distributed lock service. This critical system facilitates resource synchronization, manages shared data access, and coordinates background tasks for numerous Google services.\\\\nA high volume of rapid configuration changes triggered the regional failure, preventing Workspace servers from reliably processing traffic. To mitigate the impact, Google Workspace traffic was redirected to alternative regions.\\\\nGoogle engineers have begun a full root cause analysis and will provide additional information once it is available.\\\\n## Remediation\\\\nGoogle engineers were alerted to the issue via automated monitoring at 20:28 US/Pacific on Saturday, 30 May 2026, and immediately began an investigation. The investigation identified elevated error rates and high latency affecting multiple Google Workspace backend services. Upon investigation our engineering team confirmed errors were primarily originating from some regional servers due to coordination service failures in the region.\\\\nTo mitigate the impact, Google engineers redirected traffic away from the affected servers.\\\\n## Description of Impact\\\\nAffected customers encountered difficulties loading applications or experienced failures when performing tasks within the following products:\\\\n* Gmail\\\\n* Google Calendar\\\\n* Google Chat\\\\n* Google Docs\\\\n* Google Drive\\\\n* Google Meet\\\\n* Google Voice\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-05-31T05:11:14+00:00\\\",\\\"modified\\\":\\\"2026-06-02T03:50:13+00:00\\\",\\\"when\\\":\\\"2026-05-31T05:11:14+00:00\\\",\\\"text\\\":\\\"**Title**\\\\n[Mitigated] Multiple Workspace Products experienced Issues.\\\\n**Description**\\\\nThe issue with Google Voice, Google Meet, Google Drive, Google Chat, Google Calendar, Gmail, Google Docs has been resolved for all affected users as of Saturday, 2026-05-30 21:30 PDT.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nCustomers may have experienced issues loading the application/ intermittent high latency or errors while using the application.\\\\n**Workaround**\\\\nNone at this time as the issue stands mitigated\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-05-31T04:59:04+00:00\\\",\\\"modified\\\":\\\"2026-05-31T05:11:14+00:00\\\",\\\"when\\\":\\\"2026-05-31T04:59:04+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nMultiple Workspace Products experiencing Issues.\\\\n**Description**\\\\nWe are experiencing an issue with Google Voice, Google Meet, Google Drive, Google Chat, Google Calendar, Gmail, Google Docs beginning on Saturday, 2026-05-30 20:28 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Saturday, 2026-05-30 22:10 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nCustomers may experience issues loading the application/ intermittent high latency or errors while using the application.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-06-04T20:11:26+00:00\\\",\\\"modified\\\":\\\"2026-06-04T20:11:27+00:00\\\",\\\"when\\\":\\\"2026-06-04T20:11:26+00:00\\\",\\\"text\\\":\\\"# Incident Report\\\\n## Summary\\\\nStarting at 20:15 US/Pacific on Saturday, 30 May 2026, various Google Workspace services encountered service-level disruptions lasting approximately 75 minutes. The scope of impact spanned several products, specifically affecting Gmail, Google Docs, Drive, Calendar, Chat, Meet, and Google Voice.\\\\nThroughout the duration of this event, users faced obstacles while attempting to initialize applications, in addition to experiencing sporadic instances of elevated latency and request failures.\\\\nGoogle offers a sincere apology for the operational interference this issue introduced to your organization. Recognizing the critical nature of Google Cloud to your daily operations, we deeply regret any resulting loss in productivity. Engineering efforts are currently focused on remediating the underlying cause to ensure future stability.\\\\n## Root Cause\\\\nThe disruption originated from failures in a critical cluster infrastructure component responsible for Google's service discovery and configuration management systems. This system facilitates resource synchronization, manages shared data access, and coordinates background tasks for numerous Google services.\\\\nA high volume of rapid configuration changes triggered the regional failure in Workspace Services, preventing Workspace servers from reliably processing traffic. To mitigate the impact, Google Workspace traffic was redirected to alternative regions.\\\\n## Remediation and Prevention\\\\nGoogle engineers were alerted to the issue via automated monitoring on Saturday, 30 May 2026 at 20:28 US/Pacific and immediately began an investigation. The investigation identified elevated error rates and high latency affecting multiple Google Workspace backend services. Upon investigation our engineering team confirmed errors were primarily originating from some regional servers due to coordination service failures in the region.\\\\nTo mitigate the impact, Google engineers redirected traffic away from the affected servers.\\\\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\\\\n- Prevent recurrence of this incident by rolling out the fix for the latent encryption library issue, which improves the handling of internal log compaction compactions during active file writes, to prevent recurrence of the cluster instability.\\\\n- Investigating additional resilience in our database layer.\\\\n## Detailed Description of Impact\\\\nOn 30 May 2026 from 20:15 to 21:30 US/Pacific, affected customers encountered difficulties while loading applications or experienced failures when performing tasks within the following products:\\\\n- Gmail\\\\n- Google Calendar\\\\n- Google Chat\\\\n- Google Docs\\\\n- Google Drive\\\\n- Google Meet\\\\n- Google Voice\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_DISRUPTION\\\",\\\"severity\\\":\\\"medium\\\",\\\"service_key\\\":\\\"zall\\\",\\\"service_name\\\":\\\"Multiple Products\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Gmail\\\",\\\"id\\\":\\\"prPt2Yra2CbGsbEm9cpC\\\"},{\\\"title\\\":\\\"Google Calendar\\\",\\\"id\\\":\\\"n5hVexuq1PM9onY9qrmo\\\"},{\\\"title\\\":\\\"Google Chat\\\",\\\"id\\\":\\\"Ht9Vx5PFzDPHYvrnrvSk\\\"},{\\\"title\\\":\\\"Google Docs\\\",\\\"id\\\":\\\"wNHuVFtZEWU5Mmj2eRCK\\\"},{\\\"title\\\":\\\"Google Drive\\\",\\\"id\\\":\\\"VHNA7p3Z5p3iakj5sA8V\\\"},{\\\"title\\\":\\\"Google Meet\\\",\\\"id\\\":\\\"sUH4BQXzYXma7NiS94Hi\\\"},{\\\"title\\\":\\\"Google Voice\\\",\\\"id\\\":\\\"gh79SrQk6wqonWoxPdCm\\\"}],\\\"uri\\\":\\\"incidents/vagmms1y6yRxwg6RMQZp\\\"},{\\\"id\\\":\\\"CzVZ5zk2Gf2ixPSt6RT3\\\",\\\"number\\\":\\\"17128406613861610004\\\",\\\"begin\\\":\\\"2026-05-21T07:00:00+00:00\\\",\\\"created\\\":\\\"2026-05-21T15:37:18+00:00\\\",\\\"end\\\":\\\"2026-05-21T15:47:00+00:00\\\",\\\"modified\\\":\\\"2026-05-21T16:10:17+00:00\\\",\\\"external_desc\\\":\\\"**Summary**\\\\nGmail users are experiencing delayed delivery of messages sent from the bank.in domains.\\\\n**Description**\\\\nWe are experiencing an issue with Gmail beginning on Thursday, 2026-05-21 at 00:00 PDT.\\\\nMitigation efforts are currently underway by our engineering team.\\\\nWe will provide the next update by Thursday, 2026-05-21 at 09:30 PDT.\\\\n**Diagnosis / Customer Symptoms**\\\\nGmail users are experiencing delayed delivery of messages sent from the bank.in domains.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-05-21T16:09:55+00:00\\\",\\\"modified\\\":\\\"2026-05-21T16:10:17+00:00\\\",\\\"when\\\":\\\"2026-05-21T16:09:55+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGmail users experienced delayed delivery of messages sent from the bank.in domains.\\\\n**Description**\\\\nOur internal telemetry shows that the issue with Gmail has been resolved for all affected users as of Thursday, 2026-05-21 at 08:47 US/Pacific.\\\\nOur engineering team is actively monitoring the system to ensure continued stability.\\\\n**Diagnosis / Customer Symptoms**\\\\nCustomers should expect new messages to be delivered normally, older emails could take some time as the sender\n\n[...truncated 13350 characters...]\n\nou for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nImpacted users might have experienced authentication and sync failures while using Microsoft Exchange Online accounts through Gmail App on Android devices.\\\\n**Workaround**\\\\nUpdate to the latest version of Gmail Android (2026.05.11) in Google Play Store.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-05-22T00:04:40+00:00\\\",\\\"modified\\\":\\\"2026-05-22T18:57:24+00:00\\\",\\\"when\\\":\\\"2026-05-22T00:04:40+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGmail Android users using Microsoft Exchange Online accounts may fail to Log In and Sync due to authentication issues.\\\\n**Description**\\\\nOur engineering team identified the issue and released a new version of the Gmail Android application (version 2026.05.11) to the Google Play Store, which contains a fix to unblock affected users.\\\\nIf you are currently encountering this issue, please upgrade to the latest version of the Gmail Android app. Users operating on the new app version should be able to successfully log in and sync their accounts.\\\\nWe will provide more information by Friday, 2026-05-22 17:00 US/Pacific with current details.\\\\nWe thank you for your patience while we are working on resolving the issue.\\\\n**Symptoms**\\\\nImpacted users may experience authentication and sync failures while using Microsoft Exchange Online accounts through Gmail App on Android devices.\\\\n**Workaround**\\\\nAffected users could update to the latest version of Gmail Android (version 2026.05.11) in Google Play Store.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-19T21:55:32+00:00\\\",\\\"modified\\\":\\\"2026-05-22T00:04:40+00:00\\\",\\\"when\\\":\\\"2026-05-19T21:55:32+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGmail Android users using Microsoft Exchange Online accounts may fail to Log In and Sync due to authentication issues.\\\\n**Description**\\\\nWe are experiencing an issue with Gmail Android client connecting with Microsoft Exchange Online accounts beginning on Saturday 2026-05-06.\\\\nOur engineering team has identified the issue and started rolling out a new version of Gmail Android release that contains the fix to unblock users.\\\\nUsers already on the new app version will be able to Log In and Sync their accounts successfully.\\\\nWe will provide more information by Thursday, 2026-05-21 17:00 US/Pacific with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Customer Symptoms**\\\\nImpacted users may experience authentication and sync failures while using Microsoft Exchange Online accounts through Gmail App on Android devices.\\\\n**Workaround**\\\\nAffected users can use Microsoft Web App to access the Inbox.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-18T21:45:35+00:00\\\",\\\"modified\\\":\\\"2026-05-19T21:55:32+00:00\\\",\\\"when\\\":\\\"2026-05-18T21:45:35+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGmail Android users using Microsoft Exchange Online accounts may fail to Log In and Sync due to authentication issues.\\\\n**Description**\\\\nWe are experiencing an issue with Gmail Android client connecting with Microsoft Exchange Online accounts beginning on Saturday 2026-05-06.\\\\nOur engineering team has identified the issue and is actively working on rolling out a fix to unblock all the affected customers.\\\\nWe will provide more information by Tuesday, 2026-05-19 15:00 US/Pacific with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Customer Symptoms**\\\\nImpacted users may experience authentication and sync failures while using Microsoft Exchange Online accounts through Gmail App on Android devices.\\\\n**Workaround**\\\\nAffected users can use Microsoft Web App to access the Inbox.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-16T02:05:03+00:00\\\",\\\"modified\\\":\\\"2026-05-18T21:45:35+00:00\\\",\\\"when\\\":\\\"2026-05-16T02:05:03+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGmail Android users using Microsoft Exchange Online may fail to log in due to authentication issues.\\\\n**Description**\\\\nWe are experiencing an issue with Gmail beginning on Saturday 2026-05-06.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide more information by Monday, 2026-05-18 15:00 US/Pacific with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Customer Symptoms**\\\\nImpacted users may experience authentication failures while logging in to Microsoft Exchange Online through\\\\nGmail App on Android devices.\\\\n**Workaround**\\\\nAffected users can use Microsoft web app to access the Inbox.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-05-22T18:57:24+00:00\\\",\\\"modified\\\":\\\"2026-05-22T18:57:41+00:00\\\",\\\"when\\\":\\\"2026-05-22T18:57:24+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\n[Resolved] Gmail Android users utilizing Microsoft Exchange Online accounts experienced authentication and synchronization failures.\\\\n**Description**\\\\nThe issue with Gmail has been resolved in the new Gmail android release (version 2026.05.11) as of Thursday, 2026-05-21 15:00 PDT.\\\\nOur engineering team identified the issue and released a new version of the Gmail Android application to the Google Play Store, which contains a fix to unblock affected users.\\\\nIf you are currently encountering this issue, please upgrade to the latest version of the Gmail Android app. Users operating on the new app version will be able to successfully log in and sync their accounts.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nImpacted users might have experienced authentication and sync failures while using Microsoft Exchange Online accounts through Gmail App on Android devices.\\\\n**Workaround**\\\\nUpdate to the latest version of Gmail Android (2026.05.11) in Google Play Store.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_INFORMATION\\\",\\\"severity\\\":\\\"low\\\",\\\"service_key\\\":\\\"prPt2Yra2CbGsbEm9cpC\\\",\\\"service_name\\\":\\\"Gmail\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Gmail\\\",\\\"id\\\":\\\"prPt2Yra2CbGsbEm9cpC\\\"}],\\\"uri\\\":\\\"incidents/kWSuRqLdsXGzT7Qxjdmz\\\"},{\\\"id\\\":\\\"1hr418kjGyBZ9ugXy3W3\\\",\\\"number\\\":\\\"5513210214958819667\\\",\\\"begin\\\":\\\"2026-05-04T16:40:00+00:00\\\",\\\"created\\\":\\\"2026-05-05T23:35:49+00:00\\\",\\\"end\\\":\\\"2026-05-08T18:53:00+00:00\\\",\\\"modified\\\":\\\"2026-05-08T18:58:23+00:00\\\",\\\"external_desc\\\":\\\"**Summary**\\\\nInvestigating elevated error rates with Gemini conversations.\\\\n**Description**\\\\nMitigation work is currently underway by our engineering team.\\\\nWe do not have an ETA for mitigation at this point.\\\\nWe will provide more information by Tuesday, 2026-05-05 20:30 PDT.\\\\n**Symptoms**\\\\nAffected customers are experiencing \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-05-08T18:58:09+00:00\\\",\\\"modified\\\":\\\"2026-05-08T18:58:23+00:00\\\",\\\"when\\\":\\\"2026-05-08T18:58:09+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGemini conversations may have experienced elevated error rates.\\\\n**Description**\\\\nThe issue with Gemini has been resolved for all affected users as of Friday, 2026-05-08 11:53 PDT.\\\\nThe issue was triggered due to a combination of insufficient database resources, and application bugs. The database capacity issues were resolved, and further code changes are being made to mitigate the application issue.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nAffected customers may have experienced \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-05-08T15:53:13+00:00\\\",\\\"modified\\\":\\\"2026-05-08T18:58:09+00:00\\\",\\\"when\\\":\\\"2026-05-08T15:53:13+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGemini conversations may experience elevated error rates.\\\\n**Description**\\\\nOur engineering team has isolated the underlying issue and is in the process of reverting the specific code deployments that triggered the impact.\\\\nWe are actively monitoring the progress of the rollback, alongside system health indicators, to confirm that the restoration is proceeding effectively.\\\\nWe will provide more information by Friday, 2026-05-08 12:00 PDT.\\\\n**Symptoms**\\\\nAffected customers may encounter \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-08T10:23:42+00:00\\\",\\\"modified\\\":\\\"2026-05-08T15:53:13+00:00\\\",\\\"when\\\":\\\"2026-05-08T10:23:42+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGemini conversations may experience elevated error rates.\\\\n**Description**\\\\nOur engineering team has isolated the underlying issue and is in the process of reverting the specific code deployments that triggered the impact.\\\\nWe are actively monitoring the progress of the rollback, alongside system health indicators, to confirm that the restoration is proceeding effectively.\\\\nWe will provide more information by Friday, 2026-05-08 09:00 PDT.\\\\n**Symptoms**\\\\nAffected customers may encounter \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-08T07:29:25+00:00\\\",\\\"modified\\\":\\\"2026-05-08T10:23:42+00:00\\\",\\\"when\\\":\\\"2026-05-08T07:29:25+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGemini conversations may experience elevated error rates.\\\\n**Description**\\\\nOur engineering team has isolated the underlying issue and is in the process of reverting the specific code deployments that triggered the impact.\\\\nWe are actively monitoring the progress of the rollback, alongside system health indicators, to confirm that the restoration is proceeding effectively.\\\\nWe will provide more information by Friday, 2026-05-08 04:00 PDT.\\\\n**Symptoms**\\\\nAffected customers may encounter \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-08T05:16:23+00:00\\\",\\\"modified\\\":\\\"2026-05-08T07:29:25+00:00\\\",\\\"when\\\":\\\"2026-05-08T05:16:23+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGemini conversations are experiencing elevated error rates.\\\\n**Description**\\\\nOur engineering team has determined that further investigation is required to mitigate the issue.\\\\nWe do not have an ETA for mitigation at this point.\\\\nWe will provide more information by Friday, 2026-05-08 01:00 PDT.\\\\n**Symptoms**\\\\nAffected customers may encounter \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-08T01:00:01+00:00\\\",\\\"modified\\\":\\\"2026-05-08T05:16:23+00:00\\\",\\\"when\\\":\\\"2026-05-08T01:00:01+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGemini conversations may have experienced elevated error rates.\\\\n**Description**\\\\nThe issue with Gemini has been resolved for all affected users as of Thursday, 2026-05-07 17:00 PDT.\\\\nThe issue was triggered due to a combination of insufficient database resources, and application issues. The database capacity issues were resolved, and further code changes are being made to mitigate the application issues.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nAffected customers may have experienced the \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-07T23:52:49+00:00\\\",\\\"modified\\\":\\\"2026-05-08T01:00:01+00:00\\\",\\\"when\\\":\\\"2026-05-07T23:52:49+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGemini conversations are experiencing elevated error rates.\\\\n**Description**\\\\nOur engineering team has determined that further investigation is required to mitigate the issue.\\\\nWe do not have an ETA for mitigation at this point.\\\\nWe will provide more information by Thursday, 2026-05-07 20:30 PDT.\\\\n**Symptoms**\\\\nAffected customers are experiencing \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-06T19:11:02+00:00\\\",\\\"modified\\\":\\\"2026-05-07T23:52:49+00:00\\\",\\\"when\\\":\\\"2026-05-06T19:11:02+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGemini conversations are experiencing elevated error rates.\\\\n**Description**\\\\nOur engineering team has determined that further investigation is required to mitigate the issue.\\\\nWe do not have an ETA for mitigation at this point.\\\\nWe will provide more information by Thursday, 2026-05-07 17:00 PDT.\\\\n**Symptoms**\\\\nAffected customers are experiencing \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-06T03:22:40+00:00\\\",\\\"modified\\\":\\\"2026-05-06T19:11:02+00:00\\\",\\\"when\\\":\\\"2026-05-06T03:22:40+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nInvestigating elevated error rates with Gemini conversations.\\\\n**Description**\\\\nMitigation work is currently underway by our engineering team.\\\\nWe do not have an ETA for mitigation at this point.\\\\nWe will provide more information by Wednesday, 2026-05-06 09:00 PDT.\\\\n**Symptoms**\\\\nAffected customers are experiencing \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-05T23:35:49+00:00\\\",\\\"modified\\\":\\\"2026-05-06T03:22:40+00:00\\\",\\\"when\\\":\\\"2026-05-05T23:35:49+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nInvestigating elevated error rates with Gemini conversations.\\\\n**Description**\\\\nMitigation work is currently underway by our engineering team.\\\\nWe do not have an ETA for mitigation at this point.\\\\nWe will provide more information by Tuesday, 2026-05-05 20:30 PDT.\\\\n**Symptoms**\\\\nAffected customers are experiencing \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-05-08T18:58:09+00:00\\\",\\\"modified\\\":\\\"2026-05-08T18:58:23+00:00\\\",\\\"when\\\":\\\"2026-05-08T18:58:09+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGemini conversations may have experienced elevated error rates.\\\\n**Description**\\\\nThe issue with Gemini has been resolved for all affected users as of Friday, 2026-05-08 11:53 PDT.\\\\nThe issue was triggered due to a combination of insufficient database resources, and application bugs. The database capacity issues were resolved, and further code changes are being made to mitigate the application issue.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nAffected customers may have experienced \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_INFORMATION\\\",\\\"severity\\\":\\\"low\\\",\\\"service_key\\\":\\\"npdyhgECDJ6tB66MxXyo\\\",\\\"service_name\\\":\\\"Gemini\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Gemini\\\",\\\"id\\\":\\\"npdyhgECDJ6tB66MxXyo\\\"}],\\\"uri\\\":\\\"incidents/1hr418kjGyBZ9ugXy3W3\\\"},{\\\"id\\\":\\\"GDxbSoojBb9yyEhU2Cua\\\",\\\"number\\\":\\\"5147219210877965776\\\",\\\"begin\\\":\\\"2026-04-29T18:30:00+00:00\\\",\\\"created\\\":\\\"2026-04-29T19:17:28+00:00\\\",\\\"end\\\":\\\"2026-04-29T20:45:00+00:00\\\",\\\"modified\\\":\\\"2026-04-29T21:07:19+00:00\\\",\\\"external_desc\\\":\\\"**Title:**\\\\nInvestigating issues with Classroom.\\\\n**Description:**\\\\nWe are experiencing an issue with Classroom beginning at Wednesday, 2026-04-29 11:30 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide more information by Wednesday, 2026-04-29 13:00 PDT.\\\\n**Symptoms:**\\\\nAffected users may experience issues accessing classroom.\\\\n**Workaround:**\\\\nNo workarounds available at this time.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-04-29T21:07:19+00:00\\\",\\\"modified\\\":\\\"2026-04-29T21:07:20+00:00\\\",\\\"when\\\":\\\"2026-04-29T21:07:19+00:00\\\",\\\"text\\\":\\\"**Title:**\\\\nAn issue with Classroom is resolved.\\\\n**Description:**\\\\nA fix has been deployed, and services have stabilized for all users.\\\\nThe issue with Google Classroom has been resolved for all affected users as of Wednesday, 2026-04-29 13:45 PDT.\\\\nThe issue was due to an internal software migration. Our engineers have mitigated the issue by rolling back the migration for impacted users.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms:**\\\\nAffected users may have experienced issues accessing the classroom.\\\\n**Workaround:**\\\\nThe issue is resolved.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-04-29T20:13:38+00:00\\\",\\\"modified\\\":\\\"2026-04-29T21:07:19+00:00\\\",\\\"when\\\":\\\"2026-04-29T20:13:38+00:00\\\",\\\"text\\\":\\\"**Title:**\\\\nInvestigating issues with Classroom.\\\\n**Description:**\\\\nA fix has been deployed, and services have stabilized for the majority of users.\\\\nWe are aware that a small subset of users may still encounter errors. Our engineering team is actively working on a secondary resolution to address these remaining users.\\\\nFull resolution for remaining users is expected to complete by Wednesday, 2026-04-29 14:00 PDT.\\\\nWe will provide an update by Wednesday, 2026-04-29 14:15 PDT with current details.\\\\n**Symptoms:**\\\\nAffected users may experience issues accessing classroom.\\\\n**Workaround:**\\\\nNo workarounds available at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-04-29T19:53:46+00:00\\\",\\\"modified\\\":\\\"2026-04-29T20:13:38+00:00\\\",\\\"when\\\":\\\"2026-04-29T19:53:46+00:00\\\",\\\"text\\\":\\\"**Title:**\\\\nInvestigating issues with Classroom.\\\\n**Description:**\\\\nOur engineering team has identified the cause and working on mitigation of the issue .\\\\nThe mitigation is expected to complete by Wednesday, 2026-04-29 15:00 PDT.\\\\nWe will provide more information by Wednesday, 2026-04-29 15:15 PDT.\\\\n**Symptoms:**\\\\nAffected users may experience issues accessing classroom.\\\\n**Workaround:**\\\\nNo workarounds available at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-04-29T19:17:28+00:00\\\",\\\"modified\\\":\\\"2026-04-29T19:53:46+00:00\\\",\\\"when\\\":\\\"2026-04-29T19:17:28+00:00\\\",\\\"text\\\":\\\"**Title:**\\\\nInvestigating issues with Classroom.\\\\n**Description:**\\\\nWe are experiencing an issue with Classroom beginning at Wednesday, 2026-04-29 11:30 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide more information by Wednesday, 2026-04-29 13:00 PDT.\\\\n**Symptoms:**\\\\nAffected users may experience issues accessing classroom.\\\\n**Workaround:**\\\\nNo workarounds available at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-04-29T21:07:19+00:00\\\",\\\"modified\\\":\\\"2026-04-29T21:07:20+00:00\\\",\\\"when\\\":\\\"2026-04-29T21:07:19+00:00\\\",\\\"text\\\":\\\"**Title:**\\\\nAn issue with Classroom is resolved.\\\\n**Description:**\\\\nA fix has been deployed, and services have stabilized for all users.\\\\nThe issue with Google Classroom has been resolved for all affected users as of Wednesday, 2026-04-29 13:45 PDT.\\\\nThe issue was due to an internal software migration. Our engineers have mitigated the issue by rolling back the migration for impacted users.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms:**\\\\nAffected users may have experienced issues accessing the classroom.\\\\n**Workaround:**\\\\nThe issue is resolved.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_INFORMATION\\\",\\\"severity\\\":\\\"low\\\",\\\"service_key\\\":\\\"3GoNVrVz48jeNZSyLJMV\\\",\\\"service_name\\\":\\\"Classroom\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Classroom\\\",\\\"id\\\":\\\"3GoNVrVz48jeNZSyLJMV\\\"}],\\\"uri\\\":\\\"incidents/GDxbSoojBb9yyEhU2Cua\\\"},{\\\"id\\\":\\\"tVLvwJSkYhEtcLAMsekv\\\",\\\"number\\\":\\\"6658602565448188991\\\",\\\"begin\\\":\\\"2026-04-28T07:30:00+00:00\\\",\\\"created\\\":\\\"2026-04-28T15:20:36+00:00\\\",\\\"end\\\":\\\"2026-04-28T15:50:00+00:00\\\",\\\"modified\\\":\\\"2026-04-28T16:50:51+00:00\\\",\\\"external_desc\\\":\\\"**Title:**\\\\nInvestigating issues with slow loading of AppSheet\\\\n**Description:**\\\\nWe are experiencing an issue with AppSheet beginning at Tuesday, 2026-04-28 00:30 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Tuesday, 2026-04-28 09:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms:**\\\\nAffected customers would experience slow loading times and internal server errors for AppSheet apps.\\\\n**Workaround:**\\\\nNone at this time.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-04-28T16:50:32+00:00\\\",\\\"modified\\\":\\\"2026-04-28T16:50:51+00:00\\\",\\\"when\\\":\\\"2026-04-28T16:50:32+00:00\\\",\\\"text\\\":\\\"**Title:**\\\\nMitigated issues with slow loading of AppSheet apps.\\\\n**Description:**\\\\nThe issue with AppSheet has been resolved for all affected users as of Tuesday, 2026-04-28 08:50 PDT.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms:**\\\\nAffected customers would have experienced slow loading times and internal server errors for AppSheet apps.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-04-28T16:00:44+00:00\\\",\\\"modified\\\":\\\"2026-04-28T16:50:32+00:00\\\",\\\"when\\\":\\\"2026-04-28T16:00:44+00:00\\\",\\\"text\\\":\\\"**Title:**\\\\nInvestigating issues with slow loading of AppSheet.\\\\n**Description:**\\\\nOur engineering team has implemented a fix and system telemetry indicates that latency and error rates have returned to normal levels. We are continuing to monitor the environment closely to ensure full stability.\\\\nWe will provide an update by Tuesday, 2026-04-28 10:30 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms:**\\\\nAffected customers would experience slow loading times and internal server errors for AppSheet apps.\\\\n**Workaround:**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-04-28T15:20:36+00:00\\\",\\\"modified\\\":\\\"2026-04-28T16:00:44+00:00\\\",\\\"when\\\":\\\"2026-04-28T15:20:36+00:00\\\",\\\"text\\\":\\\"**Title:**\\\\nInvestigating issues with slow loading of AppSheet\\\\n**Description:**\\\\nWe are experiencing an issue with AppSheet beginning at Tuesday, 2026-04-28 00:30 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Tuesday, 2026-04-28 09:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms:**\\\\nAffected customers would experience slow loading times and internal server errors for AppSheet apps.\\\\n**Workaround:**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-04-28T16:50:32+00:00\\\",\\\"modified\\\":\\\"2026-04-28T16:50:51+00:00\\\",\\\"when\\\":\\\"2026-04-28T16:50:32+00:00\\\",\\\"text\\\":\\\"**Title:**\\\\nMitigated issues with slow loading of AppSheet apps.\\\\n**Description:**\\\\nThe issue with AppSheet has been resolved for all affected users as of Tuesday, 2026-04-28 08:50 PDT.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms:**\\\\nAffected customers would have experienced slow loading times and internal server errors for AppSheet apps.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_INFORMATION\\\",\\\"severity\\\":\\\"low\\\",\\\"service_key\\\":\\\"FWjKi5U7KX4FUUPThHAJ\\\",\\\"service_name\\\":\\\"AppSheet\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"AppSheet\\\",\\\"id\\\":\\\"FWjKi5U7KX4FUUPThHAJ\\\"}],\\\"uri\\\":\\\"incidents/tVLvwJSkYhEtcLAMsekv\\\"},{\\\"id\\\":\\\"JTgxXCtXTaw3aEVzUUfa\\\",\\\"number\\\":\\\"3709370212894413327\\\",\\\"begin\\\":\\\"2026-04-10T02:08:00+00:00\\\",\\\"created\\\":\\\"2026-04-10T10:07:12+00:00\\\",\\\"end\\\":\\\"2026-04-10T14:50:00+00:00\\\",\\\"modified\\\":\\\"2026-04-10T15:06:53+00:00\\\",\\\"external_desc\\\":\\\"**Summary**\\\\nGoogle Workspace users using Google IdP are unable to sign into their accounts using security keys and hardware based passkeys.\\\\n**Description**\\\\nWe are experiencing an issue with Workspace users beginning on Thursday, 2026-04-09 19:08 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Friday, 2026-04-10 04:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Customer Symptoms**\\\\nGoogle Workspace users using Google IdP are unable to sign into their accounts using security keys and hardware based passkeys.\\\\n**Workaround**\\\\nBelow steps should be performed by Admin as the workaround:\\\\n- Admin should revoke the key.\\\\n- Admin to create the verification code to sign-in.\\\\n- The user should sign-in using Backup verification code and then re-enrolls the security key.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-04-10T15:06:51+00:00\\\",\\\"modified\\\":\\\"2026-04-10T15:06:53+00:00\\\",\\\"when\\\":\\\"2026-04-10T15:06:51+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGoogle Workspace users using Google IdP were unable to sign into their accounts using security keys and hardware based passkeys.\\\\n**Description**\\\\nThe issue with Admin Console has been resolved for all affected users as of Friday, 2026-04-10 07:50 PDT.\\\\nSuspected CL have been rolled back and we are seeing in our graphs errors rate have gone down. Customers also confirmed that issues have been mitigated.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Customer Symptoms**\\\\nGoogle Workspace users using Google IdP were unable to sign into their accounts using security keys and hardware based passkeys.\\\\n**Workaround**\\\\nBelow steps should be performed by Admin as the workaround:\\\\n* Admin should revoke the key.\\\\n* Admin to create the verification code to sign-in.\\\\n* The user should sign-in using Backup verification code and then re-enrolls the security key.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-04-10T11:27:56+00:00\\\",\\\"modified\\\":\\\"2026-04-10T15:06:51+00:00\\\",\\\"when\\\":\\\"2026-04-10T11:27:56+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGoogle Workspace users using Google IdP are unable to sign into their accounts using security keys and hardware based passkeys.\\\\n**Description**\\\\nMitigation work is currently underway by our engineering team.\\\\nWe do not have an ETA for mitigation at this point.\\\\nWe will provide more information by Friday, 2026-04-10 09:00 PDT.\\\\n**Customer Symptoms**\\\\nGoogle Workspace users using Google IdP are unable to sign into their accounts using security keys and hardware based passkeys.\\\\n**Workaround**\\\\nBelow steps should be performed by Admin as the workaround:\\\\n- Admin should revoke the key.\\\\n- Admin to create the verification code to sign-in.\\\\n- The user should sign-in using Backup verification code and then re-enrolls the security key.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-04-10T10:55:50+00:00\\\",\\\"modified\\\":\\\"2026-04-10T11:27:56+00:00\\\",\\\"when\\\":\\\"2026-04-10T10:55:50+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGoogle Workspace users using Goog\n[...truncated]",
"error": "",
"elapsed": 0.025531217
}
The redacted, numbered record a structured judge grades. Check rows above link to the IDs they cite.
{"dispatch_id": "toolu_01FYfmMV6r2KJBtHG3c86jk4", "elapsed": 0.025531217, "error": "", "result": "{\"content\":\"[{\\\"id\\\":\\\"fJ4691Pe2g9KsTbP8g5F\\\",\\\"number\\\":\\\"11425782758836097339\\\",\\\"begin\\\":\\\"2026-09-11T13:00:00+00:00\\\",\\\"created\\\":\\\"2026-09-11T20:45:19+00:00\\\",\\\"end\\\":\\\"2026-09-11T20:22:00+00:00\\\",\\\"modified\\\":\\\"2026-09-11T20:45:42+00:00\\\",\\\"external_desc\\\":\\\"**Summary:**\\\\nMitigated intermittent elevated error rates with Gemini 3.1 Pro models on Gemini Web and Gemini App.\\\\n**Description:**\\\\nBetween 6:00 PDT and 13:22 PDT on 11 September 2026, a subset of Gemini users may have experienced elevated error rates with Gemini 3.1 Pro models on Gemini Web and Gemini App.\\\\nFrom preliminary analysis, the issue was triggered by a recent rollout. Our engineers identified the cause and mitigated it with a code change.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms:**\\\\nCustomers may have encountered intermittent errors when using Gemini 3.1 Pro models on Gemini Web and Gemini App.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-09-11T20:45:19+00:00\\\",\\\"modified\\\":\\\"2026-09-11T20:45:42+00:00\\\",\\\"when\\\":\\\"2026-09-11T20:45:19+00:00\\\",\\\"text\\\":\\\"**Summary:**\\\\nMitigated intermittent elevated error rates with Gemini 3.1 Pro models on Gemini Web and Gemini App.\\\\n**Description:**\\\\nBetween 6:00 PDT and 13:22 PDT on 11 September 2026, a subset of Gemini users may have experienced elevated error rates with Gemini 3.1 Pro models on Gemini Web and Gemini App.\\\\nFrom preliminary analysis, the issue was triggered by a recent rollout. Our engineers identified the cause and mitigated it with a code change.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms:**\\\\nCustomers may have encountered intermittent errors when using Gemini 3.1 Pro models on Gemini Web and Gemini App.\\\",\\\"status\\\":\\\"AVAILABLE\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-09-11T20:45:19+00:00\\\",\\\"modified\\\":\\\"2026-09-11T20:45:42+00:00\\\",\\\"when\\\":\\\"2026-09-11T20:45:19+00:00\\\",\\\"text\\\":\\\"**Summary:**\\\\nMitigated intermittent elevated error rates with Gemini 3.1 Pro models on Gemini Web and Gemini App.\\\\n**Description:**\\\\nBetween 6:00 PDT and 13:22 PDT on 11 September 2026, a subset of Gemini users may have experienced elevated error rates with Gemini 3.1 Pro models on Gemini Web and Gemini App.\\\\nFrom preliminary analysis, the issue was triggered by a recent rollout. Our engineers identified the cause and mitigated it with a code change.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms:**\\\\nCustomers may have encountered intermittent errors when using Gemini 3.1 Pro models on Gemini Web and Gemini App.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_INFORMATION\\\",\\\"severity\\\":\\\"low\\\",\\\"service_key\\\":\\\"npdyhgECDJ6tB66MxXyo\\\",\\\"service_name\\\":\\\"Gemini\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Gemini\\\",\\\"id\\\":\\\"npdyhgECDJ6tB66MxXyo\\\"}],\\\"uri\\\":\\\"incidents/fJ4691Pe2g9KsTbP8g5F\\\"},{\\\"id\\\":\\\"y2ufhCB886p7rveYNnko\\\",\\\"number\\\":\\\"1967299596760361422\\\",\\\"begin\\\":\\\"2026-08-28T17:50:00+00:00\\\",\\\"created\\\":\\\"2026-08-28T19:25:37+00:00\\\",\\\"end\\\":\\\"2026-08-28T20:34:00+00:00\\\",\\\"modified\\\":\\\"2026-09-04T03:43:27+00:00\\\",\\\"external_desc\\\":\\\"**Summary:**\\\\nGoogle Chat users are experiencing an issue where they receive notifications for new messages, but the actual messages fail to appear in their chat application unless the user refreshes their view.\\\\n**Description:**\\\\nWe are experiencing an intermittent issue with Google Chat Friday, 2026-08-28 10:25 PDT.\\\\nOur engineering team has investigated the issue and are currently working on mitigating the issue.\\\\nWe will provide more information by Friday, 2026-08-28 14:15 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Customer Symptoms:**\\\\nUsers are experiencing an issue where they receive notifications for new messages, but the actual messages fail to appear in their chat application unless the user refreshes their view\\\\n**Workaround:**\\\\nUsers can refresh to see new messages\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-09-04T03:43:07+00:00\\\",\\\"modified\\\":\\\"2026-09-04T03:43:27+00:00\\\",\\\"when\\\":\\\"2026-09-04T03:43:07+00:00\\\",\\\"text\\\":\\\"## Incident Report\\\\n## Summary\\\\nOn Friday, August 28, 2026, between 10:50 US/Pacific and 13:34 US/Pacific for a duration of 2 hours and 44 minutes, Google Chat users experienced delays in receiving real-time incoming messages within active conversation streams across web, desktop, and mobile applications. While users continued to receive notifications for new messages, the message contents did not automatically render in active conversation streams until the user manually refreshed the application or sent a reply message.\\\\nTo our valued Google Workspace customers, we are deeply sorry for the service interruption you experienced. We are taking immediate and decisive action to address the issue and strengthen our systems against future incidents.\\\\n## Root Cause\\\\nThe incident was triggered during a global database schema rollout. An incompatible database schema change caused database writes required for message delivery to start failing.\\\\nSpecifically, a mechanism for schema synchronization—designed to ensure high consistency between primary storage and live message delivery—encountered a rare race condition bug. This caused backend message dispatchers to attempt writes against an unready schema state, leading to write failures.\\\\nBecause live message delivery had an active dependency on these writes completing successfully before pushing events to user screens, the failure caused incoming messages to experience synchronization delays. Separate push notification services and core message storage operated independently and were unaffected, which is why users still received alerts for new messages while active conversation streams experienced delays.\\\\n## Remediation and Prevention\\\\nThe issue was detected through automated backend alerting on dispatch error rates, alongside corresponding customer reports. Google Engineering initiated parallel workstreams to restore service.\\\\n* At 12:44 US/Pacific, Google Engineering directly mitigated the source of the issue by removing the dependency that live message delivery had on the failing database write. This allowed new incoming messages to bypass the failed write and immediately resume normal real-time delivery to user screens.\\\\n* At 13:00 US/Pacific, Engineering halted and reverted the database schema change to its previous verified stable version, restoring full database compatibility.\\\\n* By 13:34 US/Pacific, as queued messages were processed, real-time message synchronization delays returned to 0% across all customer domains. Engineering verified that full service health was restored with zero message loss.\\\\nTo prevent this from happening again, we are executing the following preventative actions:\\\\n* **Hardened internal detection:** Strengthen automated validation and verification to cover this specific failure scenario across pre-production and internal rollout stages. This aims to catch and mitigate schema incompatibilities and synchronization edge cases internally before any customer traffic can be impacted in the future.\\\\n* **Resilience Against Race Conditions:** Implement Architectural improvements to make the schema synchronization and message delivery architecture resilient to race conditions. This aims to ensure that background schema and config synchronization inconsistencies cannot disrupt real-time message delivery.\\\\n \\\\n## Detailed Description of Impact\\\\nThe incident impact window occurred between 10:50 US/Pacific and 13:34 US/Pacific on Friday, 28 August 2026\\\\\\\\. This issue applied globally to all Google Chat environments.\\\\n**Google Chat:**\\\\n* **Delayed message rendering:** New messages did not appear in active conversation streams automatically. Users were required to manually refresh their client or send a new message/react to force the stream to synchronize.\\\\n* **Stalled read states and inbox sorting:** Background actions, such as updating unread badges and moving active spaces to the top of the inbox, experienced heavy internal error rates and were significantly delayed for users.\\\\n* **Search indexing delays:** New messages sent during the outage window stalled in the indexing pipeline, experiencing delays in becoming searchable.\\\\n**Scope Exclusions:**\\\\n* **No Data Loss:** At no point was customer data lost or compromised. All messages transmitted during the incident window were securely captured in primary storage and successfully delivered to conversation streams once the backlog cleared.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-08-28T20:55:51+00:00\\\",\\\"modified\\\":\\\"2026-09-04T03:43:07+00:00\\\",\\\"when\\\":\\\"2026-08-28T20:55:51+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGoogle Chat users experienced an issue where they received notifications for new messages, but the actual messages failed to appear in their chat application unless the user refreshed their view.\\\\n**Description**\\\\nThe issue with Google Chat has been resolved for all affected users as of Friday, 2026-08-28 13:45 PDT.\\\\nFrom preliminary analysis the issue was triggered due to a recent change which was rolled back to mitigate the impact.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Customer Symptoms**\\\\nUsers were experiencing an issue where they received notifications for new messages, but the actual messages failed to appear in their chat application unless the user refreshed their view.\\\\n**Workaround**\\\\nNone as the issue is resolved.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-08-28T19:25:37+00:00\\\",\\\"modified\\\":\\\"2026-08-28T20:55:51+00:00\\\",\\\"when\\\":\\\"2026-08-28T19:25:37+00:00\\\",\\\"text\\\":\\\"**Summary:**\\\\nGoogle Chat users are experiencing an issue where they receive notifications for new messages, but the actual messages fail to appear in their chat application unless the user refreshes their view.\\\\n**Description:**\\\\nWe are experiencing an intermittent issue with Google Chat Friday, 2026-08-28 10:25 PDT.\\\\nOur engineering team has investigated the issue and are currently working on mitigating the issue.\\\\nWe will provide more information by Friday, 2026-08-28 14:15 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Customer Symptoms:**\\\\nUsers are experiencing an issue where they receive notifications for new messages, but the actual messages fail to appear in their chat application unless the user refreshes their view\\\\n**Workaround:**\\\\nUsers can refresh to see new messages\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-09-04T03:43:07+00:00\\\",\\\"modified\\\":\\\"2026-09-04T03:43:27+00:00\\\",\\\"when\\\":\\\"2026-09-04T03:43:07+00:00\\\",\\\"text\\\":\\\"## Incident Report\\\\n## Summary\\\\nOn Friday, August 28, 2026, between 10:50 US/Pacific and 13:34 US/Pacific for a duration of 2 hours and 44 minutes, Google Chat users experienced delays in receiving real-time incoming messages within active conversation streams across web, desktop, and mobile applications. While users continued to receive notifications for new messages, the message contents did not automatically render in active conversation streams until the user manually refreshed the application or sent a reply message.\\\\nTo our valued Google Workspace customers, we are deeply sorry for the service interruption you experienced. We are taking immediate and decisive action to address the issue and strengthen our systems against future incidents.\\\\n## Root Cause\\\\nThe incident was triggered during a global database schema rollout. An incompatible database schema change caused database writes required for message delivery to start failing.\\\\nSpecifically, a mechanism for schema synchronization—designed to ensure high consistency between primary storage and live message delivery—encountered a rare race condition bug. This caused backend message dispatchers to attempt writes against an unready schema state, leading to write failures.\\\\nBecause live message delivery had an active dependency on these writes completing successfully before pushing events to user screens, the failure caused incoming messages to experience synchronization delays. Separate push notification services and core message storage operated independently and were unaffected, which is why users still received alerts for new messages while active conversation streams experienced delays.\\\\n## Remediation and Prevention\\\\nThe issue was detected through automated backend alerting on dispatch error rates, alongside corresponding customer reports. Google Engineering initiated parallel workstreams to restore service.\\\\n* At 12:44 US/Pacific, Google Engineering directly mitigated the source of the issue by removing the dependency that live message delivery had on the failing database write. This allowed new incoming messages to bypass the failed write and immediately resume normal real-time delivery to user screens.\\\\n* At 13:00 US/Pacific, Engineering halted and reverted the database schema change to its previous verified stable version, restoring full database compatibility.\\\\n* By 13:34 US/Pacific, as queued messages were processed, real-time message synchronization delays returned to 0% across all customer domains. Engineering verified that full service health was restored with zero message loss.\\\\nTo prevent this from happening again, we are executing the following preventative actions:\\\\n* **Hardened internal detection:** Strengthen automated validation and verification to cover this specific failure scenario across pre-production and internal rollout stages. This aims to catch and mitigate schema incompatibilities and synchronization edge cases internally before any customer traffic can be impacted in the future.\\\\n* **Resilience Against Race Conditions:** Implement Architectural improvements to make the schema synchronization and message delivery architecture resilient to race conditions. This aims to ensure that background schema and config synchronization inconsistencies cannot disrupt real-time message delivery.\\\\n \\\\n## Detailed Description of Impact\\\\nThe incident impact window occurred between 10:50 US/Pacific and 13:34 US/Pacific on Friday, 28 August 2026\\\\\\\\. This issue applied globally to all Google Chat environments.\\\\n**Google Chat:**\\\\n* **Delayed message rendering:** New messages did not appear in active conversation streams automatically. Users were required to manually refresh their client or send a new message/react to force the stream to synchronize.\\\\n* **Stalled read states and inbox sorting:** Background actions, such as updating unread badges and moving active spaces to the top of the inbox, experienced heavy internal error rates and were significantly delayed for users.\\\\n* **Search indexing delays:** New messages sent during the outage window stalled in the indexing pipeline, experiencing delays in becoming searchable.\\\\n**Scope Exclusions:**\\\\n* **No Data Loss:** At no point was customer data lost or compromised. All messages transmitted during the incident window were securely captured in primary storage and successfully delivered to conversation streams once the backlog cleared.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_INFORMATION\\\",\\\"severity\\\":\\\"low\\\",\\\"service_key\\\":\\\"Ht9Vx5PFzDPHYvrnrvSk\\\",\\\"service_name\\\":\\\"Google Chat\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Google Chat\\\",\\\"id\\\":\\\"Ht9Vx5PFzDPHYvrnrvSk\\\"}],\\\"uri\\\":\\\"incidents/y2ufhCB886p7rveYNnko\\\"},{\\\"id\\\":\\\"sjDiph2NXGAfkYYLfK7T\\\",\\\"number\\\":\\\"9682992681334163277\\\",\\\"begin\\\":\\\"2026-08-18T23:58:00+00:00\\\",\\\"created\\\":\\\"2026-08-19T03:57:37+00:00\\\",\\\"end\\\":\\\"2026-08-19T12:40:00+00:00\\\",\\\"modified\\\":\\\"2026-08-24T16:31:00+00:00\\\",\\\"external_desc\\\":\\\"**Summary**\\\\nSome customers may experience slowness while using Google Drive.\\\\n**Description**\\\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\\\nOur engineering team is currently working on a potential fix while continuing to investigate the full root cause.\\\\nWe will provide an update by Tuesday, 2026-08-18 22:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nAffected customers may experience slowness while using Google Drive.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-08-24T16:31:00+00:00\\\",\\\"modified\\\":\\\"2026-08-24T16:31:00+00:00\\\",\\\"when\\\":\\\"2026-08-24T16:31:00+00:00\\\",\\\"text\\\":\\\"# Incident Report\\\\n## Summary\\\\nOn Tuesday, 18 August 2026 at 16:58 US/Pacific, an incident affecting Google Drive occurred for a duration of 12 hours and 42 minutes. Customers located in Australia, Philippines and New Zealand experienced severe latency, sporadic loading failures, and errors when accessing Google Drive. \\\\nTo our valued Google Workspace customers, we are deeply sorry for the service interruption you experienced. We are taking immediate and decisive action to address the issue and strengthen our systems against future incidents. Other Google Workspace services were not affected.\\\\n## Root Cause\\\\nGoogle utilizes a multi-tiered global load balancing and network routing architecture to serve traffic efficiently. To ensure fair use and prevent congestion, bandwidth allocation is managed by attributing traffic to specific downstream services.\\\\nA misconfiguration in the attribution of traffic caused the bandwidth usage to be incorrectly attributed to the frontend network load balancing tier itself, rather than to the Drive service.\\\\nThe sequence of events leading to customer impact was as follows:\\\\n* Traffic from the regional load balancing tier was incorrectly attributed to the load balancing infrastructure rather than the Drive service.\\\\n* Unrelated traffic caused network bandwidth usage attributed to the load balancing infrastructure in this region to exceed its quota.\\\\n* Network bandwidth management systems applied throttling to this traffic to enforce quotas, causing severe network congestion affecting Drive traffic in this region.\\\\n* This congestion led to increased latency, slowness, and error rates for users accessing Drive in the affected regions.\\\\n## Remediation and Prevention\\\\nGoogle engineers were alerted to the issue via automated monitoring and an influx of customer support cases, with initial incident response teams engaged by 17:21 US/Pacific on 18 August , 2026\\\\\\\\.\\\\nEngineers initially attempted to alleviate the congestion by draining regional frontend traffic to an alternate geographic location, but this proved ineffective and was reverted. The team then mitigated the issue by migrating regional traffic to an upgraded internal network routing architecture, which correctly set the traffic attribution for Drive.\\\\nNormal operations were fully restored at 05:40 once 100% of the regional traffic was migrated to the upgraded network architecture.\\\\nTo prevent a recurrence of this issue, our engineering teams are undertaking the following structural preventative actions:\\\\n* **Resilient Network Protocol Migration:** We are expediting the complete migration of all regional frontend traffic to the newer, resilient network routing protocol to ensure accurate bandwidth attribution under all congestion scenarios.\\\\n* **Delegation Root Cause Remediation:** We are addressing the underlying mechanism that caused the incorrect delegation and analyzing similar cases across the fleet to ensure it does not reoccur.\\\\n## Detailed Description of Impact\\\\nFrom 18 August 2026, 16:58 to 19 August 2026, 05:40 US/Pacific for a duration of 12 hours, 42 minutes, Google Drive users in Australia, Philippines and New Zealand experienced severe performance degradation, including high latency and errors accessing Drive or listing files via Editors. Service was fully restored as of 19 August, 05:40 US/Pacific.\\\\n**Google Drive:**\\\\n* Elevated Latency: Users experienced extreme slowness when loading the Drive interface or accessing files, with initial data retrieval times heavily delayed.\\\\n* Loading Failures and Errors: Many user requests timed out or returned HTTP 502 errors due to the aggressive network throttling dropping the connections.\\\\n**Scope Exclusion:**\\\\n* Customers accessing Google Drive from outside the Australia, Philippines and New Zealand regions were largely unaffected by this localized throttling event.\\\\n* Other Google Workspace applications not reliant on this specific routing path remained stable during the incident window.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-08-20T20:00:17+00:00\\\",\\\"modified\\\":\\\"2026-08-24T16:31:00+00:00\\\",\\\"when\\\":\\\"2026-08-20T20:00:17+00:00\\\",\\\"text\\\":\\\"# Preliminary Incident Report\\\\nWe sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Workspace, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future. \\\\nPlease note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. \\\\nIf you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article [**https://support.google.com/a/answer/1047213**](https://support.google.com/a/answer/1047213).\\\\n#[REDACTED] Date/Time of the Issue (All time US/Pacific)\\\\nIncident Start: 18 August 2026 16:58\\\\nIncident End: 19 August 2026 05:40\\\\nDuration: 12 hours, 42 minutes \\\\n## Summary\\\\nOn August 18, 2026 from 16:58 to August 19, 2026 05:40 US/Pacific, Drive customers in Australia, Philippines and New Zealand experienced degraded performance and slowness. During this period users also experienced 502 errors. Other Google Workspace services were not affected. \\\\n## Preliminary Root Cause\\\\nGoogle utilizes a multi-tiered global load balancing and network routing architecture to serve traffic efficiently. To ensure fair use and prevent congestion, bandwidth allocation is managed by attributing traffic to specific downstream services.\\\\nA misconfiguration in the attribution of traffic caused the bandwidth usage to be incorrectly attributed to the frontend network load balancing tier itself, rather than to the Drive service.\\\\nThe sequence of events leading to customer impact was as follows:\\\\n* Traffic from the regional load balancing tier was incorrectly attributed to the load balancing infrastructure rather than the Drive service.\\\\n* Unrelated traffic caused network bandwidth usage attributed to the load balancing infrastructure in this region to exceed its quota.\\\\n* Network bandwidth management systems applied throttling to this traffic to enforce quotas, causing severe network congestion affecting Drive traffic in this region.\\\\n* This congestion led to increased latency, slowness, and error rates for users accessing Drive in the affected regions.\\\\n## Remediation \\\\nGoogle engineers were alerted to the issue via automated monitoring and an influx of customer support cases, with initial incident response teams engaged by 17:21 US/Pacific on August 18, 2026\\\\\\\\.\\\\nEngineers initially attempted to alleviate the congestion by draining regional frontend traffic to an alternate geographic location, but this proved ineffective and was reverted. The team then mitigated the issue by migrating regional traffic to an upgraded internal network routing architecture, which correctly set the traffic attribution for Drive.\\\\nNormal operations were fully restored once 100% of the regional traffic was migrated to the upgraded network architecture.\\\\n## Description of Impact\\\\nBetween August 18, 2026, at 16:58 US/Pacific and August 19, 2026, at 05:40 US/Pacific, Google Drive users in Australia, Philippines and New Zealand experienced severe latency, degraded performance and 502 errors.\\\\nDuring this roughly 12-hour and 42-minute window, affected customers may have observed significant slowness or errors when attempting to access Google Drive or listing files through Editors (e.g. [docs.google.com](http://docs.google.com)), with initial load times significantly delayed. The issue was fully mitigated as of August 19, 2026, at 05:40 US/Pacific, and normal service has been restored.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-08-19T13:05:05+00:00\\\",\\\"modified\\\":\\\"2026-08-20T20:00:17+00:00\\\",\\\"when\\\":\\\"2026-08-19T13:05:05+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nMultiple customers in Australia and New Zealand may have experienced slowness or errors while accessing Google Drive.\\\\n**Description**\\\\nThe issue with Google Drive has been resolved for all affected customers as of Wednesday, 2026-08-19 05:40 PDT.\\\\nOur engineering teams determined the slowness was the result of a localized capacity and routing issue within our regional infrastructure.\\\\nTo restore service, we implemented a fix to alleviate the network congestion and accelerated a planned infrastructure upgrade in the affected region to ensure normal performance levels.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nAffected customers in Australia and New Zealand may have experienced slowness or 502 errors, with some users briefly unable to access Drive.\\\\n**Workaround**\\\\nThe issue is now mitigated\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-08-19T11:26:14+00:00\\\",\\\"modified\\\":\\\"2026-08-19T13:05:05+00:00\\\",\\\"when\\\":\\\"2026-08-19T11:26:14+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nMultiple customers in Australia and New Zealand may experience slowness or errors while accessing Google Drive.\\\\n**Description**\\\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\\\nOur engineering team is actively working on a mitigation strategy. There is no ETA for full resolution.\\\\nWe will provide an update by Wednesday, 2026-08-19 06:00 PDT with current details.\\\\n**Symptoms**\\\\nAffected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-08-19T09:54:28+00:00\\\",\\\"modified\\\":\\\"2026-08-19T11:26:14+00:00\\\",\\\"when\\\":\\\"2026-08-19T09:54:28+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nMultiple customers in Australia and New Zealand may experience slowness or errors while accessing Google Drive.\\\\n**Description**\\\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\\\nOur engineering team is actively working on a mitigation strategy. There is no ETA for full resolution.\\\\nWe will provide an update by Wednesday, 2026-08-19 04:30 PDT with current details.\\\\n**Symptoms**\\\\nAffected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-08-19T08:54:05+00:00\\\",\\\"modified\\\":\\\"2026-08-19T09:54:28+00:00\\\",\\\"when\\\":\\\"2026-08-19T08:54:05+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nMultiple customers in Australia and New Zealand may experience slowness or errors while accessing Google Drive.\\\\n**Description**\\\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\\\nOur engineering team is actively working on a mitigation strategy. There is no ETA for full resolution.\\\\nWe will provide an update by Wednesday, 2026-08-19 03:00 PDT with current details.\\\\n**Symptoms**\\\\nAffected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-08-19T07:59:51+00:00\\\",\\\"modified\\\":\\\"2026-08-19T08:54:05+00:00\\\",\\\"when\\\":\\\"2026-08-19T07:59:51+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nMultiple customers in Australia and New Zealand may experience slowness or errors while accessing Google Drive.\\\\n**Description**\\\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\\\nOur engineering team has identified the root cause of the issue and is actively working on a mitigation strategy. There is no ETA for full resolution.\\\\nWe will provide an update by Wednesday, 2026-08-19 02:00 PDT with current details.\\\\n**Symptoms**\\\\nAffected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-08-19T06:49:41+00:00\\\",\\\"modified\\\":\\\"2026-08-19T07:59:51+00:00\\\",\\\"when\\\":\\\"2026-08-19T06:49:41+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nMultiple customers in Australia and New Zealand may experience slowness or errors while accessing Google Drive.\\\\n**Description**\\\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\\\nOur engineering team has identified the root cause of the issue and is actively working on a mitigation strategy. There is no ETA for full resolution.\\\\nWe will provide an update by Wednesday, 2026-08-19 01:30 PDT with current details.\\\\n**Symptoms**\\\\nAffected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-08-19T04:59:07+00:00\\\",\\\"modified\\\":\\\"2026-08-19T06:49:41+00:00\\\",\\\"when\\\":\\\"2026-08-19T04:59:07+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nMultiple customers in Australia and New Zealand may experience slowness or errors while using Google Drive.\\\\n**Description**\\\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\\\nOur engineering team is currently working on a potential fix while continuing to investigate the full root cause.\\\\nWe will provide an update by Wednesday, 2026-08-19 00:30 PDT with current details.\\\\n**Symptoms**\\\\nAffected customers in Australia and New Zealand may experience slowness or errors while using Google Drive. Some customers also may observe 502 errors with Drive inaccessible for a brief period.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-08-19T03:57:37+00:00\\\",\\\"modified\\\":\\\"2026-08-19T05:21:15+00:00\\\",\\\"when\\\":\\\"2026-08-19T03:57:37+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nSome customers may experience slowness while using Google Drive.\\\\n**Description**\\\\nWe are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT.\\\\nOur engineering team is currently working on a potential fix while continuing to investigate the full root cause.\\\\nWe will provide an update by Tuesday, 2026-08-18 22:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nAffected customers may experience slowness while using Google Drive.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-08-24T16:31:00+00:00\\\",\\\"modified\\\":\\\"2026-08-24T16:31:00+00:00\\\",\\\"when\\\":\\\"2026-08-24T16:31:00+00:00\\\",\\\"text\\\":\\\"# Incident Report\\\\n## Summary\\\\nOn Tuesday, 18 August 2026 at 16:58 US/Pacific, an incident affecting Google Drive occurred for a duration of 12 hours and 42 minutes. Customers located in Australia, Philippines and New Zealand experienced severe latency, sporadic loading failures, and errors when accessing Google Drive. \\\\nTo our valued Google Workspace customers, we are deeply sorry for the service interruption you experienced. We are taking immediate and decisive action to address the issue and strengthen our systems against future incidents. Other Google Workspace services were not affected.\\\\n## Root Cause\\\\nGoogle utilizes a multi-tiered global load balancing and network routing architecture to serve traffic efficiently. To ensure fair use and prevent congestion, bandwidth allocation is managed by attributing traffic to specific downstream services.\\\\nA misconfiguration in the attribution of traffic caused the bandwidth usage to be incorrectly attributed to the frontend network load balancing tier itself, rather than to the Drive service.\\\\nThe sequence of events leading to customer impact was as follows:\\\\n* Traffic from the regional load balancing tier was incorrectly attributed to the load balancing infrastructure rather than the Drive service.\\\\n* Unrelated traffic caused network bandwidth usage attributed to the load balancing infrastructure in this region to exceed its quota.\\\\n* Network bandwidth management systems applied throttling to this traffic to enforce quotas, causing severe network congestion affecting Drive traffic in this region.\\\\n* This congestion led to increased latency, slowness, and error rates for users accessing Drive in the affected regions.\\\\n## Remediation and Prevention\\\\nGoogle engineers were alerted to the issue via automated monitoring and an influx of customer support cases, with initial incident response teams engaged by 17:21 US/Pacific on 18 August , 2026\\\\\\\\.\\\\nEngineers initially attempted to alleviate the congestion by draining regional frontend traffic to an alternate geographic location, but this proved ineffective and was reverted. The team then mitigated the issue by migrating regional traffic to an upgraded internal network routing architecture, which correctly set the traffic attribution for Drive.\\\\nNormal operations were fully restored at 05:40 once 100% of the regional traffic was migrated to the upgraded network architecture.\\\\nTo prevent a recurrence of this issue, our engineering teams are undertaking the following structural preventative actions:\\\\n* **Resilient Network Protocol Migration:** We are expediting the complete migration of all regional frontend traffic to the newer, resilient network routing protocol to ensure accurate bandwidth attribution under all congestion scenarios.\\\\n* **Delegation Root Cause Remediation:** We are addressing the underlying mechanism that caused the incorrect delegation and analyzing similar cases across the fleet to ensure it does not reoccur.\\\\n## Detailed Description of Impact\\\\nFrom 18 August 2026, 16:58 to 19 August 2026, 05:40 US/Pacific for a duration of 12 hours, 42 minutes, Google Drive users in Australia, Philippines and New Zealand experienced severe performance degradation, including high latency and errors accessing Drive or listing files via Editors. Service was fully restored as of 19 August, 05:40 US/Pacific.\\\\n**Google Drive:**\\\\n* Elevated Latency: Users experienced extreme slowness when loading the Drive interface or accessing files, with initial data retrieval times heavily delayed.\\\\n* Loading Failures and Errors: Many user requests timed out or returned HTTP 502 errors due to the aggressive network throttling dropping the connections.\\\\n**Scope Exclusion:**\\\\n* Customers accessing Google Drive from outside the Australia, Philippines and New Zealand regions were largely unaffected by this localized throttling event.\\\\n* Other Google Workspace applications not reliant on this specific routing path remained stable during the incident window.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_DISRUPTION\\\",\\\"severity\\\":\\\"medium\\\",\\\"service_key\\\":\\\"VHNA7p3Z5p3iakj5sA8V\\\",\\\"service_name\\\":\\\"Google Drive\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Google Drive\\\",\\\"id\\\":\\\"VHNA7p3Z5p3iakj5sA8V\\\"}],\\\"uri\\\":\\\"incidents/sjDiph2NXGAfkYYLfK7T\\\"},{\\\"id\\\":\\\"MpR24eKjibRq1YEEaTfp\\\",\\\"number\\\":\\\"17752500813029482969\\\",\\\"begin\\\":\\\"2026-08-18T19:03:00+00:00\\\",\\\"created\\\":\\\"2026-08-18T19:35:27+00:00\\\",\\\"end\\\":\\\"2026-08-18T22:30:00+00:00\\\",\\\"modified\\\":\\\"2026-08-18T23:02:54+00:00\\\",\\\"external_desc\\\":\\\"**Summary:**\\\\nInvestigating access issues with Classroom’s new web homepage.\\\\n**Description:**\\\\nWe are investigating reports of access issues with Classroom’s new web homepage beginning on Thursday, 2026-08-13.\\\\nOur engineering team is currently working on a potential fix while continuing to investigate the full root cause.\\\\nWe will provide an update by Tuesday, 2026-08-18 13:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms:**\\\\nAdmins and some Teachers may experience permission errors when accessing Classroom’s new web homepage.\\\\n**Workaround:**\\\\nUsers should still be able to navigate to other views from the left hand pane of the interface.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-08-18T23:02:54+00:00\\\",\\\"modified\\\":\\\"2026-08-18T23:02:54+00:00\\\",\\\"when\\\":\\\"2026-08-18T23:02:54+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\n[Mitigated] Access issues for Workspace Admins with Classroom’s web homepage.\\\\n**Description**\\\\nThe issue with Classroom has been resolved for all affected users as of Tuesday, 2026-08-18 15:30 PDT.\\\\nFrom preliminary analysis, the issue was triggered due to a recent change which caused the permission mismatch, our Engineering team has rolled out the fix to mitigate the impact.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nWorkspace Admins may have experienced permission errors when accessing Classroom’s web homepage.\\\\nRemaining Classroom functionality remained unaffected.\\\\nThe Classroom Mobile App and the Classroom API were unaffected by the issue.\\\\n**Workaround**\\\\nNone as the issue is resolved\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-08-18T20:27:49+00:00\\\",\\\"modified\\\":\\\"2026-08-18T23:02:54+00:00\\\",\\\"when\\\":\\\"2026-08-18T20:27:49+00:00\\\",\\\"text\\\":\\\"**Summary:**\\\\nMitigating access issues for Workspace Admins with Classroom’s web homepage.\\\\n**Description:**\\\\nOur engineering has identified the cause and mitigation work is currently underway.\\\\nThe mitigation rollout is expected to take several hours.\\\\nWe will provide more information by Wednesday, 2026-08-19 8:00 PDT.\\\\n**Symptoms:**\\\\nWorkspace Admins may experience permission errors when accessing Classroom’s web homepage.\\\\nRemaining Classroom functionality remains unaffected.\\\\nThe Classroom Mobile App and the Classroom API are unaffected by the issue.\\\\n**Workaround:**\\\\nAll other pages are unaffected and admins can navigate to other pages using the left hand pane on the interface.\\\\nWorkspace Admins can still create classes without any issue.\\\\nWorkspace Admins can also view the courses that they are in by going to https://classroom.google.com/h/tv\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-08-18T20:00:36+00:00\\\",\\\"modified\\\":\\\"2026-08-18T20:27:49+00:00\\\",\\\"when\\\":\\\"2026-08-18T20:00:36+00:00\\\",\\\"text\\\":\\\"**Summary:**\\\\nMitigating access issues with Classroom’s new web homepage.\\\\n**Description:**\\\\nMitigation work is currently underway by our engineering team.\\\\nThe mitigation is expected to complete by Tuesday, 2026-08-18 14:00 PDT.\\\\nWe will provide more information by Tuesday, 2026-08-18 14:15 PDT.\\\\n**Symptoms:**\\\\nAdmins and some Teachers may experience permission errors when accessing Classroom’s new web homepage.\\\\n**Workaround:**\\\\nUsers should still be able to navigate to other views from the left hand pane of the interface.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-08-18T19:35:27+00:00\\\",\\\"modified\\\":\\\"2026-08-18T20:11:32+00:00\\\",\\\"when\\\":\\\"2026-08-18T19:35:27+00:00\\\",\\\"text\\\":\\\"**Summary:**\\\\nInvestigating access issues with Classroom’s new web homepage.\\\\n**Description:**\\\\nWe are investigating reports of access issues with Classroom’s new web homepage beginning on Thursday, 2026-08-13.\\\\nOur engineering team is currently working on a potential fix while continuing to investigate the full root cause.\\\\nWe will provide an update by Tuesday, 2026-08-18 13:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms:**\\\\nAdmins and some Teachers may experience permission errors when accessing Classroom’s new web homepage.\\\\n**Workaround:**\\\\nUsers should still be able to navigate to other views from the left hand pane of the interface.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-08-18T23:02:54+00:00\\\",\\\"modified\\\":\\\"2026-08-18T23:02:54+00:00\\\",\\\"when\\\":\\\"2026-08-18T23:02:54+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\n[Mitigated] Access issues for Workspace Admins with Classroom’s web homepage.\\\\n**Description**\\\\nThe issue with Classroom has been resolved for all affected users as of Tuesday, 2026-08-18 15:30 PDT.\\\\nFrom preliminary analysis, the issue was triggered due to a recent change which caused the permission mismatch, our Engineering team has rolled out the fix to mitigate the impact.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nWorkspace Admins may have experienced permission errors when accessing Classroom’s web homepage.\\\\nRemaining Classroom functionality remained unaffected.\\\\nThe Classroom Mobile App and the Classroom API were unaffected by the issue.\\\\n**Workaround**\\\\nNone as the issue is resolved\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_INFORMATION\\\",\\\"severity\\\":\\\"low\\\",\\\"service_key\\\":\\\"3GoNVrVz48jeNZSyLJMV\\\",\\\"service_name\\\":\\\"Classroom\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Classroom\\\",\\\"id\\\":\\\"3GoNVrVz48jeNZSyLJMV\\\"}],\\\"uri\\\":\\\"incidents/MpR24eKjibRq1YEEaTfp\\\"},{\\\"id\\\":\\\"AWyE1SfEzpe8Qed49Rog\\\",\\\"number\\\":\\\"12918412236754833550\\\",\\\"begin\\\":\\\"2026-06-30T09:03:00+00:00\\\",\\\"created\\\":\\\"2026-06-30T11:56:57+00:00\\\",\\\"end\\\":\\\"2026-07-01T03:45:00+00:00\\\",\\\"modified\\\":\\\"2026-07-01T04:58:32+00:00\\\",\\\"external_desc\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nWe are experiencing an issue with Google Cloud Support beginning on Tuesday, 2026-06-30 02:03 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Tuesday, 2026-06-30 05:30 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation continues to work.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-07-01T04:58:32+00:00\\\",\\\"modified\\\":\\\"2026-07-01T04:58:32+00:00\\\",\\\"when\\\":\\\"2026-07-01T04:58:32+00:00\\\",\\\"text\\\":\\\"**Title**\\\\n[Resolved] Inbound emails from customers to Google Support were experiencing issues.\\\\n**Description**\\\\nThe issue with inbound emails to Google Cloud Support has been resolved for all affected customers as of Tuesday, 2026-06-30 20:45 PDT.\\\\nFrom preliminary analysis, the engineering team identified a recent change which was preventing incoming emails from being processed. We have successfully rolled back the change to mitigate the impact.\\\\nBacklog processing of all the affected emails is now complete.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support were experiencing issues.\\\\nCase creation and updates continued to work without any issues.\\\\n**Workaround**\\\\nNone as the issue is resolved.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-07-01T00:44:02+00:00\\\",\\\"modified\\\":\\\"2026-07-01T04:58:32+00:00\\\",\\\"when\\\":\\\"2026-07-01T00:44:02+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nThe mitigation rollout was completed as of 09:00 AM PDT on 30 June 2026, and all incoming emails are currently being processed without further disruption.\\\\nOur engineers are continuing to work replaying the inbound messages that were impacted between 25 June 2026 and 09:00 AM PDT on 30 June 2026. We expect the email replays to take several hours, please be advised that this backlog processing may cause case statuses to update dynamically or cause certain closed cases to temporarily reopen.\\\\nWe will provide more information by Tuesday, 2026-06-30 21:30 PDT.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support after 09:00 AM PDT on 30 June 2026 are working as expected. The affected emails from the time of the issue are being replayed.\\\\nCase creation and updates continues to work without any issues.\\\\n**Workaround**\\\\nCustomers can continue to use the portal to add comments / notes.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T19:03:19+00:00\\\",\\\"modified\\\":\\\"2026-07-01T00:44:02+00:00\\\",\\\"when\\\":\\\"2026-06-30T19:03:19+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nThe mitigation rollout was completed as of 09:00 AM PDT on 30 June 2026, and all incoming emails are currently being processed without further disruption.\\\\nOur engineers are continuing to work replaying the inbound messages that were impacted between 25 June 2026 and 09:00 AM PDT on 30 June 2026. We expect the email replays to take several hours, please be advised that this backlog processing may cause case statuses to update dynamically or cause certain closed cases to temporarily reopen.\\\\nWe will provide more information by Tuesday, 2026-06-30 17:00 PDT.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support after 09:00 AM PDT on 30 June 2026 are working as expected. The affected emails from the time of the issue are being replayed.\\\\nCase creation and updates continues to work without any issues.\\\\n**Workaround**\\\\nCustomers can continue to use the portal to add comments / notes.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T17:53:29+00:00\\\",\\\"modified\\\":\\\"2026-06-30T19:03:19+00:00\\\",\\\"when\\\":\\\"2026-06-30T17:53:29+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nThe mitigation rollout was completed as of 09:00 AM PDT on 30 June 2026, and all incoming emails are currently being processed without further disruption.\\\\nOur engineers are continuing to work replaying the inbound messages that were impacted between 25 June 2026 and 09:00 AM PDT on 30 June 2026. We expect the email replays to take several hours, please be advised that this backlog processing may cause case statuses to update dynamically or cause certain closed cases to temporarily reopen.\\\\nWe will provide more information by Tuesday, 2026-06-30 12:15 PDT.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation and updates continues to work without any issues.\\\\n**Workaround**\\\\nCustomers can continue to use the portal to add comments / notes.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T16:36:46+00:00\\\",\\\"modified\\\":\\\"2026-06-30T17:53:29+00:00\\\",\\\"when\\\":\\\"2026-06-30T16:36:46+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nThe mitigation rollout was completed as of 09:00 AM PDT on 30 June 2026, and all incoming emails are currently being processed without further disruption.\\\\nOur engineering teams have shifted focus to replaying the inbound messages that were impacted between 25 June 2026 and 09:00 AM PDT on 30 June 2026.\\\\nWhile there is currently no specific ETA for the email replay to finish, please be advised that this backlog processing may cause case statuses to update dynamically or cause certain closed cases to temporarily reopen.\\\\nWe will provide more information by Tuesday, 2026-06-30 11:00 PDT.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation and updates continues to work without any issues.\\\\n**Workaround**\\\\nCustomers can continue to use the portal to add comments / notes.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T15:48:35+00:00\\\",\\\"modified\\\":\\\"2026-06-30T16:36:46+00:00\\\",\\\"when\\\":\\\"2026-06-30T15:48:35+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nMitigation work is currently underway by our engineering team.\\\\nThe mitigation is expected to complete by Tuesday, 2026-06-30 09:45 PDT.\\\\nWe will provide more information by Tuesday, 2026-06-30 10:00 PDT.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation and updates continues to work without any issues.\\\\n**Workaround**\\\\nCustomers can continue to use the portal to add comments / notes.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T14:51:10+00:00\\\",\\\"modified\\\":\\\"2026-06-30T15:48:35+00:00\\\",\\\"when\\\":\\\"2026-06-30T14:51:10+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nMitigation work is currently underway by our engineering team.\\\\nWe do not have an ETA for mitigation at this point.\\\\nWe will provide more information by Tuesday, 2026-06-30 08:45 PDT.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation and updates continues to work without any issues.\\\\n**Workaround**\\\\nCustomers can continue to use the portal to add comments / notes.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T14:06:26+00:00\\\",\\\"modified\\\":\\\"2026-06-30T14:51:10+00:00\\\",\\\"when\\\":\\\"2026-06-30T14:06:26+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nWe are experiencing an issue with Google Cloud Support beginning on Tuesday, 2026-06-30 02:03 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Tuesday, 2026-06-30 08:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation continues to work.\\\\n**Workaround**\\\\nCustomers can continue to use the portal to add comments / notes.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T13:28:46+00:00\\\",\\\"modified\\\":\\\"2026-06-30T14:06:26+00:00\\\",\\\"when\\\":\\\"2026-06-30T13:28:46+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nWe believe the issue with Google Support is mitigated.\\\\nOur engineering team continues to investigate the root cause.\\\\nCustomers are being advised if they have an urgent issue to resend previous communication to Google Support.\\\\nWe will provide an update by Tuesday, 2026-06-30 08:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation continues to work.\\\\n**Workaround**\\\\nCustomers can continue to use the portal to add comments / notes.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T12:28:59+00:00\\\",\\\"modified\\\":\\\"2026-06-30T13:28:46+00:00\\\",\\\"when\\\":\\\"2026-06-30T12:28:59+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nWe believe the issue with Google Support is mitigated.\\\\nOur engineering team continues to investigate the root cause.\\\\nCustomers are being advised if they have an urgent issue to resend previous communication to Google Support.\\\\nWe will provide an update by Tuesday, 2026-06-30 06:30 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation continues to work.\\\\n**Workaround**\\\\nCustomers can continue to use the portal to add comments / notes.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T12:17:19+00:00\\\",\\\"modified\\\":\\\"2026-06-30T12:28:59+00:00\\\",\\\"when\\\":\\\"2026-06-30T12:17:19+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nWe believe the issue with Google Support is mitigated.\\\\nOur engineering team continues to investigate the root cause.\\\\nCustomers are being advised if they have an urgent issue to resend previous communication to Google Support.\\\\nWe will provide an update by Tuesday, 2026-06-30 06:30 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation continues to work.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-30T11:56:57+00:00\\\",\\\"modified\\\":\\\"2026-06-30T12:17:19+00:00\\\",\\\"when\\\":\\\"2026-06-30T11:56:57+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\n**Description**\\\\nWe are experiencing an issue with Google Cloud Support beginning on Tuesday, 2026-06-30 02:03 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Tuesday, 2026-06-30 05:30 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support are experiencing issues.\\\\nCase creation continues to work.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-07-01T04:58:32+00:00\\\",\\\"modified\\\":\\\"2026-07-01T04:58:32+00:00\\\",\\\"when\\\":\\\"2026-07-01T04:58:32+00:00\\\",\\\"text\\\":\\\"**Title**\\\\n[Resolved] Inbound emails from customers to Google Support were experiencing issues.\\\\n**Description**\\\\nThe issue with inbound emails to Google Cloud Support has been resolved for all affected customers as of Tuesday, 2026-06-30 20:45 PDT.\\\\nFrom preliminary analysis, the engineering team identified a recent change which was preventing incoming emails from being processed. We have successfully rolled back the change to mitigate the impact.\\\\nBacklog processing of all the affected emails is now complete.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nInbound emails from customers to Google Support were experiencing issues.\\\\nCase creation and updates continued to work without any issues.\\\\n**Workaround**\\\\nNone as the issue is resolved.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_DISRUPTION\\\",\\\"severity\\\":\\\"medium\\\",\\\"service_key\\\":\\\"dnz8HVUHoMMcW9wV91LY\\\",\\\"service_name\\\":\\\"Google Workspace Support\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Google Workspace Support\\\",\\\"id\\\":\\\"dnz8HVUHoMMcW9wV91LY\\\"}],\\\"uri\\\":\\\"incidents/AWyE1SfEzpe8Qed49Rog\\\"},{\\\"id\\\":\\\"qqWhZSpoVLJWur1GfHfu\\\",\\\"number\\\":\\\"6760817193806533896\\\",\\\"begin\\\":\\\"2026-06-22T19:30:00+00:00\\\",\\\"created\\\":\\\"2026-06-22T22:28:55+00:00\\\",\\\"end\\\":\\\"2026-06-23T00:46:00+00:00\\\",\\\"modified\\\":\\\"2026-06-23T01:08:59+00:00\\\",\\\"external_desc\\\":\\\"**Title**\\\\nSome consumer Google Voice users are experiencing account suspensions.\\\\n**Description**\\\\nWe are experiencing an issue with Google Voice beginning at Monday, 2026-06-22 12:30 PDT.\\\\nOur engineering team has identified the root cause and is actively working to mitigate the issue.\\\\nWe will provide an update by Monday, 2026-06-22 17:30 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Customer Symptoms**\\\\nSome consumer Google Voice users are experiencing account suspensions\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-06-23T01:08:59+00:00\\\",\\\"modified\\\":\\\"2026-06-23T01:08:59+00:00\\\",\\\"when\\\":\\\"2026-06-23T01:08:59+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nSome consumer Google Voice users experienced account suspensions.\\\\n**Description**\\\\nThe issue with Google Voice has been resolved for all affected users as of Monday, 2026-06-22 17:46 PDT\\\\nOur engineers have mitigated the issue by rolling back to a previous version and reinstating suspended accounts.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nSome consumer Google Voice users experienced account suspensions\\\\n**Workaround**\\\\nThe issue is now mitigated.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-06-23T00:29:23+00:00\\\",\\\"modified\\\":\\\"2026-06-23T01:08:59+00:00\\\",\\\"when\\\":\\\"2026-06-23T00:29:23+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nSome consumer Google Voice users are experiencing account suspensions.\\\\n**Description**\\\\nWe are experiencing an issue with Google Voice beginning at Monday, 2026-06-22 12:30 PDT.\\\\nOur engineering team has identified a mitigation strategy and is actively working on mitigation.\\\\nWe do not have an ETA for mitigation at this time.\\\\nWe will provide an update by Monday, 2026-06-22 22:30 PDT with current details.\\\\n**Symptoms**\\\\nSome consumer Google Voice users are experiencing account suspensions\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-06-22T22:28:55+00:00\\\",\\\"modified\\\":\\\"2026-06-23T00:29:23+00:00\\\",\\\"when\\\":\\\"2026-06-22T22:28:55+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nSome consumer Google Voice users are experiencing account suspensions.\\\\n**Description**\\\\nWe are experiencing an issue with Google Voice beginning at Monday, 2026-06-22 12:30 PDT.\\\\nOur engineering team has identified the root cause and is actively working to mitigate the issue.\\\\nWe will provide an update by Monday, 2026-06-22 17:30 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Customer Symptoms**\\\\nSome consumer Google Voice users are experiencing account suspensions\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-06-23T01:08:59+00:00\\\",\\\"modified\\\":\\\"2026-06-23T01:08:59+00:00\\\",\\\"when\\\":\\\"2026-06-23T01:08:59+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nSome consumer Google Voice users experienced account suspensions.\\\\n**Description**\\\\nThe issue with Google Voice has been resolved for all affected users as of Monday, 2026-06-22 17:46 PDT\\\\nOur engineers have mitigated the issue by rolling back to a previous version and reinstating suspended accounts.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nSome consumer Google Voice users experienced account suspensions\\\\n**Workaround**\\\\nThe issue is now mitigated.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_INFORMATION\\\",\\\"severity\\\":\\\"low\\\",\\\"service_key\\\":\\\"gh79SrQk6wqonWoxPdCm\\\",\\\"service_name\\\":\\\"Google Voice\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Google Voice\\\",\\\"id\\\":\\\"gh79SrQk6wqonWoxPdCm\\\"}],\\\"uri\\\":\\\"incidents/qqWhZSpoVLJWur1GfHfu\\\"},{\\\"id\\\":\\\"CzZUn98mhTcEiCJo27Kv\\\",\\\"number\\\":\\\"11672950073358311792\\\",\\\"begin\\\":\\\"2026-06-10T10:20:00+00:00\\\",\\\"created\\\":\\\"2026-06-10T12:40:45+00:00\\\",\\\"end\\\":\\\"2026-06-10T17:30:00+00:00\\\",\\\"modified\\\":\\\"2026-06-16T17:41:16+00:00\\\",\\\"external_desc\\\":\\\"**Title**\\\\nGemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.\\\\n**Description**\\\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Wednesday, 2026-06-10 06:30 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nGemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-06-16T17:41:16+00:00\\\",\\\"modified\\\":\\\"2026-06-16T17:41:17+00:00\\\",\\\"when\\\":\\\"2026-06-16T17:41:16+00:00\\\",\\\"text\\\":\\\"# Incident Report\\\\n## Summary\\\\nOn Wednesday, June 10, 2026, at 03:30 US/Pacific, an incident occurred affecting the availability of Gemini services across all surfaces, including web, mobile (iOS and Android), and integrations within Google Chrome. The incident resulted in an elevated error rate for approximately 6 hours and 55 minutes, with full service restoration confirmed after a total duration of 14 hours and 49 minutes.\\\\nWe sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Cloud, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future.\\\\n## Root Cause\\\\nThe disruption was caused by extreme read contention within the foundational database service that manages tool deployment metadata.\\\\nThe primary cause was an index design issue within the database where a column used for tracking deployment expirations contained a high volume of rows with similar values. This resulted in \\\\\\\"hotspotting,\\\\\\\" where traffic was concentrated on a very small number of database shards, overwhelming them. Additionally, the service utilized an in-memory cache with a very short Time To Live (TTL) of only one minute, requiring frequent database refreshes.\\\\nThe failure sequence was as follows:\\\\n1. **Technical Trigger:** The trigger was an upward spike in the Queries Per Second (QPS) from the frontend. The underlying system was already running close to maximum usage and the incoming QPS crossed a tipping threshold beyond which the underlying spanner started getting overloaded.\\\\n2. **Systemic Effect:** This short window, combined with an indexing anomaly where specific metadata fields containing empty values were clustered on a single database partition, led to a \\\\\\\"hot shard\\\\\\\" condition.\\\\n3. **Contention and Latency:** As request volume increased, the internal tool management service experienced a more than 10x surge in database calls because the cache hit rate dropped significantly.\\\\n4. **Failure State:** The underlying storage layer reached its capacity for handling incoming queries, and database failure rates skyrocketed to 60%. This led to cache hit rate dropping down to 50%.\\\\nAutomated validation failed to prevent this because the indexing strategy did not sufficiently account for the concentration of data rows when specific identifiers were absent.\\\\n## Remediation and Prevention\\\\nThe issue was detected by internal real-time monitoring alerts indicating high processing error rates. Engineering teams performed several targeted actions to restore the service:\\\\n* **Throttling:** Applied Rate Access Control Lists (RateACLs) to the most affected database shards to allow internal caches to recover.\\\\n* **Index Redistribution:** Gradually redistributed common values in the database index to random values in a specific target range, effectively spreading the load across multiple shards.\\\\n* **Cache Policy Update:** Deployed a configuration change to increase the in-memory cache Time To Live (TTL) from 1 minute to 20 minutes, significantly reducing database load.\\\\n* **Deadline Adjustment:** Shortened the Remote Procedure Call (RPC) deadlines for database queries to prevent request pile-ups during periods of high latency.\\\\n## Prevention\\\\nTo prevent a recurrence, Google engineering will be implementing the following actions:\\\\n* **Index Design Standards:** Restructure of database index to prevent \\\\\\\"hotspotting\\\\\\\".\\\\n* **Adaptive Cache Tuning:** Implement more robust cache policies that prevent load amplification during periods of backend stress.\\\\n* **Enhanced Hotspot Monitoring:** Deploy improved monitoring and alerting specifically designed to detect uneven database load distribution before it impacts user traffic.\\\\n* **Service Resiliency:** Implement advanced load protection mechanisms, such as smart task resizing and query coalescing for preventing spanner overloads.\\\\n## Detailed Description of Impact\\\\nThe impact window occurred between 03:30 US/Pacific and 10:25 US/Pacific on June 10, 2026\\\\\\\\.\\\\n**Gemini App (Web and Mobile):**\\\\n* High Failure Rates: Users experienced a 50% error rate when attempting to send prompts, receiving \\\\\\\"Something went wrong\\\\\\\" messages .\\\\n* Service Unavailability: The issue affected both consumer accounts and enterprise Google Workspace customers .\\\\n* Enterprise Google Workspace Users encountered disruptions when attempting to attach assets from Google Drive within the web application (gemini.google.com). Affected users observed the \\\\\\\"Add from Drive\\\\\\\" functionality to be visually disabled and unresponsive following interaction with the \\\\\\\"+\\\\\\\" prompt.\\\\n**Integrations:**\\\\n* **Google Chrome Integration:** The Gemini Side Panel and integrated conversational features encountered persistent timeouts.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-06-12T11:14:34+00:00\\\",\\\"modified\\\":\\\"2026-06-16T17:41:16+00:00\\\",\\\"when\\\":\\\"2026-06-12T11:14:34+00:00\\\",\\\"text\\\":\\\"# Preliminary Incident Report\\\\nWe apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article [https://support.google.com/a/answer/1047213](https://support.google.com/a/answer/1047213).\\\\n#[REDACTED] Date/Time of the Issue (All time US/Pacific)\\\\nIncident Start: 10 June 2026 03:20\\\\nIncident End: 10 June 2026 10:25\\\\nDuration: 7 hours, 5 minutes\\\\n## Summary\\\\nOn Wednesday, 10 June 2026 03:20 US/Pacific, an incident occurred affecting the availability of Gemini services across all surfaces, including web, mobile (iOS and Android), and integrations within Google Chrome. The incident resulted in an elevated error rate for 7 hours and 5 minutes.\\\\n## Preliminary Root Cause\\\\nThe disruption was caused by extreme read contention within the foundational database service that manages tool deployment metadata.\\\\nThe primary cause was back-pressure on specific database replicas that experienced high loads. The issue was made worse because of an index design issue within the database where a column used for tracking deployment expirations contained a high volume of rows with a similar fixed value. This resulted in a condition where traffic was concentrated on a very small number of database instances, overwhelming them. Additionally, the service utilized an in-memory cache with a short Time To Live (TTL) of only one minute, requiring frequent database refreshes.\\\\nThe failure sequence was as follows:\\\\n1. **Technical Trigger:** The database service became overloaded, causing back-pressure on specific replicas.\\\\n2. **Systemic Effect:** An application level caching layer configuration is set to a short 1-minute expiration window for deployed tools. This short window, combined with an indexing anomaly where specific metadata fields containing empty values were clustered on a single database partition. The failed DB queries resulted in failure to populate application level cache.\\\\n3. **Contention and Latency:** As request volume increased, the internal tool management service experienced \\\\u003e10x surge in database calls because the cache hit rate dropped significantly.\\\\n4. **Failure State:** The underlying storage layer reached its transaction limits, with database response success rates dropping as low as 10% on the affected partitions.\\\\n## Remediation\\\\nThe issue was detected by internal real-time monitoring alerts indicating high processing error rates. Engineering teams performed several targeted actions to restore the service:\\\\n* **Throttling:** Applied rate limiting to the most affected database instances to allow internal caches to recover.\\\\n* **Index Redistribution:** Gradually redistributed fixed index values in the database to random values in a specific range, effectively spreading the load across multiple instances.\\\\n* **Cache Policy Update:** Deployed a configuration change to increase the in-memory cache Time To Live (TTL) from 1 minute to 20 minutes, significantly reducing database load.\\\\n* **Deadline Adjustment:** Shortened the Remote Procedure Call (RPC) deadlines for database queries to prevent request pile-ups during periods of high latency.\\\\n## Description of Impact\\\\nThe impact window occurred between 03:20 US/Pacific and 10:25 US/Pacific on 10 June 2026.\\\\n**Gemini App (Web and Mobile):**\\\\n* Users experienced a 50% error rate when attempting to send prompts, receiving \\\\\\\"Something went wrong\\\\\\\" messages. The issue affected both consumer accounts and enterprise Google Workspace customers .\\\\n**Integrations:**\\\\n* **Google Chrome Integration:** The Gemini Side Panel and integrated conversational features encountered persistent timeouts.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-06-11T01:07:25+00:00\\\",\\\"modified\\\":\\\"2026-06-12T11:14:34+00:00\\\",\\\"when\\\":\\\"2026-06-11T01:07:25+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nGemini App in Workspace users were experiencing “Something Went Wrong” errors.\\\\n**Description**\\\\nThe issue with Gemini has been resolved for all the affected users as of Wednesday, 2026-06-10 10:30 US/Pacific.\\\\nFrom preliminary analysis, the issue was triggered by a performance issue in our backend database which impacted the retrieval of Gemini App tools catalog.\\\\nOur engineers mitigated the issue by optimizing the load distribution across our backend database fleet.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nAffected users may have seen \\\\\\\"Something went wrong\\\\\\\" errors (codes 1099, 1076) when prompting Gemini while using the following services.\\\\nGemini App in Workspace includes the following platforms: - Web, MacOS, iOS and Android - Gemini in Chrome\\\\n**Workaround**\\\\nNone as the issue is now resolved.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-06-10T20:19:31+00:00\\\",\\\"modified\\\":\\\"2026-06-11T01:07:25+00:00\\\",\\\"when\\\":\\\"2026-06-10T20:19:31+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nGemini App in Workspace customers are experiencing “Something Went Wrong” errors.\\\\n**Description**\\\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\\\nOur engineering team has applied mitigations to reduce impact and confirmed the majority of users are no longer observing impact as of Wednesday, 2026-06-10 10:30 PDT. We will continue to monitor the service for stability.\\\\nWe will provide an update by Wednesday, 2026-06-10 20:30 PDT with current details.\\\\n**Symptoms**\\\\nThe following services are experiencing “Something Went Wrong 1099 or 1076” errors:\\\\nGemini app in Workspace includes the following platforms:\\\\n* Web, MacOS, iOS and Android\\\\n* Gemini in Chrome\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-10T18:37:53+00:00\\\",\\\"modified\\\":\\\"2026-06-10T20:19:31+00:00\\\",\\\"when\\\":\\\"2026-06-10T18:37:53+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nGemini App in Workspace customers are experiencing “Something Went Wrong” errors.\\\\n**Description**\\\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\\\nOur engineering team has applied mitigations to reduce impact and continues to investigate the root cause. We are seeing signs of recovery and will continue to monitor progress.\\\\nWe will provide an update by Wednesday, 2026-06-10 14:30 PDT with current details.\\\\n**Symptoms**\\\\nThe following services are experiencing “Something Went Wrong 1099 or 1076” errors:\\\\nGemini app in Workspace includes the following platforms:\\\\n* Web, MacOS, iOS and Android\\\\n* Gemini in Chrome\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-10T16:24:54+00:00\\\",\\\"modified\\\":\\\"2026-06-10T18:37:53+00:00\\\",\\\"when\\\":\\\"2026-06-10T16:24:54+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nGemini App in Workspace customers are experiencing “Something Went Wrong” errors.\\\\n**Description**\\\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\\\nOur engineering team has identified a mitigation which is currently in progress. We do not have an ETA for mitigation at this time.\\\\nWe will provide an update by Wednesday, 2026-06-10 12:30 PDT with current details.\\\\n**Symptoms**\\\\nThe following services are experiencing “Something Went Wrong 1099 or 1076” errors:\\\\nGemini app in Workspace includes the following platforms:\\\\n* Web, MacOS, iOS and Android\\\\n* Gemini in Chrome\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-10T15:10:52+00:00\\\",\\\"modified\\\":\\\"2026-06-10T16:24:54+00:00\\\",\\\"when\\\":\\\"2026-06-10T15:10:52+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nGemini App in Workspace customers are experiencing “Something Went Wrong” errors.\\\\n**Description**\\\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\\\nOur engineering team continues to investigate the issue and identify a mitigation strategy.\\\\nWe will provide an update by Wednesday, 2026-06-10 09:30 PDT with current details.\\\\n**Symptoms**\\\\nThe following services are experiencing “Something Went Wrong” errors.:\\\\n* Gemini app in Workspace includes the following platforms;\\\\n* Web, MacOS, iOS and Android\\\\n* Gemini in Chrome\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-10T13:21:18+00:00\\\",\\\"modified\\\":\\\"2026-06-10T15:10:52+00:00\\\",\\\"when\\\":\\\"2026-06-10T13:21:18+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nGemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.\\\\n**Description**\\\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Wednesday, 2026-06-10 08:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nGemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-06-10T12:40:45+00:00\\\",\\\"modified\\\":\\\"2026-06-10T13:21:18+00:00\\\",\\\"when\\\":\\\"2026-06-10T12:40:45+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nGemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.\\\\n**Description**\\\\nWe are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Wednesday, 2026-06-10 06:30 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nGemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-06-16T17:41:16+00:00\\\",\\\"modified\\\":\\\"2026-06-16T17:41:17+00:00\\\",\\\"when\\\":\\\"2026-06-16T17:41:16+00:00\\\",\\\"text\\\":\\\"# Incident Report\\\\n## Summary\\\\nOn Wednesday, June 10, 2026, at 03:30 US/Pacific, an incident occurred affecting the availability of Gemini services across all surfaces, including web, mobile (iOS and Android), and integrations within Google Chrome. The incident resulted in an elevated error rate for approximately 6 hours and 55 minutes, with full service restoration confirmed after a total duration of 14 hours and 49 minutes.\\\\nWe sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Cloud, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future.\\\\n## Root Cause\\\\nThe disruption was caused by extreme read contention within the foundational database service that manages tool deployment metadata.\\\\nThe primary cause was an index design issue within the database where a column used for tracking deployment expirations contained a high volume of rows with similar values. This resulted in \\\\\\\"hotspotting,\\\\\\\" where traffic was concentrated on a very small number of database shards, overwhelming them. Additionally, the service utilized an in-memory cache with a very short Time To Live (TTL) of only one minute, requiring frequent database refreshes.\\\\nThe failure sequence was as follows:\\\\n1. **Technical Trigger:** The trigger was an upward spike in the Queries Per Second (QPS) from the frontend. The underlying system was already running close to maximum usage and the incoming QPS crossed a tipping threshold beyond which the underlying spanner started getting overloaded.\\\\n2. **Systemic Effect:** This short window, combined with an indexing anomaly where specific metadata fields containing empty values were clustered on a single database partition, led to a \\\\\\\"hot shard\\\\\\\" condition.\\\\n3. **Contention and Latency:** As request volume increased, the internal tool management service experienced a more than 10x surge in database calls because the cache hit rate dropped significantly.\\\\n4. **Failure State:** The underlying storage layer reached its capacity for handling incoming queries, and database failure rates skyrocketed to 60%. This led to cache hit rate dropping down to 50%.\\\\nAutomated validation failed to prevent this because the indexing strategy did not sufficiently account for the concentration of data rows when specific identifiers were absent.\\\\n## Remediation and Prevention\\\\nThe issue was detected by internal real-time monitoring alerts indicating high processing error rates. Engineering teams performed several targeted actions to restore the service:\\\\n* **Throttling:** Applied Rate Access Control Lists (RateACLs) to the most affected database shards to allow internal caches to recover.\\\\n* **Index Redistribution:** Gradually redistributed common values in the database index to random values in a specific target range, effectively spreading the load across multiple shards.\\\\n* **Cache Policy Update:** Deployed a configuration change to increase the in-memory cache Time To Live (TTL) from 1 minute to 20 minutes, significantly reducing database load.\\\\n* **Deadline Adjustment:** Shortened the Remote Procedure Call (RPC) deadlines for database queries to prevent request pile-ups during periods of high latency.\\\\n## Prevention\\\\nTo prevent a recurrence, Google engineering will be implementing the following actions:\\\\n* **Index Design Standards:** Restructure of database index to prevent \\\\\\\"hotspotting\\\\\\\".\\\\n* **Adaptive Cache Tuning:** Implement more robust cache policies that prevent load amplification during periods of backend stress.\\\\n* **Enhanced Hotspot Monitoring:** Deploy improved monitoring and alerting specifically designed to detect uneven database load distribution before it impacts user traffic.\\\\n* **Service Resiliency:** Implement advanced load protection mechanisms, such as smart task resizing and query coalescing for preventing spanner overloads.\\\\n## Detailed Description of Impact\\\\nThe impact window occurred between 03:30 US/Pacific and 10:25 US/Pacific on June 10, 2026\\\\\\\\.\\\\n**Gemini App (Web and Mobile):**\\\\n* High Failure Rates: Users experienced a 50% error rate when attempting to send prompts, receiving \\\\\\\"Something went wrong\\\\\\\" messages .\\\\n* Service Unavailability: The issue affected both consumer accounts and enterprise Google Workspace customers .\\\\n* Enterprise Google Workspace Users encountered disruptions when attempting to attach assets from Google Drive within the web application (gemini.google.com). Affected users observed the \\\\\\\"Add from Drive\\\\\\\" functionality to be visually disabled and unresponsive following interaction with the \\\\\\\"+\\\\\\\" prompt.\\\\n**Integrations:**\\\\n* **Google Chrome Integration:** The Gemini Side Panel and integrated conversational features encountered persistent timeouts.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_DISRUPTION\\\",\\\"severity\\\":\\\"medium\\\",\\\"service_key\\\":\\\"npdyhgECDJ6tB66MxXyo\\\",\\\"service_name\\\":\\\"Gemini\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Gemini\\\",\\\"id\\\":\\\"npdyhgECDJ6tB66MxXyo\\\"}],\\\"uri\\\":\\\"incidents/CzZUn98mhTcEiCJo27Kv\\\"},{\\\"id\\\":\\\"vagmms1y6yRxwg6RMQZp\\\",\\\"number\\\":\\\"6811500350919047262\\\",\\\"begin\\\":\\\"2026-05-31T03:28:00+00:00\\\",\\\"created\\\":\\\"2026-05-31T04:59:04+00:00\\\",\\\"end\\\":\\\"2026-05-31T04:24:00+00:00\\\",\\\"modified\\\":\\\"2026-06-04T20:11:26+00:00\\\",\\\"external_desc\\\":\\\"**Title**\\\\nMultiple Workspace Products experiencing Issues.\\\\n**Description**\\\\nWe are experiencing an issue with Google Voice, Google Meet, Google Drive, Google Chat, Google Calendar, Gmail, Google Docs beginning on Saturday, 2026-05-30 20:28 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Saturday, 2026-05-30 22:10 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nCustomers may experience issues loading the application/ intermittent high latency or errors while using the application.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-06-04T20:11:26+00:00\\\",\\\"modified\\\":\\\"2026-06-04T20:11:27+00:00\\\",\\\"when\\\":\\\"2026-06-04T20:11:26+00:00\\\",\\\"text\\\":\\\"# Incident Report\\\\n## Summary\\\\nStarting at 20:15 US/Pacific on Saturday, 30 May 2026, various Google Workspace services encountered service-level disruptions lasting approximately 75 minutes. The scope of impact spanned several products, specifically affecting Gmail, Google Docs, Drive, Calendar, Chat, Meet, and Google Voice.\\\\nThroughout the duration of this event, users faced obstacles while attempting to initialize applications, in addition to experiencing sporadic instances of elevated latency and request failures.\\\\nGoogle offers a sincere apology for the operational interference this issue introduced to your organization. Recognizing the critical nature of Google Cloud to your daily operations, we deeply regret any resulting loss in productivity. Engineering efforts are currently focused on remediating the underlying cause to ensure future stability.\\\\n## Root Cause\\\\nThe disruption originated from failures in a critical cluster infrastructure component responsible for Google's service discovery and configuration management systems. This system facilitates resource synchronization, manages shared data access, and coordinates background tasks for numerous Google services.\\\\nA high volume of rapid configuration changes triggered the regional failure in Workspace Services, preventing Workspace servers from reliably processing traffic. To mitigate the impact, Google Workspace traffic was redirected to alternative regions.\\\\n## Remediation and Prevention\\\\nGoogle engineers were alerted to the issue via automated monitoring on Saturday, 30 May 2026 at 20:28 US/Pacific and immediately began an investigation. The investigation identified elevated error rates and high latency affecting multiple Google Workspace backend services. Upon investigation our engineering team confirmed errors were primarily originating from some regional servers due to coordination service failures in the region.\\\\nTo mitigate the impact, Google engineers redirected traffic away from the affected servers.\\\\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\\\\n- Prevent recurrence of this incident by rolling out the fix for the latent encryption library issue, which improves the handling of internal log compaction compactions during active file writes, to prevent recurrence of the cluster instability.\\\\n- Investigating additional resilience in our database layer.\\\\n## Detailed Description of Impact\\\\nOn 30 May 2026 from 20:15 to 21:30 US/Pacific, affected customers encountered difficulties while loading applications or experienced failures when performing tasks within the following products:\\\\n- Gmail\\\\n- Google Calendar\\\\n- Google Chat\\\\n- Google Docs\\\\n- Google Drive\\\\n- Google Meet\\\\n- Google Voice\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-06-02T03:50:13+00:00\\\",\\\"modified\\\":\\\"2026-06-04T20:11:26+00:00\\\",\\\"when\\\":\\\"2026-06-02T03:50:13+00:00\\\",\\\"text\\\":\\\"# Preliminary Incident Report\\\\nWe sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Workspace, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future.\\\\nPlease note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete.\\\\nIf you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using help article https://support.google.com/a/answer/1047213\\\\n#[REDACTED] Date/Time of the Issue (All time US/Pacific)\\\\nIncident Start: 30 May 2026 20:15\\\\nIncident End: 30 May 2026 21:30\\\\nDuration: 1 hour, 15 minutes\\\\n## Summary\\\\nStarting on Saturday, 30 May 2026 at 20:15 US/Pacific, multiple Google Workspace products experienced service disruptions for a duration of 1 hour and 15 minutes. The affected services included Google Voice, Google Meet, Google Drive, Google Chat, Google Calendar, Gmail, and Google Docs. During this incident, customers encountered difficulties loading applications, as well as intermittent high latency or errors.\\\\n## Preliminary Root Cause\\\\nThe disruption originated from failures in a regional infrastructure component responsible for Google's distributed lock service. This critical system facilitates resource synchronization, manages shared data access, and coordinates background tasks for numerous Google services.\\\\nA high volume of rapid configuration changes triggered the regional failure, preventing Workspace servers from reliably processing traffic. To mitigate the impact, Google Workspace traffic was redirected to alternative regions.\\\\nGoogle engineers have begun a full root cause analysis and will provide additional information once it is available.\\\\n## Remediation\\\\nGoogle engineers were alerted to the issue via automated monitoring at 20:28 US/Pacific on Saturday, 30 May 2026, and immediately began an investigation. The investigation identified elevated error rates and high latency affecting multiple Google Workspace backend services. Upon investigation our engineering team confirmed errors were primarily originating from some regional servers due to coordination service failures in the region.\\\\nTo mitigate the impact, Google engineers redirected traffic away from the affected servers.\\\\n## Description of Impact\\\\nAffected customers encountered difficulties loading applications or experienced failures when performing tasks within the following products:\\\\n* Gmail\\\\n* Google Calendar\\\\n* Google Chat\\\\n* Google Docs\\\\n* Google Drive\\\\n* Google Meet\\\\n* Google Voice\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-05-31T05:11:14+00:00\\\",\\\"modified\\\":\\\"2026-06-02T03:50:13+00:00\\\",\\\"when\\\":\\\"2026-05-31T05:11:14+00:00\\\",\\\"text\\\":\\\"**Title**\\\\n[Mitigated] Multiple Workspace Products experienced Issues.\\\\n**Description**\\\\nThe issue with Google Voice, Google Meet, Google Drive, Google Chat, Google Calendar, Gmail, Google Docs has been resolved for all affected users as of Saturday, 2026-05-30 21:30 PDT.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nCustomers may have experienced issues loading the application/ intermittent high latency or errors while using the application.\\\\n**Workaround**\\\\nNone at this time as the issue stands mitigated\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-05-31T04:59:04+00:00\\\",\\\"modified\\\":\\\"2026-05-31T05:11:14+00:00\\\",\\\"when\\\":\\\"2026-05-31T04:59:04+00:00\\\",\\\"text\\\":\\\"**Title**\\\\nMultiple Workspace Products experiencing Issues.\\\\n**Description**\\\\nWe are experiencing an issue with Google Voice, Google Meet, Google Drive, Google Chat, Google Calendar, Gmail, Google Docs beginning on Saturday, 2026-05-30 20:28 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Saturday, 2026-05-30 22:10 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms**\\\\nCustomers may experience issues loading the application/ intermittent high latency or errors while using the application.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-06-04T20:11:26+00:00\\\",\\\"modified\\\":\\\"2026-06-04T20:11:27+00:00\\\",\\\"when\\\":\\\"2026-06-04T20:11:26+00:00\\\",\\\"text\\\":\\\"# Incident Report\\\\n## Summary\\\\nStarting at 20:15 US/Pacific on Saturday, 30 May 2026, various Google Workspace services encountered service-level disruptions lasting approximately 75 minutes. The scope of impact spanned several products, specifically affecting Gmail, Google Docs, Drive, Calendar, Chat, Meet, and Google Voice.\\\\nThroughout the duration of this event, users faced obstacles while attempting to initialize applications, in addition to experiencing sporadic instances of elevated latency and request failures.\\\\nGoogle offers a sincere apology for the operational interference this issue introduced to your organization. Recognizing the critical nature of Google Cloud to your daily operations, we deeply regret any resulting loss in productivity. Engineering efforts are currently focused on remediating the underlying cause to ensure future stability.\\\\n## Root Cause\\\\nThe disruption originated from failures in a critical cluster infrastructure component responsible for Google's service discovery and configuration management systems. This system facilitates resource synchronization, manages shared data access, and coordinates background tasks for numerous Google services.\\\\nA high volume of rapid configuration changes triggered the regional failure in Workspace Services, preventing Workspace servers from reliably processing traffic. To mitigate the impact, Google Workspace traffic was redirected to alternative regions.\\\\n## Remediation and Prevention\\\\nGoogle engineers were alerted to the issue via automated monitoring on Saturday, 30 May 2026 at 20:28 US/Pacific and immediately began an investigation. The investigation identified elevated error rates and high latency affecting multiple Google Workspace backend services. Upon investigation our engineering team confirmed errors were primarily originating from some regional servers due to coordination service failures in the region.\\\\nTo mitigate the impact, Google engineers redirected traffic away from the affected servers.\\\\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\\\\n- Prevent recurrence of this incident by rolling out the fix for the latent encryption library issue, which improves the handling of internal log compaction compactions during active file writes, to prevent recurrence of the cluster instability.\\\\n- Investigating additional resilience in our database layer.\\\\n## Detailed Description of Impact\\\\nOn 30 May 2026 from 20:15 to 21:30 US/Pacific, affected customers encountered difficulties while loading applications or experienced failures when performing tasks within the following products:\\\\n- Gmail\\\\n- Google Calendar\\\\n- Google Chat\\\\n- Google Docs\\\\n- Google Drive\\\\n- Google Meet\\\\n- Google Voice\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_DISRUPTION\\\",\\\"severity\\\":\\\"medium\\\",\\\"service_key\\\":\\\"zall\\\",\\\"service_name\\\":\\\"Multiple Products\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Gmail\\\",\\\"id\\\":\\\"prPt2Yra2CbGsbEm9cpC\\\"},{\\\"title\\\":\\\"Google Calendar\\\",\\\"id\\\":\\\"n5hVexuq1PM9onY9qrmo\\\"},{\\\"title\\\":\\\"Google Chat\\\",\\\"id\\\":\\\"Ht9Vx5PFzDPHYvrnrvSk\\\"},{\\\"title\\\":\\\"Google Docs\\\",\\\"id\\\":\\\"wNHuVFtZEWU5Mmj2eRCK\\\"},{\\\"title\\\":\\\"Google Drive\\\",\\\"id\\\":\\\"VHNA7p3Z5p3iakj5sA8V\\\"},{\\\"title\\\":\\\"Google Meet\\\",\\\"id\\\":\\\"sUH4BQXzYXma7NiS94Hi\\\"},{\\\"title\\\":\\\"Google Voice\\\",\\\"id\\\":\\\"gh79SrQk6wqonWoxPdCm\\\"}],\\\"uri\\\":\\\"incidents/vagmms1y6yRxwg6RMQZp\\\"},{\\\"id\\\":\\\"CzVZ5zk2Gf2ixPSt6RT3\\\",\\\"number\\\":\\\"17128406613861610004\\\",\\\"begin\\\":\\\"2026-05-21T07:00:00+00:00\\\",\\\"created\\\":\\\"2026-05-21T15:37:18+00:00\\\",\\\"end\\\":\\\"2026-05-21T15:47:00+00:00\\\",\\\"modified\\\":\\\"2026-05-21T16:10:17+00:00\\\",\\\"external_desc\\\":\\\"**Summary**\\\\nGmail users are experiencing delayed delivery of messages sent from the bank.in domains.\\\\n**Description**\\\\nWe are experiencing an issue with Gmail beginning on Thursday, 2026-05-21 at 00:00 PDT.\\\\nMitigation efforts are currently underway by our engineering team.\\\\nWe will provide the next update by Thursday, 2026-05-21 at 09:30 PDT.\\\\n**Diagnosis / Customer Symptoms**\\\\nGmail users are experiencing delayed delivery of messages sent from the bank.in domains.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-05-21T16:09:55+00:00\\\",\\\"modified\\\":\\\"2026-05-21T16:10:17+00:00\\\",\\\"when\\\":\\\"2026-05-21T16:09:55+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGmail users experienced delayed delivery of messages sent from the bank.in domains.\\\\n**Description**\\\\nOur internal telemetry shows that the issue with Gmail has been resolved for all affected users as of Thursday, 2026-05-21 at 08:47 US/Pacific.\\\\nOur engineering team is actively monitoring the system to ensure continued stability.\\\\n**Diagnosis / Customer Symptoms**\\\\nCustomers should expect new messages to be delivered normally, older emails could take some time as the sender\n\n[...truncated 13350 characters...]\n\nou for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nImpacted users might have experienced authentication and sync failures while using Microsoft Exchange Online accounts through Gmail App on Android devices.\\\\n**Workaround**\\\\nUpdate to the latest version of Gmail Android (2026.05.11) in Google Play Store.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-05-22T00:04:40+00:00\\\",\\\"modified\\\":\\\"2026-05-22T18:57:24+00:00\\\",\\\"when\\\":\\\"2026-05-22T00:04:40+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGmail Android users using Microsoft Exchange Online accounts may fail to Log In and Sync due to authentication issues.\\\\n**Description**\\\\nOur engineering team identified the issue and released a new version of the Gmail Android application (version 2026.05.11) to the Google Play Store, which contains a fix to unblock affected users.\\\\nIf you are currently encountering this issue, please upgrade to the latest version of the Gmail Android app. Users operating on the new app version should be able to successfully log in and sync their accounts.\\\\nWe will provide more information by Friday, 2026-05-22 17:00 US/Pacific with current details.\\\\nWe thank you for your patience while we are working on resolving the issue.\\\\n**Symptoms**\\\\nImpacted users may experience authentication and sync failures while using Microsoft Exchange Online accounts through Gmail App on Android devices.\\\\n**Workaround**\\\\nAffected users could update to the latest version of Gmail Android (version 2026.05.11) in Google Play Store.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-19T21:55:32+00:00\\\",\\\"modified\\\":\\\"2026-05-22T00:04:40+00:00\\\",\\\"when\\\":\\\"2026-05-19T21:55:32+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGmail Android users using Microsoft Exchange Online accounts may fail to Log In and Sync due to authentication issues.\\\\n**Description**\\\\nWe are experiencing an issue with Gmail Android client connecting with Microsoft Exchange Online accounts beginning on Saturday 2026-05-06.\\\\nOur engineering team has identified the issue and started rolling out a new version of Gmail Android release that contains the fix to unblock users.\\\\nUsers already on the new app version will be able to Log In and Sync their accounts successfully.\\\\nWe will provide more information by Thursday, 2026-05-21 17:00 US/Pacific with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Customer Symptoms**\\\\nImpacted users may experience authentication and sync failures while using Microsoft Exchange Online accounts through Gmail App on Android devices.\\\\n**Workaround**\\\\nAffected users can use Microsoft Web App to access the Inbox.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-18T21:45:35+00:00\\\",\\\"modified\\\":\\\"2026-05-19T21:55:32+00:00\\\",\\\"when\\\":\\\"2026-05-18T21:45:35+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGmail Android users using Microsoft Exchange Online accounts may fail to Log In and Sync due to authentication issues.\\\\n**Description**\\\\nWe are experiencing an issue with Gmail Android client connecting with Microsoft Exchange Online accounts beginning on Saturday 2026-05-06.\\\\nOur engineering team has identified the issue and is actively working on rolling out a fix to unblock all the affected customers.\\\\nWe will provide more information by Tuesday, 2026-05-19 15:00 US/Pacific with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Customer Symptoms**\\\\nImpacted users may experience authentication and sync failures while using Microsoft Exchange Online accounts through Gmail App on Android devices.\\\\n**Workaround**\\\\nAffected users can use Microsoft Web App to access the Inbox.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-16T02:05:03+00:00\\\",\\\"modified\\\":\\\"2026-05-18T21:45:35+00:00\\\",\\\"when\\\":\\\"2026-05-16T02:05:03+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGmail Android users using Microsoft Exchange Online may fail to log in due to authentication issues.\\\\n**Description**\\\\nWe are experiencing an issue with Gmail beginning on Saturday 2026-05-06.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide more information by Monday, 2026-05-18 15:00 US/Pacific with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Customer Symptoms**\\\\nImpacted users may experience authentication failures while logging in to Microsoft Exchange Online through\\\\nGmail App on Android devices.\\\\n**Workaround**\\\\nAffected users can use Microsoft web app to access the Inbox.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-05-22T18:57:24+00:00\\\",\\\"modified\\\":\\\"2026-05-22T18:57:41+00:00\\\",\\\"when\\\":\\\"2026-05-22T18:57:24+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\n[Resolved] Gmail Android users utilizing Microsoft Exchange Online accounts experienced authentication and synchronization failures.\\\\n**Description**\\\\nThe issue with Gmail has been resolved in the new Gmail android release (version 2026.05.11) as of Thursday, 2026-05-21 15:00 PDT.\\\\nOur engineering team identified the issue and released a new version of the Gmail Android application to the Google Play Store, which contains a fix to unblock affected users.\\\\nIf you are currently encountering this issue, please upgrade to the latest version of the Gmail Android app. Users operating on the new app version will be able to successfully log in and sync their accounts.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nImpacted users might have experienced authentication and sync failures while using Microsoft Exchange Online accounts through Gmail App on Android devices.\\\\n**Workaround**\\\\nUpdate to the latest version of Gmail Android (2026.05.11) in Google Play Store.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_INFORMATION\\\",\\\"severity\\\":\\\"low\\\",\\\"service_key\\\":\\\"prPt2Yra2CbGsbEm9cpC\\\",\\\"service_name\\\":\\\"Gmail\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Gmail\\\",\\\"id\\\":\\\"prPt2Yra2CbGsbEm9cpC\\\"}],\\\"uri\\\":\\\"incidents/kWSuRqLdsXGzT7Qxjdmz\\\"},{\\\"id\\\":\\\"1hr418kjGyBZ9ugXy3W3\\\",\\\"number\\\":\\\"5513210214958819667\\\",\\\"begin\\\":\\\"2026-05-04T16:40:00+00:00\\\",\\\"created\\\":\\\"2026-05-05T23:35:49+00:00\\\",\\\"end\\\":\\\"2026-05-08T18:53:00+00:00\\\",\\\"modified\\\":\\\"2026-05-08T18:58:23+00:00\\\",\\\"external_desc\\\":\\\"**Summary**\\\\nInvestigating elevated error rates with Gemini conversations.\\\\n**Description**\\\\nMitigation work is currently underway by our engineering team.\\\\nWe do not have an ETA for mitigation at this point.\\\\nWe will provide more information by Tuesday, 2026-05-05 20:30 PDT.\\\\n**Symptoms**\\\\nAffected customers are experiencing \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-05-08T18:58:09+00:00\\\",\\\"modified\\\":\\\"2026-05-08T18:58:23+00:00\\\",\\\"when\\\":\\\"2026-05-08T18:58:09+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGemini conversations may have experienced elevated error rates.\\\\n**Description**\\\\nThe issue with Gemini has been resolved for all affected users as of Friday, 2026-05-08 11:53 PDT.\\\\nThe issue was triggered due to a combination of insufficient database resources, and application bugs. The database capacity issues were resolved, and further code changes are being made to mitigate the application issue.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nAffected customers may have experienced \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-05-08T15:53:13+00:00\\\",\\\"modified\\\":\\\"2026-05-08T18:58:09+00:00\\\",\\\"when\\\":\\\"2026-05-08T15:53:13+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGemini conversations may experience elevated error rates.\\\\n**Description**\\\\nOur engineering team has isolated the underlying issue and is in the process of reverting the specific code deployments that triggered the impact.\\\\nWe are actively monitoring the progress of the rollback, alongside system health indicators, to confirm that the restoration is proceeding effectively.\\\\nWe will provide more information by Friday, 2026-05-08 12:00 PDT.\\\\n**Symptoms**\\\\nAffected customers may encounter \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-08T10:23:42+00:00\\\",\\\"modified\\\":\\\"2026-05-08T15:53:13+00:00\\\",\\\"when\\\":\\\"2026-05-08T10:23:42+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGemini conversations may experience elevated error rates.\\\\n**Description**\\\\nOur engineering team has isolated the underlying issue and is in the process of reverting the specific code deployments that triggered the impact.\\\\nWe are actively monitoring the progress of the rollback, alongside system health indicators, to confirm that the restoration is proceeding effectively.\\\\nWe will provide more information by Friday, 2026-05-08 09:00 PDT.\\\\n**Symptoms**\\\\nAffected customers may encounter \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-08T07:29:25+00:00\\\",\\\"modified\\\":\\\"2026-05-08T10:23:42+00:00\\\",\\\"when\\\":\\\"2026-05-08T07:29:25+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGemini conversations may experience elevated error rates.\\\\n**Description**\\\\nOur engineering team has isolated the underlying issue and is in the process of reverting the specific code deployments that triggered the impact.\\\\nWe are actively monitoring the progress of the rollback, alongside system health indicators, to confirm that the restoration is proceeding effectively.\\\\nWe will provide more information by Friday, 2026-05-08 04:00 PDT.\\\\n**Symptoms**\\\\nAffected customers may encounter \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-08T05:16:23+00:00\\\",\\\"modified\\\":\\\"2026-05-08T07:29:25+00:00\\\",\\\"when\\\":\\\"2026-05-08T05:16:23+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGemini conversations are experiencing elevated error rates.\\\\n**Description**\\\\nOur engineering team has determined that further investigation is required to mitigate the issue.\\\\nWe do not have an ETA for mitigation at this point.\\\\nWe will provide more information by Friday, 2026-05-08 01:00 PDT.\\\\n**Symptoms**\\\\nAffected customers may encounter \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-08T01:00:01+00:00\\\",\\\"modified\\\":\\\"2026-05-08T05:16:23+00:00\\\",\\\"when\\\":\\\"2026-05-08T01:00:01+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGemini conversations may have experienced elevated error rates.\\\\n**Description**\\\\nThe issue with Gemini has been resolved for all affected users as of Thursday, 2026-05-07 17:00 PDT.\\\\nThe issue was triggered due to a combination of insufficient database resources, and application issues. The database capacity issues were resolved, and further code changes are being made to mitigate the application issues.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nAffected customers may have experienced the \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-07T23:52:49+00:00\\\",\\\"modified\\\":\\\"2026-05-08T01:00:01+00:00\\\",\\\"when\\\":\\\"2026-05-07T23:52:49+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGemini conversations are experiencing elevated error rates.\\\\n**Description**\\\\nOur engineering team has determined that further investigation is required to mitigate the issue.\\\\nWe do not have an ETA for mitigation at this point.\\\\nWe will provide more information by Thursday, 2026-05-07 20:30 PDT.\\\\n**Symptoms**\\\\nAffected customers are experiencing \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-06T19:11:02+00:00\\\",\\\"modified\\\":\\\"2026-05-07T23:52:49+00:00\\\",\\\"when\\\":\\\"2026-05-06T19:11:02+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGemini conversations are experiencing elevated error rates.\\\\n**Description**\\\\nOur engineering team has determined that further investigation is required to mitigate the issue.\\\\nWe do not have an ETA for mitigation at this point.\\\\nWe will provide more information by Thursday, 2026-05-07 17:00 PDT.\\\\n**Symptoms**\\\\nAffected customers are experiencing \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-06T03:22:40+00:00\\\",\\\"modified\\\":\\\"2026-05-06T19:11:02+00:00\\\",\\\"when\\\":\\\"2026-05-06T03:22:40+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nInvestigating elevated error rates with Gemini conversations.\\\\n**Description**\\\\nMitigation work is currently underway by our engineering team.\\\\nWe do not have an ETA for mitigation at this point.\\\\nWe will provide more information by Wednesday, 2026-05-06 09:00 PDT.\\\\n**Symptoms**\\\\nAffected customers are experiencing \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-05-05T23:35:49+00:00\\\",\\\"modified\\\":\\\"2026-05-06T03:22:40+00:00\\\",\\\"when\\\":\\\"2026-05-05T23:35:49+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nInvestigating elevated error rates with Gemini conversations.\\\\n**Description**\\\\nMitigation work is currently underway by our engineering team.\\\\nWe do not have an ETA for mitigation at this point.\\\\nWe will provide more information by Tuesday, 2026-05-05 20:30 PDT.\\\\n**Symptoms**\\\\nAffected customers are experiencing \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-05-08T18:58:09+00:00\\\",\\\"modified\\\":\\\"2026-05-08T18:58:23+00:00\\\",\\\"when\\\":\\\"2026-05-08T18:58:09+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGemini conversations may have experienced elevated error rates.\\\\n**Description**\\\\nThe issue with Gemini has been resolved for all affected users as of Friday, 2026-05-08 11:53 PDT.\\\\nThe issue was triggered due to a combination of insufficient database resources, and application bugs. The database capacity issues were resolved, and further code changes are being made to mitigate the application issue.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms**\\\\nAffected customers may have experienced \\\\\\\"Something went wrong(13)\\\\\\\" error message in their Gemini conversations.\\\\n**Workaround**\\\\nNone at this time.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_INFORMATION\\\",\\\"severity\\\":\\\"low\\\",\\\"service_key\\\":\\\"npdyhgECDJ6tB66MxXyo\\\",\\\"service_name\\\":\\\"Gemini\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Gemini\\\",\\\"id\\\":\\\"npdyhgECDJ6tB66MxXyo\\\"}],\\\"uri\\\":\\\"incidents/1hr418kjGyBZ9ugXy3W3\\\"},{\\\"id\\\":\\\"GDxbSoojBb9yyEhU2Cua\\\",\\\"number\\\":\\\"5147219210877965776\\\",\\\"begin\\\":\\\"2026-04-29T18:30:00+00:00\\\",\\\"created\\\":\\\"2026-04-29T19:17:28+00:00\\\",\\\"end\\\":\\\"2026-04-29T20:45:00+00:00\\\",\\\"modified\\\":\\\"2026-04-29T21:07:19+00:00\\\",\\\"external_desc\\\":\\\"**Title:**\\\\nInvestigating issues with Classroom.\\\\n**Description:**\\\\nWe are experiencing an issue with Classroom beginning at Wednesday, 2026-04-29 11:30 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide more information by Wednesday, 2026-04-29 13:00 PDT.\\\\n**Symptoms:**\\\\nAffected users may experience issues accessing classroom.\\\\n**Workaround:**\\\\nNo workarounds available at this time.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-04-29T21:07:19+00:00\\\",\\\"modified\\\":\\\"2026-04-29T21:07:20+00:00\\\",\\\"when\\\":\\\"2026-04-29T21:07:19+00:00\\\",\\\"text\\\":\\\"**Title:**\\\\nAn issue with Classroom is resolved.\\\\n**Description:**\\\\nA fix has been deployed, and services have stabilized for all users.\\\\nThe issue with Google Classroom has been resolved for all affected users as of Wednesday, 2026-04-29 13:45 PDT.\\\\nThe issue was due to an internal software migration. Our engineers have mitigated the issue by rolling back the migration for impacted users.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms:**\\\\nAffected users may have experienced issues accessing the classroom.\\\\n**Workaround:**\\\\nThe issue is resolved.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-04-29T20:13:38+00:00\\\",\\\"modified\\\":\\\"2026-04-29T21:07:19+00:00\\\",\\\"when\\\":\\\"2026-04-29T20:13:38+00:00\\\",\\\"text\\\":\\\"**Title:**\\\\nInvestigating issues with Classroom.\\\\n**Description:**\\\\nA fix has been deployed, and services have stabilized for the majority of users.\\\\nWe are aware that a small subset of users may still encounter errors. Our engineering team is actively working on a secondary resolution to address these remaining users.\\\\nFull resolution for remaining users is expected to complete by Wednesday, 2026-04-29 14:00 PDT.\\\\nWe will provide an update by Wednesday, 2026-04-29 14:15 PDT with current details.\\\\n**Symptoms:**\\\\nAffected users may experience issues accessing classroom.\\\\n**Workaround:**\\\\nNo workarounds available at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-04-29T19:53:46+00:00\\\",\\\"modified\\\":\\\"2026-04-29T20:13:38+00:00\\\",\\\"when\\\":\\\"2026-04-29T19:53:46+00:00\\\",\\\"text\\\":\\\"**Title:**\\\\nInvestigating issues with Classroom.\\\\n**Description:**\\\\nOur engineering team has identified the cause and working on mitigation of the issue .\\\\nThe mitigation is expected to complete by Wednesday, 2026-04-29 15:00 PDT.\\\\nWe will provide more information by Wednesday, 2026-04-29 15:15 PDT.\\\\n**Symptoms:**\\\\nAffected users may experience issues accessing classroom.\\\\n**Workaround:**\\\\nNo workarounds available at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-04-29T19:17:28+00:00\\\",\\\"modified\\\":\\\"2026-04-29T19:53:46+00:00\\\",\\\"when\\\":\\\"2026-04-29T19:17:28+00:00\\\",\\\"text\\\":\\\"**Title:**\\\\nInvestigating issues with Classroom.\\\\n**Description:**\\\\nWe are experiencing an issue with Classroom beginning at Wednesday, 2026-04-29 11:30 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide more information by Wednesday, 2026-04-29 13:00 PDT.\\\\n**Symptoms:**\\\\nAffected users may experience issues accessing classroom.\\\\n**Workaround:**\\\\nNo workarounds available at this time.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-04-29T21:07:19+00:00\\\",\\\"modified\\\":\\\"2026-04-29T21:07:20+00:00\\\",\\\"when\\\":\\\"2026-04-29T21:07:19+00:00\\\",\\\"text\\\":\\\"**Title:**\\\\nAn issue with Classroom is resolved.\\\\n**Description:**\\\\nA fix has been deployed, and services have stabilized for all users.\\\\nThe issue with Google Classroom has been resolved for all affected users as of Wednesday, 2026-04-29 13:45 PDT.\\\\nThe issue was due to an internal software migration. Our engineers have mitigated the issue by rolling back the migration for impacted users.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms:**\\\\nAffected users may have experienced issues accessing the classroom.\\\\n**Workaround:**\\\\nThe issue is resolved.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_INFORMATION\\\",\\\"severity\\\":\\\"low\\\",\\\"service_key\\\":\\\"3GoNVrVz48jeNZSyLJMV\\\",\\\"service_name\\\":\\\"Classroom\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"Classroom\\\",\\\"id\\\":\\\"3GoNVrVz48jeNZSyLJMV\\\"}],\\\"uri\\\":\\\"incidents/GDxbSoojBb9yyEhU2Cua\\\"},{\\\"id\\\":\\\"tVLvwJSkYhEtcLAMsekv\\\",\\\"number\\\":\\\"6658602565448188991\\\",\\\"begin\\\":\\\"2026-04-28T07:30:00+00:00\\\",\\\"created\\\":\\\"2026-04-28T15:20:36+00:00\\\",\\\"end\\\":\\\"2026-04-28T15:50:00+00:00\\\",\\\"modified\\\":\\\"2026-04-28T16:50:51+00:00\\\",\\\"external_desc\\\":\\\"**Title:**\\\\nInvestigating issues with slow loading of AppSheet\\\\n**Description:**\\\\nWe are experiencing an issue with AppSheet beginning at Tuesday, 2026-04-28 00:30 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Tuesday, 2026-04-28 09:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms:**\\\\nAffected customers would experience slow loading times and internal server errors for AppSheet apps.\\\\n**Workaround:**\\\\nNone at this time.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-04-28T16:50:32+00:00\\\",\\\"modified\\\":\\\"2026-04-28T16:50:51+00:00\\\",\\\"when\\\":\\\"2026-04-28T16:50:32+00:00\\\",\\\"text\\\":\\\"**Title:**\\\\nMitigated issues with slow loading of AppSheet apps.\\\\n**Description:**\\\\nThe issue with AppSheet has been resolved for all affected users as of Tuesday, 2026-04-28 08:50 PDT.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms:**\\\\nAffected customers would have experienced slow loading times and internal server errors for AppSheet apps.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-04-28T16:00:44+00:00\\\",\\\"modified\\\":\\\"2026-04-28T16:50:32+00:00\\\",\\\"when\\\":\\\"2026-04-28T16:00:44+00:00\\\",\\\"text\\\":\\\"**Title:**\\\\nInvestigating issues with slow loading of AppSheet.\\\\n**Description:**\\\\nOur engineering team has implemented a fix and system telemetry indicates that latency and error rates have returned to normal levels. We are continuing to monitor the environment closely to ensure full stability.\\\\nWe will provide an update by Tuesday, 2026-04-28 10:30 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms:**\\\\nAffected customers would experience slow loading times and internal server errors for AppSheet apps.\\\\n**Workaround:**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"},{\\\"created\\\":\\\"2026-04-28T15:20:36+00:00\\\",\\\"modified\\\":\\\"2026-04-28T16:00:44+00:00\\\",\\\"when\\\":\\\"2026-04-28T15:20:36+00:00\\\",\\\"text\\\":\\\"**Title:**\\\\nInvestigating issues with slow loading of AppSheet\\\\n**Description:**\\\\nWe are experiencing an issue with AppSheet beginning at Tuesday, 2026-04-28 00:30 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Tuesday, 2026-04-28 09:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Symptoms:**\\\\nAffected customers would experience slow loading times and internal server errors for AppSheet apps.\\\\n**Workaround:**\\\\nNone at this time.\\\",\\\"status\\\":\\\"SERVICE_INFORMATION\\\"}],\\\"most_recent_update\\\":{\\\"created\\\":\\\"2026-04-28T16:50:32+00:00\\\",\\\"modified\\\":\\\"2026-04-28T16:50:51+00:00\\\",\\\"when\\\":\\\"2026-04-28T16:50:32+00:00\\\",\\\"text\\\":\\\"**Title:**\\\\nMitigated issues with slow loading of AppSheet apps.\\\\n**Description:**\\\\nThe issue with AppSheet has been resolved for all affected users as of Tuesday, 2026-04-28 08:50 PDT.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Symptoms:**\\\\nAffected customers would have experienced slow loading times and internal server errors for AppSheet apps.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},\\\"status_impact\\\":\\\"SERVICE_INFORMATION\\\",\\\"severity\\\":\\\"low\\\",\\\"service_key\\\":\\\"FWjKi5U7KX4FUUPThHAJ\\\",\\\"service_name\\\":\\\"AppSheet\\\",\\\"affected_products\\\":[{\\\"title\\\":\\\"AppSheet\\\",\\\"id\\\":\\\"FWjKi5U7KX4FUUPThHAJ\\\"}],\\\"uri\\\":\\\"incidents/tVLvwJSkYhEtcLAMsekv\\\"},{\\\"id\\\":\\\"JTgxXCtXTaw3aEVzUUfa\\\",\\\"number\\\":\\\"3709370212894413327\\\",\\\"begin\\\":\\\"2026-04-10T02:08:00+00:00\\\",\\\"created\\\":\\\"2026-04-10T10:07:12+00:00\\\",\\\"end\\\":\\\"2026-04-10T14:50:00+00:00\\\",\\\"modified\\\":\\\"2026-04-10T15:06:53+00:00\\\",\\\"external_desc\\\":\\\"**Summary**\\\\nGoogle Workspace users using Google IdP are unable to sign into their accounts using security keys and hardware based passkeys.\\\\n**Description**\\\\nWe are experiencing an issue with Workspace users beginning on Thursday, 2026-04-09 19:08 PDT.\\\\nOur engineering team continues to investigate the issue.\\\\nWe will provide an update by Friday, 2026-04-10 04:00 PDT with current details.\\\\nWe apologize to all who are affected by the disruption.\\\\n**Customer Symptoms**\\\\nGoogle Workspace users using Google IdP are unable to sign into their accounts using security keys and hardware based passkeys.\\\\n**Workaround**\\\\nBelow steps should be performed by Admin as the workaround:\\\\n- Admin should revoke the key.\\\\n- Admin to create the verification code to sign-in.\\\\n- The user should sign-in using Backup verification code and then re-enrolls the security key.\\\",\\\"updates\\\":[{\\\"created\\\":\\\"2026-04-10T15:06:51+00:00\\\",\\\"modified\\\":\\\"2026-04-10T15:06:53+00:00\\\",\\\"when\\\":\\\"2026-04-10T15:06:51+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGoogle Workspace users using Google IdP were unable to sign into their accounts using security keys and hardware based passkeys.\\\\n**Description**\\\\nThe issue with Admin Console has been resolved for all affected users as of Friday, 2026-04-10 07:50 PDT.\\\\nSuspected CL have been rolled back and we are seeing in our graphs errors rate have gone down. Customers also confirmed that issues have been mitigated.\\\\nWe thank you for your patience while we worked on resolving the issue.\\\\n**Customer Symptoms**\\\\nGoogle Workspace users using Google IdP were unable to sign into their accounts using security keys and hardware based passkeys.\\\\n**Workaround**\\\\nBelow steps should be performed by Admin as the workaround:\\\\n* Admin should revoke the key.\\\\n* Admin to create the verification code to sign-in.\\\\n* The user should sign-in using Backup verification code and then re-enrolls the security key.\\\",\\\"status\\\":\\\"AVAILABLE\\\"},{\\\"created\\\":\\\"2026-04-10T11:27:56+00:00\\\",\\\"modified\\\":\\\"2026-04-10T15:06:51+00:00\\\",\\\"when\\\":\\\"2026-04-10T11:27:56+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGoogle Workspace users using Google IdP are unable to sign into their accounts using security keys and hardware based passkeys.\\\\n**Description**\\\\nMitigation work is currently underway by our engineering team.\\\\nWe do not have an ETA for mitigation at this point.\\\\nWe will provide more information by Friday, 2026-04-10 09:00 PDT.\\\\n**Customer Symptoms**\\\\nGoogle Workspace users using Google IdP are unable to sign into their accounts using security keys and hardware based passkeys.\\\\n**Workaround**\\\\nBelow steps should be performed by Admin as the workaround:\\\\n- Admin should revoke the key.\\\\n- Admin to create the verification code to sign-in.\\\\n- The user should sign-in using Backup verification code and then re-enrolls the security key.\\\",\\\"status\\\":\\\"SERVICE_DISRUPTION\\\"},{\\\"created\\\":\\\"2026-04-10T10:55:50+00:00\\\",\\\"modified\\\":\\\"2026-04-10T11:27:56+00:00\\\",\\\"when\\\":\\\"2026-04-10T10:55:50+00:00\\\",\\\"text\\\":\\\"**Summary**\\\\nGoogle Workspace users using Goog\n[...truncated]", "status": "completed", "tool_name": "web_fetch"}