Google Cloud GCVE incident details

What happened? Summary

During the initial deployment of Google Cloud VMware Engine (GCVE) private clouds for customers using our internal tools, a parameter was left blank, causing a Google operator to misconfigure the GCVE service. This had the unintended and subsequently unknown consequence of causing customer GCVE private clouds to default to a fixed duration and then be automatically deleted at the end of that duration. Both the incident trigger and the behavior of our downstream systems have been fixed to ensure this does not happen again.

This incident did not affect any Google Cloud services other than this customer's GCVE private cloud. No other customers were affected by this incident.

Digging deeper:

Deployment using exception processing

In early 2023, our operators deployed one of the customer's GCVE private clouds using our internal tool to meet a specific capacity placement need. This internal tool for capacity management was retired in Q4 2023 and is no longer required as it has become fully automated (i.e. no human intervention is required).

Blank input parameters caused unintended behavior

Google operators followed their internal management protocols, but left one input parameter blank when using their internal tools to provision a customer's private cloud. Because the parameter was blank, the system assigned it a then-unknown default fixed duration value of one year.

Your GCVE Private Cloud was deleted after the system-allocated 1-year period expired. No notification was sent to you as the deletion was triggered as a result of a Google operator leaving a parameter blank using an internal tool, not as a deletion request from you. In the case of a customer-initiated deletion, you are notified in advance.

recovery

The customer and our teams worked around the clock over several days to recover the customer's GCVE private cloud, restore network and security configurations, restore applications, and recover data to restore full operations.

This was supported by the customer's robust and resilient architectural approach to managing the risk of outages and failures.

Backups of data stored in Google Cloud Storage in the same region were not affected by the deletion and, along with third-party backup software, helped with quick restores.

repair

Google Cloud has taken several steps to prevent this incident from happening again:

The internal tool that triggered this chain of events has been deprecated; this aspect is now fully automated and controllable by the customer via the user interface, even if specific capacity management is required.

We scrubbed our system databases and manually reviewed all GCVE private clouds to ensure that no other GCVE deployments were at risk.

We have corrected the system behavior of marking GCVE private clouds for deletion in such deployment workflows.

Conclusion

Prior to this incident, there have been no incidents of this nature within Google Cloud, and this is not a systemic issue.

Google Cloud services have strong safeguards in place that combine soft deletion, advance notification, and human intervention when necessary.

We have confirmed that these safety measures remain in place.

Close collaboration with the customer was essential to a rapid recovery, and the customer's CIO and technical teams deserve credit for the speed and precision with which they worked closely with the Google Cloud team to execute the recovery 24/7.

Fail-safe, resilient and robust risk management is essential to recover quickly in the event of an unexpected incident.

Google Cloud continues to deliver the most resilient and stable cloud infrastructure in the world, and despite this one-time incident, Google Cloud's uptime and resiliency has been independently verified to be the best of any major cloud.

Sources 1/ https://Google.com/ 2/ https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident The mention sources can contact us to remove/changing this article

What Are The Main Benefits Of Comparing Car Insurance Quotes Online

LOS ANGELES, CA / ACCESSWIRE / June 24, 2020, / Compare-autoinsurance.Org has launched a new blog post that presents the main benefits of comparing multiple car insurance quotes. For more info and free online quotes, please visit https://compare-autoinsurance.Org/the-advantages-of-comparing-prices-with-car-insurance-quotes-online/ The modern society has numerous technological advantages. One important advantage is the speed at which information is sent and received. With the help of the internet, the shopping habits of many persons have drastically changed. The car insurance industry hasn't remained untouched by these changes. On the internet, drivers can compare insurance prices and find out which sellers have the best offers. View photos The advantages of comparing online car insurance quotes are the following: Online quotes can be obtained from anywhere and at any time. Unlike physical insurance agencies, websites don't have a specific schedule and they are available at any time. Drivers that have busy working schedules, can compare quotes from anywhere and at any time, even at midnight. Multiple choices. Almost all insurance providers, no matter if they are well-known brands or just local insurers, have an online presence. Online quotes will allow policyholders the chance to discover multiple insurance companies and check their prices. Drivers are no longer required to get quotes from just a few known insurance companies. Also, local and regional insurers can provide lower insurance rates for the same services. Accurate insurance estimates. Online quotes can only be accurate if the customers provide accurate and real info about their car models and driving history. Lying about past driving incidents can make the price estimates to be lower, but when dealing with an insurance company lying to them is useless. Usually, insurance companies will do research about a potential customer before granting him coverage. Online quotes can be sorted easily. Although drivers are recommended to not choose a policy just based on its price, drivers can easily sort quotes by insurance price. Using brokerage websites will allow drivers to get quotes from multiple insurers, thus making the comparison faster and easier. For additional info, money-saving tips, and free car insurance quotes, visit https://compare-autoinsurance.Org/ Compare-autoinsurance.Org is an online provider of life, home, health, and auto insurance quotes. This website is unique because it does not simply stick to one kind of insurance provider, but brings the clients the best deals from many different online insurance carriers. In this way, clients have access to offers from multiple carriers all in one place: this website. On this site, customers have access to quotes for insurance plans from various agencies, such as local or nationwide agencies, brand names insurance companies, etc. "Online quotes can easily help drivers obtain better car insurance deals. All they have to do is to complete an online form with accurate and real info, then compare prices", said Russell Rabichev, Marketing Director of Internet Marketing Company. CONTACT: Company Name: Internet Marketing CompanyPerson for contact Name: Gurgu CPhone Number: (818) 359-3898Email: [email protected]: https://compare-autoinsurance.Org/ SOURCE: Compare-autoinsurance.Org View source version on accesswire.Com:https://www.Accesswire.Com/595055/What-Are-The-Main-Benefits-Of-Comparing-Car-Insurance-Quotes-Online View photos