Catching up with Google BigQuery

With the ink just drying Google all-new acquisition of Looker, all eyes are on BigQuery as for plans to extend the footprint of the platform. Fueling anticipation is the fact that GCP's cloud data warehousing competes, including Microsoft, Oracle, and SAP, have recently extended the scope of their offerings to include back-end data integration or self-service front-end BI visualization. While Google claims that Looker will retain its multi-cloud platform support, in GCP, BigQuery seems to be the logical target for improved integration.

We recently participated in an update call that looked at recently introduced features, ranging from the general availability of Redshift and S3 migration tools to memory in memory. BI engine to beta versions for Flexible locations and column level security. BigQuery has been one of GCP's fastest growing services, with the customer base having grown significantly over the past 18 months, and more importantly, the number of flat rate customers (as opposed to the card per request ) having doubled in number in the past year. . In one barely published blog, Google highlighted big wins with customers such as KeyBank, Wayfair, Lowe & # 39; s, Saber and Lufthansa.

Big Query is unique in that, unlike most cloud data warehousing services, it is serverless. Traditionally, you used it on an ad hoc basis and didn't worry about supply nodes, although later on location pricing was introduced to make BigQuery costs more predictable for large-scale users. Serverless is also useful for handling highly competitive scenarios, with Google claiming that some BigQuery users have executed up to 10,000 queries at a time.

A typical scenario for adopting BigQuery is to take advantage of the platform scale, both in terms of data volumes (with quirky petabyte size queries) and competition high. Although there is no equivalent of the CAP theorem when it comes to evolving compared to the competition in analytical databases, for most data warehousing platforms, it is usually about A choice between one or the other.

At the origin of the growth of Google's log processing system, BigQuery is the platform on which Dremel query engine was developed; this is the engine on which Apache Drill was developed. BigQuery can store a variety of data going beyond typical relational structured data to formats such as Parquet, JSON or CSV and can use cloud object storage as a source; while such extensibility is not unusual today among other cloud storage platforms, BigQuery was one of the first to offer such extensibility.

Originally, BigQuery did not look like a typical data warehouse, because it worked best when the data was organized in nested structures which, at first glance, looked more like JSON documents than relational schemas or in typical SQL star. Since then, Google claims that BigQuery has evolved so that it can now work efficiently with more traditional data warehouse schemas.

So customers will likely need help moving data to BigQuery given its unique layout. Partners such as Datométrie and CompilerWorks have developed migration tools to move workloads without having to rewrite queries. Informatica has developed a BigQuery codeless / low-code integration tool that includes a six-step wizard for less technical business users to guide them through the process. In turn, global ISs such as Accenture, Infosys and Wipro have developed migration tools as part of their own BigQuery practices. Google recently expands its partnership with SADA systems, a global cloud consulting and managed services provider, which was also one of GCP's first partners. They have returned to a $ 500 million deal that will include support for migrations from Netezza, Teradata, and Hadoop to BigQuery.

With regard to tooling, Google subscribes to a coopetition model; in the past year, he has made several acquisitions. At NEXT last year, Google announced Cloud Data Fusion, the result of its acquisition of the open source company behind the development of open source CDAP technology, which manages data transformation pipelines within Google Cloud Dataproc, the Hadoop service of GCP. Google later acquired Alooma, which instead uses a staging server approach that is akin to AWS and Azure blue Database migration services. While these offerings help complete the GCP portfolio, as a newbie to the cloud platform ecosystem, we don't expect Google to aggressively sell these services in competition to its partners.

One of the main selling points of cloud data platform providers is to exploit synergies between their portfolios. The history of BigQuery's federated queries should soon become GA. Today, it can reach Cloud SQL (GCP's MySQL and PostgreSQL services) and Bigtable (the NoSQL database that was the inspiration for Hadoop's HBase). We believe that in the future, GCP will add Spanner to this list.

BigQuery has also gotten wet with machine learning (ML) by making it more accessible to SQL developers. Typically, these features allow developers to run ML models without having to write Python or R code, and for BigQuery, they now support various training models for linear regression (to predict numerical values ); K-means clustering (for customer segmentation); matrix factorization (in Alpha, for recommendation systems); XGBoost; (for regression, classification and classification); Deep neural networks (using TensorFlow) and others.

Google is hardly the only one here with the ability to fire ML models from SQL code so that they can run inside the database without having to moving the data starts to become a checkbox item. But most of the others (for example, Amazon Redshift, Oracle, and SQL Server on-premises) generally treat the R or Python programs used for ML as user-defined functions, rather than BigQuery's storage of the models in them. data sets themselves. In addition, BigQuery's serverless architecture has made the platform better suited to training models compared to most cloud data warehousing services.

So what's the next step? We expect the obvious answer to be how Google will combine Looker's data integration and visualization capabilities into its broader data platform offering. With Microsoft recently unveiling Synapse, which places Azure Data Factory under a common service, Oracle is expanding the stand-alone data warehouse to incorporate self-service data integration tools, while SAP has expanded its HANA data warehouse to use cloud analytics SAP Analytics, Looker and BigQuery is looking more and more like one another. But, we are also interested to see if Google will designate BigQuery as one of the services that could be supported as part of itsAnthos hybrid platform. We expect to receive some of these responses in April at Google NEXT.