Problem 23: How Do We Query Massive Data?

Query Data Lakes

Concepts appear as they come up…

Our Data Lake is growing every

00:00/01:32

1 / 1

Query Data Lakes

Certisured is an Edtech based out of Bangalore and is Co-founding team of Wizori - built to provide quick fundamental clarity in form of short vibes.

www.certisured.com

Query Data Lakes

Loading comments…

Continue your learning — your way

Sign in to unlock the full playlist, save your progress, and get a personalized feed.

Up next

Problem 6: Do We Really Need a Server?
1:29
Problem 22: How Do We Analyze Massive Data?
1:39
Problem 4: Who Should Handle the Next User?
1:32
Problem 5: what if demand for machines increases?
1:26
Problem 7: Are Files Enough for Business?
1:45
Problem 2 : Computer is temporary but, Data is permanent
1:20
Problem 21: Where Should We Keep Our Secrets?
1:44
Every Cloud Service Begins With a Problem
1:16
Problem 20: How Do Services Complete One Workflow?
1:38
Problem 3: Shared Object Storage
1:32
Engineering Problem 1: Renting a Computer
1:23
Problem 8: Must We Ask the Database Every Time?
1:38

Problem 23: How Do We Query Massive Data? — full transcript

Query Data Lakes

Query Data Lakes

Our Data Lake is growing every day. It stores application logs. Sales records. Customer activity. Sensor readings. Business reports. Everything is safely stored. But now the business team has a question. "How many customers purchased this product last month? " Does that mean we must first copy all the data into a database? For petabytes of data, that would be slow, expensive and unnecessary. The engineering problem became clear. How do we analyze massive datasets without first loading them into a database? The engineering concept that solved this problem is Serverless Query Engines. Instead of moving data to the query engine, the query engine goes to the data. Engineers simply write SQL queries, and the cloud scans only the required files directly from the Data Lake. No database servers. No data migration. Just query the data where it already lives. Amazon Web Services provides this through Amazon Athena. Microsoft Azure provides Azure Synapse Analytics (Serverless SQL). Google Cloud provides BigQuery. Different names. One engineering concept. Organizations could now analyze terabytes and even petabytes of data in minutes without building dedicated analytics databases. But another challenge soon emerged. Running SQL queries answers questions. What if we want the system to learn from the data and make predictions?