Data Lake
Data Lake
Our application has been running for years. Every click. Every purchase. Every login. Every sensor reading. Every application log. The data keeps growing. Soon, traditional databases begin to struggle. Not because they're slow... ... but because they were designed for running applications, not storing and analyzing enormous volumes of raw data. The engineering problem became clear. How do we store massive amounts of structured and unstructured data at low cost so it can be analyzed later? The engineering concept that solved this problem is the Data Lake. Instead of deciding how data should be organized before storing it, a Data Lake stores everything in its original form. Images. Videos. CSV files. JSON. Logs. Documents. Sensor data. Nothing is discarded. The data is simply collected and made available for future analysis. Amazon Web Services builds Data Lakes primarily on Amazon Simple Storage Service (S3). Microsoft Azure provides Azure Data Lake Storage. Google Cloud uses Cloud Storage as the foundation of its Data Lake architecture. Different names. One engineering concept. Organizations could now preserve virtually unlimited amounts of data without forcing it into a database first. But storing data was only half the solution. How do we ask questions directly on this massive Data Lake without first moving the data into a database?