Cloud Solutions for Data Engineering

From one pipeline to a platform

In Introduction to Data Engineering, you built the mental model: engineers move data reliably under clear contracts.

Then you added:

• Data Warehousing: facts, dimensions, and analytical grain
• Database Querying: SQL as a reliable interface to modeled data
• ETL Pipelines: extraction state, transformations, safe loads, and operations

Now we’ll place those ideas in the cloud. The cloud is not magic; it is a set of managed building blocks for storage, compute, orchestration, security, and observability.

By the end, you should be able to look at a cloud architecture and ask: Where does data live? Who computes on it? How is it secured? How do we know it worked?

Think in layers

Most cloud data platforms are combinations of five layers:

Users / BI / ML
      |
Warehouse + Query Engines
      |
Transform + Orchestration
      |
Object Storage / Tables
      |
Sources: apps, events, APIs, databases

The key shift from local infrastructure is that these layers are often managed services.

That means the cloud provider handles much of the undifferentiated work: machines, disks, replication, patching, and scaling. Your team still owns the important engineering decisions: schemas, data quality, access, cost, reliability, and business correctness.

A useful question for every cloud component is: What responsibility did we outsource, and what responsibility stayed with us?

You are designing an analytics platform with large historical datasets and bursty SQL/Spark workloads. Some days analysts scan terabytes; other days the system is quiet.

Which architecture best fits this usage pattern?

The cloud pricing mindset

A major cloud data engineering idea is separating storage from compute.

• Storage: durable files, tables, snapshots, metadata
• Compute: Spark clusters, SQL warehouses, serverless jobs, streaming workers

This separation gives flexibility, but it also creates a new engineering responsibility: cost awareness.

A query that scans unnecessary partitions is not just slow; it may directly cost money. This is where your earlier lessons connect: warehouse grain, query filters, and pipeline design all influence cloud cost.

```python
# Simple cloud data cost estimator.
# Numbers are illustrative, not vendor-specific pricing.

GB = 1024 ** 3
TB = 1024 * GB

storage_tb = 8
scan_tb_per_day = 2.5
compute_hours_per_day = 3

object_storage_per_gb_month = 0.023
query_per_tb_scanned = 5.00
compute_per_hour = 2.40

days = 30
storage_monthly = storage_tb * 1024 * object_storage_per_gb_month
query_monthly = scan_tb_per_day * query_per_tb_scanned * days
compute_monthly = compute_hours_per_day * compute_per_hour * days

total = storage_monthly + query_monthly + compute_monthly

print(f"Storage: ${storage_monthly:,.2f}/month")
print(f"Query scans: ${query_monthly:,.2f}/month")
print(f"Compute: ${compute_monthly:,.2f}/month")
print(f"Estimated total: ${total:,.2f}/month")

# What if better partition filters cut scanned data by 60%?
optimized_scan_tb_per_day = scan_tb_per_day * 0.40
optimized_query_monthly = optimized_scan_tb_per_day * query_per_tb_scanned * days
optimized_total = storage_monthly + optimized_query_monthly + compute_monthly

print(f"Optimized total: ${optimized_total:,.2f}/month")
print(f"Monthly savings: ${total - optimized_total:,.2f}")
```

What gets managed?

Cloud platforms offer managed versions of the pieces you used in ETL Pipelines:

Pipeline need Common cloud solution Raw durable landing zone Object storage buckets/containers Batch compute Managed Spark, serverless jobs, containers SQL analytics Cloud warehouse or lakehouse query engine Scheduling Managed workflow orchestrator Streaming Managed event bus or streaming service Metadata Catalog, schema registry, lineage tools Secrets Secret manager / key vault

The design goal is not to use every service. The goal is to pick the smallest set that satisfies your contracts, latency needs, scale, and team maturity.

A simple batch analytics platform might only need object storage, a scheduler, a transformation engine, and a warehouse.

True or false:

Using managed cloud services means your team no longer owns data quality, schema contracts, access policy, or pipeline monitoring.

Production cloud data needs guardrails

Cloud data systems are powerful because access is easy to grant, storage is easy to create, and compute is easy to scale. Those same strengths can become risks.

Key guardrails:

• IAM and least privilege: jobs get only the permissions they need
• Secrets management: credentials are never hard-coded in notebooks or scripts
• Encryption: protect data at rest and in transit
• Network boundaries: private endpoints, restricted ingress, controlled egress
• Data classification: know which datasets contain sensitive fields
• Audit logs: track who accessed what and when

This connects directly to the earlier idea of data contracts. A contract is not only about column names and types; in production, it also includes ownership, access rules, freshness, and expected usage.

In many cloud lakehouse architectures, raw files first land in durable ____ before Spark or warehouse query engines read them.

Type the two-word term.

Cloud pipelines must be measurable

In ETL Pipelines, you learned to operate the pipeline, not just write it. In the cloud, that means emitting metrics that can drive alerts, dashboards, and cost decisions.

Useful metrics include:

• Rows extracted, transformed, and loaded
• Bytes scanned or written
• Runtime and retry count
• Freshness delay
• Failed contract checks
• Cost-related dimensions such as warehouse size or cluster time

If you cannot measure a pipeline, you cannot confidently scale it.

```python
import json
from datetime import datetime, timezone

pipeline_run = {
    "pipeline": "orders_daily_etl",
    "run_id": "2026-04-27T09:00:00Z",
    "rows_loaded": 1_240_500,
    "bytes_scanned_gb": 180,
    "runtime_minutes": 42,
    "freshness_minutes": 75,
    "contract_failures": 0,
}

thresholds = {
    "max_runtime_minutes": 60,
    "max_freshness_minutes": 90,
    "max_contract_failures": 0,
}

status = "ok"
alerts = []

if pipeline_run["runtime_minutes"] > thresholds["max_runtime_minutes"]:
    status = "alert"
    alerts.append("runtime exceeded threshold")

if pipeline_run["freshness_minutes"] > thresholds["max_freshness_minutes"]:
    status = "alert"
    alerts.append("freshness exceeded threshold")

if pipeline_run["contract_failures"] > thresholds["max_contract_failures"]:
    status = "alert"
    alerts.append("data contract failures detected")

log_event = {
    "event_time": datetime.now(timezone.utc).isoformat(),
    "status": status,
    "alerts": alerts,
    **pipeline_run,
}

print(json.dumps(log_event, indent=2))
```

You now have the full map

This course followed the path of a real data engineering system:

1. Introduction to Data Engineering: reliability, ownership, contracts
2. Data Warehousing: model data for analytics
3. Database Querying: ask correct, performant questions
4. ETL Pipelines: move and transform data safely
5. Cloud Solutions for Data Engineering: scale the system with managed infrastructure

When you evaluate a cloud data platform, use this checklist:

• Where does raw, staged, and curated data live?
• What compute engine transforms or queries it?
• What warehouse or lakehouse model exposes trusted data?
• How are jobs scheduled, retried, and monitored?
• How are secrets, permissions, and sensitive data protected?
• How does the design control cost?
• What contracts prove the data is correct enough to use?

That is the core mindset of a data engineer: not just moving data, but building trustworthy data systems that survive real usage.