Welcome to DATA 707!
Introduction to Data Engineering
This course introduces the data engineering lifecycle, which is the foundation for building reliable data systems that power analytics and machine learning. Students learn how data moves from source systems through ingestion, transformation, and serving layers, and how to design pipelines that are reproducible, testable, and maintainable. Hands-on work uses Python and cloud computing to build end-to-end pipelines on real datasets. No prior experience with cloud tools or data engineering is assumed.
Course Philosophy
- We focus on the fundamental principles for this half-semester introductory course.
- Topics not planned for in-depth coverage include LLM-focused systems (e.g., RAG), open-source tools, computer networking, cybersecurity, vector and graph databases, distributed computing, and advanced software engineering.
- We will use Microsoft Fabric for course exercises and projects, but DATA 707 is tool-agnostic. The goal is not to turn the course into a product tutorial or bootcamp.
- For hands-on tutorials, start with free resources from DataCamp (SQL, Databricks, Snowflake, Kafka, Airflow, Spark, Power BI, Tableau, dbt, AWS, and Google Cloud) and Microsoft Learn.
Learning Objectives
By the end of the course, students should be able to:
- Explain the data engineering lifecycle and the role of data engineers in modern data-driven organizations.
- Design and implement end-to-end data pipelines that ingest, transform, and serve data.
- Select appropriate storage systems and pipeline architectures for a given use case, justifying tradeoffs.
- Apply DataOps best practices to build reliable, maintainable pipelines.
Course Materials
Read the sections relevant to each lecture, and feel free to skip material you already know or that goes beyond the depth needed for this course.
- [RH] Fundamentals of Data Engineering: Plan and Build Robust Data Systems — Joe Reis and Matt Housley
- [D] Data Pipelines Pocket Reference: Moving and Processing Data for Analytics — James Densmore
Lecture Content
Each lecture will take up 1~2 class meetings.
| Lecture | Title | Topics | Recommended Readings |
|---|---|---|---|
| 1 | Introduction to Data Engineering | • What is data engineering? Learning objectives • Data engineer vs. data scientist roles • Importance of cloud computing: UNC computing resources and Microsoft Fabric • The data engineering lifecycle and system design • Medallion architecture • Cross-cutting undercurrents: security, DataOps, architecture, and software engineering • Structured, semi-structured, and unstructured data • Modern data architectures: warehouse, lake, and lakehouse |
[RH] Ch. 1; selected Ch. 2 (lifecycle overview and undercurrents) [D] Ch. 1 |
| 2 | Stage 1: Data Generation | • Source systems and data quality • Files, APIs, databases, and event streams • REST APIs • Databases: OLTP vs. OLAP, CRUD, and SQL • Metadata |
[RH] Ch. 5 |
| 3 | Stage 2: Storage | • Storage in the data pipeline • Storage systems: RDBMS, NoSQL, and streaming; ACID vs. BASE • Storage technology: object, block, and file • Data layout: row vs. column; file formats: Avro, Parquet, and ORC • Revisiting OLTP vs. OLAP and warehouse, lake, and lakehouse architectures |
[RH] Ch. 6; selected Ch. 3 (architecture) [D] Ch. 2 |
| 4 | Stage 3: Ingestion | • Engineering considerations: bounded vs. unbounded data, ingestion frequency, asynchronous vs. synchronous, and push vs. pull vs. poll • Batch ingestion and change data capture (CDC) • Streaming and watermarks • ELT vs. ETL • Kappa vs. Lambda architectures |
[RH] Ch. 7 [D] Ch. 3 (ETL/ELT patterns); selected parts of Chs. 4–5 (extracting and loading data) |
| 5 | Stage 4: Transformation | • Cleaning • Query operations: filtering, joins, grouping, and aggregation • Data modeling: normalization vs. denormalization, star vs. snowflake schemas, and Kimball vs. Inmon (data vault) |
[RH] Ch. 8 [D] Ch. 6; selected Ch. 8 (validation) |
| 6 | Stage 5: Serving | • Data products, definitions, logic, and semantic models • Use cases and users • Business, operational, and embedded analytics • Dashboards and reports • Reverse ETL • APIs as a serving layer • Data-product consumers • Data contracts and SLAs |
[RH] Ch. 9 |
| 7 | Orchestration and Governance | • Orchestration: dependencies, DAGs, and scheduling • Security and privacy • Data management • Observability |
[RH] Selected Chs. 2 (undercurrents) and 10 (security and privacy) [D] Selected Chs. 7 (orchestration), 8 (validation), 9 (pipeline maintenance), and 10 (monitoring) |
Grading
Final course grades will be determined using the UNC Graduate School grading scale.
| Component | Weight | Description |
|---|---|---|
| Participation & Engagement | 10% | Assesses preparation, active participation in class and project work, engagement with guest speakers, and constructive contribution to the learning environment. |
| Quizzes | 15% | Short, in-class, individual pen-and-paper quizzes assess understanding of core data engineering concepts and the ability to apply them to realistic scenarios. Questions focus on design concepts rather than programming. |
| Project Milestones | 25% | Incremental group submissions assess progress through the data engineering lifecycle and provide opportunities for feedback and revision before the final project. |
| Final Project Report & Demo | 20% | The final group deliverable demonstrates a complete, working data pipeline and communicates the team’s engineering decisions, validation, and lessons learned. |
| Individual Oral Exam | 30% | Assesses each student’s independent ability to explain, defend, troubleshoot, and adapt data engineering systems, including work completed with AI assistance. |
Quizzes [15%]
There will be seven short in-class, individual, pen-and-paper quizzes after each lecture is concluded. Quizzes emphasize application and engineering reasoning, rather than memorization of terminology or programming. Quizzes are designed to take approximately 10–15 minutes.
Project Milestones [25%]
Projects are completed in groups of two. The milestones are not separate mini-projects: they are successive iterations of the same end-to-end data engineering project. You are expected to revise earlier design decisions as you learn more.
| Milestone | Due | Weight |
|---|---|---|
| 1. Project Proposal & Feasibility | Oct. 27 | 3% |
| 2. Storage Architecture | Nov. 5 | 4% |
| 3. Ingestion | Nov. 10 | 5% |
| 4. Transformation & Data Quality | Nov. 17 | 5% |
| 5. Serving & Data Contract | Nov. 24 | 4% |
| 6. Production Design | Dec. 1 | 4% |
Final Project [20%]
Final Blog Post / Report [15%]
Due Sunday, Dec. 6 at 11:59 PM.
Ideally, submit a blog post suitable for a public portfolio. A standard PDF report is also accepted. The report must cover:
- Problem and Consumer
- Data and Source System
- Pipeline Architecture
- Ingestion
- Transformation
- Serving
- Data Contract
- Engineering Decisions — Data Architecture
- Orchestration (Design Only)
- Data Governance (Design Only)
- DataOps (Design Only)
- Lessons Learned
The final submission should demonstrate not only that the pipeline works, but that the team can explain and justify its engineering decisions.
Demo Recording [5%]
Submit a short demonstration of the working pipeline and final data product. The demo should show the system rather than repeat the written report.
Individual Oral Exam [30%]
Each student will complete an individual oral exam of approximately 20 minutes. Prepared slides, electronic devices, and AI tools are not permitted during the exam. The oral exam is designed to assess understanding and engineering reasoning, not memorization.
Generative AI Use
Generative AI tools may be used in this course, but students remain fully responsible for the accuracy, relevance, quality, and understanding of all submitted work.
Written English Text
AI tools may be used to assist with drafting, grammar, formatting, outlining, or organizing ideas, but students must substantially review, revise, and integrate the output into their own work. Unmodified AI-generated text is not acceptable.
You are responsible for removing irrelevant or unnecessary details, correcting inaccuracies, improving organization and flow, and ensuring that the discussion stays within the scope of this course and is directly relevant to your project. Generic, overly verbose, poorly organized, or project-irrelevant writing may receive reduced credit, regardless of whether AI was used. Appropriate citations must be provided for external sources.
Programming Code
Students are allowed and encouraged to use AI tools to assist with programming, including “vibe coding,” as long as they verify correctness and take full responsibility for the final output.
You are responsible for understanding how the code works, why it is appropriate for your pipeline, and what assumptions or limitations it has. You should be able to explain, troubleshoot, and modify any code you submit, including code generated with AI assistance. Code that runs successfully but is not understood, is unnecessarily complex, or does not appropriately address the project requirements may receive reduced credit.
Oral Exam
Students are allowed and encouraged to use AI tools to prepare for the oral exam, but electronic devices are not permitted during the exam.
Project Guidelines
You are not restricted to the options below. If you would like to use another dataset or source, check with the instructor as early as possible to confirm that it is appropriately scoped and technically suitable.
The goal is to choose data that creates a meaningful data engineering problem, not simply an interesting data analysis problem. Your project should give you opportunities to make and justify decisions about storage, ingestion, transformation, data quality, serving, and production design.
Suggested datasets available in OneLake
- Synthea — synthetic electronic health records: A good option if you are interested in healthcare, though medical terminology and healthcare data structures may have a learning curve.
- TPC-DS — business data benchmark: Synthetic business data designed as a benchmark for data systems. Although it is less messy than real-world data, it provides substantial scale and multiple related tables.
- NYC 311 Service Requests: A very large service-request dataset. Because the primary dataset is relatively flat, consider restructuring or normalizing it, or enriching it with additional sources.
- MBTA Performance Data: Public transportation performance data from the Massachusetts Bay Transportation Authority. Consider combining it with geographic, weather, real-estate, or other contextual data.
Optional enrichment sources for NYC 311 or MBTA projects
- Maps: Mapbox, OpenStreetMap, or Google Maps Platform (usage may incur costs).
- Weather: NOAA/NCEI API.