Read some about Luigi and how to do pipeline/workflow stuff in standard python leveraging a package like Luigi. Then use your understanding to look into how Airflow works. Luigi https://luigi.readthedocs.

Often times, early in cohorts, students begin to discover two things. The first is that materials they are using are incomplete, vague and contain mistakes or errors that cause confusion. The second thing

Before the OS Concepts lecture, I'd like you to watch these videos. They break down the complexities of the CPU, Memory and so on within a modern microprocessor. They move at a decent

Cellular Automata. https://bitstorm.org/gameoflife/ good sim on page, several great patterns pre-built. https://www.quantamagazine.org/john-conways-life-in-games-20150828/ bio of Conway, plus good desc of game. https://web.stanford.edu/class/sts145/

Let’s say you have a bunch of files you don't want in your repo (like .class or .ctxt files?) but you've already pushed them! (doh!) And you'd like add them to your

Now, let's do some python with our kafka. (Hey, your kafka and zookeeper should still be running.) (and if not, start 'em up again) in a directory named (lines10) in your projects somewhere,

So make sure you have java 1.8 running $ java -version java version "1.8.0_66" Java(TM) SE Runtime Environment (build 1.8.0_66-b17) Java HotSpot(TM) 64-Bit

As part of your weekend agenda, read these articles, as many as you can. You might even decide to follow one through to the end. But that's up to you. Kafka is one

Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a

Read these articles, to help cement and settle some of the things you've been learning over these last weeks. None of them are challenging, you should start to see things you recognize, and

Finding cool datasets should be something you do continuously as a Data Engineer. Ponder some of these: https://www.dataquest.io/blog/free-datasets-for-projects/ https://www.analyticsindiamag.com/popular-data-science-projects-for-aspiring-data-scientists/ https://www.springboard.com/blog/

Books for Data Engineering Data Engineering Data Engineering Cookbook Plumber of DS Python Primary: Think Python Python Notes SQL Primary: Learning SQL: Master SQL Fundamentals Database Design SQL Notes PostgreSQL Notes SQL 1

Data Engineering Data Engineering Cookbook Plumber of DS Python Primary: Think Python Hitchhiker's Guide to Python Python Notes SQL Primary: Learning SQL: Master SQL Fundamentals SQL: The Complete Reference Database Design SQL Notes

To load a missing python library into your notebook server import sys print(sys.executable) will tell you what the executable path is for your python used within your notebooks. /Users/foo/anaconda3/