Data Engineering

Sign in Subscribe

Latest — Nov 22, 2024

Goodbye Netlify, Hello Ghost: My Blog's New Home

In October 2021, I published my first blog post on Medium: Pub/Sub to BigQuery: How to Build a Data Pipeline Using Dataflow, Apache Beam, and Java. The feedback I received was incredible, and it motivated me—I wanted a personal blog, a space where I could share my thoughts

More issues

Who’s Really Following You on Dev.to? A Guide to Analyzing Your Audience

The reason I’m writing this post is to shed some light on an aspect of Dev.to that many of us don’t think twice about: our followers. We put so much effort into creating content, hoping it resonates with readers and builds our community, but have you ever

12 Steps to Organize and Maintain Your Python Codebase for Beginners

The reason why I’m writing this post is to share some insights on keeping a project clean, even with lots of contributors. This is especially important for data engineers, given the ever-changing nature of data and the processing demands in Python libraries and applications. The title might sound a

My Journey Learning AI for Songwriting: LSTMs and Taylor Swift

I spent last 4 months learning with the Machine Learning Zoomcamp. When it was time for the final project, I wanted to try something new that wasn’t covered in our sessions. Then, I heard all the hype about Taylor Swift’s tour and thought, ‘Why not try creating lyrics

The Data Engineering Docker-Compose Starter Kit

We’ve all been there. Starting a new data engineering project and, instead of diving right into the fun stuff, we’re stuck setting up configurations. Again. So, I’ve put together a Docker Compose config with a bunch of services you’ll often see in data engineering projects. Just

The Golang Saga: A Coder’s Journey There and Back Again. Part 3: The Graphing Conundrum

Welcome back to the third part of “The Golang Saga: A Coder’s Journey There and Back Again.” In the first part of this series, I started work on a personal project, the Climate Change Visualizer, using Go as my chosen programming language. In the second part, we delved into

The Golang Saga: A Coder’s Journey There and Back Again. Part 2: The Data Expedition

Welcome back to the second part of “The Golang Saga: A Coder’s Journey There and Back Again.” In the first part of this series, I started working on a personal project, the Climate Change Visualizer, using Golang as my chosen programming language. Just a quick recap, I had already

The Golang Saga: A Coder’s Journey There and Back Again. Part 1: Leaving the Shire

The idea of learning Golang has been circulating in my head for quite some time. However, like many of us, I constantly found excuses to put it off. Just the other day, though, I was reading this compelling article that talked about how the key to effective learning lies in

Unleash Your Pipeline Creativity: Local Development with Argo Workflows and MinIO on Minikube

As a data engineer, you know the joy of wrangling massive datasets and navigating complex data pipelines. Argo Workflows, a popular workflow engine for Kubernetes, is your trusty companion in this data-driven journey, allowing you to define, run, and manage data pipelines as code. However, like any adventure, there are

From Kafka to Amazon S3: Partitioning Outputs

So you’ve got a ton of data that needs to be processed in real-time, huh? Don’t worry, in this tutorial I’ll show you how to stream data from Kafka compatible streaming platform to Amazon S3. But wait, there’s more! I’ll also cover how to create