Data Engineering

Sign in Subscribe

Latest — Feb 2, 2025

Scaling Data Analytics: Building a Starter Kit with Neon, Airflow, and Streamlit

Setting up a data analytics project from scratch can be a headache. You need a database, a way to pull in data, and a dashboard to make sense of it all. I wanted to make that process easier—something lightweight, flexible, and beginner-friendly. That’s how I ended up building

More issues

Handling Dates in Argo Workflows

Working with dates in Argo Workflows can range from simple to unexpectedly challenging depending on your needs. Whether you're scheduling tasks, extracting specific parts of a date, or looping through hours for complex workflows there’s plenty you can achieve—but figuring it out might take some trial

Logs Don’t Lie: Debugging My Data Engineering Crisis in 2025

Every data pipeline has its breaking point. Mine came in late 2024, throwing errors I couldn’t ignore. Logs showed signs of stagnation, over-processing, and the need for a refreshed perspective. But here’s the thing: I don’t really have anything to complain about. I hold a classical Software

Goodbye Netlify, Hello Ghost: My Blog's New Home

In October 2021, I published my first blog post on Medium: Pub/Sub to BigQuery: How to Build a Data Pipeline Using Dataflow, Apache Beam, and Java. The feedback I received was incredible, and it motivated me—I wanted a personal blog, a space where I could share my thoughts

Who’s Really Following You on Dev.to? A Guide to Analyzing Your Audience

The reason I’m writing this post is to shed some light on an aspect of Dev.to that many of us don’t think twice about: our followers. We put so much effort into creating content, hoping it resonates with readers and builds our community, but have you ever

12 Steps to Organize and Maintain Your Python Codebase for Beginners

The reason why I’m writing this post is to share some insights on keeping a project clean, even with lots of contributors. This is especially important for data engineers, given the ever-changing nature of data and the processing demands in Python libraries and applications. The title might sound a

My Journey Learning AI for Songwriting: LSTMs and Taylor Swift

I spent last 4 months learning with the Machine Learning Zoomcamp. When it was time for the final project, I wanted to try something new that wasn’t covered in our sessions. Then, I heard all the hype about Taylor Swift’s tour and thought, ‘Why not try creating lyrics

The Data Engineering Docker-Compose Starter Kit

We’ve all been there. Starting a new data engineering project and, instead of diving right into the fun stuff, we’re stuck setting up configurations. Again. So, I’ve put together a Docker Compose config with a bunch of services you’ll often see in data engineering projects. Just

The Golang Saga: A Coder’s Journey There and Back Again. Part 3: The Graphing Conundrum

Welcome back to the third part of “The Golang Saga: A Coder’s Journey There and Back Again.” In the first part of this series, I started work on a personal project, the Climate Change Visualizer, using Go as my chosen programming language. In the second part, we delved into

The Golang Saga: A Coder’s Journey There and Back Again. Part 2: The Data Expedition

Welcome back to the second part of “The Golang Saga: A Coder’s Journey There and Back Again.” In the first part of this series, I started working on a personal project, the Climate Change Visualizer, using Golang as my chosen programming language. Just a quick recap, I had already