Sign in Subscribe

Olga Braginskaya

Unleash Your Pipeline Creativity: Local Development with Argo Workflows and MinIO on Minikube

As a data engineer, you know the joy of wrangling massive datasets and navigating complex data pipelines. Argo Workflows, a popular workflow engine for Kubernetes, is your trusty companion in this data-driven journey, allowing you to define, run, and manage data pipelines as code. However, like any adventure, there are

From Kafka to Amazon S3: Partitioning Outputs

So you’ve got a ton of data that needs to be processed in real-time, huh? Don’t worry, in this tutorial I’ll show you how to stream data from Kafka compatible streaming platform to Amazon S3. But wait, there’s more! I’ll also cover how to create

How we mastered dbt: A true story

Staying sane in the modern world of data engineering is a non-trivial mission. As data engineers we ask ourselves the same question several times a day: is everything okay with my data? If you know what I’m talking about, you also know that, let’s be honest, our primary

PubSub to BigQuery: How to Build a Data Pipeline Using Dataflow, Apache Beam, and Java

I’ve recently worked on a project that required me to collect data from Google PubSub and load it into different BigQuery tables. I’ve faced many challenges during this process so I would like to share my experience building a complete data pipeline in Google Cloud Platform. Problem statement