r/dataengineering • • 3h ago

Rant I love corporate America

109 Upvotes

Reviewing PRs from our new corporate office in India makes me want to baptize myself in freshly brewed coffee. That’s all


r/dataengineering • • 14h ago

Help Pitchbook Share to Internal schema pipeline

5 Upvotes

Hello fellow data engineers, I need to build a new dashboard and backend data pipelines to show the data of the company that user search for…

So our company has subscribed to pitchbook snowflake share which provides different types of data and we need to fetch it from there and dump it into internal snowflake schemas.. and the frontend dashboard where user can search for the company and get information…

The data is most likely about company profile, investors, funding, news….

The backend pipelines we need to build using airflow jobs…


r/dataengineering • • 2h ago

Help Which of these two courses should I choose to transition from SQL only DEV to a DE?

4 Upvotes

Idk any Python, have never worked with Python. But now for DE, I want to learn so which one should I do?

  1. Data Engineering for Beginners: Learn SQL, Python & Spark by Durga Viswanatha Gadiraju / ITVersity
  2. The Complete Python Bootcamp From Zero to Hero in Python by Jose Portilla, Pierian

r/dataengineering • • 9h ago

Discussion Inherited a PB scale data mess. Looking for tech & people strategies.

25 Upvotes

Hello good folks of /r/dataengineering,

I need opinion &/ guidance on how you would handle such a situation.

Inherited a large data data platform PB level, with a lot of pipelines, tables and tools. Built mostly on OSS and K8s stack.

  1. Company grew very fast (think non-tech startup, public now), BigQuery is the primary warehouse, with a mix of batch and streaming.
  2. Engineers came and went, lot of the platform is not maintained and people do not wish to touch what works. Eg. There are teams that still use Airflow 1.10.
  3. There are a lot of pipelines and data sets are that are no longer in use, they're running on schedule and cost $$.
  4. Schedules are all over the place. I found a pipeline that runs daily, powering a report that only gets sent out every two weeks.

I was planning on mapping everything first then use that data to make culling decisions, but there is internal resistance from business. Folks are not even ready to spend time to add metadata on their own datasets.

What do you folks recommend I do? Any suggestions on both technical and people strategies would be greatly appreciated.

Please don't plug your tool here.


r/dataengineering • • 22h ago

Help Coming from ADO – dbt Cloud + GitHub CI triggering is a pain. How do you handle this?

9 Upvotes

We're migrating from Azure DevOps to GitHub and honestly, the dbt Cloud integration feels like pain. In ADO it just worked out of the box.

With GitHub, connecting via Deploy/Access Key gives us zero webhooks, so no automatic PR triggering. Meanwhile, the custom GitHub App option in dbt is still in Preview, which is a tough sell to our IT team.

How are you guys handling PR CI in stricter orgs? Triggering dbt Cloud jobs via GitHub Actions ?

Anyone using the beta custom Github App? How is it ?