r/dataengineering • • 3h ago

Rant I love corporate America

109 Upvotes

Reviewing PRs from our new corporate office in India makes me want to baptize myself in freshly brewed coffee. That’s all


r/dataengineering • • 21h ago

Discussion Airbyte has identified a security incident that they believe is part of a broader, global cyberattack.

Thumbnail
linkedin.com
75 Upvotes

Oof


r/dataengineering • • 9h ago

Discussion Inherited a PB scale data mess. Looking for tech & people strategies.

25 Upvotes

Hello good folks of /r/dataengineering,

I need opinion &/ guidance on how you would handle such a situation.

Inherited a large data data platform PB level, with a lot of pipelines, tables and tools. Built mostly on OSS and K8s stack.

  1. Company grew very fast (think non-tech startup, public now), BigQuery is the primary warehouse, with a mix of batch and streaming.
  2. Engineers came and went, lot of the platform is not maintained and people do not wish to touch what works. Eg. There are teams that still use Airflow 1.10.
  3. There are a lot of pipelines and data sets are that are no longer in use, they're running on schedule and cost $$.
  4. Schedules are all over the place. I found a pipeline that runs daily, powering a report that only gets sent out every two weeks.

I was planning on mapping everything first then use that data to make culling decisions, but there is internal resistance from business. Folks are not even ready to spend time to add metadata on their own datasets.

What do you folks recommend I do? Any suggestions on both technical and people strategies would be greatly appreciated.

Please don't plug your tool here.


r/dataengineering • • 22h ago

Help Coming from ADO – dbt Cloud + GitHub CI triggering is a pain. How do you handle this?

9 Upvotes

We're migrating from Azure DevOps to GitHub and honestly, the dbt Cloud integration feels like pain. In ADO it just worked out of the box.

With GitHub, connecting via Deploy/Access Key gives us zero webhooks, so no automatic PR triggering. Meanwhile, the custom GitHub App option in dbt is still in Preview, which is a tough sell to our IT team.

How are you guys handling PR CI in stricter orgs? Triggering dbt Cloud jobs via GitHub Actions ?

Anyone using the beta custom Github App? How is it ?


r/dataengineering • • 14h ago

Help Pitchbook Share to Internal schema pipeline

5 Upvotes

Hello fellow data engineers, I need to build a new dashboard and backend data pipelines to show the data of the company that user search for…

So our company has subscribed to pitchbook snowflake share which provides different types of data and we need to fetch it from there and dump it into internal snowflake schemas.. and the frontend dashboard where user can search for the company and get information…

The data is most likely about company profile, investors, funding, news….

The backend pipelines we need to build using airflow jobs…