r/dataengineering • u/IronAntlers • 3h ago
Rant I love corporate America
Reviewing PRs from our new corporate office in India makes me want to baptize myself in freshly brewed coffee. That’s all
r/dataengineering • u/IronAntlers • 3h ago
Reviewing PRs from our new corporate office in India makes me want to baptize myself in freshly brewed coffee. That’s all
r/dataengineering • u/Existing_Wealth6142 • 21h ago
Oof
r/dataengineering • u/brickkcirb • 9h ago
Hello good folks of /r/dataengineering,
I need opinion &/ guidance on how you would handle such a situation.
Inherited a large data data platform PB level, with a lot of pipelines, tables and tools. Built mostly on OSS and K8s stack.
I was planning on mapping everything first then use that data to make culling decisions, but there is internal resistance from business. Folks are not even ready to spend time to add metadata on their own datasets.
What do you folks recommend I do? Any suggestions on both technical and people strategies would be greatly appreciated.
Please don't plug your tool here.
r/dataengineering • u/Salt_Anteater3307 • 22h ago
We're migrating from Azure DevOps to GitHub and honestly, the dbt Cloud integration feels like pain. In ADO it just worked out of the box.
With GitHub, connecting via Deploy/Access Key gives us zero webhooks, so no automatic PR triggering. Meanwhile, the custom GitHub App option in dbt is still in Preview, which is a tough sell to our IT team.
How are you guys handling PR CI in stricter orgs? Triggering dbt Cloud jobs via GitHub Actions ?
Anyone using the beta custom Github App? How is it ?
r/dataengineering • u/According-Mud-6472 • 14h ago
Hello fellow data engineers, I need to build a new dashboard and backend data pipelines to show the data of the company that user search for…
So our company has subscribed to pitchbook snowflake share which provides different types of data and we need to fetch it from there and dump it into internal snowflake schemas.. and the frontend dashboard where user can search for the company and get information…
The data is most likely about company profile, investors, funding, news….
The backend pipelines we need to build using airflow jobs…