r/DistributedComputing • u/Paige_Ward44 • 4h ago
is optimizing distributed batch jobs worth it vs just adding more nodes?
I've been thinking about something lately: when a distributed batch job starts slowing down, is adding more workers really the best way to deal with it, or would it make more sense to improve how the job runs I'm mainly looking at operations like sorting, joins and shuffles, since they can end up taking a fair amount of processing time.
The idea would be to figure out which parts of the workload are slowing things down, make those operations more efficient and see how that compares to adding more nodes. I haven't tried this on any large workloads yet, so most of what I've read comes from documentation and benchmarks. trino, clickhouse and dualbird are a few names I've come across while researching different options.
I'm interested in hearing from people who've worked with distributed systems. Is optimizing execution usually worth the effort, or does scaling out tend to be more practical?
Would appreciate any thoughts on data movement, overhead and whether the performance gains justify the added complexity.