Ask HN: Any pragmatic guides to building large data pipelines?
I am rebuilding a data pipeline that processes billions of records. An overview of what I built is as follows: collect data from (n*k) sources-> derive new data -> generate a unified/merged collection of data (n) data. The current solution is all hand crafted code. I know this is a 10,000 foot view of the problem, but are there any guides or books on how to better design and implement this type of solution? 0 comments on Hacker News.
I am rebuilding a data pipeline that processes billions of records. An overview of what I built is as follows: collect data from (n*k) sources-> derive new data -> generate a unified/merged collection of data (n) data. The current solution is all hand crafted code. I know this is a 10,000 foot view of the problem, but are there any guides or books on how to better design and implement this type of solution?
I am rebuilding a data pipeline that processes billions of records. An overview of what I built is as follows: collect data from (n*k) sources-> derive new data -> generate a unified/merged collection of data (n) data. The current solution is all hand crafted code. I know this is a 10,000 foot view of the problem, but are there any guides or books on how to better design and implement this type of solution? 0 comments on Hacker News.
I am rebuilding a data pipeline that processes billions of records. An overview of what I built is as follows: collect data from (n*k) sources-> derive new data -> generate a unified/merged collection of data (n) data. The current solution is all hand crafted code. I know this is a 10,000 foot view of the problem, but are there any guides or books on how to better design and implement this type of solution?
Hacker News story: Ask HN: Any pragmatic guides to building large data pipelines?
![Hacker News story: Ask HN: Any pragmatic guides to building large data pipelines?]()
Reviewed by Tha Kur
on
November 01, 2021
Rating:
No comments: