Ahmed, thanks for reading.

I think the approach you are proposing is spot on. It can be difficult to achieve depending on the problem you are working on. Also, in the way you are obtaining the data. But it should do the work.

I think it should helpful for you to check out Maxime's post about Functional Data Engineering. Particularly, the section where he is talking about "**Past dependencies**." I think it can shed some light in your approach.

https://medium.com/@maximebeauchemin/functional-data-engineering-a-modern-paradigm-for-batch-data-processing-2327ec32c42a

Writing to learn! | LinkedIn profile: https://www.linkedin.com/in/ajhenaor | Buy me a coffee: https://www.buymeacoffee.com/ajhenaor

Writing to learn! | LinkedIn profile: https://www.linkedin.com/in/ajhenaor | Buy me a coffee: https://www.buymeacoffee.com/ajhenaor