Well, sometimes it is not the best thing to do to have transaction that last that long (it prevents vacuuming for example). So sometimes I have used “tuple iterator” where you keep tuple of components that provide strict and monotonic ordering of elements and then you can process entries in batches by fetching all elements that are next in order. Additionally it is important to remember that it will block one connection from pool, so sometimes it can be better to start new connection just for sake of that single action to not degrade main repository pool.
Well, sometimes it is not the best thing to do to have transaction that last that long (it prevents vacuuming for example). So sometimes I have used “tuple iterator” where you keep tuple of components that provide strict and monotonic ordering of elements and then you can process entries in batches by fetching all elements that are next in order. Additionally it is important to remember that it will block one connection from pool, so sometimes it can be better to start new connection just for sake of that single action to not degrade main repository pool.
Nice examples of Stream/GenStage/Flow are in https://www.slideshare.net/quatermain1/elixir-after-2-years-in-action-code-webup (without Ecto)