I’ve never personally used this function but I’ve seen it get recommended before and there are benchmarks like this one that show that pooledMapConcurrently is the fastest.
Namely, @marcosh you’d use something like mapTasks_, which has almost exactly the signature you’re looking for (you just need to create a task group first):
main = withTaskGroup 5 $ \tg -> do
mapTasks_ tg [io1, io2, io3, io4]
If the tasks don’t return anything and they are bare IO, you could build something simple around QSem from Control.Concurrent.QSem.
EDIT:
So this appears to work:
import Control.Monad (forM_)
import Control.Concurrent (forkIO)
import Control.Concurrent.QSem qualified as QSem
runJobs :: Int -> [IO ()] -> IO ()
runJobs max jobs = do
sem <- QSem.newQSem max
forM_ jobs $ \job -> forkIO $ do
QSem.waitQSem sem
job
QSem.signalQSem sem
pooledMapConcurrentlyN only works in simple situations, so QSem is definitely something that should be in a Haskeller’s engineering toolkit.
For instance, I wanted my tool to respect a -j param when generating an import graph pruned to a specific module. The algorithm is
Parse imports from the file we are pruning too.
Recursively parse all its imports.
Repeat until we hit the leaves.
I want a stable -j across that concurrency. Using pooledMapConcurrentlyN naively to do step 2 would violate -j since each recursive call would also call it. So a QSem is a better fit. (Code here)
And I wouldn’t really consider it reinventing the wheel. Programming with semaphores is a pretty fundamental engineering skill that every programmer who does concurrency should be able to do
Yup. For any production/work use, you’d probably look into what one can find in libraries you already use and depend on. I.e use async instead of forkIO, maybe even a resource manager that expands over what bracket does, and so on.
Though, if one is interested in reinventing the wheel, then as QSem is implemented in terms of MVars and exception masking, then one might look into using MVars and exceptions directly.