Skip to content

Instantly share code, notes, and snippets.

@didibus

didibus/control_pmap_concurrency_level_with_rechunking.clj

Last active January 5, 2021 06:32
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save didibus/708990cbf47e091fd881d9883496ba66 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save didibus/708990cbf47e091fd881d9883496ba66 to your computer and use it in GitHub Desktop.
Download ZIP
Example to control Clojure pmap concurrency level using the chunk size of lazy-seqs.
Raw
control_pmap_concurrency_level_with_rechunking.clj
;; From: https://clojuredocs.org/clojure.core/chunk#example-5c9cebc3e4b0ca44402ef6ec
(defn re-chunk [n xs]
(lazy-seq
(when-let [s (seq (take n xs))]
(let [cb (chunk-buffer n)]
(doseq [x s] (chunk-append cb x))
(chunk-cons (chunk cb) (re-chunk n (drop n xs)))))))
;; Simply wrap the collection or seq/lazy-seq with re-chunk and the configured chunk size will make pmap's concurrency level grow to the size of the chunk.
(time (dorun (pmap (fn[e] (Thread/sleep 100) (inc e)) (re-chunk 100 (range 1000)))))
"Elapsed time: 1038.57 msecs"
@didibus
Copy link
Author

didibus commented Jan 5, 2021 •
edited
Loading

In Clojure, pmap has a concurrency level (parallelism level) equal to numberOfCpuCores + 2 or the size of the chunk of the lazy-seq if it is passed a lazy-seq as collection. That means you can take advantage of the fact its concurrency level is the size of lazy-seq by passing it a lazy-seq of a custom chunk size. Lucky for us, there is a convenient function to re-chunk a lazy-seq and thus let us control pmap's parallelization level.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment