Skip to content

Instantly share code, notes, and snippets.

@cnuernber

cnuernber/numpy-byte-array-zc.clj

Created April 11, 2021 22:39
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save cnuernber/b986ca2ff8dd4defe33b1cbd2f854e4d to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save cnuernber/b986ca2ff8dd4defe33b1cbd2f854e4d to your computer and use it in GitHub Desktop.
Download ZIP
Create a Java ByteBuffer from a numpy array via zerocopy pathways
Raw
numpy-byte-array-zc.clj
```clojure
user> (require '[libpython-clj2.python :as py])
nil
user> ;;Enable zero copy access to numpy objects
user> (require '[libpython-clj2.python.np-array])
nil
user> (require '[tech.v3.datatype :as dtype]
               '[tech.v3.datatype.native-buffer :as native-buffer]
               '[tech.v3.datatype.nio-buffer :as nio-buffer])
nil
user> (py/initialize!)
Apr 11, 2021 4:30:22 PM clojure.tools.logging$eval6672$fn__6675 invoke
INFO: Detecting startup info
Apr 11, 2021 4:30:22 PM clojure.tools.logging$eval6672$fn__6675 invoke
INFO: Startup info {:lib-version "3.8", :java-library-path-addendum "/usr/lib", :exec-prefix "/usr", :executable "/usr/bin/python3", :libnames ("python3.8m" "python3.8"), :prefix "/usr", :base-prefix "/usr", :libname "python3.8m", :base-exec-prefix "/usr", :python-home "/usr", :version [3 8 5], :platform "linux"}
Apr 11, 2021 4:30:22 PM clojure.tools.logging$eval6672$fn__6675 invoke
INFO: Prefixing java library path: /usr/lib
Apr 11, 2021 4:30:22 PM clojure.tools.logging$eval6672$fn__6675 invoke
INFO: Loading python library: python3.8
Apr 11, 2021 4:30:22 PM clojure.tools.logging$eval6672$fn__6675 invoke
INFO: Reference thread starting
:ok
user> (def np (py/import-module "numpy"))
       
#'user/np
user> (def ary (py/call-attr np "array" (range 100)))
#'user/ary
user> ;;Zerocopy implies at least a translation to a native buffer
user> (dtype/as-native-buffer ary)
#native-buffer@0x00007F83C10902B0<int64>[100]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19...]
user> ;;Native buffers can change their base datatype
user> (native-buffer/set-native-datatype *1 :uint8)
#native-buffer@0x00007F83C10902B0<uint8>[800]
[0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0...]
user> ;;Native buffers also have a zero-copy pathway to nio buffers
user> ;;We disable the moderately expensive resource management as long
user> ;;as we know the native buffer will be reachable as long as or
user> ;;longer than the nio buffer.
user> (nio-buffer/native-buf->nio-buf *1 {:resource-type nil})
#object[java.nio.DirectByteBuffer 0x15d5f2b5 "java.nio.DirectByteBuffer[pos=0 lim=800 cap=800]"]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment