When neural network frameworks are built, they are dynamically linked to CUDA and cuDNN libraries. These are so or shared object files that are loaded at runtime. The LD_LIBRARY_PATH environment variable tells Ubuntu where to look for these files.
On Windows, these are dll or dynamic link library files.
GPUs were originally designed for graphics. When you're running a neural network, you're not using the GPU for graphics. CUDA (Compute Unified Device Architecture) is a general-purpose computing on GPUs (GPGPU) platform that allows C-code to run on the GPU. cuDNN (CUDA Deep Neural Network) is a library of primitives like matrix multiplication and convolution that are optimized for GPUs.
To ensure everything works, you want your system to provide the versions of CUDA and cuDNN that your software expects.