Bug#920659: caffe-cuda: Fails to find the right cuda driver: "CUDA driver version is insufficient for CUDA runtime version"
Hans Ekbrand
hans.ekbrand at gmail.com
Mon Jan 28 01:00:51 GMT 2019
Package: caffe-cuda
Version: 1.0.0+git20180821.99bd997-2
Severity: normal
Dear Maintainer,
* What led up to the situation?
$ caffe time -model examples/mnist/lenet.prototxt -gpu 0
I0128 01:30:26.382874 7238 caffe.cpp:339] Use GPU with device ID 0
F0128 01:30:26.384541 7238 common.cpp:152] Check failed: error == cudaSuccess (35 vs. 0) CUDA driver version is insufficient for CUDA runtime version
*** Check failure stack trace: ***
@ 0x7ff771464cad google::LogMessage::Fail()
@ 0x7ff771466b13 google::LogMessage::SendToLog()
@ 0x7ff771464808 google::LogMessage::Flush()
@ 0x7ff771467579 google::LogMessageFatal::~LogMessageFatal()
@ 0x7ff7717d17f2 caffe::Caffe::SetDevice()
@ 0x55c3a4d7bf03 time()
@ 0x55c3a4d76a71 main
@ 0x7ff770b8409b __libc_start_main
@ 0x55c3a4d775ea _start
Aborted
Not quite sure which package contains the CUDA driver, but here is a list of suspects:
and yes I have cuda9.0 (nvidia-packages) installed side by side with cuda9.1 from Debian, libcudart9.2 was installed as a dependency by libcaffe-cuda1. Perhaps libcudart9.2 requires a newer kernel driver than 390.87?
$ LC_ALL=C dpkg -l "*cuda*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==========================================-===========================-============-=======================================================
ii caffe-cuda 1.0.0+git20180821.99bd997-2 amd64 Fast, open framework for Deep Learning (Meta)
ii caffe-tools-cuda 1.0.0+git20180821.99bd997-2 amd64 Tools for fast, open framework for Deep Learning (CUDA)
rc cuda-command-line-tools-9-0 9.0.176-1 amd64 CUDA command-line tools
ii cuda-core-9-0 9.0.176-1 amd64 CUDA core tools
ii cuda-cublas-9-0 9.0.176.4-1 amd64 CUBLAS native runtime libraries
ii cuda-cublas-dev-9-0 9.0.176.4-1 amd64 CUBLAS native dev links, headers
ii cuda-cudart-9-0 9.0.176-1 amd64 CUDA Runtime native Libraries
ii cuda-cudart-dev-9-0 9.0.176-1 amd64 CUDA Runtime native dev links, headers
ii cuda-cufft-9-0 9.0.176-1 amd64 CUFFT native runtime libraries
ii cuda-cufft-dev-9-0 9.0.176-1 amd64 CUFFT native dev links, headers
ii cuda-curand-9-0 9.0.176-1 amd64 CURAND native runtime libraries
ii cuda-curand-dev-9-0 9.0.176-1 amd64 CURAND native dev links, headers
ii cuda-cusolver-9-0 9.0.176-1 amd64 CUDA solver native runtime libraries
ii cuda-cusolver-dev-9-0 9.0.176-1 amd64 CUDA solver native dev links, headers
ii cuda-cusparse-9-0 9.0.176-1 amd64 CUSPARSE native runtime libraries
ii cuda-cusparse-dev-9-0 9.0.176-1 amd64 CUSPARSE native dev links, headers
ii cuda-driver-dev-9-0 9.0.176-1 amd64 CUDA Driver native dev stub library
ii cuda-libraries-9-0 9.0.176-1 amd64 CUDA Libraries 9.0 meta-package
ii cuda-libraries-dev-9-0 9.0.176-1 amd64 CUDA Libraries 9.0 development meta-package
un cuda-license-7-5 <none> <none> (no description available)
ii cuda-license-9-0 9.0.176-1 amd64 CUDA licenses
ii cuda-misc-headers-9-0 9.0.176-1 amd64 CUDA miscellaneous headers
ii cuda-npp-9-0 9.0.176-1 amd64 NPP native runtime libraries
ii cuda-npp-dev-9-0 9.0.176-1 amd64 NPP native dev links, headers
ii cuda-nvgraph-9-0 9.0.176-1 amd64 NVGRAPH native runtime libraries
ii cuda-nvgraph-dev-9-0 9.0.176-1 amd64 NVGRAPH native dev links, headers
ii cuda-nvrtc-9-0 9.0.176-1 amd64 NVRTC native runtime libraries
ii cuda-nvrtc-dev-9-0 9.0.176-1 amd64 NVRTC native dev links, headers
un cuda-repo-ubuntu1204 <none> <none> (no description available)
un cuda-repo-ubuntu1404 <none> <none> (no description available)
un cuda-repo-ubuntu1404-7-5-local <none> <none> (no description available)
un cuda-repo-ubuntu1410 <none> <none> (no description available)
un cuda-repo-ubuntu1504 <none> <none> (no description available)
un cuda-repo-ubuntu1504-7-5-local <none> <none> (no description available)
ii cuda-repo-ubuntu1704-9-0-176-local-patch-4 1.0-1 amd64 cuda repository configuration files
ii cuda-repo-ubuntu1704-9-0-local 9.0.176-1 amd64 cuda repository configuration files
un libcaffe-cuda-dev <none> <none> (no description available)
ii libcaffe-cuda1:amd64 1.0.0+git20180821.99bd997-2 amd64 library of Caffe, deep leanring framework (CUDA)
un libcuda-5.0-1 <none> <none> (no description available)
un libcuda-5.5-1 <none> <none> (no description available)
un libcuda-6.0-1 <none> <none> (no description available)
un libcuda-6.5-1 <none> <none> (no description available)
un libcuda-7.0-1 <none> <none> (no description available)
un libcuda-7.5-1 <none> <none> (no description available)
un libcuda-8.0-1 <none> <none> (no description available)
un libcuda-9.0-1 <none> <none> (no description available)
un libcuda-9.1-1 <none> <none> (no description available)
un libcuda-9.2-1 <none> <none> (no description available)
ii libcuda1:amd64 390.87-6 amd64 NVIDIA CUDA Driver Library
in libcuda1-384 <none> amd64 (no description available)
un libcuda1-any <none> <none> (no description available)
un libcuda1-i386 <none> <none> (no description available)
ii libcudart9.1:amd64 9.1.85-8 amd64 NVIDIA CUDA Runtime Library
ii libcudart9.2:amd64 9.2.148-5 amd64 NVIDIA CUDA Runtime Library
ii nvidia-cuda-dev 9.1.85-8 amd64 NVIDIA CUDA development files
ii nvidia-cuda-doc 9.1.85-8 all NVIDIA CUDA and OpenCL documentation
ii nvidia-cuda-gdb 9.1.85-8 amd64 NVIDIA CUDA Debugger (GDB)
un nvidia-cuda-mps <none> <none> (no description available)
ii nvidia-cuda-toolkit 9.1.85-8 amd64 NVIDIA CUDA development toolkit
ii python3-caffe-cuda 1.0.0+git20180821.99bd997-2 amd64 Python3 interface of Caffe (CUDA)
un python3-pycuda <none> <none> (no description available)
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-================================-===============-============-=================================================================
un bumblebee-nvidia <none> <none> (no description available)
ii glx-alternative-nvidia 0.9.0 amd64 allows the selection of NVIDIA as GLX provider
un libegl-nvidia-legacy-390xx0 <none> <none> (no description available)
ii libegl-nvidia0:amd64 390.87-6 amd64 NVIDIA binary EGL library
un libegl1-glvnd-nvidia <none> <none> (no description available)
un libegl1-nvidia <none> <none> (no description available)
un libgl1-glvnd-nvidia-glx <none> <none> (no description available)
ii libgl1-nvidia-glvnd-glx:amd64 390.87-6 amd64 NVIDIA binary OpenGL/GLX library (GLVND variant)
un libgl1-nvidia-glx <none> <none> (no description available)
un libgl1-nvidia-glx-390.87 <none> <none> (no description available)
un libgl1-nvidia-glx-any <none> <none> (no description available)
un libgl1-nvidia-legacy-390xx-glx <none> <none> (no description available)
un libgldispatch0-nvidia <none> <none> (no description available)
ii libgles-nvidia1:amd64 390.87-6 amd64 NVIDIA binary OpenGL|ES 1.x library
ii libgles-nvidia2:amd64 390.87-6 amd64 NVIDIA binary OpenGL|ES 2.x library
un libgles1-glvnd-nvidia <none> <none> (no description available)
un libgles2-glvnd-nvidia <none> <none> (no description available)
un libglvnd0-nvidia <none> <none> (no description available)
ii libglx-nvidia0:amd64 390.87-6 amd64 NVIDIA binary GLX library
un libglx0-glvnd-nvidia <none> <none> (no description available)
ii libnvidia-cfg1:amd64 390.87-6 amd64 NVIDIA binary OpenGL/GLX configuration library
un libnvidia-cfg1-any <none> <none> (no description available)
ii libnvidia-compiler:amd64 390.87-6 amd64 NVIDIA runtime compiler library
un libnvidia-compiler-390.87 <none> <none> (no description available)
ii libnvidia-eglcore:amd64 390.87-6 amd64 NVIDIA binary EGL core libraries
un libnvidia-eglcore-390.87 <none> <none> (no description available)
un libnvidia-encode1 <none> <none> (no description available)
ii libnvidia-fatbinaryloader:amd64 390.87-6 amd64 NVIDIA FAT binary loader
un libnvidia-fatbinaryloader-390.87 <none> <none> (no description available)
un libnvidia-fbc1 <none> <none> (no description available)
ii libnvidia-glcore:amd64 390.87-6 amd64 NVIDIA binary OpenGL/GLX core libraries
un libnvidia-glcore-390.87 <none> <none> (no description available)
un libnvidia-ifr1 <none> <none> (no description available)
ii libnvidia-ml1:amd64 390.87-6 amd64 NVIDIA Management Library (NVML) runtime library
ii libnvidia-ptxjitcompiler1:amd64 390.87-6 amd64 NVIDIA PTX JIT Compiler
un libopengl0-glvnd-nvidia <none> <none> (no description available)
in nvidia-384 <none> amd64 (no description available)
ii nvidia-alternative 390.87-6 amd64 allows the selection of NVIDIA as GLX provider
un nvidia-alternative--kmod-alias <none> <none> (no description available)
un nvidia-alternative-legacy-173xx <none> <none> (no description available)
un nvidia-alternative-legacy-71xx <none> <none> (no description available)
un nvidia-alternative-legacy-96xx <none> <none> (no description available)
ii nvidia-cuda-dev 9.1.85-8 amd64 NVIDIA CUDA development files
ii nvidia-cuda-doc 9.1.85-8 all NVIDIA CUDA and OpenCL documentation
ii nvidia-cuda-gdb 9.1.85-8 amd64 NVIDIA CUDA Debugger (GDB)
un nvidia-cuda-mps <none> <none> (no description available)
ii nvidia-cuda-toolkit 9.1.85-8 amd64 NVIDIA CUDA development toolkit
un nvidia-current <none> <none> (no description available)
un nvidia-current-updates <none> <none> (no description available)
ii nvidia-driver 390.87-6 amd64 NVIDIA metapackage
ii nvidia-driver-bin 390.87-6 amd64 NVIDIA driver support binaries
un nvidia-driver-bin-390.87 <none> <none> (no description available)
un nvidia-driver-binary <none> <none> (no description available)
ii nvidia-driver-libs:amd64 390.87-6 amd64 NVIDIA metapackage (OpenGL/GLX/EGL/GLES libraries)
un nvidia-driver-libs-i386 <none> <none> (no description available)
un nvidia-driver-libs-nonglvnd <none> <none> (no description available)
ii nvidia-egl-common 390.87-6 amd64 NVIDIA binary EGL driver - common files
ii nvidia-egl-icd:amd64 390.87-6 amd64 NVIDIA EGL installable client driver (ICD)
un nvidia-glx <none> <none> (no description available)
un nvidia-glx-any <none> <none> (no description available)
ri nvidia-installer-cleanup 20151021+8 amd64 cleanup after driver installation with the nvidia-installer
un nvidia-kernel-390.87 <none> <none> (no description available)
ii nvidia-kernel-common 20151021+8 amd64 NVIDIA binary kernel module support files
ii nvidia-kernel-dkms 390.87-6 amd64 NVIDIA binary kernel module DKMS source
un nvidia-kernel-source <none> <none> (no description available)
ii nvidia-kernel-support 390.87-6 amd64 NVIDIA binary kernel module support files
un nvidia-kernel-support--v1 <none> <none> (no description available)
un nvidia-kernel-support-any <none> <none> (no description available)
un nvidia-legacy-304xx-alternative <none> <none> (no description available)
un nvidia-legacy-304xx-vdpau-driver <none> <none> (no description available)
un nvidia-legacy-340xx-alternative <none> <none> (no description available)
un nvidia-legacy-340xx-vdpau-driver <none> <none> (no description available)
un nvidia-legacy-390xx-opencl-icd <none> <none> (no description available)
un nvidia-legacy-390xx-vulkan-icd <none> <none> (no description available)
ii nvidia-legacy-check 390.87-6 amd64 check for NVIDIA GPUs requiring a legacy driver
un nvidia-libopencl1 <none> <none> (no description available)
un nvidia-libopencl1-384 <none> <none> (no description available)
un nvidia-libopencl1-dev <none> <none> (no description available)
ii nvidia-modprobe 390.25-1 amd64 utility to load NVIDIA kernel modules and create device nodes
un nvidia-nonglvnd-vulkan-common <none> <none> (no description available)
un nvidia-nonglvnd-vulkan-icd <none> <none> (no description available)
rc nvidia-opencl-common 390.87-5 amd64 NVIDIA OpenCL driver - common files
ii nvidia-opencl-dev:amd64 9.1.85-8 amd64 NVIDIA OpenCL development files
un nvidia-opencl-icd <none> <none> (no description available)
rc nvidia-opencl-icd-384 384.81-0ubuntu1 amd64 NVIDIA OpenCL ICD
ii nvidia-openjdk-8-jre 9.1.85-8 amd64 NVIDIA provided OpenJDK Java runtime, using Hotspot JIT
ii nvidia-persistenced 390.25-1 amd64 daemon to maintain persistent software state in the NVIDIA driver
ii nvidia-profiler 9.1.85-8 amd64 NVIDIA Profiler for CUDA and OpenCL
ri nvidia-settings 390.67-1 amd64 tool for configuring the NVIDIA graphics driver
un nvidia-settings-gtk-390.67 <none> <none> (no description available)
ii nvidia-smi 390.87-6 amd64 NVIDIA System Management Interface
ii nvidia-support 20151021+8 amd64 NVIDIA binary graphics driver support files
ii nvidia-vdpau-driver:amd64 390.87-6 amd64 Video Decode and Presentation API for Unix - NVIDIA driver
ii nvidia-visual-profiler 9.1.85-8 amd64 NVIDIA Visual Profiler for CUDA and OpenCL
ii nvidia-vulkan-common 390.87-6 amd64 NVIDIA Vulkan driver - common files
ii nvidia-vulkan-icd:amd64 390.87-6 amd64 NVIDIA Vulkan installable client driver (ICD)
ii xserver-xorg-video-nvidia 390.87-6 amd64 NVIDIA binary Xorg driver
un xserver-xorg-video-nvidia-any <none> <none> (no description available)
-- System Information:
Debian Release: buster/sid
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)
Kernel: Linux 4.19.0-1-amd64 (SMP w/6 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to sv_SE.utf8), LANGUAGE=en_US:en (charmap=UTF-8) (ignored: LC_ALL set to sv_SE.utf8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages caffe-cuda depends on:
ii caffe-tools-cuda 1.0.0+git20180821.99bd997-2
ii libcaffe-cuda1 1.0.0+git20180821.99bd997-2
ii python3-caffe-cuda 1.0.0+git20180821.99bd997-2
ii python3-protobuf 3.6.1.3-1
caffe-cuda recommends no packages.
Versions of packages caffe-cuda suggests:
ii caffe-doc 1.0.0+git20180821.99bd997-2
pn libcaffe-cuda-dev <none>
-- no debconf information
More information about the debian-science-maintainers
mailing list