Of Stack

caffe environment construction method based on docker

  • 2020-11-20 06:21:27
  • OfStack

Why should I use docker? I have been familiar with docker for a long time. I have never had the opportunity to use it since I thought it was a sharp weapon during my internship.

Requirement: Build a stand-alone container with all caffe dependencies installed and run it when you need to run code.

Advantages: Various dependencies can be resolved, such as this software needs to install gcc 4.7, another need to install gcc 4.8 and other mutually exclusive environment requirements.

Docker installation

For the installation and basic use of docker, please refer to my two blogs above: Installation and Use.

Build the mirror

There are two ways to build an image:

1, Dockerfile, the advantage is easy to share;

2, commit from the container, the advantage is simple and convenient, but not easy to share.

Since installing the caffe environment requires a large number of dependent packages, which are often not installed once due to network reasons, it is convenient to simply start a basic container and install the dependent packages from within.

Here, select basic ubuntu:14.04 to start building the environment,

1. Start the container: 

sudo docker run --dns 8.8.8.8 --dns 8.8.4.4 --name ubuntu_caffe -i -t ubuntu:14.04 /bin/bash

You need to change dns because without it, the container won't be able to connect to the web.

2, dependent package installation

Once inside the container, all operations are as normal as before.

First install 1 of the basic tools: 


apt-get install wget
apt-get install unzip
apt-get install python-pip

Create a new path and put all the caffe code in there 


mkdir /home/crw/Caffe
cd /home/crw/Caffe

Installation of opencv:

github has 1 key installation on it, but it cannot be installed in the container with 1 key. Later, I will execute the script sentence by sentence. 


#opencv1 other 1 The operation of the other 
arch=$(uname -m)
if [ "$arch" == "i686" -o "$arch" == "i386" -o "$arch" == "i486" -o "$arch" == "i586" ]; then flag=1; else flag=0; fi
echo "Installing OpenCV 2.4.9"
mkdir OpenCV
cd OpenCV
sudo apt-get -y install libopencv-dev
sudo apt-get -y install build-essential checkinstall cmake pkg-config yasm
sudo apt-get -y install libtiff4-dev libjpeg-dev libjasper-dev
sudo apt-get -y install libavcodec-dev libavformat-dev libswscale-dev libdc1394-22-dev libxine-dev libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev libv4l-dev
sudo apt-get -y install python-dev python-numpy
sudo apt-get -y install libtbb-dev
sudo apt-get -y install libqt4-dev libgtk2.0-dev
sudo apt-get -y install libfaac-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev
wget http://archive.ubuntu.com/ubuntu/pool/multiverse/f/faac/faac_1.28-6.debian.tar.gz
vi /etc/hosts
ifconfig
sudo apt-get -y install x264 v4l-utils ffmpeg
wget -O OpenCV-2.4.9.zip http://sourceforge.net/projects/opencvlibrary/files/opencv-unix/2.4.9/opencv-2.4.9.zip/download
unzip OpenCV-2.4.9.zip
cd opencv-2.4.9
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_TBB=ON -D BUILD_NEW_PYTHON_SUPPORT=ON -D WITH_V4L=ON -D INSTALL_C_EXAMPLES=ON -D INSTALL_PYTHON_EXAMPLES=ON -D BUILD_EXAMPLES=ON -D WITH_QT=ON -D WITH_OPENGL=ON ..
make -j4
sudo make install
sudo sh -c 'echo "/usr/local/lib" > /etc/ld.so.conf.d/opencv.conf'
sudo ldconfig
cd ..

caffe and python dependency packages: 


sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler
sudo apt-get install --no-install-recommends libboost-all-dev
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
sudo apt-get install libatlas-base-dev

Install cuda:

The trick here is to import data from the host host into the container using the following command (run on host) 


sudo cp cuda_7.5.18_linux.run /var/lib/docker/aufs/mnt/92562f161e51994949dd8496360265e5d54d32fbe301d693300916cd56d4e0a2/home/crw/Caffe

sudo cp cudnn-7.0-linux-x64-v3.0-prod.tgz /var/lib/docker/aufs/mnt/92562f161e51994949dd8496360265e5d54d32fbe301d693300916cd56d4e0a2/home/crw/Caffe

sudo cp caffe-master.zip /var/lib/docker/aufs/mnt/92562f161e51994949dd8496360265e5d54d32fbe301d693300916cd56d4e0a2/home/crw/Caffe

Where the 1 long string is the complete id of your container, you can use the command 

docker inspect -f   '{{.Id}}' ubuntu_caffe #ubuntu_caffe  It's the name of the container 


./cuda_*_linux.run -extract=`pwd`
./NVIDIA-Linux-x86_64-*.run -s --no-kernel-module
./cuda-linux64-rel-*.run -noprompt

Install cudnn: 


tar -xvf cudnn-7.0-linux-x64-v3.0-prod.tgz 
cp cuda/include/cudnn.h /usr/local/cuda/include/
cp cuda/lib64/* /usr/local/cuda/lib64/

Install caffe: 


cd caffe-15.12.07/
cp Makefile.config.example Makefile.config
vi Makefile.config
make all
make test

Install the python binding for caffe 


cd python/

apt-get install python-pip
for req in $(cat requirements.txt); do pip install $req; done

sudo apt-get install python-numpy python-scipy python-matplotlib ipython ipython-notebook python-pandas python-sympy python-nose

The following packages need to be installed manually by themselves. It is not good to use command 1 above to install them directly. 


apt-get install python-scipy
for req in $(cat requirements.txt); do pip install $req; done
pip install -U scikit-learn
for req in $(cat requirements.txt); do pip install $req; done
pip install scikit-image
for req in $(cat requirements.txt); do pip install $req; done

then 


mkdir /home/crw/Caffe
cd /home/crw/Caffe
0

Finally, create a new path to facilitate disk mapping 


mkdir /home/crw/Caffe
cd /home/crw/Caffe
1

With all the environments in place, you are ready to commit 


mkdir /home/crw/Caffe
cd /home/crw/Caffe
2

So, in the local environment, you have a container that turns the caffe gpu environment into one.

Start caffe Start container: 


sudo docker run -ti \
  --device /dev/nvidia0:/dev/nvidia0 \
  --device /dev/nvidiactl:/dev/nvidiactl \
  --device /dev/nvidia-uvm:/dev/nvidia-uvm \
  -v /media/crw/MyBook:/media/crw/MyBook \
  my-ubuntu-caffe /bin/bash

1. Graphics card through use,
2, file mapping, mount 1 host host disk to container path, here set to the same, can reduce 1 unnecessary trouble.

Run caffe model training

There will be a prompt saying that cuda can't be found, just set the environment variable under 1. 


mkdir /home/crw/Caffe
cd /home/crw/Caffe
4 

mkdir /home/crw/Caffe
cd /home/crw/Caffe
5
Related articles: