3D Depth Sensor for Raspberry Pi 3

Standard

Depth sensors such as OPT8241 CDK from Texas Instruments enable a wide array of new applications, ranging from people counting to gesture control to robotics and other dimensioning applications.  3D sensing together with a WiFi-enabled embedded system, will enable powerful IoT applications.  This blog post describes how to integrate OPT8241 CDK with Raspberry Pi 3.

Download Ubuntu MATE

Download UbuntuMATE 16.04 LTS image into your Ubuntu Linux box and unzip the *.img.xz file:

     xz -d filename.img.xz

where filename.img.xz is the downloaded image.

Program Micro SD Card

The next step  is program the image into the micro SD card.  Insert the micro SD card into your Ubuntu Linux PC and check the SD card’s assigned device name this way:

     dmesg | tail

Usually you should see something like mmcblk0 or sdc. Remember the device name. Eject or unmount the micro SD card, and then program the micro SD card using this command:

     dd if=filename.img of=/dev/mmblk0 bs=1M

Replace filename and mmblk0 with your own filename and device name.

Check Disk Partitions

The Ubuntu Mate image downloaded may not be configured to use of the full storage capacity of your micro SD card. To ensure the full storage capacity is used, run gparted:

     sudo apt-get install gparted
     sudo gparted

Click on the Linux partition (not the boot partition), and select Check from the menu, and option will be provided to resize the selected partition to the full capacity.

Booting the Image

Once programming is completed, insert the micro SD card into RPi3 and power it up with HDMI and USB keyboard and mouse connected.  Answer a series of questions and enable WiFi connection. WiFi may become disconnected, but no worries, just reboot. When booting and general OS setup is completed, login and create a subdirectory Software at the home directory to keep all the new software we will download:

     cd
     mkdir Software

Setup Swapfile

Create a swap file to supplement the limited 1GB RAM available on RPi3. A swap file is space on the storage device used as virtual memory, useful in compiling massive projects.

First, check that there is no active swap by entering the command:

     free -h

Create a 2 GB swap file by the following commands:

     sudo fallocate -l 2G /swapfile
     ls -lh /swapfile   # Check that the file is indeed 2GB
     sudo chmod 600 /swapfile  # Make the file only accessible to root
     sudo mkswap /swapfile  # Identifies the file as swap space
     sudo swapon /swapfile  # Enables swap file
     sudo swapon --show    # Verifies that the swap file is working

To make the swap permanent, add the below line to /etc/fstab and reboot:

     /swapfile   none   swap   sw   0   0

Setup Prerequisites

To prepare the software development on RPi3 we need setup some prerequisite tools:

     sudo apt-get update
     sudo apt-get install git build-essential linux-libc-dev
     sudo apt-get install cmake cmake-gui swig3.0
     sudo apt-get install libusb-1.0-0-dev libusb-dev libudev-dev
     sudo apt-get install mpi-default-dev openmpi-bin openmpi-common  
     sudo apt-get install libflann1.8 libflann-dev
     sudo apt-get install libeigen3-dev
     sudo apt-get install libboost-all-dev
     sudo apt-get install libvtk5.10-qt4 libvtk5.10 libvtk5-dev
     sudo apt-get install libqhull* libgtest-dev
     sudo apt-get install freeglut3-dev pkg-config
     sudo apt-get install libxmu-dev libxi-dev 
     sudo apt-get install mono-complete
     sudo apt-get install qt-sdk openjdk-8-jdk openjdk-8-jre

Build Point Cloud Library

Point Cloud Library is needed by Voxel SDK, and can be obtained by:

     sudo add-apt-repository ppa:v-launchpad-jochen-sprickerhof-de/pcl
     sudo apt-get update
     git clone https://github.com/PointCloudLibrary/pcl.git

Now you should find pcl in the ~/Software directory.  Create a release directory and follow the cmake build process:

     mkdir release
     cd release
     cmake -DCMAKE_BUILD_TYPE=None -DCMAKE_INSTALL_PREFIX=/usr 
           -DBUILD_GPU=ON -DBUILD_apps=ON -DBUILD_examples=ON 
           -DCMAKE_INSTALL_PREFIX=/usr ..
     make

The make will take over 7 hours. Once the build finishes, install it by:

     sudo make install

Build Voxel SDK

In the same Software directory, clone Voxel SDK:

     cd ~/Software
     git clone https://github.com/3dtof/voxelsdk.git

Go the voxelsdk directory and edit the top level CMakeList.txt with the below change:

     # set(ARCH ${CMAKE_SYSTEM_PROCESSOR})
     set(ARCH armhf)

and

     #add_definitions(-msse2 -pthread -std=c++11 -fPIC -ffast-math)
     add_definitions(-pthread -std=c++11 -fPIC -ffast-math -fpermissive)

Also the CMakeList.txt in the VoxelPCL subdirectory change the “wily” option by removing “1.7” from all the libpcl-*1.7, then change “wily” to “xenial”. (Note: Ubuntu 16.04 codename is “xenial”).

Create a build directory and run cmake-gui:

     mkdir build
     cd build
     cmake-gui ..

Change CMAKE_INSTALL_PREFIXto /usr, and make all Python to use Python 2.7 include and library instead of Python 3.5m. After you made these changes, click Configure and then Generate to create the makefile.

To build Voxel SDK enter:

     make

During build you may get an error about duplicated “standard container” macro definition between one in swig3.0 and one in Voxel SDK–resolve it by commenting out the problematic macro in the Voxel SDK SWIG directory.

After a successful build, install Voxel SDK by:

     sudo make install

This will install need header files to /usr/include/voxel-0.6.5 and libraries to /usr/lib/voxel. Create a symbolic link as follow:

     sudo ln -s /usr/include/voxel-0.6.5 /usr/include/voxel

This is required before building demo applications below.

Build DemoApplications

Now we will test the installed Voxel SDK with demo applications available on GitHub:

     cd ~/Software
     git clone https://github.com/3dtof/DemoApplications.git
     cd DemoApplications/TinTin/people_tracking

Now remove the -msse2 flag from CMakeLists.txt.

     #add_definitions(-msse2 -pthread -std=c++11 -fPIC -ffast-math)
     add_definitions(-pthread -std=c++11 -fPIC -ffast-math -fpermissive)

If your machine does not have OpenCV installed, install it this way:

     sudo apt-get install libopencv-dev

Now we are ready to build the demo application:

     mkdir build
     cd build
     cmake ..
     make
     ./PeopleTracking

and you should see the demo running. If the demo is dropping frames or having the “slow forward pipeline” warnings, reduce the frame dimension by editing the main() inside the source file PeopleTracking.cpp so it looks like this:

     //Horus eye(80, 60)
     Horus eye(160, 120)
     //Horus eye(320, 240)

Now run make again and everything should be working smoothly.

Below is a video of the setup running one of the applications in the DemoApplications repository.

34 thoughts on “3D Depth Sensor for Raspberry Pi 3”

  1. Muriel

    Hi Larry, is this possible to use sensor_msgs::PointCloud2 with this camera. If yes, why there is no support on Voxel SDK. Many thanks in advance for your kind reply!

  2. Dee

    Hi Larry,
    I have PCL 1.9.1 and Voxel SDK 0.6.11. I am trying to connect OPT8241 with ROS however I couldn’t make it yet. I have checked TI support forum however they confirm that there is no API for ros interfacing of this camera. Do you have any suggestion for me so after interfacing with ROS I can use it in rviz and create a 3D mapping? I’ll appreciate if you could answer for a beginner level one.

    Thanks a lot!
    Best regards.

  3. Sebastian

    Hallo,
    I am running Ubuntu Mate as suggested by TI, but on my raspberry Pi 3B+, 64GB Card it fails when I want to compile PCL. It tells me Virtual Memory is exhaustetd (at about 61%), but I have enabled a swapfile and already tried to increase that to 4GB, but I got stuck at the same place (PCL Surface).
    Did you have any similar problem. Well, I needed to configure the makefile via ccmake .. as the command given by TI and above here did produce error (GPU Module configuration). Do you have any suggestions? Probably with the config of the makefile, but actually I am unsure…

  5. Satheesh Ramasamy

    Hi,,
    Really Thanks for the great steps provided to work with PCL …
    I am using Occipital Structure sensor with Raspberry pi 3, I was able to build properly. but runnig a demo application shows below output message
    ./PeopleTracking
    Cannot connect
    I found that, this ​_sys.scan();​ function returns 0 and hence not able to connect.

    i coundn’t trace completely the issue, kindly help me to resolve the issue ..
    Thanks

    • dibet

      In your CMakeLists.txt you need something like that

      include_directories(/usr/include/voxel-0.6.11/ti3dtof)
      include_directories(/usr/include/voxel-0.6.11/ ${EIGEN_INCLUDE_DIRS} )

      FIND_PACKAGE(Voxel 0.1.0 REQUIRED)
      find_package(OpenCV REQUIRED)

      set(THREADS_PREFER_PTHREAD_FLAG ON)
      find_package (Threads REQUIRED )

      TARGET_LINK_LIBRARIES(myapp ${OpenCV_LIBS} ${VOXEL_LIBRARIES} ${Boost_LIBRARIES} )

    • Nikolaos Athanassopoulos

      Hey SATHEESH RAMASAMY i am trying to do the same thing and i am having a really hard time.Did you manage to make it finally?And if yes would you please help me to resolve the problem?

  6. dibet

    Hi Larry, your blog is very interesting for me. I have learned a lot about the R PI3 and the OPT8241. I developed an application but I wanted to know how to capture the error when the camera is disconnected. Can you send a suggestion?

  7. Dragoneo

    Hi Larry,
    and thanks a lot for your work.
    I got troubles running the last version of Jessie on a 16Gb SD card. I was able to install all packages, and when i do the make from PCL, it took to my computer 4 day to compile for finish with a C++ internal compiler error:
    sample_consensus/CMakeFiles/pcl_sample_consensus.dir/build.make:134
    make[2]: *** [sample_consensus/CMakeFiles/pcl_sample_consensus.dir/src/sac_model_cylinder.cpp.o] error 4
    CMakeFiles/Makefile2:339
    sample_consensus/CMakeFiles/pcl_sample_consensus.dir/all
    make[1]: *** [sample_consensus/CMakeFiles/pcl_sample_consensus.dir/all] error 2
    Makefile:160
    make: *** [all] error 2

  8. fadetoblackdh

    Hello Larry,

    I’m having a bit problem when i try to run cmake-gui .. in “build” file. I made build folder like this “/Software/voxelsdk/build” and after typing cmake-gui .. i get this problem “libEGL warning: DRI2: failed to authenticate” and after that a pop-up comes up and tells me to select a source code. I tried to make build folder in different directories but the error is the same. i’m kinda new to this so i don’t know where i made the mistake. Until this step everything worked out smoothly.

    It would be a huge help if you can help. Thanks.

    • Kagan

      Finally solved the problems while building from source! I had to upgrade to Vtk6 and install OpenNI since this time it was giving me the error at PCL building too. Due to updated version of PCL(v1.8.1) and Voxel(v0.6.11) the prerequisites must be updated too, i guess.

      Just for your information.

  9. Kagan

    Hello Larry,

    I’m having a bit problem when i try to run cmake-gui .. in “build” file. I made build folder like this “/Software/voxelsdk/build” and after typing cmake-gui .. i get this problem “libEGL warning: DRI2: failed to authenticate” and after that a pop-up comes up and tells me to select a source code. i’m kinda new to this so i don’t know where i made the mistake.

    I would be a huge help if you can help. Thanks.

    • Kagan

      Hi chen,

      Try to update prerequisites such as Vtk6 and OpenNI. PCL and Voxel are updated so the required dependencies must be updated too.

  10. KAI

    Hey Larry, I’m stuck at making the demo application. I’m getting missing file error and if I’m following the steps correctly. This demoapplication should be a subfolder of Software, not in voxelsdk or anywhere else. From there, I should make the build file inside the people_tracking folder and build the file from there, right?

    Cheers!

    • From your command line prompt it looks like you launched cmake from ‘Software/release’ but the build files are written to ‘Software/pcl/release’. Please first reconcile that issue.

  12. For your info, using the latest Raspian Jessie Pixel, I was not able to install the libvtk stuff). BTW, there are some lines around QT, Mono and JDK. Do we really need these for getting the Kinect to run? They need another GB on flash…

  13. Ralph

    Hi there!

    I was just trying to go through all these steps when two errors occured:
    1st: Download failed: http/ppa.launchpad.net/v-launchpad-jochen-sprickerhof-de/pcl/ubuntu/dists/xenial/main/binary-armhf/Packager
    –> ignored or older ones used.

    2nd error came up when I enter the cmake -DCMAKE (…)
    “The source directory “/home/ralph” does not appear to contain CMakeLists.txt.”

    I am anything but pro at programming but I somehow have to make this work. Is there a connection between these errors or do you have any suggestion on how to make this work? 🙂

    Of course, you can contact me directly via email.

    Best regards,
    Ralph

    • Ralph

      I am running UbuntuMATE.
      Also, I re-entered the cmake command today (without the “..” at the end) and was able to start the make process. Though, after 3-4% the following errors occur:
      c++: internal compiler error: Killed (program cc1plus)
      …
      full error can be found here (screenshot): https://prnt.sc/fop2yd

      Thanks for your help!

    • Ralph,

      cmake works needs to find CMakeLists.txt, which sets in the root directory containing the source. To build the source, create a ‘build’ directory under the source directory, then ‘cd’ into it. Then issue the “cmake .. ” command. That’s the correct way build.

Leave a Reply to DeeCancel reply