= Vision Framework = Mentors: Joy Cho Students: Reid Huntley, Benji Albert, Nathan Misner == Background == From 2015-2017, team 2537 developed a machine vision platform for autonomous targeting and navigation. It was first used successfully during the 2016-2017 season and helped the team reach the finals at the world championship. The vision platform is based on the [https://www.raspberrypi.org/products/raspberry-pi-3-model-b/ Raspberry Pi 3] with [https://www.raspberrypi.org/products/camera-module-v2/ camera module] in a [https://www.amazon.com/Distributed-MCM-83-17540-Enclosure-Raspberry/dp/B01KGTD3I0 clear case]. The 2016-2017 version used an [https://www.amazon.com/eBoot-MP1584EN-Converter-Adjustable-Module/dp/B01MQGMOKI MP1584EN power supply] to convert the 12v battery voltage to the 5v required for the machine vision hardware. It also included a transistor switch to allow the Pi to control power to a [http://www.andymark.com/product-p/am-3597.htm green 12v LED light ring]. Communication between the Pi and roboRIO was uni-directional using PWM; the PWM duty cycle represented the angle for the robot to rotate to be perfectly aligned with the target (the gear peg in Steamworks). Many lessons were learned including: * Mount the vision system securely, the more it moves relative to the robot, the less accuracy it will deliver * Don't try to use vision while the robot is driving, the vibration makes the images move too much * Protect your vision module from collisions: we used a custom polycarbonate shield == 2017-2018 Objectives == Extend the vision module software to make it more accessible to other teams and more user friendly generally. Changes planned include: * Execute a base program on startup that continuously reads the camera feed (subscribes to camera) and publishes corresponding data via the Pi's serial output (located on the Pi's GPIO connector). This data can then be fairly simply read by the RoboRIO. * Improve processing speed to yield better resolution/frame rate * Replace the 12v monochrome LED ring with a smart Neopixel-compatible 12x RGB LED light ring based on the WS2812 controller. This will allow the module to run entirely from 5v, allow teams to select their light-ring color, and even make fancy light shows. Smart light rings are available from [https://www.adafruit.com/product/1643 adafruit] and [https://www.ebay.com/itm/12Bit-WS2812-5050-RGB-LED-Ring-Strip-Integrated-Drivers-for-Arduino-Module/391717334432 eBay] among other sources. You can read about the neopixel [https://learn.adafruit.com/neopixels-on-raspberry-pi/overview here]. Neopixel control can be achieved using many libraries including [http://diozero.readthedocs.io/en/stable/ diozero]; see also [http://rtd.diozero.com/en/latest/LEDStrips/ here] == Materials == === Vision Module === * [https://www.amazon.com/Distributed-MCM-83-17540-Enclosure-Raspberry/dp/B01KGTD3I0 Clear plastic case] * [https://www.raspberrypi.org/products/raspberry-pi-3-model-b/ Raspberry Pi 3] * [https://www.raspberrypi.org/products/camera-module-v2/ Pi camera module w/cable] * [https://www.adafruit.com/product/1643 NeoPixel Light Ring] * Qty 3 M-F and qty 3 F-F [https://www.amazon.com/gp/product/B017NEGTXC jumper wires] * 16GB or larger Micro SD card === Additional materials needed during development === * Usb Keyboard and Mouse (for developing on the Pi) * HDMI cable or [https://www.amazon.com/gp/product/B016HL49OS HDMI-to-VGA adapter] (for developing on the Pi) * OPTIONAL Console Cable (This option can be used in the case when HDMI cables are unavailable!) [https://learn.adafruit.com/adafruits-raspberry-pi-lesson-5-using-a-console-cable/enabling-serial-console] * Monitor (that accepts HDMI or VGA input) * Micro USB Cable * USB charger capable of supplying 5v at 2A or greater == Raspberry Pi Setup == === Raspbian Lite === Install [https://sourceforge.net/projects/win32diskimager/ Win32 Disk Imager][[BR]] Get the [https://drive.google.com/file/d/17DSNVyRUJDLNU41tlGdiv0dif2-_QGQ7/view?usp=sharing premade image] and write it on to a 16 gb or larger card or if you want to "do it yourself"... Install [https://downloads.raspberrypi.org/raspbian_lite_latest Raspian Lite] onto MicroSD card via Win32 Disk Imager Load Raspberry Pi 3 (rpi) with the newly imaged MicroSD {{{ username: pi password: raspberry }}} connect to wifi by adding the following to the /etc/network/interfaces file: {{{ auto wlan0 iface wlan0 inet dhcp wpa-ssid "your-ssid" wpa-psk "your-password" }}} reboot the pi: {{{ sudo shutdown -r now }}} update with the following commands: {{{ sudo apt-get update sudo apt-get upgrade sudo apt-get dist-upgrade sudo apt-get clean }}} set localization configuration options: {{{ sudo raspi-config }}} reboot: {{{ sudo shutdown -r now }}} === X server === Install Xorg and Xinit: {{{ sudo apt-get install --no-install-recommends xserver-xorg sudo apt-get install --no-install-recommends xinit }}} Install the MATE GUI: {{{ sudo apt-get install mate-desktop-environment-core }}} Install LightDM login manager: {{{ sudo apt-get install lightdm }}} reboot: {{{ sudo shutdown -r now }}} login to MATE[[BR]] At the top left:[[BR]] navigate to {{{preferences -> hardware -> keyboard shortcuts}}} to set Run a Terminal to {{{Ctrl + Alt + T}}} or the like open a terminal[[BR]] navigate to {{{Edit -> Profile Preferences -> Colors}}} to deselect {{{Use colors from system theme}}}[[BR]] select {{{Built-in schemas: White on black}}} === Development Tools === ==== Java 8 ==== Now that the terminal color scheme is not killing you, we are going to start off by installing java: {{{ sudo apt-get install oracle-java8-jdk }}} test the java installation (output should include "1.8.0_65" and should not include "openjdk"): {{{ java -version }}} Let's also set the JAVA_HOME variable to be exported on startup: {{{ echo "export JAVA_HOME=/usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt" >> ~/.bashrc source ~/.bashrc }}} Finally, let's install ANT: {{{ sudo apt-get install ant }}} ==== Pi4J (Java-GPIO Interface) ==== Now we can install [http://pi4j.com/install.html pi4j], a Java interface for the pi GPIO: {{{ curl -s get.pi4j.com | sudo bash }}} ==== rpi_ws281x (NeoPixel Control Library) ==== Then we can use the rpi_ws281x library for controlling the [https://www.adafruit.com/category/168 Adafruit NeoPixel lightring][[BR]] But first, let's install some tools and dependencies: {{{ sudo apt-get install build-essential python-dev git scons swig }}} Now we can install the rpi_ws281x library: {{{ git clone https://github.com/jgarff/rpi_ws281x.git cd rpi_ws281x scons }}} ==== OpenCV ==== First of all, order some pizza or something because this is going to take about 3 hours (if you are lucky). Also, you need a heatsink for your processor before building OpenCV. If you see this overheating symbol, you will die. Straight up.[[BR]] [[Image(https://i.imgur.com/kYydNgVm.jpg)]][[BR]] '''EDIT: we have tried with a heatsink and by itself, it didn't work. You need to used forced air (fan) or water *and* a heatsink to cool your pi. Please skip to the WATER COOLING section.'''[[BR]] If you don't have a heatsink on hand, tape some pennies to the CPU like we did:[[BR]] [[Image(https://i.imgur.com/KqWLvKJm.jpg)]][[BR]][[BR]][[BR]] '''================WATER COOLING================'''[[BR]][[BR]] Poor man's water cooling system:[[BR]] [[Image(https://i.imgur.com/DPOazIJm.jpg)]][[BR]] Make sure to change the ice cube at least once every half hour. Also, only single bag the ice cube.[[BR]][[BR]][[BR]] Now we are going to install OpenCV and its dependencies.[[BR]] We only install OpenCV for C++ and Java, not for Python (we ain't scrubs): {{{ sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev cd git clone https://github.com/opencv/opencv.git cd opencv mkdir build cd build cmake .. make -j $[$(nproc)+1] sudo make install }}} This builds a single shared object file (libopencv_java320.so), a jar file (opencv-320.jar) and a ton of static libraries (*.a)[[BR]] It copies the shared object file and the jar file to the install directory: {{{/usr/local/share/OpenCV/java/...}}}[[BR]] NOTE: we could not get the java wrapper to work with ffmpeg enabled (we ran into a Segmentation fault with opencv libs 3.2.0, 3.3.0, and 3.3.1). We gave up and disabled it.[[BR]] NOTE: All of the below steps work with OpenCV versions 3.2.0, 3.3.0, and 3.3.1. Use whichever you prefer. If you do not want to git clone the master opencv repository, download a zipped version: [https://github.com/opencv/opencv/archive/3.2.0.zip opencv-3.2.0.zip][[BR]] === Java Projects with OpenCV === ==== Enable VideoCapture ==== First, to allow OpenCV VideoCapture to access the ribbon cable camera, execute the following: {{{ echo "sudo modprobe bcm2835-v4l2" >> ~/.bashrc source ~/.bashrc }}} ===== Compiling ===== {{{ javac -d -classpath /*.java }}} where: * {{{build_path = directory to store class files}}} * {{{external_jars = string specifying individual jar dependencies (such as OpenCV and Pi4J) with delimiters as follows:}}} * {{{:/path_to_jars/jar1.jar:/path_to_jars/jar2.jar}}} * {{{souce_path = directory containing all of your java files to be compiled}}} We recommend creating a {{{libs}}} folder in which you store all of your external jars such as the {{{opencv-320.jar}}} and all the Pi4J jars rather than having your libraries scattered all over your machine. ===== Running ===== {{{ sudo java -Djava.library.path= -Dpi4j.debug -Dpi4j.linking=dynamic -classpath }}} where: * {{{external_jars = same string that you used in the Compilation step}}} * {{{Main_class = name of the class file with main method}}} * For example, where {{{Vision.class}}} is the class file, {{{Main_class = Vision}}} === C++ Projects with OpenCV === '''NOTE: our performance doubled when we transitioned to C++ from Java'''[[BR]][[BR]] '''With C++, we can process (capture and segment) 640x480 frames at 66 fps and 1080x720 frames at 28 fps'''[[BR]][[BR]] '''With Java, we could only get about 30 fps with 640x480 frames''' Let's install an IDE so we do not have to keep writing in vi/vim/nano: {{{ sudo apt-get install codeblocks }}} We experienced a bug that constantly crashed CodeBlocks:[[BR]] To prevent crashing, navigate to: {{{Settings -> Editor -> Code completion -> Symbols browser}}} and check {{{Disable symbols browser}}}[[BR]] (the {{{Code completion}}} tab is on the west panel of the configure editor.) Navigate to: {{{File -> New -> Project -> Console Application}}} And fill in the fields that come up. If your {{{Management}}} tab is not visible, open it by selecting: {{{View -> Manager}}}[[BR]] Also, if your {{{Logs & others}}} tab is not open already, select {{{View -> Logs}}} Now right click your project in the {{{Management}}} tab and navigate to: {{{Build options... -> Linker settings}}} Under the {{{Link libraries}}} panel, select {{{Add}}} and select all OpenCV shared object (.so) files located in {{{/home/pi/opencv/build/lib/*.so}}}[[BR]] (You can [ctrl + left-click] multiple *.so] files to add them all at once) Now let's navigate to: {{{Search directories -> Linker}}}[[BR]] In the {{{Linker}}} tab, select {{{Add}}} and add the path to your opencv lib: {{{/home/pi/opencv/build/lib}}} ===== Tricks and Tips ===== Using OpenCV with the Raspberry Pi 3 ribbon cable camera is a little cumbersome because you cannot use many of the VideoCapture.set(char*,int) functions.[[BR]] Instead, we use system commands to set camera settings via the v4l2-ctl library. For FRC retroreflective tape segmentation, use the following system commands to prevent auto adjustment features such as {{{auto exposure}}}, {{{white balance}}}, and {{{exposure time}}}: * v4l2-ctl --set-ctrl=auto_exposure=1 * v4l2-ctl --set-ctrl=white_balance_auto_preset=0 * v4l2-ctl --set-ctrl=auto_exposure_bias=0 * v4l2-ctl --set-ctrl=exposure_time_absolute=100 Make your C++ program set these options by using the {{{stdlib.h}}} {{{system(char*)}}} command which simply takes a string command as input.[[BR]] Alternatively you could add these options to your {{{bashrc}}} file like we did with the modprobe, but you may want to tweak these options or experiment with them, so putting them into your C++ program makes it a little easier to manipulate.