Introduction

Localization for robots can be done in different ways. Machine vision plays a major role when considering the relative localization. There are point based and line based approaches to create the surrounding environment other than the vision based approach. Point based and line based technologies use either fixed or rotating Lidar sensors where more mechanical parts and expensive technologies are used. In this research we have developed a vision based real time low cost depth map generation system that can be used for UAV and robotic applications.

This paper is based on the implementation of the depth map generation system for FPGA. The output was tested on Xilinx Basys3 FPGA which is based on Artix-7 architecture.

Implementation and architecture

The Disparity generation system on FPGA mainly consists of five major units (Figure 1) image acquisition, Image rectification, Image buffers, Disparity generator and output buffers. Each and individual unit has subunits for handling the stereo images effectively.

To decide the parameters such as resolution of stereo image pair, Frame rate for the final output and the disparity comparison depth for the design, a suitable entry level FPGA was needed to be selected. After doing a market research it was found that the BASYS 3 FPGA was a perfect candidate since it consists of required amount of Input Output pins (IO) and due to the fact that it consists of a 100 MHz hardware clock and supports generating higher frequency clock rates through inbuilt Mixed-Mode Clock Manager (MMCM). Then for the camera sensor OV7670 was used due to its 8bit parallel image output bus that can be used to fetch images directly through the FPGA. Another reason for selecting this module was that it is commonly available and the lower price. But the Major limitation of this camera module was the frames rate which was 30fps. The next major bottleneck is the memory of the FPGA. BASYS 3 has 50 cells of BRAM-36k (each cell containing 36,000 bits), providing a total of 1800 Kbits on the chip.

Based on the image size of 320 × 240 the estimations were done to the prototype system. The average image created from left and right images can be eliminated later and it was introduced to retain the debugging capability of the system. For the left, right and average image of resolution 320 × 240 pixels with data of half byte, 115,200bytes (320 × 240 × 4 × 3bits) are required. For storing the disparity image under 256 gray levels 76,800bytes are required. The total memory requirement adds up to 192,000bytes. This is 85% of the available total memory (225000bytes) of the FPGA.

According to the analysis of the specifications related to the FPGA and the camera, the fps was selected as 25fps and the ideal resolution was selected as 320 × 240.

The Implementation plan was created under 3 major sections as

1. Functional verification stage.
2. Stereo Camera integration and testing stage.
3. Disparity implementation on FPGA.

For the functional verification stage, the major concern was given to the Disparity generator unit in the design pipeline. For the disparity calculation module, Sum of Squared Difference (SSD) comparison technique was selected based on the results from the literature review. To reduce the LUTRAM¹ usage of the algorithm a block-wise disparity calculation approach was suggested.

In the stage of Stereo Camera integration and testing the major concern was given to image acquisition, image rectification and output buffer of the design pipeline. Based on the selected hardware communication protocols were planned. (I2C for camera configuration and 8bit parallel bus for image capturing). Sizes of the memory buffers were decided based on the calculations. Multiple clock domains were decided to be used for different modules. Based on the maximum capabilities of the camera, a 25 MHz clock was selected to drive the camera. For the generation of the disparity a variable clock was selected and for memory operations and other operations a 50 MHz clock was proposed.

During the final stage of Disparity implementation on FPGA additional memory buffers were decided to be used in-order to store the intermediate results and several adaptations and optimizations were planned to previously stage outputs in order to integrate them together.

Quality control

Design has been tested and validated using functional verification techniques in simulation environment “ModelSim”. More details and functional verification sources are available at https://github.com/Archfx/FPGA_depthMap.

Operating system

Can be synthesized on Linux and Windows using Xilinx Vivado 2019.1 tool-chain.

Programming language

VHDL, Verilog, Python (Prototyping and code generation)

Additional system requirements

For synthesizing: Minimum requirements to run Xilinx Vivado 2019.1 tool-chain.

For hardware testing: Xilinx FPGA, Two identical Camera Sensors (OV7670).

Dependencies

Block Memory Generator v8.3 LogiCORE IP Product (For manually Configuring BRAM), Clocking Wizard v6.0 LogiCORE IP Product Guide (For manually Configuring Clocks).

List of contributors

For Stereo camera integration prior work done by Engineer Mike Field was used.

Software location

Archive https://github.com/Archfx/FPGA-DepthMap-Basys3/releases

Code repository https://github.com/Archfx/FPGA-DepthMap-Basys3

Name: FPGA-DepthMap-Basys3

Persistent identifier: 10.5281/zenodo.3907295

Licence: MIT License

Date published: 30/06/2019

Language

Documentation available in English.