Start Submission Become a Reviewer

Reading: Register Transfer Level Disparity generator with Stereo Vision

Download

A- A+
Alt. Display

Software Metapapers

Register Transfer Level Disparity generator with Stereo Vision

Author:

Aruna Jayasena

RTL developer, University of Moratuwa, LK
X close

Abstract

This tool can generate binary files for Xilinx FPGAs for “Stereo Vision-based disparity generation”. With related vendor tools this project can be ported to other types of FPGAs (such as Intel technology). Implementation is done using VHDL and Verilog hardware description languages. The tool is available in 3 stages 1). Functional Verification 2). Stereo Camera integration 3). Disparity Generation. Documentation is available for users who are interested in modifying for different platforms. Components that are used during physical setup are explained in the documentation and based on the requirements they can be changed. In order to use this tool, the user must have prior experience in hardware description languages, experience in Xilinx tools will be an additional advantage.

How to Cite: Jayasena, A., 2021. Register Transfer Level Disparity generator with Stereo Vision. Journal of Open Research Software, 9(1), p.18. DOI: http://doi.org/10.5334/jors.339
17
Views
2
Downloads
  Published on 19 Jul 2021
 Accepted on 07 Jul 2021            Submitted on 01 Jul 2020

(1) Overview

Introduction

Localization for robots can be done in different ways. Machine vision plays a major role when considering the relative localization. There are point based and line based approaches to create the surrounding environment other than the vision based approach. Point based and line based technologies use either fixed or rotating Lidar sensors where more mechanical parts and expensive technologies are used. In this research we have developed a vision based real time low cost depth map generation system that can be used for UAV and robotic applications.

This paper is based on the implementation of the depth map generation system for FPGA. The output was tested on Xilinx Basys3 FPGA which is based on Artix-7 architecture.

Implementation and architecture

The Disparity generation system on FPGA mainly consists of five major units (Figure 1) image acquisition, Image rectification, Image buffers, Disparity generator and output buffers. Each and individual unit has subunits for handling the stereo images effectively.

Figure 1 

Design Pipeline.

To decide the parameters such as resolution of stereo image pair, Frame rate for the final output and the disparity comparison depth for the design, a suitable entry level FPGA was needed to be selected. After doing a market research it was found that the BASYS 3 FPGA was a perfect candidate since it consists of required amount of Input Output pins (IO) and due to the fact that it consists of a 100 MHz hardware clock and supports generating higher frequency clock rates through inbuilt Mixed-Mode Clock Manager (MMCM). Then for the camera sensor OV7670 was used due to its 8bit parallel image output bus that can be used to fetch images directly through the FPGA. Another reason for selecting this module was that it is commonly available and the lower price. But the Major limitation of this camera module was the frames rate which was 30fps. The next major bottleneck is the memory of the FPGA. BASYS 3 has 50 cells of BRAM-36k (each cell containing 36,000 bits), providing a total of 1800 Kbits on the chip.

Based on the image size of 320 × 240 the estimations were done to the prototype system. The average image created from left and right images can be eliminated later and it was introduced to retain the debugging capability of the system. For the left, right and average image of resolution 320 × 240 pixels with data of half byte, 115,200bytes (320 × 240 × 4 × 3bits) are required. For storing the disparity image under 256 gray levels 76,800bytes are required. The total memory requirement adds up to 192,000bytes. This is 85% of the available total memory (225000bytes) of the FPGA.

According to the analysis of the specifications related to the FPGA and the camera, the fps was selected as 25fps and the ideal resolution was selected as 320 × 240.

The Implementation plan was created under 3 major sections as

  1. Functional verification stage.
  2. Stereo Camera integration and testing stage.
  3. Disparity implementation on FPGA.

For the functional verification stage, the major concern was given to the Disparity generator unit in the design pipeline. For the disparity calculation module, Sum of Squared Difference (SSD) comparison technique was selected based on the results from the literature review. To reduce the LUTRAM1 usage of the algorithm a block-wise disparity calculation approach was suggested.

In the stage of Stereo Camera integration and testing the major concern was given to image acquisition, image rectification and output buffer of the design pipeline. Based on the selected hardware communication protocols were planned. (I2C for camera configuration and 8bit parallel bus for image capturing). Sizes of the memory buffers were decided based on the calculations. Multiple clock domains were decided to be used for different modules. Based on the maximum capabilities of the camera, a 25 MHz clock was selected to drive the camera. For the generation of the disparity a variable clock was selected and for memory operations and other operations a 50 MHz clock was proposed.

During the final stage of Disparity implementation on FPGA additional memory buffers were decided to be used in-order to store the intermediate results and several adaptations and optimizations were planned to previously stage outputs in order to integrate them together.

Quality control

Design has been tested and validated using functional verification techniques in simulation environment “ModelSim”. More details and functional verification sources are available at https://github.com/Archfx/FPGA_depthMap.

(2) Availability

Operating system

Can be synthesized on Linux and Windows using Xilinx Vivado 2019.1 tool-chain.

Programming language

VHDL, Verilog, Python (Prototyping and code generation)

Additional system requirements

For synthesizing: Minimum requirements to run Xilinx Vivado 2019.1 tool-chain.

For hardware testing: Xilinx FPGA, Two identical Camera Sensors (OV7670).

Dependencies

Block Memory Generator v8.3 LogiCORE IP Product (For manually Configuring BRAM), Clocking Wizard v6.0 LogiCORE IP Product Guide (For manually Configuring Clocks).

List of contributors

For Stereo camera integration prior work done by Engineer Mike Field was used.

Software location

Archive https://github.com/Archfx/FPGA-DepthMap-Basys3/releases

Code repository https://github.com/Archfx/FPGA-DepthMap-Basys3

Name: FPGA-DepthMap-Basys3

Persistent identifier: 10.5281/zenodo.3907295

Licence: MIT License

Date published: 30/06/2019

Language

Documentation available in English.

(3) Reuse potential

This tool can be used to generate the binary files for any type of Xilinx FPGA boards (Tuned and optimized for Basys 3 FPGA to fit into available resources). For Altera and other types of boards, source code can be directly used and should be configured using relevant vendor tools. Can be used for robotics projects as a budget solution for disparity generation setup for use cases such as obstacle avoidance, collision avoidance, point cloud generation, etc.

For any issues or inquires please use the GitHub source page.

Notes

1LUTRAM is distributed memory implemented within the FPGA’s logical lookup tables and completely independent of BRAM. 

Acknowledgements

This project was done with the help and resources from IntelliSense Lab, Computer Science and Engineering department, University of Moratuwa, Sri Lanka.

Competing Interests

The author has no competing interests to declare.

comments powered by Disqus