Lin
Blind Interaction
Used computer vision algorithms to identify whether the pin was punched in the correct position for each millimeter hole. Improved the hardware defect of the tactile terminal for blind people. Also, used Unity to develop modules that connect to the hardware program.
Improve Hardware Defects of the Device (Large-format Graphical Tactile Display Terminal) Based on Computer Vision and Deep Learning
Hexun Lin, Yang Jiao (Supervisor)
Future Lab (Lead by Prof. Yingqing Xu), Tsinghua University
CCTV (China Central Television) News about This Project:
http://tv.cctv.com/2021/04/25/VIDE38Bty7Euyo4ES494JhNE210425.shtml
Internship Report: https://drive.google.com/file/d/1DEHoKxDzQrLRPFfPy3R2uc4oLrn1GGLH/view?usp=sharing
Due to impaired vision, blind people mainly rely on their sense of hearing and touch to access information. With the development of technology and the popularity of the Internet, when visual information such as pictures and videos become the primary source of information in knowledge dissemination, blind people urgently need accessible devices that can read image information conveniently.
Large-format Graphical Tactile Display Terminal is a device independently developed by Tsinghua University's Future Lab. The device consists of a concave and convex dot-matrix controlled by a computer. It turns the pictures in the computer into touchable dot graphics, allowing blind people to "see" the images by touching those dots.
The device is more affordable than existing touchable devices for the blind because it is made with inexpensive open-source hardware and has used our own control technology. If mass production is achieved, the price is only one-tenth to one-twentieth of similar foreign devices.
However, due to the hardware defects, the holes of the dot matrix are too narrow, and it often happens that the pins cannot go through the holes. Therefore, I was independently responsible for optimizing the problem using computer vision techniques in this project. Let the camera capture the graphical, tactile display position, establish the dot matrix coordinates, and automatically identify whether the dot matrix at the corresponding pixel location successfully protrudes through the hole, and if not, push the pin in the corresponding pixel location again. Also, I used Unity to develop modules that connect to the hardware program.
