Final Year Project ~ Syntax Lines

Introduction

Project Introduction

The target of this project is allowing the patient to control a Remote Control Car(RCC) based on head movement. The RCC is combination of micro controller and camera. The purpose of micro controller is able to create communication between the RCC and the patient. The camera function a the patient “eyes” on the RCC. It will transmit a video feedback to the patient.
The patient will only require only a head movement, this allowing patient with various kinds of injuries or diseases capable using the system. The Patient able to use this system if have a capabilities to move their head either sit or laying using this system. As long the system still able detect patient head and patient position does not constrain the movement.

Project Use Case

At the end of the project, the user able to navigate the RCC wireless via head movement. The system will be involving image processing, signal transmission and electrical ingenuity.
This project also continues from the previous FYP. The image processing process is to enable detect and track user head motion. Based on the head motion it produces a signal input. On the RCC side, based on signal receive. The RCC will respond corresponding.

Reason You Pick This Topic

The reason I pick this topic because during my second last semester, I was helping one of my senior with his FYP. I'm very invest with the project, the project is very similar with mine. However I found out there is a couple problem with the overall design.

Using eye pupil as RCC base direction.
By using eye pupil as direction, the user is facing problem when steering the RCC. After a few testing, the user having difficulty looking where the RCC is going.
Change the original RCC
The original RCC is a normal out of box 4x4 remote control. The problem occurs when additional hardware install on top of RCC. It mostly due to RCC cant accommodate additional add on's.
Problem in transferring image from RCC to application
The previous version an other 3rd party application to transfer real time image.
Problem connectivity between RCC and application
The previous point, I mention a 3rd party application. This application require an internet connection. The directional input from application to RCC is using radio wave. From here we can see there is a more connection required. My suggestion is to use internet as a single connection.

Project Plan

The following is the timeline for final year project.

Week 1 - 4

For the first 4 week I spent mostly on research purpose. Research on micro-controller, image processing and little bit on engineering basic. Beside researching I did consult with my senior and lectures.

Week 5 - 6

After I finish the rough draft from my planning and moving into implementation. I split the implementation into couple of module

User to Application
Main Application to (RCC)
RCC to Engine
Camera to Micro-controller

Week 7 - 8

Base on previous module. I finishing up the skeleton(prototype) application. It can be consider as a Alpha state since every module is still separated.

Week 9

Preparation for presentation. This will be the first presentation, the purpose is to check rather our current FYP fulfill the requirement.

Week 10

Presentation Day
During the presentation I explain the functionality of each module and how it is fulfill the requirement. At this current moment only the controller unit and image process is finish.

Week 11-14

For this week, I postpone my current progress to focus my exams.

Week 1 -7 (New Semester)

In this phase, I try moving the project from Alpha state to Beta state

Base on the first presentation. Do a correct or redesign
Try connect all the module
Build the RCC
Keep update the documentation

Week 8 (New Semester)

Bug fixing and optimizing.

Week 9 (New Semester)

Final presentation preparation. This include presentation slide,poster and documentation .

Week 10 (New Semester)

Final presentation day.

Modules

To simplify, I separate the project into multiple modules Module 1,2 and 3. The purpose of breaking the project into multiple module is easier to standardize the input and output on each module.

Module Name/Entity	Module Description
Patient	-
Module 1	Application & Camera
Module 2	RCC
Module 3	Engine(s) + Camera(s)

Module Description

In this section, it will give a brief explanation on how the overall modules process. The more details explanation will follow section.

In module 1 is where the interaction happen between the patient and the application. The application also handle process of visual image processing between patient and application, processing and and application and RCC. In additional, the application also provide a patient camera configuration for patient comfort and provide a video feedback

The Module 2 is handle on RCC side. The RCC require handle process after receive command from application. In respond RCC also require a video feedback back to the application.
The last module is the I/O for RCC. The engine(s) is will move forward,stop or backward base on RCC command. While the input is provided from the camera(s), the purpose of the camera is to capture a video feedback. There is a 2 camera located on RCC, one at the back and other one at the front. This allow to to capture a visual video at the both end.

At the end the micro controller will receive a command from the application and transmitted a video from the RCC camera. After receiving the command, the RCC will act accordingly. For transmitted a video, the RCC will receive frame by frame by the RCC camera. The single frame later will be converted into matrix to be compressed and finally convert into byte. The reason of compression is to minimize the cost of transmitting. The result byte later will be repackage with a header and tail, the same method in module 1 After receiving the package, the application will convert back into a frame following an inverse process. Therefore the video feedback it is a string of a frame transmitted from the RCC camera. In micro controller side, I was suggested using a multi threading. By using multi threading able to avoid hard reset if due to lost connection. If it enter reconnecting phase, instead of restarting multi threading simply remove old thread and create a new thread for both receiving and transmitting.

Sequence Diagram

Module 1

In Module 1, this is where the heavy lift occurs.For the head movement, I used image processing(OpenCV) for the detection. The purpose of this module is to calculate the user head movement. The result will be transmitted to the Raspberry Pi. The following is the initial step of detecting user face:

The user can adjust the angle of the camera comfortable for his head level.
If comfortable, the user will press capture. The application will calculate the current position.
The next frame will be compared with the 'savePoint'.

Packet Packaging

The command will be repackaged with a Initialize.length, Data_length. length and Data.length. Initialize.length is a cipher protection. If the receiver initialize does not have the same initialize, they are unable to read the content of the package. Data_length purpose to check rather the package is safely send during the transmission. The Data.length is the data after been compress. This process will prevent inconsistency during the communication. Example if the package only able transmitted half, the RCC will discard it.

Calculation Head Motion

In the calculation of head motion, it requires a two point. The first point is from the savePoint and the second is from the current frame(currentFrame). The GU display center will be position compare to savePoint. The calculation algorithm is divided into two parts since it has two axis. It will check the x-axis first follow with the y-axis. The reason x-axis have -15/+15 and y-axis -10/+10 allowing a rest spot for the user as long under it boundaries.

Current Position	Command
a.	Turn Left
b.	Foward
c.	Turn right
d.	Reverse toward left
e.	Reverse
f.	Reverse toward right
Normal line	(Stop) rest spot

The centerPoint.x is the current x point from the current frame. The savePoint is from the generate image.The calculation algorithm is divided into two parts since it has two axis (x-axis and y-axis). It will check the x-axis first follow with the y-axis.

Calculate x - axis	X-axis Result
If (centerPoint.x < savePoint.x - 15)	Left
If (centerPoint.x < savePoint.x +15)	Right
else	Normal

Calculate y - axis	Y-axis Result
If (centerPoint.y < savePoint.y - 10)	Foward
If (centerPoint.y < savePoint.y + 10)	Reverse
else	Normal

The centerPoint.x is the current x point from the current frame. The savePoint is from the generate image.

Calculate X and Y		Result
X	Y	Result
Normal	Normal	Normal
Left	Foward	Turning left
Normal	Foward	Foward
Right	Foward	Turning right
Left	Reverse	Reverse to the left
Normal	Reverse	Reverse
Right	Reverse	Reverse tot the right

Module 2

In this module, The micro controller will receive a command from the application and transmitted a video from the RCCcamera. After receive the command, the RCC will act accordingly. For transmitted a video, the RCC will receive frame by frame by the RCC camera. The single frame later will be converted into matrix to be compressed and finally convert into byte. The reason of compression is to minimize the cost of transmitting. The result byte later will be repackage with a header and tail, the same method in module 1

After receiving the package, the application will convert back into a frame following an inverse process. Therefore the video feedback it is a string of a frame transmitted from the RCCcamera.

In micro controller side, I was suggested using a multi threading. By using multi threading able to avoid hard reset if due to lost connection. If it enter reconnecting phase, instead of restarting multi threading simply remove old thread and create a new thread for both receiving and transmitting.

RCC Component

The most important component in the RCC it is the micro-controller. The micro-controller is important because it act as a receiver from the user and a controller for the engine. The micro-controller will be connected with a engine driver. The engine driver is dual motor controller. The engine driver able for direct drive of two bi-directional DC motors, forward an reverse.

Pin no	Name	Connection
2	5.0 VDC Power	Connected to the positive engine volt input
4	5.0 VDC Power	Connected with a positive wire from battery pack
6	Ground	Connected with a negative wire from battery pack
3	GPIO8 SDA1 (12 C)	Connected with engine driver on the right side. This pin able output high and low.
5	GPIO9 SCL1 (12 C)	Connected with engine driver on the left side. This pin able output high and low.
12	GPIO1 PCM_CLK/PWMO	Connected with the enable pin right side of the engine driver. This pin able output high and low.
23	GPIO1 PWM1	Connected with the enable pin right side of the engine driver. This pin able output high and low.
39	Ground	Connected to the Ground in engine driver.

E	M	Run
Low	Low/High	Stop
High	High	Back Direction
High	Low	Foward Direction
PWM	Low/High	Speed

Module 3

In Module 3 mostly handle to I/O for the RCC. The input will come from RCCcamera while the output is the motion provide from the engine. At the very last moment I get suggestion to add an additional camera. One will be facing front while the other will facing to the back. If reversing the user able to rear view.

Finalize Application GUI

For the head movement, I used image processing(OpenCV) for the detection. The purpose of this module is to calculate the user head movement. The result will be transmitted to the micro controller. The following is the initial step of detecting user face:

The user can adjust the angle of the camera comfortable for his head level.
If comfortable, the user will press capture. The application will calculate the current position.
The next frame will be compared with the savePoint.

The command will be repackaged with a Initialize.length, Data_length.length and Data.length. Initialize.length is a cipher protection. If the receiver initialize does not have the same initialize, they unable read the content of the package. Data_length purpose to check rather the package is safely send during the transmission. The Data.length is the data after been compress. This process will prevention inconsistency during the communication. Example if the package only able transmitted half, the RCC will discard it.

Server Side

In the server side, I use a multi thread for capturing frame and receiving command. The reason for this dedication is to prevent any latency while capturing and listening to the user. When the server is properly connected, Command thread will start. In the thread it will only create frame base on the camera.

In ClientWorker work same as the Command class. But instead of create a new frame, this class a wait command from the user. ReadStream function purpose is to change the data in byte type in String type. The command latter will be pass to the GPIO class. The GPIO class will execute the command base on the command receive.

RCC Final Version

Application Setting

The application is included a setting option, that allow patient adjust that suitable with his preference.

Brightness adjust setting

RCC connection

Flipping video feed

Test Scenario

In this chapter, the application will be tested under a control environment. The test module will receive pass if the application able to detect user at least 5 second without interrupt. If the detection fail to detect at least for 5 second, then the test module is fail.

	Test Result	Result and Observation Statement
		Pass, able to detect user head.
		Pass, able to detect user head.
		Pass, able to detect user head.
		Pass, able to detect user head.
		Pass, able to detect user head.

Second Scenario

In head detection testing, the testing is divided into four scenario. The first scenario is test under control environment. The user while be sitting under 2 light build. The user face will be facing 90 degrees horizontal toward the web camera. The scenario will be testing does position of the light source effect the detect action

	Test Result	Result and Observation Statement
		Pass, able to detect user head. However if the user wear glasses, the application will have difficulty in detecting user.
		Pass, able to detect user head. However if the user wear glasses, the application will have difficulty in detecting user.
		Pass, able to detect user head. However if the user wear glasses, the application will have difficulty in detecting user.
		Pass, able to detect user head. However if the user wear glasses, the application will have difficulty in detecting user.

Second Scenario

Result

No	Test Module	Purpose	Result
A. Application
1	Application able connect with micro-controller	Test rather the system can connect with the micro-controller	Pass
2	Application able connect to the web camera	Test rather the application able connect to web camera	Pass
3	Application able connect with micro-controller camera	The test to check, if the application able to connect to the micro-controller camera	Pass
B. Application Interface
4	Click Setting menu	To test rather the menu is appears when the user is click	Pass
5	Click the Flip Image menu	To test rather the video feedback in inverted when the user is click	Pass
6	Click the Generate Image menu	To test rather the Generate Panel is created when the Generated button is press	Pass
7	Brightness Slider manipulate the user video feedback	To check rather user video feedback corresponding to the Brightness slider	Pass
8	Toggle Display User Line	To check if the Display User Line will appear if the Display User Line mode is on	Pass
9	Click the START button	To check rather the face tracking start tracking	Pass
10	Click the Generate button	To check rather the system able to create the generate image is created	Pass
11	Click the Stop button	To check rather the face tracking is stop	Pass
12	Open with web camera	Able to detect if the system does not requirement of the system	Pass
C. Camera
13	Video able capture from web camera	To check weather the PC web camera able capture feed from the web camera	Pass
14	Web camera video able display in a proper visual standard	To check rather the video feedback is reach a certain standard	Pass
15	Video capture from the microcontroller	To check weather the computer can get feed back from the mircocontroller	Pass
D. RCC
16	Able detect to the RCC	Testing weather the user able to communicate with the RCC	Pass
17	Test RCC able receive command	Testing in RCC able receive command from the user	Pass
18	RCC able to move	Testing rather the RCC able support the additional weight	Pass
19	RCC able to turn	Testing weather the RCC able to change direction during moving	Pass
E. Detection and Tracking Method
20	Detecting user head	Testing weather the system able detect the user head	Pass
21	Tracking user head motion	Testing weather the system able detect the current user head motion	Pass
22	User partially cover vertically half of user face	Testing weather the system able to detect if user partially cover his face	Fail, the system unable to detect.
23	User wearing glasses	To test weather the system able detect the user head if user wear a glasses	Pass, however if the room brightness is very low, it may effect the efficiency
24	Web camera is position is more 20 cm from the user	Testing the distance limit between the user and the user web camera	Pass
25	User rotate less 35 degree to right	Testing the system, whether the system able compute and display the direction of the user facing	Pass
26	User rotate less 35 degree to left	Testing the system, whether the system able compute and display the direction of the user facing	Pass
27	User rotate less 25 degree to up	Testing the system, whether the system able compute and display the direction of the user facing	Pass
28	User rotate less 35 degree to down	Testing the system, whether the system able compute and display the direction of the user facing	Pass
29	User rotate more 35 degree to right	Testing the system, whether the system able compute and display the direction of the user facing	Fail. But it is not recommended, dangerous due to user unable see the screen
30	User rotate more 35 degree to left	Testing the system, whether the system able compute and display the direction of the user facing	Pass
31	User rotate more 35 degree to down	Testing the system, whether the system able compute and display the direction of the user facing	Pass
32	User move head more 35 degree upward	Testing the system, whether the system able compute and display the direction of the user facing	Pass

Test Result

Test Result Alhamdulillah, after 2 semester struggling completing this project. I receive A- for all my hard work and mark as my greatest achievement.

Syntax Lines

My portfolio and hobby

Menu

Final Year Project