Future Work & Improvements

Limitations

At Present the two approaches have been implemented and compared
Approach I is not very realistic in the sense that the problem modelling does not take into account realistic constraints, which would be faced by a real robot. For eg

Due to a state space decomposition the change in bat velocity at the instant when bat hits ball is abrupt.
Also between timesteps the bat motion is not continuous since change in bat position is brought about by increments in bat coordinates. (discrete values).

Some of the problems being faced in Approach II are

Even though the weights of the neural net are dumped and loaded in successive runs to tune the net better , increasing the number of iterations of this process does not help us achieve total 100 % learning.
In fact it has been observed that beyond 6- 7 iterations the values sometimes stabilise and in some cases worsen

Future Work

Improvements in Learning Approach

A Comparison of the two approaches demonstrates that the more complex the problem formulation is, the better the nature of solution obtained but the more difficult it is achieve learning.
The first approach which involved a state space decomposition was a simpler strategy which showed that the Q-Learning Network Algorithm works well for a problem of this stature. However the second approach which provides a more realistic solution is more difficult from the Q-learning point of view . This is because the reward function , state space description etc are quite complex.

The following things can be changed in the second approach, which may give fruitful results.

state space description ,

action set

reward functions

probabilistic model ( exploration vs exploitation)

Effect of spin can be considered , this would effectively

Increase the number of state variables (currently 12)- 3 extra variables - bat angles (orientation wrt ground ) would be required.
Increase the action set size (currently 3*3*3=27), apart from velocity increments, there would have to be independents increments in each of the 3 angles.
Cause an enormous increase in state space size.

Some sort of a state space discretization should be attempted.

Improvements in the simulator

A more realistic model chosen for simulating the collision between bat & ball and also between table surface , net & ball.
An error factor should be added which should diminish with the distance between ball and bat. This error factor is nothing but a measure of the uncertainty that a human player faces when he tries to judge the trajectory of the ball. This will be important when vision-based tracking of the ball at some timestep.