A method for a hand tracking of a hand of a user in a Head Mounted Display (HMD) device, includes: detecting, during at least one current frame associated with a field-of-view (FOV) of the HMD device, a disproportionate hand movement while the hand tracking of the hand is performed; predicting, when the disproportionate hand movement is detected, a trajectory of a movement of the hand, based on a correlation between an eye gaze location of the user and a hand location of the user in one or more previous frames associated with the FOV of the HMD device; and performing, when the disproportionate hand movement is detected, the hand tracking, based on the predicted trajectory of the movement of the hand.