Systems and methods for six-degree of freedom (6-DoF) pose estimation of a user input device, e.g., in a three-dimensional (3D) display system rendering interactive augmented reality (AR) and/or virtual reality (VR) experiences include the user input device capturing, via a camera disposed at a forward-facing tip of the user input device, images in a direction that the user input device is directed and determining, via an inertial measurement unit (IMU), motion of the user input device in three-dimensional (3D) space. The user input device may then determine pose information associated with the user input device based on the images and motion of the user input device. The determination of the pose information may be via usage of at least one of a neural network model, estimation model trained on a set of unique and identifiable patterns, and/or an estimation model trained on a dataset of images.