A robotic system is configured to automatically identify surgical instruments used during a bronchoscopy procedure. The robotic system can include a video capture device, a robotic manipulator, sensors configured to detect a configuration of the robotic manipulator, and control circuitry communicatively coupled to the robotic manipulator. The control circuitry is configured to perform, using a machine learning classifier, a first analysis of a bronchoscopy video of a patient site to track a medical instrument in the bronchoscopy video. The control circuitry can then identify a set of possible instrument identifications for the medical instrument in the bronchoscopy video based on the first analysis and an identified phase of the bronchoscopy procedure. The control circuitry can then track a motion of the medical instrument in the bronchoscopy video and select an identification from the set of possible instrument identification for the medical instrument based at least on the tracked motion.