Robots can now learn to perform surgery just by watching videos

For decades, surgeons have spent hours learning and practicing to master sophisticated surgical skills and painstakingly perfecting each movement to ensure patient safety. But now, robots with advanced artificial intelligence (AI) technology are learning to perform surgical tasks just by watching videos, opening up a promising era of automated surgery.

Researchers at Johns Hopkins University (JHU) and Stanford University have made a breakthrough in the field of robotic surgery. They taught a robotic system called the 'da Vinci Surgical System' to perform a variety of surgical tasks by training it via video. The da Vinci is an advanced robotic system that is often remotely controlled by doctors to perform precise operations such as dissection, bloodletting, cutting, and suturing. Such systems typically cost more than $2 million, and that does not include the cost of accessories, sterilization equipment, or training.

However, the JHU study has shown the potential of imitation learning , a new machine learning method that allows robotic systems to learn how to perform surgical tasks by watching videos of sample surgeries.

With the help of an AI model trained on surgical videos, a robotic system has successfully performed difficult surgical tasks as skillfully as humans.

Using a learning approach, the team helped the da Vinci Surgical System learn to perform three typical surgical tasks: needle manipulation, tissue lifting, and suturing. Not only did the robot perform these tasks with high precision, it also had the ability to self-correct errors that arose during the process.

Axel Krieger, an assistant professor at JHU and co-author of the study, said the system was able to automatically pick up a needle when it accidentally dropped it, a task the team had not programmed for. This ability marks a major advance, because previously, robotic systems often required human intervention when they made mistakes.

The da Vinci surgical system at work.

To control the da Vinci Surgical System , the team applied an advanced AI model similar to what is used in chatbots like ChatGPT. However, instead of processing text, the robotic system processes kinematic information – a specialized language for describing motion. With complex mathematical elements such as numbers and equations, this language helps simulate and control the precise movements of the robotic arm during surgical operations.

The system was trained using hundreds of videos of surgeries recorded from the perspective of a camera mounted on a robotic arm. Through imitation learning, the robot can learn to replicate the surgeon's actions and even develop the ability to self-correct when it makes mistakes. This ability helps reduce errors and improve surgical accuracy.

Krieger and his team hope that this approach could open up the possibility of training robots to perform any type of surgery much more quickly and efficiently than traditional step-by-step hand coding. Instead of spending years programming each action, AI systems can now learn quickly from video data. This could help realize the goal of automated surgery much sooner than expected.

'What's new here is that we can simply collect data from different procedures, and we can train the robot to learn within a few days, ' said Krieger . 'This allows us to move towards our goal of automating surgery, helping to reduce medical errors and improve accuracy.'

The robotic surgical system will be accompanied by a human surgeon to guide it step by step, allowing it to operate with greater precision.

Despite these remarkable achievements, automated surgery still faces many challenges. Currently, some advanced surgical robotic systems such as Corindus' CorPath system are used in complex cardiovascular procedures. However, the capabilities of these systems are still limited in certain stages of the surgical procedure.

Coding each step of a surgical procedure for a robotic system is a complex and time-consuming process, according to Krieger . 'It could take a human a decade to model the suturing process for a single type of surgery ,' he said. That's why developing a system that can learn by observation is opening up a whole new direction.

In 2022, Krieger's team developed an autonomous robotic system called the Smart Tissue Autonomous Robot (STAR), which can sew complex pig intestinal tissues without human intervention. STAR uses a 3D camera and machine learning tracking algorithms to complete complex surgical tasks.

Researchers at JHU continue to develop robots to perform comprehensive surgical procedures. While fully automated surgery is still a long way off, these advances will contribute to increased surgical safety and precision. With advances in AI and machine learning, complex surgeries could become more accessible to more patients around the world.

However, this also raises questions about the role of surgeons in the future. Can robots completely replace humans on the operating table? Krieger believes that humans and robots will coexist and support each other in medicine. Robotic surgery is not about replacing humans, but about helping and supporting doctors to perform difficult tasks more accurately and quickly.