OpenAI is also apparently boosting its own robots in the field of robotics. Last week, Caitlin Kalinowski, who previously led the development of virtual and augmented reality headsets at Meta, announced on LinkedIn that she is joining OpenAI to work on hardware, including robotics.
Lachy Groom, a friend of OpenAI CEO Sam Altman and an investor and co-founder of Physical Intelligence, joins the team in the conference room to discuss the business side of the plan. The groom wears an expensive looking hood and looks remarkably young. He emphasizes that Physical Intelligence has many opportunities to pursue a breakthrough in robot learning. “I just got off the phone with Kushner,” he says of Joshua Kushner, founder and managing partner of Thrive Capital, who led the startup’s seed investment round. He is also, of course, the brother of Donald Trump’s son-in-law Jared Kushner.
Several other companies are now pursuing the same kind of breakthrough. One, called Skild, founded by roboticists from Carnegie Mellon University, raised $300 million in July. “Just as OpenAI built ChatGPT for language, we’re building a general-purpose brain for robots,” says Deepak Pathak, CEO of Skild and assistant professor at CMU.
Not everyone is sure that this can be achieved in the same way that OpenAI cracked the AI language code.
There is simply no Internet-scale repository of robot actions similar to the text and image data available for LLM training. Achieving a breakthrough in physical intelligence may require exponentially more data anyway.
“Words in a sequence are, dimensionally, a tiny little toy compared to all the movement and activity of objects in the physical world,” says Illah Nurbakhsh, a roboticist at CMU who does not work with Skild. “The degrees of freedom we have in the physical world are much more than just letters in the alphabet.”
Ken Goldberg, an academic at the University of California at Berkeley who works on applying AI to robots, warns that the excitement building around the idea of a revolution of data-powered robots as well as humanoids is reaching hype-like proportions. “To achieve the expected performance levels, we will need ‘good old engineering’, modularity, algorithms and metrics,” he says.
Russ Tedrake, a computer scientist at MIT and vice president of robotics research at the Toyota Research Institute, says the success of the LLMs has led many roboticists, including himself, to rethink their research priorities and focus on finding ways to pursue robotic learning on a more ambitious scale. But he admits huge challenges remain.
