How Is Mark Zuckerberg’s AI Butler Jarvis Working Out?

After introducing virtual butler Jarvis in a Morgan Freeman voice, Facebook founder and CEO Mark Zuckerberg is now seeking suggestions from his Facebook followers to make Jarvis more useful and make it do much more.


In a video post, which shows Mark, Priscilla, his child Max and dog Beast, at home in different situations, the co-founder showcases some use cases of Jarvis and wants to do more.

Jarvis, named after Marvel comics’ superhero Iron Man or Tony Stark, can respond to text and voice commands and it could run music, air conditioning, doors, and other systems. It could recognize visitors, start a toaster and even shoot t-shirts from a cannon in his closet.

The video also shows that Jarvis can make decisions like who to tickle next, in this case, it was Max. It can also shoot t-shirts through tubes and a lot more.

However, Zuck wants to do more. With more effort to broaden Jarvis’ use beyond Zuckerberg’s own house, the experiment “could be a great foundation to build a new product,” he wrote.

A dearth of internet-connected devices, lack of common standards for connected devices to communicate and challenges related to speech recognition and machine learning were all obstacles, he said. At the same time, he said challenges lead to eureka moments.

Adjustments made to help Jarvis recognize context in commands ultimately helped the system respond to less specific requests in a better fashion, such as asking the system to “play me some music”.

“I’ve found we use these more open-ended requests more frequently than more specific asks. No commercial products I know of do this today, and this seems like a big opportunity,” he wrote.

Creating Jarvis proved humanity is “both closer and farther off” from an AI breakthrough than we imagine, Zuckerberg wrote.

Computers are getting very good at pick out patterns, such as face recognition, but it is difficult to teach them new things, he wrote.

“Everything I did this year — natural language, face recognition, speech recognition and so on — are all variants of the same fundamental pattern recognition techniques,” he wrote. “But even if I spent 1,000 more hours, I probably wouldn’t be able to build a system that could learn completely new skills on its own.”

Meanwhile, Zuck has been sweet to his wife as well as he posted another clip on his timeline showing the harrowing time that his wife went through as he was building the virtual butler.