Facebook Deploys Robots to Save Blu-ray From Extinction
Facebook will start storing data on Blu-ray discs, the same discs that let you play high-def movies on your living room TV. Image: Kyle Owen
One day, your Facebook photos will sit in the hands of robots.
Behind the scenes at Mark Zuckerberg’s social networking giant, Facebook engineers have already built these robots, and one of them was on display last week in downtown San Jose, at a gathering of companies dedicated to exploring new technology inside the massive data centers that underpin the internet. This robot is little more than a mechanical arm — a device that moves up and down and side to side, grabbing things and carrying them from place to place — but it works entirely on its own, without human intervention, and it’s designed to grab things you’d never expect to find in the state-of-the-art computing centers operated by a company like Facebook.
The company’s social network — an online service now used by over one billion people — runs on custom-built computer servers packed with hard drives, DRAM memory, and the new breed of high-speed flash storage devices. But using its robotic arms, Facebook will store information on something very different: Blu-ray discs, the same discs that let you watch Stanley Kubrick’s 2001 on your living room TV in high-definition.
As those one billion users generate increasingly enormous amounts of data — an endless stream of status posts, likes, comments, photos, and videos — the team of engineers working under Facebook vice president of infrastructure engineering Jay Parikh are always looking for ways to more efficiently store all this digital information. With tongue in cheek, they call themselves “Jay’s commandos,” and their latest project is to move some of the older Facebook data onto Blu-ray, which is significantly cheaper than hard disks and flash and better suited to longterm storage. The robotic arms will help move the discs to and from the drives where the data is written and retrieved. Yes, this is a rather slow process compared to storing stuff on hard disk or flash, but it will only be used with data that rarely gets accessed, like very old photos.
Facebook calls it cold storage, and it’s yet another way that the giants of the web are pushing data center design into the future, seeking more efficient ways of processing and, yes, storing your data. It could also signal a larger movement towards the use of robotics in the data center, but at the same time, Facebook is venturing into the past with this rather surprising project. Just when the Blu-ray seemed to be on the way out — undercut by Netflix and other services that stream movies over the internet — another part of the internet has thrown it a lifeline.
“We went to Japan to understand the industry, and we were just — embraced,” says Giovanni Coglitore, the Facebook director of engineering who dreamed up this cold storage project. “I don’t want to call us a savior, but there are a lot of companies who see us as the new opportunity for optical discs.”
The Man Who Saved Google From the Fire Marshal
Though rarely written about in the press — even the hardware trade press — Giovanni Coglitore is one of the key figures in the rapid evolution of data center hardware over the last decade. As the co-founder of server maker Rackable Systems, he helped build the seminal computing centers fashioned by Google at the turn of the millennium, and at Facebook, he helps the company not only explore an even newer type of hardware, but share its designs with the rest of the computing world through the Open Compute Project, a Facebook creation that’s rapidly changing the worldwide hardware market.
Even in its earliest days, Google wasn’t one to buy commercial servers from big-name hardware makers like Dell and HP. Larry Page and company ran their search engine on machines pieced together from dirt-cheap parts and motherboards placed onto baking trays — literally, baking trays — that could be quickly slotted into racks. Some called them “bread rack” servers. The problem is that these baking trays were lined with cork board, and at one point, the fire marshal told the company that hot motherboards on cork wasn’t a good idea. So, Google called in Coglitore and his partner to help them build some bread-rack servers that met the fire code. “We built out the first two years of Google,” he says.
At the time, Coglitore’s hardware business — eventually renamed Rackable — was a tiny operation, but it grew into a major supplier as countless other companies followed Google’s lead into the world of low-cost custom servers, including Amazon, Yahoo, and Hotmail, the online email provider now owned by Microsoft. The new breed of web startup, you see, needed a cheaper type of system to run their ever growing services, and Rackable helped deliver it. “They would say: ‘Just build this, Gio,’” he remembers. “And we would do it.”
Taking this trend even further, Google soon started designing its own servers and contracting with manufacturers in Asia to build them, and as Rackable was acquired by big-name hardware maker SGI, it moved away from the sort of streamlined systems it helped pioneer. But Coglitore is now back in the game with Facebook.
Facebook’s Blu-ray rack. Image: Kyle Owen
Like Google, Facebook designs its own gear in tandem with manufacturers in Asia, but unlike its rival, it open sources these designs in an effort to encourage collaboration with other web players and drive down costs even further. Coglitore is one of the chief minds in the middle of this effort. He helps designs not only servers, but all sorts of other gear, including entirely new ways of plugging servers into the network. “He has this unbelievable knack for connecting seemingly unrelated dots,” says Frank Frankovsky, the Facebook hardware guru who helps oversee the Open Compute Project. “I’m just in awe when he brings up this stuff.”
The cold storage project is a prime example. As the company explored various ways to storing older data on the cheap, Coglitore suggested optical disks. It’s a counterintuitive suggestion. But Blu-rays are actually more reliable and last longer than hard disks, and they’re significantly cheaper. Yes, as anyone who bought CDs in the 80s and 90s will tell you, they scratch easily. But if you take out the human element, that’s not an issue, and through discussions with suppliers in Japan and other parts of Asia, Coglitore eventually found a cost-effective way of juggling Blu-rays with robots. “We’re really just repackaging old technologies,” he says. “But I think we’re looking at a whole era of evolution for optical discs.”
Why Robots Are the Future of the Data Center
Coglitore likens the project to the sort of thing he would do while working in the local hardware store in high school. When a customer walked in with a problem, he would grab stuff from across the store and piece together a solution. In solving Facebook’s cold storage problem, he paired the Blu-ray discs and drives with a company called HIT Archive Corp. This Asian manufacturer has long offered a small robotic arm that helps store data on CDs inside hospitals and small offices. Working with the company, Coglitore and crew refashioned this arm for use inside a data center rack. “We just scaled a robot that had been around for decades,” he says. “We made the arm longer.”
The company’s new cold storage racks hold about 10,000 Blu-ray discs, dividing them among dozens small trays. With each disc holding up to 50GB of data, you can stuff about a petabyte, aka about a million gigabytes, into a single rack. Each tray holds a handful of discs, and the robotic arm is programmed to find the right disk, lift it from the tray, and move to a drive where data can be written and read. It’s like those CD jukeboxes that turned up in high-end stereos and cars in the late 90s — only much bigger.
Facebook yet to install these racks inside its data centers, but it soon will. It has already built a separate cold storage facility beside its data center on the high desert in Prineville, Oregon, and this where the company will first make the move to Blu-ray. When Facebook initially discussed the facility last year, Jay Parikh indicated that the company could use Blu-ray to store older photos. But at least initially, it will use the discs to store rarely used data that it’s legally required to keep. “With legal stuff, you don’t care about retrieval time,” Colitore says. “You just want to have it there when someones asks for it.”
Some question whether this type of cold storage will ever make sense for photos. On Facebook, even old photos get accessed from time to time, and if they’re stored on Blu-ray, you’ll have to wait several minutes for the robots to retrieve them. “With today’s web services, some of the ‘cold storage’ gets activated pretty often,” says Mike Yang, the vice president and general manager at Quanta, one of the Asian manufacturer helping to drive the new hardware revolution inside the data center. “Is it really practical to build someone just for cold storage?”
But Coglitore believes this sort of cold storage is a real possibility not only for photos, but all sorts of older Facebook data. Facebook will use a variety of cold storage methods, and the trick will be to pair the right method with the right data. When you consider just how must data Facebookers generate on a daily basis — and how little of it still gets looked it — it only stands to reason that Blu-ray will play a major role. “The amount of high performance data is growing, but the amount of tepid to cold data is growing like crazy,” says Frankovsky. “That’s where the real opportunity is.”
But Facebook’s robotic arms are also a sign of even bigger things to come. Just as robots have begun to remake the distribution centers that drive Amazon’s retail empire, they will surely reinvent the data center as well. Ken Patchett, who oversees Facebook’s Prineville data center, has long seen robotics as the future of the data center, and Giovanni Coglitore, the man dreamt up the company’s Blu-ray robots, agrees. “I think our friends in Mountain View have shown that this is their desire,” he says, nodding to Google, which is buying up robotics companies left and right. “I would be foolish to say this isn’t inevitable. Everyone wants to increase efficiency and automation. It would be silly not to.”
Image: Kyle Owen