The Big Data Foundry: A Petabyte-Scale Algorithm Machine

Last week, I had the opportunity to participate in a fireside conversation at the OpenStack Enterprise Forum, moderated by Lydia Leong, Vice President and Distinguished Analyst at Gartner. SiliconANGLE’s theCUBE captured the conversation in an article titled “[OpenStack: a candid conversation](https://siliconangle.com/2014/01/31/openstack-a-candid-conversation-oeforum/),” which highlighted some of our experience at PARC using OpenStack as the infrastructure foundation for big data research and open innovation.

At PARC, a Xerox company, our work is centered on creating breakthrough technologies and translating research into real-world impact. We operate at the intersection of physical systems, data, algorithms, and enterprise problems. Increasingly, that work requires an infrastructure model that is flexible, scalable, and accessible to researchers who are focused on discovery rather than infrastructure management.

That is the context in which we adopted OpenStack.

For us, OpenStack is not simply a private cloud platform. It is becoming an enabling layer for data-centric research. Our researchers work with large volumes of sensor data and apply parallel algorithms to extract meaning, patterns, and insight. The infrastructure must support that rhythm. It must allow researchers to provision resources quickly, run experiments, and move on without waiting for long deployment cycles or depending heavily on a large operations team.

One of the important points I made during the discussion is that researchers are not necessarily programmers or DevOps engineers. They should not have to become infrastructure specialists in order to do their work. We needed an environment where they could begin using compute resources with minimal programming effort around the automation layer. That was one of the reasons OpenStack became attractive to us. (SiliconANGLE)

The practical benefit is agility.

With a small support team, we need to give researchers tools that allow them to focus on their primary area of research. OpenStack helps us create lightweight environments that can be deployed quickly, used for experimentation, and removed when the work is complete. This is especially valuable in big data research, where different experiments may require different compute configurations, storage needs, and analytical pipelines.

One example we discussed was our ability to deploy what we call a “disposable Hadoop cluster.” With OpenStack, we have built a framework that allows us to launch a large Hadoop cluster, run experiments, and then tear it down after the work is done. SiliconANGLE captured this as one of the key examples from the conversation. (SiliconANGLE)

This capability changes the economics and pace of experimentation. Instead of treating infrastructure as a fixed asset that must be manually configured and preserved, we can treat it as an elastic research utility. That matters because modern research is increasingly iterative. The faster we can move from question to experiment to result, the faster we can learn.

But the conversation was also candid about the challenges.

OpenStack is powerful, but it is still maturing. At PARC, we are running the Folsom release, even though the community is already moving toward newer releases. The reason is straightforward: upgrades are difficult. When we moved from Essex to Folsom, the transition required significant effort. For an enterprise or research organization, that kind of upgrade friction creates a real decision point. If the current release supports most of the workload, there must be a compelling reason to take on the risk and effort of a major upgrade. (SiliconANGLE)

This is why I emphasized the need for what I called a “zero loss upgrade.” For OpenStack to become more widely adopted in enterprise environments, upgrades must become seamless. Organizations need to move forward without redoing complex deployments, disrupting users, or risking the stability of existing workloads. (SiliconANGLE)

That is not a criticism of OpenStack’s direction. It is a practical requirement for enterprise maturity.

In research environments like PARC, infrastructure must serve innovation, not slow it down. We cannot afford to rebuild everything each time the platform advances. The more seamless the upgrade path becomes, the easier it will be for organizations like ours to stay current and take advantage of new capabilities.

Despite these challenges, I remain very encouraged by what OpenStack enables.

Our next step is to expand this infrastructure beyond individual research environments and make it part of a broader open innovation model. We want to make these capabilities available across labs and, over time, involve customers and hardware partners in a more collaborative data-centric research ecosystem. The vision is to allow customers to connect sensors and devices into the network, generate data, and work with PARC researchers to analyze that data and create new forms of value. (SiliconANGLE)

This is where OpenStack becomes more than cloud infrastructure.

It becomes a platform for democratizing algorithmic research. It allows researchers, customers, and partners to participate in a shared environment where data, devices, compute, and expertise can come together. For PARC, that is a natural extension of our open innovation practice.

The broader lesson is simple.

The real question is not whether an organization has adopted cloud. The real question is whether cloud helps people experiment faster, collaborate more effectively, and turn data into insight. For research organizations, that is the difference between infrastructure as a cost center and infrastructure as an innovation platform.

OpenStack is helping us move toward that future.

It is not perfect. The ecosystem still needs to improve upgradeability, operational simplicity, and enterprise readiness. But the direction is promising. For organizations working with big data, sensor systems, and algorithmic research, OpenStack offers a practical path toward flexible, on-demand research infrastructure.

At PARC, our goal is to use that infrastructure to make data-centric innovation more accessible — first to our researchers, then to our broader labs, and ultimately to customers and partners who want to turn real-world data into meaningful insight.

That is why this moment matters. OpenStack is not just changing how we deploy infrastructure. It is beginning to change who can participate in large-scale, data-driven research.

Watch the embedded video

Source; <http://siliconangle.com/blog/2014/01/31/openstack-a-candid-conversation-oeforum/> photo credit: <http://www.flickr.com/photos/mamchenkov/363542398/>