Big Data Did Not Make Enterprises Wise

The Foundry

My role as CTO and Executive in Residence for Big Data and AI/ML Research sat between two worlds. On one side were researchers thinking deeply about graph analytics, machine learning, recommender systems, and pattern discovery. On the other were enterprises with real problems, budgets, legacy environments, business pressure, and expectations of value. The work was to build a bridge between them, and the Big Data Foundry became that bridge.

The word foundry mattered to me. A foundry is not a warehouse; it is a place where raw material is heated, shaped, tested, and turned into something usable. At PARC the raw material was enterprise data, research methods, and computational infrastructure, and the goal was an environment where data could be ingested, processed, modeled, analyzed, and turned into enterprise capability. This was not a toy lab, and that distinction mattered more than almost anything else.

Many research environments produce impressive prototypes under controlled conditions. Enterprise systems live in mess, where data is incomplete, schemas vary, quality is uneven, and security, performance, integration, and scale all matter at once. The customer does not want a demonstration. The customer wants something trustworthy enough to inform a decision, and the Foundry had to respect that from the start.

We built the environment from OpenStack, Kubernetes, OpenShift, SAP HANA, and Hadoop, each layer with a purpose. Hadoop gave us distributed storage and processing for large-scale data, SAP HANA brought high-performance in-memory analytics and enterprise relevance, and the cloud platform layers let us manage and deploy capabilities flexibly. The Foundry had to ingest hundreds of terabytes of enterprise data and still support experimentation, analytics, and production pathways. It was a demanding architecture for a demanding moment, and yet the architecture was never the point by itself. The point was what enterprises could understand from the data they were collecting.

This is where David’s method earned its place in the actual work. Standing in front of a customer’s data, the team’s instinct was to move quickly toward an answer, and David’s questions slowed that motion down. Before we decided what a dataset meant, we asked what we were truly observing as opposed to what we assumed. Before we trusted a pattern, we asked which beliefs we had carried into it and which perspectives were missing. The seven squares became a way to hold a messy enterprise problem still long enough to see its structure, which is precisely what a petabyte of data does not do on its own.

The market was full of big data excitement, and many companies were struggling to turn excitement into repeatable capability. They could stand up clusters, hire data scientists, run pilots, and produce dashboards, and still the gap between a pilot and a production capability stayed wide. I had seen the shape of this before. A factory planning system meant nothing until it changed how people planned, a cloud strategy meant nothing until it changed provisioning behavior, and a SaaS workflow meant nothing until people adopted it in real work. At PARC the same law held, since a model or analytic method was useful only when it could operate inside a customer’s business context.

Researchers tend to begin with what is possible, and enterprises begin with what is painful, and the bridge between the two is never automatic. A researcher may see a powerful graph algorithm while a business leader sees customer churn, promotion waste, supply-chain inefficiency, or missed revenue. The algorithm matters only when it reaches the pain, the platform matters only when it carries the algorithm into a usable environment, and the insight matters only when someone can act on it. That became the organizing discipline of the Foundry, and the question behind it was plain. Could we take advanced research and make it operational enough to solve a real enterprise problem?

The SAP partnership was central to that work. SAP brought enterprise gravity, an understanding of large customers, business processes, and the importance of trust, and SAP HANA gave us a serious platform for high-performance analytics. Around that foundation we explored graph intelligence, hypergraph recommender systems, and next-best-action capabilities. Those words can sound technical, but the business problem underneath was simple. Companies needed to understand relationships, not only records.

The enterprise world had spent decades building systems of record, and those systems were necessary, carrying customers, contracts, orders, invoices, inventory, and service history. By the early 2010s many companies were discovering that records alone were not enough, because the value was increasingly in relationships, patterns, context, and prediction. Graph thinking mattered because it matched the shape of real problems. Customers relate to products, households, regions, channels, campaigns, and behaviors, suppliers relate to parts, orders, delays, and risks, and financial entities relate to accounts, counterparties, and anomalies. Enterprises are not simple tables. They are networks of relationships.

That idea reached something old in me. I had been drawn to systems and relationships since childhood, in forms that looked nothing alike, soil and circuits and factories and transit systems and telecommunications and databases and supply chains, each teaching me that the relationships between parts often matter more than the parts themselves. At PARC that instinct found a technical expression in graph intelligence, and the Foundry was built to help the enterprise see the connections it could not otherwise see.

From data to decision

The point became sharpest in recommender systems and next-best-action thinking. A recommender at enterprise scale was not really about suggesting a product. It was a way of turning patterns into action, asking what should be offered, changed, prioritized, investigated, or done next given what we already knew.

That was a real shift, because a dashboard can tell an organization what happened while a next-best-action system asks what should happen now. To answer that, the system needed far more than data volume. It needed context, patterns, business logic, machine learning, feedback, and an understanding of where the recommendation would enter the workflow. A recommendation without a workflow is an orphan, statistically interesting and operationally useless.

The global brands we worked with kept the work honest. Companies like P&G, Unilever, Coca-Cola, PepsiCo, and Nestlé were not interested in technology for its own sake, and they cared about customer intelligence, promotion effectiveness, targeting, and measurable outcomes. A hypergraph recommender mattered to them only when its relationships improved a decision, and next-best-action thinking mattered only when the action fit a real process. That pressure did not cheapen the research. It honored it, because research becomes more powerful when it finds a pathway into use.

At PARC that pathway was never automatic. Researchers are rewarded for novelty, depth, and contribution, while enterprises reward usefulness, reliability, integration, and economic impact, and the two operate on different clocks. A model may be promising and hard to explain, or strong in evaluation and dependent on data the customer cannot provide, and a platform may be powerful and difficult to integrate into an environment full of legacy systems and governance requirements. The Foundry forced these tensions into the open, which is exactly why I valued it. It created a middle ground where researchers, engineers, enterprise partners, and customers could meet around real data, a place where possibility had to meet consequence.

The capability we were trying to create had many layers: data ingestion, storage, processing, model development, graph structures, recommender logic, in-memory analytics, experimentation, customer use cases, and commercial pathways. Each layer mattered, and the value emerged only when the layers connected. A brilliant model without usable data sits idle, a large dataset without context is noise, a platform without a use case is infrastructure waiting for meaning, and a recommendation without action is unfinished.

The Foundry existed to arrange these elements into something useful, and that turned big data, for me, from a slogan into a question of arrangement. How do data, context, models, platforms, business logic, workflow, and human judgment come together so the enterprise can act better? That question was still forming in me, and it would later become central.

At PARC I watched the enterprise begin to move from systems of record toward systems of insight. The database and application eras had digitized operations, and the big data era promised to interpret their traces. Companies no longer wanted only to store what happened. They wanted to understand why it happened, what might happen next, and what to do about it. The tools were immature, the talent scarce, the data messy, and the expectations sometimes unrealistic, and still the direction was unmistakable. Data was becoming a strategic material, analytics a competitive capability, and machine learning practical enough to matter, while enterprise platforms were being asked to move from memory toward sensemaking.

That word, sensemaking, grew more important to me through this period, and I knew where I had first learned to practice it. It was the name for what David had been teaching me by hand for years, the disciplined movement from a fragment to its meaning, from observation to interpretation to belief to choice. Enterprises had spent years collecting and managing data, and the next challenge was interpretation, converting signals into meaning in time to act. The Foundry let me see the same need at a scale no individual could meet alone with a notebook, and the enterprise needed David’s discipline built into a system. Data without sensemaking remains unfinished, and that became one of the deeper lessons of PARC.

A customer never simply wanted to know that we could ingest hundreds of terabytes. They wanted to know what that made possible, whether we could identify patterns, recommend actions, improve targeting, understand their customers, and produce measurable outcomes. The platform mattered because it made those questions answerable.

This was also where I began to see the limits of an isolated model. A model by itself is not a product, since it is part of a system that includes data pipelines, feature preparation, training, serving, interfaces, feedback loops, business rules, security, governance, monitoring, and adoption. If any of those pieces is weak, the model’s value is limited. Research often highlights the model, while enterprise adoption exposes the system.

The work also made me more careful about decision quality. More data did not create better decisions on its own. Better decisions required the right data, organized in the right context, interpreted through the right models, delivered into the right workflow, and trusted by the right people.

This is where the difference between data infrastructure and decision infrastructure began to form in my mind.

At PARC we were moving along that path, though not completely, and not yet with the later language. The direction was there.

The enterprise did not need more dashboards as the final answer. Dashboards, reports, and exploration each had a place, but the deeper need was a system that could connect fragmented data, hidden relationships, models, context, human judgment, and action. I could see the gap clearly. Enterprises were standing up clusters, hiring data scientists, and producing more visualizations, and yet when a consequential decision arrived, many of them fell back on fragmented workflows, manual interpretation, and institutional habit, with no coherent intelligence layer beneath the judgment. The problem was not a lack of signals. It was a lack of sense.

I did not leave PARC with a finished product in my hand, which would make the story too clean. I left with a sharper question, formed through the Foundry, the SAP partnership, graph analytics, recommender systems, the enterprise customers, and the hard translation between research and use. How does an enterprise turn fragmented data, hidden relationships, and human judgment into decisions it can trust? I also left with something quieter and more durable, a way of working that David had pressed into me, the refusal to let signal masquerade as sense.

That question and that discipline together became the soil from which Sensemaker and Quantiply would grow, and when I launched the company from PARC, David’s method went with me into it. In those same years the discipline was also taking a more deliberate form, a canvas I would come to call Clearstory, though its fuller origin belongs to a note of its own.

This note belongs to PARC. It belongs to the Foundry, to the researchers and engineers turning advanced ideas into enterprise capability, to datasets that were valuable only when understood, to graph intelligence showing that relationships matter, and to the recommender and next-best-action work that pushed analytics closer to decision.

It belongs, most of all, to the realization that big data was never the destination. PARC sharpened the question that would define the next chapter of my work. The enterprise did not need another dashboard. It needed a new layer for sensemaking, and I had already met the man who first taught me what that layer should do.