Building the Great Pyramid at Giza required millions of stone blocks, which had to be carved and shipped to the site. It involved thousands of workers, each of whom required a daily ration of food, which of course had to be grown, harvested and delivered on schedule.
All of this had to be synchronized, otherwise the project would stall. In short, it required a system—a system tracked, 4,500 years ago, with ink on papyrus. Yes, the ancient pyramids relied not only on labor and raw materials, but on data collection and analysis.
Data collection is what we do
Today, we think of Big Data as a modern concept. Cloud storage, text mining and social network analytics are vital 21st century tools. But the need for data is a thread that stretches back millennia, from today’s CEO looking for business analytics to the managers trying to get a 481-foot-tall pyramid built in the ancient desert.
Tens of thousands of years ago, notches on sticks were used to record trading activity or figure out how long supplies would last. Over time, records became more voluminous, more detailed. Around 300 B.C.E., King Ptolemy I Soter set about creating the largest collection of data (then) known to man, an institution known as the Library of Alexandria.
Libraries had been around for centuries, but this one would be different—filled with written records in numerous languages about the world’s knowledge. Religion, science, the arts, law, medicine, history, literature and more were represented here. Estimates of the collection range from 42,000 to 700,000. Here, scholars could immerse themselves in a bottomless sea of knowledge.
Then we started analyzing
But data collection alone wasn’t enough. By analyzing data, we began to solve problems.
- In the 1600s, a businessman named John Graunt tabulated London death records and noticed patterns. The Black Death, he concluded, had killed more people during this period than any other cause. By plotting the deaths over time, he also realized something else: looking at population data, you could disprove some of the era’s “scientific” theories. Contrary to popular opinion, he concluded, the plague wasn’t caused by a particular alignment of the planets.
- Florence Nightingale continued in that vein in the 19th century. Many of us know “The Lady with the Lamp” as a nurse tending soldiers during the Crimean War. But her larger contribution was in statistics. Collecting data at military hospitals, she realized that infectious disease and poor sanitation were killing more soldiers than war injuries. Just as important, she devised a way to display the data visually, so it was easy-to-understand at a glance. As a result, military medical care was reformed.
The power of analyzing data fueled data collection. Over time, data accumulated and started to become unwieldy, even useless.
How our ancestors tried to solve the data collection dilemma
After collecting census records for about 100 years, the U.S. government realized the records were so voluminous it would take a decade to analyze the results—and by then, it was time for a new census.
An engineer named Herman Hollerith came up with a mechanized system, using punched cards and a tabulating machine, that could tally the results in months, not years. That invention turned languishing data into useful data. Hollerith went on to found the company that became IBM, another giant in the world of data science.
The dilemma of how to best store and analyze data, however, continued.
In the 20th century, an ever-evolving set of storage systems and analytical tools were developed to deal with this problem. In the 1920s, an engineer figured out a way to store sound on magnetic tape. By the 1950s, IBM came up with a (then) high-speed system that could store two million digits on a single magnetic tape. By the 1970s, Intel developed a 1KB memory chip. In the 1980s, Fujio Masuoka invented flash memory.
Over time we went from floppy disks to hard disks to the cloud. Meanwhile, retrieval that once required a trained expert could now be done by a schoolchild on a tablet.
As Tesla predicted, then came the smartphone
By the 21st century, Nikola Tesla’s famous 1926 prediction about the “new art of applied electricity” had come true.
“We shall be able to communicate with one another instantly, irrespective of distance…and the instruments through which we shall be able to do this will be amazingly simple compared with our present telephone. A man will be able to carry one in his vest pocket.”
Considering about half the world has a smartphone, Tesla pretty much nailed it. But his larger point, about how these new technologies connect us, is just as profound.
How United by Data™ connects us
Data is a thread that unites past with present, nation with nation, individual with community. It connects a vaccine manufacturer in India with researchers in the U.S. and epidemiologists in South Africa. It connects employees to teams and teams to solutions.
Today, researchers and policy makers are looking to data to help solve some of the world’s biggest problems. Data researchers are identifying patterns of racial bias, looking for tools to fight malaria, detecting ever-tinier evidence of pathogens in our food supply. One day, data could help identify a cure for intractable diseases such as ALS. Data is informing policies seeking to address inequality, poverty and childhood mortality.
But to do any of that, data needs to be efficiently stored, easily retrieved, and 100% reliable.