What happened to Big Data?
Remember? A few years ago, BIg Data was a big deal. Lots of folks offered courses and suggested riches would be theirs if they only learned the difference between SQL and NoSQL.
Enterprises spent months and tens of thousands of dollars with regional or national business technology consulting firms. The resulting findings seldom matched the quality of existing reports used by a high-performance business development group within the company.
So, what happened? Are companies not using SQL to reliably track transactions? Is anyone not using graph databases and social media data? Is scale really an issue? If the current range of AI solutions to problems we didn’t know existed is to gain a foothold, they must do better than the data cognoscenti has done with governance and master data concepts.
Variations of SaaS have been a normal part of information management for developers and other professionals. Slack is the standard secure communications channel. The flow and verbiage are strong clues to the culture of the participants. It speaks for the clarity of the problem to be solved, and the pace of problem resolution.
What Happened?
Just the name – Big Data – suggested current in-house systems could not/should not exist in such a scale. Ergo, Big Data is something that would live remotely. The C-suites agreed banking and mortgage and insurance and government were already operating securely and reliably in remote datacenters.
The grail has been sought for years by folks with five or more words in their titles. Real-time access to each action in the supply chain is often described as an event seen on the dashboard of relevant people. It is seldom achieved. They know of the event, though a corporate dashboard is not involved. Are dashboards at the corporate level used for long?
Big Data will manage information without a schema, it was said. Just drop PDFs and other data in standard formats into a lake or warehouse, and a variety of tools are useful to assess or view related information. A foundational model had not been shown. Loading a lake or warehouse with digital assets seems quaint when considering the years of ingestion and billions of documents in the foundational large language models.