By Bryan Kirschner, Vice President, Strategy at DataStax
Data scientists have long struggled with silos and cycle time. That’s partly because of an underlying structural tension between the traditional data science mission of turning “data into insights” versus the on-the-ground game of turning “context into action.”
The latter is something business teams and managers strive to do every day in real time. Consider a file share full of briefing documents and presentations, the team’s email inboxes, and even the moment in a sales presentation where the prospect seems to perk up. Whether the goal is making a sale, solving a customer problem, or preventing regretted attrition, they try to squeeze signals out of them in close to real time to improve their plan of attack.
Doing so requires straddling the boundaries of qualitative and quantitative judgment. It’s not a process that’s 100% reliable or replicable. But agility is prized, and wins are celebrated.
By comparison, the data playbook typically involves collecting a lot of data, ensuring that it’s clean and well-structured, and applying rigorous math to it. Proven reliability is expected–and once it’s achieved, algorithms can operate at machine speed and scale, delivering a lot of value.
But this makes the process much slower by comparison. There’s a constant risk of data science projects failing by (for example) arriving at an insight that managers already figured out by hook or by crook—or correctly finding an insight that isn’t a business priority.
GenAI: Harnessing unstructured data to improve workflows
Generative AI (genAI) offers an opportunity to square the circle and find new common cause and common ground. And some of the biggest challenges to making the most of it are well-suited to the skills and mindset of data scientists.
Consider this description of what a highly capable agentic genAI system might be able to do:
Based on hundreds of variables and patterns … the [genAI] agent is 92% confident this new customer-reported issue will have significant financial implications and is therefore reprioritizing all of dev team x’s work and a hot-fix deadline by EOD.
Today, this probably sounds like slightly alarming science fiction to most. But let’s break down a realistic journey toward a system like that:
First, one capable of raising a flag, sooner, to alert a human to investigate a problem or opportunity.
Second, one capable of making a recommendation, more quickly, to a human to evaluate.
Each step along the path would improve workflow and a potential leg up over competitors.
GenAI makes it possible for unstructured data to be handled at machine speed and scale to help move from context to action. It doesn’t mean traditional AI and data science-based assets should go away: agentic systems can be powered by both new signals from context and traditional ones, like propensity scores based on past purchases.
The case for a close partnership between data science and business
Just in terms of getting off the ground, data scientists bring the skills and mindset to help workflow owners “incorporate unstructured data sources into analyses, translate business problems into analytical models, and understand and explain models’ results.”
But genAI also means learning to build, operate, and work alongside non-deterministic systems. The data science practices of testing and iterating, experimenting, and diagnosing the interplay of “what data you’ve chosen to use and why” in the context of “what are acceptable boundary conditions for success and failure” are on point.
For workflow owners who want to aim high with genAI, one accomplished data scientist nails the case for close partnership:
[O]nce you’re doing enterprise scale automation with no human to check the output before it leaves your system, your biggest concern should be: does it work? In other words, does this massive system that’s automating your process at scale do so safely and effectively? And that’s where the data scientist shines! … Unlike traditional software systems, it’s not possible to “read the code” to figure out how well an automation solution is performing in your production environment, which is why you’ll need expert data scientists handling the process of understanding the efficacy and value of the AI solutions you’ll be relying on.
In the long term, that may have implications for formal job descriptions and organizational structures. But in the short term, the way past the challenges of silos and cycle time is to learn by doing. Match business workflow owners keen to accelerate context to action with data scientists keen to master working with genAI. Give them room to be agile, and document and celebrate both the wins and failures with learning.
It’s an opportunity to write a new playbook while stealing a march on organizations still sitting on the sidelines.
Learn how DataStax enables enterprises and developers to get GenAI apps to production fast.
About Bryan Kirschner:
Bryan is Vice President, Strategy at DataStax. For more than 20 years he has helped large organizations build and execute strategy when they are seeking new ways forward and a future materially different from their past. He specializes in removing fear, uncertainty, and doubt from strategic decision-making through empirical data and market sensing.
