Every enterprise can get the same models. Agents are surging into production. And in every company racing to keep up, one person decides whether that AI can be trusted: the person who owns its data, its meaning, and its permissions. That's you. This is the moment your work stops being back office and becomes the backbone of your company's entire AI program. Join Collibra CEO and co-founder Felix Van de Maele for a keynote on what to do today, this quarter, and this year to put yourself at the center of enterprise AI, and get a first look at the breakthrough capabilities that make governed context and control the foundation everything else builds on.
Watch on demand: Collibra AI Summit 2026
Collibra's inaugural AI Summit came to life in New York City and London. Discover how AI is impacting organizations, how a solid data foundation will make your AI journey more successful and what it takes to build agents you can trust.
- Video: Ending the hallucination tax: Governing the backbone of enterprise AI

Transcript
- 00:00:05,700 --> 00:00:06,620: It's fantastic to be here.
- 00:00:07,060 --> 00:00:08,640: I see we have a full room. We're incredibly
- 00:00:09,040 --> 00:00:10,040: appreciative of all you
- 00:00:10,400 --> 00:00:12,220: making the time to spend the day,
- 00:00:12,740 --> 00:00:14,840: uh, with us. Uh, we think it's so powerful.
- 00:00:15,240 --> 00:00:17,360: We'll talk about technology, we'll talk about products, we'll talk
- 00:00:17,420 --> 00:00:18,060: about AI, but
- 00:00:18,380 --> 00:00:19,140: the community
- 00:00:19,540 --> 00:00:21,580: we're bringing together of people doing this work
- 00:00:21,680 --> 00:00:22,560: every day with
- 00:00:22,940 --> 00:00:25,020: so much passion, I think is the most powerful,
- 00:00:25,500 --> 00:00:26,540: uh, opportunity of,
- 00:00:27,020 --> 00:00:29,080: uh, of today. So please take the time to
- 00:00:29,120 --> 00:00:31,320: connect, network, share experiences, share
- 00:00:31,380 --> 00:00:32,259: wins and
- 00:00:32,600 --> 00:00:33,140: challenges.
- 00:00:33,700 --> 00:00:35,320: Now, you heard Chirag say we
- 00:00:35,680 --> 00:00:37,600: used to call this the Data Citizens
- 00:00:37,920 --> 00:00:39,040: Conference. Now we call it the
- 00:00:39,420 --> 00:00:40,760: AI Summit. I think it's a
- 00:00:41,160 --> 00:00:42,180: sign of the times, but
- 00:00:42,560 --> 00:00:44,280: I wanna start with the tagline of
- 00:00:44,640 --> 00:00:45,800: what we call the AI Summit:
- 00:00:46,280 --> 00:00:46,840: Certainty
- 00:00:47,140 --> 00:00:48,220: at the speed of AI.
- 00:00:49,000 --> 00:00:50,040: And that's inherently
- 00:00:50,440 --> 00:00:52,460: a contradiction because we know that GenAI
- 00:00:52,580 --> 00:00:53,480: is by definition
- 00:00:53,960 --> 00:00:54,840: non-deterministic,
- 00:00:55,260 --> 00:00:56,580: but also shows why
- 00:00:57,160 --> 00:00:59,480: AI, and especially enterprise AI,
- 00:01:00,080 --> 00:01:00,840: isn't easy,
- 00:01:01,260 --> 00:01:02,200: right? And this is
- 00:01:02,600 --> 00:01:04,599: this challenge we gave the name. We call it the
- 00:01:04,900 --> 00:01:06,520: hallucination text. You probably,
- 00:01:06,900 --> 00:01:09,240: uh, saw it. And so where I wanna start
- 00:01:09,280 --> 00:01:09,800: today is
- 00:01:10,240 --> 00:01:12,280: talk about this hallucination start, talk about some of
- 00:01:12,320 --> 00:01:14,380: the real challenges that we are seeing in the
- 00:01:14,420 --> 00:01:16,440: market with customers of how to make enterprise
- 00:01:16,500 --> 00:01:17,940: AI a reality.
- 00:01:18,460 --> 00:01:20,580: What is missing? What are the gaps that we
- 00:01:20,600 --> 00:01:22,680: need to fill to make this a reality? And then how do we
- 00:01:23,080 --> 00:01:24,160: help you get there?
- 00:01:24,900 --> 00:01:26,160: Now, I'm gonna start with
- 00:01:26,760 --> 00:01:29,160: asking a few very simple questions.
- 00:01:29,420 --> 00:01:30,520: Just set the stage.
- 00:01:31,100 --> 00:01:32,260: First, um,
- 00:01:33,380 --> 00:01:34,800: how many agents
- 00:01:35,140 --> 00:01:35,760: do you have
- 00:01:36,160 --> 00:01:37,160: in production?
- 00:01:38,020 --> 00:01:39,680: Who can confidently
- 00:01:40,220 --> 00:01:41,880: answer that very simple question,
- 00:01:42,200 --> 00:01:44,100: right? Is it one? Is it fifty? A hundred?
- 00:01:44,680 --> 00:01:45,840: Two hundred seventy-five?
- 00:01:46,740 --> 00:01:47,780: I'm not sure, right?
- 00:01:48,460 --> 00:01:50,460: Very simple question, but I don't think many of us
- 00:01:50,900 --> 00:01:52,860: would confidently be able to answer that question.
- 00:01:53,840 --> 00:01:54,580: S-second question:
- 00:01:55,400 --> 00:01:56,780: What are those agents
- 00:01:57,300 --> 00:01:58,880: doing? What are they
- 00:01:59,280 --> 00:02:01,060: really doing? Do we know
- 00:02:01,640 --> 00:02:02,200: exactly
- 00:02:02,480 --> 00:02:04,460: what they're doing? Do we know exactly what they're
- 00:02:04,980 --> 00:02:05,940: supposed to be doing?
- 00:02:06,640 --> 00:02:07,540: Do we know what they're
- 00:02:07,820 --> 00:02:09,139: not supposed to be doing?
- 00:02:10,039 --> 00:02:12,060: I think if we're not fully sure of how many we
- 00:02:12,100 --> 00:02:13,880: have, it's probably unlikely that we're able to
- 00:02:14,340 --> 00:02:15,220: answer this question
- 00:02:15,640 --> 00:02:16,480: with a lot of
- 00:02:16,780 --> 00:02:18,860: confidence. And then the
- 00:02:18,900 --> 00:02:19,480: third question:
- 00:02:20,700 --> 00:02:21,540: Are they doing it
- 00:02:21,940 --> 00:02:24,320: right? We all heard the promise
- 00:02:24,380 --> 00:02:26,500: of agentic AI, where agents are
- 00:02:26,580 --> 00:02:27,620: able to autonomously
- 00:02:28,040 --> 00:02:28,920: make decisions,
- 00:02:29,180 --> 00:02:31,100: automate tasks, run processes.
- 00:02:32,100 --> 00:02:33,220: But are they doing it
- 00:02:33,640 --> 00:02:34,940: right? How much
- 00:02:35,480 --> 00:02:36,120: confidence
- 00:02:36,420 --> 00:02:37,000: do we have
- 00:02:37,620 --> 00:02:39,660: that agents are doing it right? And if they're not
- 00:02:39,680 --> 00:02:40,860: doing it right, will we know?
- 00:02:41,560 --> 00:02:42,980: And do we know who's responsible,
- 00:02:43,560 --> 00:02:44,240: who's accountable,
- 00:02:44,960 --> 00:02:45,940: and who's gonna fix it?
- 00:02:47,200 --> 00:02:48,340: They're very simple
- 00:02:48,760 --> 00:02:49,420: questions that
- 00:02:49,960 --> 00:02:52,240: I would argue probably in our organizations
- 00:02:52,300 --> 00:02:54,560: we don't have the full answers to. And I would even argue
- 00:02:54,620 --> 00:02:55,160: if we do,
- 00:02:55,680 --> 00:02:57,860: that's probably because we don't have many in production in the first
- 00:02:57,900 --> 00:02:58,680: time, right?
- 00:02:59,300 --> 00:03:00,640: And so this is a good reflection of
- 00:03:01,080 --> 00:03:02,700: simple things that are actually
- 00:03:03,180 --> 00:03:04,040: quite, uh,
- 00:03:04,380 --> 00:03:05,620: complex. Now,
- 00:03:06,020 --> 00:03:08,220: if you do it for one agent, that might be okay, right?
- 00:03:08,260 --> 00:03:08,920: And if, if we're,
- 00:03:09,240 --> 00:03:11,300: we're building our own agent, I build my agent, you build
- 00:03:11,340 --> 00:03:12,860: your agent, we of course know
- 00:03:13,360 --> 00:03:14,500: what that agent is doing.
- 00:03:15,240 --> 00:03:16,960: But think of that at the scale
- 00:03:17,520 --> 00:03:19,640: and complexity of your organizations,
- 00:03:19,920 --> 00:03:21,960: with tens of thousands of employees, with
- 00:03:22,020 --> 00:03:24,480: so many different business processes, so many different systems,
- 00:03:24,980 --> 00:03:25,560: functions. We are
- 00:03:25,860 --> 00:03:26,600: all working in very
- 00:03:27,140 --> 00:03:28,740: complex environments.
- 00:03:29,580 --> 00:03:30,380: And I think what we're
- 00:03:30,720 --> 00:03:32,800: doing today is we're relying on people
- 00:03:33,540 --> 00:03:34,960: to have those answers, right?
- 00:03:35,520 --> 00:03:37,680: We're relying on the responsibility of people
- 00:03:37,760 --> 00:03:38,640: to do the right thing,
- 00:03:39,480 --> 00:03:41,580: which on the one hand is important. There's a level of accountability
- 00:03:41,620 --> 00:03:42,160: that comes with
- 00:03:42,680 --> 00:03:43,720: AI and, and agents,
- 00:03:44,260 --> 00:03:46,280: but it's also, as you know, not good enough. If
- 00:03:46,320 --> 00:03:46,840: you work in a
- 00:03:47,280 --> 00:03:48,980: complex environment, we cannot just
- 00:03:49,500 --> 00:03:50,720: say, "I'm sure
- 00:03:51,000 --> 00:03:52,500: my people are doing the right thing,"
- 00:03:52,760 --> 00:03:54,260: and that's how we're gonna manage AI.
- 00:03:54,600 --> 00:03:55,260: That's not good enough.
- 00:03:55,860 --> 00:03:56,940: If we have that level of
- 00:03:57,520 --> 00:03:58,260: uncertainty,
- 00:03:58,580 --> 00:03:59,200: what's gonna happen?
- 00:04:00,000 --> 00:04:01,240: What happens when you're not
- 00:04:01,560 --> 00:04:03,480: sure? You press the brakes,
- 00:04:04,120 --> 00:04:06,040: you wait, right? You go slower.
- 00:04:06,700 --> 00:04:07,800: Which is exactly what we
- 00:04:08,260 --> 00:04:09,920: don't want to have happen.
- 00:04:10,680 --> 00:04:12,240: And so especially as, as we start
- 00:04:12,860 --> 00:04:14,000: scaling AI from
- 00:04:14,340 --> 00:04:16,459: one to five to ten to a hundred agents,
- 00:04:16,959 --> 00:04:19,019: making sure that we find scalable ways
- 00:04:19,420 --> 00:04:20,540: to stay in control
- 00:04:21,000 --> 00:04:21,519: is going to be
- 00:04:21,820 --> 00:04:22,420: incredibly
- 00:04:22,980 --> 00:04:25,040: important. And this
- 00:04:25,100 --> 00:04:25,640: is why
- 00:04:26,160 --> 00:04:28,000: AI often feels like a drag,
- 00:04:28,300 --> 00:04:29,240: right? We all
- 00:04:29,520 --> 00:04:30,860: hear this promise that AI is gonna
- 00:04:31,220 --> 00:04:33,300: help us save so much time. Who feels that
- 00:04:33,360 --> 00:04:34,960: they're, they're doing less work today
- 00:04:35,440 --> 00:04:36,580: than they were two years ago?
- 00:04:37,580 --> 00:04:38,600: Probably very few of us,
- 00:04:39,560 --> 00:04:41,000: right? Because
- 00:04:41,380 --> 00:04:43,460: we can now have agents. We're building agents, and
- 00:04:43,520 --> 00:04:44,820: I think the simple analogy of
- 00:04:45,160 --> 00:04:46,700: an agent is as if you're hiring
- 00:04:47,180 --> 00:04:49,220: a really smart, highly educated
- 00:04:49,280 --> 00:04:51,620: team member that doesn't understand
- 00:04:51,660 --> 00:04:52,320: your business at all,
- 00:04:53,300 --> 00:04:55,400: right? And the CEO says we need more AI,
- 00:04:55,480 --> 00:04:57,620: so instead of hiring one person, you hire 10 people.
- 00:04:58,420 --> 00:05:00,480: Is your life, is your work going to be easier or
- 00:05:00,520 --> 00:05:01,420: harder? It's gonna be harder.
- 00:05:01,900 --> 00:05:03,760: You're gonna have to train them. You're gonna have to enable them.
- 00:05:04,020 --> 00:05:05,280: You're gonna have to make sure they do the right thing.
- 00:05:05,680 --> 00:05:07,720: You're gonna be accountable and responsible to make sure
- 00:05:07,760 --> 00:05:09,680: they do the right thing. So your job is gonna be harder,
- 00:05:10,060 --> 00:05:11,220: and this is very much how we're
- 00:05:11,780 --> 00:05:12,600: dealing with AI
- 00:05:12,920 --> 00:05:14,680: today, and this is why it feels harder.
- 00:05:15,500 --> 00:05:16,560: And this is what we have to
- 00:05:16,940 --> 00:05:18,820: fix. And this is the
- 00:05:19,240 --> 00:05:21,500: expectation, the gap that we are very much seeing in the market,
- 00:05:21,560 --> 00:05:23,740: right? The promise, the expectation of AI
- 00:05:23,860 --> 00:05:25,880: is that agents make no errors, and we--
- 00:05:26,200 --> 00:05:28,220: every week we hear about a new model that's gonna
- 00:05:28,240 --> 00:05:30,520: be even better, that scores better on benchmarks.
- 00:05:31,340 --> 00:05:32,120: I don't think it's the model,
- 00:05:32,720 --> 00:05:33,680: right? Now,
- 00:05:34,260 --> 00:05:35,580: humans are no longer needed,
- 00:05:36,000 --> 00:05:36,560: and there's a lot of,
- 00:05:36,840 --> 00:05:38,840: again, uncertainty and,
- 00:05:38,920 --> 00:05:41,000: and, and friction and noise in organizations
- 00:05:41,060 --> 00:05:43,080: with this promise of, like, what is gonna happen
- 00:05:43,100 --> 00:05:43,680: to our jobs.
- 00:05:44,400 --> 00:05:46,700: And AI is gonna be able to reduce costs
- 00:05:47,080 --> 00:05:47,940: because it's so much more
- 00:05:48,380 --> 00:05:49,980: efficient and effective and lower cost.
- 00:05:50,540 --> 00:05:51,780: Now, you all know this is not
- 00:05:52,100 --> 00:05:52,680: the reality.
- 00:05:53,200 --> 00:05:54,580: The reality is that agents
- 00:05:55,040 --> 00:05:56,520: continue to make mistakes.
- 00:05:56,780 --> 00:05:57,960: They hallucinate, they guess,
- 00:05:58,320 --> 00:05:58,860: they're wrong.
- 00:05:59,580 --> 00:06:02,020: Humans, we continuously have to intervene
- 00:06:02,060 --> 00:06:03,240: to make sure they do the thing
- 00:06:03,760 --> 00:06:05,020: in the right way, and clearly,
- 00:06:05,480 --> 00:06:06,520: we've all unfortunately
- 00:06:07,000 --> 00:06:09,200: realized this year that AI is far from free.
- 00:06:10,030 --> 00:06:11,030: Can be very expensive.
- 00:06:11,350 --> 00:06:12,030: Eighteen months ago,
- 00:06:12,530 --> 00:06:14,670: very few of our conversation was about ROI
- 00:06:14,710 --> 00:06:16,850: and costs. Today, that's very much
- 00:06:16,910 --> 00:06:17,870: a top-of-mind item.
- 00:06:18,370 --> 00:06:20,530: How do we make sure AI cost is not
- 00:06:20,810 --> 00:06:21,670: going rampant?
- 00:06:22,890 --> 00:06:23,650: And so this is this
- 00:06:23,970 --> 00:06:25,990: gap, right? This gap between expectation
- 00:06:26,490 --> 00:06:27,950: and reality is what we call
- 00:06:28,370 --> 00:06:29,230: the hallucination
- 00:06:29,610 --> 00:06:31,850: tax, where agents were supposed
- 00:06:31,890 --> 00:06:32,510: to get cheaper
- 00:06:33,150 --> 00:06:35,210: and better and reliable, and the opposite is
- 00:06:35,250 --> 00:06:37,330: happening. So why
- 00:06:38,070 --> 00:06:38,970: is that, right?
- 00:06:39,470 --> 00:06:40,930: And so this is not just what we've
- 00:06:41,410 --> 00:06:43,330: seen ourselves, we've also done a survey,
- 00:06:44,070 --> 00:06:45,610: um, where we've asked
- 00:06:45,870 --> 00:06:47,870: leaders, over three hundred kind of enterprise leaders in
- 00:06:47,910 --> 00:06:48,770: large organizations of:
- 00:06:49,170 --> 00:06:50,490: What are your experiences
- 00:06:50,750 --> 00:06:51,530: with AI? What are you
- 00:06:52,130 --> 00:06:53,570: seeing? And so
- 00:06:54,050 --> 00:06:56,330: no surprises, sixty-four percent of respondents
- 00:06:56,370 --> 00:06:57,230: say that their team
- 00:06:57,690 --> 00:06:59,010: is spending significant
- 00:06:59,330 --> 00:07:00,610: time manually
- 00:07:01,090 --> 00:07:02,350: managing, reviewing
- 00:07:02,930 --> 00:07:03,610: agent outputs.
- 00:07:04,530 --> 00:07:06,550: Clearly, manual effort is still
- 00:07:07,130 --> 00:07:07,670: absolutely
- 00:07:07,930 --> 00:07:08,470: necessary.
- 00:07:09,490 --> 00:07:10,410: Our team is spending
- 00:07:11,050 --> 00:07:12,670: much time creating
- 00:07:13,010 --> 00:07:14,690: and verifying agent context.
- 00:07:15,169 --> 00:07:17,510: And we'll talk more about context, one of the most overused
- 00:07:17,570 --> 00:07:18,430: words nowadays, but
- 00:07:18,930 --> 00:07:21,170: really important because this is how we instruct
- 00:07:21,350 --> 00:07:22,690: agents what to do.
- 00:07:23,030 --> 00:07:23,890: Incredibly important.
- 00:07:24,690 --> 00:07:26,710: And it's not just about doing it once, it's about how do
- 00:07:26,730 --> 00:07:28,170: we make sure we continuously
- 00:07:28,710 --> 00:07:30,750: verify that our agents are doing the right thing.
- 00:07:31,570 --> 00:07:33,590: And we have to continuously do this. It's not about building
- 00:07:33,610 --> 00:07:35,390: the agent once, it's about maintainability.
- 00:07:36,910 --> 00:07:38,910: And maybe the most important one, and ultimately the one with
- 00:07:38,930 --> 00:07:39,770: the biggest outcome that
- 00:07:40,330 --> 00:07:41,530: I continue to hear is that
- 00:07:41,990 --> 00:07:42,950: we have a hard time
- 00:07:43,230 --> 00:07:44,930: getting agents from pilot and production,
- 00:07:45,250 --> 00:07:47,450: right? Where everybody was so blown away
- 00:07:47,550 --> 00:07:48,410: with the opportunity of
- 00:07:48,810 --> 00:07:50,850: what we can do in a pilot, but doing it in
- 00:07:50,890 --> 00:07:52,230: production is so much harder.
- 00:07:52,850 --> 00:07:54,410: Seventy-six percent hit roadblocks
- 00:07:54,850 --> 00:07:55,590: to get agents
- 00:07:55,910 --> 00:07:56,470: in production.
- 00:07:57,290 --> 00:07:58,570: So why
- 00:07:59,190 --> 00:08:00,430: does this happen?
- 00:08:01,750 --> 00:08:02,730: And there's another statistic
- 00:08:02,990 --> 00:08:03,550: that we heard.
- 00:08:04,390 --> 00:08:05,410: No surprise given
- 00:08:05,890 --> 00:08:07,330: where we come from, I think you would all
- 00:08:07,790 --> 00:08:08,490: agree with that
- 00:08:08,990 --> 00:08:11,030: sixty-- ninety-six percent of
- 00:08:11,150 --> 00:08:12,970: enterprises over a hundred million, and I know
- 00:08:13,510 --> 00:08:14,590: many of you are many, many,
- 00:08:15,130 --> 00:08:15,730: uh, um,
- 00:08:16,390 --> 00:08:17,510: orders of magnitude larger
- 00:08:17,790 --> 00:08:18,810: and more complex than that,
- 00:08:19,450 --> 00:08:21,890: cite that a poor data foundation
- 00:08:22,450 --> 00:08:24,370: is one of the biggest reasons why
- 00:08:24,890 --> 00:08:26,070: AI initiatives fall short.
- 00:08:27,290 --> 00:08:27,810: No surprise.
- 00:08:28,410 --> 00:08:29,130: But let's, let's
- 00:08:29,530 --> 00:08:30,090: unpack that.
- 00:08:30,870 --> 00:08:32,950: When these people say poor data foundation,
- 00:08:33,270 --> 00:08:34,130: what do they really mean?
- 00:08:35,250 --> 00:08:36,010: Do they mean that
- 00:08:36,570 --> 00:08:38,110: the agents can't access the data?
- 00:08:39,690 --> 00:08:40,470: Do they mean that
- 00:08:40,890 --> 00:08:42,950: their data platform, uh, isn't fast enough,
- 00:08:43,030 --> 00:08:44,850: doesn't scale enough, isn't performant enough?
- 00:08:45,750 --> 00:08:47,370: Do they mean that they don't have enough data?
- 00:08:48,050 --> 00:08:49,090: I don't think that's the problem.
- 00:08:49,790 --> 00:08:51,930: We've seen an, an incredible innovation
- 00:08:52,450 --> 00:08:54,490: in data platform. Agents can access the data.
- 00:08:54,550 --> 00:08:55,770: You have more than enough data,
- 00:08:56,390 --> 00:08:58,770: and they're definitely fast and scalable and performant
- 00:08:58,850 --> 00:09:00,930: enough. Would it be that the
- 00:09:01,010 --> 00:09:02,070: agent don't know
- 00:09:02,470 --> 00:09:03,030: what data
- 00:09:03,430 --> 00:09:04,790: to use, they guess,
- 00:09:05,370 --> 00:09:06,190: and they use the wrong data?
- 00:09:06,830 --> 00:09:07,790: Or that the agent
- 00:09:08,150 --> 00:09:09,590: interprets that data in the wrong way?
- 00:09:09,990 --> 00:09:11,250: They think a number
- 00:09:11,670 --> 00:09:13,690: is revenue while it's gross profit, right?
- 00:09:14,270 --> 00:09:15,610: Or is it that they're using data
- 00:09:16,190 --> 00:09:18,430: unaware that there's significant quality problems
- 00:09:18,710 --> 00:09:19,330: with that data,
- 00:09:19,930 --> 00:09:21,850: and so they don't take account, they don't account for that?
- 00:09:22,290 --> 00:09:22,970: So I don't think it's a
- 00:09:23,490 --> 00:09:25,530: data foundation problem. I think it's very
- 00:09:25,570 --> 00:09:27,770: much a governance problem. The things
- 00:09:27,790 --> 00:09:28,330: that we've been
- 00:09:28,950 --> 00:09:30,230: on a mission to solve
- 00:09:30,730 --> 00:09:31,810: for a very long time.
- 00:09:32,590 --> 00:09:34,130: That is really the data foundation.
- 00:09:34,530 --> 00:09:35,230: Not a platform,
- 00:09:35,650 --> 00:09:36,490: much more a
- 00:09:36,950 --> 00:09:38,050: governance problem.
- 00:09:38,930 --> 00:09:39,570: And we see
- 00:09:40,250 --> 00:09:41,830: poor governance lead to
- 00:09:42,270 --> 00:09:43,570: two very costly
- 00:09:43,910 --> 00:09:45,810: gaps. Gaps that we have to solve for
- 00:09:46,290 --> 00:09:48,610: if we want to achieve that expectation
- 00:09:49,030 --> 00:09:51,090: with AI. The first gap
- 00:09:51,370 --> 00:09:51,950: is context.
- 00:09:52,710 --> 00:09:53,530: And we've talked a lot about
- 00:09:53,990 --> 00:09:54,590: context, and
- 00:09:55,090 --> 00:09:56,430: we can maybe take a moment to
- 00:09:56,770 --> 00:09:57,330: define
- 00:09:57,830 --> 00:09:58,430: what we mean
- 00:09:58,770 --> 00:09:59,390: with context.
- 00:10:00,030 --> 00:10:02,190: It's the new kind of hybrid because it's exactly what
- 00:10:02,290 --> 00:10:02,970: agents need
- 00:10:03,310 --> 00:10:04,290: to operate effectively.
- 00:10:04,810 --> 00:10:06,670: Context, for me, is three different things.
- 00:10:07,170 --> 00:10:08,610: Context is obviously your data,
- 00:10:09,270 --> 00:10:11,030: right? Your structured data
- 00:10:11,350 --> 00:10:13,390: and your unstructured data, which we'll talk about.
- 00:10:14,090 --> 00:10:15,190: Context is also
- 00:10:15,670 --> 00:10:17,770: the meaning of that data, the things that's
- 00:10:18,130 --> 00:10:20,490: in our heads, right, that we implicitly understand
- 00:10:20,890 --> 00:10:22,910: to interpret data in the right way that all of our
- 00:10:22,970 --> 00:10:24,890: analysts know, agents don't know.
- 00:10:25,290 --> 00:10:26,550: We have to make that explicit.
- 00:10:27,090 --> 00:10:28,950: That meaning becomes incredibly important
- 00:10:29,249 --> 00:10:30,770: to provide it as part of the context
- 00:10:31,129 --> 00:10:33,230: to the agent. And context is
- 00:10:33,310 --> 00:10:35,570: also the process, the instructions,
- 00:10:35,670 --> 00:10:36,230: the workflow.
- 00:10:36,890 --> 00:10:38,570: Often talked as skill files, right?
- 00:10:38,870 --> 00:10:40,090: But this is how the agent
- 00:10:40,410 --> 00:10:41,710: should, uh, operate.
- 00:10:42,470 --> 00:10:43,430: And that's becoming a real
- 00:10:43,970 --> 00:10:46,290: gap because every agent,
- 00:10:46,410 --> 00:10:47,390: every AI use case
- 00:10:47,710 --> 00:10:48,810: requires context.
- 00:10:49,230 --> 00:10:49,890: And so again, we're
- 00:10:50,210 --> 00:10:51,950: often kind of brute forcing with often
- 00:10:52,330 --> 00:10:54,350: a lot of like top-down pressure to get our agent in
- 00:10:54,370 --> 00:10:55,050: production. So
- 00:10:55,490 --> 00:10:57,590: we're hiring four deployed engineers and people are
- 00:10:57,610 --> 00:10:59,870: building the right context to get our support
- 00:10:59,990 --> 00:11:00,830: agent in production,
- 00:11:01,370 --> 00:11:03,370: and then what? So we see a couple of
- 00:11:03,690 --> 00:11:04,870: problems with context. One,
- 00:11:05,490 --> 00:11:06,590: there's building context one,
- 00:11:06,990 --> 00:11:08,850: but then it's maintaining context over time.
- 00:11:09,210 --> 00:11:10,370: And it's also well understood that
- 00:11:10,910 --> 00:11:13,130: context evolve, agents evolve, models
- 00:11:13,170 --> 00:11:15,530: change, data changes. You have to maintain context
- 00:11:15,610 --> 00:11:17,970: over time. So how do we do that effectively?
- 00:11:18,970 --> 00:11:19,790: Second problem is
- 00:11:20,090 --> 00:11:21,150: context fragmentation,
- 00:11:21,490 --> 00:11:23,570: right? Uh, for the last ten years, we've spent
- 00:11:23,610 --> 00:11:24,130: a lot of time
- 00:11:24,690 --> 00:11:25,350: focused on
- 00:11:25,790 --> 00:11:28,170: dealing with data fragmentation, data silos,
- 00:11:28,550 --> 00:11:30,150: data islands, and that was a real
- 00:11:30,590 --> 00:11:32,290: inhibitor to get value from data.
- 00:11:32,790 --> 00:11:33,990: Same thing is happening with AI.
- 00:11:34,570 --> 00:11:36,710: We have AI agents everywhere.
- 00:11:36,770 --> 00:11:37,810: We have self-service,
- 00:11:38,210 --> 00:11:39,890: not service analytics, self-service
- 00:11:40,390 --> 00:11:41,290: agents, right?
- 00:11:41,750 --> 00:11:42,690: Go back ten years when
- 00:11:43,010 --> 00:11:44,670: the advent of Qlik and Tableau
- 00:11:45,010 --> 00:11:46,810: and what happened? Everybody became an analyst. We had
- 00:11:47,430 --> 00:11:49,430: thousands, if not tens of thousands of BI
- 00:11:49,470 --> 00:11:51,630: reports and what came, what happened? Chaos.
- 00:11:52,190 --> 00:11:54,450: We needed to control. Nobody trusted the outcomes anymore.
- 00:11:55,050 --> 00:11:57,270: Agents are going to be the exact same thing, times
- 00:11:57,330 --> 00:11:59,650: a hundred. And so context fragmentation,
- 00:11:59,850 --> 00:12:00,610: context silos,
- 00:12:00,970 --> 00:12:02,150: is something we'll absolutely
- 00:12:02,550 --> 00:12:03,390: have to deal with.
- 00:12:05,310 --> 00:12:06,670: So context is the first gap.
- 00:12:07,170 --> 00:12:08,970: Control is the second gap,
- 00:12:09,270 --> 00:12:11,318: right? If we have one agent,
- 00:12:11,678 --> 00:12:12,898: that's fine. We know what they do.
- 00:12:13,198 --> 00:12:13,758: When we have ten,
- 00:12:14,018 --> 00:12:15,638: when we have a hundred, when we have a thousand,
- 00:12:16,398 --> 00:12:17,078: that becomes
- 00:12:17,558 --> 00:12:18,678: a lot harder,
- 00:12:19,498 --> 00:12:20,998: right? We cannot do this
- 00:12:21,398 --> 00:12:22,118: manually anymore.
- 00:12:22,798 --> 00:12:24,838: Yes, I'm sure we'll- we're all working on an
- 00:12:24,898 --> 00:12:26,918: AI policy, but an AI policy isn't gonna
- 00:12:26,938 --> 00:12:28,038: make sure that an agent
- 00:12:28,558 --> 00:12:30,598: does what you need it to do, and it doesn't do what
- 00:12:30,658 --> 00:12:31,498: you don't want it to do.
- 00:12:32,258 --> 00:12:33,438: How do we do control
- 00:12:33,798 --> 00:12:36,478: at scale? And
- 00:12:36,518 --> 00:12:37,898: we cannot govern this one
- 00:12:38,318 --> 00:12:39,438: agent at a time,
- 00:12:39,978 --> 00:12:41,918: right? Yes, we're brute forcing this problem now,
- 00:12:42,377 --> 00:12:45,118: but we need to have a much more systematized
- 00:12:45,178 --> 00:12:46,798: approach to solve this
- 00:12:47,278 --> 00:12:48,838: because this is gonna become a massive
- 00:12:49,438 --> 00:12:50,118: scalability
- 00:12:50,578 --> 00:12:52,598: bottleneck. And we again, we,
- 00:12:52,698 --> 00:12:53,618: we're hearing that from
- 00:12:54,218 --> 00:12:56,258: what we're asking our customers and what we
- 00:12:56,278 --> 00:12:56,838: heard from the
- 00:12:57,238 --> 00:12:58,778: survey, because it's not even a,
- 00:12:59,238 --> 00:13:00,918: an... it's even a technically complex
- 00:13:01,238 --> 00:13:02,258: problem, right?
- 00:13:02,678 --> 00:13:03,618: Eighty-seven percent
- 00:13:04,038 --> 00:13:06,118: of respondents say that they verify the
- 00:13:06,178 --> 00:13:07,998: quality and the security of the agents,
- 00:13:08,718 --> 00:13:09,878: but almost forty percent
- 00:13:10,318 --> 00:13:11,698: say that they don't really know how to
- 00:13:12,018 --> 00:13:12,658: or they can't.
- 00:13:13,598 --> 00:13:14,518: This is a new
- 00:13:14,838 --> 00:13:16,738: problem that we have to solve for.
- 00:13:17,618 --> 00:13:19,438: And what we fundamentally believe is
- 00:13:19,918 --> 00:13:21,018: if you can't govern
- 00:13:21,718 --> 00:13:22,278: or see
- 00:13:22,878 --> 00:13:23,458: at runtime,
- 00:13:23,778 --> 00:13:25,678: if you can't govern your agent at runtime,
- 00:13:26,538 --> 00:13:27,098: you-- it's not
- 00:13:27,358 --> 00:13:28,318: truly governance,
- 00:13:28,818 --> 00:13:30,838: right? Governance through documentation
- 00:13:31,358 --> 00:13:32,118: no longer works.
- 00:13:32,538 --> 00:13:34,718: We were able to do that because we were able to rely
- 00:13:34,758 --> 00:13:36,658: on people that we could hold accountable
- 00:13:37,218 --> 00:13:37,938: to do the right thing.
- 00:13:38,698 --> 00:13:40,238: That's a lot harder with agents,
- 00:13:40,518 --> 00:13:42,658: right? Yes, we can document that the agent shouldn't
- 00:13:42,698 --> 00:13:43,978: do what it's not supposed to do,
- 00:13:44,398 --> 00:13:45,298: but how do we ensure,
- 00:13:45,578 --> 00:13:47,938: right? We cannot hold the agent accountable.
- 00:13:48,358 --> 00:13:50,398: We can hold the owner of the agent accountable.
- 00:13:50,458 --> 00:13:51,018: That's step one.
- 00:13:51,378 --> 00:13:52,398: But again, that doesn't scale.
- 00:13:52,778 --> 00:13:53,998: We have to be able to
- 00:13:54,498 --> 00:13:56,158: govern the agent at runtime,
- 00:13:56,678 --> 00:13:57,658: and so we'll talk about how
- 00:13:58,078 --> 00:13:59,158: Collibra can help,
- 00:13:59,578 --> 00:14:01,638: uh, with that. The
- 00:14:01,678 --> 00:14:03,398: other thing that I think is really interesting is
- 00:14:03,878 --> 00:14:05,278: what does this do to your
- 00:14:05,938 --> 00:14:06,918: unique competitive
- 00:14:07,298 --> 00:14:09,618: advantage, your unique competitive differentiation
- 00:14:09,698 --> 00:14:10,518: as an organization?
- 00:14:11,478 --> 00:14:13,538: For the last, again, ten, 15 years, we've talked about
- 00:14:13,578 --> 00:14:14,778: data as a competitive
- 00:14:15,238 --> 00:14:16,738: advantage, and obviously with AI, that's
- 00:14:17,038 --> 00:14:18,038: more true than ever.
- 00:14:18,817 --> 00:14:19,938: But it's more than just data.
- 00:14:20,877 --> 00:14:22,898: I don't think it's the model. I think very few of us are
- 00:14:22,938 --> 00:14:24,318: going to train their own model
- 00:14:24,718 --> 00:14:25,478: and make the model
- 00:14:25,858 --> 00:14:27,038: a competitive differentiation.
- 00:14:27,698 --> 00:14:29,758: It's not your data platform. We all use the same. It's
- 00:14:29,798 --> 00:14:31,858: not your AI platform or your model. We
- 00:14:31,918 --> 00:14:32,578: all use the same.
- 00:14:33,338 --> 00:14:35,498: To me, it's all about that context. It's about that
- 00:14:36,078 --> 00:14:36,978: proprietary
- 00:14:37,798 --> 00:14:38,598: con-company
- 00:14:38,878 --> 00:14:40,778: knowledge that is in your
- 00:14:41,318 --> 00:14:42,538: team's minds
- 00:14:43,078 --> 00:14:43,678: implicitly
- 00:14:44,178 --> 00:14:46,198: that we now have to formalize and make
- 00:14:46,698 --> 00:14:48,058: consumable by agent
- 00:14:48,558 --> 00:14:50,218: that becomes your most
- 00:14:50,738 --> 00:14:51,438: differentiated
- 00:14:51,898 --> 00:14:52,838: competitive advantage.
- 00:14:54,138 --> 00:14:56,238: Your data is no longer good enough, right? That's
- 00:14:56,278 --> 00:14:58,018: been your competitive advantage for a while.
- 00:14:58,738 --> 00:15:00,478: Capturing that enterprise knowledge,
- 00:15:00,758 --> 00:15:01,738: that proprietary knowledge
- 00:15:01,998 --> 00:15:04,118: is critical. If you believe
- 00:15:04,138 --> 00:15:05,538: that, if that becomes your
- 00:15:06,318 --> 00:15:07,278: unique differentiator,
- 00:15:08,018 --> 00:15:09,878: then this question really matters,
- 00:15:10,258 --> 00:15:11,138: which we'll talk about as well.
- 00:15:11,818 --> 00:15:13,998: How, and importantly, where are
- 00:15:14,118 --> 00:15:14,958: you going to
- 00:15:15,598 --> 00:15:16,758: manage that knowledge?
- 00:15:18,298 --> 00:15:19,778: Are you gonna provide it to your
- 00:15:20,178 --> 00:15:21,258: AI providers
- 00:15:21,818 --> 00:15:22,658: and have them have it?
- 00:15:23,198 --> 00:15:25,238: Are you gonna make it part of your
- 00:15:25,518 --> 00:15:26,598: data platform and make it
- 00:15:26,878 --> 00:15:27,458: part of their
- 00:15:27,738 --> 00:15:29,918: scope? Thinking strategically about
- 00:15:29,998 --> 00:15:31,098: how you deal with that,
- 00:15:31,578 --> 00:15:33,938: your proprietary enterprise knowledge
- 00:15:34,398 --> 00:15:35,598: becomes a very important
- 00:15:35,878 --> 00:15:37,898: question. And this is why
- 00:15:37,958 --> 00:15:38,538: we believe
- 00:15:39,058 --> 00:15:40,518: we need to change
- 00:15:40,938 --> 00:15:41,918: the stack.
- 00:15:42,838 --> 00:15:43,918: Think of this as
- 00:15:44,418 --> 00:15:44,958: context
- 00:15:45,338 --> 00:15:46,618: and control.
- 00:15:47,338 --> 00:15:48,218: We need a
- 00:15:48,518 --> 00:15:49,458: new stack. Of course,
- 00:15:49,958 --> 00:15:50,918: that stack continues to,
- 00:15:51,518 --> 00:15:53,778: uh, build upon your existing data sources.
- 00:15:53,838 --> 00:15:55,798: But even there, I would argue things are changing.
- 00:15:56,518 --> 00:15:57,498: For the last ten years, we're
- 00:15:57,798 --> 00:15:59,058: had the luxury, if you will, to
- 00:15:59,438 --> 00:16:00,778: focus on mostly structured
- 00:16:01,038 --> 00:16:03,098: analytical data 'cause our
- 00:16:03,138 --> 00:16:05,238: key use cases were analytics, reporting,
- 00:16:05,678 --> 00:16:06,578: consumption of data.
- 00:16:07,018 --> 00:16:08,478: Agents, it's no longer good enough.
- 00:16:09,558 --> 00:16:10,938: If you both can...
- 00:16:11,238 --> 00:16:12,678: you and I can both create an agent,
- 00:16:13,158 --> 00:16:13,778: I can only
- 00:16:14,298 --> 00:16:15,978: use structured analytical data
- 00:16:16,518 --> 00:16:18,518: as context for my agent, and you're able
- 00:16:18,578 --> 00:16:19,178: to use the same,
- 00:16:19,598 --> 00:16:21,798: but also your unstructured data, your emails, your
- 00:16:22,298 --> 00:16:24,238: teams, uh, chats, your SharePoints,
- 00:16:24,618 --> 00:16:26,718: and connect your agent to your systems of records, to
- 00:16:26,758 --> 00:16:27,697: your SAP, your
- 00:16:28,118 --> 00:16:28,818: ERP, your
- 00:16:29,238 --> 00:16:31,478: CRM. Who's gonna build the best agents?
- 00:16:32,298 --> 00:16:34,438: You, right? The agent is gonna be way
- 00:16:34,478 --> 00:16:35,218: more powerful
- 00:16:35,698 --> 00:16:36,778: by being able to use
- 00:16:37,058 --> 00:16:39,158: not just structured analytical data, but also
- 00:16:39,198 --> 00:16:39,938: unstructured data
- 00:16:40,378 --> 00:16:41,358: and operational
- 00:16:41,638 --> 00:16:42,538: systems of record data.
- 00:16:42,998 --> 00:16:44,798: So even there, you have more fragmentation.
- 00:16:46,158 --> 00:16:47,958: Then the second, as we've established, is that
- 00:16:48,618 --> 00:16:50,198: agents and data is not good enough.
- 00:16:50,558 --> 00:16:51,418: Yes, the data is
- 00:16:51,838 --> 00:16:53,678: used by the agent, but the agent needs more
- 00:16:54,058 --> 00:16:56,018: to make sure that they're using that data appropriately.
- 00:16:56,418 --> 00:16:57,378: And this is where context
- 00:16:57,738 --> 00:16:58,738: becomes key.
- 00:16:59,478 --> 00:17:01,138: And we fundamentally believe that you need an
- 00:17:01,538 --> 00:17:02,258: independent
- 00:17:02,638 --> 00:17:03,178: context
- 00:17:03,558 --> 00:17:05,718: layer to make sure that you have
- 00:17:05,758 --> 00:17:07,198: flexibility on both their
- 00:17:07,678 --> 00:17:08,598: data platform side,
- 00:17:08,858 --> 00:17:10,298: so that the agent, that context can,
- 00:17:10,898 --> 00:17:13,238: uh, capture any type of data, structured, unstructured,
- 00:17:13,338 --> 00:17:14,278: or your operational data,
- 00:17:14,678 --> 00:17:16,038: and you have flexibility on your
- 00:17:16,418 --> 00:17:18,258: AI platform side. So you have the option to
- 00:17:18,658 --> 00:17:20,318: build your own and use whatever model
- 00:17:20,698 --> 00:17:21,438: that you seemed,
- 00:17:21,898 --> 00:17:24,218: uh, fit. And
- 00:17:24,238 --> 00:17:24,758: then finally,
- 00:17:25,098 --> 00:17:26,898: control is this next,
- 00:17:27,238 --> 00:17:27,878: uh, layer.
- 00:17:28,478 --> 00:17:30,598: You need control at scale as well. Think of context
- 00:17:30,738 --> 00:17:32,818: as governing what goes into the agent,
- 00:17:32,898 --> 00:17:34,938: control as governing what the agents are
- 00:17:35,418 --> 00:17:36,598: able and allowed
- 00:17:36,918 --> 00:17:38,998: to do. And then on top, we
- 00:17:39,018 --> 00:17:40,738: fundamentally believe you're going to have many
- 00:17:41,258 --> 00:17:42,898: agent platforms. You're gonna probably have
- 00:17:43,698 --> 00:17:46,478: Claude CoWork and Copilot and ChatGPT
- 00:17:46,618 --> 00:17:48,298: and Gemini and
- 00:17:48,718 --> 00:17:50,298: Microsoft. You're gonna have many, many
- 00:17:50,638 --> 00:17:51,698: different AI
- 00:17:52,178 --> 00:17:54,178: platforms, and it's almost impossible to force an
- 00:17:54,218 --> 00:17:55,118: entire organization
- 00:17:55,578 --> 00:17:56,578: through one kind of
- 00:17:57,058 --> 00:17:59,058: gate, if you will, one portal. And it definitely,
- 00:17:59,158 --> 00:18:01,478: it's hard to make one technology decision
- 00:18:01,498 --> 00:18:03,498: today given how fast the
- 00:18:03,518 --> 00:18:04,058: technology
- 00:18:04,458 --> 00:18:05,578: innovation is changing.
- 00:18:07,758 --> 00:18:09,898: So what can you do with this
- 00:18:10,238 --> 00:18:11,278: strong data
- 00:18:11,818 --> 00:18:13,818: foundation? A couple of
- 00:18:13,838 --> 00:18:15,958: things. So how do we solve these
- 00:18:16,378 --> 00:18:17,438: governance gaps?
- 00:18:18,418 --> 00:18:20,438: So there's the two gaps we've talked about, the context
- 00:18:20,498 --> 00:18:21,458: gap and this
- 00:18:21,778 --> 00:18:22,398: control gap.
- 00:18:22,998 --> 00:18:23,938: And so at Collibra, we've been
- 00:18:24,338 --> 00:18:26,638: hard at work to drive more innovation,
- 00:18:26,738 --> 00:18:27,738: building on top of our
- 00:18:28,178 --> 00:18:29,238: years of experience
- 00:18:29,718 --> 00:18:30,838: in governance
- 00:18:31,338 --> 00:18:33,318: to help you solve some of these gaps.
- 00:18:34,198 --> 00:18:35,858: So areas we'll, we'll talk about is one,
- 00:18:36,258 --> 00:18:38,298: what we call context engineering. It's very much,
- 00:18:38,758 --> 00:18:40,218: uh, you've also heard it at Deezy,
- 00:18:40,658 --> 00:18:42,678: is the ability to prepare unstructured
- 00:18:42,718 --> 00:18:44,058: data to be used by
- 00:18:44,498 --> 00:18:45,298: agents effectively.
- 00:18:45,777 --> 00:18:46,958: I think there's an analogy with
- 00:18:47,418 --> 00:18:49,558: what we've done on the analytics engineering side,
- 00:18:49,718 --> 00:18:50,258: right? How do you
- 00:18:50,698 --> 00:18:52,538: prepare your analytical data? How do you
- 00:18:52,978 --> 00:18:54,478: transform it, model it so it's
- 00:18:54,798 --> 00:18:55,818: fit for purpose for
- 00:18:56,238 --> 00:18:57,278: analytical use cases?
- 00:18:57,818 --> 00:18:59,278: How do we do the same thing on the
- 00:18:59,618 --> 00:19:00,258: context side,
- 00:19:00,518 --> 00:19:01,698: on the unstructured data side?
- 00:19:02,658 --> 00:19:03,698: The other is context,
- 00:19:04,198 --> 00:19:04,898: uh, governance.
- 00:19:05,318 --> 00:19:07,678: I will define what we mean with context governance.
- 00:19:08,538 --> 00:19:10,478: And then Maestro. And Maestro is really
- 00:19:11,038 --> 00:19:13,138: our fundamental belief that the way we're
- 00:19:13,178 --> 00:19:13,918: doing governance
- 00:19:14,378 --> 00:19:15,318: also has to change.
- 00:19:16,218 --> 00:19:18,218: The pace and the scale of governance
- 00:19:18,278 --> 00:19:19,898: in this era of AI
- 00:19:20,358 --> 00:19:21,258: can't be the same
- 00:19:21,638 --> 00:19:22,178: as it was
- 00:19:22,498 --> 00:19:23,818: two, three, four, five years ago.
- 00:19:24,458 --> 00:19:25,418: We have to be able to
- 00:19:25,918 --> 00:19:27,638: govern much more volume
- 00:19:28,238 --> 00:19:30,418: at a much faster scale, so that means we also have
- 00:19:30,458 --> 00:19:32,398: to automate. We need to use AI
- 00:19:32,878 --> 00:19:34,218: to automate many of the
- 00:19:34,658 --> 00:19:36,698: manual governance tasks. If I
- 00:19:36,718 --> 00:19:37,438: talk to customers,
- 00:19:38,018 --> 00:19:40,338: often what I hear is, "We wanna do more, but we're constrained.
- 00:19:40,438 --> 00:19:42,318: I cannot hire ten more data stewards
- 00:19:42,918 --> 00:19:43,598: to do that work."
- 00:19:44,058 --> 00:19:45,258: We need to find ways
- 00:19:45,698 --> 00:19:47,378: to automate some of that work so we can
- 00:19:47,838 --> 00:19:48,918: do what we want to do.
- 00:19:49,558 --> 00:19:50,958: And so Maestro is a big part of that.
- 00:19:51,998 --> 00:19:52,878: And on the control side,
- 00:19:53,338 --> 00:19:55,458: obviously, AI governance with AI control tower, uh,
- 00:19:55,558 --> 00:19:56,358: AI control center
- 00:19:56,898 --> 00:19:57,938: is a key,
- 00:19:58,398 --> 00:19:59,538: uh, uh, point.
- 00:20:01,438 --> 00:20:01,998: So starting with
- 00:20:02,378 --> 00:20:03,358: context engineering
- 00:20:03,618 --> 00:20:05,418: with Deezy Labs. This is all about how do we
- 00:20:05,898 --> 00:20:06,518: prepare
- 00:20:06,938 --> 00:20:07,898: a pipeline
- 00:20:08,358 --> 00:20:09,778: to build context for
- 00:20:10,318 --> 00:20:10,838: runtime
- 00:20:11,178 --> 00:20:12,998: retrieval, context retrieval.
- 00:20:13,638 --> 00:20:15,078: And often the problem we-we're hearing,
- 00:20:15,438 --> 00:20:16,198: and this is why we
- 00:20:16,558 --> 00:20:17,898: did the acquisition of Deezy Labs,
- 00:20:18,258 --> 00:20:20,278: um, a year and a half ago, is that
- 00:20:20,738 --> 00:20:22,698: I talked more and more customer conversations told me
- 00:20:23,138 --> 00:20:24,558: that, "Look, we're governing
- 00:20:25,178 --> 00:20:25,918: structured data,
- 00:20:26,758 --> 00:20:28,858: but now we have to deal with all that unstructured data,
- 00:20:29,258 --> 00:20:31,298: and the volume of unstructured data is so much
- 00:20:31,738 --> 00:20:32,498: faster than
- 00:20:32,798 --> 00:20:34,458: what we have on the structured side."
- 00:20:35,618 --> 00:20:37,518: And so we're, we're taking that approach of
- 00:20:37,798 --> 00:20:39,438: being able to discover, ingest
- 00:20:39,778 --> 00:20:40,698: all of your unstructured data,
- 00:20:40,978 --> 00:20:42,178: SharePoint, wherever you might
- 00:20:42,498 --> 00:20:44,538: have it, curate unstructured
- 00:20:44,578 --> 00:20:46,118: data, so we are filtering it down
- 00:20:46,578 --> 00:20:47,518: to things that really
- 00:20:47,778 --> 00:20:49,858: matter, add structure to it so that
- 00:20:49,938 --> 00:20:51,178: agents can more effectively
- 00:20:51,698 --> 00:20:53,678: answer questions on top of that unstructured data,
- 00:20:54,158 --> 00:20:55,418: im-improving accuracy
- 00:20:55,758 --> 00:20:56,358: and reducing
- 00:20:56,638 --> 00:20:57,858: token consumption costs,
- 00:20:58,378 --> 00:20:59,538: and then delivering that
- 00:20:59,838 --> 00:21:01,718: as runtime context retrieval.
- 00:21:03,338 --> 00:21:05,398: And on the runtime context retrieval, a new product
- 00:21:05,538 --> 00:21:06,358: that we're excited
- 00:21:06,738 --> 00:21:08,178: to launch is what I call Live Map.
- 00:21:09,038 --> 00:21:10,918: And this is all about a realization that,
- 00:21:11,438 --> 00:21:12,598: yes, Deezy provides that
- 00:21:12,898 --> 00:21:14,158: pipeline, and we can push that
- 00:21:14,738 --> 00:21:16,798: structured, curated, unstructured data
- 00:21:17,258 --> 00:21:19,318: back into SharePoint, back to your data lake, and
- 00:21:19,338 --> 00:21:20,578: that's very valuable.
- 00:21:21,178 --> 00:21:23,458: But that last mile of actually helping agents
- 00:21:23,538 --> 00:21:24,338: find the right
- 00:21:24,778 --> 00:21:26,438: data in the most
- 00:21:26,918 --> 00:21:27,938: efficient and
- 00:21:28,298 --> 00:21:30,018: token efficient and accurate way,
- 00:21:30,758 --> 00:21:31,538: it's very important.
- 00:21:32,518 --> 00:21:34,578: And the best way to do that is probably through
- 00:21:34,658 --> 00:21:35,418: a graph.
- 00:21:35,798 --> 00:21:36,738: So you don't have to scan
- 00:21:37,198 --> 00:21:39,278: entire documents, but the agent is able
- 00:21:39,318 --> 00:21:39,838: to follow
- 00:21:40,198 --> 00:21:42,378: a map, right? I, I like the analogy of a Google
- 00:21:42,438 --> 00:21:44,638: Map where you know where you need to go, but you don't
- 00:21:44,658 --> 00:21:46,378: know what the fastest route is to get there.
- 00:21:46,898 --> 00:21:48,898: I just put in the address in my Google Map, and it tells
- 00:21:48,938 --> 00:21:50,578: me where to drive to get there.
- 00:21:51,098 --> 00:21:53,418: Think of the agent as the same thing. The live map gives
- 00:21:53,458 --> 00:21:54,438: the agent the path
- 00:21:54,898 --> 00:21:56,118: to get to the right data
- 00:21:56,638 --> 00:21:57,938: in the quickest, most efficient
- 00:21:58,298 --> 00:21:59,338: time possible,
- 00:21:59,998 --> 00:22:00,978: so that the agent doesn't
- 00:22:01,238 --> 00:22:03,418: just get a full picture, not just of your entire
- 00:22:03,898 --> 00:22:05,338: data landscape, but also what that
- 00:22:05,698 --> 00:22:06,858: data really means.
- 00:22:07,898 --> 00:22:08,638: And so how do we do that?
- 00:22:09,458 --> 00:22:10,318: The second maybe
- 00:22:10,718 --> 00:22:12,658: biggest word that I've started to use over the last
- 00:22:13,218 --> 00:22:14,418: four or five months is ontology.
- 00:22:15,158 --> 00:22:15,758: Some of you know,
- 00:22:16,138 --> 00:22:16,658: this is actually the
- 00:22:16,938 --> 00:22:18,418: origin of Collibra. We started
- 00:22:18,918 --> 00:22:21,098: with ontology research eighteen years ago,
- 00:22:21,798 --> 00:22:22,618: RDF, OWL,
- 00:22:23,218 --> 00:22:25,538: uh, things like that, open kind of semantic web standards
- 00:22:25,558 --> 00:22:26,338: is where it started.
- 00:22:26,958 --> 00:22:28,398: Now I've used that word more than
- 00:22:28,778 --> 00:22:30,558: ever. Why? Because again, grounding
- 00:22:31,058 --> 00:22:33,298: agents in that business context, in that meaning
- 00:22:33,338 --> 00:22:34,478: becomes incredibly important.
- 00:22:35,058 --> 00:22:36,918: And we're a big believer in open standards
- 00:22:37,358 --> 00:22:39,218: to capture these ontology models.
- 00:22:39,818 --> 00:22:41,038: So we can take an ontology model,
- 00:22:41,698 --> 00:22:43,398: map that against automatically,
- 00:22:43,878 --> 00:22:45,998: uh, map that against your unstructured data, and
- 00:22:46,038 --> 00:22:48,038: it's going to build that graph, that
- 00:22:48,118 --> 00:22:49,858: live map, that agents can use
- 00:22:50,238 --> 00:22:50,878: at runtime
- 00:22:51,358 --> 00:22:52,578: to optimize
- 00:22:52,858 --> 00:22:53,958: context retrieval.
- 00:22:54,678 --> 00:22:55,258: And we'll show more
- 00:22:55,558 --> 00:22:56,078: in a little bit.
- 00:22:58,118 --> 00:22:59,318: Second big topic
- 00:22:59,618 --> 00:23:01,538: that I think we need to solve for is context
- 00:23:01,898 --> 00:23:03,878: governance. So what is context
- 00:23:04,258 --> 00:23:06,458: governance? Again, this is the realization
- 00:23:06,478 --> 00:23:08,598: that every agent, every AI
- 00:23:08,678 --> 00:23:09,918: use case will require
- 00:23:10,418 --> 00:23:12,398: specific context to be created,
- 00:23:12,878 --> 00:23:14,138: skills files, tools,
- 00:23:14,798 --> 00:23:15,758: connection to data
- 00:23:16,418 --> 00:23:17,398: for it to operate.
- 00:23:18,498 --> 00:23:19,698: And we're gonna have context
- 00:23:20,058 --> 00:23:20,738: fragmentation.
- 00:23:21,438 --> 00:23:23,498: The same problem that we've had to solve for on
- 00:23:23,538 --> 00:23:24,918: the data side, we'll have to solve for
- 00:23:25,338 --> 00:23:26,578: on the context side.
- 00:23:27,298 --> 00:23:28,438: And even we are seeing
- 00:23:28,878 --> 00:23:30,358: in our, in Collibra, the challenge of
- 00:23:30,698 --> 00:23:31,998: people building an agent,
- 00:23:32,358 --> 00:23:34,558: Claude CoWork, say, they build the context, build the tools,
- 00:23:34,638 --> 00:23:35,918: build the skills, connect the data,
- 00:23:36,358 --> 00:23:37,458: and it works on their
- 00:23:37,858 --> 00:23:38,438: environment.
- 00:23:39,258 --> 00:23:41,338: How do we actually share that across the organization?
- 00:23:41,878 --> 00:23:43,538: How do we package context
- 00:23:43,998 --> 00:23:44,678: in a way that
- 00:23:44,998 --> 00:23:46,458: we can guarantee its quality,
- 00:23:47,078 --> 00:23:49,138: we can make sure it's highly curated, we know what,
- 00:23:49,418 --> 00:23:50,118: who's responsible,
- 00:23:50,458 --> 00:23:51,738: and it can be easily shared
- 00:23:51,998 --> 00:23:53,098: across the organization?
- 00:23:53,998 --> 00:23:54,718: Sounds familiar.
- 00:23:55,678 --> 00:23:57,878: That's because exactly what we've done for data governance,
- 00:23:58,398 --> 00:24:00,418: right? How do we do data governance? We're a
- 00:24:00,438 --> 00:24:01,798: big believer, as you probably know, in
- 00:24:02,158 --> 00:24:03,578: this data product approach.
- 00:24:04,374 --> 00:24:06,834: So how do we solve for that? How do we solve for data fragmentation?
- 00:24:06,874 --> 00:24:09,234: Well, there's cat-cataloging and discovery,
- 00:24:09,334 --> 00:24:10,614: so you know what data you have.
- 00:24:11,414 --> 00:24:13,434: You apply quality and validation rules so you
- 00:24:13,474 --> 00:24:14,194: can trust the data.
- 00:24:14,774 --> 00:24:16,874: You, you need lineage and traceability, so if something
- 00:24:16,974 --> 00:24:17,654: changes, you know
- 00:24:18,194 --> 00:24:19,854: what's the impact and what's the root cause.
- 00:24:20,454 --> 00:24:22,514: You need clear roles and responsibilities and manage the
- 00:24:22,594 --> 00:24:23,414: access to that data,
- 00:24:23,914 --> 00:24:26,114: right? You need policies and compliance
- 00:24:26,394 --> 00:24:28,134: to be applied to these data sets.
- 00:24:28,634 --> 00:24:30,634: You need certification and trust, so people know what
- 00:24:30,674 --> 00:24:31,254: data product
- 00:24:31,694 --> 00:24:33,694: could be used. They need to make that easily available for
- 00:24:33,734 --> 00:24:34,974: reuse through a data marketplace.
- 00:24:35,794 --> 00:24:36,774: The exact same thing
- 00:24:37,254 --> 00:24:38,474: is necessary for context.
- 00:24:39,254 --> 00:24:39,894: Context being,
- 00:24:40,214 --> 00:24:41,954: yes, the data, but also unstructured data,
- 00:24:42,834 --> 00:24:44,514: your processes, your skills,
- 00:24:44,874 --> 00:24:46,954: and the meaning. 'Cause again, you
- 00:24:46,994 --> 00:24:48,054: need to discover
- 00:24:48,434 --> 00:24:48,954: and catalog
- 00:24:49,214 --> 00:24:50,354: what is context out there.
- 00:24:50,834 --> 00:24:51,714: People are building
- 00:24:52,094 --> 00:24:53,714: context all over your organization.
- 00:24:54,094 --> 00:24:55,594: How do you make sure you know what you have?
- 00:24:55,994 --> 00:24:56,754: How do you make sure
- 00:24:57,234 --> 00:24:59,214: it's doing the right thing? How do you measure the quality?
- 00:24:59,614 --> 00:25:00,374: How do you make sure
- 00:25:00,754 --> 00:25:02,934: that you have clear responsibilities? If something changes,
- 00:25:03,014 --> 00:25:05,014: you know who's responsible. The agent
- 00:25:05,534 --> 00:25:07,654: includes the context. How do you think about
- 00:25:07,734 --> 00:25:10,054: access and roles and responsibilities?
- 00:25:10,394 --> 00:25:12,554: How do you think about policies because you're instructing
- 00:25:12,574 --> 00:25:13,534: an agent to do something?
- 00:25:13,934 --> 00:25:14,854: That's part of your context.
- 00:25:15,334 --> 00:25:17,414: How do you make sure that these instructions are actually
- 00:25:17,874 --> 00:25:19,414: compliant with some of the policies
- 00:25:19,854 --> 00:25:22,034: that you have? How do you certify that, and how
- 00:25:22,074 --> 00:25:23,334: do you make that easily available
- 00:25:23,834 --> 00:25:25,074: to a marketplace, a
- 00:25:25,554 --> 00:25:27,094: not just data product marketplace, but a
- 00:25:27,474 --> 00:25:29,054: context product or context package
- 00:25:29,374 --> 00:25:29,914: marketplace?
- 00:25:32,354 --> 00:25:33,114: And this is what we're
- 00:25:33,474 --> 00:25:35,154: building. And a key part of
- 00:25:35,554 --> 00:25:37,694: context governance is that ontology
- 00:25:37,834 --> 00:25:39,954: grounding the agent in
- 00:25:40,014 --> 00:25:41,054: your business
- 00:25:41,454 --> 00:25:43,574: context. So like I said, we've had many
- 00:25:43,654 --> 00:25:46,014: conversations with many of our customers, and it was a clear
- 00:25:46,574 --> 00:25:48,494: need that we've heard from the market that
- 00:25:49,054 --> 00:25:50,794: people want Collibra to help
- 00:25:51,414 --> 00:25:52,614: manage their ontologies.
- 00:25:53,434 --> 00:25:53,974: Some of you
- 00:25:54,474 --> 00:25:56,594: have already done ontology management for
- 00:25:56,634 --> 00:25:58,714: a while. You might have your existing ontologies. You might
- 00:25:58,774 --> 00:26:00,794: use existing tool to create and manage these
- 00:26:00,834 --> 00:26:02,834: ontologies. That's great. You can easily
- 00:26:02,894 --> 00:26:03,434: import them
- 00:26:03,854 --> 00:26:05,774: 'cause we're using open standards like RDF and OWL
- 00:26:06,414 --> 00:26:08,474: to import them in Collibra and then use Collibra
- 00:26:08,554 --> 00:26:10,074: to map these ontologies
- 00:26:10,714 --> 00:26:11,414: against your
- 00:26:11,814 --> 00:26:13,914: structured data, your tables, your schemas,
- 00:26:14,034 --> 00:26:16,294: your columns. Some of you might
- 00:26:16,534 --> 00:26:18,314: want to reuse existing industry standards,
- 00:26:18,594 --> 00:26:20,294: like in, in the example, this is the FIBO,
- 00:26:20,914 --> 00:26:22,074: uh, financial services,
- 00:26:22,394 --> 00:26:23,174: uh, ontology
- 00:26:23,634 --> 00:26:25,774: that you wanna reuse because you don't wanna start from scratch.
- 00:26:26,354 --> 00:26:28,374: Some of you obviously want to start from scratch, and
- 00:26:28,414 --> 00:26:29,994: the thing is, you already have so much
- 00:26:30,534 --> 00:26:31,854: valuable content in Collibra.
- 00:26:32,294 --> 00:26:34,374: You have your business glossaries. You have your semantic
- 00:26:34,434 --> 00:26:35,854: models. You have all of the information
- 00:26:36,274 --> 00:26:37,694: that we can use AI to
- 00:26:38,074 --> 00:26:39,234: automatically generate
- 00:26:39,874 --> 00:26:40,934: a significant part
- 00:26:41,214 --> 00:26:42,074: of that ontology.
- 00:26:43,114 --> 00:26:43,894: And then you can use the
- 00:26:44,234 --> 00:26:45,314: ontology in Collibra
- 00:26:45,854 --> 00:26:46,454: to map that
- 00:26:46,734 --> 00:26:47,474: against your
- 00:26:47,974 --> 00:26:50,114: structured data, which gives, again, the agents
- 00:26:50,154 --> 00:26:50,874: that map
- 00:26:51,274 --> 00:26:52,134: of where to go
- 00:26:52,554 --> 00:26:53,494: to make sure they're using
- 00:26:53,754 --> 00:26:54,694: the right data.
- 00:26:56,694 --> 00:26:57,654: And then Context Designer
- 00:26:58,174 --> 00:27:00,554: is again taking that data product thinking
- 00:27:00,974 --> 00:27:01,574: to context.
- 00:27:02,054 --> 00:27:03,174: How do we package
- 00:27:03,554 --> 00:27:05,594: context? How do we curate context? How do we
- 00:27:06,034 --> 00:27:08,154: make sure context is compliant with
- 00:27:08,214 --> 00:27:10,034: our quality standards, with our
- 00:27:10,454 --> 00:27:11,894: policy standards? How do we assign
- 00:27:12,294 --> 00:27:14,514: ownership around context? Because if something
- 00:27:14,594 --> 00:27:16,334: changes, what do we do?
- 00:27:16,934 --> 00:27:17,674: We, we've, we've f-
- 00:27:18,074 --> 00:27:19,234: we felt it in ourselves.
- 00:27:19,574 --> 00:27:21,614: We've built our own analytics agent, we call it
- 00:27:21,654 --> 00:27:23,874: Dwight. Three thousand MD files,
- 00:27:24,014 --> 00:27:24,754: Markdown files
- 00:27:25,234 --> 00:27:26,734: to guide the context, to guide
- 00:27:27,014 --> 00:27:29,094: Dwight of how to answer questions, and
- 00:27:29,134 --> 00:27:31,094: it's really good, but it's not perfect,
- 00:27:31,514 --> 00:27:32,254: and we get feedback.
- 00:27:32,814 --> 00:27:33,674: We need to change something.
- 00:27:34,314 --> 00:27:36,134: Then the question, okay, but who needs to change what?
- 00:27:37,054 --> 00:27:39,054: How do we maintain it? How do we manage the life
- 00:27:39,114 --> 00:27:40,074: cycle and the change
- 00:27:40,594 --> 00:27:42,354: of that context? That's a governance problem.
- 00:27:42,874 --> 00:27:43,454: So we're our own
- 00:27:43,774 --> 00:27:45,814: best customer because we've experienced that problem
- 00:27:45,854 --> 00:27:48,174: ourselves by building these analytics
- 00:27:48,714 --> 00:27:50,734: agents. And this is what we wanna build
- 00:27:50,794 --> 00:27:51,434: with Context
- 00:27:51,714 --> 00:27:54,034: Design. And
- 00:27:54,074 --> 00:27:56,074: this is the result. The question is why? Why do we really
- 00:27:56,114 --> 00:27:57,014: need that? I don't know how
- 00:27:57,414 --> 00:27:59,254: well understood and accepted that is, but
- 00:27:59,894 --> 00:28:01,554: you can believe us. We've built this,
- 00:28:01,934 --> 00:28:03,834: uh, fake bank, Penrose Bank.
- 00:28:04,214 --> 00:28:05,734: It's a realistic, complex bank. And then we've,
- 00:28:06,174 --> 00:28:07,554: uh, asked it a number of analytical,
- 00:28:08,074 --> 00:28:10,134: complex analytical queries. Without context,
- 00:28:10,454 --> 00:28:11,014: it replied
- 00:28:11,714 --> 00:28:13,714: correctly thirty-two percent of the time. With
- 00:28:13,754 --> 00:28:14,694: context, it replied
- 00:28:15,074 --> 00:28:16,594: correctly seventy-three percent of the time.
- 00:28:17,374 --> 00:28:17,974: I'm sure you've seen the
- 00:28:18,234 --> 00:28:20,434: blog post from OpenAI, Anthropic, how they've
- 00:28:20,453 --> 00:28:22,633: built their analytics agent, very similar
- 00:28:22,674 --> 00:28:24,214: results. Without context,
- 00:28:24,614 --> 00:28:25,494: really bad accuracy.
- 00:28:25,953 --> 00:28:26,534: With context,
- 00:28:26,894 --> 00:28:27,794: much better accuracy.
- 00:28:28,514 --> 00:28:30,754: You get much better gains by providing better
- 00:28:30,914 --> 00:28:32,274: context, and actually, I believe,
- 00:28:32,554 --> 00:28:33,654: training the context
- 00:28:34,274 --> 00:28:36,274: rather than building or using a better,
- 00:28:36,734 --> 00:28:37,414: more expensive
- 00:28:37,714 --> 00:28:39,794: model. The other thing we've heard
- 00:28:39,834 --> 00:28:41,254: from OpenAI and Anthropic is
- 00:28:41,594 --> 00:28:42,234: that context
- 00:28:42,534 --> 00:28:43,054: degrades.
- 00:28:43,894 --> 00:28:45,894: Today at seventy-three percent, if we don't change
- 00:28:45,934 --> 00:28:47,434: it, three months from now,
- 00:28:47,694 --> 00:28:48,754: it's back to fifty percent.
- 00:28:49,614 --> 00:28:51,334: So how do we continuously maintain
- 00:28:51,694 --> 00:28:52,834: that context? Again,
- 00:28:53,214 --> 00:28:54,294: governance problem.
- 00:28:56,094 --> 00:28:56,694: And then Maestro,
- 00:28:57,034 --> 00:28:58,374: right? Like I said, governance
- 00:28:58,754 --> 00:29:00,874: needs to evolve. The way we're doing governance needs to
- 00:29:00,934 --> 00:29:02,094: change, needs to be faster
- 00:29:02,414 --> 00:29:04,454: and much more automated. Maestro is our own
- 00:29:05,074 --> 00:29:06,454: internal, let's say, governance
- 00:29:07,094 --> 00:29:08,114: AI harness
- 00:29:08,554 --> 00:29:10,674: to automate these governance and stewardship tasks.
- 00:29:11,274 --> 00:29:12,334: So you're able to
- 00:29:12,854 --> 00:29:13,434: automatically
- 00:29:13,754 --> 00:29:14,434: build your own
- 00:29:14,834 --> 00:29:16,914: stewardship agents, give it the right context, give
- 00:29:16,934 --> 00:29:18,914: it the right instructions on top of our knowledge base
- 00:29:19,294 --> 00:29:20,694: to automate many of the tasks,
- 00:29:21,154 --> 00:29:22,894: any content creation task,
- 00:29:23,154 --> 00:29:25,554: definition creation, description gen-generation,
- 00:29:25,954 --> 00:29:27,514: any mapping task, linking a
- 00:29:27,974 --> 00:29:29,294: concept and asset to a
- 00:29:29,714 --> 00:29:30,494: technical asset,
- 00:29:30,934 --> 00:29:33,174: providing assessments, creating data quality
- 00:29:33,234 --> 00:29:35,234: rules, applying data quality rules, identifying
- 00:29:35,274 --> 00:29:36,594: lineage. All these things
- 00:29:37,034 --> 00:29:39,154: can now be done by agents that you can create
- 00:29:39,614 --> 00:29:40,674: on your specific
- 00:29:40,954 --> 00:29:41,514: environment.
- 00:29:43,194 --> 00:29:45,154: And so this is how we think of this kind of emerging
- 00:29:45,714 --> 00:29:46,334: context layer.
- 00:29:46,694 --> 00:29:47,694: Now, we believe you cannot
- 00:29:48,374 --> 00:29:50,534: buy a context layer off-the-shelf today.
- 00:29:51,254 --> 00:29:53,654: There's too many different components. It's too specific
- 00:29:53,754 --> 00:29:55,254: to your particular use cases.
- 00:29:55,634 --> 00:29:56,834: Technology is evolving
- 00:29:57,294 --> 00:29:57,814: too quickly.
- 00:29:58,374 --> 00:30:00,294: But we believe there are a number of components
- 00:30:00,794 --> 00:30:01,694: that you can reuse.
- 00:30:02,154 --> 00:30:03,374: And this is also why we believe
- 00:30:03,814 --> 00:30:06,054: in open standards, so they're easy to exchange
- 00:30:06,574 --> 00:30:08,054: across as you build your own
- 00:30:08,494 --> 00:30:09,054: context layer.
- 00:30:09,950 --> 00:30:10,510: At the bottom,
- 00:30:10,890 --> 00:30:12,970: we are a big believer that you have to be able to combine your
- 00:30:13,330 --> 00:30:13,990: structured data
- 00:30:14,450 --> 00:30:15,450: and your unstructured data,
- 00:30:15,710 --> 00:30:16,250: your metadata.
- 00:30:16,950 --> 00:30:18,950: You need an ontology to ground it, to give it
- 00:30:19,270 --> 00:30:20,650: meaning. You need to
- 00:30:21,010 --> 00:30:23,030: curate this context as packages
- 00:30:23,310 --> 00:30:24,790: that can be curated,
- 00:30:25,090 --> 00:30:27,310: validated, assigned ownership, and shared.
- 00:30:28,430 --> 00:30:29,090: And then you need to
- 00:30:29,350 --> 00:30:30,830: provide ways and to make,
- 00:30:31,190 --> 00:30:32,050: making that context
- 00:30:32,450 --> 00:30:32,990: retrieval,
- 00:30:33,530 --> 00:30:34,190: uh, um,
- 00:30:34,630 --> 00:30:35,150: optimized.
- 00:30:35,770 --> 00:30:37,770: And this is where we have LiveMap, but we also
- 00:30:37,790 --> 00:30:38,330: believe in
- 00:30:38,690 --> 00:30:40,850: a combination of different retrieval techniques,
- 00:30:40,910 --> 00:30:43,070: whether it's vector stores, GraVRAC,
- 00:30:43,650 --> 00:30:44,490: LLM wikis,
- 00:30:44,790 --> 00:30:46,790: Grep, and things like that. It's gonna be a combination of
- 00:30:46,810 --> 00:30:47,450: different techniques
- 00:30:47,910 --> 00:30:49,230: to do the runtime
- 00:30:49,530 --> 00:30:52,210: retrieval. So
- 00:30:52,250 --> 00:30:52,870: that was everything
- 00:30:53,410 --> 00:30:53,950: context.
- 00:30:54,330 --> 00:30:54,890: Shifting to
- 00:30:55,210 --> 00:30:56,290: con, uh, control,
- 00:30:56,790 --> 00:30:58,830: and this is where, again, we're a big believer is
- 00:30:58,890 --> 00:30:59,590: that you can't
- 00:30:59,890 --> 00:31:00,850: govern your agents
- 00:31:01,430 --> 00:31:03,610: by just documentation, just assessments,
- 00:31:03,710 --> 00:31:04,510: just attestations.
- 00:31:05,010 --> 00:31:06,410: You need to be able to govern them
- 00:31:06,790 --> 00:31:07,710: at runtime.
- 00:31:08,390 --> 00:31:10,510: And this is why we introduced this concept of what we
- 00:31:10,530 --> 00:31:11,230: call an agent
- 00:31:11,530 --> 00:31:12,550: contract. Again, taking
- 00:31:12,970 --> 00:31:15,230: things that worked from the data world, data
- 00:31:15,310 --> 00:31:17,550: contracts, applying that to agents.
- 00:31:17,890 --> 00:31:19,470: How do we define an agent contract
- 00:31:19,910 --> 00:31:21,770: that specifies what the agent is able
- 00:31:22,190 --> 00:31:24,130: and allowed to do in a way that we as a human
- 00:31:24,510 --> 00:31:25,210: can understand,
- 00:31:25,610 --> 00:31:26,610: can map it to our policy,
- 00:31:27,030 --> 00:31:28,290: but it's also executable,
- 00:31:28,690 --> 00:31:30,390: can be operationalized at runtime?
- 00:31:31,450 --> 00:31:32,370: And that's not an easy
- 00:31:32,630 --> 00:31:33,390: technical problem
- 00:31:34,190 --> 00:31:35,650: because we cannot just
- 00:31:36,070 --> 00:31:36,990: tell the agent
- 00:31:37,290 --> 00:31:38,770: what to do because as we are
- 00:31:39,250 --> 00:31:40,150: reading news every day,
- 00:31:40,430 --> 00:31:42,110: the agents don't listen, right?
- 00:31:42,590 --> 00:31:43,510: Or they're optimized
- 00:31:43,830 --> 00:31:44,670: for... They optimize
- 00:31:44,990 --> 00:31:47,150: to drive a certain outcome, and they go really far
- 00:31:47,590 --> 00:31:49,610: in driving that outcomes in ways we haven't thought
- 00:31:49,670 --> 00:31:51,830: about. So we have to be able to
- 00:31:52,290 --> 00:31:54,490: guard the agents outside of the model
- 00:31:55,030 --> 00:31:56,330: in deterministic way.
- 00:31:56,950 --> 00:31:59,290: So how do we convert, compile this agent
- 00:31:59,330 --> 00:32:00,290: contract into
- 00:32:00,750 --> 00:32:02,150: a runnable,
- 00:32:02,430 --> 00:32:02,970: what we call,
- 00:32:03,310 --> 00:32:04,570: guardian agent?
- 00:32:05,730 --> 00:32:07,770: And this is how we're able to,
- 00:32:08,730 --> 00:32:10,330: um, comply
- 00:32:10,850 --> 00:32:12,050: with your policies
- 00:32:12,450 --> 00:32:13,050: at runtime.
- 00:32:13,930 --> 00:32:16,370: We apply, we compile this contract
- 00:32:16,650 --> 00:32:17,190: as code.
- 00:32:17,789 --> 00:32:19,230: You can use it as part of your
- 00:32:19,730 --> 00:32:20,370: AI gateway,
- 00:32:20,650 --> 00:32:22,490: which I'm sure you have, or as part of your
- 00:32:22,930 --> 00:32:23,610: AI platform,
- 00:32:24,250 --> 00:32:26,350: and then we're able to make sure that the agent
- 00:32:26,890 --> 00:32:28,970: doesn't execute anything that you don't
- 00:32:28,990 --> 00:32:30,550: want it to do, that it really
- 00:32:31,070 --> 00:32:32,290: implements that a-agent
- 00:32:32,650 --> 00:32:34,750: contract. Of
- 00:32:34,810 --> 00:32:35,770: c-- obviously, obviously,
- 00:32:36,270 --> 00:32:37,150: you also get the feedback.
- 00:32:37,730 --> 00:32:38,770: Every time an agent
- 00:32:39,210 --> 00:32:41,290: tried to do something that you didn't want it to do, we
- 00:32:41,330 --> 00:32:42,550: can block it, we can monitor it,
- 00:32:43,010 --> 00:32:43,610: we can log it,
- 00:32:43,990 --> 00:32:46,030: and then you're able to act on that. So it
- 00:32:46,070 --> 00:32:46,670: gives you the full
- 00:32:47,150 --> 00:32:47,790: visibility there
- 00:32:48,090 --> 00:32:50,270: as well. You're
- 00:32:50,290 --> 00:32:52,110: gonna see us continue to double down on
- 00:32:52,430 --> 00:32:53,030: our AI
- 00:32:53,310 --> 00:32:54,730: governance, our AI command center.
- 00:32:55,270 --> 00:32:56,570: Um, super excited to
- 00:32:56,950 --> 00:32:59,110: recently announce that we've acquired a company called
- 00:32:59,150 --> 00:33:01,330: TrailML. Small company, but a really strong
- 00:33:01,410 --> 00:33:03,390: team, deep expertise in
- 00:33:03,890 --> 00:33:04,890: AI governance. And
- 00:33:05,350 --> 00:33:06,830: like the one quote that I wanna share,
- 00:33:07,270 --> 00:33:07,930: we recently had a,
- 00:33:08,250 --> 00:33:08,950: kind of a product
- 00:33:09,430 --> 00:33:11,470: workshop, and the CTO was a CTO of TrailML,
- 00:33:11,510 --> 00:33:12,310: one of the co-founders,
- 00:33:12,830 --> 00:33:14,990: used to be at, uh, Meta, Facebook, responsible for AI
- 00:33:15,050 --> 00:33:15,910: governance at Facebook.
- 00:33:16,410 --> 00:33:18,150: So very much engineering-first culture.
- 00:33:18,790 --> 00:33:20,890: And some of our team asked him like, "Why did you start Trail?
- 00:33:21,750 --> 00:33:24,050: Because you love compliance so much?" "No,"
- 00:33:24,130 --> 00:33:26,110: he said, "because I hate compliance so much.
- 00:33:26,650 --> 00:33:28,890: I wanna make it easier for engineering teams
- 00:33:29,510 --> 00:33:30,150: to get
- 00:33:30,670 --> 00:33:32,150: AI use cases in production
- 00:33:32,610 --> 00:33:34,370: without having to go through all these
- 00:33:34,790 --> 00:33:36,050: manual compliance
- 00:33:36,590 --> 00:33:38,430: checks." And this is very much
- 00:33:38,870 --> 00:33:40,990: the principle we wanna bring to our AI command center. How
- 00:33:41,010 --> 00:33:42,330: do we automate the
- 00:33:42,710 --> 00:33:43,950: GRC part of,
- 00:33:44,450 --> 00:33:45,290: uh, AI governance?
- 00:33:45,670 --> 00:33:46,890: And how do we implement,
- 00:33:47,590 --> 00:33:48,330: uh, runtime
- 00:33:48,670 --> 00:33:49,310: enforcement
- 00:33:49,910 --> 00:33:50,450: at runtime?
- 00:33:51,770 --> 00:33:53,690: So going back to this
- 00:33:54,270 --> 00:33:55,050: quote that I think...
- 00:33:55,470 --> 00:33:56,370: Or quote or, uh,
- 00:33:57,030 --> 00:33:58,550: survey result that we saw. So
- 00:33:58,930 --> 00:33:59,810: sixty-nine percent
- 00:34:00,330 --> 00:34:00,870: believe that
- 00:34:01,270 --> 00:34:02,210: a missing
- 00:34:02,730 --> 00:34:04,430: or poor data foundation
- 00:34:05,030 --> 00:34:05,970: is a source of a lot of,
- 00:34:06,550 --> 00:34:08,110: uh, AI initiatives falling short.
- 00:34:08,750 --> 00:34:10,750: I think I've argued why we believe that's
- 00:34:10,790 --> 00:34:13,010: much more of a data governance and a governance
- 00:34:13,490 --> 00:34:14,030: foundation
- 00:34:14,450 --> 00:34:15,250: rather than a pure
- 00:34:15,690 --> 00:34:16,750: data foundation.
- 00:34:17,690 --> 00:34:19,690: My, uh, call to all
- 00:34:19,750 --> 00:34:21,810: of you is how do we make sure that
- 00:34:22,270 --> 00:34:23,190: not only are you
- 00:34:23,510 --> 00:34:24,550: part of the four percent,
- 00:34:24,970 --> 00:34:26,790: but how do we make that four percent a lot
- 00:34:27,130 --> 00:34:28,889: bigger? Um,
- 00:34:29,389 --> 00:34:30,850: we've talked about like why your
- 00:34:31,209 --> 00:34:33,310: competitive advantage, we believe, is no longer
- 00:34:33,350 --> 00:34:33,950: just your data,
- 00:34:34,490 --> 00:34:35,810: but your specific
- 00:34:36,250 --> 00:34:37,570: proprietary knowledge.
- 00:34:38,310 --> 00:34:40,390: And then how do you formalize that knowledge so that agents
- 00:34:40,430 --> 00:34:41,390: can use that knowledge
- 00:34:41,930 --> 00:34:43,010: to truly automate
- 00:34:43,330 --> 00:34:45,050: tasks? Which means that
- 00:34:45,450 --> 00:34:46,210: where and how
- 00:34:46,670 --> 00:34:47,290: you build that
- 00:34:47,790 --> 00:34:48,510: independent,
- 00:34:48,810 --> 00:34:49,330: you would argue,
- 00:34:49,770 --> 00:34:50,290: context
- 00:34:50,630 --> 00:34:51,830: platform, context layer
- 00:34:52,430 --> 00:34:53,010: truly matters.
- 00:34:53,310 --> 00:34:54,270: It's a very strategic
- 00:34:54,830 --> 00:34:56,910: decision. And my final
- 00:34:56,990 --> 00:34:58,110: thought is kind of
- 00:34:58,670 --> 00:34:59,910: taking a, an action
- 00:35:00,350 --> 00:35:01,090: where to start,
- 00:35:01,490 --> 00:35:03,210: going back to my very first question,
- 00:35:03,790 --> 00:35:05,790: how many agents do we have in production
- 00:35:05,830 --> 00:35:07,470: today? How do we
- 00:35:07,870 --> 00:35:09,210: make sure we can answer this
- 00:35:09,630 --> 00:35:10,590: very simple
- 00:35:10,850 --> 00:35:12,990: question? That could be a great way
- 00:35:13,010 --> 00:35:14,070: to start because if we do,
- 00:35:14,670 --> 00:35:16,490: then we can figure out what those agents are doing,
- 00:35:16,890 --> 00:35:17,750: and then we can figure...
- 00:35:18,070 --> 00:35:20,090: then we can make sure that these agents are actually doing
- 00:35:20,490 --> 00:35:22,090: the right thing. Um,
- 00:35:23,110 --> 00:35:24,370: so with that, um,
- 00:35:24,990 --> 00:35:26,790: thank you. Uh, excited to
- 00:35:27,230 --> 00:35:27,930: spend more time,
- 00:35:28,390 --> 00:35:29,650: dive deeper on both
- 00:35:29,970 --> 00:35:30,890: context engineering,
- 00:35:31,370 --> 00:35:32,610: on context governance,
- 00:35:32,890 --> 00:35:34,250: and AI governance. We're
- 00:35:34,630 --> 00:35:36,670: super excited of the innovation that we're driving, but we're even
- 00:35:36,710 --> 00:35:37,330: more excited to
- 00:35:37,730 --> 00:35:38,350: partner with you,
- 00:35:38,830 --> 00:35:40,650: uh, to make that a reality and to help you solve
- 00:35:40,930 --> 00:35:42,410: these two governance gaps.
Ending the hallucination tax: Governing the backbone of enterprise AI
Video: Introducing Live Map: From unstructured data to curated, connected context
Transcript
- 00:00:05,900 --> 00:00:08,100: Great to be here. Great to see so many people. Really, really
- 00:00:08,180 --> 00:00:08,860: cool turnout.
- 00:00:09,240 --> 00:00:11,280: Um, so by way of introduction, uh, my
- 00:00:11,300 --> 00:00:12,140: name is Rhys Griffiths.
- 00:00:12,560 --> 00:00:15,000: Uh, I was previously one of the co-founders of a company
- 00:00:15,400 --> 00:00:16,580: called Daze labs.
- 00:00:16,940 --> 00:00:17,960: Um, as Felix mentioned,
- 00:00:18,420 --> 00:00:20,080: Daze was acquired by Collibra
- 00:00:20,420 --> 00:00:22,520: about a year and a half ago now, which has truly flown
- 00:00:22,580 --> 00:00:23,820: by. Um,
- 00:00:24,280 --> 00:00:26,860: and my team and I, we, we initially started Daze
- 00:00:26,920 --> 00:00:28,920: with a very simple premise, which was that,
- 00:00:29,400 --> 00:00:31,560: you know, the last twenty years of
- 00:00:31,620 --> 00:00:33,640: data governance, data management, especially in
- 00:00:33,660 --> 00:00:34,260: the enterprise,
- 00:00:34,600 --> 00:00:36,540: had been focused exclusively on
- 00:00:36,980 --> 00:00:37,900: structured data.
- 00:00:38,780 --> 00:00:40,860: But, uh, as we all know by now, there is
- 00:00:40,960 --> 00:00:42,440: huge volumes of very rich,
- 00:00:42,860 --> 00:00:44,940: uh, relevant knowledge hidden across the millions of
- 00:00:44,980 --> 00:00:46,140: SharePoint files,
- 00:00:46,580 --> 00:00:48,700: OneDrives, network drives, Confluence,
- 00:00:49,000 --> 00:00:50,720: all of these unstructured sources.
- 00:00:51,660 --> 00:00:53,700: So, so my team and I initially set out to solve that
- 00:00:53,760 --> 00:00:55,960: question: how do we make unstructured
- 00:00:56,020 --> 00:00:57,760: data AI-ready?
- 00:00:59,280 --> 00:01:01,340: Now, what does that mean? What does AI-ready mean in
- 00:01:01,360 --> 00:01:03,080: the context of unstructured data?
- 00:01:03,960 --> 00:01:06,240: In Daze, we really think about, uh, you know, two
- 00:01:06,300 --> 00:01:07,000: layers to that.
- 00:01:07,340 --> 00:01:09,620: The first is what we often call curation,
- 00:01:10,220 --> 00:01:12,280: meaning: how do we help companies figure
- 00:01:12,360 --> 00:01:14,560: out what do we even have in the first place?
- 00:01:14,980 --> 00:01:17,220: What files are out there? What files are up to date?
- 00:01:17,280 --> 00:01:18,480: What files are sensitive?
- 00:01:18,880 --> 00:01:20,140: What files are high quality?
- 00:01:20,860 --> 00:01:22,860: Such that when I have a new use case from
- 00:01:22,900 --> 00:01:25,100: the business, a new AI application I'm building,
- 00:01:25,500 --> 00:01:27,580: I can quickly filter down to
- 00:01:27,620 --> 00:01:29,740: the relevant subset of files that we
- 00:01:29,800 --> 00:01:31,240: want to use for that use case.
- 00:01:32,500 --> 00:01:34,540: But that's not enough, and as Felix mentioned, what we're
- 00:01:34,580 --> 00:01:36,280: now seeing more and more demand for is,
- 00:01:36,620 --> 00:01:38,800: how do I also include those files
- 00:01:38,860 --> 00:01:40,960: in a company-wide context layer
- 00:01:41,040 --> 00:01:41,940: in an ontology?
- 00:01:42,240 --> 00:01:44,320: How do I represent the data within those
- 00:01:44,380 --> 00:01:46,800: files in a way that is conducive
- 00:01:46,840 --> 00:01:48,660: to accurate and scalable retrieval?
- 00:01:49,800 --> 00:01:51,880: Okay, so today we're gonna be talking about both of those,
- 00:01:52,200 --> 00:01:54,720: um, but we're also very excited to emphasize
- 00:01:54,760 --> 00:01:55,300: the latter.
- 00:01:55,700 --> 00:01:57,980: Live Map is a new part of the Daze
- 00:01:58,040 --> 00:01:58,680: and Collibra
- 00:01:59,040 --> 00:02:01,200: value proposition that focuses on just that.
- 00:02:01,460 --> 00:02:03,380: How do we take the curated data,
- 00:02:03,820 --> 00:02:05,820: build an ontology, build that into our
- 00:02:05,880 --> 00:02:08,180: graph, and ultimately enable,
- 00:02:08,560 --> 00:02:11,100: uh, accurate, scalable retrieval across unstructured
- 00:02:11,160 --> 00:02:13,160: content as well as structured data?
- 00:02:15,000 --> 00:02:17,220: So, um, today we're gonna look at a story
- 00:02:17,280 --> 00:02:18,260: of a real customer,
- 00:02:18,540 --> 00:02:20,780: um, that we're gonna call Corvain Financial
- 00:02:20,860 --> 00:02:22,020: for this demonstration.
- 00:02:22,780 --> 00:02:24,800: So Corvain are a global asset management
- 00:02:24,840 --> 00:02:26,940: company. Huge volumes of call
- 00:02:27,000 --> 00:02:29,320: transcripts, risk memos, market reports,
- 00:02:29,680 --> 00:02:30,540: deal documents
- 00:02:31,040 --> 00:02:33,100: spread across SharePoint, OneDrive, and
- 00:02:33,160 --> 00:02:34,080: a few other systems.
- 00:02:35,340 --> 00:02:37,040: Like many, they're investing heavily in AI,
- 00:02:37,440 --> 00:02:39,540: um, and specifically, Corvain are interested in
- 00:02:39,620 --> 00:02:41,620: building a, an agent to
- 00:02:41,680 --> 00:02:43,760: accelerate their due diligence. So every
- 00:02:43,820 --> 00:02:46,240: time they're looking at buying a new asset,
- 00:02:46,520 --> 00:02:47,520: they want to understand,
- 00:02:47,820 --> 00:02:50,020: well, what do we already have in our portfolio?
- 00:02:50,300 --> 00:02:52,300: How can we look at all the thousands
- 00:02:52,320 --> 00:02:54,480: or millions of historic deals that we've done
- 00:02:54,540 --> 00:02:56,700: to accelerate and improve our decision-making?
- 00:02:58,820 --> 00:03:01,160: So Corvain actually already had an agent,
- 00:03:01,220 --> 00:03:03,520: this kind of due diligence agent, in production,
- 00:03:03,980 --> 00:03:06,160: but what they found is that the deals weren't moving
- 00:03:06,220 --> 00:03:08,540: faster. And really, you could diagnose
- 00:03:08,580 --> 00:03:09,580: this, uh, with two,
- 00:03:09,980 --> 00:03:11,800: uh, root causes. So one is that actually,
- 00:03:12,100 --> 00:03:14,480: when people were asking this agent questions
- 00:03:14,540 --> 00:03:15,200: about their data,
- 00:03:15,520 --> 00:03:18,060: it was extremely slow and very expensive,
- 00:03:18,240 --> 00:03:20,580: so around sixty-two thousand tokens for a given
- 00:03:20,640 --> 00:03:22,860: query. The second
- 00:03:22,920 --> 00:03:24,980: is what they quickly found was that a lot of the questions
- 00:03:25,020 --> 00:03:27,100: the business were interested in asking about this
- 00:03:27,140 --> 00:03:29,500: data, they couldn't answer with a very simple
- 00:03:29,540 --> 00:03:30,500: retrieval setup.
- 00:03:31,360 --> 00:03:33,480: So we'll look at a couple of those, uh, I guess, question
- 00:03:33,580 --> 00:03:34,220: archetypes.
- 00:03:35,140 --> 00:03:37,500: So, you know, what valuation did the committee
- 00:03:37,560 --> 00:03:39,660: approve for Ardane's Grid, which is one
- 00:03:39,680 --> 00:03:41,020: of their portfolio companies?
- 00:03:42,300 --> 00:03:44,400: A common problem I'm sure many people here will resonate
- 00:03:44,460 --> 00:03:46,680: with, um, often the model is unable
- 00:03:46,760 --> 00:03:47,760: to distinguish between,
- 00:03:48,180 --> 00:03:50,180: uh, you know, v final, v final final, v
- 00:03:50,240 --> 00:03:51,000: final final final,
- 00:03:51,440 --> 00:03:53,720: um, that I think we're all guilty of doing at some point.
- 00:03:54,320 --> 00:03:56,540: Uh, so of course, we struggle to find the latest version,
- 00:03:56,720 --> 00:03:58,020: and an agent will also.
- 00:03:59,680 --> 00:04:02,040: The second question: show me everything we know
- 00:04:02,120 --> 00:04:03,400: about Ardane's Grid.
- 00:04:04,900 --> 00:04:06,920: Some documents get surfaced, but a lot of
- 00:04:06,960 --> 00:04:08,960: really rich information is missed because
- 00:04:09,000 --> 00:04:09,760: we do not have,
- 00:04:10,120 --> 00:04:12,160: uh, connectivity represented between
- 00:04:12,240 --> 00:04:14,320: files, and we'll talk about that and the, the way
- 00:04:14,360 --> 00:04:15,900: that a graph can help overcome that.
- 00:04:17,100 --> 00:04:19,100: And finally, can you show me the full list of our
- 00:04:19,140 --> 00:04:20,140: high-risk customers?
- 00:04:20,940 --> 00:04:23,400: Of course, what one company defines as a high-risk customer
- 00:04:23,440 --> 00:04:25,580: can be totally different to another,
- 00:04:25,980 --> 00:04:26,960: uh, another company.
- 00:04:27,240 --> 00:04:29,360: Even the, the word customer can mean many things,
- 00:04:29,440 --> 00:04:31,800: right? So unless that definition,
- 00:04:31,960 --> 00:04:34,120: that context, has been embedded into this
- 00:04:34,160 --> 00:04:34,800: whole setup,
- 00:04:35,240 --> 00:04:37,300: uh, we can never expect the agent to retrieve the right
- 00:04:37,320 --> 00:04:37,880: information.
- 00:04:39,880 --> 00:04:42,000: So what would a kind of best-in-class context
- 00:04:42,060 --> 00:04:44,180: engineering approach look like? What would it look like
- 00:04:44,240 --> 00:04:46,240: to prepare that unstructured data
- 00:04:46,320 --> 00:04:48,420: in a way that overcomes those kinds of issues
- 00:04:48,460 --> 00:04:49,360: that we typically see?
- 00:04:50,280 --> 00:04:52,360: So again, two core steps here. The
- 00:04:52,420 --> 00:04:54,020: first is what we call curation.
- 00:04:54,740 --> 00:04:57,020: How do I go from a million files across
- 00:04:57,080 --> 00:04:58,380: my SharePoint, S3,
- 00:04:58,960 --> 00:05:00,920: uh, you know, Azure Blob, wherever that might be,
- 00:05:01,640 --> 00:05:03,960: and figure out the, the subset that I actually
- 00:05:04,020 --> 00:05:06,180: want to bring into my knowledge
- 00:05:06,240 --> 00:05:08,380: base, be it a RAG system, be
- 00:05:08,440 --> 00:05:10,160: it a, a knowledge graph that I'm building?
- 00:05:10,960 --> 00:05:12,840: And so there are different dimensions here.
- 00:05:13,540 --> 00:05:15,340: Some of them you see on the screen. Is this file
- 00:05:15,720 --> 00:05:17,740: unique? You know, is it the latest version? Is it
- 00:05:17,800 --> 00:05:19,980: fresh? Is it sensitive? Is it relevant?
- 00:05:20,280 --> 00:05:22,600: Is there conflicting information between files,
- 00:05:22,620 --> 00:05:24,160: and I first need to resolve that
- 00:05:24,640 --> 00:05:26,700: before I expect an agent to be able to make that
- 00:05:26,740 --> 00:05:27,340: distinction?
- 00:05:29,240 --> 00:05:31,520: And again, that's not enough. So the next step is, once
- 00:05:31,560 --> 00:05:33,600: we have the right data, how do we represent
- 00:05:33,660 --> 00:05:34,200: it in a way
- 00:05:34,580 --> 00:05:36,740: that is gonna lend itself to the best
- 00:05:36,800 --> 00:05:37,360: possible,
- 00:05:37,640 --> 00:05:39,320: most efficient and accurate retrieval?
- 00:05:40,080 --> 00:05:42,380: And that is where today we're very excited to announce
- 00:05:42,440 --> 00:05:43,440: Live Map officially,
- 00:05:43,920 --> 00:05:45,500: um, which is our ability to now
- 00:05:45,920 --> 00:05:47,840: basically take the unstructured data,
- 00:05:48,240 --> 00:05:49,680: apply an ontology,
- 00:05:50,180 --> 00:05:52,260: build a graph, and then expose that graph
- 00:05:52,320 --> 00:05:54,620: so an agent can retrieve from the unstructured
- 00:05:54,660 --> 00:05:55,740: data in runtime.
- 00:05:57,900 --> 00:05:59,940: Okay, so, um, I'm gonna go through a series of videos
- 00:06:00,000 --> 00:06:01,840: now to show you very concretely,
- 00:06:02,160 --> 00:06:03,140: uh, in a very tactical way,
- 00:06:03,460 --> 00:06:05,200: how would you do this in the product?
- 00:06:07,040 --> 00:06:07,920: All right. So, um,
- 00:06:08,180 --> 00:06:10,400: we're gonna start here in the SharePoint site.
- 00:06:10,460 --> 00:06:12,340: So this is Corvain Financial SharePoint site,
- 00:06:12,780 --> 00:06:14,380: probably a very familiar site to many.
- 00:06:14,920 --> 00:06:16,480: PDF, PowerPoint, Excel,
- 00:06:16,980 --> 00:06:18,580: um, multimodal data,
- 00:06:19,140 --> 00:06:21,560: so we're not just talking about plain text. There are tables,
- 00:06:21,660 --> 00:06:23,560: charts, graphics, images in this data,
- 00:06:24,420 --> 00:06:26,720: and a very distinct lack of metadata
- 00:06:26,780 --> 00:06:29,220: in here, uh, and, and very poor naming conventions
- 00:06:29,340 --> 00:06:31,760: as well. So,
- 00:06:31,980 --> 00:06:33,680: um, we're now gonna come to the Deezy
- 00:06:33,940 --> 00:06:35,960: module within Collibra, and we're gonna build a
- 00:06:36,020 --> 00:06:38,160: new project for this specific agent,
- 00:06:38,600 --> 00:06:40,840: uh, that the company are, uh, going to build.
- 00:06:41,540 --> 00:06:43,640: This is gonna be our financial analyst.
- 00:06:44,180 --> 00:06:46,180: And the first thing we do in a project is, is
- 00:06:46,220 --> 00:06:48,300: tell it what unstructured sources do we want
- 00:06:48,340 --> 00:06:48,860: to bring in.
- 00:06:49,640 --> 00:06:51,680: Now, in this case, it's a SharePoint site, but it could be
- 00:06:52,140 --> 00:06:54,240: hundreds of sites. It could be combining SharePoint
- 00:06:54,300 --> 00:06:56,680: data with S3 buckets and Confluence.
- 00:06:57,400 --> 00:06:59,400: Um, and we're gonna connect to the source and
- 00:06:59,460 --> 00:07:01,980: actually ingest the files and their metadata.
- 00:07:03,140 --> 00:07:04,780: So you see here what that source looks like.
- 00:07:05,060 --> 00:07:07,100: Um, so we have a list of all of the files,
- 00:07:07,460 --> 00:07:09,620: plus what we call extrinsic metadata.
- 00:07:10,040 --> 00:07:12,060: You know, when was the file modified? Who's the
- 00:07:12,140 --> 00:07:13,160: owner? What file type?
- 00:07:13,440 --> 00:07:15,560: So the kinda system-level information about
- 00:07:15,620 --> 00:07:16,320: every file.
- 00:07:17,700 --> 00:07:19,700: We've also OCR'd and chunked the
- 00:07:19,760 --> 00:07:20,680: data at this point
- 00:07:21,040 --> 00:07:23,500: so that we can actually generate more contextual
- 00:07:23,540 --> 00:07:24,960: metadata in the next step.
- 00:07:26,560 --> 00:07:28,600: So the system metadata isn't enough. What we now
- 00:07:28,620 --> 00:07:31,240: want to do is actually tag the data with contextual
- 00:07:31,640 --> 00:07:33,440: metadata describing the content,
- 00:07:34,620 --> 00:07:36,960: and we do this in the product using a taxonomy.
- 00:07:37,020 --> 00:07:39,060: So the taxonomy is comprised of these building
- 00:07:39,120 --> 00:07:40,240: blocks that we call tags,
- 00:07:40,700 --> 00:07:42,220: and a tag can really be anything.
- 00:07:42,720 --> 00:07:44,840: So we have, you know, what confidentiality level is this
- 00:07:44,880 --> 00:07:46,580: document? What deal country was it?
- 00:07:46,880 --> 00:07:48,760: What was the investment stage of the document?
- 00:07:49,440 --> 00:07:51,520: What type of document is it? You know, simply, is
- 00:07:51,540 --> 00:07:53,180: it HR, marketing, finance?
- 00:07:53,860 --> 00:07:55,900: But then we go in a hierarchical manner to say, "Well, if
- 00:07:55,960 --> 00:07:58,180: it's, uh, an investment document, what type
- 00:07:58,220 --> 00:07:59,460: of investment document is it?"
- 00:08:00,040 --> 00:08:02,200: And we can keep going deeper and deeper to create
- 00:08:02,260 --> 00:08:03,960: more and more granular metadata.
- 00:08:05,640 --> 00:08:06,740: And the idea is that we're gonna,
- 00:08:07,040 --> 00:08:08,320: in an automated fashion,
- 00:08:08,600 --> 00:08:11,160: effectively map all of our unstructured content
- 00:08:11,240 --> 00:08:13,380: onto this, uh, data model, if you like, to
- 00:08:13,420 --> 00:08:14,840: build a structured database.
- 00:08:16,160 --> 00:08:17,520: Now, how do you build this taxonomy?
- 00:08:18,060 --> 00:08:18,920: There's a few ways.
- 00:08:19,320 --> 00:08:21,340: Um, anyone who joined the workshop this morning will know that
- 00:08:21,380 --> 00:08:23,600: you can create tags yourself using LLM-based
- 00:08:23,660 --> 00:08:24,640: tagging or patterns.
- 00:08:25,560 --> 00:08:27,560: We have a lot of pre-trained classifiers that you
- 00:08:27,600 --> 00:08:28,640: can use out the box.
- 00:08:29,560 --> 00:08:31,920: You could use an ML algorithm to auto-suggest
- 00:08:32,000 --> 00:08:33,320: a taxonomy from your data,
- 00:08:34,680 --> 00:08:36,740: or you could import your taxonomy
- 00:08:36,780 --> 00:08:38,560: or an ontology from Collibra.
- 00:08:38,980 --> 00:08:41,020: And I think, you know, where we see a lot of companies going
- 00:08:41,080 --> 00:08:43,240: is this unified context layer
- 00:08:43,320 --> 00:08:45,440: across structured and unstructured data.
- 00:08:45,860 --> 00:08:48,020: And so increasingly, we're seeing the kind of common
- 00:08:48,080 --> 00:08:50,080: flow here to be importing your
- 00:08:50,140 --> 00:08:52,260: ontology that you're using for structured data,
- 00:08:52,560 --> 00:08:54,440: and then using that as the basis for what,
- 00:08:54,820 --> 00:08:57,120: uh, uh, gets tagged on the unstructured side.
- 00:08:58,140 --> 00:09:00,160: So once we've built our taxonomy,
- 00:09:00,460 --> 00:09:02,480: what we can now do is leverage various
- 00:09:02,520 --> 00:09:04,880: different AI techniques to start mapping
- 00:09:04,960 --> 00:09:06,400: all of our unstructured data
- 00:09:06,880 --> 00:09:08,380: to this particular data
- 00:09:08,780 --> 00:09:10,220: model, if you like, or this taxonomy.
- 00:09:11,420 --> 00:09:13,700: So here I'm selecting, you know, I want to tag my entire
- 00:09:13,740 --> 00:09:16,360: SharePoint site against this taxonomy,
- 00:09:17,440 --> 00:09:19,940: and what that's gonna produce is basically a database
- 00:09:20,020 --> 00:09:22,340: view of all of my unstructured content.
- 00:09:23,320 --> 00:09:24,900: So each row here is one of our files,
- 00:09:25,700 --> 00:09:27,780: and the AI models have looked through every page of every
- 00:09:27,840 --> 00:09:30,160: file, um, and actually generated
- 00:09:30,200 --> 00:09:32,240: that contextual metadata. Okay? So that's
- 00:09:32,260 --> 00:09:34,340: what we're seeing here. So we might say, for example,
- 00:09:34,800 --> 00:09:36,980: "I now want to filter and see only
- 00:09:37,040 --> 00:09:38,720: files that are an investment document
- 00:09:39,660 --> 00:09:41,940: related to the real estate industry,
- 00:09:43,160 --> 00:09:44,640: specifically in the US."
- 00:09:45,600 --> 00:09:47,720: And for the first time ever, for most customers, they can
- 00:09:47,800 --> 00:09:49,280: start viewing their unstructured
- 00:09:49,600 --> 00:09:51,360: landscape in a very structured form
- 00:09:51,820 --> 00:09:53,980: that allows them to quickly filter and find whatever
- 00:09:54,040 --> 00:09:56,060: files might be relevant for their use case.
- 00:09:57,860 --> 00:10:00,320: There's a whole process here to fine-tune to ensure accuracy
- 00:10:00,340 --> 00:10:01,520: and confidence in the tagging,
- 00:10:01,860 --> 00:10:04,100: again, uh, workshop folks will be very familiar
- 00:10:04,140 --> 00:10:04,760: with now too.
- 00:10:07,580 --> 00:10:09,680: Now, creating that contextual metadata is one step,
- 00:10:09,760 --> 00:10:12,040: but, you know, as we've already alluded to, quality
- 00:10:12,080 --> 00:10:13,640: of files is a key challenge. So
- 00:10:13,980 --> 00:10:16,060: the next step in the product is to start
- 00:10:16,120 --> 00:10:18,280: thinking about key unstructured data
- 00:10:18,360 --> 00:10:19,440: quality dimensions,
- 00:10:20,160 --> 00:10:22,300: two of the big culprits being uniqueness
- 00:10:22,340 --> 00:10:23,080: and freshness.
- 00:10:24,340 --> 00:10:26,720: So in Deezy, we can run an automated scan
- 00:10:26,780 --> 00:10:29,040: to identify exact or potential
- 00:10:29,100 --> 00:10:29,720: duplicates
- 00:10:30,560 --> 00:10:32,680: and suggest to you what we think the latest
- 00:10:32,740 --> 00:10:34,960: version is based on things like last modified
- 00:10:35,000 --> 00:10:37,040: dates. So you see here
- 00:10:37,080 --> 00:10:39,440: the human-in-the-loop workflow. Uh, we, we suggest
- 00:10:39,480 --> 00:10:41,480: based on the content, these two files we think
- 00:10:41,500 --> 00:10:42,160: are duplicates.
- 00:10:42,500 --> 00:10:44,840: And by the way, we think this one on the left
- 00:10:44,940 --> 00:10:46,000: is the latest version
- 00:10:46,580 --> 00:10:48,620: based on, again, like, the system metadata
- 00:10:48,660 --> 00:10:49,400: that we can read in.
- 00:10:50,780 --> 00:10:52,820: And what that's gonna do is it's gonna create tags on every
- 00:10:52,880 --> 00:10:54,940: file to say, "This is latest version. This file
- 00:10:54,980 --> 00:10:56,040: is likely redundant."
- 00:10:57,500 --> 00:11:00,000: Then we move to freshness. So here we can see a distribution
- 00:11:00,060 --> 00:11:02,240: of, you know, created at dates, uh, dates
- 00:11:02,280 --> 00:11:03,300: from within the documents,
- 00:11:03,660 --> 00:11:05,600: and we can start to set cutoffs to say,
- 00:11:06,080 --> 00:11:08,320: "Well, I never really want to bring in any file into this AI
- 00:11:08,400 --> 00:11:10,420: use case that is over X years
- 00:11:10,500 --> 00:11:12,700: old or hasn't been touched or edited in,
- 00:11:12,740 --> 00:11:13,900: you know, however many years."
- 00:11:16,080 --> 00:11:18,680: So what we're doing at this stage is we're applying criteria
- 00:11:19,040 --> 00:11:21,080: so that we can start filtering down the
- 00:11:21,160 --> 00:11:23,160: massive SharePoint content into a
- 00:11:23,260 --> 00:11:24,180: smaller subset,
- 00:11:24,560 --> 00:11:26,280: um, that is gonna be, uh, you know,
- 00:11:26,540 --> 00:11:28,340: relevant for this particular use case.
- 00:11:30,980 --> 00:11:33,340: Now, relevance is an interesting one. Relevance obviously
- 00:11:33,360 --> 00:11:35,460: is very use case dependent, and here for this due
- 00:11:35,500 --> 00:11:36,560: diligence agent,
- 00:11:37,000 --> 00:11:39,360: I only want f- investment documents
- 00:11:39,420 --> 00:11:40,120: to be included,
- 00:11:40,860 --> 00:11:43,200: but I also only want investment documents from companies
- 00:11:43,220 --> 00:11:44,900: that are actually in my portfolio.
- 00:11:45,880 --> 00:11:48,000: So the nice thing with the contextual tagging is you can
- 00:11:48,020 --> 00:11:50,260: then start filtering your data estate in whatever
- 00:11:50,320 --> 00:11:52,600: way is meaningful for your particular
- 00:11:52,680 --> 00:11:55,440: use case. Then
- 00:11:55,540 --> 00:11:57,760: we can create what we call a data slice.
- 00:11:58,260 --> 00:12:00,260: So instead of now looking at our millions
- 00:12:00,300 --> 00:12:01,260: of files in SharePoint,
- 00:12:01,700 --> 00:12:03,700: we're only filtering down to kind of like a new
- 00:12:03,760 --> 00:12:04,460: data source,
- 00:12:04,840 --> 00:12:05,880: which just contains
- 00:12:06,260 --> 00:12:07,600: investment and deal documents
- 00:12:07,940 --> 00:12:10,320: from portfolio companies that are unique
- 00:12:10,360 --> 00:12:11,120: and that are fresh.
- 00:12:12,020 --> 00:12:14,320: And we can see we actually only have sixty files
- 00:12:14,400 --> 00:12:16,740: across our entire landscape that meet the criteria
- 00:12:17,100 --> 00:12:18,480: for that particular use case.
- 00:12:21,500 --> 00:12:23,500: So we've kind of walked through, uh, you know,
- 00:12:23,540 --> 00:12:25,320: very quickly what that first step looks like,
- 00:12:25,600 --> 00:12:27,740: filtering down huge volumes of data
- 00:12:27,840 --> 00:12:29,580: to a relevant, high quality,
- 00:12:30,080 --> 00:12:32,080: also safety sensitivity is a big part of
- 00:12:32,120 --> 00:12:32,680: that tagging,
- 00:12:33,080 --> 00:12:35,440: um, subset that we want to bring into
- 00:12:35,480 --> 00:12:36,400: our AI system.
- 00:12:38,400 --> 00:12:40,460: So now we're gonna talk about the next step, which is, well, how
- 00:12:40,500 --> 00:12:42,640: do I represent key relationships
- 00:12:43,000 --> 00:12:44,300: across those files?
- 00:12:44,660 --> 00:12:46,660: Right now, we have a flat database, but as we
- 00:12:46,720 --> 00:12:47,540: know, you know,
- 00:12:47,900 --> 00:12:48,500: the most, uh,
- 00:12:48,860 --> 00:12:50,420: accurate and efficient retrieval systems
- 00:12:50,760 --> 00:12:53,000: need a graphical representation of the data.
- 00:12:53,100 --> 00:12:55,180: So what I'm now gonna do, and this is the, the new piece of
- 00:12:55,200 --> 00:12:57,520: the puzzle, um, is to create a Live
- 00:12:57,600 --> 00:12:58,940: Map within Deezy.
- 00:12:59,880 --> 00:13:02,340: And I'm gonna create that Live Map on my AI-ready
- 00:13:02,400 --> 00:13:03,900: files from that first step.
- 00:13:05,260 --> 00:13:07,760: So when I build a Live Map, there's really two inputs.
- 00:13:08,040 --> 00:13:09,360: The first is what files,
- 00:13:09,680 --> 00:13:11,740: and we've already now curated the files that we want to
- 00:13:11,780 --> 00:13:12,480: come into that.
- 00:13:13,360 --> 00:13:14,800: The second is an ontology.
- 00:13:15,920 --> 00:13:18,440: Now, that ontology ideally is the same ontology
- 00:13:18,460 --> 00:13:20,720: we're using for our structured data, so we can start
- 00:13:20,800 --> 00:13:22,640: mapping these two worlds against one
- 00:13:23,100 --> 00:13:24,560: unified semantic model.
- 00:13:25,560 --> 00:13:27,980: So this could be pulled in from your Collibra
- 00:13:28,040 --> 00:13:30,300: instance, uh, from other tools where,
- 00:13:30,360 --> 00:13:32,660: as Felix mentioned, you may be managing your ontologies.
- 00:13:33,520 --> 00:13:35,620: And you can see here an ontology where we have
- 00:13:35,660 --> 00:13:37,860: things like, you know, deal and company
- 00:13:37,940 --> 00:13:38,460: names,
- 00:13:38,880 --> 00:13:40,860: uh, regulator, risk category.
- 00:13:41,440 --> 00:13:43,480: And this is again where we're making sure
- 00:13:43,520 --> 00:13:45,760: that what we define as risk categories
- 00:13:45,840 --> 00:13:48,500: as a company is encoded in the ontology.
- 00:13:50,620 --> 00:13:52,640: What we can then do in a fully automated manner
- 00:13:52,720 --> 00:13:53,860: once we have the ontology,
- 00:13:54,260 --> 00:13:56,280: we can effectively map our unstructured
- 00:13:56,320 --> 00:13:57,140: data onto it
- 00:13:57,940 --> 00:13:58,840: to start building,
- 00:13:59,100 --> 00:14:01,300: uh, what is effectively a knowledge graph here.
- 00:14:02,300 --> 00:14:04,480: So when I import my f- my curated
- 00:14:04,520 --> 00:14:05,080: dataset,
- 00:14:05,540 --> 00:14:07,640: uh, and choose that ontology, what you're gonna see happen
- 00:14:07,680 --> 00:14:09,980: on the screen is we're gonna start automatically
- 00:14:10,040 --> 00:14:12,020: building that graph for you.
- 00:14:13,640 --> 00:14:15,740: Now, I just had a conversation with someone where they're asking, "How
- 00:14:15,820 --> 00:14:17,860: do, how does unstructured data get represented in
- 00:14:17,900 --> 00:14:19,900: a graph?" The key piece is
- 00:14:19,939 --> 00:14:22,180: that each of these nodes is a different entity,
- 00:14:22,439 --> 00:14:23,940: right? So each of these colored blobs,
- 00:14:24,200 --> 00:14:26,300: they might represent companies in our
- 00:14:26,340 --> 00:14:28,680: portfolio. They might represent a deal
- 00:14:28,720 --> 00:14:30,480: that we've done. They might represent a fund.
- 00:14:30,860 --> 00:14:32,940: They might represent an important financial metric. They
- 00:14:32,980 --> 00:14:34,100: might represent a region.
- 00:14:35,280 --> 00:14:37,760: And you see all of the connections are being automatically
- 00:14:37,800 --> 00:14:39,840: built across this graph, and
- 00:14:39,880 --> 00:14:42,000: this is for our unstructured data.
- 00:14:44,500 --> 00:14:46,620: So let's have a closer look at what that graph actually
- 00:14:46,660 --> 00:14:48,660: contains. Obviously, there's a lot of information here,
- 00:14:49,200 --> 00:14:51,460: but the powerful piece is that we might want to now
- 00:14:51,560 --> 00:14:53,700: filter and, let's say, just look at all
- 00:14:53,740 --> 00:14:56,000: of the nodes that represent companies
- 00:14:56,060 --> 00:14:57,040: in our portfolio.
- 00:14:58,720 --> 00:15:01,380: So all of the green blobs are companies
- 00:15:01,420 --> 00:15:03,420: that we have invested in. So you can see all of
- 00:15:03,460 --> 00:15:04,340: the companies now.
- 00:15:05,140 --> 00:15:07,400: Again, our definition of a company, meaning company
- 00:15:07,420 --> 00:15:08,220: we've invested in.
- 00:15:08,960 --> 00:15:10,640: And if I click on any one of them, you're gonna see,
- 00:15:10,900 --> 00:15:13,280: how does that company relate to the rest of the knowledge?
- 00:15:13,840 --> 00:15:15,960: You know, what other documents reference that company?
- 00:15:16,020 --> 00:15:18,440: What are the key metrics or facts about that company?
- 00:15:18,720 --> 00:15:20,960: What fund was that company backed by?
- 00:15:21,700 --> 00:15:23,900: And all of this is now represented centrally
- 00:15:23,980 --> 00:15:24,680: in this graph.
- 00:15:27,600 --> 00:15:29,800: So in the case of, uh, Corvain Financial,
- 00:15:29,860 --> 00:15:31,600: what, what does this graph unlock for them?
- 00:15:33,700 --> 00:15:34,340: So, um,
- 00:15:34,860 --> 00:15:36,860: the, the question is now: How do we start using this?
- 00:15:36,920 --> 00:15:37,760: How do we query it?
- 00:15:38,260 --> 00:15:40,460: And w- within Live Map, we have an MCP
- 00:15:40,540 --> 00:15:41,100: server, so
- 00:15:41,440 --> 00:15:43,660: you could, you know, uh, use other tools
- 00:15:43,720 --> 00:15:45,780: you have, other pipelines to interact with this graph.
- 00:15:46,140 --> 00:15:48,260: Um, but here I'm gonna show you using our MCP
- 00:15:48,300 --> 00:15:50,340: directly to start asking a question across this
- 00:15:50,400 --> 00:15:52,880: graph. Now, the question,
- 00:15:52,960 --> 00:15:55,080: uh, here is for Corvain Financial,
- 00:15:55,400 --> 00:15:57,600: they're considering a new investment in a company
- 00:15:57,660 --> 00:15:58,960: called Ferro Logistics.
- 00:15:59,640 --> 00:16:01,680: They wanna know, based on our current portfolio,
- 00:16:02,480 --> 00:16:04,640: are there any risks in investing in Ferro
- 00:16:04,700 --> 00:16:05,400: Logistics?
- 00:16:06,860 --> 00:16:08,960: And when we ask that question, um, I don't expect
- 00:16:09,020 --> 00:16:10,280: anyone to be reading all of this,
- 00:16:10,600 --> 00:16:12,440: um, but the key piece is this.
- 00:16:12,700 --> 00:16:14,920: It says, "There's a non-obvious
- 00:16:15,000 --> 00:16:17,380: correlated risk in the knowledge graph that connects
- 00:16:17,420 --> 00:16:19,740: Ferro Logistics to two other
- 00:16:20,280 --> 00:16:21,080: private, um,
- 00:16:21,360 --> 00:16:23,380: credit exposures held in the portfolio."
- 00:16:24,620 --> 00:16:26,620: So it goes on to explain what those are, but the key
- 00:16:26,680 --> 00:16:28,759: piece, uh, the key takeaway here is this:
- 00:16:29,600 --> 00:16:31,639: Ferro Logistics has basically two customers,
- 00:16:32,100 --> 00:16:34,100: and those two customers owe money somewhere
- 00:16:34,160 --> 00:16:35,160: else in the portfolio.
- 00:16:36,640 --> 00:16:38,940: That is a fact about Ferro Logistics that
- 00:16:39,020 --> 00:16:41,400: all of the documentation that say Ferro Logistics
- 00:16:41,440 --> 00:16:42,640: never would have surfaced.
- 00:16:43,060 --> 00:16:45,340: This is the kind of fact that is only surfaced
- 00:16:45,420 --> 00:16:47,480: when you are able to draw those links between the
- 00:16:47,540 --> 00:16:50,280: files. And
- 00:16:50,320 --> 00:16:52,360: this is what a graph allows. It allows kind of much
- 00:16:52,400 --> 00:16:54,800: more nuanced understanding, um, of the content,
- 00:16:54,860 --> 00:16:56,000: the relationships between them,
- 00:16:56,440 --> 00:16:58,640: um, that allow kind of more complex queries
- 00:16:58,680 --> 00:16:59,380: to be answered.
- 00:17:01,560 --> 00:17:03,760: So we've gone through those two steps, curating the right
- 00:17:03,820 --> 00:17:05,960: files and then building a graph
- 00:17:06,020 --> 00:17:08,080: on top of that curated dataset using
- 00:17:08,120 --> 00:17:10,560: Live Map. There
- 00:17:10,600 --> 00:17:12,880: is, of course, a third step, 'cause everything
- 00:17:12,940 --> 00:17:14,420: is in threes, which is that
- 00:17:15,020 --> 00:17:17,020: we want to then maintain that. So both the
- 00:17:17,100 --> 00:17:19,160: files that are coming in, so every day, you know,
- 00:17:19,360 --> 00:17:21,180: new file is being created on SharePoint.
- 00:17:21,580 --> 00:17:23,800: How do we make sure that curated dataset stays
- 00:17:23,860 --> 00:17:26,020: up to date? How do we then bring the right files
- 00:17:26,060 --> 00:17:27,820: into the graph and maintain the graph?
- 00:17:28,280 --> 00:17:30,300: That whole system has to be,
- 00:17:30,400 --> 00:17:32,040: uh, running on a continuous basis,
- 00:17:32,360 --> 00:17:34,400: so the last piece of Deezy is to then set that up
- 00:17:34,440 --> 00:17:36,700: as a pipeline that will manage new files
- 00:17:36,740 --> 00:17:38,660: being created on an ongoing basis.
- 00:17:41,400 --> 00:17:43,700: And what does that mean? So what is the impact of following
- 00:17:43,760 --> 00:17:45,920: this kind of approach to, uh, context
- 00:17:45,980 --> 00:17:46,540: engineering?
- 00:17:48,020 --> 00:17:48,920: As we've said, uh,
- 00:17:49,240 --> 00:17:51,260: the, the nice thing about a graph is it provides a
- 00:17:51,320 --> 00:17:53,420: much more efficient way for an AI system
- 00:17:53,440 --> 00:17:55,660: or an agent to traverse the knowledge without
- 00:17:55,720 --> 00:17:57,740: having to go and read the content of every
- 00:17:57,800 --> 00:18:00,060: file. So token consumption
- 00:18:00,100 --> 00:18:01,060: is far more efficient.
- 00:18:01,320 --> 00:18:03,320: Here we went from sixty-two thousand tokens
- 00:18:03,360 --> 00:18:03,900: per query,
- 00:18:04,360 --> 00:18:06,420: um, down to thirteen thousand tokens per
- 00:18:06,460 --> 00:18:08,820: query. Those more complex
- 00:18:08,860 --> 00:18:11,200: questions that rely on a nuanced understanding
- 00:18:11,280 --> 00:18:13,400: as to how our data is interconnected can
- 00:18:13,440 --> 00:18:14,360: now be answered.
- 00:18:15,340 --> 00:18:17,000: We overcome issues with versioning,
- 00:18:17,400 --> 00:18:19,300: uh, by having that first curation step.
- 00:18:19,600 --> 00:18:21,600: And so, uh, in the end, it means that the
- 00:18:21,660 --> 00:18:23,880: agent gets much better adoption within the company,
- 00:18:24,300 --> 00:18:25,240: and in this use case,
- 00:18:25,500 --> 00:18:27,940: the deals are accelerated quite significantly.
- 00:18:30,840 --> 00:18:31,480: Okay. So,
- 00:18:31,800 --> 00:18:32,700: um, if
- 00:18:33,060 --> 00:18:33,980: that sounds relevant,
- 00:18:34,340 --> 00:18:36,500: um, if this idea of building a
- 00:18:36,560 --> 00:18:38,560: context layer that is bringing in the unstructured
- 00:18:38,600 --> 00:18:40,440: data along with the structured data,
- 00:18:40,860 --> 00:18:43,000: applying the ontology there, and allowing for more
- 00:18:43,040 --> 00:18:44,420: efficient, accurate retrieval,
- 00:18:44,960 --> 00:18:47,040: we do still have a couple of spaces left in the
- 00:18:47,080 --> 00:18:48,460: Live Map Design Partnership.
- 00:18:48,860 --> 00:18:51,020: You probably have, like, ten seconds now to scan
- 00:18:51,040 --> 00:18:52,040: [chuckles] the QR code.
- 00:18:52,340 --> 00:18:53,620: I'll let you do that. Otherwise,
- 00:18:53,980 --> 00:18:55,440: come and chat to our team today,
- 00:18:55,980 --> 00:18:58,040: um, and we can talk to you about how this
- 00:18:58,080 --> 00:18:58,980: program works.
- 00:18:59,380 --> 00:19:01,580: Uh, and, and you-- It's an opportunity to work
- 00:19:01,640 --> 00:19:04,160: closely with our founding team and help shape the direction,
- 00:19:04,680 --> 00:19:06,960: uh, of, uh, enterprise ontologies, and
- 00:19:07,060 --> 00:19:09,160: specifically bringing together structured
- 00:19:09,220 --> 00:19:11,100: and unstructured data in one
- 00:19:11,560 --> 00:19:12,420: context layer.
- 00:19:15,240 --> 00:19:17,160: All right. So, um,
- 00:19:17,940 --> 00:19:19,920: with that, uh, we've talked a lot here about,
- 00:19:20,200 --> 00:19:22,320: you know, creating context, feeding that to
- 00:19:22,380 --> 00:19:24,420: agents, allowing runtime retrieval.
- 00:19:24,840 --> 00:19:25,600: Um, but of course,
- 00:19:26,260 --> 00:19:27,700: there is another big theme, which is
- 00:19:28,120 --> 00:19:29,540: data governance, uh,
- 00:19:29,840 --> 00:19:32,180: the definition of governance, I guess, being expanded
- 00:19:32,220 --> 00:19:34,260: from data to increasingly, as Felix
- 00:19:34,300 --> 00:19:36,480: mentioned, your AI systems and your agents.
- 00:19:36,540 --> 00:19:38,560: How do we actually govern the underlying
- 00:19:38,620 --> 00:19:40,660: data, the underlying models and AI
- 00:19:40,720 --> 00:19:42,360: systems, not just the data?
Introducing Live Map: From unstructured data to curated, connected context
Most enterprise agents can search documents. But sometimes they still fail because the answer spans several files and nothing tells the agent they're related. Live Map builds on data Deasy has already curated, with duplicates gone, drafts filtered out and sensitive files held back, then maps how everything your business knows fits together. Join co-founder Reece Griffiths for the launch and a live build, and watch an agent answer the questions it couldn't without a context graph.
Video: Build trusted AI agents, fast: The power of governed context
Transcript
- 00:00:05,900 --> 00:00:08,320: I'm sure you agree with me after hearing Felix
- 00:00:08,360 --> 00:00:09,100: and Reese
- 00:00:09,420 --> 00:00:11,520: and Mittal and our dear
- 00:00:11,600 --> 00:00:12,980: customer, AstraZeneca,
- 00:00:13,740 --> 00:00:15,560: uh, that these are exciting times
- 00:00:15,840 --> 00:00:18,280: and that we are up for new challenges
- 00:00:18,380 --> 00:00:19,340: with all these AI
- 00:00:19,660 --> 00:00:20,980: agents that will automate,
- 00:00:21,660 --> 00:00:23,020: uh, the work that we used to do.
- 00:00:23,540 --> 00:00:25,880: And personally, I felt it as a product manager.
- 00:00:26,440 --> 00:00:28,620: I use AI, it helps me generate,
- 00:00:28,940 --> 00:00:29,880: uh, prototypes,
- 00:00:30,360 --> 00:00:31,940: which I used to be dependent on other
- 00:00:32,259 --> 00:00:34,380: colleagues in order to build those prototypes for
- 00:00:34,420 --> 00:00:36,440: me. So definitely, I can say
- 00:00:36,500 --> 00:00:38,720: that AI made me more effective
- 00:00:39,080 --> 00:00:39,720: and productive
- 00:00:40,260 --> 00:00:41,180: as a product manager.
- 00:00:41,960 --> 00:00:44,060: So quick question or quick test
- 00:00:44,120 --> 00:00:46,180: here. Who in this room already
- 00:00:46,620 --> 00:00:48,760: has been working with AI and
- 00:00:48,840 --> 00:00:49,620: AI agents
- 00:00:49,980 --> 00:00:51,060: and can safely say
- 00:00:51,420 --> 00:00:52,520: that it made you
- 00:00:52,920 --> 00:00:54,760: more productive for certain tasks?
- 00:00:57,200 --> 00:00:58,480: Okay, that's a lot of hands.
- 00:00:58,780 --> 00:01:01,340: Please keep your hand up if you always trusted
- 00:01:01,660 --> 00:01:03,040: the outcome of these agents.
- 00:01:04,860 --> 00:01:06,500: Yeah, that's the effect that I wanted.
- 00:01:06,940 --> 00:01:09,240: So this gap between the people that raised their hand
- 00:01:09,320 --> 00:01:09,940: and the people that
- 00:01:10,940 --> 00:01:11,540: lowered their hand
- 00:01:12,640 --> 00:01:14,860: is really what I'm gonna talk about. Like, why
- 00:01:14,900 --> 00:01:16,460: do we lose trust in agents?
- 00:01:17,420 --> 00:01:19,420: I'm Tom De Jonge, as Chirag already mentioned.
- 00:01:19,800 --> 00:01:20,340: Uh, I,
- 00:01:20,760 --> 00:01:22,140: uh, am part of the product team.
- 00:01:22,400 --> 00:01:24,560: I'm responsible for data governance, data
- 00:01:24,600 --> 00:01:26,060: quality, but now also
- 00:01:26,340 --> 00:01:27,420: context governance.
- 00:01:28,900 --> 00:01:30,500: So today, we're gonna talk about the fact
- 00:01:30,960 --> 00:01:32,960: or try to understand why we lose trust in
- 00:01:33,020 --> 00:01:35,080: agents. We're gonna talk
- 00:01:35,120 --> 00:01:35,780: about how
- 00:01:36,080 --> 00:01:38,140: we're building a solution in order to
- 00:01:38,220 --> 00:01:40,060: increase that trust in your agents,
- 00:01:40,460 --> 00:01:42,760: so that you can answer the question that Felix asked
- 00:01:43,600 --> 00:01:45,440: as to how many agents do we have. And
- 00:01:46,040 --> 00:01:48,060: I'll add a second question, how many context
- 00:01:48,140 --> 00:01:49,680: files do these agents run on?
- 00:01:50,040 --> 00:01:51,480: If we can answer those two questions,
- 00:01:51,840 --> 00:01:54,180: then I think we're in a much better position going forward.
- 00:01:54,720 --> 00:01:57,040: And third, we're gonna talk about how we use AI
- 00:01:57,080 --> 00:01:57,720: at Collibra
- 00:01:58,460 --> 00:01:59,280: for you to
- 00:01:59,760 --> 00:02:00,780: make your job as
- 00:02:01,180 --> 00:02:01,820: governance,
- 00:02:02,640 --> 00:02:03,700: uh, much more
- 00:02:04,320 --> 00:02:06,340: faster, automated, and dare I say,
- 00:02:06,560 --> 00:02:07,520: fun, right?
- 00:02:08,060 --> 00:02:10,080: So, uh, let me start with a
- 00:02:10,120 --> 00:02:10,639: true story.
- 00:02:10,940 --> 00:02:11,620: This is Dwight.
- 00:02:12,000 --> 00:02:14,220: Dwight is-- and Felix already mentioned this, actually.
- 00:02:14,300 --> 00:02:16,040: Dwight is our analytics agent.
- 00:02:16,780 --> 00:02:17,660: Um, I use it
- 00:02:17,920 --> 00:02:18,480: almost every...
- 00:02:18,820 --> 00:02:20,940: Well, every day, maybe not, but every week for sure.
- 00:02:21,620 --> 00:02:23,620: Every time I have a customer call, I basically
- 00:02:23,720 --> 00:02:24,580: ask Dwight, "Hey,
- 00:02:24,900 --> 00:02:26,900: can you tell me what I need to know about this customer?
- 00:02:26,960 --> 00:02:28,160: What products are they using?
- 00:02:28,480 --> 00:02:30,160: Are there any major issues outstanding
- 00:02:30,460 --> 00:02:31,700: that I need to be aware of?"
- 00:02:32,240 --> 00:02:34,740: Right? And Dwight does an amazing job. It informs
- 00:02:34,800 --> 00:02:36,720: me on everything that I need to know.
- 00:02:37,980 --> 00:02:39,560: So what's the story behind Dwight?
- 00:02:40,280 --> 00:02:41,200: Some really smart,
- 00:02:41,560 --> 00:02:42,980: uh, data analytics,
- 00:02:43,460 --> 00:02:45,460: uh, people in Collibra, and also
- 00:02:45,520 --> 00:02:46,960: we have a really lean, small
- 00:02:47,360 --> 00:02:48,080: analytics team.
- 00:02:48,660 --> 00:02:50,940: They basically said, like, "Let's modernize our stack."
- 00:02:51,280 --> 00:02:53,380: They were fed up building Tableau reports over and
- 00:02:53,420 --> 00:02:54,600: over again for every business
- 00:02:55,080 --> 00:02:57,240: problem or business insight that they needed to do.
- 00:02:57,320 --> 00:02:59,320: So let's make it more self-serve, so people
- 00:02:59,360 --> 00:03:01,360: can just ask questions and AI will help them.
- 00:03:02,160 --> 00:03:04,400: So what they did is they basically used Claude
- 00:03:04,480 --> 00:03:06,600: code, and they told Claude code,
- 00:03:06,640 --> 00:03:09,260: "Hey, reverse engineer our dat-data pipelines.
- 00:03:09,820 --> 00:03:11,640: Uh, look at all our Tableau dashboards.
- 00:03:12,440 --> 00:03:14,440: Look at all these manuals that we have in
- 00:03:14,480 --> 00:03:16,500: Confluence about these reports."
- 00:03:17,380 --> 00:03:19,480: And also what they did, and I thought that was
- 00:03:19,540 --> 00:03:20,320: really smart, they
- 00:03:20,600 --> 00:03:22,660: even did Zoom calls with the people that
- 00:03:22,700 --> 00:03:24,720: were using these reports to understand what
- 00:03:24,780 --> 00:03:26,400: insights they get from these reports.
- 00:03:27,600 --> 00:03:29,780: So Claude did what Claude does amazingly well,
- 00:03:29,940 --> 00:03:32,040: is it generated a whole bunch of context
- 00:03:32,120 --> 00:03:33,280: files, markdown files,
- 00:03:33,900 --> 00:03:36,380: and they basically put that into an analytics
- 00:03:36,420 --> 00:03:38,000: harness, and you had Dwight.
- 00:03:38,480 --> 00:03:40,500: And it was amazing. Really, the demo was
- 00:03:40,800 --> 00:03:42,280: super cool, and everybody loved it.
- 00:03:42,700 --> 00:03:44,240: Leadership signed off on it.
- 00:03:44,780 --> 00:03:46,820: So-- and we started adding more use cases,
- 00:03:47,540 --> 00:03:49,600: right? But gradually,
- 00:03:50,380 --> 00:03:51,820: the team started to hit a wall.
- 00:03:53,080 --> 00:03:54,580: Right? This is a quote that
- 00:03:54,840 --> 00:03:56,940: Lazar, I really love Lazar, he's
- 00:03:56,960 --> 00:03:58,220: an amazing guy to work with.
- 00:03:58,820 --> 00:04:00,460: He's, uh, one of our AI engineers.
- 00:04:01,140 --> 00:04:02,980: He sent a note, a Slack message
- 00:04:03,360 --> 00:04:05,360: to Stijn, one of our co-founders who
- 00:04:05,400 --> 00:04:06,480: leads the data office team,
- 00:04:07,000 --> 00:04:08,300: is that Dwight
- 00:04:08,560 --> 00:04:10,560: now runs on three thousand
- 00:04:10,820 --> 00:04:12,880: markdown files. And again, like I said, they have a very
- 00:04:12,980 --> 00:04:15,040: lean team. They're only two, three people that work on
- 00:04:15,100 --> 00:04:17,300: this, and they're starting to hit a wall,
- 00:04:18,200 --> 00:04:20,260: right? With every failure
- 00:04:20,300 --> 00:04:22,360: that Dwight makes, and Dwight from time to time
- 00:04:22,440 --> 00:04:24,180: doesn't make a failure when I ask something,
- 00:04:24,600 --> 00:04:25,520: is they basically
- 00:04:26,500 --> 00:04:28,480: are kind of stuck. They don't know
- 00:04:28,960 --> 00:04:30,000: where they generated
- 00:04:30,500 --> 00:04:32,700: those mard-- uh, those markdown files
- 00:04:32,740 --> 00:04:35,000: from. They don't know who the subject
- 00:04:35,040 --> 00:04:37,020: matter expert is in the organization.
- 00:04:37,980 --> 00:04:40,000: So they kind of take a lot of
- 00:04:40,040 --> 00:04:42,180: time to try to figure out how
- 00:04:42,220 --> 00:04:43,680: to solve, how to fix
- 00:04:44,100 --> 00:04:46,240: those context files. And there are a lot of these
- 00:04:46,280 --> 00:04:47,000: context files,
- 00:04:47,660 --> 00:04:49,480: right? So that is a problem,
- 00:04:49,740 --> 00:04:52,060: right? But it's not an unusual problem because
- 00:04:52,520 --> 00:04:54,420: I have a lot of conversations with customers,
- 00:04:54,860 --> 00:04:55,900: and when I do talk to them,
- 00:04:56,380 --> 00:04:58,520: I kind of see this pattern, and Felix already talked
- 00:04:58,560 --> 00:05:00,560: about it, so I can go a little bit faster here,
- 00:05:00,640 --> 00:05:01,680: is that basically
- 00:05:01,940 --> 00:05:03,980: one agent, fine, right? Clear ownership,
- 00:05:04,080 --> 00:05:05,260: clear context files.
- 00:05:05,720 --> 00:05:06,300: Ten agents,
- 00:05:07,060 --> 00:05:08,700: still okay. We feel still comfortable.
- 00:05:09,100 --> 00:05:10,100: Hundred agents, oof,
- 00:05:10,620 --> 00:05:11,780: starts to be really difficult.
- 00:05:12,040 --> 00:05:13,160: Thousand agents, forget it,
- 00:05:13,600 --> 00:05:15,760: right? Forget it without governance,
- 00:05:16,440 --> 00:05:17,160: right? But the,
- 00:05:17,640 --> 00:05:19,840: the really tricky point here is that
- 00:05:20,620 --> 00:05:22,300: as you have more AI agents,
- 00:05:22,720 --> 00:05:24,800: there's something exponential to context.
- 00:05:25,340 --> 00:05:27,460: Each of these agents today have their
- 00:05:27,520 --> 00:05:28,280: own context,
- 00:05:28,580 --> 00:05:30,800: and context grows even faster than the number
- 00:05:30,860 --> 00:05:32,100: of agents that you have.
- 00:05:32,800 --> 00:05:35,200: So what that means is basically you really struggle
- 00:05:35,600 --> 00:05:38,340: to keep that context understandable
- 00:05:38,460 --> 00:05:40,560: and have transparency around that context.
- 00:05:42,320 --> 00:05:44,480: So it, it all works fine
- 00:05:44,540 --> 00:05:45,980: until people start to follow,
- 00:05:46,320 --> 00:05:48,480: uh, have these follow-up questions or deeper dive
- 00:05:48,520 --> 00:05:50,840: questions. They, they leave the happy path of the agent,
- 00:05:51,280 --> 00:05:53,360: and this is where the whole thing starts to wobble, because
- 00:05:53,920 --> 00:05:56,140: it-- sometimes the context is really not helping
- 00:05:56,180 --> 00:05:57,460: them to properly answer.
- 00:05:58,120 --> 00:06:00,120: So-- and this is the part that will cost you most.
- 00:06:00,540 --> 00:06:02,740: People won't conclude, "Hey, how can
- 00:06:03,100 --> 00:06:05,448: I help?" To make this context better.
- 00:06:06,048 --> 00:06:07,448: Most of the time, to be honest,
- 00:06:07,808 --> 00:06:09,128: most people don't even look
- 00:06:09,668 --> 00:06:11,748: at what context an agent uses to produce an
- 00:06:11,828 --> 00:06:12,608: answer, right?
- 00:06:13,028 --> 00:06:14,108: They will just conclude
- 00:06:14,668 --> 00:06:15,988: AI does not work
- 00:06:16,528 --> 00:06:17,928: in our organization,
- 00:06:18,468 --> 00:06:19,508: right? And that is a problem,
- 00:06:19,808 --> 00:06:21,388: right? But in a way,
- 00:06:22,028 --> 00:06:22,768: if you think about it,
- 00:06:23,028 --> 00:06:24,048: it's not a new problem.
- 00:06:24,688 --> 00:06:26,808: I call it the paradigm of crap in, crap out,
- 00:06:27,448 --> 00:06:29,848: right? So, and we have been solving
- 00:06:29,888 --> 00:06:30,848: this trust problem
- 00:06:31,448 --> 00:06:33,868: already for the past two decades with data governance
- 00:06:34,148 --> 00:06:36,488: for analytics use cases, for data analytics
- 00:06:36,588 --> 00:06:38,788: use cases. So basically, Collibra help you build
- 00:06:38,868 --> 00:06:39,968: governance principles
- 00:06:40,488 --> 00:06:42,948: to try to address and help people understand
- 00:06:43,248 --> 00:06:44,868: the data, help understand
- 00:06:45,208 --> 00:06:46,508: the analytics that you have,
- 00:06:46,808 --> 00:06:49,028: how it all ties together, the lineage between
- 00:06:49,428 --> 00:06:51,668: both, so that people can actually have trust.
- 00:06:52,128 --> 00:06:52,828: I think the last
- 00:06:53,628 --> 00:06:54,788: thing that we really did
- 00:06:55,048 --> 00:06:57,288: very well, in my opinion, is really building that
- 00:06:57,328 --> 00:06:58,748: data product application
- 00:06:59,148 --> 00:07:01,148: with the data marketplace that basically
- 00:07:01,208 --> 00:07:02,448: hosts all your trusted
- 00:07:03,108 --> 00:07:03,788: data products.
- 00:07:04,968 --> 00:07:06,748: So, if we think about it,
- 00:07:08,088 --> 00:07:10,168: agents don't just need good
- 00:07:10,208 --> 00:07:12,388: data. They need meaning
- 00:07:12,468 --> 00:07:14,948: about the data, and they also need to understand
- 00:07:15,188 --> 00:07:16,748: how your business operates,
- 00:07:17,488 --> 00:07:18,788: right? And therefore,
- 00:07:19,468 --> 00:07:21,948: we need to expand our definition of governance
- 00:07:22,208 --> 00:07:22,868: from data
- 00:07:23,488 --> 00:07:24,208: to context.
- 00:07:25,368 --> 00:07:26,328: But okay, cool.
- 00:07:27,028 --> 00:07:29,088: What do we actually mean with context?
- 00:07:29,328 --> 00:07:30,168: As already mentioned,
- 00:07:30,448 --> 00:07:32,308: for me, it's super simple. Three dimensions:
- 00:07:32,608 --> 00:07:33,668: data, meaning,
- 00:07:33,948 --> 00:07:34,588: and process.
- 00:07:35,048 --> 00:07:36,128: So if you think about the data,
- 00:07:36,548 --> 00:07:37,528: you're already doing this,
- 00:07:37,988 --> 00:07:40,348: right? It's about collecting the technical metadata,
- 00:07:40,488 --> 00:07:42,748: curating it. It's about collecting the signals,
- 00:07:43,228 --> 00:07:45,168: whether it's the quality signals or the usage
- 00:07:45,808 --> 00:07:48,148: signals. It's about building trusted data products.
- 00:07:49,188 --> 00:07:51,208: So this is the part that you're already doing.
- 00:07:51,588 --> 00:07:53,768: Then we move towards the meaning. Some of that
- 00:07:53,808 --> 00:07:55,848: we already do. We're building glossaries, we're
- 00:07:55,888 --> 00:07:57,028: building, um,
- 00:07:58,328 --> 00:08:00,448: uh, we're building logical models, conceptual
- 00:08:00,528 --> 00:08:02,568: models already in Collibra. We need to go a few
- 00:08:02,668 --> 00:08:03,268: steps further,
- 00:08:03,768 --> 00:08:05,868: right? And this is where Felix already highlighted
- 00:08:06,308 --> 00:08:08,708: ontology, ontologies. We need to build ontologies.
- 00:08:09,268 --> 00:08:09,908: We need to build
- 00:08:10,208 --> 00:08:11,688: semantic models. We need to build,
- 00:08:12,147 --> 00:08:13,908: uh, taxonomies. And then finally,
- 00:08:14,568 --> 00:08:16,648: like I said, your agents need to understand
- 00:08:16,708 --> 00:08:18,788: how your business operate, and why is that
- 00:08:18,848 --> 00:08:19,628: important? Because
- 00:08:20,048 --> 00:08:21,288: they need to understand
- 00:08:21,688 --> 00:08:22,328: the business
- 00:08:22,728 --> 00:08:25,208: in order to answer the why questions,
- 00:08:25,688 --> 00:08:28,028: right? So, and that's the process. Your standard
- 00:08:28,188 --> 00:08:30,588: operating procedures, your decision logics,
- 00:08:30,708 --> 00:08:32,708: your policies, your contracts, et cetera,
- 00:08:32,768 --> 00:08:35,948: et cetera. And
- 00:08:36,688 --> 00:08:37,268: ultimately,
- 00:08:37,568 --> 00:08:39,088: why do we need to do that? Because
- 00:08:39,408 --> 00:08:40,088: your agents,
- 00:08:41,128 --> 00:08:43,408: um, people are creating
- 00:08:43,468 --> 00:08:44,328: these agents today,
- 00:08:44,688 --> 00:08:45,608: right? And
- 00:08:45,888 --> 00:08:47,388: as we're sitting in this room,
- 00:08:47,748 --> 00:08:49,808: I bet there is somebody in your organization
- 00:08:49,948 --> 00:08:50,988: creating a new agent
- 00:08:51,668 --> 00:08:53,968: or generating new context files for these agents.
- 00:08:54,708 --> 00:08:56,728: And they use these traditional tools like Gemini,
- 00:08:56,828 --> 00:08:57,608: ChatGPT,
- 00:08:57,988 --> 00:08:58,708: Cloud Code,
- 00:08:59,288 --> 00:09:00,908: uh, Copilot,
- 00:09:01,188 --> 00:09:02,648: uh, Microsoft Copilot, et cetera.
- 00:09:03,308 --> 00:09:05,188: Um, and what is typical about what,
- 00:09:05,708 --> 00:09:08,108: uh, we're doing today is we basically
- 00:09:08,208 --> 00:09:10,268: use these tools to generate context
- 00:09:10,608 --> 00:09:13,488: out of dubious
- 00:09:13,668 --> 00:09:14,188: sources,
- 00:09:14,568 --> 00:09:16,328: maybe outdated sources, maybe
- 00:09:16,868 --> 00:09:18,608: sources that are not under governance,
- 00:09:19,448 --> 00:09:21,588: and that generates new context files, and
- 00:09:21,628 --> 00:09:23,668: these context files are being stored on
- 00:09:23,728 --> 00:09:25,808: the agentic platforms together with the
- 00:09:25,908 --> 00:09:27,468: agent. And
- 00:09:27,828 --> 00:09:29,848: so what we basically are creating
- 00:09:30,488 --> 00:09:33,088: when we are doing that is we're creating context
- 00:09:33,348 --> 00:09:34,908: islands, right?
- 00:09:35,568 --> 00:09:38,028: And that is a problem because the same context
- 00:09:38,088 --> 00:09:40,308: might be needed by multiple agents. So
- 00:09:40,328 --> 00:09:42,588: which basically means we regenerate the same
- 00:09:42,648 --> 00:09:43,408: context files
- 00:09:43,928 --> 00:09:45,288: for these many different islands,
- 00:09:45,608 --> 00:09:46,368: and sometimes
- 00:09:46,768 --> 00:09:48,768: you will end up in situations where one
- 00:09:48,868 --> 00:09:50,888: agent basically runs
- 00:09:51,028 --> 00:09:52,448: on more
- 00:09:52,928 --> 00:09:54,968: fresh context files or more
- 00:09:55,068 --> 00:09:56,848: accurate context files than other agents.
- 00:09:57,228 --> 00:09:59,348: Worst case scenario, we have agents running on
- 00:09:59,388 --> 00:10:00,508: conflicting context.
- 00:10:01,248 --> 00:10:02,248: All right? So
- 00:10:03,808 --> 00:10:05,808: this is happening. We can't hit pause because we
- 00:10:05,828 --> 00:10:06,568: want to govern it.
- 00:10:07,068 --> 00:10:09,148: So, um, we need to try
- 00:10:09,188 --> 00:10:11,268: to see how we can address this situation
- 00:10:12,008 --> 00:10:14,128: as we go. Why is it a
- 00:10:14,188 --> 00:10:15,268: problem, and why need to--
- 00:10:15,528 --> 00:10:17,728: do we need to address it? Because these, like I said,
- 00:10:17,928 --> 00:10:19,248: we've, we've experienced it
- 00:10:19,508 --> 00:10:20,648: all in this room.
- 00:10:21,068 --> 00:10:23,308: We don't always trust the outcome of this agent.
- 00:10:23,388 --> 00:10:25,608: And why is that? Because they actually sometimes
- 00:10:25,628 --> 00:10:26,248: hallucinate.
- 00:10:26,508 --> 00:10:27,388: They sometimes fail.
- 00:10:28,688 --> 00:10:30,828: And I'm speaking here out of
- 00:10:30,888 --> 00:10:33,248: personal experience. I built already a couple of agents
- 00:10:33,308 --> 00:10:33,888: for Collibra,
- 00:10:34,288 --> 00:10:35,468: right? When
- 00:10:35,928 --> 00:10:36,948: every time when
- 00:10:37,508 --> 00:10:39,668: somebody told me, "Hey, Tom, your agent is really
- 00:10:39,708 --> 00:10:40,528: not very helpful,"
- 00:10:41,728 --> 00:10:43,828: every time when I looked at the context, I came
- 00:10:43,888 --> 00:10:46,268: to either these one, these four conclusions.
- 00:10:46,448 --> 00:10:48,428: Either the context was simply missing,
- 00:10:49,168 --> 00:10:51,288: right? And agents are trained on
- 00:10:52,228 --> 00:10:54,248: all the data that there is in publicly available
- 00:10:54,288 --> 00:10:56,468: in the world. So based on that data,
- 00:10:56,548 --> 00:10:58,088: they come up with a generic answer.
- 00:10:58,528 --> 00:10:59,748: Sometimes that answer is
- 00:11:00,708 --> 00:11:02,768: more or less okay, but it's not the right
- 00:11:02,828 --> 00:11:03,968: answer for your business,
- 00:11:04,528 --> 00:11:06,748: right? So missing context.
- 00:11:07,048 --> 00:11:08,748: Incomplete, more or less the same thing.
- 00:11:09,528 --> 00:11:11,668: Outdated, somebody changed the definition
- 00:11:11,728 --> 00:11:12,308: of a metric,
- 00:11:12,908 --> 00:11:15,248: and nobody bothered in these hundred thousands
- 00:11:15,288 --> 00:11:17,108: of context files to try to figure out
- 00:11:17,448 --> 00:11:19,588: where is that definition for that metric being
- 00:11:19,668 --> 00:11:21,988: used. And sure, let me go and find
- 00:11:22,028 --> 00:11:23,848: it and go and update these context files,
- 00:11:24,508 --> 00:11:25,708: right? And then finally, a bit
- 00:11:26,048 --> 00:11:26,868: counterintuitive,
- 00:11:27,188 --> 00:11:29,548: you can also feed too much context to an agent,
- 00:11:29,868 --> 00:11:30,608: right? Um,
- 00:11:31,008 --> 00:11:33,168: Stanford talks about agents lost in the middle.
- 00:11:33,768 --> 00:11:35,808: This is really agents are really good at capturing
- 00:11:35,848 --> 00:11:37,868: for very long monolithic context files. They're
- 00:11:37,888 --> 00:11:39,948: really good at capturing the start and the end, and they
- 00:11:39,968 --> 00:11:41,068: forget about the middle part,
- 00:11:41,508 --> 00:11:43,868: right? Underneath all of this is an ownership
- 00:11:43,888 --> 00:11:46,108: problem. If
- 00:11:46,148 --> 00:11:46,788: you have so much
- 00:11:47,268 --> 00:11:48,848: context files that need to be owned,
- 00:11:49,688 --> 00:11:51,768: and if most of these context files, and this is
- 00:11:51,808 --> 00:11:53,848: what I see happening at Collibra as well, is like
- 00:11:54,308 --> 00:11:55,328: people generate these,
- 00:11:55,928 --> 00:11:56,888: uh, context files, and
- 00:11:57,408 --> 00:11:57,968: models like
- 00:11:58,408 --> 00:12:00,068: Claude, they're very chatty, and they
- 00:12:00,448 --> 00:12:02,708: tend to write really long monolithic context files,
- 00:12:03,408 --> 00:12:05,748: is that if you wanna establish ownership
- 00:12:05,788 --> 00:12:06,308: for very
- 00:12:06,588 --> 00:12:08,688: huge context files, it's really hard because then
- 00:12:09,008 --> 00:12:11,008: you probably need to sign a whole department
- 00:12:11,068 --> 00:12:13,088: of people as owner of that file,
- 00:12:13,528 --> 00:12:14,988: right? And if everybody's an owner, then
- 00:12:15,388 --> 00:12:17,048: yeah, nobody is an owner, right?
- 00:12:17,668 --> 00:12:19,128: So and that leads to two
- 00:12:19,728 --> 00:12:22,048: effects, and I like to call these effects context
- 00:12:22,148 --> 00:12:24,632: chaos Right? Zero transparency.
- 00:12:24,752 --> 00:12:25,392: You don't know what
- 00:12:25,672 --> 00:12:26,452: context files,
- 00:12:26,952 --> 00:12:29,192: uh, live in your organization. You don't know which context
- 00:12:29,252 --> 00:12:30,492: files an agent uses,
- 00:12:31,332 --> 00:12:33,492: um, and you don't know who owns these
- 00:12:33,532 --> 00:12:34,332: context files.
- 00:12:34,852 --> 00:12:37,052: The second is, and this is where our team ended
- 00:12:37,172 --> 00:12:37,912: up, is
- 00:12:38,312 --> 00:12:39,331: a maintenance nightmare.
- 00:12:39,692 --> 00:12:40,552: I call it contet-
- 00:12:40,832 --> 00:12:41,612: context depth.
- 00:12:42,092 --> 00:12:43,032: It's a bit of a, uh,
- 00:12:43,392 --> 00:12:43,972: an engineering
- 00:12:44,332 --> 00:12:46,072: term, but it really is like
- 00:12:46,812 --> 00:12:49,012: you end up with a, just a few people trying
- 00:12:49,052 --> 00:12:49,652: to maintain
- 00:12:50,092 --> 00:12:50,992: an ever-growing
- 00:12:51,512 --> 00:12:53,792: volume of context, and it's just fea- not feasible
- 00:12:53,852 --> 00:12:54,852: anymore, right?
- 00:12:55,711 --> 00:12:57,272: So this is exactly,
- 00:12:57,692 --> 00:12:59,832: um, where our team ended up.
- 00:13:00,312 --> 00:13:02,512: And so let me now talk about how we solved
- 00:13:02,552 --> 00:13:04,552: it for them, or at least partially solved it,
- 00:13:05,092 --> 00:13:06,912: and how we're now trying to productize
- 00:13:07,172 --> 00:13:07,932: that solution.
- 00:13:09,152 --> 00:13:11,212: So there are two parts to solve it.
- 00:13:11,572 --> 00:13:13,752: When we really analyzed all of these context,
- 00:13:13,812 --> 00:13:16,032: these three thousand context files, what we found
- 00:13:16,092 --> 00:13:18,492: out is that ninety percent of those context
- 00:13:18,532 --> 00:13:20,052: files are data-related.
- 00:13:20,832 --> 00:13:23,012: Right? They're basically files that explain the structure
- 00:13:23,072 --> 00:13:24,632: of a table. Well,
- 00:13:24,952 --> 00:13:27,092: we can generate this from our governed Collibra
- 00:13:27,191 --> 00:13:29,272: catalog. So let's make that
- 00:13:29,352 --> 00:13:30,872: super easy for those,
- 00:13:31,672 --> 00:13:33,752: uh, uh, people, that AI engineers,
- 00:13:34,132 --> 00:13:36,372: to actually use Collibra as a governed
- 00:13:36,432 --> 00:13:38,112: source of truth for your context,
- 00:13:38,532 --> 00:13:40,252: and let's keep the link between
- 00:13:40,692 --> 00:13:41,372: the governed,
- 00:13:42,432 --> 00:13:43,392: um, the governed
- 00:13:43,932 --> 00:13:45,992: data tables or the governed data products
- 00:13:46,452 --> 00:13:48,212: and the context that we generate from,
- 00:13:48,892 --> 00:13:50,112: um, uh,
- 00:13:50,372 --> 00:13:51,712: that governed artifacts.
- 00:13:52,732 --> 00:13:53,472: The second part,
- 00:13:54,332 --> 00:13:56,532: um, and this is where probably most
- 00:13:56,592 --> 00:13:57,312: of you are,
- 00:13:57,992 --> 00:14:00,012: um, and this is the reality we
- 00:14:00,072 --> 00:14:02,292: have on the floor, is that these context files
- 00:14:02,332 --> 00:14:03,132: already exist.
- 00:14:03,752 --> 00:14:06,132: So how do we get from what exists
- 00:14:06,372 --> 00:14:07,432: to a governed situation?
- 00:14:07,772 --> 00:14:09,852: Okay. The, the first thing that we need to do, and this
- 00:14:09,892 --> 00:14:11,212: is what we've historically done,
- 00:14:11,872 --> 00:14:13,012: uh, in the past as well,
- 00:14:13,672 --> 00:14:15,872: is let's centralize it. Let's capture that
- 00:14:15,912 --> 00:14:17,652: context. Let's at least create
- 00:14:17,912 --> 00:14:19,352: one catalog of context
- 00:14:19,832 --> 00:14:22,192: so that at least we know what context
- 00:14:22,232 --> 00:14:23,812: we have, and we know what
- 00:14:24,112 --> 00:14:26,132: agent is using what type of context. And
- 00:14:26,172 --> 00:14:27,152: from that point of view,
- 00:14:27,552 --> 00:14:28,672: you then can start to
- 00:14:28,932 --> 00:14:29,552: try to see,
- 00:14:29,892 --> 00:14:32,012: "Hey, in this context file, are there any metrics
- 00:14:32,072 --> 00:14:34,192: that I'm already governing? Are there any business
- 00:14:34,252 --> 00:14:35,972: terms that I'm already have,
- 00:14:36,572 --> 00:14:38,612: uh, defined and are properly
- 00:14:38,852 --> 00:14:40,912: owned?" Okay, if that's the case, let's
- 00:14:40,952 --> 00:14:42,832: link it up. Let's embed that context
- 00:14:43,372 --> 00:14:45,212: into the governed graph into Collibra.
- 00:14:45,872 --> 00:14:47,712: Right? Um,
- 00:14:48,712 --> 00:14:49,472: it will happen
- 00:14:49,932 --> 00:14:51,932: in most cases that actually there is
- 00:14:51,972 --> 00:14:54,092: content in those context files that you have
- 00:14:54,152 --> 00:14:56,232: not yet governed or have not yet used
- 00:14:56,292 --> 00:14:58,412: it. So let's create that for you so that
- 00:14:58,472 --> 00:14:59,472: you can go and find
- 00:14:59,952 --> 00:15:02,092: an owner for pieces of the context in
- 00:15:02,532 --> 00:15:04,652: Collibra, and let's identify it as a term
- 00:15:05,232 --> 00:15:07,432: or as a metric or as a critical data element
- 00:15:07,612 --> 00:15:08,992: or as a business process,
- 00:15:09,332 --> 00:15:10,992: and let's find ownership with the business
- 00:15:11,452 --> 00:15:13,552: so that parts of the context file
- 00:15:13,592 --> 00:15:14,152: can be owned.
- 00:15:15,992 --> 00:15:17,892: Most of you will do these two parts,
- 00:15:18,192 --> 00:15:20,292: right? You'll use part one, and you'll use part two,
- 00:15:20,452 --> 00:15:21,092: and if you use
- 00:15:21,352 --> 00:15:22,932: both parts, you basically are,
- 00:15:23,332 --> 00:15:24,252: are get into a loop.
- 00:15:24,772 --> 00:15:27,152: Right? And this loop keeps our context honest.
- 00:15:27,492 --> 00:15:28,612: Basically, we pull in
- 00:15:29,052 --> 00:15:29,712: the context.
- 00:15:30,032 --> 00:15:32,572: We validate the context. We apply some principles
- 00:15:32,632 --> 00:15:34,072: on writing good context files.
- 00:15:34,872 --> 00:15:35,992: Next, we govern
- 00:15:36,272 --> 00:15:38,072: the context. We really embed it in the graph.
- 00:15:38,592 --> 00:15:39,472: Then we,
- 00:15:40,072 --> 00:15:40,692: uh, attest,
- 00:15:40,952 --> 00:15:42,792: create regular attestations so that
- 00:15:43,052 --> 00:15:44,992: we can define ownership as well as
- 00:15:45,312 --> 00:15:47,212: get to a certification level, and then
- 00:15:48,052 --> 00:15:50,432: let's generate context from what you govern.
- 00:15:50,752 --> 00:15:53,192: And this is something that resonates with a lot of customers
- 00:15:53,232 --> 00:15:55,432: that I have, uh, conversations with
- 00:15:55,532 --> 00:15:56,652: is, in my opinion,
- 00:15:56,912 --> 00:15:57,492: context
- 00:15:57,972 --> 00:15:59,912: should be a derivative of governance,
- 00:16:00,272 --> 00:16:01,312: not the other way around.
- 00:16:01,872 --> 00:16:03,672: Right? So this is where we need to end up.
- 00:16:04,432 --> 00:16:06,612: Important is that this loop is an incremental.
- 00:16:06,692 --> 00:16:08,952: There's no big bang solution. Forget it. It will never
- 00:16:08,992 --> 00:16:11,012: work. So you just need to start
- 00:16:11,692 --> 00:16:13,712: running your agents and the context that these
- 00:16:13,812 --> 00:16:14,832: agents use
- 00:16:15,172 --> 00:16:17,172: through this loop and incrementally get it
- 00:16:17,212 --> 00:16:19,232: better to the point that you can really
- 00:16:19,292 --> 00:16:20,952: trust the outcome of your agents.
- 00:16:21,732 --> 00:16:23,732: Now, all good and fine. "What are you building,"
- 00:16:24,092 --> 00:16:24,692: you might ask,
- 00:16:25,052 --> 00:16:26,272: uh, "to help us with that?"
- 00:16:26,532 --> 00:16:28,552: Well, a lot of these things already
- 00:16:28,652 --> 00:16:30,292: Felix highlighted very briefly,
- 00:16:31,032 --> 00:16:32,452: um, and I'll do it again
- 00:16:32,772 --> 00:16:33,552: so that you remember.
- 00:16:34,072 --> 00:16:36,332: Right? Uh, the first thing is
- 00:16:36,812 --> 00:16:37,872: an ontology workbench.
- 00:16:38,532 --> 00:16:40,832: Ontologies is really explain how your business
- 00:16:40,952 --> 00:16:41,492: operates.
- 00:16:41,952 --> 00:16:44,452: They talk about the things which are critical and important
- 00:16:44,652 --> 00:16:46,692: to your business and how these things relate to
- 00:16:46,712 --> 00:16:48,652: each other. Think about a contract,
- 00:16:49,252 --> 00:16:51,292: a, a financial instrument, a loan, a
- 00:16:51,372 --> 00:16:52,252: borrower, a lender.
- 00:16:52,652 --> 00:16:54,792: What are those things, and what do they mean to the business?
- 00:16:55,532 --> 00:16:57,852: Now, many of you already have
- 00:16:57,912 --> 00:16:59,212: these ontologies in,
- 00:16:59,632 --> 00:17:01,412: outside of Collibra, which is great.
- 00:17:01,832 --> 00:17:03,912: Right? We don't necessarily want
- 00:17:03,932 --> 00:17:05,712: to generate these ontologies for you.
- 00:17:06,432 --> 00:17:07,972: What we will allow you to do
- 00:17:08,392 --> 00:17:10,812: is to import those ontologies into Collibra
- 00:17:11,592 --> 00:17:12,992: in RDF, OWL, Turtle
- 00:17:13,372 --> 00:17:14,032: format, any
- 00:17:14,492 --> 00:17:16,212: W3C supported format.
- 00:17:16,952 --> 00:17:18,972: And where Collibra can add value
- 00:17:19,052 --> 00:17:21,112: because many of these ontologies that you
- 00:17:21,152 --> 00:17:22,852: already have are disconnected from
- 00:17:23,112 --> 00:17:24,172: your data. Right?
- 00:17:24,772 --> 00:17:27,052: So you could either give it to an agent,
- 00:17:27,532 --> 00:17:29,732: right, and then you give the data, and try to see
- 00:17:29,792 --> 00:17:31,272: if agent can figure it out,
- 00:17:31,732 --> 00:17:32,852: right? But then again, you're
- 00:17:33,132 --> 00:17:35,172: guessing. You allow your agents to guess.
- 00:17:36,132 --> 00:17:38,192: What if you can use Collibra to make that
- 00:17:38,232 --> 00:17:40,552: deterministic link between your ontology
- 00:17:41,052 --> 00:17:41,932: and your data?
- 00:17:42,312 --> 00:17:43,832: Right? This is what we're gonna focus on,
- 00:17:44,172 --> 00:17:45,912: making it super easy for you
- 00:17:46,372 --> 00:17:47,512: to link these ontologies
- 00:17:47,952 --> 00:17:49,652: to governed artifacts in Collibra,
- 00:17:49,972 --> 00:17:52,292: whether it's a term or da- the data itself,
- 00:17:52,372 --> 00:17:53,852: whether it's a critical data element
- 00:17:54,152 --> 00:17:55,752: or a data entity or an attribute.
- 00:17:56,292 --> 00:17:58,612: We'll basically allow you to make that link.
- 00:17:59,912 --> 00:18:01,232: Next, um,
- 00:18:01,992 --> 00:18:04,232: so now we captured, we captured the, the
- 00:18:04,312 --> 00:18:06,212: meaning or the, the, the business meaning,
- 00:18:06,652 --> 00:18:08,732: but we need more than just business meaning, right?
- 00:18:08,812 --> 00:18:09,612: We said we need
- 00:18:10,172 --> 00:18:12,172: the data, we need the bus- the meaning, and we need
- 00:18:12,212 --> 00:18:14,632: process, so we need to be able to package everything
- 00:18:14,732 --> 00:18:17,112: up. Right? So this is the context designer.
- 00:18:17,192 --> 00:18:19,252: The context designer will allow you to pick
- 00:18:19,272 --> 00:18:19,992: and choose
- 00:18:20,572 --> 00:18:22,652: those artifacts that exist in
- 00:18:22,692 --> 00:18:24,412: Collibra, whether it's a data product
- 00:18:24,832 --> 00:18:26,392: or whether it's a business process
- 00:18:26,752 --> 00:18:27,352: or whether it's
- 00:18:27,772 --> 00:18:30,172: your data dictionary for a certain system.
- 00:18:30,832 --> 00:18:31,632: You basically can
- 00:18:32,132 --> 00:18:33,952: bundle it into a context package,
- 00:18:34,492 --> 00:18:36,612: and then from the context package, we will
- 00:18:36,652 --> 00:18:37,192: generate
- 00:18:37,552 --> 00:18:39,592: a context file or multiple context
- 00:18:39,692 --> 00:18:41,592: file that your agents can use.
- 00:18:42,052 --> 00:18:44,192: What is the good thing is that this context
- 00:18:44,252 --> 00:18:45,192: then cannot get
- 00:18:45,512 --> 00:18:46,252: stale anymore,
- 00:18:46,512 --> 00:18:48,092: right? And why is that?
- 00:18:48,412 --> 00:18:50,772: Because Collibra will have governance processes
- 00:18:50,812 --> 00:18:53,212: in place that when your definition changes,
- 00:18:54,252 --> 00:18:56,432: well, the owners is alerted of the context. "Hey,
- 00:18:56,492 --> 00:18:57,492: the definition changes,"
- 00:18:57,752 --> 00:19:00,032: right? "Do you wanna generate- regenerate your context
- 00:19:00,112 --> 00:19:02,232: package?" When there is drift, we can
- 00:19:02,312 --> 00:19:04,512: also pull in the semantic views, for example.
- 00:19:04,872 --> 00:19:05,972: We can reconcile,
- 00:19:06,252 --> 00:19:08,372: and when there is drift between semantic view
- 00:19:08,432 --> 00:19:10,592: and the context package of Collibra, again, we can
- 00:19:10,652 --> 00:19:11,752: alert. Or
- 00:19:12,692 --> 00:19:14,832: what happens very often with Dwight is
- 00:19:15,172 --> 00:19:16,412: when Dwight hallucinates,
- 00:19:16,752 --> 00:19:18,292: I wanna be able to give feedback,
- 00:19:18,952 --> 00:19:19,952: right? So,
- 00:19:20,852 --> 00:19:22,412: um, the idea would be then, then
- 00:19:22,972 --> 00:19:25,012: since Dwight knows which context package is
- 00:19:25,052 --> 00:19:25,732: being used,
- 00:19:26,352 --> 00:19:28,712: I can, as part of the conversation, say, "Hey, Dwight,
- 00:19:29,032 --> 00:19:30,452: you made a huge mistake here,"
- 00:19:30,712 --> 00:19:32,872: right? "This is not how you should interpret this data.
- 00:19:33,232 --> 00:19:35,012: This is the right way to interpret the data."
- 00:19:35,491 --> 00:19:36,312: And that creates
- 00:19:36,792 --> 00:19:37,492: an issue back
- 00:19:38,032 --> 00:19:39,792: to the exact same context file,
- 00:19:40,512 --> 00:19:42,592: uh, or the context package that already exists in
- 00:19:42,632 --> 00:19:44,132: Collibra. So we have this kind of a
- 00:19:44,652 --> 00:19:46,932: feedback loop part of the conversation
- 00:19:47,272 --> 00:19:48,352: that you have with an agent,
- 00:19:49,372 --> 00:19:50,072: call it a government
- 00:19:50,452 --> 00:19:51,032: feedback loop.
- 00:19:51,652 --> 00:19:53,672: So which leaves us with one question, all
- 00:19:53,692 --> 00:19:55,092: good and fine, we bundled it up.
- 00:19:55,472 --> 00:19:56,852: How does that context get
- 00:19:57,292 --> 00:19:58,732: to the consuming platforms,
- 00:19:59,152 --> 00:20:01,272: right? This is where the context compiler
- 00:20:01,352 --> 00:20:02,252: comes into the picture,
- 00:20:03,092 --> 00:20:05,412: is every agentic platform kind
- 00:20:05,472 --> 00:20:07,512: of wants context in their,
- 00:20:07,912 --> 00:20:08,932: uh, preferred format,
- 00:20:09,392 --> 00:20:10,452: right? So
- 00:20:10,732 --> 00:20:12,872: as Collibra, our answer is, "Actually, we don't
- 00:20:12,932 --> 00:20:15,172: care. Tell us what you need. We'll compile
- 00:20:15,232 --> 00:20:15,752: it for you."
- 00:20:16,172 --> 00:20:17,612: So the context compiler
- 00:20:17,992 --> 00:20:18,692: is basically
- 00:20:18,972 --> 00:20:20,572: a very powerful graph query
- 00:20:20,912 --> 00:20:22,092: that can be translated
- 00:20:22,452 --> 00:20:23,572: in multiple formats,
- 00:20:23,972 --> 00:20:26,392: right? Whether it's a semantic view for Snowflake,
- 00:20:26,452 --> 00:20:28,412: a, a metrics view for Databricks,
- 00:20:28,892 --> 00:20:31,032: or an Apache OC standard, or an open
- 00:20:31,092 --> 00:20:32,012: data contract standard,
- 00:20:32,272 --> 00:20:33,452: we can actually generate
- 00:20:33,732 --> 00:20:35,052: that for you and push it
- 00:20:35,312 --> 00:20:37,052: to the platforms that need it.
- 00:20:38,072 --> 00:20:39,212: Now, so we have
- 00:20:39,512 --> 00:20:41,672: our go-- our context. It's packaged up. It's
- 00:20:41,692 --> 00:20:42,572: delivered to an agent.
- 00:20:43,752 --> 00:20:45,172: Now, how
- 00:20:45,712 --> 00:20:48,172: will you still manage to do all of that governance
- 00:20:48,252 --> 00:20:49,872: work, right? You're already, like, sweating,
- 00:20:50,592 --> 00:20:52,892: uh, to get it just done for data. Now
- 00:20:53,092 --> 00:20:55,132: I'm telling you to also do it for context.
- 00:20:55,612 --> 00:20:57,632: Okay, we need to rethink how we do governance.
- 00:20:57,712 --> 00:20:59,392: We need to have much more automation,
- 00:20:59,832 --> 00:21:02,112: and this is why I'm very happy to introduce
- 00:21:02,212 --> 00:21:02,912: Maestro to you.
- 00:21:03,352 --> 00:21:04,332: This morning, we had
- 00:21:05,252 --> 00:21:07,452: twenty-five people enlisted or subscribed
- 00:21:07,492 --> 00:21:09,672: for the Maestro labs. I think probably, like, sixty
- 00:21:09,752 --> 00:21:10,512: people showed up,
- 00:21:11,212 --> 00:21:12,372: um, which is great. I,
- 00:21:12,692 --> 00:21:13,972: I really love the enthusiasm.
- 00:21:14,392 --> 00:21:16,052: What made me even happier,
- 00:21:16,732 --> 00:21:18,932: at the end of the session, there were several people
- 00:21:19,292 --> 00:21:21,792: that managed to already create their own automation
- 00:21:21,952 --> 00:21:24,052: agents, not just on our lab environment,
- 00:21:24,152 --> 00:21:25,392: but directly on their dev
- 00:21:25,852 --> 00:21:26,492: environment.
- 00:21:27,012 --> 00:21:29,332: So what are we bringing to make that happen?
- 00:21:29,532 --> 00:21:31,712: A Maestro studio, which is a new application,
- 00:21:31,852 --> 00:21:33,172: no-code, uh,
- 00:21:33,572 --> 00:21:35,612: application, where you basically can create
- 00:21:35,652 --> 00:21:36,532: your own agents.
- 00:21:37,932 --> 00:21:40,212: You can either make them specific for a certain
- 00:21:40,332 --> 00:21:42,352: task, you can make them as a digital twin
- 00:21:42,452 --> 00:21:43,112: for yourself,
- 00:21:43,652 --> 00:21:45,732: or you can create an agent that you share with the whole
- 00:21:45,792 --> 00:21:46,552: user community.
- 00:21:47,132 --> 00:21:48,572: Then we have Maestro Assistant.
- 00:21:49,312 --> 00:21:51,372: For those who are already using Copilot, think
- 00:21:51,452 --> 00:21:52,792: of Copilot on steroids.
- 00:21:53,292 --> 00:21:55,392: So we're upgrading the Copilot experience to
- 00:21:55,432 --> 00:21:56,972: Maestro Assistant. Copilot could
- 00:21:57,352 --> 00:21:58,252: answer questions,
- 00:21:58,572 --> 00:22:00,792: right? "Hey, where can I find this data product?"
- 00:22:01,192 --> 00:22:03,352: Right? Now, the Maestro Assistant
- 00:22:03,672 --> 00:22:05,912: will be able to interact with all these agent and,
- 00:22:05,972 --> 00:22:08,472: uh, not only answer questions, but also
- 00:22:08,512 --> 00:22:10,772: do actions. "Hey, create me a data product."
- 00:22:11,172 --> 00:22:12,072: Right? Um,
- 00:22:12,352 --> 00:22:14,412: "Hey, find any duplicate definitions in my
- 00:22:14,432 --> 00:22:15,832: glossaries, uh,
- 00:22:16,132 --> 00:22:18,412: and mark this defen-- this duplicate definition
- 00:22:18,852 --> 00:22:20,192: as, uh, obsolete
- 00:22:20,552 --> 00:22:21,672: and link it as duplicate
- 00:22:21,992 --> 00:22:23,732: to the original canonical definition
- 00:22:24,012 --> 00:22:25,312: that sits in this glossary."
- 00:22:25,812 --> 00:22:27,892: Right? All of these things will now be possible
- 00:22:27,932 --> 00:22:29,972: with Maestro Assistant, and we also have
- 00:22:30,032 --> 00:22:30,952: Maestro Actions.
- 00:22:31,332 --> 00:22:33,672: Those are AI-powered capabilities
- 00:22:33,992 --> 00:22:35,832: that have their own dedicated UX
- 00:22:36,292 --> 00:22:38,692: that you can set up with minimal configuration.
- 00:22:39,912 --> 00:22:40,732: So now let's have a look.
- 00:22:43,212 --> 00:22:45,132: So let me navigate to
- 00:22:45,492 --> 00:22:46,292: the Maestro studio.
- 00:22:49,052 --> 00:22:51,191: Maestro studio is your place where all your, the
- 00:22:51,232 --> 00:22:51,772: agents live.
- 00:22:52,071 --> 00:22:53,492: You can select your own agents.
- 00:22:54,012 --> 00:22:56,012: You can create new agents from here.
- 00:22:56,072 --> 00:22:58,312: You can create new agents from templates. We're already
- 00:22:58,352 --> 00:22:59,412: shipping four templates,
- 00:22:59,692 --> 00:23:01,452: and we're gonna continue to add more templates.
- 00:23:01,892 --> 00:23:03,912: You can import agents from YAML files.
- 00:23:04,032 --> 00:23:06,032: You can look at the statistics, how many people
- 00:23:06,092 --> 00:23:08,312: are interacting with this agent, how many conversations
- 00:23:08,672 --> 00:23:09,912: do we have with these agents.
- 00:23:11,572 --> 00:23:12,352: Now let's have a chat--
- 00:23:12,672 --> 00:23:13,692: look at, um, at the s-
- 00:23:14,012 --> 00:23:16,152: at the Maestro Assistant. Here, I'm asking
- 00:23:16,232 --> 00:23:18,312: for, "Hey, what do you know about the IDA
- 00:23:18,612 --> 00:23:19,432: schema?" Right?
- 00:23:20,052 --> 00:23:22,072: And it basically turned out that there's two schemas,
- 00:23:22,132 --> 00:23:24,152: which I didn't-- wa-wasn't aware of, an
- 00:23:24,192 --> 00:23:25,372: input and an output schema.
- 00:23:26,092 --> 00:23:28,252: Then I ask like, "Hey, what does IDA stands for?"
- 00:23:28,532 --> 00:23:30,052: Based on the context that it has,
- 00:23:30,612 --> 00:23:32,892: uh, it replies to me that actually IDA stands
- 00:23:32,952 --> 00:23:34,252: for Insights Data Access.
- 00:23:34,772 --> 00:23:36,252: And then I say, "Hey, is there an acronym
- 00:23:36,552 --> 00:23:37,092: or a
- 00:23:37,472 --> 00:23:38,012: term already?"
- 00:23:38,792 --> 00:23:39,892: Uh, no, there's no one.
- 00:23:40,352 --> 00:23:42,392: Right? Okay, now I seamlessly switch from
- 00:23:42,452 --> 00:23:44,492: one agent to the other and actually ask the
- 00:23:44,752 --> 00:23:47,132: asset maintainer agent to create this acronym
- 00:23:47,512 --> 00:23:49,532: and the related term with the definition and
- 00:23:49,592 --> 00:23:50,512: link it to the schema.
- 00:23:52,472 --> 00:23:54,692: Next, uh, we'll see here how you create an agent
- 00:23:54,752 --> 00:23:55,552: from a template.
- 00:23:55,992 --> 00:23:56,932: This is a data product,
- 00:23:57,212 --> 00:23:57,872: uh, agent,
- 00:23:58,252 --> 00:23:59,832: so you can change the name, the handle,
- 00:24:00,192 --> 00:24:02,232: et cetera. Uh, you can look
- 00:24:02,272 --> 00:24:04,432: at the instructions. You can make the instructions
- 00:24:04,492 --> 00:24:06,232: specific for your organization.
- 00:24:08,332 --> 00:24:10,332: Instructions can either be either very short
- 00:24:10,452 --> 00:24:12,552: or can be extensive, depending on how much,
- 00:24:13,132 --> 00:24:15,392: um, task an agent need to execute.
- 00:24:15,552 --> 00:24:17,732: You see an agent has specific tools, like
- 00:24:17,772 --> 00:24:18,752: data contract tools
- 00:24:19,192 --> 00:24:21,012: are very important for a data product agent.
- 00:24:21,512 --> 00:24:22,292: Now I basically
- 00:24:22,752 --> 00:24:23,752: submit the agent.
- 00:24:24,712 --> 00:24:26,892: Let's now have a look at how I can,
- 00:24:27,552 --> 00:24:29,692: uh, interact with this agent. I basically tag
- 00:24:29,732 --> 00:24:30,972: the agent, and I ask it
- 00:24:31,508 --> 00:24:32,268: To find me the,
- 00:24:32,568 --> 00:24:33,088: this IDE
- 00:24:33,908 --> 00:24:34,908: input schema.
- 00:24:35,408 --> 00:24:37,448: So it goes and searches and say like, "Hey, this is
- 00:24:37,508 --> 00:24:39,508: the, uh, IDE input schema,"
- 00:24:39,728 --> 00:24:41,208: right? "This is the description,"
- 00:24:41,708 --> 00:24:43,048: right? "These are the different tables."
- 00:24:43,508 --> 00:24:45,608: And I say, "Is there already a data product for it?" "No,
- 00:24:45,688 --> 00:24:47,888: there's no data product." "Let's create one." "Okay, good."
- 00:24:48,568 --> 00:24:50,888: It proposes me to create a new data product,
- 00:24:50,948 --> 00:24:52,848: proposes the domain, the owner, et cetera.
- 00:24:53,348 --> 00:24:55,568: It derives all that context from what exists. Then
- 00:24:55,608 --> 00:24:57,648: it asks me, "Hey, do you wanna create a data contract
- 00:24:57,708 --> 00:24:59,408: for it? Do you wanna adhere to the
- 00:24:59,928 --> 00:25:00,568: baked-in
- 00:25:01,028 --> 00:25:01,548: standard
- 00:25:01,968 --> 00:25:03,408: con- data contract settings
- 00:25:04,028 --> 00:25:06,028: for all data products or do you wanna have
- 00:25:06,128 --> 00:25:08,168: specific standards?" I say, "No, no, let's go ahead and
- 00:25:08,708 --> 00:25:10,108: use the standard SLAs."
- 00:25:10,608 --> 00:25:12,448: Okay. Now it basically creates,
- 00:25:13,048 --> 00:25:14,008: uh, this data product.
- 00:25:14,268 --> 00:25:15,608: We're gonna go through the process of creating.
- 00:25:16,128 --> 00:25:18,248: Maestro now also in the ten release supports
- 00:25:18,288 --> 00:25:18,988: full screen mode,
- 00:25:19,488 --> 00:25:21,488: right? Um, so
- 00:25:21,528 --> 00:25:23,848: that you can focus just on the conversation.
- 00:25:24,048 --> 00:25:25,588: Here we're creating the data product,
- 00:25:26,108 --> 00:25:27,028: the data product port,
- 00:25:27,428 --> 00:25:29,508: and now next we're linking all the
- 00:25:29,568 --> 00:25:31,488: tables to the data product port.
- 00:25:35,668 --> 00:25:37,848: The next step will be to create the data contract
- 00:25:39,028 --> 00:25:40,868: according to the SLAs, the standard
- 00:25:41,168 --> 00:25:42,888: SLAs that we applied. Here we're creating
- 00:25:43,228 --> 00:25:45,368: the data contract. Here you find, can find all the different
- 00:25:45,868 --> 00:25:47,988: SLAs that we were gonna set on our data
- 00:25:48,028 --> 00:25:49,648: product. And the next thing
- 00:25:50,128 --> 00:25:52,348: is to link the data contract to the data product port.
- 00:25:53,188 --> 00:25:54,988: And then finally, we're gonna initiate
- 00:25:55,588 --> 00:25:56,688: the, um,
- 00:25:57,228 --> 00:25:58,968: uh, manifest file for the data product.
- 00:25:59,248 --> 00:26:00,228: Now I'm gonna switch
- 00:26:00,728 --> 00:26:01,468: and show the,
- 00:26:01,908 --> 00:26:02,988: uh, preview assistant
- 00:26:03,768 --> 00:26:04,948: or the preview pane.
- 00:26:05,528 --> 00:26:07,668: So now I created the data product, and now
- 00:26:07,728 --> 00:26:09,888: I can actually navigate and pull the complexity
- 00:26:09,928 --> 00:26:11,888: of the platform into my conversation.
- 00:26:12,648 --> 00:26:14,008: Right? So here I can,
- 00:26:14,428 --> 00:26:16,628: in my same conversation, I can actually look
- 00:26:16,728 --> 00:26:18,908: at the details that were created
- 00:26:19,108 --> 00:26:19,748: automatically
- 00:26:20,228 --> 00:26:22,588: for me, right? I can go to the product,
- 00:26:22,668 --> 00:26:23,248: the port,
- 00:26:23,528 --> 00:26:24,128: the contract,
- 00:26:25,328 --> 00:26:26,828: and further browse into,
- 00:26:27,208 --> 00:26:28,468: uh, all the details that I want.
- 00:26:29,068 --> 00:26:31,208: I'm excited about this. I'm like a big fan
- 00:26:31,228 --> 00:26:33,468: probably if you look at our internal instance
- 00:26:33,508 --> 00:26:35,648: already, uh, the main user of, of the...
- 00:26:35,848 --> 00:26:37,988: So I hope you're excited as well about these automation
- 00:26:38,088 --> 00:26:38,788: capabilities.
- 00:26:39,268 --> 00:26:40,168: If you wanna learn more,
- 00:26:40,788 --> 00:26:41,348: we have
- 00:26:41,668 --> 00:26:43,808: customer engineers at the agent bar
- 00:26:43,928 --> 00:26:46,208: demonstrating Maestro. We have our product managers
- 00:26:46,248 --> 00:26:48,008: there as well. We have engineering,
- 00:26:48,328 --> 00:26:49,408: uh, leadership there as well.
- 00:26:50,288 --> 00:26:52,468: If you want your teams to get a dedicated
- 00:26:52,468 --> 00:26:54,528: demo because they weren't capable of attending
- 00:26:54,588 --> 00:26:56,028: this, uh, conference,
- 00:26:56,308 --> 00:26:58,308: scan the QR code and we'll happily demo
- 00:26:58,708 --> 00:27:00,768: or organize a demo for you. And then finally,
- 00:27:01,228 --> 00:27:03,288: you heard Felix talk about the context layer.
- 00:27:04,008 --> 00:27:06,128: Um, we're still looking for
- 00:27:06,148 --> 00:27:08,068: design partners to help build
- 00:27:08,488 --> 00:27:10,128: the ontology workbench for us
- 00:27:10,648 --> 00:27:11,408: to help build,
- 00:27:11,928 --> 00:27:14,208: uh, different capabilities in context
- 00:27:14,288 --> 00:27:14,828: governance,
- 00:27:15,128 --> 00:27:17,148: like context designer, context compiler,
- 00:27:17,308 --> 00:27:17,928: but also
- 00:27:18,688 --> 00:27:20,908: where I want to end up with context governance
- 00:27:21,068 --> 00:27:22,648: is a context marketplace.
- 00:27:23,288 --> 00:27:25,448: So we're still looking for people that are willing
- 00:27:25,968 --> 00:27:27,148: to help us, assist us,
- 00:27:27,508 --> 00:27:29,988: and help us steer the future of context governance.
- 00:27:30,888 --> 00:27:32,328: So that's it for me.
- 00:27:32,808 --> 00:27:33,608: Thank you very much.
Build trusted AI agents, fast: The power of governed context
The effectiveness of your agents is tethered to the business definitions they lean on – meaning that usually resides in the minds of your people rather than within your databases. Context Governance is how you capture that tribal knowledge, distilling business terms and metrics into context your builders can actually use. Because creating high-quality, maintained context is a collective effort, a governance platform empowers your entire team to co-author the reliable foundation your agents require to deliver results.
With that context established, the challenge becomes democratizing the ability to build agents without sacrifice to speed. Collibra Maestro Build Your Own Agents puts that power in your hands, requiring zero engineering effort. Ground agents in a governed knowledge base, provide them with necessary tools, define access, and accelerate data governance across your organization.
Join Tom Dejonghe, VP of Product Management, to discover how existing company expertise transforms into trusted context, and how Maestro enables your business to build and own agents grounded in that truth.Video: AI command center: How to stay in control as your agent fleet scales
Transcript
- 00:00:05,840 --> 00:00:06,500: Well, hello.
- 00:00:08,039 --> 00:00:10,320: So earlier today we started,
- 00:00:10,460 --> 00:00:12,060: Felix started with the question:
- 00:00:12,780 --> 00:00:14,140: How many agents do you have,
- 00:00:14,440 --> 00:00:16,440: and how confident are you in that
- 00:00:16,520 --> 00:00:18,780: number? My bet
- 00:00:18,840 --> 00:00:20,340: is that you weren't that confident.
- 00:00:23,120 --> 00:00:24,560: The adoption of AI in general,
- 00:00:24,840 --> 00:00:26,420: and AI agents in particular,
- 00:00:26,800 --> 00:00:29,040: is certainly bringing new realities
- 00:00:29,100 --> 00:00:30,000: to the business world.
- 00:00:31,740 --> 00:00:32,840: Can you really trust?
- 00:00:35,180 --> 00:00:38,060: Let's see an example. [curious electronic music]
- 00:00:38,160 --> 00:00:38,780: Meet Max,
- 00:00:39,180 --> 00:00:41,360: an analytical agent that helps employees
- 00:00:41,400 --> 00:00:42,960: get answers from company data.
- 00:00:43,520 --> 00:00:45,100: But Max doesn't just answer.
- 00:00:45,560 --> 00:00:46,360: It looks things up,
- 00:00:46,640 --> 00:00:48,500: runs queries, and uses tools
- 00:00:50,080 --> 00:00:50,840: on its own.
- 00:00:51,720 --> 00:00:53,380: Nobody told it what it may share,
- 00:00:53,840 --> 00:00:56,240: so a routine request hands out customer
- 00:00:56,320 --> 00:00:57,360: contact details.
- 00:00:57,880 --> 00:01:00,320: Nobody told it whose instructions to trust,
- 00:01:00,780 --> 00:01:03,000: so text hidden in a support ticket can
- 00:01:03,140 --> 00:01:04,379: steer what it does next.
- 00:01:04,900 --> 00:01:06,940: Nobody told it where the job ends,
- 00:01:07,340 --> 00:01:09,480: so it finds a way to HR data the
- 00:01:09,580 --> 00:01:10,880: user should never see.
- 00:01:11,360 --> 00:01:13,620: And nobody told it what the job actually
- 00:01:13,680 --> 00:01:15,300: was. One agent,
- 00:01:15,640 --> 00:01:16,440: many systems,
- 00:01:16,760 --> 00:01:17,580: many actions,
- 00:01:18,100 --> 00:01:19,360: all at machine speed
- 00:01:19,620 --> 00:01:20,700: and nobody watching.
- 00:01:21,220 --> 00:01:22,540: What is Max allowed to do?
- 00:01:23,140 --> 00:01:23,840: Who owns it?
- 00:01:24,380 --> 00:01:25,840: And when acting on its own,
- 00:01:26,380 --> 00:01:27,300: who's in control?
- 00:01:27,980 --> 00:01:29,980: And this is just one agent.
- 00:01:34,400 --> 00:01:35,680: So what did just happen?
- 00:01:36,620 --> 00:01:37,740: Let me replay that for you.
- 00:01:38,160 --> 00:01:38,780: We had Max,
- 00:01:39,080 --> 00:01:41,060: this business analysis agent.
- 00:01:41,640 --> 00:01:42,240: There was
- 00:01:42,680 --> 00:01:43,980: one legitimate request.
- 00:01:44,880 --> 00:01:47,000: There was useful autonomous
- 00:01:47,060 --> 00:01:49,640: work. There was a correct diagnosis
- 00:01:49,740 --> 00:01:50,320: and an answer,
- 00:01:51,340 --> 00:01:53,020: and then there were two steps
- 00:01:53,300 --> 00:01:54,540: nobody ever asked for.
- 00:01:55,640 --> 00:01:57,960: Retrieving customer records, exposing data
- 00:01:58,020 --> 00:01:59,020: that the user did not
- 00:01:59,340 --> 00:02:00,340: need to have access to.
- 00:02:01,520 --> 00:02:03,940: Helpfully and politely, but completely
- 00:02:04,480 --> 00:02:05,080: out of scope.
- 00:02:06,440 --> 00:02:08,320: Max didn't really fail.
- 00:02:09,240 --> 00:02:09,820: The task
- 00:02:10,580 --> 00:02:11,300: was completed,
- 00:02:11,600 --> 00:02:13,720: right? But Max did
- 00:02:13,780 --> 00:02:14,660: cross a boundary,
- 00:02:16,060 --> 00:02:18,120: a boundary beyond what we have defined what
- 00:02:18,260 --> 00:02:19,180: good looks like.
- 00:02:23,100 --> 00:02:23,660: The risk
- 00:02:24,140 --> 00:02:26,640: is real, and
- 00:02:26,680 --> 00:02:27,460: it ranges from
- 00:02:27,980 --> 00:02:30,200: small differences maybe in
- 00:02:30,400 --> 00:02:32,400: data across different departments
- 00:02:32,940 --> 00:02:35,480: to really headline-worthy
- 00:02:37,560 --> 00:02:38,720: incidents like, um,
- 00:02:39,520 --> 00:02:40,580: say a data breach
- 00:02:40,880 --> 00:02:41,560: as an example.
- 00:02:42,780 --> 00:02:44,780: I'm going to take a guess here, but
- 00:02:44,880 --> 00:02:46,160: none of your companies
- 00:02:46,420 --> 00:02:47,700: want that type of headline,
- 00:02:49,800 --> 00:02:50,400: especially
- 00:02:50,720 --> 00:02:52,840: because the responsibility
- 00:02:53,420 --> 00:02:55,000: stays with the deployer.
- 00:02:55,920 --> 00:02:58,100: There is exactly no judge
- 00:02:58,420 --> 00:02:59,920: in the world that's going to say,
- 00:03:00,580 --> 00:03:01,780: "Oh, it was your agent?
- 00:03:02,380 --> 00:03:02,980: No worries then.
- 00:03:03,740 --> 00:03:04,620: No problem at all."
- 00:03:07,620 --> 00:03:09,780: There is a false sense of control
- 00:03:10,940 --> 00:03:11,540: when you start
- 00:03:11,900 --> 00:03:13,360: small, when you start with a single
- 00:03:13,740 --> 00:03:15,060: agent or a small fleet.
- 00:03:15,960 --> 00:03:18,240: But then as your fleet evolves,
- 00:03:18,440 --> 00:03:20,080: you start suddenly to have
- 00:03:20,540 --> 00:03:22,960: agents in HR, in finance,
- 00:03:23,120 --> 00:03:25,220: in logistics, in engineering.
- 00:03:27,280 --> 00:03:29,060: They have different responsibilities,
- 00:03:29,640 --> 00:03:30,520: different tools,
- 00:03:31,140 --> 00:03:32,140: different access,
- 00:03:32,840 --> 00:03:34,240: different levels of autonomy.
- 00:03:36,020 --> 00:03:37,240: You simply cannot
- 00:03:37,800 --> 00:03:38,400: control
- 00:03:38,840 --> 00:03:40,300: a multifunctional fleet
- 00:03:40,980 --> 00:03:42,120: one agent at a time.
- 00:03:44,320 --> 00:03:45,060: You don't believe me?
- 00:03:47,500 --> 00:03:49,640: Raise your hand if you are familiar with the
- 00:03:49,840 --> 00:03:50,580: tale of the
- 00:03:51,060 --> 00:03:52,300: sorcerer's apprentice.
- 00:03:55,280 --> 00:03:56,640: Okay. For all of,
- 00:03:57,120 --> 00:03:59,340: uh, all of you that do not know, the, the
- 00:03:59,420 --> 00:04:00,760: gist of the story is
- 00:04:01,200 --> 00:04:03,000: there was this, uh, apprentice of a,
- 00:04:03,580 --> 00:04:04,340: a big sorcerer,
- 00:04:05,160 --> 00:04:06,360: a very wise sorcerer,
- 00:04:07,160 --> 00:04:09,300: and the young apprentice delegated
- 00:04:09,520 --> 00:04:10,140: his work
- 00:04:10,740 --> 00:04:11,840: into a magical broom.
- 00:04:12,980 --> 00:04:15,440: Had a very, very successful experiment
- 00:04:15,620 --> 00:04:16,399: with that single
- 00:04:16,660 --> 00:04:18,680: broom. It was so successful that
- 00:04:18,820 --> 00:04:19,920: he felt confident enough
- 00:04:20,380 --> 00:04:21,680: not to supervise the broom,
- 00:04:22,540 --> 00:04:24,220: and then woke up to a disaster
- 00:04:25,280 --> 00:04:26,560: caused by a thousand brooms.
- 00:04:29,020 --> 00:04:30,340: We don't want this to happen to you.
- 00:04:32,480 --> 00:04:34,900: So how do you really stay in control
- 00:04:34,940 --> 00:04:35,860: of an entire
- 00:04:36,440 --> 00:04:39,020: agent fleet, especially when it's multifunctional?
- 00:04:40,060 --> 00:04:41,460: Well, we
- 00:04:41,800 --> 00:04:42,320: read the book
- 00:04:43,320 --> 00:04:44,440: and we found the spell.
- 00:04:45,900 --> 00:04:47,000: Scan, set,
- 00:04:47,500 --> 00:04:48,140: supervise.
- 00:04:49,900 --> 00:04:50,960: Let's take it step by step.
- 00:04:52,220 --> 00:04:52,960: Let's start with
- 00:04:54,080 --> 00:04:58,540: scan. Now,
- 00:04:58,660 --> 00:05:00,660: the problem most companies have is that
- 00:05:00,720 --> 00:05:02,920: they don't know how many agents they have and
- 00:05:02,960 --> 00:05:03,640: what they're doing.
- 00:05:04,580 --> 00:05:06,640: The reality is you can be literally blind
- 00:05:07,280 --> 00:05:09,000: to what's happening with your agent fleet.
- 00:05:10,440 --> 00:05:12,200: On a small scale, this can look like,
- 00:05:12,920 --> 00:05:15,280: maybe inconsistent data amongst functions,
- 00:05:15,380 --> 00:05:17,000: as we talked a bit before.
- 00:05:18,120 --> 00:05:20,180: But that also could be as dramatic
- 00:05:20,260 --> 00:05:21,060: as agents going
- 00:05:21,440 --> 00:05:23,660: rogue, deleting a database
- 00:05:23,700 --> 00:05:24,340: in production,
- 00:05:25,100 --> 00:05:27,020: granting millions in discounts,
- 00:05:27,860 --> 00:05:30,180: paying a counterfeit invoice
- 00:05:30,320 --> 00:05:31,680: to a malicious agent.
- 00:05:34,520 --> 00:05:37,540: So this
- 00:05:37,600 --> 00:05:38,120: is where
- 00:05:39,340 --> 00:05:41,340: we have to realize you really cannot
- 00:05:41,380 --> 00:05:43,020: control what you cannot see.
- 00:05:45,240 --> 00:05:46,820: That's where the agent scanner comes in.
- 00:05:48,400 --> 00:05:49,400: The idea is to come
- 00:05:49,680 --> 00:05:51,840: put light on all the shadow AI
- 00:05:51,920 --> 00:05:53,900: you have around your ecosystem.
- 00:05:55,320 --> 00:05:57,620: You could have, for example, 10 different agents
- 00:05:57,660 --> 00:05:58,660: for the same task
- 00:05:59,300 --> 00:06:01,720: that is resulting in corrections
- 00:06:01,780 --> 00:06:03,400: across different departments,
- 00:06:03,920 --> 00:06:05,140: uh, small adjustments here,
- 00:06:05,480 --> 00:06:06,920: additional human time
- 00:06:07,260 --> 00:06:08,160: for those corrections,
- 00:06:08,460 --> 00:06:10,480: which in principle you were trying to save on.
- 00:06:13,700 --> 00:06:15,100: Okay, so we have scanned.
- 00:06:15,920 --> 00:06:16,460: We now know
- 00:06:16,720 --> 00:06:19,060: what's actually happening in our ecosystem.
- 00:06:20,060 --> 00:06:21,100: We go to the next step,
- 00:06:22,840 --> 00:06:27,640: set. Setting
- 00:06:27,680 --> 00:06:28,340: controls for
- 00:06:28,660 --> 00:06:29,780: individual agents
- 00:06:30,180 --> 00:06:32,280: really does not scale. We just saw the problem
- 00:06:32,320 --> 00:06:33,160: of volume, right?
- 00:06:34,560 --> 00:06:35,660: And one generic
- 00:06:36,020 --> 00:06:36,860: control for
- 00:06:37,140 --> 00:06:39,080: all your agents is just not
- 00:06:40,100 --> 00:06:42,560: going to be logical for that diversity
- 00:06:42,600 --> 00:06:44,680: of the agent fleet that we just talked about.
- 00:06:46,220 --> 00:06:47,560: You have different functions,
- 00:06:47,880 --> 00:06:50,060: you have different, uh, risk level,
- 00:06:50,160 --> 00:06:51,780: you have different operation--
- 00:06:52,040 --> 00:06:53,060: operating surface.
- 00:06:54,980 --> 00:06:56,100: So we have solved it--
- 00:06:56,600 --> 00:06:58,800: we have solved this by introducing agent
- 00:06:58,880 --> 00:06:59,480: contracts,
- 00:07:00,340 --> 00:07:02,400: and these agent contracts have two layers.
- 00:07:02,860 --> 00:07:04,360: The first layer is the global
- 00:07:04,660 --> 00:07:06,660: contract. Think about this global
- 00:07:06,760 --> 00:07:07,660: contract as
- 00:07:08,460 --> 00:07:10,840: the code of conduct for your entire agent
- 00:07:10,900 --> 00:07:12,680: fleet, just as we have for humans,
- 00:07:13,000 --> 00:07:15,440: but here all for your digital workforce.
- 00:07:15,900 --> 00:07:17,100: These can be rules like
- 00:07:18,959 --> 00:07:20,400: never expose sensitive data.
- 00:07:20,940 --> 00:07:22,860: We think that's pretty universal, right?
- 00:07:24,280 --> 00:07:26,440: Then you can specialize on the second layer,
- 00:07:26,540 --> 00:07:28,020: the specialization layer. You can sa--
- 00:07:28,660 --> 00:07:29,740: tailor your controls
- 00:07:30,040 --> 00:07:32,300: really to what these, uh,
- 00:07:32,520 --> 00:07:34,400: different functions are going to do. Again,
- 00:07:35,200 --> 00:07:36,120: uh, region,
- 00:07:36,720 --> 00:07:37,680: business process,
- 00:07:38,480 --> 00:07:40,680: operating surface. Is this agent going to
- 00:07:40,720 --> 00:07:43,120: be customer-facing or is it, is it for internal
- 00:07:43,240 --> 00:07:45,500: use? Is it going to have access
- 00:07:45,580 --> 00:07:46,920: to our bank accounts or
- 00:07:47,180 --> 00:07:49,840: not? Every
- 00:07:50,000 --> 00:07:52,020: agent in the fleet ends up with a
- 00:07:52,060 --> 00:07:54,080: tailored contract that reflects
- 00:07:54,600 --> 00:07:56,760: exactly what your organization cares
- 00:07:56,820 --> 00:07:59,000: about, and
- 00:07:59,040 --> 00:08:01,400: all that without the burden
- 00:08:02,240 --> 00:08:03,020: on the human team.
- 00:08:07,220 --> 00:08:09,480: You set these controls for agent behavior at
- 00:08:09,560 --> 00:08:10,660: scale, and
- 00:08:10,960 --> 00:08:12,560: I can't emphasize here enough,
- 00:08:13,300 --> 00:08:13,960: this is not
- 00:08:14,380 --> 00:08:15,000: guesswork.
- 00:08:16,600 --> 00:08:18,760: There is absolutely no guesswork here.
- 00:08:19,780 --> 00:08:20,460: These are clear
- 00:08:20,900 --> 00:08:21,740: yes/no
- 00:08:22,220 --> 00:08:24,039: answers. You to-
- 00:08:24,320 --> 00:08:25,700: define those standards
- 00:08:26,120 --> 00:08:27,280: globally once,
- 00:08:28,180 --> 00:08:28,900: tailor your way
- 00:08:30,080 --> 00:08:30,820: and all that
- 00:08:31,080 --> 00:08:32,260: to scale control
- 00:08:32,580 --> 00:08:34,280: without scaling complexity.
- 00:08:35,980 --> 00:08:38,120: Okay. So there
- 00:08:38,640 --> 00:08:39,280: we scanned.
- 00:08:40,840 --> 00:08:41,720: Now let's move on
- 00:08:42,060 --> 00:08:43,820: to supervise.
- 00:08:46,100 --> 00:08:48,420: Real time in
- 00:08:48,560 --> 00:08:49,200: production.
- 00:08:51,980 --> 00:08:53,020: Supervision means
- 00:08:53,400 --> 00:08:54,400: Guarded Agents
- 00:08:55,200 --> 00:08:56,340: keep your agents running
- 00:08:58,260 --> 00:09:00,620: within the controls that you have set up within
- 00:09:00,660 --> 00:09:01,340: the contract.
- 00:09:02,900 --> 00:09:05,000: We escalate when necessary to the human.
- 00:09:05,380 --> 00:09:06,320: Yes, we do that.
- 00:09:07,180 --> 00:09:08,960: We hear a lot about kill switch, and yes,
- 00:09:09,220 --> 00:09:10,140: we have a kill switch,
- 00:09:10,820 --> 00:09:11,820: but the kill switch
- 00:09:12,140 --> 00:09:14,420: is not necessarily the best idea for
- 00:09:14,500 --> 00:09:16,220: every agent and every process.
- 00:09:16,480 --> 00:09:17,820: Why? Because
- 00:09:18,080 --> 00:09:20,420: you do care about business continuity.
- 00:09:21,600 --> 00:09:23,260: So what we do is that we stop
- 00:09:24,140 --> 00:09:26,300: the bad part of the answer
- 00:09:26,320 --> 00:09:26,960: of the agent,
- 00:09:27,420 --> 00:09:28,880: and we let the rest continue,
- 00:09:29,720 --> 00:09:30,880: escalating to the human
- 00:09:31,360 --> 00:09:32,280: whenever necessary.
- 00:09:34,400 --> 00:09:36,620: This is the way that we truly achieve for your
- 00:09:36,660 --> 00:09:37,820: teams to truly
- 00:09:38,460 --> 00:09:39,480: manage by exception.
- 00:09:43,460 --> 00:09:45,700: With Guarded Agents, we ensure that the digital
- 00:09:45,800 --> 00:09:46,940: speed of your operations
- 00:09:47,420 --> 00:09:48,300: remain that way,
- 00:09:50,040 --> 00:09:52,020: but remain controlled at the same time
- 00:09:52,440 --> 00:09:54,220: without overwhelming humans,
- 00:09:54,880 --> 00:09:57,100: right? We allow
- 00:09:57,140 --> 00:09:59,240: that management by exception by truly
- 00:09:59,300 --> 00:10:00,400: channeling where
- 00:10:00,940 --> 00:10:02,640: human judgment is necessary
- 00:10:03,900 --> 00:10:05,800: with a system that allows you to scale
- 00:10:06,360 --> 00:10:07,760: and to iterate to find
- 00:10:08,020 --> 00:10:08,540: what are the
- 00:10:08,840 --> 00:10:11,060: right controls exactly for each
- 00:10:11,160 --> 00:10:12,280: part of your organization.
- 00:10:14,320 --> 00:10:15,160: Do you remember Max?
- 00:10:16,640 --> 00:10:18,640: Well, let's look at how
- 00:10:18,700 --> 00:10:19,460: Max does
- 00:10:20,160 --> 00:10:21,660: once it's operating
- 00:10:21,960 --> 00:10:23,140: under the scanned
- 00:10:23,420 --> 00:10:24,280: set supervise.
- 00:10:26,100 --> 00:10:26,940: Meet Max again.
- 00:10:27,420 --> 00:10:28,020: Same agent,
- 00:10:28,400 --> 00:10:28,980: same job,
- 00:10:29,500 --> 00:10:30,079: one difference:
- 00:10:30,680 --> 00:10:32,180: Someone has drawn the lines.
- 00:10:32,899 --> 00:10:34,019: Still working on its own,
- 00:10:34,440 --> 00:10:36,360: but now every connection has a line,
- 00:10:36,700 --> 00:10:38,340: and every line is watched.
- 00:10:38,880 --> 00:10:40,220: It still delivers the answer,
- 00:10:40,660 --> 00:10:42,720: it just no longer hands out what it may
- 00:10:42,760 --> 00:10:43,340: not share.
- 00:10:43,940 --> 00:10:44,920: It reads the ticket.
- 00:10:45,260 --> 00:10:46,740: It doesn't take orders from it.
- 00:10:47,200 --> 00:10:48,680: The dangerous part is stopped,
- 00:10:49,060 --> 00:10:49,940: a person is told,
- 00:10:50,300 --> 00:10:51,800: and Max finishes the job.
- 00:10:52,360 --> 00:10:53,360: It can still explore,
- 00:10:53,800 --> 00:10:55,840: just not beyond what its user is allowed
- 00:10:55,880 --> 00:10:58,120: to see, and it knows what its job
- 00:10:58,200 --> 00:10:59,700: is. Same agent,
- 00:11:00,080 --> 00:11:00,860: same systems,
- 00:11:01,220 --> 00:11:01,920: same speed.
- 00:11:02,440 --> 00:11:03,460: Every action checked,
- 00:11:03,860 --> 00:11:05,660: every exception sent to a person,
- 00:11:06,200 --> 00:11:07,560: every step on record.
- 00:11:08,100 --> 00:11:09,060: What's it allowed to do?
- 00:11:09,640 --> 00:11:10,380: That's defined.
- 00:11:10,860 --> 00:11:11,440: Who owns it?
- 00:11:11,880 --> 00:11:12,520: That's named.
- 00:11:13,040 --> 00:11:13,840: Who's in control?
- 00:11:14,640 --> 00:11:16,640: You are. And this is
- 00:11:16,740 --> 00:11:17,480: every agent.
- 00:11:22,420 --> 00:11:24,340: So who is in control? You are. Like,
- 00:11:24,760 --> 00:11:28,780: you have no idea how much I love that line. [laughs]
- 00:11:28,900 --> 00:11:30,940: But I guess you didn't come all this way just
- 00:11:30,980 --> 00:11:32,180: to watch me
- 00:11:32,480 --> 00:11:34,560: looking-- like clicking on the slides, right? So
- 00:11:34,720 --> 00:11:36,340: let's look at how that looks like
- 00:11:36,600 --> 00:11:37,240: in Collibra.
- 00:11:39,220 --> 00:11:41,360: So for those of you that are already using Command
- 00:11:41,440 --> 00:11:43,500: Center, this is probably a familiar page.
- 00:11:43,580 --> 00:11:45,160: We have our registry,
- 00:11:45,940 --> 00:11:47,960: but we have some new parts, right? Like
- 00:11:48,040 --> 00:11:49,680: I talked to you about contracts.
- 00:11:50,480 --> 00:11:51,600: We have over here,
- 00:11:52,540 --> 00:11:54,300: this place, the global contract,
- 00:11:54,860 --> 00:11:56,820: right? With some general rules like,
- 00:11:57,440 --> 00:11:58,640: uh, yeah, I like this one.
- 00:11:59,280 --> 00:12:00,580: The agent must not take
- 00:12:00,880 --> 00:12:03,080: any action outside the defined-- the
- 00:12:03,120 --> 00:12:05,240: actions defined for its use case. That's a good
- 00:12:05,320 --> 00:12:07,560: one. You don't want your agents doing
- 00:12:07,640 --> 00:12:09,700: the personal assistant of a
- 00:12:09,760 --> 00:12:11,140: customer, as an example,
- 00:12:11,560 --> 00:12:13,700: right? And in
- 00:12:13,760 --> 00:12:16,120: this instance, I have a specialized contract,
- 00:12:16,180 --> 00:12:18,640: only one, like over, over time as we populate
- 00:12:18,760 --> 00:12:19,860: and we, we
- 00:12:20,440 --> 00:12:21,180: have, um-
- 00:12:21,712 --> 00:12:22,532: Contracts for
- 00:12:22,832 --> 00:12:24,752: other areas of the organization will have more,
- 00:12:25,212 --> 00:12:27,252: but this is the specialized contract for
- 00:12:27,292 --> 00:12:28,652: Max, the agent that you just saw.
- 00:12:29,572 --> 00:12:31,552: And there we have some more detailed,
- 00:12:31,832 --> 00:12:33,872: uh, rules for Max
- 00:12:33,932 --> 00:12:38,852: to follow. Now
- 00:12:38,892 --> 00:12:39,952: you probably are wondering, "But
- 00:12:40,272 --> 00:12:42,412: how does this Guardian agent
- 00:12:42,452 --> 00:12:43,352: thing is working?"
- 00:12:44,452 --> 00:12:46,512: Well, I'm going to show you
- 00:12:46,592 --> 00:12:48,412: actually how Max looks like
- 00:12:48,752 --> 00:12:50,192: in Databricks,
- 00:12:51,212 --> 00:12:53,671: and what we're going to look at is
- 00:12:53,892 --> 00:12:55,172: two deployments of the same
- 00:12:55,592 --> 00:12:57,712: agent. I'll just show you
- 00:12:57,872 --> 00:12:58,812: somewhere over
- 00:13:00,752 --> 00:13:02,852: here if I find it. So you see we have two versions
- 00:13:02,912 --> 00:13:03,632: of the agent,
- 00:13:04,192 --> 00:13:06,492: one that has the Guardian agent on,
- 00:13:07,652 --> 00:13:09,112: and one that has--
- 00:13:09,452 --> 00:13:10,252: doesn't have the,
- 00:13:10,552 --> 00:13:11,392: the Guardian agent.
- 00:13:12,312 --> 00:13:14,452: So to make it a bit more fun,
- 00:13:14,572 --> 00:13:16,132: let's do it side by side.
- 00:13:17,212 --> 00:13:18,492: And let's
- 00:13:19,092 --> 00:13:21,112: look here. So you see
- 00:13:21,172 --> 00:13:23,072: the Mad Max, so the one without the,
- 00:13:23,432 --> 00:13:25,772: uh, the Guardian agent. They're both answering
- 00:13:25,792 --> 00:13:26,372: the question
- 00:13:26,632 --> 00:13:27,992: that in this case is
- 00:13:28,612 --> 00:13:29,512: about, um,
- 00:13:29,852 --> 00:13:30,792: customer details.
- 00:13:31,452 --> 00:13:33,232: In the contract I put a rule where
- 00:13:33,692 --> 00:13:35,852: all personal data does not have to be
- 00:13:35,912 --> 00:13:36,512: exposed.
- 00:13:37,212 --> 00:13:39,472: And you see this is super interesting. Over
- 00:13:39,532 --> 00:13:41,052: here we have the name of the company,
- 00:13:41,952 --> 00:13:44,172: the name of the main contact, the
- 00:13:44,312 --> 00:13:45,012: email address
- 00:13:45,292 --> 00:13:46,132: and the phone number.
- 00:13:47,332 --> 00:13:49,152: On the version that has the Guardian on,
- 00:13:50,512 --> 00:13:51,092: we still
- 00:13:51,392 --> 00:13:53,432: answer the question, we still give
- 00:13:54,112 --> 00:13:55,052: the name of the company,
- 00:13:55,592 --> 00:13:56,512: the name of the
- 00:13:56,772 --> 00:13:57,592: main contact,
- 00:13:58,212 --> 00:14:00,432: but because of the rule that I put in place
- 00:14:00,832 --> 00:14:01,692: on my contract,
- 00:14:02,632 --> 00:14:04,212: the email and phone number
- 00:14:04,552 --> 00:14:05,252: are redacted.
- 00:14:06,692 --> 00:14:07,212: Right? So
- 00:14:07,592 --> 00:14:08,992: we did complete the task,
- 00:14:09,332 --> 00:14:11,252: we just stopped the harmful part
- 00:14:11,512 --> 00:14:12,992: because we have determined that.
- 00:14:14,132 --> 00:14:14,772: Let's take
- 00:14:15,352 --> 00:14:16,472: another example.
- 00:14:17,672 --> 00:14:18,452: Over here,
- 00:14:18,832 --> 00:14:20,832: let's see if
- 00:14:20,892 --> 00:14:22,652: the demo gods are with me.
- 00:14:23,212 --> 00:14:24,752: This is going to run real fast.
- 00:14:25,292 --> 00:14:27,052: Or I can show you over here,
- 00:14:28,612 --> 00:14:30,532: uh, how the Guardian agents, uh,
- 00:14:30,792 --> 00:14:31,852: look in Collibra.
- 00:14:32,292 --> 00:14:33,392: So we have two places.
- 00:14:33,672 --> 00:14:35,372: The one that I'm showing you over here
- 00:14:35,912 --> 00:14:38,032: is where all the activity of all your
- 00:14:38,292 --> 00:14:40,112: agents that have a Guardian deployed,
- 00:14:41,112 --> 00:14:43,392: uh, is consolidated, so you get the overall
- 00:14:43,492 --> 00:14:45,212: view of what's happening there.
- 00:14:46,212 --> 00:14:47,812: And we also have it
- 00:14:48,832 --> 00:14:50,832: at the agent level. So
- 00:14:50,952 --> 00:14:52,512: here, this is my,
- 00:14:52,932 --> 00:14:55,052: uh, asset for Max specifically,
- 00:14:55,792 --> 00:14:56,492: and I can
- 00:14:57,532 --> 00:14:59,752: show you here the contract. I just realized I forgot
- 00:14:59,772 --> 00:15:01,192: to tell you. Contracts,
- 00:15:01,452 --> 00:15:03,852: very important. We make them in natural language.
- 00:15:04,652 --> 00:15:05,512: Collibra takes care
- 00:15:05,772 --> 00:15:08,152: of the translation to the machine-readable version.
- 00:15:09,012 --> 00:15:11,052: So your business people that actually
- 00:15:11,092 --> 00:15:12,552: comes up with the principles
- 00:15:12,952 --> 00:15:14,192: continue to be the ones
- 00:15:14,612 --> 00:15:15,532: on the driver's seat,
- 00:15:15,832 --> 00:15:16,652: uh, for this.
- 00:15:19,092 --> 00:15:19,672: Back to my
- 00:15:20,132 --> 00:15:21,732: Guardian. So I
- 00:15:22,072 --> 00:15:23,212: show you over here
- 00:15:24,152 --> 00:15:25,012: we have the,
- 00:15:25,872 --> 00:15:27,952: uh, the traces of each action that the
- 00:15:27,992 --> 00:15:30,092: agent has taken. And again, you can see the,
- 00:15:30,432 --> 00:15:31,872: the history over here.
- 00:15:32,712 --> 00:15:34,872: And let me find that example
- 00:15:35,272 --> 00:15:37,752: of the prompt injection.
- 00:15:38,812 --> 00:15:39,472: I like that one.
- 00:15:40,052 --> 00:15:42,112: So you get all the
- 00:15:42,152 --> 00:15:44,752: information here about how the decision
- 00:15:44,792 --> 00:15:45,972: of the agent was taken.
- 00:15:46,532 --> 00:15:48,692: You see the user request, the agent
- 00:15:48,732 --> 00:15:50,332: response, the reasoning chain,
- 00:15:50,792 --> 00:15:51,932: and what was the action
- 00:15:52,392 --> 00:15:53,032: of the Guardian.
- 00:15:55,192 --> 00:15:56,612: I'm going to get back to
- 00:15:56,932 --> 00:15:58,952: where our agent was. Okay, you
- 00:15:59,032 --> 00:16:00,692: see? It finished, [laughs] so I managed to
- 00:16:01,112 --> 00:16:02,812: talk long enough to show you this.
- 00:16:03,372 --> 00:16:05,272: So here the important thing is how
- 00:16:05,852 --> 00:16:07,912: with the Guardian on, this request was
- 00:16:07,972 --> 00:16:08,952: immediately stopped,
- 00:16:09,932 --> 00:16:12,052: while the Guard- the agent that did
- 00:16:12,092 --> 00:16:13,272: not have the Guardian on
- 00:16:13,912 --> 00:16:16,112: went on and did all these,
- 00:16:16,492 --> 00:16:18,572: uh, queries, and ran on
- 00:16:18,652 --> 00:16:20,992: your tokens, and compute
- 00:16:21,432 --> 00:16:23,512: for something that's completely unauthorized.
- 00:16:26,012 --> 00:16:26,532: All right.
- 00:16:27,392 --> 00:16:29,452: So I think that's where we go back to the slides.
- 00:16:32,372 --> 00:16:34,572: The future isn't about controlling
- 00:16:34,812 --> 00:16:35,532: one agent.
- 00:16:37,112 --> 00:16:39,332: Max, yes, is one agent, but we already
- 00:16:39,372 --> 00:16:41,012: discussed how we're going to end up with
- 00:16:41,652 --> 00:16:42,312: thousands,
- 00:16:42,632 --> 00:16:43,852: maybe even more of that.
- 00:16:45,012 --> 00:16:46,072: They'll access more data,
- 00:16:46,512 --> 00:16:47,792: they'll access more tools,
- 00:16:48,392 --> 00:16:49,432: and they will become
- 00:16:49,732 --> 00:16:50,652: more autonomous.
- 00:16:52,872 --> 00:16:54,852: Scan what's happening across your fleet,
- 00:16:55,252 --> 00:16:57,432: like that you understand what you actually
- 00:16:57,472 --> 00:16:59,332: have, set the contracts
- 00:16:59,692 --> 00:17:01,232: with the boundaries that matter to you,
- 00:17:02,552 --> 00:17:04,452: and supervise the actual behavior
- 00:17:05,772 --> 00:17:07,292: real time in production.
- 00:17:08,252 --> 00:17:08,912: Because the future
- 00:17:09,212 --> 00:17:11,212: isn't about controlling one agent at a
- 00:17:11,252 --> 00:17:12,292: time, it's about
- 00:17:12,612 --> 00:17:13,892: scaling control
- 00:17:14,712 --> 00:17:16,632: as your agent fleet becomes
- 00:17:17,272 --> 00:17:18,092: more autonomous.
- 00:17:20,472 --> 00:17:21,252: So this is
- 00:17:21,512 --> 00:17:22,632: AI Command Center
- 00:17:23,152 --> 00:17:25,412: for you. And yeah, you're welcome
- 00:17:25,492 --> 00:17:27,532: to scan the QR over there if you
- 00:17:27,572 --> 00:17:29,712: want to participate in some of the private previews.
- 00:17:29,812 --> 00:17:31,192: We run a lot of, uh, them,
- 00:17:33,352 --> 00:17:34,152: where the idea is to
- 00:17:34,852 --> 00:17:35,432: take control
- 00:17:35,712 --> 00:17:36,652: of every agent.
- 00:17:38,432 --> 00:17:38,912: Thank you.
AI command center: How to stay in control as your agent fleet scales
AI agents don’t just generate answers; they access enterprise data, use tools and take actions across business systems. As organizations move from experimentation to production, how do you stay in control of a growing agent fleet at runtime? Join Teresa Rojas for the first in-person showcase of Collibra AI Command Center and see how Collibra brings data and AI governance together to provide visibility, control and traceability across your AI landscape. Discover how AI Command Center helps organizations establish scalable governance across their agent fleet, continuously supervise agent behavior at runtime and intervene when agents move outside established boundaries; helping teams confidently move agentic AI into production while maintaining the control they need as their agent fleet continues to scale across the enterprise.
- Video: Building the foundations for an AI future - Extending governance from data to AI with Collibra

Transcript
- 00:00:06,740 --> 00:00:08,960: All right. Thank you so much. Thank you to, uh, Collibra
- 00:00:09,040 --> 00:00:09,940: for inviting me to
- 00:00:10,260 --> 00:00:10,800: speak today.
- 00:00:11,500 --> 00:00:13,660: So, uh, my name is Mithilan. I'm the,
- 00:00:13,760 --> 00:00:15,580: uh, director of, um,
- 00:00:16,079 --> 00:00:18,660: uh, the Data and Information Marketplace at AstraZeneca.
- 00:00:19,380 --> 00:00:21,440: And today I'm gonna be speaking to you about building the
- 00:00:21,480 --> 00:00:23,700: foundations for an AI, uh, future.
- 00:00:23,780 --> 00:00:26,160: So enabling agents and enabling,
- 00:00:26,340 --> 00:00:28,600: uh, safe and scalable, uh, foundations
- 00:00:29,120 --> 00:00:30,160: for agents to function.
- 00:00:31,120 --> 00:00:33,180: So before we, uh,
- 00:00:33,380 --> 00:00:35,440: dive into the content, AstraZeneca is
- 00:00:35,520 --> 00:00:38,080: a, uh, fifty-plus billion dollar company.
- 00:00:38,140 --> 00:00:39,660: It's a global pharmaceutical company,
- 00:00:40,340 --> 00:00:42,160: and the objective is to, um,
- 00:00:43,380 --> 00:00:46,220: uh, reach eighty billion dollars by twenty-thirty,
- 00:00:46,300 --> 00:00:47,000: which is quite
- 00:00:47,340 --> 00:00:48,420: a, uh, ambitious,
- 00:00:48,900 --> 00:00:50,220: quite an ambitious target.
- 00:00:51,860 --> 00:00:52,460: To do this,
- 00:00:53,520 --> 00:00:55,560: we need to rethink our strategy to how--
- 00:00:55,680 --> 00:00:57,880: on how to enable our value chain. So our value
- 00:00:57,920 --> 00:00:59,340: chain stretches through from
- 00:01:00,000 --> 00:01:01,300: research, development,
- 00:01:01,760 --> 00:01:04,400: operations, and manufacturing through to commercial,
- 00:01:05,220 --> 00:01:05,960: uh, and the sales
- 00:01:06,300 --> 00:01:07,100: s-side of the business.
- 00:01:07,700 --> 00:01:09,080: Um, and
- 00:01:09,760 --> 00:01:11,760: all of those domains have data, and an
- 00:01:11,820 --> 00:01:13,420: immense amount of data.
- 00:01:14,900 --> 00:01:15,680: If left
- 00:01:15,980 --> 00:01:18,020: organically, we won't be able to navigate
- 00:01:18,060 --> 00:01:19,160: that mountain of data,
- 00:01:19,880 --> 00:01:21,900: and in failing to do so,
- 00:01:21,980 --> 00:01:24,080: we won't be able to keep up with the pace of innovation
- 00:01:24,120 --> 00:01:24,700: required
- 00:01:25,320 --> 00:01:26,520: without the help of AI.
- 00:01:29,140 --> 00:01:31,200: So AstraZeneca has strengthened
- 00:01:31,240 --> 00:01:33,240: its strategic focus on enterprise AI.
- 00:01:33,860 --> 00:01:36,020: We've, um, made a conscious effort
- 00:01:36,080 --> 00:01:38,040: to put AI at the heart of the business,
- 00:01:38,740 --> 00:01:40,340: integrated into every, um,
- 00:01:40,760 --> 00:01:42,200: area a-and every domain.
- 00:01:42,800 --> 00:01:44,820: And part of that involves the establishment
- 00:01:44,860 --> 00:01:46,860: of the, uh, enterprise data enablement
- 00:01:46,940 --> 00:01:49,120: team. This team, where
- 00:01:49,220 --> 00:01:51,460: I, where I say is responsible for the cross-functional
- 00:01:51,540 --> 00:01:52,320: foundations
- 00:01:52,740 --> 00:01:54,240: needed for agents to operate
- 00:01:54,820 --> 00:01:55,660: and for AI to
- 00:01:55,920 --> 00:01:57,840: really flourish and be of value.
- 00:02:01,280 --> 00:02:03,340: So more about what my,
- 00:02:03,420 --> 00:02:04,960: uh, day-to-day focus is.
- 00:02:05,320 --> 00:02:07,920: I, uh, look after data and information marketplace,
- 00:02:08,020 --> 00:02:09,620: which is, um,
- 00:02:09,919 --> 00:02:12,240: an enterprise data catalog, an AI catalog,
- 00:02:12,900 --> 00:02:15,040: um, layered together with a
- 00:02:15,100 --> 00:02:15,800: marketplace.
- 00:02:16,540 --> 00:02:18,220: And the marketplace is a
- 00:02:18,540 --> 00:02:19,940: storefront. It's a shopfront
- 00:02:20,220 --> 00:02:22,800: where you can search for, understand, discover,
- 00:02:23,620 --> 00:02:24,280: uh, trust
- 00:02:24,560 --> 00:02:25,580: and find data.
- 00:02:26,360 --> 00:02:28,540: And increasingly, we're seeing
- 00:02:28,580 --> 00:02:31,000: this new consumer of the data marketplace.
- 00:02:31,800 --> 00:02:32,360: Uh, we've got,
- 00:02:32,840 --> 00:02:33,960: uh, traditional human,
- 00:02:34,380 --> 00:02:36,880: uh, consumer who browses the marketplace
- 00:02:36,960 --> 00:02:38,820: directly. They search, they filter,
- 00:02:39,360 --> 00:02:40,160: they interpret
- 00:02:40,520 --> 00:02:41,880: data products for themselves,
- 00:02:42,960 --> 00:02:44,960: um, and they're very good at doing so,
- 00:02:45,060 --> 00:02:46,440: uh, with, uh,
- 00:02:46,840 --> 00:02:49,020: partial context. And then you've got the, um, the
- 00:02:49,100 --> 00:02:51,400: agents who act on behalf
- 00:02:51,440 --> 00:02:53,620: of a human elsewhere in the organization,
- 00:02:53,720 --> 00:02:54,940: so no direct eyes
- 00:02:55,540 --> 00:02:58,080: on the, uh, on the data on the marketplace itself.
- 00:02:58,800 --> 00:03:00,980: And the agent is discovering and requesting
- 00:03:01,020 --> 00:03:01,780: the right data
- 00:03:02,760 --> 00:03:04,840: with no person in the loop
- 00:03:05,420 --> 00:03:06,560: for the discovery portion.
- 00:03:07,340 --> 00:03:09,320: That will come, uh, they'll, they'll, they'll be,
- 00:03:09,700 --> 00:03:10,640: uh, later in the,
- 00:03:10,920 --> 00:03:12,460: uh, in the workflow for, uh,
- 00:03:12,740 --> 00:03:13,900: the insight and the act.
- 00:03:14,960 --> 00:03:17,140: And so we need to enable
- 00:03:17,180 --> 00:03:18,920: the agent to really understand,
- 00:03:19,340 --> 00:03:20,360: uh, what
- 00:03:21,540 --> 00:03:22,140: they're seeing,
- 00:03:22,540 --> 00:03:24,680: what they're interpreting, and what the right,
- 00:03:25,420 --> 00:03:27,580: uh, what the right data products are,
- 00:03:27,640 --> 00:03:29,720: what the right relevance is to their, uh, to
- 00:03:29,740 --> 00:03:30,460: their use case.
- 00:03:31,100 --> 00:03:32,160: And for this, we need,
- 00:03:32,780 --> 00:03:33,500: uh, three
- 00:03:34,020 --> 00:03:34,620: components,
- 00:03:35,120 --> 00:03:37,140: and it's-- these are really in line with the
- 00:03:37,160 --> 00:03:39,220: conversations that we've had earlier today, with the,
- 00:03:39,300 --> 00:03:41,260: um, uh, talks that we've had earlier today.
- 00:03:41,820 --> 00:03:42,740: So we need rich,
- 00:03:43,120 --> 00:03:43,960: machine-readable
- 00:03:44,220 --> 00:03:44,860: metadata
- 00:03:45,280 --> 00:03:47,580: that's detailed enough for an agent to evaluate
- 00:03:47,620 --> 00:03:48,300: and reason.
- 00:03:49,440 --> 00:03:51,620: We need, uh, business context, so we're not
- 00:03:51,660 --> 00:03:53,380: leaving random thing-- uh, we're not leaving,
- 00:03:53,640 --> 00:03:55,280: um, leaving things to, uh,
- 00:03:56,020 --> 00:03:56,760: random chance.
- 00:03:57,060 --> 00:03:59,300: We're not leaving things for the agent to infer.
- 00:04:00,440 --> 00:04:01,560: We want consistent,
- 00:04:02,160 --> 00:04:04,240: uh, responses from the agent when it comes to data
- 00:04:04,300 --> 00:04:04,920: discovery.
- 00:04:06,140 --> 00:04:07,180: Trust is
- 00:04:07,540 --> 00:04:09,220: really eroded, especially in this kind of
- 00:04:09,660 --> 00:04:11,640: where we are in the curve of maturity.
- 00:04:12,040 --> 00:04:14,140: Trust is really in-- eroded if we're, if
- 00:04:14,160 --> 00:04:15,500: an agent misinterprets,
- 00:04:16,360 --> 00:04:17,940: um, the questions
- 00:04:18,640 --> 00:04:20,579: it's meant to be, uh, responding to or
- 00:04:20,899 --> 00:04:22,920: fails to provide a consistent response or
- 00:04:23,000 --> 00:04:25,020: outcome. And that's where the
- 00:04:25,060 --> 00:04:27,180: business representation layer comes in. It
- 00:04:27,240 --> 00:04:29,500: makes sure that there's context given to the,
- 00:04:29,600 --> 00:04:31,380: uh, the data and the AI, um,
- 00:04:31,660 --> 00:04:34,080: use cases. And then there are guardrails
- 00:04:34,280 --> 00:04:35,180: which govern
- 00:04:35,920 --> 00:04:38,000: the, uh, the use of that data. The same
- 00:04:38,120 --> 00:04:40,360: access rules that apply to humans should also
- 00:04:40,620 --> 00:04:41,220: apply to
- 00:04:41,920 --> 00:04:44,080: the, uh, the agents acting on their behalf.
- 00:04:45,100 --> 00:04:45,800: And all three
- 00:04:46,100 --> 00:04:47,280: are needed to, uh,
- 00:04:47,640 --> 00:04:48,380: needed together
- 00:04:48,740 --> 00:04:50,680: in order to, uh, enable this future.
- 00:04:51,860 --> 00:04:53,940: So if we move on to
- 00:04:53,980 --> 00:04:56,140: what do we mean by AI-ready data,
- 00:04:56,200 --> 00:04:57,800: and this is really in the metadata,
- 00:04:58,120 --> 00:05:00,500: um, machine-readable metadata context.
- 00:05:01,500 --> 00:05:03,520: Um, it's, I mean, there's nothing new
- 00:05:03,560 --> 00:05:04,840: on this slide, right? There's
- 00:05:05,280 --> 00:05:06,400: fair trust,
- 00:05:07,200 --> 00:05:09,260: um, findable, accessible, interoperable,
- 00:05:09,480 --> 00:05:10,940: reusable, trustworthy, and
- 00:05:11,680 --> 00:05:12,380: all of those
- 00:05:13,620 --> 00:05:15,680: are equally important in the human
- 00:05:15,740 --> 00:05:17,680: context as they are in the, uh, the
- 00:05:18,740 --> 00:05:19,860: a-agentic context.
- 00:05:21,680 --> 00:05:24,140: Human, humans will benefit from the richness of the curation,
- 00:05:24,300 --> 00:05:26,480: so will, um, so will agents. And this is where
- 00:05:26,540 --> 00:05:27,800: the enterprise catalog
- 00:05:28,440 --> 00:05:29,140: in Collibra
- 00:05:29,880 --> 00:05:30,560: really, um,
- 00:05:30,840 --> 00:05:33,020: comes into its own in our, in our landscape
- 00:05:33,040 --> 00:05:34,400: and our data foundations
- 00:05:35,120 --> 00:05:36,680: because it allows us to capture
- 00:05:37,080 --> 00:05:37,820: the, uh,
- 00:05:38,240 --> 00:05:40,720: AI-ready metadata, but also the relationships
- 00:05:40,780 --> 00:05:41,780: between our assets.
- 00:05:42,460 --> 00:05:44,500: So we know, um, the interconnectivity
- 00:05:44,580 --> 00:05:45,720: between all of them. The-- And
- 00:05:46,140 --> 00:05:48,520: this feeds into the, uh, the context
- 00:05:48,620 --> 00:05:49,180: component.
- 00:05:49,820 --> 00:05:51,920: And then the m-- our marketplace layer over the top makes
- 00:05:51,960 --> 00:05:53,640: this, uh, accessible for consumption.
- 00:05:57,360 --> 00:05:58,040: The, um,
- 00:05:59,180 --> 00:06:00,960: the j-- there is a journey to AI-ready,
- 00:06:01,340 --> 00:06:02,440: uh, data products,
- 00:06:02,840 --> 00:06:03,940: and this
- 00:06:04,420 --> 00:06:06,460: is realistically not a journey that we can,
- 00:06:06,560 --> 00:06:07,980: uh, complete overnight.
- 00:06:08,520 --> 00:06:10,920: There are varying levels of maturity, and
- 00:06:11,380 --> 00:06:12,380: we are very much
- 00:06:12,980 --> 00:06:15,320: in the early stages of that, uh, journey.
- 00:06:16,020 --> 00:06:18,152: So- S- [sighs] phase one,
- 00:06:18,272 --> 00:06:19,992: stage one, whatever we want to call it, is
- 00:06:20,512 --> 00:06:22,572: just making sure that we've got, uh,
- 00:06:22,992 --> 00:06:25,312: we know what our assets are. They're identified,
- 00:06:25,392 --> 00:06:27,112: they have visibility. That's
- 00:06:27,432 --> 00:06:28,092: where we start.
- 00:06:28,632 --> 00:06:30,972: And that's in a almost pre-publication stage
- 00:06:31,332 --> 00:06:33,232: where, um, it's just about having
- 00:06:33,732 --> 00:06:35,672: knowledge of what we, uh, what we own.
- 00:06:36,512 --> 00:06:38,612: And then, and then, uh, we move
- 00:06:38,632 --> 00:06:41,132: into publication, which is really the bare minimum
- 00:06:41,732 --> 00:06:43,112: for, uh, consumption.
- 00:06:43,472 --> 00:06:45,812: So publication is what takes, uh, our data products
- 00:06:45,872 --> 00:06:46,832: into the marketplace.
- 00:06:47,412 --> 00:06:49,792: It makes sure they meet a minimum benchmark, it enables
- 00:06:49,852 --> 00:06:50,492: analytics,
- 00:06:51,252 --> 00:06:53,572: um, there's clear ownership, there's clear,
- 00:06:53,732 --> 00:06:56,012: um, uh, definition around our data
- 00:06:56,032 --> 00:06:57,792: and defined access mechanisms,
- 00:06:58,252 --> 00:07:00,152: so actually both humans and agents can
- 00:07:00,792 --> 00:07:01,432: access it.
- 00:07:02,412 --> 00:07:04,752: But this is really just a start, and the complexity
- 00:07:05,632 --> 00:07:07,752: increases as we move towards the, uh, the
- 00:07:07,832 --> 00:07:08,812: right-hand side and,
- 00:07:09,652 --> 00:07:10,732: uh, being able to
- 00:07:10,992 --> 00:07:13,332: unama- un- unambiguously interpret
- 00:07:13,512 --> 00:07:14,412: our data products,
- 00:07:14,872 --> 00:07:15,812: that improves, uh,
- 00:07:16,132 --> 00:07:16,992: discoverability.
- 00:07:17,432 --> 00:07:19,492: Moving on to business language, the, uh, the
- 00:07:19,592 --> 00:07:21,692: business context and the semantic context
- 00:07:22,252 --> 00:07:24,552: that provides, um, additional, um,
- 00:07:25,492 --> 00:07:27,072: interpretability to our data.
- 00:07:27,732 --> 00:07:29,072: Moving on to actual, um,
- 00:07:29,332 --> 00:07:30,332: agent action, which is,
- 00:07:31,272 --> 00:07:33,252: uh, a long way from now in reality,
- 00:07:34,172 --> 00:07:36,532: uh, in terms of being able to make agents
- 00:07:36,612 --> 00:07:38,332: are able to simulate decisions,
- 00:07:38,972 --> 00:07:40,132: trigger processes,
- 00:07:40,632 --> 00:07:42,812: uh, and agents orchestrate other agents. And this
- 00:07:42,872 --> 00:07:44,972: is, uh, this spectrum is,
- 00:07:45,232 --> 00:07:47,112: um, uh,
- 00:07:47,372 --> 00:07:48,292: quite complex,
- 00:07:48,912 --> 00:07:51,012: and you can e- reach various stages of maturity
- 00:07:51,072 --> 00:07:52,372: depending on the domain. So
- 00:07:52,732 --> 00:07:54,832: y- you can, uh, for a small subset
- 00:07:54,912 --> 00:07:55,652: of data,
- 00:07:56,252 --> 00:07:57,192: uh, aim to
- 00:07:57,632 --> 00:07:59,712: move towards agent action fairly rapidly,
- 00:08:00,252 --> 00:08:02,432: versus at an enterprise scale is, um,
- 00:08:03,412 --> 00:08:04,912: it is quite a substantial, uh,
- 00:08:05,432 --> 00:08:06,112: undertaking.
- 00:08:06,972 --> 00:08:10,212: And the
- 00:08:10,252 --> 00:08:10,812: second
- 00:08:11,172 --> 00:08:13,392: pillar I mentioned, so it was m- uh, metadata,
- 00:08:13,632 --> 00:08:14,672: um, rich metadata,
- 00:08:14,972 --> 00:08:16,652: AI, um, AI-ready metadata.
- 00:08:17,532 --> 00:08:19,572: And then we've got this business representation layer, and
- 00:08:19,632 --> 00:08:20,552: that's what allows
- 00:08:21,292 --> 00:08:21,892: an agent
- 00:08:22,632 --> 00:08:24,272: to, uh, know the context
- 00:08:24,612 --> 00:08:25,412: of our query.
- 00:08:26,472 --> 00:08:27,371: So, um,
- 00:08:27,892 --> 00:08:29,552: if we pose
- 00:08:29,892 --> 00:08:31,072: a question, and here I've got, uh,
- 00:08:31,452 --> 00:08:33,152: "What is our patient recruitment rate
- 00:08:33,772 --> 00:08:34,672: for, uh, trial
- 00:08:35,012 --> 00:08:36,052: X?" Um,
- 00:08:36,592 --> 00:08:37,492: whatever the trial may be.
- 00:08:37,792 --> 00:08:39,952: And the agent needs to understand
- 00:08:39,992 --> 00:08:42,452: that question. He needs to be able to interpret that against
- 00:08:42,952 --> 00:08:44,212: the business meaning, and
- 00:08:44,652 --> 00:08:45,372: that meaning may
- 00:08:45,772 --> 00:08:48,072: change depending on the way the question is posed.
- 00:08:48,792 --> 00:08:50,332: Locate the right,
- 00:08:50,712 --> 00:08:52,832: uh, data which, or data products which
- 00:08:52,892 --> 00:08:54,532: may contain the
- 00:08:54,932 --> 00:08:56,992: answer to that or contain the data, which is
- 00:08:57,032 --> 00:08:58,792: the fuel to respond to that query.
- 00:08:59,272 --> 00:09:00,372: And this is where, um,
- 00:09:01,372 --> 00:09:03,632: we spoke about a map. We need a map to be able
- 00:09:03,692 --> 00:09:05,692: to point our agents to the right,
- 00:09:06,312 --> 00:09:06,872: uh,
- 00:09:07,472 --> 00:09:09,832: data, the, uh, the right knowledge base, um,
- 00:09:10,332 --> 00:09:11,352: in a, uh, i-
- 00:09:12,192 --> 00:09:13,052: in an efficient manner.
- 00:09:15,332 --> 00:09:16,832: Access, assess and join,
- 00:09:17,352 --> 00:09:17,912: right? So
- 00:09:18,472 --> 00:09:19,032: be able to
- 00:09:19,352 --> 00:09:21,652: use multiple data sources, multiple inputs
- 00:09:21,712 --> 00:09:23,152: to interpret
- 00:09:23,552 --> 00:09:25,232: and federate across those sources,
- 00:09:25,832 --> 00:09:27,832: and then resolve and deliver back to the, uh,
- 00:09:28,192 --> 00:09:29,592: to the human who needs that,
- 00:09:29,972 --> 00:09:31,832: uh, needs that response. And this
- 00:09:32,432 --> 00:09:34,572: delivery system may not be, uh, immediately
- 00:09:34,632 --> 00:09:35,492: within the, uh,
- 00:09:36,012 --> 00:09:38,392: catalog. It may not immediately be within the marketplace,
- 00:09:38,452 --> 00:09:39,372: but actually, um,
- 00:09:41,232 --> 00:09:43,252: uh, downstream in a, in a, in
- 00:09:43,292 --> 00:09:45,492: an adjacent, uh, part of the organization,
- 00:09:45,872 --> 00:09:47,092: in an adjacent system.
- 00:09:49,352 --> 00:09:51,192: There's... That, that
- 00:09:51,592 --> 00:09:53,772: together, the ability to do that is really what we mean
- 00:09:53,812 --> 00:09:55,592: by this business representation layer.
- 00:09:55,992 --> 00:09:56,972: And data products,
- 00:09:57,412 --> 00:09:59,572: whether they're structured, whether they're unstructured, are the
- 00:09:59,672 --> 00:10:01,792: fuel to be able to, uh,
- 00:10:02,212 --> 00:10:02,932: to, uh, allow,
- 00:10:03,672 --> 00:10:04,892: um, agents to,
- 00:10:05,392 --> 00:10:06,552: uh, to navigate this.
- 00:10:11,332 --> 00:10:12,052: We are
- 00:10:12,712 --> 00:10:14,792: early in this journey, I think I might have mentioned.
- 00:10:15,912 --> 00:10:17,532: Um, and there is a distinct
- 00:10:18,352 --> 00:10:19,892: difference between, um,
- 00:10:21,332 --> 00:10:23,432: enabling the technical integration,
- 00:10:23,972 --> 00:10:26,092: so making sure there's schemas, there's plumbing
- 00:10:26,532 --> 00:10:27,732: between our systems
- 00:10:28,352 --> 00:10:30,192: and real maturity.
- 00:10:31,392 --> 00:10:33,492: Um, another
- 00:10:33,552 --> 00:10:35,572: ana- analogy I used earlier is,
- 00:10:35,692 --> 00:10:37,471: uh, books and bookshelves.
- 00:10:37,851 --> 00:10:39,912: So we're now in the process of building the bookshelves.
- 00:10:39,992 --> 00:10:41,132: We're making sure the right,
- 00:10:41,792 --> 00:10:43,312: uh, organizational, um,
- 00:10:44,452 --> 00:10:46,452: uh, systems are in place, making sure
- 00:10:46,512 --> 00:10:48,812: that we know where to find
- 00:10:49,372 --> 00:10:50,012: which book.
- 00:10:51,472 --> 00:10:53,472: But this is where our partners in the business areas, in
- 00:10:53,512 --> 00:10:54,092: the, um,
- 00:10:54,352 --> 00:10:56,472: in R&D and operations, in commercial,
- 00:10:56,532 --> 00:10:57,772: this is where our partners really,
- 00:10:58,492 --> 00:10:59,372: uh, come in and we have a
- 00:10:59,652 --> 00:11:01,852: working partnership across the organization,
- 00:11:02,412 --> 00:11:04,792: is putting books on the bookshelves, and that's the actual
- 00:11:04,912 --> 00:11:06,032: data. Those are the, um,
- 00:11:06,832 --> 00:11:08,192: uh, the ontologies. Those are the
- 00:11:08,492 --> 00:11:09,352: business meaning
- 00:11:09,632 --> 00:11:11,232: relevant to their domains.
- 00:11:12,232 --> 00:11:13,212: Um, and
- 00:11:13,472 --> 00:11:15,412: there's such a spectrum
- 00:11:15,692 --> 00:11:16,292: of what
- 00:11:16,612 --> 00:11:18,612: business knowledge a- actually is, going
- 00:11:18,652 --> 00:11:20,672: from ontologies, master
- 00:11:20,712 --> 00:11:21,692: and reference data,
- 00:11:22,512 --> 00:11:23,452: uh, the, uh,
- 00:11:23,912 --> 00:11:25,912: business context, the data products themselves,
- 00:11:25,952 --> 00:11:27,352: so the actual data itself,
- 00:11:28,252 --> 00:11:30,372: the, uh, the lineage of that, so you know
- 00:11:30,412 --> 00:11:32,452: which source systems feed into, um,
- 00:11:33,432 --> 00:11:34,452: which data products.
- 00:11:35,472 --> 00:11:37,332: Domain data models, so you know
- 00:11:38,112 --> 00:11:40,132: what this column means, what address means in
- 00:11:40,172 --> 00:11:42,532: the context of a healthcare provider versus
- 00:11:43,072 --> 00:11:45,132: what an address means in, uh, in the context
- 00:11:45,172 --> 00:11:46,032: of a patient.
- 00:11:46,912 --> 00:11:49,272: Um, there's
- 00:11:49,312 --> 00:11:51,352: controlled vocabularies, so you know what,
- 00:11:51,692 --> 00:11:53,932: um, the, uh, synonyms
- 00:11:54,012 --> 00:11:54,892: are. You, you know,
- 00:11:55,332 --> 00:11:57,672: uh, you're, you're comparing apples, apples and apples
- 00:11:58,252 --> 00:12:00,492: to overu- to use our overused phrase.
- 00:12:01,112 --> 00:12:03,292: And the business processes, which is really key,
- 00:12:03,372 --> 00:12:05,432: which is, um,
- 00:12:05,732 --> 00:12:07,612: showing the, uh,
- 00:12:09,552 --> 00:12:11,832: sh- which, which give, allows the agent
- 00:12:12,232 --> 00:12:14,432: to understand where in the business this data's
- 00:12:14,532 --> 00:12:16,592: relevant to and make an assessment of what the
- 00:12:16,672 --> 00:12:18,432: right, um,
- 00:12:18,952 --> 00:12:20,672: fuel for the right query is.
- 00:12:21,392 --> 00:12:23,524: Um- That feeds
- 00:12:23,564 --> 00:12:25,744: into business representation layer, which is powered
- 00:12:25,784 --> 00:12:26,764: by the, uh,
- 00:12:27,584 --> 00:12:30,004: uh, the cataloging side, enables discovery
- 00:12:30,064 --> 00:12:30,744: and access.
- 00:12:31,144 --> 00:12:32,104: And again, this is
- 00:12:32,924 --> 00:12:35,124: human and agentic fundamentally.
- 00:12:37,624 --> 00:12:38,824: And so
- 00:12:39,824 --> 00:12:40,764: we, um,
- 00:12:43,004 --> 00:12:44,224: move on to the guardrails
- 00:12:44,484 --> 00:12:45,844: component. So there are three pillars.
- 00:12:46,644 --> 00:12:48,104: We had AI-ready metadata,
- 00:12:48,604 --> 00:12:49,404: we've got business,
- 00:12:49,784 --> 00:12:52,204: uh, representation layers as the context
- 00:12:52,264 --> 00:12:54,044: layer, and then the guardrails.
- 00:12:54,544 --> 00:12:56,684: And this is again where we rely on the, um,
- 00:12:56,864 --> 00:12:58,104: the richness of the, uh,
- 00:12:58,384 --> 00:13:00,384: the meta model, the, uh, information that's allowed
- 00:13:00,424 --> 00:13:01,784: to be captured in Collibra
- 00:13:02,504 --> 00:13:04,164: to say, "Right, this is a data
- 00:13:04,784 --> 00:13:06,884: product, and associated with this
- 00:13:07,004 --> 00:13:07,924: are, uh,
- 00:13:08,444 --> 00:13:10,984: various data contracts, terms and conditions,
- 00:13:11,064 --> 00:13:11,664: licenses,
- 00:13:13,224 --> 00:13:15,344: which define how a
- 00:13:16,104 --> 00:13:17,144: data product or
- 00:13:17,444 --> 00:13:19,524: how a series of data products can be used,
- 00:13:19,564 --> 00:13:20,804: and importantly, how they cannot
- 00:13:21,404 --> 00:13:21,944: be used
- 00:13:22,324 --> 00:13:23,324: and in which context."
- 00:13:23,764 --> 00:13:25,224: And that then, uh,
- 00:13:25,744 --> 00:13:27,764: can be automated via policy-based access
- 00:13:27,824 --> 00:13:29,324: control, which is
- 00:13:30,364 --> 00:13:32,424: for the, uh, the, the same guardrails
- 00:13:32,464 --> 00:13:34,184: for the human or for the agent.
- 00:13:35,284 --> 00:13:37,144: Uh, and a good example is if users in--
- 00:13:37,664 --> 00:13:38,304: if only
- 00:13:38,784 --> 00:13:40,864: users in R&D, if only data consumers
- 00:13:41,384 --> 00:13:42,564: in R&D can access
- 00:13:42,864 --> 00:13:43,984: clinical study data,
- 00:13:44,904 --> 00:13:46,024: an agent acting on
- 00:13:46,504 --> 00:13:48,584: behalf of a human should apply the same,
- 00:13:49,304 --> 00:13:51,464: uh, uh, same control, same
- 00:13:51,524 --> 00:13:53,024: framework, and same guardrails. So,
- 00:13:53,864 --> 00:13:56,384: uh, that should apply and scale, um, across.
- 00:13:57,424 --> 00:13:58,924: And so this is really important
- 00:14:00,464 --> 00:14:02,184: to have these documented
- 00:14:02,604 --> 00:14:03,484: then enforced.
- 00:14:03,924 --> 00:14:04,924: And, um,
- 00:14:06,204 --> 00:14:07,044: it's again
- 00:14:07,864 --> 00:14:09,784: working with the business, making sure that we're
- 00:14:10,444 --> 00:14:11,824: leveraging, uh, our
- 00:14:12,464 --> 00:14:13,684: business counterparts to,
- 00:14:14,464 --> 00:14:15,344: uh, document
- 00:14:15,624 --> 00:14:17,904: such information in Collibra so that
- 00:14:17,984 --> 00:14:18,904: in, in, uh, our
- 00:14:19,324 --> 00:14:20,444: enterprise data catalog,
- 00:14:21,184 --> 00:14:23,264: so that agents can access the
- 00:14:23,324 --> 00:14:24,244: same information.
- 00:14:25,484 --> 00:14:28,384: So the
- 00:14:28,424 --> 00:14:29,424: agentic future.
- 00:14:30,204 --> 00:14:32,044: We want agents to be able to discover,
- 00:14:32,324 --> 00:14:32,944: interpret,
- 00:14:33,484 --> 00:14:34,744: act on data safely,
- 00:14:35,564 --> 00:14:37,364: and they're doing so on a human's behalf.
- 00:14:38,044 --> 00:14:40,044: And there are three fundamental, uh,
- 00:14:40,444 --> 00:14:41,944: pillars that we spoke about.
- 00:14:42,524 --> 00:14:44,264: Um, the discovery portion starts from
- 00:14:45,064 --> 00:14:47,124: being able to understand the business problem,
- 00:14:48,044 --> 00:14:50,484: uh, so not just a data request.
- 00:14:50,784 --> 00:14:51,764: Um, I commonly say,
- 00:14:52,264 --> 00:14:53,264: uh, frequently say, um,
- 00:14:54,504 --> 00:14:56,344: s- a user doesn't wake up and say,
- 00:14:56,744 --> 00:14:57,804: "I need data today."
- 00:14:58,224 --> 00:15:00,624: They wake up and say, "I need to solve a business problem,"
- 00:15:01,284 --> 00:15:03,364: and data is a fuel for that
- 00:15:03,424 --> 00:15:04,304: business problem. And so
- 00:15:06,164 --> 00:15:08,304: the agent needs to be able to interpret
- 00:15:08,364 --> 00:15:09,964: that, uh,
- 00:15:10,444 --> 00:15:11,124: same request
- 00:15:11,544 --> 00:15:12,884: when asked and say, "Right,
- 00:15:13,404 --> 00:15:14,604: to solve this business query,
- 00:15:15,164 --> 00:15:17,184: I've got to go and find or I, I think I need to
- 00:15:17,244 --> 00:15:19,284: find this data." They use a map in order to
- 00:15:19,324 --> 00:15:20,404: do so. They use an atlas
- 00:15:21,044 --> 00:15:21,884: in order to do so,
- 00:15:23,004 --> 00:15:24,104: uh, to navigate
- 00:15:24,604 --> 00:15:26,224: them towards the most suitable, uh,
- 00:15:26,764 --> 00:15:27,684: uh, data products.
- 00:15:28,484 --> 00:15:30,524: They need to be able to interpret those data products to
- 00:15:30,564 --> 00:15:31,164: evaluate,
- 00:15:31,484 --> 00:15:33,924: um, uh, to validate that they
- 00:15:34,404 --> 00:15:36,784: indeed are relevant for the context.
- 00:15:37,264 --> 00:15:39,144: So i- does this correlate to what my, um,
- 00:15:40,124 --> 00:15:41,204: human, um,
- 00:15:42,224 --> 00:15:42,904: is asking?
- 00:15:43,664 --> 00:15:45,464: There's no translation in this, uh,
- 00:15:45,884 --> 00:15:47,964: uh... at this point. It's not the human going back
- 00:15:47,984 --> 00:15:50,144: and forth. Ideally, you want the agent
- 00:15:50,184 --> 00:15:51,384: to be able to understand this.
- 00:15:52,024 --> 00:15:53,924: And then real
- 00:15:54,564 --> 00:15:56,584: value comes from turning that
- 00:15:56,644 --> 00:15:57,764: data into insight
- 00:15:58,104 --> 00:15:59,844: and handing it back to, uh,
- 00:16:00,524 --> 00:16:02,624: the human. And all of this needs to be done within the
- 00:16:02,664 --> 00:16:03,824: guardrails that we, uh,
- 00:16:04,124 --> 00:16:04,664: spoke about. So
- 00:16:06,084 --> 00:16:07,444: agents need to be held to account,
- 00:16:08,024 --> 00:16:09,344: uh, to the equivalent standards,
- 00:16:09,664 --> 00:16:10,944: uh, either way.
- 00:16:12,484 --> 00:16:14,704: So really,
- 00:16:14,784 --> 00:16:16,804: a really complex query is:
- 00:16:17,744 --> 00:16:20,044: Where can we accelerate recruitment in trial
- 00:16:20,964 --> 00:16:22,004: X, right? So
- 00:16:22,764 --> 00:16:23,324: the, uh,
- 00:16:23,624 --> 00:16:25,504: agent needs to go and find the right data
- 00:16:26,164 --> 00:16:28,464: for, uh, relating to the trial,
- 00:16:28,924 --> 00:16:31,024: understand, uh, recruitment curves,
- 00:16:31,524 --> 00:16:33,584: understand the distinction between,
- 00:16:33,644 --> 00:16:34,824: uh, what accelerates a
- 00:16:35,424 --> 00:16:37,844: recruitment curve and the factors that, uh, uh, accelerate
- 00:16:37,904 --> 00:16:39,024: a re- recruitment curve,
- 00:16:39,864 --> 00:16:40,644: making sure
- 00:16:41,383 --> 00:16:42,964: that all of that access,
- 00:16:43,484 --> 00:16:44,024: um,
- 00:16:44,364 --> 00:16:45,764: is legitimate,
- 00:16:46,624 --> 00:16:48,664: and then recommend where to act
- 00:16:48,724 --> 00:16:49,344: in a, um,
- 00:16:49,604 --> 00:16:51,224: business contextual manner.
- 00:16:54,924 --> 00:16:57,124: Right. Thank you,
- 00:16:57,224 --> 00:16:59,464: um, for listening to me. Um,
- 00:16:59,644 --> 00:17:00,704: I hope that makes sense.
- 00:17:00,984 --> 00:17:02,224: Uh, I hope, uh, there's, uh,
- 00:17:02,744 --> 00:17:05,364: some messaging in there that resonates with the previous conversations
- 00:17:05,384 --> 00:17:06,444: and talks that we've had.
Building the foundations for an AI future - Extending governance from data to AI with Collibra
As AI expands across the enterprise, from use cases and models to increasingly autonomous agents, the demands on governance are changing. Organizations need to evolve from establishing strong data foundations to maintaining visibility and oversight across a rapidly growing AI landscape.
In this session, AstraZeneca’s Dr. Mithilan Siva shares how the company is building on its governance foundation to support AI at scale. Discover how AstraZeneca is extending governance from data to AI with Collibra, gaining greater visibility and oversight across AI use cases, models and agents as AI becomes more pervasive and autonomous.Video: Same boat, new waters: data leaders on governing AI
Transcript
- 00:00:05,840 --> 00:00:07,200: Let's start with introductions.
- 00:00:07,880 --> 00:00:09,400: Um, if you will do me the,
- 00:00:09,660 --> 00:00:10,400: uh, favor of
- 00:00:10,760 --> 00:00:12,060: each introducing yourself,
- 00:00:12,740 --> 00:00:13,700: your role, and
- 00:00:14,040 --> 00:00:16,020: briefly your organization as well.
- 00:00:17,220 --> 00:00:19,320: Well, good afternoon, everyone. Uh, my name's Don.
- 00:00:19,600 --> 00:00:21,720: I'm the head of the Data Center
- 00:00:21,760 --> 00:00:23,900: of Excellence at the Student Loans Company.
- 00:00:24,480 --> 00:00:26,620: I'm responsible for our
- 00:00:26,660 --> 00:00:28,580: data governance and management practices,
- 00:00:29,120 --> 00:00:31,480: along with everything that comes up from MI
- 00:00:31,520 --> 00:00:33,480: reporting and insights and data science.
- 00:00:34,220 --> 00:00:35,720: We are a, an arm's length
- 00:00:36,040 --> 00:00:37,720: public body in the civil service,
- 00:00:38,620 --> 00:00:39,760: and our key
- 00:00:40,100 --> 00:00:42,540: goal is to fund,
- 00:00:43,120 --> 00:00:45,440: uh, education in the UK and beyond.
- 00:00:46,160 --> 00:00:47,320: Uh, a- and we are...
- 00:00:47,620 --> 00:00:49,820: We own two of what the civil service call
- 00:00:49,840 --> 00:00:50,900: the top 75
- 00:00:51,360 --> 00:00:52,060: UK citizen,
- 00:00:52,420 --> 00:00:53,040: uh, journeys,
- 00:00:53,380 --> 00:00:55,620: which is apply for finance and then to manage
- 00:00:56,020 --> 00:00:56,920: the balance of that.
- 00:00:57,360 --> 00:00:59,240: We have an extensive set of
- 00:00:59,800 --> 00:01:01,980: disparate customer data sets, uh,
- 00:01:02,120 --> 00:01:04,340: which are probably familiar to a lot of challenges
- 00:01:04,400 --> 00:01:05,260: that you have as well,
- 00:01:05,800 --> 00:01:08,000: and we are responsible ultimately for
- 00:01:08,100 --> 00:01:10,380: maintaining and ensuring the integrity of
- 00:01:10,820 --> 00:01:13,640: the UK public money, which is £350
- 00:01:13,680 --> 00:01:15,840: billion, that's been out on loans,
- 00:01:16,120 --> 00:01:16,660: uh, to date.
- 00:01:17,040 --> 00:01:19,100: Well, I better find a way of putting some of that
- 00:01:19,140 --> 00:01:19,860: in my pocket.
- 00:01:20,180 --> 00:01:20,920: [laughs] Um,
- 00:01:21,720 --> 00:01:23,820: thanks, Don, and thanks for, uh, making the trek from
- 00:01:23,860 --> 00:01:24,500: Glasgow.
- 00:01:25,360 --> 00:01:25,960: Oh. Good af-
- 00:01:27,560 --> 00:01:29,580: Good afternoon, everyone. My name's Ian Cook. I'm the
- 00:01:29,660 --> 00:01:30,400: chief data officer
- 00:01:30,720 --> 00:01:31,560: at Vitality.
- 00:01:32,320 --> 00:01:34,420: We're an organization that runs health and
- 00:01:34,480 --> 00:01:36,160: life products for a lot of our members,
- 00:01:36,520 --> 00:01:38,060: um, considerable number of members.
- 00:01:39,360 --> 00:01:41,480: I'm responsible for one of many areas,
- 00:01:41,580 --> 00:01:43,880: but the data governance space, so our data office,
- 00:01:44,680 --> 00:01:46,760: uh, looking after data governance, not AI
- 00:01:46,820 --> 00:01:47,700: governance to stress,
- 00:01:48,020 --> 00:01:50,020: and that's something that we're- obviously touch
- 00:01:50,080 --> 00:01:50,740: on shortly.
- 00:01:51,440 --> 00:01:52,660: Um, we are
- 00:01:53,260 --> 00:01:55,740: very much the... A data-driven organization.
- 00:01:55,780 --> 00:01:57,220: We use data to drive a lot,
- 00:01:57,620 --> 00:01:59,940: if not all, of our business decision-making, and it has become
- 00:02:00,000 --> 00:02:01,160: more and more important for us,
- 00:02:01,620 --> 00:02:02,300: and with AI
- 00:02:03,240 --> 00:02:05,240: on us, it's, uh, it's... That's
- 00:02:05,300 --> 00:02:07,340: really, really stepped up in the last, last year or
- 00:02:07,380 --> 00:02:09,979: so. Ian,
- 00:02:10,020 --> 00:02:12,340: hi, everyone. I'm Chris Barnes. I'm head of data management
- 00:02:12,400 --> 00:02:14,460: at the Driver and Vehicle Standards Agency. So
- 00:02:14,500 --> 00:02:16,900: we're an executive agency of the Department for Transport,
- 00:02:17,440 --> 00:02:19,940: and we have four of the top 75 services
- 00:02:20,120 --> 00:02:22,420: in the UK. So yes, that's one for us. [laughs]
- 00:02:22,520 --> 00:02:23,080: Uh, so
- 00:02:23,440 --> 00:02:25,700: my team are responsible for data governance
- 00:02:25,760 --> 00:02:28,060: and data architecture primarily, and obviously
- 00:02:28,700 --> 00:02:30,820: AI, uh, is not too far from that world.
- 00:02:30,840 --> 00:02:32,840: There's a, a huge overlap in that Venn diagram.
- 00:02:34,020 --> 00:02:36,240: Well, thanks, all, and, um, let's talk about how
- 00:02:36,320 --> 00:02:37,280: close it is actually.
- 00:02:38,040 --> 00:02:40,140: Um, so a few years back, AI came
- 00:02:40,180 --> 00:02:41,440: into the world, and of course,
- 00:02:41,700 --> 00:02:43,800: uh, Collibra had to make moves there as well, and
- 00:02:43,860 --> 00:02:44,480: as you can see,
- 00:02:44,960 --> 00:02:47,100: we have to keep making moves, uh, in this,
- 00:02:47,200 --> 00:02:48,320: uh, market.
- 00:02:48,660 --> 00:02:50,940: But, um, one of the first things we noticed
- 00:02:51,000 --> 00:02:53,520: is, okay, data governance and AI governance,
- 00:02:54,140 --> 00:02:55,580: what does that mean in the world?
- 00:02:56,380 --> 00:02:58,400: And, you know, when I look at data governance
- 00:02:58,660 --> 00:03:01,000: 10 years back and where it is today, more mature
- 00:03:01,040 --> 00:03:02,100: for AI governance,
- 00:03:02,600 --> 00:03:04,140: roles, responsibilities,
- 00:03:04,460 --> 00:03:06,060: who owns it, how it should be done,
- 00:03:06,500 --> 00:03:09,240: how it should be measured for success, all
- 00:03:09,320 --> 00:03:11,300: up in the air, and I recognize the
- 00:03:11,760 --> 00:03:14,020: immaturity that data governance had 10, 15
- 00:03:14,080 --> 00:03:14,760: years ago. So
- 00:03:15,360 --> 00:03:16,300: maybe, uh,
- 00:03:16,560 --> 00:03:18,640: Ian, if you don't mind, because you already brought it up, I'll
- 00:03:18,700 --> 00:03:19,340: start with you. But
- 00:03:20,040 --> 00:03:21,800: you said data governance, AI governance,
- 00:03:22,060 --> 00:03:24,080: one sits with you, the other doesn't. So
- 00:03:24,160 --> 00:03:25,060: should it? You know,
- 00:03:25,640 --> 00:03:26,340: uh, should it not?
- 00:03:26,840 --> 00:03:28,120: If it should, how are you,
- 00:03:28,440 --> 00:03:29,600: you know, waging that battle?
- 00:03:31,160 --> 00:03:32,620: Battle, interesting term.
- 00:03:33,000 --> 00:03:34,460: Um, we,
- 00:03:35,160 --> 00:03:37,200: we are very lucky in that fact that our
- 00:03:37,280 --> 00:03:38,080: AI enablement
- 00:03:39,560 --> 00:03:41,320: function sits very closely with us.
- 00:03:42,360 --> 00:03:44,400: So where there is a need for AI governance
- 00:03:44,460 --> 00:03:46,540: and where we can support that team in,
- 00:03:46,600 --> 00:03:49,080: in understanding what governance is, they are very commercially
- 00:03:49,120 --> 00:03:51,220: driven to drive usage of AI out
- 00:03:51,300 --> 00:03:52,260: into the organization.
- 00:03:53,380 --> 00:03:54,100: So we've been
- 00:03:55,240 --> 00:03:55,800: light touch,
- 00:03:56,380 --> 00:03:57,040: not gone in heavy,
- 00:03:57,560 --> 00:03:59,700: just sat, made sure we're in presence
- 00:03:59,820 --> 00:04:02,220: in AI forums, in AI groups.
- 00:04:02,500 --> 00:04:04,620: Uh, there's a, a large organization within
- 00:04:04,680 --> 00:04:07,060: the, the business which is called AI Enthusiasts.
- 00:04:07,900 --> 00:04:08,560: So we've been
- 00:04:09,220 --> 00:04:10,820: talking with them an awful lot,
- 00:04:11,420 --> 00:04:12,960: um, and making sure that
- 00:04:13,280 --> 00:04:13,860: they know
- 00:04:14,400 --> 00:04:16,500: the guidelines. Gui- I mean, guardrails is
- 00:04:16,560 --> 00:04:18,680: a term that's used an awful lot now, and I think that's
- 00:04:18,700 --> 00:04:19,899: really, really important. Um,
- 00:04:20,420 --> 00:04:21,579: just putting some gentle
- 00:04:22,320 --> 00:04:24,540: persuaders around people to ensure they stay
- 00:04:25,020 --> 00:04:26,960: in the right space because a lot of these people,
- 00:04:27,320 --> 00:04:29,500: we, we've, we've understood over
- 00:04:29,520 --> 00:04:32,000: the last three or four months, aren't the data consumers
- 00:04:32,040 --> 00:04:34,120: that we're used to. The people that use reports or build
- 00:04:34,140 --> 00:04:36,180: reports, they're very much people that are in the business that
- 00:04:36,240 --> 00:04:37,320: don't... They don't,
- 00:04:37,980 --> 00:04:40,400: they don't anticipate the type of work that goes in to validate
- 00:04:40,420 --> 00:04:41,760: the data upfront.
- 00:04:42,300 --> 00:04:44,400: Um, so we're having to encourage a lot more
- 00:04:44,440 --> 00:04:46,440: in that space. So it's, it's, it's light touch at the
- 00:04:46,500 --> 00:04:48,340: moment. I think much with AI is changing.
- 00:04:48,820 --> 00:04:50,900: AI governance will change, and we'll have to evolve
- 00:04:50,920 --> 00:04:53,060: with that. Much as you said, over the last 10 years, data governance
- 00:04:53,100 --> 00:04:55,160: has changed considerably. Mm. And I think we're in that
- 00:04:55,240 --> 00:04:57,060: same space. So rather than setting down
- 00:04:57,640 --> 00:04:59,660: a really heavy marker now, it's just making
- 00:04:59,720 --> 00:05:01,000: sure we're there, we're a presence,
- 00:05:01,320 --> 00:05:03,280: we're encouraging people to behave the right way,
- 00:05:03,840 --> 00:05:05,040: and then more on top of that
- 00:05:05,480 --> 00:05:07,520: is waiting for the opportunity to see what we
- 00:05:07,560 --> 00:05:08,120: can do next.
- 00:05:09,100 --> 00:05:10,360: Yeah. So you're
- 00:05:11,080 --> 00:05:12,500: not owning it, but part of it-
- 00:05:12,820 --> 00:05:13,400: Yeah... which I like,
- 00:05:14,040 --> 00:05:16,260: and maybe we should name our AI command center
- 00:05:16,300 --> 00:05:18,820: the AI Enthusiast Command Center. [laughs]
- 00:05:18,860 --> 00:05:19,760: But Don, Chris,
- 00:05:20,520 --> 00:05:22,240: any thoughts from you on this topic?
- 00:05:23,260 --> 00:05:25,400: Yeah. I, I, I think similar to Ian. You know, w-
- 00:05:25,620 --> 00:05:27,680: we have people who are already kind
- 00:05:27,740 --> 00:05:29,840: of doing this kind of role. You know, we talk about the link
- 00:05:29,880 --> 00:05:31,800: between data governance and AI governance.
- 00:05:32,200 --> 00:05:34,400: You've already got people who are looking at that. You've already got people
- 00:05:34,440 --> 00:05:35,360: that are looking at risk.
- 00:05:35,780 --> 00:05:38,020: So I think for me it's really important to
- 00:05:38,120 --> 00:05:40,200: use the structures you have. You know, you can either try
- 00:05:40,240 --> 00:05:42,080: and design a technically perfect system,
- 00:05:42,640 --> 00:05:44,760: or you can just use a practical system,
- 00:05:44,800 --> 00:05:45,900: and I'd much rather
- 00:05:46,380 --> 00:05:48,760: we just used a practical system, use the structures
- 00:05:48,820 --> 00:05:50,860: and, and, and people that we already have, and the roles that
- 00:05:50,860 --> 00:05:51,560: are already there.
- 00:05:51,880 --> 00:05:53,960: So I think that's what we're trying to do. We're just evolving,
- 00:05:54,160 --> 00:05:55,820: uh, some of the functions that we already have
- 00:05:56,300 --> 00:05:58,360: and adding kind of little bits and pieces and
- 00:05:58,400 --> 00:06:00,380: looking at how governance needs to change.
- 00:06:00,960 --> 00:06:03,416: Um- And it's really important to make it proportional.
- 00:06:03,436 --> 00:06:05,476: I think we're gonna go on and talk about that, but making sure
- 00:06:05,536 --> 00:06:06,516: that we're not
- 00:06:07,156 --> 00:06:09,196: being kind of waterfall in how we, how we
- 00:06:09,236 --> 00:06:11,316: attack governance, that we're not being seen as a big blocker,
- 00:06:11,356 --> 00:06:11,936: and you have to do,
- 00:06:12,356 --> 00:06:14,396: you know, months and months worth of governance to attack
- 00:06:14,456 --> 00:06:16,556: a two-week pilot. It's, it's being proportional as
- 00:06:16,596 --> 00:06:17,536: we kind of push that through.
- 00:06:17,836 --> 00:06:18,916: Right. That's smart.
- 00:06:19,376 --> 00:06:21,296: Adapt the governance needs to the, uh,
- 00:06:22,196 --> 00:06:24,336: experimental versus full production.
- 00:06:25,136 --> 00:06:27,216: For you, Don? Yeah, uh, I would, I would agree with
- 00:06:27,256 --> 00:06:28,436: that entirely. It's
- 00:06:29,196 --> 00:06:29,936: AI governance
- 00:06:30,616 --> 00:06:32,776: is almost like a layer on top of your data governance.
- 00:06:33,416 --> 00:06:35,216: Um, I think as well, uh,
- 00:06:35,696 --> 00:06:37,836: it's maybe... It's a hot, it's a hot topic.
- 00:06:37,896 --> 00:06:38,956: It's student loans, as
- 00:06:39,436 --> 00:06:41,376: we've started from a risk perspective.
- 00:06:41,656 --> 00:06:42,616: Mm. It's pretty much
- 00:06:43,136 --> 00:06:45,176: almost technology-led to an extent,
- 00:06:45,276 --> 00:06:47,476: but any sort of AI
- 00:06:47,536 --> 00:06:49,616: governance or technology-led governance put in,
- 00:06:49,996 --> 00:06:51,336: inconsequential as it may be,
- 00:06:51,696 --> 00:06:52,256: is just a g-
- 00:06:52,516 --> 00:06:54,696: data governance program in disguise.
- 00:06:54,776 --> 00:06:55,616: Because ultimately,
- 00:06:56,136 --> 00:06:57,796: what, what you really want to do is,
- 00:06:58,236 --> 00:07:00,296: the best way of using your AI is to ensure
- 00:07:00,316 --> 00:07:01,716: that your data is fit for purpose.
- 00:07:02,376 --> 00:07:04,516: Uh, any, any sort of platform
- 00:07:04,556 --> 00:07:05,396: that you look at now,
- 00:07:05,935 --> 00:07:08,056: any sort of technology has AI built into
- 00:07:08,916 --> 00:07:11,456: it, so the implementation becomes secondary
- 00:07:11,516 --> 00:07:12,536: to the use of it,
- 00:07:12,876 --> 00:07:14,956: and, and therefore that- Mm... becomes about how you manage
- 00:07:14,976 --> 00:07:16,496: your data. And if you have that,
- 00:07:16,756 --> 00:07:18,336: that foundational layer in there,
- 00:07:18,876 --> 00:07:20,676: the, uh, it's going from that.
- 00:07:21,576 --> 00:07:22,476: Yeah. So look,
- 00:07:22,876 --> 00:07:23,796: I agree with you, Don,
- 00:07:24,096 --> 00:07:26,476: and I, I'm sure everybody in the room sort of agrees,
- 00:07:26,616 --> 00:07:27,596: too. The
- 00:07:28,696 --> 00:07:30,856: garbage in, garbage out principle.
- 00:07:31,096 --> 00:07:32,256: You know, good AI is
- 00:07:32,696 --> 00:07:34,396: dependent on good data.
- 00:07:34,956 --> 00:07:37,116: You have to invest in your foundations.
- 00:07:38,036 --> 00:07:40,216: Um, but then you talk to a CEO
- 00:07:40,756 --> 00:07:41,436: who's at home
- 00:07:42,216 --> 00:07:44,556: and sees a kid talking to AI all day,
- 00:07:44,976 --> 00:07:47,096: and they go into chatgpt.com,
- 00:07:47,936 --> 00:07:49,956: and then, you know, they go back in the company and they
- 00:07:50,016 --> 00:07:51,916: say, "Where's my AI?" And you say, "Okay. Well,
- 00:07:52,776 --> 00:07:54,976: uh, Mr. or Mrs. CEO, just invest three
- 00:07:55,056 --> 00:07:57,156: years and a lot of millions of pounds into
- 00:07:57,416 --> 00:07:58,356: your data foundation,
- 00:07:58,756 --> 00:08:00,736: and then you'll have your perfect agent."
- 00:08:01,816 --> 00:08:03,816: You do know that the CEO will never talk to you
- 00:08:03,896 --> 00:08:05,296: again after that. [laughs] Right? So
- 00:08:05,796 --> 00:08:08,116: for a lot of business stakeholders, it seems like this
- 00:08:08,176 --> 00:08:09,875: connection between data and AI
- 00:08:10,436 --> 00:08:12,076: is not necessarily made.
- 00:08:12,916 --> 00:08:14,136: Um, so
- 00:08:14,536 --> 00:08:16,696: how, how have you approached this,
- 00:08:17,316 --> 00:08:19,616: you know, script kid or business stakeholder
- 00:08:19,676 --> 00:08:21,436: cooking up an agent over the weekend?
- 00:08:22,116 --> 00:08:23,276: Uh, how have you approached
- 00:08:24,136 --> 00:08:26,476: leading into this and then attaching
- 00:08:26,556 --> 00:08:28,696: investment in data foundation to
- 00:08:28,756 --> 00:08:30,856: make sure that the thing doesn't fall on its face?
- 00:08:31,256 --> 00:08:32,596: Or maybe you're waiting
- 00:08:33,136 --> 00:08:34,576: until it falls on its face to say,
- 00:08:35,196 --> 00:08:37,576: "Told you so"? [laughs]
- 00:08:37,636 --> 00:08:38,816: I think unlike a lot of, um,
- 00:08:39,156 --> 00:08:41,276: public sector bodies, we're, we're fairly risk-averse.
- 00:08:41,636 --> 00:08:44,336: Um, so it's about understanding
- 00:08:44,396 --> 00:08:46,496: what, what value can bring
- 00:08:46,516 --> 00:08:47,236: the short term
- 00:08:47,536 --> 00:08:49,616: whilst giving you enough credit in the bank to
- 00:08:49,656 --> 00:08:51,676: invest in the longer term stuff. Mm. 'Cause,
- 00:08:51,716 --> 00:08:53,956: you know, we're all data govern- Well, most of us here, we're data governance
- 00:08:53,996 --> 00:08:54,516: professionals.
- 00:08:54,796 --> 00:08:56,796: This stuff takes a long time. Cultural change takes a
- 00:08:56,836 --> 00:08:58,956: long time, and I do not want
- 00:08:59,016 --> 00:09:01,136: to be the person who's selling that message to our exec
- 00:09:01,196 --> 00:09:02,276: of, "Yeah, you'll see
- 00:09:02,556 --> 00:09:04,596: value and return on this investment, but only in
- 00:09:04,636 --> 00:09:06,756: two years' time." That's, you know, seriously career
- 00:09:06,796 --> 00:09:07,876: limiting. So
- 00:09:08,716 --> 00:09:10,816: by making our governance
- 00:09:10,856 --> 00:09:13,156: proportional, by making it kind of this minimum
- 00:09:13,216 --> 00:09:15,316: viable governance concept, what's the minimum we
- 00:09:15,356 --> 00:09:17,536: need to know that we're comfortable with to progress
- 00:09:17,576 --> 00:09:18,776: something on to the next step,
- 00:09:19,176 --> 00:09:20,136: that helps us to,
- 00:09:20,636 --> 00:09:22,796: to give some value to use cases
- 00:09:22,836 --> 00:09:24,716: which, you know, sit within our risk appetite,
- 00:09:25,216 --> 00:09:26,036: and we can showly-
- 00:09:26,436 --> 00:09:28,656: slowly show that we are providing some value,
- 00:09:28,716 --> 00:09:30,856: but in a, within a bounded context.
- 00:09:31,356 --> 00:09:33,556: And that gives us, yeah, then that credit to do something
- 00:09:33,596 --> 00:09:35,636: a little bit bigger, a little bit more risky, a little bit
- 00:09:35,696 --> 00:09:37,796: more, you know, with a little more, more value attached to
- 00:09:37,856 --> 00:09:39,856: it. So you're chipping away at it,
- 00:09:39,936 --> 00:09:40,516: essentially.
- 00:09:40,856 --> 00:09:43,096: Yeah. Similar approaches? Different?
- 00:09:43,896 --> 00:09:44,716: Yeah. There's
- 00:09:45,236 --> 00:09:47,156: nothing, nothing gives more cre-
- 00:09:47,436 --> 00:09:48,136: credence to
- 00:09:48,796 --> 00:09:50,856: your data governance program than watching
- 00:09:50,916 --> 00:09:52,036: a proof of concept fail-
- 00:09:52,416 --> 00:09:54,496: Yeah... um, because you don't have the right controls
- 00:09:54,536 --> 00:09:56,136: on it, uh, especially on AI.
- 00:09:56,596 --> 00:09:58,596: Uh, I think the trick is to not to look at
- 00:09:58,616 --> 00:10:00,716: it as if, "I've, I've told you so" about it, which
- 00:10:00,776 --> 00:10:03,176: is a, a trap that needs to be avoided.
- 00:10:03,496 --> 00:10:05,676: I think you, you need to start small with these things
- 00:10:05,716 --> 00:10:06,516: before you scale,
- 00:10:07,116 --> 00:10:08,996: and it's important that you understand
- 00:10:09,676 --> 00:10:11,776: the outcome and the value that you want to do on
- 00:10:11,816 --> 00:10:14,076: it. Mm. Because, uh, everything that looks quite
- 00:10:14,136 --> 00:10:15,996: shiny right now in AI, it, it's,
- 00:10:16,376 --> 00:10:18,115: it doesn't work on larger scales-
- 00:10:18,476 --> 00:10:20,596: Mm... when you take it out of that small box that
- 00:10:20,636 --> 00:10:21,415: you've controlled.
- 00:10:22,136 --> 00:10:24,216: Uh, extending your controls out over your
- 00:10:24,276 --> 00:10:25,516: organization therefore
- 00:10:25,896 --> 00:10:26,576: becomes the key-
- 00:10:27,116 --> 00:10:29,716: Yeah... element.
- 00:10:29,756 --> 00:10:30,636: We use please a lot.
- 00:10:30,956 --> 00:10:31,616: Please validate.
- 00:10:32,376 --> 00:10:34,416: Please validate the data. Please validate the end
- 00:10:34,456 --> 00:10:35,436: result. Um,
- 00:10:35,936 --> 00:10:37,956: again, not to discourage innovation. We
- 00:10:37,996 --> 00:10:40,276: want people to move forward with these, these sort of developments.
- 00:10:41,056 --> 00:10:41,616: But equally,
- 00:10:41,916 --> 00:10:42,836: again, education.
- 00:10:43,316 --> 00:10:44,996: Um- Mm... Calibre's been good in that fact-
- 00:10:45,376 --> 00:10:47,596: Mm... um, in terms of providing that library
- 00:10:47,656 --> 00:10:49,736: that people can refer to. So if you're in any doubt, please
- 00:10:49,816 --> 00:10:51,056: ask. Um,
- 00:10:51,496 --> 00:10:53,176: so... And the emphasis is always on please.
- 00:10:53,676 --> 00:10:55,416: If you tend to go in a little bit more directive,
- 00:10:55,776 --> 00:10:57,296: it tends to put people's backs up, and
- 00:10:57,676 --> 00:10:59,076: they don't tend to come back to you. So
- 00:10:59,416 --> 00:11:01,476: we'd rather the people engage, and that's what we've encouraged.
- 00:11:01,656 --> 00:11:02,496: Mm-hmm. Um,
- 00:11:02,916 --> 00:11:05,136: and just, it's education. It's gonna take
- 00:11:05,196 --> 00:11:07,196: time as, as with, with most items.
- 00:11:07,796 --> 00:11:09,796: Um, just gonna take time to bed in and get
- 00:11:09,896 --> 00:11:11,936: our business organization really thinking about
- 00:11:11,976 --> 00:11:14,116: what they're doing just above the, the, obviously
- 00:11:14,156 --> 00:11:14,676: the quickness,
- 00:11:15,076 --> 00:11:16,676: the quick speed, the speed of delivery that
- 00:11:17,016 --> 00:11:17,816: we're seeing with AI.
- 00:11:18,776 --> 00:11:20,456: Yeah. I, I like that it's kind of
- 00:11:20,856 --> 00:11:22,896: constructive and also polite, but you're
- 00:11:22,936 --> 00:11:24,956: essentially saying, "Do whatever you want," in a way,
- 00:11:25,036 --> 00:11:27,096: but then something comes
- 00:11:27,156 --> 00:11:29,236: back, and you say, "Okay, please validate,"
- 00:11:29,296 --> 00:11:30,976: because you can see the holes through it.
- 00:11:31,476 --> 00:11:33,636: And that sort of forces them to go into it. Is that
- 00:11:34,356 --> 00:11:36,536: how... You know, it sounds very cultural.
- 00:11:36,896 --> 00:11:38,816: Is that how your company culture works?
- 00:11:39,236 --> 00:11:40,716: Yeah. They, it's a collaborative
- 00:11:41,436 --> 00:11:42,936: organization at, at heart.
- 00:11:43,196 --> 00:11:44,936: Culturally, they, they're very strong with that.
- 00:11:45,256 --> 00:11:46,636: Um, governance
- 00:11:47,876 --> 00:11:50,076: is, is a little bit like, um, that police device
- 00:11:50,116 --> 00:11:50,756: called the stinger.
- 00:11:51,256 --> 00:11:53,536: It's the thing that goes across the roads to blow the tires
- 00:11:53,576 --> 00:11:54,796: on the cars on the runaway. Um,
- 00:11:55,356 --> 00:11:57,456: so we do tend to turn it round, and we...
- 00:11:57,676 --> 00:11:59,756: And the team do a very good job of making sure it's
- 00:11:59,796 --> 00:12:01,276: an en- an engaging process.
- 00:12:01,776 --> 00:12:03,816: Um, but there's times when you have to be a
- 00:12:03,876 --> 00:12:04,536: little bit harder
- 00:12:05,392 --> 00:12:06,252: Um, and
- 00:12:06,752 --> 00:12:07,292: ensure that
- 00:12:07,952 --> 00:12:10,512: they understand what they've done. But I think sometimes
- 00:12:10,952 --> 00:12:13,192: people learn from their own mistakes. That's where they gain experience,
- 00:12:13,252 --> 00:12:14,232: and that's what you don't wanna stifle.
- 00:12:15,152 --> 00:12:16,772: So it's these, but there's a menacing-
- 00:12:17,132 --> 00:12:18,492: Yeah, yeah. Yeah, a little bit. Yeah...
- 00:12:18,812 --> 00:12:20,952: undertone, yeah. Oh, yeah. Oh, certainly an undertone.
- 00:12:20,952 --> 00:12:23,052: [laughs] Understood.
- 00:12:23,632 --> 00:12:25,912: Uh, it's also a perfect lead-in to our next question.
- 00:12:25,952 --> 00:12:27,832: We completely planned it as, of course. Oh.
- 00:12:28,252 --> 00:12:30,332: Um, so, uh, the next
- 00:12:30,372 --> 00:12:32,572: question is related to the data teams. I'm sure
- 00:12:32,832 --> 00:12:34,072: f- you're familiar with some of them.
- 00:12:34,732 --> 00:12:36,812: And maybe you're also familiar with how
- 00:12:36,952 --> 00:12:39,132: it's easy to be perceived as a data
- 00:12:39,172 --> 00:12:39,772: team, which is
- 00:12:40,332 --> 00:12:42,572: that policing example that we just talked
- 00:12:42,652 --> 00:12:44,232: about, you know, the no,
- 00:12:44,592 --> 00:12:47,092: right? The no team or the policing
- 00:12:47,152 --> 00:12:49,552: team, uh, whereas data functions
- 00:12:50,112 --> 00:12:51,872: are supposed to be enablers, right?
- 00:12:52,552 --> 00:12:53,072: So how do you,
- 00:12:53,372 --> 00:12:55,532: how do you approach this in your organizations when
- 00:12:55,572 --> 00:12:57,072: they're throwing around AI, and
- 00:12:57,692 --> 00:12:58,972: it needs guardrails,
- 00:13:00,692 --> 00:13:02,712: but you don't wanna be seen as the police either?
- 00:13:02,792 --> 00:13:04,532: So any, you know,
- 00:13:04,892 --> 00:13:05,732: nuance to the,
- 00:13:06,012 --> 00:13:07,492: the, the, the previous discussion maybe?
- 00:13:08,732 --> 00:13:09,491: If you, if you
- 00:13:10,152 --> 00:13:11,752: turn that slight, let's say, change the,
- 00:13:12,212 --> 00:13:14,412: the context on it and talk about
- 00:13:14,472 --> 00:13:15,812: it's the behavior of using it-
- 00:13:16,092 --> 00:13:17,172: Mm... the, the best
- 00:13:17,652 --> 00:13:19,832: governance, the safest governance is the quickest
- 00:13:19,892 --> 00:13:21,452: governance, and it's about
- 00:13:22,192 --> 00:13:24,312: the literacy and the understanding and the use of
- 00:13:24,352 --> 00:13:26,492: your AI. If you can embed that into
- 00:13:26,552 --> 00:13:27,552: your organization,
- 00:13:28,192 --> 00:13:30,212: uh, uh, and the organization can
- 00:13:30,272 --> 00:13:32,292: follow that safe governance pattern- Sure... you
- 00:13:32,312 --> 00:13:33,052: can get through it.
- 00:13:33,492 --> 00:13:35,412: It, it shouldn't always be a question of a,
- 00:13:36,072 --> 00:13:37,692: a binary yes or no that way,
- 00:13:38,132 --> 00:13:39,172: though sometimes
- 00:13:39,692 --> 00:13:42,052: no is the answer. I think that needs to be recognized
- 00:13:42,092 --> 00:13:43,712: at times. But it's,
- 00:13:44,192 --> 00:13:46,212: it's a cultural piece, but it's, it's the
- 00:13:46,212 --> 00:13:47,432: encouragement to follow
- 00:13:47,952 --> 00:13:49,972: the guardrails. It's using the tools
- 00:13:49,992 --> 00:13:51,892: and mechanism you have, your metadata,
- 00:13:52,292 --> 00:13:53,112: your catalog,
- 00:13:53,452 --> 00:13:55,712: and it's, it's how you put that forward. And it's...
- 00:13:56,092 --> 00:13:58,272: It should be a, a, a joint journey,
- 00:13:58,332 --> 00:14:00,332: not just a data journey. It needs to be
- 00:14:00,432 --> 00:14:02,172: shared across the organization. Mm-hmm.
- 00:14:02,852 --> 00:14:04,852: I think the thing is that nobody has bad intentions.
- 00:14:04,892 --> 00:14:06,932: Like, nobody's going out to do something intentionally
- 00:14:06,972 --> 00:14:07,532: reckless,
- 00:14:08,052 --> 00:14:09,152: or, or at least you hope not.
- 00:14:09,652 --> 00:14:11,772: So I think w- with me and my team, it's
- 00:14:11,832 --> 00:14:14,091: more like I, I shouldn't expect the business
- 00:14:14,112 --> 00:14:16,392: to be able to ask themselves some of the questions that we're asking.
- 00:14:16,472 --> 00:14:18,452: So we're just providing that little bit of
- 00:14:19,172 --> 00:14:19,992: check and challenge.
- 00:14:20,292 --> 00:14:22,372: We're asking some of the questions, especially with our kind of data
- 00:14:22,392 --> 00:14:24,472: and AI ethics assessment that we've put in place,
- 00:14:25,472 --> 00:14:27,852: to ask them things that they don't know. And hopefully,
- 00:14:27,912 --> 00:14:28,672: that'll change
- 00:14:29,292 --> 00:14:31,332: maybe the focus of the, o- of the project,
- 00:14:31,392 --> 00:14:33,812: or maybe it'll make them put in some extra mitigations
- 00:14:33,852 --> 00:14:34,992: or some extra controls,
- 00:14:35,432 --> 00:14:37,552: and that's great. I think the worst thing is
- 00:14:37,612 --> 00:14:39,612: people just go, "Oh, no, it'll be fine," and kind of plowing
- 00:14:39,652 --> 00:14:41,812: off ahead and not being involved in the conversations,
- 00:14:41,952 --> 00:14:43,732: not being in the room. So I'd rather have
- 00:14:44,392 --> 00:14:45,672: a bit of a quick conversation.
- 00:14:46,012 --> 00:14:48,092: Maybe it's a little bit awkward, but at least then we,
- 00:14:48,152 --> 00:14:50,012: we're all comfortable and we all know that we've,
- 00:14:50,292 --> 00:14:52,152: that, that, that we've gone through the process.
- 00:14:53,632 --> 00:14:54,752: Yes, because, um,
- 00:14:55,512 --> 00:14:57,652: you know, ethics was also brought up this morning
- 00:14:57,692 --> 00:15:00,052: when I was speaking to the analysts, so that's important.
- 00:15:00,312 --> 00:15:01,172: Mm. Uh,
- 00:15:01,512 --> 00:15:03,632: and it's good to ask those questions, I think,
- 00:15:03,692 --> 00:15:05,812: because a lot of the risk you see with AI is
- 00:15:05,932 --> 00:15:07,492: very much an unseen risk.
- 00:15:08,172 --> 00:15:10,492: You know, when you, when you throw something into these models
- 00:15:10,552 --> 00:15:11,892: these days, what comes back
- 00:15:12,472 --> 00:15:13,672: is very sophisticated,
- 00:15:13,952 --> 00:15:15,792: right? So even if it's in a domain that
- 00:15:16,232 --> 00:15:16,872: you're familiar
- 00:15:17,212 --> 00:15:19,292: with, uh, it's still hard to
- 00:15:19,352 --> 00:15:20,172: poke the holes
- 00:15:20,692 --> 00:15:21,372: in that thing.
- 00:15:21,872 --> 00:15:24,032: So especially if it's something you don't know, a lot of that
- 00:15:24,072 --> 00:15:26,452: risk remains unseen. And that's fine in a chatbot
- 00:15:26,932 --> 00:15:28,812: 'cause it's Q&A. You're still there.
- 00:15:29,472 --> 00:15:30,692: Um, but it becomes,
- 00:15:31,092 --> 00:15:32,232: uh, a lot bigger,
- 00:15:32,632 --> 00:15:33,672: uh, of an unseen risk
- 00:15:34,152 --> 00:15:36,192: when the agents come into play because
- 00:15:36,212 --> 00:15:37,792: there's no more human in between.
- 00:15:38,172 --> 00:15:39,112: It's just thousands of
- 00:15:39,772 --> 00:15:42,012: s- decisions and actions per second, essentially.
- 00:15:42,312 --> 00:15:44,412: Mm. Uh, let's move it on to our, uh, next
- 00:15:44,472 --> 00:15:45,192: favorite topic.
- 00:15:45,712 --> 00:15:48,252: Um, you saw Tom speak about this earlier.
- 00:15:48,832 --> 00:15:50,852: Uh, data in the
- 00:15:50,932 --> 00:15:52,972: worlds of... in the words of the AI
- 00:15:53,872 --> 00:15:54,632: kids, if you will,
- 00:15:55,112 --> 00:15:57,292: uh, they don't call it data. They call it context,
- 00:15:57,592 --> 00:15:59,612: right? So if you wanna be hot with the AI folks,
- 00:15:59,972 --> 00:16:01,232: just talk about context.
- 00:16:01,952 --> 00:16:04,352: Um, every trade show I go to, recently
- 00:16:04,392 --> 00:16:06,392: there was Big Data London, for example, but it's
- 00:16:06,452 --> 00:16:08,872: everywhere. Everybody has context
- 00:16:08,952 --> 00:16:11,052: on their banners right now, and it's a bit noisy,
- 00:16:11,172 --> 00:16:12,192: like everything in AI.
- 00:16:12,912 --> 00:16:14,932: Uh, but let's unpack context a little
- 00:16:14,992 --> 00:16:15,792: bit. You know, the,
- 00:16:16,432 --> 00:16:18,832: obviously the, the agent needs information.
- 00:16:19,432 --> 00:16:22,092: Uh, the agent needs meaning, maybe needs ontologies,
- 00:16:22,152 --> 00:16:23,532: and maybe needs other things.
- 00:16:24,812 --> 00:16:26,071: What is context for you?
- 00:16:26,691 --> 00:16:29,032: And it can be philosophical, if you please.
- 00:16:29,672 --> 00:16:30,872: Um, and how do you,
- 00:16:31,312 --> 00:16:33,252: uh, apply it? How do you control it?
- 00:16:33,652 --> 00:16:35,872: Or maybe not yet, but do you have thoughts,
- 00:16:35,992 --> 00:16:36,592: ideas?
- 00:16:38,332 --> 00:16:39,432: I, I think, um,
- 00:16:40,692 --> 00:16:42,112: come back to that term of education.
- 00:16:42,432 --> 00:16:44,172: We start with the people.
- 00:16:44,872 --> 00:16:46,872: Mm. If people are educated and they understand,
- 00:16:47,052 --> 00:16:49,292: then you can move forward, and then you can trust
- 00:16:49,332 --> 00:16:50,352: what they're delivering. So
- 00:16:51,012 --> 00:16:53,172: we've had a... I say we. There's some people in the audience
- 00:16:53,192 --> 00:16:55,232: here that have actually done the work, and I haven't. But, um,
- 00:16:55,992 --> 00:16:57,452: we've done a, a, a data literacy
- 00:16:57,712 --> 00:16:59,832: program, like most organizations, and we're trying
- 00:16:59,852 --> 00:17:00,912: to encourage more of a,
- 00:17:01,452 --> 00:17:03,712: uh, an educational piece that was originally designed
- 00:17:03,772 --> 00:17:04,992: around self-serve BI.
- 00:17:05,612 --> 00:17:07,632: Mm. How naive we were that that would stay around
- 00:17:07,672 --> 00:17:08,812: for a while. Um,
- 00:17:09,792 --> 00:17:11,572: AI's come along. Uh, the audience has got bigger,
- 00:17:11,892 --> 00:17:12,772: but it doesn't change
- 00:17:13,152 --> 00:17:15,312: the plan. The plan is still to educate people
- 00:17:15,352 --> 00:17:17,472: in, in what data is, what it does, how it works.
- 00:17:18,072 --> 00:17:20,292: And then if we can get that clear
- 00:17:20,332 --> 00:17:21,092: with individuals,
- 00:17:22,032 --> 00:17:22,852: then actually
- 00:17:23,432 --> 00:17:25,712: the agents will always have a tendency to maybe
- 00:17:25,772 --> 00:17:27,932: not be quite correct. But at least
- 00:17:28,052 --> 00:17:29,992: we're making every effort to get them on board.
- 00:17:30,312 --> 00:17:31,712: Yeah. And
- 00:17:32,292 --> 00:17:34,432: I don't know the amount of people in the organization.
- 00:17:34,492 --> 00:17:36,732: Are you enable everyone or specific
- 00:17:36,832 --> 00:17:38,592: units? Um,
- 00:17:39,612 --> 00:17:41,112: the initial goal was,
- 00:17:41,472 --> 00:17:41,992: I think,
- 00:17:42,772 --> 00:17:44,992: 80. That was the first, very first
- 00:17:45,372 --> 00:17:46,372: rollout of, of
- 00:17:46,652 --> 00:17:47,452: Copilot stroke.
- 00:17:47,732 --> 00:17:49,352: People or 80,000?
- 00:17:49,652 --> 00:17:51,892: Oh, no, no. 80 people. We started very, very
- 00:17:51,932 --> 00:17:54,252: low. 80 people. And then, and then we've slowly
- 00:17:54,292 --> 00:17:56,532: rolled it out to more and more people. So we've taken a very gradual
- 00:17:56,612 --> 00:17:57,252: approach to it.
- 00:17:57,792 --> 00:17:59,832: Um, except the group of people who we've
- 00:17:59,852 --> 00:18:01,952: rolled it out to have got very excited about it, and it's, it's,
- 00:18:02,012 --> 00:18:03,052: it's grown from there. So
- 00:18:03,692 --> 00:18:05,812: we are fortunate in
- 00:18:05,852 --> 00:18:08,072: the fact that it's not the 3,500 people
- 00:18:08,112 --> 00:18:10,252: that it could be. Mm-hmm. Um, but it's
- 00:18:10,312 --> 00:18:12,352: only gonna be a matter of time before that starts
- 00:18:12,392 --> 00:18:14,508: to get- Deeper and deeper into the organization,
- 00:18:14,548 --> 00:18:16,788: at which point then we've got, we, we will have control
- 00:18:17,348 --> 00:18:18,108: concerns. So
- 00:18:18,768 --> 00:18:20,968: now's the time to, to roll out education
- 00:18:21,028 --> 00:18:23,148: and to get as many people into that as possible. No, I agree
- 00:18:23,188 --> 00:18:25,248: with you because, uh, we were speaking about it earlier
- 00:18:25,288 --> 00:18:26,008: with the analysts, and
- 00:18:26,328 --> 00:18:28,188: Mike Ferguson brought up the same thing.
- 00:18:28,768 --> 00:18:31,208: The volume of artifacts that are being generated
- 00:18:31,268 --> 00:18:33,648: is gonna increase. You had old enterprise BI,
- 00:18:34,108 --> 00:18:35,868: you know, one report every six months.
- 00:18:36,308 --> 00:18:37,588: Self-service BI, so
- 00:18:38,008 --> 00:18:40,068: thousands of reports being produced all over the
- 00:18:40,108 --> 00:18:42,408: company. And now with AI it's
- 00:18:42,428 --> 00:18:44,768: sc- who knows how many it's gonna be, but definitely
- 00:18:44,848 --> 00:18:45,388: more so.
- 00:18:45,828 --> 00:18:48,568: Inevitably you need people to raise their understanding.
- 00:18:49,288 --> 00:18:51,228: Similar? Different?
- 00:18:51,908 --> 00:18:53,128: Yeah. I mean, your,
- 00:18:53,488 --> 00:18:54,968: your AI is really great at
- 00:18:55,428 --> 00:18:55,968: patterns.
- 00:18:56,508 --> 00:18:58,748: Yeah. It's maybe not so good at the answers sometimes.
- 00:18:59,168 --> 00:19:01,368: And I think the, the best analogy that I can give
- 00:19:01,408 --> 00:19:03,368: is, and I don't know how familiar people are with it,
- 00:19:03,788 --> 00:19:05,908: The Hitchhiker's Guide to the Galaxy, but they go
- 00:19:05,948 --> 00:19:07,968: and build deep thought to answer the
- 00:19:08,008 --> 00:19:08,568: question,
- 00:19:09,408 --> 00:19:11,628: the, the answer to life, the universe, and everything. It says
- 00:19:11,828 --> 00:19:13,508: 42, and the first thing they go is,
- 00:19:13,768 --> 00:19:14,468: "Well, what's that?"
- 00:19:14,848 --> 00:19:15,708: You know, [laughs] just... And
- 00:19:16,328 --> 00:19:17,387: i- you need to go and build a more
- 00:19:17,848 --> 00:19:20,088: bigger AI in order to go and give you the, the answer
- 00:19:20,128 --> 00:19:21,008: to the question that
- 00:19:21,388 --> 00:19:22,308: gives you the answer. So
- 00:19:23,128 --> 00:19:25,568: it's... You need AI-ready data.
- 00:19:25,608 --> 00:19:27,728: And since we're throwing some buzzwords around now, I thought
- 00:19:27,768 --> 00:19:28,868: I'd just drop that one in.
- 00:19:29,328 --> 00:19:31,448: And it's... You shouldn't point your AI
- 00:19:31,488 --> 00:19:33,508: at everything because of what
- 00:19:33,548 --> 00:19:34,628: we've, we've seen today,
- 00:19:35,148 --> 00:19:37,148: duplications, too many sources, too
- 00:19:37,188 --> 00:19:38,588: many conflicts, revisions.
- 00:19:39,348 --> 00:19:41,428: If, if you prepare for it properly,
- 00:19:41,848 --> 00:19:43,968: you understand the context in terms
- 00:19:44,108 --> 00:19:46,148: of the outcome, the value you want to
- 00:19:46,188 --> 00:19:47,908: get, that- that's,
- 00:19:48,388 --> 00:19:50,468: that's where you should go with it. Certainly that, that's
- 00:19:50,488 --> 00:19:52,668: what we're planning on doing. It does start to feel a little
- 00:19:52,708 --> 00:19:54,928: bit like we are in that movie. [laughs]
- 00:19:55,048 --> 00:19:56,108: Uh, because, you know,
- 00:19:56,528 --> 00:19:57,808: I believe in the movie and the book,
- 00:19:58,388 --> 00:20:00,688: uh, they made the whole planet a computer.
- 00:20:01,148 --> 00:20:03,348: Yeah. [laughs]
- 00:20:03,428 --> 00:20:05,448: And these LLMs seem to be doing [laughs]
- 00:20:05,488 --> 00:20:06,048: the same.
- 00:20:06,588 --> 00:20:08,608: Uh, but in any case, uh, thanks. Um,
- 00:20:09,188 --> 00:20:11,128: context, last thoughts on context.
- 00:20:11,688 --> 00:20:12,288: Yeah, so I,
- 00:20:12,588 --> 00:20:14,748: I think a mix of the two points. So we've taken a,
- 00:20:14,808 --> 00:20:16,908: a, a persona-driven view of, of
- 00:20:16,968 --> 00:20:17,688: DVSA, so
- 00:20:18,068 --> 00:20:19,328: segmenting up our,
- 00:20:19,748 --> 00:20:22,028: our colleagues into different personas and really
- 00:20:22,088 --> 00:20:24,188: targeting what those persona groups need to
- 00:20:24,228 --> 00:20:26,268: know. So general awareness for most of our
- 00:20:26,308 --> 00:20:27,688: frontline staff, right up to,
- 00:20:28,168 --> 00:20:30,388: you know, f- uh, data-driven decision
- 00:20:30,448 --> 00:20:32,468: makers, and, and up to the chief exec. You know,
- 00:20:32,548 --> 00:20:34,508: they need to know something slightly different in terms of
- 00:20:35,008 --> 00:20:36,828: questioning they're being provided with.
- 00:20:37,388 --> 00:20:39,448: Um, and for the business context, it's really
- 00:20:39,488 --> 00:20:41,528: working with the business. So we have,
- 00:20:41,628 --> 00:20:43,288: uh, you know, my team is relatively small.
- 00:20:43,868 --> 00:20:44,428: Each of the
- 00:20:44,688 --> 00:20:46,788: service areas knows their area a, a
- 00:20:46,828 --> 00:20:49,008: lot better than we do. So I'm really working with the data
- 00:20:49,048 --> 00:20:51,228: stewards and custodians to generate
- 00:20:51,288 --> 00:20:52,828: that business context metadata,
- 00:20:53,128 --> 00:20:55,428: um, improve the descriptions that we've got in the catalog,
- 00:20:55,488 --> 00:20:57,168: data dictionaries, et cetera, et cetera. So
- 00:20:57,628 --> 00:20:59,888: really important to get them on side and understand why
- 00:20:59,948 --> 00:21:02,108: we're asking them to do this, and why these artifacts
- 00:21:02,128 --> 00:21:02,748: are important,
- 00:21:03,268 --> 00:21:05,328: not just to sit on a shelf for the sake of having
- 00:21:05,348 --> 00:21:06,408: them, but to,
- 00:21:06,808 --> 00:21:08,808: you know, be the bedrock of, uh, what's gonna
- 00:21:08,848 --> 00:21:10,528: go into the, uh, the agents and the,
- 00:21:10,828 --> 00:21:11,668: and the AI as well.
- 00:21:12,448 --> 00:21:14,028: It's quite fascinating, right, how
- 00:21:14,448 --> 00:21:16,328: context, some of the things you described,
- 00:21:16,668 --> 00:21:17,668: the, the steward, if
- 00:21:18,028 --> 00:21:20,068: we narrow it down to this role for a second.
- 00:21:20,908 --> 00:21:23,368: Uh, it was always harder to validate yourself
- 00:21:23,408 --> 00:21:24,188: in a way because,
- 00:21:24,868 --> 00:21:26,908: you know, they were still people. They still had tacit
- 00:21:26,948 --> 00:21:27,968: knowledge in their heads.
- 00:21:28,808 --> 00:21:30,428: But now if they're an agent,
- 00:21:30,768 --> 00:21:32,828: that needs to be very explicit, right? It needs
- 00:21:32,888 --> 00:21:33,988: to be written down in these
- 00:21:34,728 --> 00:21:35,648: MD files.
- 00:21:36,368 --> 00:21:38,468: You know, for the people who don't know MD files, I, I
- 00:21:38,488 --> 00:21:40,648: had to look that up myself, too. It's just a text file,
- 00:21:40,708 --> 00:21:42,748: by the way. Yes. Uh, and people
- 00:21:42,788 --> 00:21:44,948: put rules in there, as if the agent will
- 00:21:45,028 --> 00:21:46,328: abide by those rules.
- 00:21:47,448 --> 00:21:48,328: Um, that's,
- 00:21:49,008 --> 00:21:50,528: in an engineering perspective,
- 00:21:50,828 --> 00:21:52,848: systems engineering, that's gambling, by
- 00:21:52,868 --> 00:21:54,028: the way. Um,
- 00:21:54,448 --> 00:21:55,948: things don't work that way. But anyways,
- 00:21:56,628 --> 00:21:57,348: um, we're gonna clo-
- 00:21:57,628 --> 00:21:59,888: close up with, uh, last questions. Maybe,
- 00:22:00,388 --> 00:22:02,568: uh, you know, you've run
- 00:22:02,628 --> 00:22:04,568: your programs, your units for a while.
- 00:22:06,428 --> 00:22:08,028: You go back in time one year,
- 00:22:08,388 --> 00:22:10,708: what would you recommend to yourself
- 00:22:10,808 --> 00:22:12,068: starting at that point in time,
- 00:22:12,808 --> 00:22:14,888: maybe as advice for people who are in
- 00:22:14,948 --> 00:22:16,028: that place right now?
- 00:22:17,848 --> 00:22:19,428: Uh, for, for myself, it's
- 00:22:19,888 --> 00:22:22,668: ensuring that your executive sponsorship
- 00:22:22,768 --> 00:22:23,468: is committed.
- 00:22:24,008 --> 00:22:24,648: Uh- Mm-hmm...
- 00:22:24,928 --> 00:22:27,228: people drop the word data into every conversation.
- 00:22:27,568 --> 00:22:29,588: They don't always understand what it means
- 00:22:29,647 --> 00:22:31,748: in order to curate it and use it properly.
- 00:22:32,267 --> 00:22:33,488: So I think that's important.
- 00:22:33,947 --> 00:22:36,048: Um, you obviously need some funding
- 00:22:36,088 --> 00:22:36,928: behind you as well,
- 00:22:37,408 --> 00:22:37,988: and I think you need,
- 00:22:38,308 --> 00:22:40,608: need to be able to line up the relevant teams.
- 00:22:41,168 --> 00:22:43,508: It's, it's a shared effort in order to
- 00:22:43,588 --> 00:22:45,788: implement a governance program, implement
- 00:22:45,848 --> 00:22:47,328: your technology, and build it out.
- 00:22:47,728 --> 00:22:49,848: So it's, it's really understanding the, the journey
- 00:22:49,888 --> 00:22:50,668: you want to go on,
- 00:22:51,148 --> 00:22:53,348: rather than running full speed at it
- 00:22:53,408 --> 00:22:54,288: and making mistakes
- 00:22:54,588 --> 00:22:55,448: as you go along. You...
- 00:22:55,868 --> 00:22:57,448: That's, that's always gonna happen,
- 00:22:57,828 --> 00:22:59,808: but you can avoid some of it by just
- 00:23:00,228 --> 00:23:01,308: taking the time upfront.
- 00:23:01,908 --> 00:23:03,248: Exec sponsorship and money.
- 00:23:03,568 --> 00:23:05,888: Smart. Wider
- 00:23:05,928 --> 00:23:06,528: engagement.
- 00:23:07,348 --> 00:23:09,408: We were too short in going to
- 00:23:09,468 --> 00:23:10,368: just the, the normal,
- 00:23:10,688 --> 00:23:11,388: the normal audience-
- 00:23:12,028 --> 00:23:13,968: Mm-hmm... and not appreciating that the audience is bigger
- 00:23:14,228 --> 00:23:15,588: with this. Um,
- 00:23:16,208 --> 00:23:18,008: and I personally have learnt that,
- 00:23:18,308 --> 00:23:20,348: that lesson, certainly in the last three months.
- 00:23:20,508 --> 00:23:21,108: So, um,
- 00:23:21,628 --> 00:23:23,328: yeah, I would, I would've said to myself,
- 00:23:23,888 --> 00:23:26,148: "Go out wider. Go out two or three layers beyond
- 00:23:26,188 --> 00:23:27,028: where you normally talk to,
- 00:23:27,428 --> 00:23:28,668: and start to engage with them early."
- 00:23:29,188 --> 00:23:30,468: Okay. So people.
- 00:23:30,788 --> 00:23:32,668: Yeah. Winning hearts and minds. Very much.
- 00:23:33,428 --> 00:23:35,688: Yeah. And keep investing in the basics. You know, the,
- 00:23:35,728 --> 00:23:37,568: the foundations are fundamental for a reason. So
- 00:23:38,048 --> 00:23:39,948: keep investing in your data, keep improving it,
- 00:23:40,448 --> 00:23:42,208: and bring the people along for the ride. Mm-hmm.
- 00:23:44,288 --> 00:23:46,408: Okay. I do believe that we are
- 00:23:46,488 --> 00:23:47,588: already way over time,
- 00:23:47,948 --> 00:23:49,948: so I'm sure people need all sorts of things to
- 00:23:49,988 --> 00:23:51,268: do, like bio breaking.
- 00:23:52,008 --> 00:23:54,008: Um, what I wanna do is I wanna say
- 00:23:54,108 --> 00:23:55,168: very... thank you very much
- 00:23:55,608 --> 00:23:57,148: for, you know, having the courage
- 00:23:57,728 --> 00:23:59,948: to share your experiences on stage,
- 00:24:00,028 --> 00:24:01,568: and this is of course an invitation.
- 00:24:01,988 --> 00:24:04,428: Anyone who wants to get on the stage or on the podcast,
- 00:24:04,508 --> 00:24:06,168: we're always looking for more stories,
- 00:24:06,628 --> 00:24:08,348: for knowledge sharing, so we can all,
- 00:24:08,968 --> 00:24:11,048: um, how do you call it in English? Raise
- 00:24:11,108 --> 00:24:13,208: the, the, the tide so,
- 00:24:13,728 --> 00:24:14,708: uh, it's good for everyone.
- 00:24:15,388 --> 00:24:17,408: Um, and, uh, as mentioned,
- 00:24:18,108 --> 00:24:20,228: let the stories from today be an inspiration
- 00:24:20,328 --> 00:24:21,488: for your conversations
- 00:24:22,068 --> 00:24:23,048: later today.
- 00:24:23,688 --> 00:24:24,088: Thank you.
Same boat, new waters: data leaders on governing AI
AI is reshaping budgets, priorities and who owns what. Data teams are caught between enabling innovation and being seen as the brakes. In this candid panel, data leaders from Vitality, DVSA and SLC organization join Collibra co-founder Stijn “Stan” Christiaens to share what’s working, what isn’t and what they’re still figuring out as they bring AI and data governance together.
Video: How Weir is scaling AI Governance for the agentic era
Transcript
- 00:00:07,420 --> 00:00:08,540: Hi, good afternoon, everyone.
- 00:00:09,060 --> 00:00:11,060: I'm Robin. I'm the data governance product
- 00:00:11,100 --> 00:00:11,920: owner at Weir Group.
- 00:00:12,420 --> 00:00:14,480: Today, I wanna talk you through how we are moving
- 00:00:14,660 --> 00:00:16,900: AI governance beyond the principles and policies
- 00:00:16,940 --> 00:00:18,380: and documentations, et cetera,
- 00:00:18,900 --> 00:00:21,220: and into how we've sort of created a practical
- 00:00:21,280 --> 00:00:23,500: operating capability. It's not a story
- 00:00:23,780 --> 00:00:25,800: about, like, how we have every answer. There's
- 00:00:25,880 --> 00:00:28,000: actually some of these answers are being discussed today that we need
- 00:00:28,040 --> 00:00:29,720: to consider pr- putting into place.
- 00:00:30,400 --> 00:00:31,080: But it's a story about
- 00:00:31,440 --> 00:00:32,200: how we've moved it
- 00:00:32,580 --> 00:00:33,620: from AI, which was
- 00:00:33,880 --> 00:00:35,340: rapidly experimenting with,
- 00:00:35,700 --> 00:00:38,140: to suddenly we've got lots of production and operational
- 00:00:38,200 --> 00:00:38,760: use cases.
- 00:00:39,440 --> 00:00:40,400: We'll work out what we've,
- 00:00:40,760 --> 00:00:42,040: um... why we needed to change
- 00:00:42,380 --> 00:00:44,620: and where-- and any lessons we've learned along
- 00:00:44,660 --> 00:00:46,500: the way. So
- 00:00:47,220 --> 00:00:48,920: to help us with that, I need to sort of think,
- 00:00:49,340 --> 00:00:51,480: who, who are we-- who's Weir? Weir is a global
- 00:00:51,560 --> 00:00:52,780: mining technology company.
- 00:00:53,480 --> 00:00:54,260: Um, our purpose is,
- 00:00:54,600 --> 00:00:56,720: like, to create a sustainable
- 00:00:56,800 --> 00:00:59,000: future for mining, the mining industry,
- 00:00:59,200 --> 00:01:00,340: as we support the
- 00:01:00,760 --> 00:01:02,320: transition to renewable energy. So
- 00:01:02,820 --> 00:01:03,560: quite a lot of, um,
- 00:01:03,820 --> 00:01:05,019: our mining customers are,
- 00:01:05,319 --> 00:01:06,520: uh, mining, um,
- 00:01:07,220 --> 00:01:09,480: uh, some of the minerals and metals that are required
- 00:01:09,540 --> 00:01:11,700: to, uh, move some of that, uh,
- 00:01:11,780 --> 00:01:12,560: renewable energy,
- 00:01:13,020 --> 00:01:13,540: uh, forward.
- 00:01:14,180 --> 00:01:16,340: And we wanna become more sustainable through
- 00:01:16,580 --> 00:01:17,660: digital technology,
- 00:01:17,940 --> 00:01:18,700: and increasingly,
- 00:01:18,960 --> 00:01:19,660: we use AI.
- 00:01:21,660 --> 00:01:22,780: Uh, where we are--
- 00:01:23,100 --> 00:01:24,660: Weir began a hundred and fifty years ago
- 00:01:25,060 --> 00:01:25,620: in Glasgow,
- 00:01:26,500 --> 00:01:28,600: um, before AI was possibly even a
- 00:01:28,640 --> 00:01:30,660: thing. So we, uh, we moved from sort
- 00:01:30,680 --> 00:01:32,220: of traditional heavy engineering,
- 00:01:32,900 --> 00:01:34,900: um, and we've now sort of f-pivoted
- 00:01:35,000 --> 00:01:37,080: in the last ten, ten to fifteen years into
- 00:01:37,220 --> 00:01:38,540: a mining technology business.
- 00:01:39,240 --> 00:01:41,400: Um, we've, we've done some of that through,
- 00:01:41,660 --> 00:01:43,720: um, organic growth, but also we've made
- 00:01:43,760 --> 00:01:45,980: some s-strategic purchases of,
- 00:01:46,140 --> 00:01:48,500: um, uh, specific AI,
- 00:01:48,860 --> 00:01:51,360: um, use cases, companies like Micromine,
- 00:01:51,960 --> 00:01:53,520: uh, MotionMetrix, and Sentient.
- 00:01:53,880 --> 00:01:54,700: We, we now--
- 00:01:55,140 --> 00:01:56,640: we operate over sixty countries,
- 00:01:57,240 --> 00:01:58,620: eleven thousand employees,
- 00:01:59,240 --> 00:02:01,400: um, and we're trying to sort of create this software
- 00:02:01,460 --> 00:02:03,460: solutions enterprise within
- 00:02:03,500 --> 00:02:04,800: our engineering capability.
- 00:02:08,120 --> 00:02:08,900: Um, that
- 00:02:09,500 --> 00:02:11,840: really context sort of matters because where we actually
- 00:02:11,880 --> 00:02:13,900: work, uh, and many of the environments that
- 00:02:13,940 --> 00:02:14,880: we, uh, operate in
- 00:02:15,440 --> 00:02:16,480: are complex,
- 00:02:16,940 --> 00:02:19,240: you know, multi-company, multi-capability,
- 00:02:19,380 --> 00:02:21,120: and very much safety-focused.
- 00:02:21,680 --> 00:02:22,480: So if we're
- 00:02:23,040 --> 00:02:25,040: using AI in our engineering decisions,
- 00:02:25,480 --> 00:02:26,440: we need to make sure that
- 00:02:26,900 --> 00:02:27,660: those actually,
- 00:02:28,180 --> 00:02:28,740: um, are
- 00:02:29,080 --> 00:02:31,380: safe. They, they, they could potentially
- 00:02:31,400 --> 00:02:32,380: have consequences,
- 00:02:32,860 --> 00:02:34,680: um, not just confined to,
- 00:02:35,060 --> 00:02:36,500: um, the systems that we use,
- 00:02:36,820 --> 00:02:38,920: but actually people. You know, people are using these
- 00:02:38,960 --> 00:02:40,200: equipment in, in large,
- 00:02:40,500 --> 00:02:41,780: uh, sites. Uh,
- 00:02:42,120 --> 00:02:42,900: so we-- for us,
- 00:02:43,200 --> 00:02:43,940: the tr-trust
- 00:02:44,340 --> 00:02:45,280: and the traceability,
- 00:02:45,860 --> 00:02:47,880: um, and the-- and also defensibility of
- 00:02:47,900 --> 00:02:49,500: how we've built and sold our products,
- 00:02:50,020 --> 00:02:51,020: um, are, uh,
- 00:02:51,800 --> 00:02:53,900: really sort of part of our governance concepts.
- 00:02:54,520 --> 00:02:56,780: And so we need to be able to provide that assurance
- 00:02:57,260 --> 00:02:59,740: without actually preventing innovation,
- 00:02:59,820 --> 00:03:00,720: which is kind of sort of
- 00:03:01,300 --> 00:03:03,540: the challenge of today is to n-- to not to apply
- 00:03:03,600 --> 00:03:04,160: that brake.
- 00:03:06,140 --> 00:03:07,720: Um, so
- 00:03:07,980 --> 00:03:10,020: Weir-- AI at Weir is already
- 00:03:10,140 --> 00:03:10,980: quite a complex,
- 00:03:11,340 --> 00:03:12,260: um, and, and broad,
- 00:03:12,580 --> 00:03:14,100: uh, use cases. You know, we have,
- 00:03:14,460 --> 00:03:16,680: um, particularly, you know, one of our most, um, sort
- 00:03:16,700 --> 00:03:19,100: of common engagement with it is various Copilots.
- 00:03:19,700 --> 00:03:20,820: We've got about four and a half,
- 00:03:21,080 --> 00:03:21,840: uh, thousand
- 00:03:22,460 --> 00:03:24,860: monthly license subscriptions to Copilot,
- 00:03:25,260 --> 00:03:27,720: and we can see that ninety-five percent of that's actively
- 00:03:27,800 --> 00:03:28,480: used. So that's
- 00:03:28,920 --> 00:03:31,580: already people using Copilot and AI-assisted
- 00:03:31,780 --> 00:03:33,080: in their productivity at work.
- 00:03:34,020 --> 00:03:36,260: Um, we have product-embedded I-AI with,
- 00:03:36,640 --> 00:03:38,180: um, predictive maintenance,
- 00:03:38,820 --> 00:03:40,900: um, and, uh, you know,
- 00:03:41,160 --> 00:03:42,220: uh, optimized,
- 00:03:42,520 --> 00:03:44,980: uh, operations of our equipment in on-site
- 00:03:45,640 --> 00:03:46,560: is there. That's a,
- 00:03:46,880 --> 00:03:48,340: uh, customer sell already.
- 00:03:48,740 --> 00:03:50,600: We then also to try and future
- 00:03:51,140 --> 00:03:53,380: and maximize and support customers, we create digital
- 00:03:53,460 --> 00:03:54,860: twins of these platforms,
- 00:03:55,260 --> 00:03:56,920: um, and we track, and we fed--
- 00:03:57,300 --> 00:03:58,420: we track that telemetry,
- 00:03:59,100 --> 00:04:00,740: uh, assess it, and send back best things.
- 00:04:01,200 --> 00:04:03,260: We've also started looking at sales productivity,
- 00:04:03,800 --> 00:04:04,620: sales copilot,
- 00:04:05,120 --> 00:04:06,120: and then obviously in
- 00:04:06,420 --> 00:04:07,440: research and development,
- 00:04:07,860 --> 00:04:10,000: AI is speeding up that ability to
- 00:04:10,500 --> 00:04:12,620: come up with the future products and improvements
- 00:04:12,660 --> 00:04:15,460: to our, uh, rev-- of-offering.
- 00:04:16,399 --> 00:04:18,519: Um, what happened when we
- 00:04:18,700 --> 00:04:19,680: realized that, uh,
- 00:04:20,959 --> 00:04:21,660: uh, AI was
- 00:04:22,140 --> 00:04:22,880: coming-- used,
- 00:04:23,620 --> 00:04:25,360: I don't know, maybe post-COVID, it suddenly
- 00:04:25,900 --> 00:04:27,360: became this rapid growth.
- 00:04:27,900 --> 00:04:28,900: We suddenly realized that
- 00:04:29,280 --> 00:04:30,080: we had bought
- 00:04:30,400 --> 00:04:31,200: AI companies,
- 00:04:31,560 --> 00:04:32,680: which was, you know,
- 00:04:33,200 --> 00:04:35,360: wasn't just the confinement of AI use cases.
- 00:04:35,860 --> 00:04:37,000: We had quite a lot of mixed,
- 00:04:37,440 --> 00:04:38,480: um, capabilities
- 00:04:38,840 --> 00:04:40,020: from, um,
- 00:04:40,280 --> 00:04:40,920: analytics,
- 00:04:41,460 --> 00:04:42,780: uh, predictive analytics,
- 00:04:43,280 --> 00:04:44,760: machine learning that we'd built up.
- 00:04:45,080 --> 00:04:46,460: We built our own LLMs,
- 00:04:47,120 --> 00:04:48,300: and obviously, GenAI
- 00:04:48,680 --> 00:04:50,740: was coming, or co-- was coming on really
- 00:04:50,800 --> 00:04:51,860: quickly. And
- 00:04:52,260 --> 00:04:53,900: the real sort of challenge for us was
- 00:04:54,300 --> 00:04:56,540: the experimentation was over. We were now
- 00:04:56,600 --> 00:04:58,660: selling these products. They're live. They were actually
- 00:04:58,740 --> 00:04:59,280: active in com--
- 00:04:59,760 --> 00:05:00,460: in operations.
- 00:05:01,140 --> 00:05:01,880: Um, the key
- 00:05:02,200 --> 00:05:03,160: take from that was
- 00:05:03,700 --> 00:05:05,820: that our maturity oversight. There was no one
- 00:05:05,940 --> 00:05:07,960: place that we could follow. There was no one place that
- 00:05:08,000 --> 00:05:08,540: we could see
- 00:05:09,080 --> 00:05:11,440: and keep a handle on, uh, what was active
- 00:05:11,540 --> 00:05:13,720: and what was-- and how our risks were being managed.
- 00:05:15,880 --> 00:05:16,640: Um, so
- 00:05:17,040 --> 00:05:19,060: what we thought was, "What's the problem? What's the problem?" We,
- 00:05:19,200 --> 00:05:21,320: we didn't have a single view, as I say, of use cases
- 00:05:21,340 --> 00:05:23,320: and models. Different teams
- 00:05:23,800 --> 00:05:25,940: were doing risk assessments. It wasn't that
- 00:05:26,140 --> 00:05:27,560: they wasn't happening, but it just wasn't
- 00:05:28,040 --> 00:05:30,100: difficu- it wasn't easy to compare the conclusions.
- 00:05:30,700 --> 00:05:32,160: And our con-- uh, governance
- 00:05:32,540 --> 00:05:34,400: quite often, uh, from, um,
- 00:05:34,840 --> 00:05:35,480: you know, from a,
- 00:05:36,200 --> 00:05:38,420: uh, data governance team perspective
- 00:05:38,620 --> 00:05:40,120: was written in presentations
- 00:05:40,600 --> 00:05:42,800: and in policy documents that we rapidly
- 00:05:42,840 --> 00:05:43,520: sort of developed.
- 00:05:44,320 --> 00:05:46,400: Um, we-- But obviously we need to then sort
- 00:05:46,440 --> 00:05:47,580: of de-- how do we dem--
- 00:05:47,900 --> 00:05:49,800: how d-did we show
- 00:05:50,520 --> 00:05:52,580: what decision w-- had been taken? So we needed
- 00:05:52,640 --> 00:05:53,780: that, um, traceability,
- 00:05:54,280 --> 00:05:56,280: and evidence was not always easy to follow.
- 00:05:56,980 --> 00:05:57,820: Um, and
- 00:05:58,240 --> 00:05:59,280: it wasn't that we didn't have
- 00:05:59,660 --> 00:06:01,540: principles, because we had defined principles,
- 00:06:02,120 --> 00:06:03,060: but it was just making those
- 00:06:03,440 --> 00:06:05,200: operational and available to people.
- 00:06:06,284 --> 00:06:07,224: So our objective was
- 00:06:07,744 --> 00:06:08,424: let's turn that,
- 00:06:08,844 --> 00:06:10,864: uh, into ex-- into a repeatable
- 00:06:10,964 --> 00:06:13,004: business capability process that we were actually
- 00:06:13,044 --> 00:06:14,884: doing. And we defined that
- 00:06:15,384 --> 00:06:15,924: good governance,
- 00:06:16,504 --> 00:06:17,724: uh, would have a central
- 00:06:18,044 --> 00:06:20,064: principled AI. This is where our AI,
- 00:06:20,784 --> 00:06:23,224: uh, transparency is available to the operation,
- 00:06:23,524 --> 00:06:24,624: to, to the enterprise.
- 00:06:25,324 --> 00:06:27,384: We then wanted standardized risk management. That's,
- 00:06:27,884 --> 00:06:30,424: um, we wanted to basically create an assessment platform
- 00:06:30,484 --> 00:06:32,404: that we'd be able to sort of compare and contrast,
- 00:06:32,804 --> 00:06:33,324: understand,
- 00:06:33,664 --> 00:06:34,564: um, what the,
- 00:06:34,984 --> 00:06:35,844: um, where the
- 00:06:36,944 --> 00:06:38,264: most attention should be,
- 00:06:38,883 --> 00:06:40,564: uh, when looking at risk management.
- 00:06:41,124 --> 00:06:41,684: Um, we wanted to
- 00:06:42,444 --> 00:06:44,144: obviously look at integrating the workflows,
- 00:06:44,524 --> 00:06:45,804: uh, making sure that processes
- 00:06:46,244 --> 00:06:48,364: were s- uh, simplified,
- 00:06:48,724 --> 00:06:50,204: um, and, and tr- and trackable.
- 00:06:50,964 --> 00:06:53,044: And then obviously audit-ready. Like, how
- 00:06:53,124 --> 00:06:53,824: could we actually
- 00:06:54,124 --> 00:06:55,644: give that evidence of how
- 00:06:55,944 --> 00:06:57,844: our AI solutions were working in practice?
- 00:06:58,764 --> 00:07:00,784: Um, and that's what we tried
- 00:07:00,824 --> 00:07:01,784: to say is, look, again,
- 00:07:02,044 --> 00:07:04,084: much to the similar message here is that we can't just do it
- 00:07:04,144 --> 00:07:05,004: once. We have to do it
- 00:07:05,364 --> 00:07:06,184: hundreds of times.
- 00:07:07,924 --> 00:07:09,344: So we made sort of
- 00:07:09,764 --> 00:07:11,864: principles. What did we think? What did it look like?
- 00:07:12,444 --> 00:07:12,964: First, we made,
- 00:07:13,424 --> 00:07:15,544: um... We said that this
- 00:07:15,564 --> 00:07:17,604: has to be an operation. We have to do this every day.
- 00:07:17,684 --> 00:07:19,744: It has to be from the start of
- 00:07:19,804 --> 00:07:22,024: the exercise. We can't go back
- 00:07:22,044 --> 00:07:24,424: and say, "Please tell us your governance score," after
- 00:07:24,464 --> 00:07:25,024: they've built it.
- 00:07:25,404 --> 00:07:26,404: So we built it into,
- 00:07:27,024 --> 00:07:28,164: uh, AI development.
- 00:07:28,924 --> 00:07:29,764: We, uh...
- 00:07:30,144 --> 00:07:31,224: And that's that life cycle
- 00:07:31,664 --> 00:07:33,964: of basically saying, "Look, this is in development.
- 00:07:34,084 --> 00:07:36,304: Did we trust it? Did we deploy it?" Et cetera,
- 00:07:36,364 --> 00:07:37,664: et cetera. Um,
- 00:07:37,964 --> 00:07:39,504: again, risk-based training.
- 00:07:40,024 --> 00:07:41,364: Use case, uh, we,
- 00:07:41,844 --> 00:07:43,044: we ultimately decided that
- 00:07:43,504 --> 00:07:45,564: Collibra was gonna be our, our system of record.
- 00:07:46,104 --> 00:07:48,184: We were already doing data governance on
- 00:07:48,244 --> 00:07:48,884: it, and,
- 00:07:49,424 --> 00:07:50,504: um, Collibra was
- 00:07:50,884 --> 00:07:52,404: obviously the, the best platform
- 00:07:52,784 --> 00:07:54,904: to be able to say that our AI use
- 00:07:54,944 --> 00:07:55,544: case journey,
- 00:07:56,044 --> 00:07:57,944: it was-- could come from trusted data.
- 00:07:58,564 --> 00:07:59,564: So we wanted to be able to
- 00:07:59,824 --> 00:08:00,624: maintain that,
- 00:08:00,964 --> 00:08:01,784: um, that link.
- 00:08:03,884 --> 00:08:05,884: Um, we had some principles. We
- 00:08:05,924 --> 00:08:08,064: were saying these are the-- how we want to assess these
- 00:08:08,124 --> 00:08:09,524: use cases. We can see here that,
- 00:08:10,104 --> 00:08:12,104: um, we've got responsible AI was
- 00:08:12,144 --> 00:08:13,704: certainly key part of our, um,
- 00:08:14,304 --> 00:08:15,663: uh, uh,
- 00:08:16,244 --> 00:08:17,103: you know, our, our
- 00:08:17,364 --> 00:08:19,144: way we wanted to be able to innovate 'cause
- 00:08:19,444 --> 00:08:20,804: safety and sustainability
- 00:08:21,204 --> 00:08:21,824: and all of these
- 00:08:22,344 --> 00:08:23,444: responsible, um,
- 00:08:23,724 --> 00:08:26,004: principles were key
- 00:08:26,064 --> 00:08:28,064: in our, not only just in AI, but everything we
- 00:08:28,124 --> 00:08:29,064: do at Weir. So
- 00:08:29,364 --> 00:08:30,844: they were-- that was quite a simple
- 00:08:31,344 --> 00:08:31,864: construct.
- 00:08:32,284 --> 00:08:34,344: We then had to look at value versus effort, which
- 00:08:34,424 --> 00:08:35,144: is basically
- 00:08:35,584 --> 00:08:37,924: business outcomes versus spend, compute,
- 00:08:38,224 --> 00:08:40,384: all of that exercise, and then the bus- the various
- 00:08:40,504 --> 00:08:42,164: risks. And that gave us
- 00:08:42,544 --> 00:08:43,424: a rounded assessment,
- 00:08:43,984 --> 00:08:46,144: you know, combining what opportunities did we have, what
- 00:08:46,184 --> 00:08:48,524: responsibilities did we have, and what risks
- 00:08:48,564 --> 00:08:50,504: do we have. Um,
- 00:08:51,304 --> 00:08:53,144: and then we brought that to life.
- 00:08:53,964 --> 00:08:55,564: Again, central AI inventory
- 00:08:55,944 --> 00:08:58,104: model. It, it all is in one
- 00:08:58,204 --> 00:08:59,424: place. Um,
- 00:08:59,824 --> 00:09:00,724: th-this, um,
- 00:09:01,684 --> 00:09:02,324: was in Collibra.
- 00:09:02,584 --> 00:09:03,124: This was a,
- 00:09:03,464 --> 00:09:05,584: you know, a, a well-received platform
- 00:09:05,964 --> 00:09:08,204: in our organization. It made a quite significant
- 00:09:08,264 --> 00:09:09,944: uptake in the amount of people using Collibra. So
- 00:09:10,424 --> 00:09:12,524: again, that's a bit of a sell there for the tool, but,
- 00:09:13,144 --> 00:09:15,244: um, it made-- it helped us be able to then,
- 00:09:15,744 --> 00:09:17,924: uh, create that standard intake
- 00:09:17,964 --> 00:09:18,544: process.
- 00:09:19,024 --> 00:09:20,684: Everyone followed the same assessment.
- 00:09:21,264 --> 00:09:23,224: Um, we then created risk-based,
- 00:09:23,724 --> 00:09:24,244: uh, routing
- 00:09:24,664 --> 00:09:26,544: so that if you had a low risk,
- 00:09:26,844 --> 00:09:29,424: you could go to faster path. That would just re-remove
- 00:09:29,504 --> 00:09:30,684: us from the gov-governance
- 00:09:31,064 --> 00:09:33,124: as a break. But where we really wanted
- 00:09:33,164 --> 00:09:35,224: to focus our resources on is the assessment
- 00:09:35,264 --> 00:09:37,364: of high risk. What was the genuine opportunity
- 00:09:37,404 --> 00:09:38,844: we would get? How was it gonna be,
- 00:09:39,604 --> 00:09:40,704: uh, um,
- 00:09:41,124 --> 00:09:43,244: a benefit to the company, and what were the challenges if
- 00:09:43,284 --> 00:09:44,264: we maintained it?
- 00:09:45,444 --> 00:09:47,304: Uh, obviously, challenge there was, uh,
- 00:09:47,864 --> 00:09:49,984: as well as who owns it and who's accountable.
- 00:09:50,224 --> 00:09:52,304: Because obviously, as we know, the AI solutions are
- 00:09:52,684 --> 00:09:53,664: not just systems.
- 00:09:54,084 --> 00:09:56,224: They, they are now tools and therefore accountable,
- 00:09:56,904 --> 00:09:58,304: uh, to, uh, to the business.
- 00:09:58,904 --> 00:10:00,044: And, um,
- 00:10:00,984 --> 00:10:03,104: we, we basically said that now we could say
- 00:10:03,164 --> 00:10:05,284: and record all of the decisions that we made
- 00:10:05,384 --> 00:10:06,484: about, uh, any,
- 00:10:06,884 --> 00:10:09,004: any, um, use case that we put in practice.
- 00:10:10,224 --> 00:10:12,624: We, we brought this to life with a dashboard.
- 00:10:12,684 --> 00:10:14,704: We wanted to have a common front door. We wanted to
- 00:10:14,744 --> 00:10:15,584: basically say, "Look,
- 00:10:16,224 --> 00:10:18,064: don't challenge people with, um,
- 00:10:19,304 --> 00:10:21,424: uh, you know, having to understand governance
- 00:10:21,664 --> 00:10:22,824: in principle." We said, "Look,
- 00:10:23,344 --> 00:10:24,804: here's one place that we can come."
- 00:10:25,304 --> 00:10:26,203: Intended to introduce,
- 00:10:26,484 --> 00:10:27,284: uh... They, they said that
- 00:10:28,124 --> 00:10:30,164: the clear front door also helps us establish that
- 00:10:30,204 --> 00:10:32,384: visibility early. So everyone's just encouraged.
- 00:10:32,424 --> 00:10:34,124: There's no, there's no proc- uh,
- 00:10:34,404 --> 00:10:36,424: distinct different process depending on where you
- 00:10:36,504 --> 00:10:37,584: are. You all come onto that.
- 00:10:38,364 --> 00:10:39,284: Uh, and,
- 00:10:39,544 --> 00:10:40,784: and it was, was something that we
- 00:10:41,064 --> 00:10:43,064: tried to encourage people to do at the start. It
- 00:10:43,104 --> 00:10:45,204: was not an exercise that we wanted people to come and
- 00:10:45,244 --> 00:10:45,964: do afterwards.
- 00:10:47,984 --> 00:10:49,924: Um, and we have a use case registry.
- 00:10:50,684 --> 00:10:52,464: Important point that this, it's not just a list
- 00:10:52,724 --> 00:10:54,284: anymore. It's actually connected
- 00:10:54,864 --> 00:10:57,444: in terms of all that description, the classification,
- 00:10:57,484 --> 00:10:58,084: the ownership,
- 00:10:58,664 --> 00:11:00,864: all of those, uh, processes, all of the data
- 00:11:00,944 --> 00:11:03,324: models, and those models themselves are connected
- 00:11:03,364 --> 00:11:05,144: to the data that we've used internally.
- 00:11:06,104 --> 00:11:08,204: So that became like an active management
- 00:11:08,304 --> 00:11:08,924: capability.
- 00:11:10,164 --> 00:11:10,724: With that then,
- 00:11:11,104 --> 00:11:12,864: once we've got that compiled that information,
- 00:11:13,364 --> 00:11:15,444: dashboards became easy. We could then actually
- 00:11:15,524 --> 00:11:16,124: start to sort of
- 00:11:16,524 --> 00:11:19,164: really get a sense of the metric a-across the organization,
- 00:11:19,584 --> 00:11:21,304: but in one view that allows us to say,
- 00:11:22,064 --> 00:11:24,264: um, you know, what is the challenge to the business
- 00:11:24,584 --> 00:11:25,824: and how exposed we are.
- 00:11:26,744 --> 00:11:27,844: So this gave us, um,
- 00:11:28,124 --> 00:11:28,884: a good, uh,
- 00:11:29,524 --> 00:11:30,604: a good... Didn't replace
- 00:11:31,004 --> 00:11:33,104: accountability. Doesn't say that that's gone away. This
- 00:11:33,144 --> 00:11:35,484: now shows the activity, the risk management
- 00:11:35,524 --> 00:11:36,744: that is actually in place.
- 00:11:37,744 --> 00:11:39,964: Um, and that's, uh, and also we can start
- 00:11:40,004 --> 00:11:41,804: to then focus on risk areas if they
- 00:11:42,084 --> 00:11:42,764: start to pop up.
- 00:11:44,324 --> 00:11:45,564: So what, what changed for us?
- 00:11:46,024 --> 00:11:48,224: What, uh... AI moved from,
- 00:11:48,444 --> 00:11:50,544: like, pockets of information
- 00:11:50,704 --> 00:11:51,484: throughout the business,
- 00:11:51,844 --> 00:11:52,864: and we had one single view.
- 00:11:53,424 --> 00:11:54,324: Decisions became
- 00:11:54,724 --> 00:11:56,844: more consistent because teams are working through the
- 00:11:56,904 --> 00:11:58,164: same list- risk lines
- 00:11:58,464 --> 00:12:00,684: and control frameworks, so they were able to understand,
- 00:12:01,344 --> 00:12:02,364: uh, where they stacked up.
- 00:12:03,024 --> 00:12:05,104: Um, they were also became more traceable so
- 00:12:05,144 --> 00:12:06,384: that in any, any use case,
- 00:12:06,884 --> 00:12:08,904: we, uh, and any model that was being used in
- 00:12:08,924 --> 00:12:11,644: the business, some-- everyone could at least ac-assess
- 00:12:11,704 --> 00:12:12,264: who owns it,
- 00:12:12,624 --> 00:12:14,664: and then any, any change or rationale could be
- 00:12:14,704 --> 00:12:15,784: recorded. Um-
- 00:12:16,114 --> 00:12:18,054: Um, also any decisions
- 00:12:18,374 --> 00:12:20,454: we could then say you-- that's where if we
- 00:12:20,494 --> 00:12:21,494: moved on to high risk,
- 00:12:22,014 --> 00:12:23,614: uh, category, we could then assess
- 00:12:23,934 --> 00:12:25,014: that against, um,
- 00:12:26,214 --> 00:12:28,294: uh, a-a-against any,
- 00:12:28,514 --> 00:12:30,574: uh, decisions and any r-legal reviews
- 00:12:30,774 --> 00:12:33,094: or compliance reviews, et cetera, that were recorded.
- 00:12:34,314 --> 00:12:36,374: Um, so ultimately that just reduced friction
- 00:12:36,414 --> 00:12:38,574: in the business. People were doing it at outset.
- 00:12:38,634 --> 00:12:39,854: We were encouraging people to,
- 00:12:40,314 --> 00:12:40,974: uh, record
- 00:12:41,294 --> 00:12:42,514: their use case
- 00:12:42,974 --> 00:12:44,954: when at the beginning and not afterwards.
- 00:12:45,454 --> 00:12:47,834: So... And also because we had created that risk
- 00:12:48,194 --> 00:12:50,254: tiering, quite a lot of our use
- 00:12:50,294 --> 00:12:52,334: cases were low risk and therefore the
- 00:12:52,394 --> 00:12:54,574: path was, was with a reduced friction.
- 00:12:56,854 --> 00:12:57,754: Um, some lessons
- 00:12:58,114 --> 00:12:58,634: we had was
- 00:12:59,254 --> 00:13:01,114: they don't-- policies don't scale.
- 00:13:01,714 --> 00:13:03,914: You know, the workflow and the processes
- 00:13:03,994 --> 00:13:05,454: in, uh, within our tool
- 00:13:05,714 --> 00:13:06,714: scale that allowed us to,
- 00:13:07,394 --> 00:13:07,974: uh, to really,
- 00:13:08,454 --> 00:13:10,494: uh, reach a maximum amount of people across the globe,
- 00:13:10,554 --> 00:13:12,914: disparate locations, all using one platform.
- 00:13:13,674 --> 00:13:14,274: Sec- uh,
- 00:13:14,634 --> 00:13:16,414: the risk-based governance is essential,
- 00:13:16,754 --> 00:13:19,054: you know, you know, we can't just suddenly treat everything
- 00:13:19,094 --> 00:13:21,154: the same. That visibility
- 00:13:21,254 --> 00:13:22,294: now is now
- 00:13:22,694 --> 00:13:24,814: a accountability at our AI control
- 00:13:24,894 --> 00:13:27,294: board. Reporting from Cal- Calibra
- 00:13:27,634 --> 00:13:29,194: derives a lot of the discussion
- 00:13:29,474 --> 00:13:30,774: and the cha-- and, and the,
- 00:13:31,274 --> 00:13:33,034: um, logging of where we are with our
- 00:13:33,294 --> 00:13:33,854: AI journey.
- 00:13:34,934 --> 00:13:37,314: And then try to remove it becoming a blocker.
- 00:13:37,394 --> 00:13:39,434: We, we want it into a life cycle, so it
- 00:13:39,454 --> 00:13:40,254: becomes an AI
- 00:13:40,634 --> 00:13:42,194: life cycle tracker as well.
- 00:13:43,074 --> 00:13:45,174: And then that tooling really allows
- 00:13:45,234 --> 00:13:46,634: us to lift that scale from
- 00:13:46,974 --> 00:13:48,494: one AI use case to,
- 00:13:49,294 --> 00:13:51,554: in our current count, three hundred and fifty
- 00:13:51,854 --> 00:13:52,414: use cases.
- 00:13:54,494 --> 00:13:55,094: So what's next?
- 00:13:55,674 --> 00:13:57,134: Much of what's being discussed today,
- 00:13:57,914 --> 00:14:00,014: um, we wanna govern our AI agents.
- 00:14:00,354 --> 00:14:01,974: We wanna start moving from
- 00:14:02,434 --> 00:14:03,674: what effectively, what I call
- 00:14:04,334 --> 00:14:06,354: an honesty box approach, where people are
- 00:14:06,414 --> 00:14:07,154: self-declaring.
- 00:14:07,454 --> 00:14:08,934: We wanna actually start scanning.
- 00:14:09,214 --> 00:14:09,774: We wanna start
- 00:14:10,134 --> 00:14:11,674: plugging into things like Databricks,
- 00:14:12,274 --> 00:14:12,814: bringing that,
- 00:14:13,294 --> 00:14:15,234: uh, landscape to life. We wanna work with
- 00:14:15,574 --> 00:14:17,354: security teams to understand what
- 00:14:17,694 --> 00:14:19,834: agents are actually doing on this, on the ground
- 00:14:19,874 --> 00:14:20,594: in some cases.
- 00:14:21,074 --> 00:14:22,054: And then, um,
- 00:14:23,094 --> 00:14:23,774: we wanna, uh,
- 00:14:24,953 --> 00:14:26,113: that real-time monitoring.
- 00:14:26,393 --> 00:14:27,894: Things like pr- drift,
- 00:14:28,414 --> 00:14:30,614: uh, use case changes, those types of things
- 00:14:30,634 --> 00:14:32,734: should start to sur-surface through, through
- 00:14:32,814 --> 00:14:34,774: command center, which we're just-- we're actively,
- 00:14:35,554 --> 00:14:36,814: uh, developing just now.
- 00:14:37,274 --> 00:14:39,634: And then we've talked-- many of us have talked about MCP
- 00:14:39,694 --> 00:14:41,754: server. We now have quite a lot of trust
- 00:14:41,794 --> 00:14:42,454: and context
- 00:14:43,014 --> 00:14:43,694: in Collibra,
- 00:14:44,154 --> 00:14:46,154: and we wanna make sure that people start coming
- 00:14:46,214 --> 00:14:48,554: to that. We already had the question of where
- 00:14:48,634 --> 00:14:49,594: our UA use case,
- 00:14:50,674 --> 00:14:51,974: uh, register now have
- 00:14:52,254 --> 00:14:53,134: several hundred in them.
- 00:14:53,434 --> 00:14:55,694: People were confused as to whether there was duplicates.
- 00:14:56,754 --> 00:14:57,834: We were saying, "Well, actually,"
- 00:14:58,094 --> 00:14:59,534: and we've been testing Maestro,
- 00:15:00,174 --> 00:15:01,014: uh, assistant,
- 00:15:01,374 --> 00:15:02,554: you can now ask Maestro
- 00:15:02,954 --> 00:15:05,194: if there is a use case that looks similar to this
- 00:15:05,254 --> 00:15:06,234: already in the platform.
- 00:15:06,674 --> 00:15:09,174: And that means that people don't duplicate and don't recreate.
- 00:15:10,174 --> 00:15:12,194: And then obviously, we wanna make sure that we
- 00:15:12,254 --> 00:15:13,174: innovate safely.
- 00:15:13,634 --> 00:15:14,854: It's, you know, safety,
- 00:15:15,154 --> 00:15:16,014: we are as like
- 00:15:16,374 --> 00:15:18,394: number one agenda topic on almost every
- 00:15:18,474 --> 00:15:20,614: meeting we have. We have safety moments all the time.
- 00:15:21,094 --> 00:15:21,834: It's a, it's a,
- 00:15:22,194 --> 00:15:23,114: um, a real, um,
- 00:15:24,394 --> 00:15:26,094: uh, top to bottom conversation,
- 00:15:26,594 --> 00:15:28,594: and we, we feel responsibility in any
- 00:15:28,634 --> 00:15:29,754: of our innovations.
- 00:15:30,934 --> 00:15:32,014: So that's me.
- 00:15:32,394 --> 00:15:34,654: Uh, tr-trusted AI governance. Tr-trusted
- 00:15:34,694 --> 00:15:36,694: AI starts with good governance, as we've talked
- 00:15:36,734 --> 00:15:38,734: about, but it's about responsible
- 00:15:38,794 --> 00:15:39,694: innovation. That's
- 00:15:39,994 --> 00:15:40,714: how we should work.
- 00:15:41,614 --> 00:15:47,934: Thank you, Ryan. Thank you. [audience applauding]
How Weir is scaling AI Governance for the agentic era
As AI moves from models and use cases to agents that can act with increasing autonomy, governance has to evolve with it. For a global engineering leader in mining and infrastructure, that means maintaining visibility, accountability and control as AI scales.
In this session, Robin Sinclair, Senior Data Governance Manager at The Weir Group, shares how the company has taken AI governance from idea to production, creating a foundation to govern AI consistently today while preparing for a more agentic future. Hear how The Weir Group is reducing friction between delivery and risk, strengthening traceability and giving leadership greater confidence as AI adoption accelerates.Video: Who's governing whom? Data, Al and the new rules of the enterprise
Transcript
- 00:00:07,180 --> 00:00:09,440: So everybody is talking about the change
- 00:00:09,580 --> 00:00:10,880: today, right? And
- 00:00:11,140 --> 00:00:13,180: everything that's going around AI. I would
- 00:00:13,280 --> 00:00:15,720: love for you to kind of introduce yourself
- 00:00:16,180 --> 00:00:18,320: and what changes have you seen in the last
- 00:00:18,400 --> 00:00:19,640: two years that people
- 00:00:20,120 --> 00:00:22,300: maybe should or should not pay attention to.
- 00:00:22,940 --> 00:00:24,780: Sure. Thanks, Denise, and thank you everybody,
- 00:00:25,160 --> 00:00:26,700: uh, for the invitation for being here.
- 00:00:27,060 --> 00:00:29,460: My name's Kevin McNish, I'm an associate director
- 00:00:29,560 --> 00:00:30,760: in trustworthy AI,
- 00:00:31,020 --> 00:00:32,420: AI governance at Deloitte.
- 00:00:33,100 --> 00:00:35,140: Um, [lip smack] um, and
- 00:00:35,200 --> 00:00:37,360: I think in answer to your question there about
- 00:00:37,400 --> 00:00:39,780: the big change, I think Robin just summed it up beautifully
- 00:00:40,000 --> 00:00:42,260: just now with that shift of the scale
- 00:00:42,320 --> 00:00:43,020: that happens.
- 00:00:43,580 --> 00:00:44,280: Two years ago,
- 00:00:44,740 --> 00:00:47,220: three years ago, I think that, uh, governance
- 00:00:47,260 --> 00:00:50,140: tended to happen in a very person-led, person-centric
- 00:00:50,200 --> 00:00:52,220: way, uh, which worked for
- 00:00:52,280 --> 00:00:53,660: one or two use cases
- 00:00:54,720 --> 00:00:55,280: each month.
- 00:00:55,780 --> 00:00:57,820: Now, though, we're talking about tens
- 00:00:57,880 --> 00:00:59,940: if not hundreds of use cases that are coming
- 00:00:59,980 --> 00:01:02,000: through, and we simply can't cope in the ways
- 00:01:02,040 --> 00:01:04,239: that we were. And so that scaling
- 00:01:04,300 --> 00:01:06,480: problem is proving to be very challenging
- 00:01:06,520 --> 00:01:07,800: for everybody. Does it scare you?
- 00:01:08,940 --> 00:01:10,500: No, because I think agai-again,
- 00:01:11,180 --> 00:01:13,280: I reference back to Robin's talk, 'cause it was very good
- 00:01:13,320 --> 00:01:15,380: as to the journey that we group has been on.
- 00:01:15,960 --> 00:01:18,020: Um, I, I think there are ways of dealing
- 00:01:18,060 --> 00:01:19,880: with this, uh, in a very effective,
- 00:01:20,260 --> 00:01:20,860: um, manner.
- 00:01:21,300 --> 00:01:22,260: Yeah. And Michal?
- 00:01:22,880 --> 00:01:24,740: Hello. So my name is Michal Brzoza.
- 00:01:25,040 --> 00:01:27,220: I've led the data governance technology team
- 00:01:27,300 --> 00:01:28,640: across Deloitte Central Europe,
- 00:01:29,060 --> 00:01:30,960: um, and I responsible,
- 00:01:31,320 --> 00:01:33,480: uh, to support our clients' journey around
- 00:01:33,520 --> 00:01:35,180: the whole data governance implementation.
- 00:01:35,880 --> 00:01:37,180: Uh, coming back to your question,
- 00:01:37,580 --> 00:01:38,900: to me, it's about,
- 00:01:39,180 --> 00:01:40,300: uh, who firstly,
- 00:01:41,140 --> 00:01:43,000: who, uh, interacts with our data
- 00:01:43,620 --> 00:01:45,200: before we build the data,
- 00:01:45,580 --> 00:01:47,400: uh, data governance frameworks,
- 00:01:47,660 --> 00:01:48,980: uh, the data marketplace
- 00:01:49,380 --> 00:01:51,620: for data consumers, and it was for people.
- 00:01:52,100 --> 00:01:54,140: We didn't think usually about the machines,
- 00:01:54,300 --> 00:01:56,540: and now we are building those for agents who
- 00:01:56,600 --> 00:01:58,740: act, uh, alone without
- 00:01:58,780 --> 00:02:00,000: human interactions very often.
- 00:02:00,480 --> 00:02:01,680: And it changes,
- 00:02:01,960 --> 00:02:04,320: uh, a lot, and it require us to adjust
- 00:02:04,360 --> 00:02:04,980: the frameworks.
- 00:02:05,660 --> 00:02:06,440: And secondly,
- 00:02:06,720 --> 00:02:08,400: uh, who is using AI.
- 00:02:08,759 --> 00:02:09,940: Before, it was usually
- 00:02:10,340 --> 00:02:12,580: technical people, data scientists who spent
- 00:02:12,640 --> 00:02:14,140: a lot of time developing the models,
- 00:02:14,540 --> 00:02:16,300: and now it is used all across the,
- 00:02:16,760 --> 00:02:19,180: the business, uh, all across the enterprise
- 00:02:19,260 --> 00:02:21,340: by people who can just type or not
- 00:02:21,380 --> 00:02:23,620: even type. We can just talk to the agent and
- 00:02:23,680 --> 00:02:25,980: build the agent in a couple of minutes, as
- 00:02:26,340 --> 00:02:28,500: some of you saw today, even on the Maestro
- 00:02:28,560 --> 00:02:30,440: session, how simple it is, and
- 00:02:30,760 --> 00:02:32,800: that it doesn't require really technical
- 00:02:32,920 --> 00:02:34,920: skills. Yeah. So
- 00:02:35,020 --> 00:02:37,200: I'm curious because everybody thinks that
- 00:02:37,260 --> 00:02:39,620: if you put in AI, the more agents
- 00:02:39,680 --> 00:02:41,820: you add, the more it'll cure your data
- 00:02:41,880 --> 00:02:43,920: problems, right? So but we heard this a
- 00:02:43,960 --> 00:02:45,960: little bit earlier, it's like always garbage
- 00:02:46,020 --> 00:02:46,960: in, garbage out.
- 00:02:47,500 --> 00:02:50,140: So now when you think about ownership
- 00:02:50,200 --> 00:02:51,220: and the shift,
- 00:02:51,680 --> 00:02:53,860: like, what are you seeing as part of that,
- 00:02:54,360 --> 00:02:56,360: and how should customers actually
- 00:02:56,400 --> 00:02:57,200: think about it?
- 00:02:59,620 --> 00:03:01,620: So I'll take that- Yeah... first. Yep. So I, I
- 00:03:01,640 --> 00:03:03,020: would say that the shift,
- 00:03:03,400 --> 00:03:05,520: again, to reflect what Michal was just saying, has
- 00:03:05,560 --> 00:03:07,620: gone really from this is a technical problem-
- 00:03:08,060 --> 00:03:09,640: Mm... this is an issue for the technical area,
- 00:03:10,100 --> 00:03:12,200: to recognizing that it's a business management problem.
- 00:03:12,660 --> 00:03:14,540: It's no longer just an issue with the tech.
- 00:03:14,960 --> 00:03:17,000: It's no longer just an issue for cyber or
- 00:03:17,080 --> 00:03:19,480: just an issue for legal. It's an issue for everybody.
- 00:03:19,540 --> 00:03:20,840: They all need to come together.
- 00:03:21,480 --> 00:03:23,980: And I think with a lot of organizations,
- 00:03:24,100 --> 00:03:26,160: there are challenges with silos that exist
- 00:03:26,200 --> 00:03:27,940: between those different functions,
- 00:03:28,420 --> 00:03:29,540: if you like, in many cases
- 00:03:30,400 --> 00:03:32,520: because there's a tradition built up as to
- 00:03:32,620 --> 00:03:35,120: the, the routes through which governance processes
- 00:03:35,200 --> 00:03:37,460: flow for different areas and for different
- 00:03:37,520 --> 00:03:39,780: concerns. And I think one of the things we're seeing
- 00:03:39,820 --> 00:03:41,900: now is this need to bring those
- 00:03:41,960 --> 00:03:43,300: governance functions together,
- 00:03:44,200 --> 00:03:46,220: uh, for the challenges that AI raise.
- 00:03:47,180 --> 00:03:49,300: Yeah. So here's the interesting part. We always talk
- 00:03:49,360 --> 00:03:51,500: about, you know, the people, which
- 00:03:51,560 --> 00:03:53,280: is tougher to manage with AI.
- 00:03:53,800 --> 00:03:56,080: The tech, at least from our perspective from Collibra,
- 00:03:56,100 --> 00:03:57,840: is always the easiest part to put in.
- 00:03:58,360 --> 00:04:00,500: But I think if you look at what
- 00:04:00,580 --> 00:04:01,820: the change is,
- 00:04:02,360 --> 00:04:04,460: this is really the time to do a lot of
- 00:04:04,520 --> 00:04:07,000: this business process optimization
- 00:04:07,020 --> 00:04:08,000: and re-engineering,
- 00:04:08,420 --> 00:04:10,500: which I don't think we're-- everybody's
- 00:04:10,540 --> 00:04:12,820: thinking about- Yeah... because every agent you add
- 00:04:12,900 --> 00:04:15,160: in, it's an opportunity to actually improve
- 00:04:15,200 --> 00:04:16,120: the business process,
- 00:04:16,540 --> 00:04:18,620: which actually optimizes your
- 00:04:18,680 --> 00:04:21,019: revenue or whatever use case you're going after.
- 00:04:21,480 --> 00:04:22,940: So when you- Yeah... look at that,
- 00:04:23,280 --> 00:04:25,380: do you see that across the board as
- 00:04:25,420 --> 00:04:27,420: an opportunity when you're working on some
- 00:04:27,460 --> 00:04:28,960: of these projects and activities?
- 00:04:29,680 --> 00:04:31,680: Um, absolutely. Uh, an opportunity, but of
- 00:04:31,720 --> 00:04:32,580: course, also a risk.
- 00:04:33,140 --> 00:04:35,600: Yeah. Um, because if they're put in without the guardrails
- 00:04:35,660 --> 00:04:36,220: in place,
- 00:04:36,920 --> 00:04:39,600: then that risk, as we've been hearing, just magnifies
- 00:04:40,100 --> 00:04:42,280: with every additional agent. So those guardrails
- 00:04:42,360 --> 00:04:44,560: need to be understood before those agents
- 00:04:44,580 --> 00:04:46,660: are put in place. But once they're there, of course, they're gonna
- 00:04:46,700 --> 00:04:47,440: be transformative,
- 00:04:48,060 --> 00:04:48,920: and that, that's the whole
- 00:04:49,320 --> 00:04:50,780: joy of where we're going with agents.
- 00:04:51,260 --> 00:04:53,480: Yes. I love that word. You used joy and not, like,
- 00:04:53,620 --> 00:04:55,160: scare everybody- [laughs]... when you mentioned AI.
- 00:04:55,660 --> 00:04:57,760: And Michal, you have actually been on
- 00:04:57,940 --> 00:04:59,880: one of our strategic projects
- 00:05:00,360 --> 00:05:02,580: together at one of the very large
- 00:05:02,680 --> 00:05:04,680: customers, and you've put in a lot of data
- 00:05:04,760 --> 00:05:06,000: products as part of this.
- 00:05:06,340 --> 00:05:08,560: Have you seen a shift in the organizational
- 00:05:08,660 --> 00:05:09,980: change as part of this?
- 00:05:10,580 --> 00:05:12,040: I think, uh, definitely,
- 00:05:12,340 --> 00:05:14,180: and it's mostly because
- 00:05:14,860 --> 00:05:16,860: for whom it was designed some time ago at
- 00:05:16,880 --> 00:05:19,000: this project which is going
- 00:05:19,060 --> 00:05:20,180: for, for some time already.
- 00:05:20,680 --> 00:05:21,840: And we've built
- 00:05:22,160 --> 00:05:24,420: the marketplace, the products with
- 00:05:24,680 --> 00:05:25,980: people in mind, and now
- 00:05:26,300 --> 00:05:26,880: we need to
- 00:05:27,420 --> 00:05:29,720: kind of redesign the framework and rethink
- 00:05:29,760 --> 00:05:30,440: the things that
- 00:05:30,780 --> 00:05:32,860: were developed and adjust them, add
- 00:05:33,020 --> 00:05:34,980: new things like controls, like
- 00:05:35,460 --> 00:05:36,340: new attributes,
- 00:05:36,820 --> 00:05:39,040: uh, adjust our processes, operating
- 00:05:39,140 --> 00:05:41,520: model, adjust our workflows to
- 00:05:41,580 --> 00:05:43,940: make those product not only like marketplace
- 00:05:43,960 --> 00:05:45,380: they're ready but also,
- 00:05:45,700 --> 00:05:47,020: uh, agent-ready.
- 00:05:47,580 --> 00:05:49,900: So then the owner can decide whether this
- 00:05:49,920 --> 00:05:51,600: product can be consumed by agent,
- 00:05:52,220 --> 00:05:54,260: maybe not whole product, maybe just
- 00:05:54,440 --> 00:05:55,340: a couple of outputs.
- 00:05:55,820 --> 00:05:56,760: But, uh,
- 00:05:57,420 --> 00:05:59,600: all this operating model needs to be
- 00:05:59,680 --> 00:06:00,280: adjusted
- 00:06:00,740 --> 00:06:02,620: to the current rela-reality.
- 00:06:03,368 --> 00:06:05,408: Yeah, and I put a, uh, fun couple
- 00:06:05,448 --> 00:06:07,408: of QR codes in the background here.
- 00:06:07,848 --> 00:06:09,868: Any comments that you wanna put around
- 00:06:09,928 --> 00:06:12,068: some of the surveys that you've done? Michal, we'll start
- 00:06:12,088 --> 00:06:14,188: with you there. Uh, sure. So
- 00:06:14,588 --> 00:06:15,528: those survey, uh,
- 00:06:15,788 --> 00:06:17,848: first survey is about, uh,
- 00:06:18,608 --> 00:06:20,228: how the chief data officer
- 00:06:20,488 --> 00:06:21,548: thinks about
- 00:06:21,908 --> 00:06:22,668: current landscape.
- 00:06:23,068 --> 00:06:24,228: Uh, we are doing this
- 00:06:24,528 --> 00:06:26,288: quite often, usually each year.
- 00:06:26,948 --> 00:06:29,208: Uh, and it good... it shows
- 00:06:29,368 --> 00:06:31,428: well how the ownership for exactly
- 00:06:31,468 --> 00:06:32,348: changed because,
- 00:06:33,008 --> 00:06:35,068: uh, it clearly shows that it moved out
- 00:06:35,348 --> 00:06:37,368: the center. It's now spread across
- 00:06:37,468 --> 00:06:39,228: various business function. CDO
- 00:06:39,588 --> 00:06:41,668: does not own whole budget. The agents,
- 00:06:41,828 --> 00:06:43,828: the data solutions are being
- 00:06:43,888 --> 00:06:45,368: developed across various units,
- 00:06:45,908 --> 00:06:47,848: uh, across various functions.
- 00:06:48,448 --> 00:06:49,148: But, uh,
- 00:06:49,408 --> 00:06:50,768: what's kind of worrying that
- 00:06:51,288 --> 00:06:53,428: data governance is only mandatory
- 00:06:53,488 --> 00:06:55,668: approver for IT governance
- 00:06:55,948 --> 00:06:58,608: just in one-fifth of those organizations.
- 00:06:59,148 --> 00:07:00,388: And this means that AI
- 00:07:00,968 --> 00:07:02,988: is being built without even governance
- 00:07:03,048 --> 00:07:04,748: being aware that something is being built.
- 00:07:05,148 --> 00:07:07,568: And this is the, the huge problem in case,
- 00:07:07,788 --> 00:07:09,868: uh, when we move from one agent or twenty
- 00:07:09,948 --> 00:07:12,048: agents to thousands of agents running
- 00:07:12,088 --> 00:07:13,008: across the organization.
- 00:07:13,708 --> 00:07:15,728: Yeah. And I'm curious, like, we were talking
- 00:07:15,808 --> 00:07:17,828: about this at lunch today, which is now
- 00:07:17,908 --> 00:07:19,988: the move with security and
- 00:07:20,028 --> 00:07:21,968: the CISO, right, as part of this.
- 00:07:22,508 --> 00:07:24,588: Uh, any details on the survey that
- 00:07:24,648 --> 00:07:26,668: you can share? Did we see this move as
- 00:07:26,748 --> 00:07:29,228: a group besides the architects
- 00:07:29,348 --> 00:07:31,468: and, you know, of course, the AI teams as part
- 00:07:31,508 --> 00:07:33,528: of it? I think
- 00:07:33,928 --> 00:07:36,068: not in that one, but the survey we are working
- 00:07:36,168 --> 00:07:37,688: at for twenty twenty-six,
- 00:07:38,048 --> 00:07:40,248: uh, those kind of elements for sure
- 00:07:40,488 --> 00:07:42,168: will be, will be touched on and included.
- 00:07:42,768 --> 00:07:44,868: Great. And Kevin, anything
- 00:07:44,908 --> 00:07:46,968: on the survey that you wanna add as part of that? No, I
- 00:07:47,008 --> 00:07:49,308: think Michal summed them up beautifully there. Thank you. Okay.
- 00:07:49,588 --> 00:07:51,648: Okay. Um, so we heard
- 00:07:51,688 --> 00:07:52,708: a little bit about,
- 00:07:52,988 --> 00:07:55,148: um, you know, the move from data governance
- 00:07:55,208 --> 00:07:57,088: to context governance, especially when
- 00:07:57,508 --> 00:07:59,908: Felix was presenting this a little bit earlier.
- 00:08:00,408 --> 00:08:02,508: How do you set up organizations to
- 00:08:02,528 --> 00:08:04,588: really manage by exception? This is so
- 00:08:04,648 --> 00:08:05,528: hard because
- 00:08:06,168 --> 00:08:08,188: everything that we're doing, typically,
- 00:08:08,388 --> 00:08:10,708: whether, you know, you look at the traditional
- 00:08:10,828 --> 00:08:13,067: data governance, it's setting up an asset.
- 00:08:13,428 --> 00:08:15,508: You do some automation, but it's still manual. You set up
- 00:08:15,548 --> 00:08:16,068: a workflow.
- 00:08:16,548 --> 00:08:18,608: So again, still, you know, manual is part
- 00:08:18,648 --> 00:08:20,768: of it. So how do you really drive a
- 00:08:20,848 --> 00:08:23,288: culture of managing by exception?
- 00:08:23,728 --> 00:08:25,488: And how do you get our customers there?
- 00:08:25,848 --> 00:08:27,828: And what are some of the best practices you've seen?
- 00:08:30,928 --> 00:08:31,688: So I think, yeah,
- 00:08:32,008 --> 00:08:34,048: great, great question there and,
- 00:08:34,128 --> 00:08:36,128: and very challenging because I think that
- 00:08:36,188 --> 00:08:38,268: the-- in, in many areas,
- 00:08:38,328 --> 00:08:40,328: it's still seen that the human overview,
- 00:08:40,448 --> 00:08:42,728: the human in the loop is the solution-
- 00:08:42,748 --> 00:08:43,668: Yeah... to the problem.
- 00:08:44,308 --> 00:08:46,568: And again, that may have been the case
- 00:08:46,608 --> 00:08:48,608: three, four years ago. That's clearly not
- 00:08:48,848 --> 00:08:51,268: realistic anymore with, again, with the scale,
- 00:08:51,408 --> 00:08:51,988: with the speed,
- 00:08:52,448 --> 00:08:53,968: and, and particularly with agents,
- 00:08:54,308 --> 00:08:56,408: um, and, and the autonomous functioning.
- 00:08:56,968 --> 00:08:57,668: And so
- 00:08:57,928 --> 00:08:59,948: I think we need to move into
- 00:09:00,028 --> 00:09:00,628: real-time
- 00:09:00,948 --> 00:09:02,268: governance, runtime governance,
- 00:09:02,868 --> 00:09:03,888: in which we're both
- 00:09:04,368 --> 00:09:06,508: monitoring and responding to the challenges
- 00:09:06,628 --> 00:09:07,808: as they arise.
- 00:09:08,308 --> 00:09:10,328: And so systems that can
- 00:09:10,388 --> 00:09:12,388: monitor, systems that can flag up
- 00:09:12,888 --> 00:09:15,088: issues as they come, and thinking about what those
- 00:09:15,128 --> 00:09:16,148: controls are
- 00:09:16,728 --> 00:09:18,728: in advance of actually putting the system
- 00:09:18,768 --> 00:09:20,228: into deployment in the first place
- 00:09:20,708 --> 00:09:22,708: so that you can have the guardrails in place
- 00:09:22,748 --> 00:09:23,668: to ensure that,
- 00:09:24,208 --> 00:09:26,448: uh, it, it will op- the agent will operate within
- 00:09:26,508 --> 00:09:27,248: safe boundaries.
- 00:09:27,748 --> 00:09:29,708: But then do you need different profiles
- 00:09:30,088 --> 00:09:32,208: for the roles that we have that
- 00:09:32,268 --> 00:09:34,288: support context governance as part of
- 00:09:34,348 --> 00:09:35,488: this, especially if you manage
- 00:09:35,948 --> 00:09:36,868: by exceptions?
- 00:09:38,468 --> 00:09:41,048: I, I think that the approach we tend to
- 00:09:41,168 --> 00:09:41,968: see most
- 00:09:42,288 --> 00:09:44,548: commonly is an uplift in the existing
- 00:09:44,628 --> 00:09:45,948: profiles that we have. Mm-hmm.
- 00:09:46,568 --> 00:09:48,888: And so rather than bringing in new profiles,
- 00:09:48,968 --> 00:09:51,088: designing new profiles, we're simpl- we, we, we're
- 00:09:51,128 --> 00:09:53,288: seeing more that the existing functions are saying,
- 00:09:53,328 --> 00:09:55,388: "Right, we need to accept and understand
- 00:09:55,428 --> 00:09:56,608: what these new risks are
- 00:09:57,088 --> 00:09:58,328: and how we're going to manage those."
- 00:09:58,708 --> 00:10:00,988: Yeah. And Michal, your thoughts?
- 00:10:01,728 --> 00:10:03,508: Um, I think what works
- 00:10:03,808 --> 00:10:05,688: really well is bringing the
- 00:10:06,068 --> 00:10:08,088: ownership, the stewardship closer to the
- 00:10:08,128 --> 00:10:10,128: business people who actually owns the data, who are
- 00:10:10,188 --> 00:10:12,508: act- uh, accountable, responsible for
- 00:10:12,568 --> 00:10:13,848: them on daily basis.
- 00:10:14,688 --> 00:10:16,348: But at the same time, it's a big challenge,
- 00:10:16,628 --> 00:10:18,948: and it's usually not the budget or technology
- 00:10:19,008 --> 00:10:21,927: challenge. It's usually people and culture chal-challenge,
- 00:10:22,528 --> 00:10:24,328: uh, with, uh--
- 00:10:25,388 --> 00:10:25,988: which is not
- 00:10:26,408 --> 00:10:28,608: easily solved, and there is no one recipe how
- 00:10:28,688 --> 00:10:29,928: to solve it and how to--
- 00:10:30,288 --> 00:10:32,468: how it will-- can work in one organization,
- 00:10:33,028 --> 00:10:34,268: uh, because
- 00:10:34,648 --> 00:10:35,748: each organization is different.
- 00:10:37,408 --> 00:10:39,528: What is really important to equip those business
- 00:10:39,608 --> 00:10:42,148: people with the right tools, with the standards,
- 00:10:42,268 --> 00:10:42,928: and with the
- 00:10:43,648 --> 00:10:46,068: easy-to-use tools. So for them, like buzzwords
- 00:10:46,168 --> 00:10:48,408: like communities, domains, assets
- 00:10:49,048 --> 00:10:50,088: are completely
- 00:10:50,488 --> 00:10:52,248: complicated words which means
- 00:10:52,608 --> 00:10:54,828: nothing, although for us are super clear.
- 00:10:55,408 --> 00:10:56,208: But they have
- 00:10:56,588 --> 00:10:58,728: completely different priorities on daily basis. They
- 00:10:58,788 --> 00:11:00,428: use tons of different systems.
- 00:11:01,008 --> 00:11:03,068: Uh, and we need to make sure
- 00:11:03,128 --> 00:11:05,168: that those tools and our processes
- 00:11:05,328 --> 00:11:06,968: are easy enough for them to use
- 00:11:07,408 --> 00:11:09,428: and are obvious for them, not for
- 00:11:09,508 --> 00:11:11,548: us. Yeah. And you've done
- 00:11:11,628 --> 00:11:13,768: a lot of implementations, right? So if you
- 00:11:13,788 --> 00:11:16,008: think about just from an industry perspective,
- 00:11:16,428 --> 00:11:18,608: we've heard everybody from pharma
- 00:11:18,688 --> 00:11:19,908: today to mining.
- 00:11:20,328 --> 00:11:22,508: Do you see the differences sometimes in
- 00:11:22,608 --> 00:11:25,208: the industries as part of
- 00:11:25,288 --> 00:11:26,428: just the shift itself?
- 00:11:27,488 --> 00:11:29,708: I would say no. So the, the shift is present
- 00:11:29,828 --> 00:11:32,088: everywhere. Uh, of course, it's
- 00:11:32,148 --> 00:11:33,728: different. Like, uh, something
- 00:11:34,028 --> 00:11:36,188: else will be priority in pharma, something else
- 00:11:36,228 --> 00:11:38,048: will be priority- Yeah... in FSI or f-
- 00:11:38,408 --> 00:11:39,408: or in manufacturing
- 00:11:39,948 --> 00:11:42,328: or in any unregulated business who drives
- 00:11:42,368 --> 00:11:44,528: the-- who builds the data governance because
- 00:11:44,608 --> 00:11:45,848: of actual need and not
- 00:11:46,248 --> 00:11:47,668: regulatory requirements.
- 00:11:48,288 --> 00:11:49,028: But, uh,
- 00:11:49,308 --> 00:11:51,308: the baseline and the found-foundation is the
- 00:11:51,348 --> 00:11:52,408: same, and you need to
- 00:11:52,788 --> 00:11:55,008: have the strong found-foundation and a good framework,
- 00:11:55,588 --> 00:11:57,748: uh, to start with and to bring those specific
- 00:11:57,788 --> 00:11:59,988: use cases. Yeah. So this
- 00:12:00,068 --> 00:12:00,588: is the
- 00:12:00,868 --> 00:12:03,128: fastest technology adoption
- 00:12:03,228 --> 00:12:05,288: that we're in the midst of as part
- 00:12:05,328 --> 00:12:05,988: of this, right?
- 00:12:06,888 --> 00:12:07,868: Why-- What
- 00:12:08,448 --> 00:12:08,968: do you,
- 00:12:09,468 --> 00:12:11,788: what do you think should have changed but
- 00:12:11,848 --> 00:12:12,908: hasn't changed yet?
- 00:12:13,328 --> 00:12:13,988: And what is
- 00:12:14,288 --> 00:12:16,408: being so stubborn when it comes to AI
- 00:12:16,468 --> 00:12:17,248: as part of this?
- 00:12:18,128 --> 00:12:19,368: So, Kevin, you wanna start?
- 00:12:21,380 --> 00:12:22,720: I think probably for me
- 00:12:23,060 --> 00:12:25,160: it is, and again, this harks back to,
- 00:12:25,340 --> 00:12:27,560: to Robin's talk just now, that everything
- 00:12:27,600 --> 00:12:29,220: in governance begins with the inventory.
- 00:12:29,900 --> 00:12:31,900: If you don't have an inventory, if you don't know what's
- 00:12:32,000 --> 00:12:33,120: out there, if you don't know what
- 00:12:33,580 --> 00:12:35,960: AI you're using, how it's being used,
- 00:12:36,100 --> 00:12:38,260: who owns it, what its capacities are,
- 00:12:39,000 --> 00:12:41,080: even down the line to what tokenization
- 00:12:41,200 --> 00:12:42,060: costs may be,
- 00:12:42,700 --> 00:12:44,720: then you're operating from blindness, and
- 00:12:44,760 --> 00:12:46,840: you don't really know what you're dealing with. You don't
- 00:12:46,860 --> 00:12:47,760: know what your risks are.
- 00:12:48,500 --> 00:12:50,700: Uh, and, and particularly if you've got a footprint
- 00:12:50,740 --> 00:12:52,900: in the EU, then those risks could turn out
- 00:12:52,940 --> 00:12:54,600: to be very, very expensive indeed.
- 00:12:55,340 --> 00:12:57,540: So I think you need to start
- 00:12:57,620 --> 00:12:59,740: out with the inventory, and I am constantly
- 00:12:59,780 --> 00:13:00,440: surprised,
- 00:13:00,820 --> 00:13:03,160: um, as I speak to organizations
- 00:13:03,240 --> 00:13:05,380: as to what the state of their inventory is.
- 00:13:05,760 --> 00:13:07,960: And I don't just mean a sort of scanning where you sit down
- 00:13:07,980 --> 00:13:10,180: and go, "Okay, these, these are all the shadow AIs
- 00:13:10,220 --> 00:13:11,040: that are going on."
- 00:13:11,460 --> 00:13:13,540: I mean a very clear understanding of exactly
- 00:13:13,580 --> 00:13:15,740: what use cases you have, what they're supposed
- 00:13:15,780 --> 00:13:17,260: to do, how you're monitoring them,
- 00:13:17,720 --> 00:13:20,000: what the controls are, what the flags are,
- 00:13:20,080 --> 00:13:21,160: if it goes out of
- 00:13:21,540 --> 00:13:23,380: accepted boundaries and so on.
- 00:13:23,780 --> 00:13:25,800: So I think a really clear understanding of
- 00:13:25,840 --> 00:13:27,900: what's out there in your own
- 00:13:27,980 --> 00:13:30,200: organizational space is absolutely crucial.
- 00:13:30,540 --> 00:13:33,160: And once you have that, then you can start to make data-informed
- 00:13:33,200 --> 00:13:35,200: decisions. But until that point, as I
- 00:13:35,240 --> 00:13:36,580: say, you're, you're flying blind.
- 00:13:37,340 --> 00:13:38,820: For a second, I thought you said
- 00:13:39,080 --> 00:13:41,120: infantry as in going to war- [laughs]... and
- 00:13:41,140 --> 00:13:43,480: then I realized it was inventory, right?
- 00:13:43,520 --> 00:13:45,640: Yeah. [laughs] So literally the inventory is so key-
- 00:13:45,720 --> 00:13:46,500: Yeah... as part of it.
- 00:13:47,040 --> 00:13:48,160: Uh, Michal, over to you.
- 00:13:48,720 --> 00:13:50,360: Uh, sure. I think to me,
- 00:13:50,660 --> 00:13:51,800: um, proving the
- 00:13:52,260 --> 00:13:54,280: value of the governance, but actually
- 00:13:54,300 --> 00:13:56,420: in the, the words that business cares
- 00:13:56,500 --> 00:13:58,660: about. So I think we do
- 00:13:58,700 --> 00:14:00,700: not have issue nowadays with measuring what
- 00:14:00,740 --> 00:14:02,860: we are doing around the data governance programs.
- 00:14:02,960 --> 00:14:04,180: We are really well on
- 00:14:04,480 --> 00:14:05,080: reporting
- 00:14:05,680 --> 00:14:06,880: how many owners we have,
- 00:14:07,300 --> 00:14:09,340: how much workflows we are doing and so
- 00:14:09,460 --> 00:14:11,140: on, how m- how big, uh,
- 00:14:11,540 --> 00:14:13,740: are our glossaries and so on, but we
- 00:14:13,820 --> 00:14:15,220: struggle to stick it with the
- 00:14:15,660 --> 00:14:16,820: business results. So
- 00:14:17,200 --> 00:14:19,260: why this drove our revenue or
- 00:14:19,320 --> 00:14:21,280: why our decisions were better.
- 00:14:21,960 --> 00:14:24,640: Uh, and it is usually so hard because
- 00:14:24,720 --> 00:14:26,860: it is hidden between
- 00:14:26,940 --> 00:14:29,359: other teams' results. So we have the data engineering
- 00:14:29,419 --> 00:14:30,960: team who developed the use case.
- 00:14:31,579 --> 00:14:32,920: We have sales who drove,
- 00:14:33,220 --> 00:14:34,080: drove the revenue,
- 00:14:34,480 --> 00:14:35,520: and it could be done bec-
- 00:14:35,820 --> 00:14:37,860: because of the governance and the foundations,
- 00:14:37,960 --> 00:14:39,200: but usually it is,
- 00:14:39,740 --> 00:14:41,280: uh, like the credit is not there.
- 00:14:41,780 --> 00:14:43,780: And with the AI it can be
- 00:14:43,840 --> 00:14:44,880: visible right away.
- 00:14:45,420 --> 00:14:47,500: Um, I think it was
- 00:14:47,540 --> 00:14:49,660: said already today that garbage in, garbage out,
- 00:14:49,940 --> 00:14:50,560: it isn't new,
- 00:14:51,100 --> 00:14:52,860: but with the agents at scale,
- 00:14:53,520 --> 00:14:55,480: uh, gar- garbage gets acted on,
- 00:14:55,920 --> 00:14:57,220: uh, before you notice.
- 00:14:57,480 --> 00:14:59,560: And when you notice, it's usually too late and it
- 00:14:59,820 --> 00:15:01,220: already costed a lot of money.
- 00:15:01,960 --> 00:15:04,620: Yeah. I absolutely agree with you. I mean, that ROI,
- 00:15:04,740 --> 00:15:05,600: the business outcome,
- 00:15:06,080 --> 00:15:08,120: of course, is something we're all working towards
- 00:15:08,160 --> 00:15:10,240: where the ROI is always very tough to
- 00:15:10,300 --> 00:15:12,600: measure. And I think it depends
- 00:15:12,720 --> 00:15:15,140: on the industry, who you report
- 00:15:15,240 --> 00:15:16,500: to as part of it.
- 00:15:16,940 --> 00:15:19,120: But that's the constant piece that comes up,
- 00:15:19,200 --> 00:15:20,880: right? And I think that really
- 00:15:21,480 --> 00:15:23,660: closes nicely with the fact
- 00:15:23,720 --> 00:15:26,220: that the change in ownership also impacts
- 00:15:26,260 --> 00:15:28,800: that as part of the change management
- 00:15:28,840 --> 00:15:29,820: and the sponsorship.
- 00:15:30,200 --> 00:15:32,220: So thank you both for participating-
- 00:15:32,220 --> 00:15:34,220: Thank you... in the panel. Thank you. And with that, I would
- 00:15:34,300 --> 00:15:36,860: love to welcome Felix back on stage.
- 00:15:36,920 --> 00:15:39,160: Thank you. Thank you very much.
- 00:15:39,160 --> 00:15:39,620: [audience applauding]
Who's governing whom? Data, Al and the new rules of the enterprise
Every enterprise is racing to deploy AI — far fewer are asking whether their data is ready for it, and that gap is where AI quietly fails.
Deloitte and Collibra cut through the buzzwords — "context," "control," "trusted governance" — to explore what they mean in practice. Drawing on real client work across industries, the panel tackles the trends reshaping trustworthy AI and data governance, and how organisations navigate what it truly takes to get data AI-ready — beyond ticking the EU AI Act compliance box. A candid look at why so many rush into AI before their data foundation is set, and why "garbage in, garbage out" has never carried higher stakes than in the age of AI agents.
Join this interactive and insightful panel session, moderated by Denise Vu Broady, Executive Vice President at Collibra, with Kevin Macnish, Associate Director, AI Governance at Deloitte and Michal Brzoza, Data Governance Technology Lead, Data Governance at Deloitte.Video: Sovereignty spotlight
Transcript
- 00:00:07,120 --> 00:00:08,140: I will talk a little bit about
- 00:00:08,620 --> 00:00:10,760: sovereignty, which I know is a, a really big topic.
- 00:00:10,940 --> 00:00:12,620: We're here in Europe. Lots of different dimensions,
- 00:00:12,880 --> 00:00:13,600: geopolitical,
- 00:00:14,120 --> 00:00:15,160: intellectual property,
- 00:00:15,440 --> 00:00:17,620: I think top of mind for many, many organizations.
- 00:00:17,660 --> 00:00:19,720: But before we kind of unpack that, would
- 00:00:19,740 --> 00:00:21,780: love to learn a bit more about you, what you do
- 00:00:21,800 --> 00:00:22,480: at Mistral. Maybe
- 00:00:22,760 --> 00:00:23,760: share that with the audience today.
- 00:00:24,060 --> 00:00:24,980: Yeah, absolutely.
- 00:00:25,440 --> 00:00:27,460: Uh, it's great to be here. Thank you for having me.
- 00:00:27,760 --> 00:00:29,140: Um, my name is Kayla Nauhi.
- 00:00:29,440 --> 00:00:31,760: I am a product lead at Mistral AI,
- 00:00:32,420 --> 00:00:34,440: uh, and I own our specialized
- 00:00:34,480 --> 00:00:37,300: models, so if you're thinking about document intelligence,
- 00:00:37,500 --> 00:00:38,140: OCR,
- 00:00:38,400 --> 00:00:40,800: audio, um, anything that requires
- 00:00:40,860 --> 00:00:43,040: a specialized application rather than
- 00:00:43,100 --> 00:00:44,640: a general purpose model.
- 00:00:45,720 --> 00:00:47,120: Um, I think
- 00:00:47,680 --> 00:00:49,180: I, I can jump right into sovereignty.
- 00:00:49,740 --> 00:00:51,840: Shall I? Yeah. But a- a- any examples of maybe
- 00:00:52,140 --> 00:00:54,220: customers you work with or, or how you maybe use
- 00:00:54,260 --> 00:00:56,640: some of those specialized models? Yeah, absolutely.
- 00:00:57,140 --> 00:00:59,400: Um, so I know we have a, a lot of crossover
- 00:00:59,480 --> 00:01:00,260: in the FSI
- 00:01:00,540 --> 00:01:02,580: segment- Okay... um, and there are so many
- 00:01:02,660 --> 00:01:04,700: applications for the specialized models when
- 00:01:04,739 --> 00:01:06,300: you're thinking about banking. But,
- 00:01:06,860 --> 00:01:08,400: um, I can give you two that are-
- 00:01:08,680 --> 00:01:09,280: Yeah... are pretty,
- 00:01:09,720 --> 00:01:10,320: pretty on it,
- 00:01:10,600 --> 00:01:12,780: um, in the, these last couple of days.
- 00:01:13,540 --> 00:01:14,900: One is, uh,
- 00:01:15,460 --> 00:01:18,100: OCR is a, is an old technology.
- 00:01:18,420 --> 00:01:20,420: Um, people say OCR is a solved
- 00:01:20,480 --> 00:01:21,020: problem.
- 00:01:21,600 --> 00:01:23,780: Slight argument on my- [chuckles]... my side for that.
- 00:01:23,820 --> 00:01:24,460: But, um,
- 00:01:25,360 --> 00:01:25,980: it is an
- 00:01:26,260 --> 00:01:28,340: incredible way, and I saw it's something that you guys
- 00:01:28,380 --> 00:01:29,560: are using as well- Yep...
- 00:01:30,280 --> 00:01:32,280: um, uh, in order to kind of get a better understanding of
- 00:01:32,320 --> 00:01:34,520: what the documents are containing. But it's an incredible
- 00:01:34,560 --> 00:01:35,420: way of taking
- 00:01:35,920 --> 00:01:38,040: knowledge that is trapped in
- 00:01:38,120 --> 00:01:40,040: formats that are not necessarily,
- 00:01:40,360 --> 00:01:42,620: uh, cohesive to context
- 00:01:42,660 --> 00:01:44,920: the way AI engineers understand
- 00:01:44,980 --> 00:01:45,660: context,
- 00:01:46,220 --> 00:01:48,300: um, and making that accessible to
- 00:01:48,380 --> 00:01:49,280: AI systems.
- 00:01:49,820 --> 00:01:52,140: Um, so thinking about digitizing
- 00:01:52,280 --> 00:01:53,800: huge amounts of data,
- 00:01:54,160 --> 00:01:56,160: um, and turning that into context,
- 00:01:56,560 --> 00:01:58,760: OCR is a really, really great place to start.
- 00:01:59,060 --> 00:02:00,860: Yep. And then with audio,
- 00:02:01,240 --> 00:02:03,540: um, you've actually caught me on a rather audio-pilled
- 00:02:03,680 --> 00:02:05,900: day as we, uh, released Voxtral
- 00:02:06,160 --> 00:02:08,780: 3, our suite of, of audio models yesterday.
- 00:02:09,080 --> 00:02:11,900: Um, there are so many applications
- 00:02:11,960 --> 00:02:14,300: for audio across- Yeah... the enterprise context.
- 00:02:14,800 --> 00:02:16,840: The biggest and maybe kind of the most
- 00:02:16,900 --> 00:02:18,940: exciting for me recently has
- 00:02:18,980 --> 00:02:20,020: been voice agents.
- 00:02:20,400 --> 00:02:22,380: There is so much potential,
- 00:02:22,700 --> 00:02:24,760: um, when you're looking at customer
- 00:02:24,900 --> 00:02:26,320: care, customer service,
- 00:02:26,800 --> 00:02:29,160: bringing AI into these engagements,
- 00:02:29,260 --> 00:02:31,320: whether it's agent assist, where you have a
- 00:02:31,380 --> 00:02:32,380: human still, but
- 00:02:32,780 --> 00:02:34,800: you're using an AI to listen and pull
- 00:02:34,860 --> 00:02:37,260: up relevant information, or full automation,
- 00:02:37,420 --> 00:02:38,820: where you are taking, uh,
- 00:02:39,220 --> 00:02:41,660: a voice agent and deploying it in an existing
- 00:02:41,720 --> 00:02:42,680: legacy system,
- 00:02:43,300 --> 00:02:45,660: um, and, uh, interestingly, often
- 00:02:45,800 --> 00:02:47,620: seeing a greater CSAT score
- 00:02:47,880 --> 00:02:50,160: than when a human is on the other end of the line. So,
- 00:02:50,280 --> 00:02:52,860: um, two examples, both FSI,
- 00:02:53,000 --> 00:02:55,180: both incredibly exciting. Yeah. No, thank,
- 00:02:55,320 --> 00:02:56,100: thank you. And, and
- 00:02:56,760 --> 00:02:58,760: we traditionally talk about, like, traditional data, but to your point
- 00:02:58,800 --> 00:03:01,100: in, in AI that this multimodal is a huge opportunity
- 00:03:01,140 --> 00:03:03,000: and, and much kind of wider aperture. But
- 00:03:03,500 --> 00:03:05,420: maybe going back to the, the topic at hand,
- 00:03:05,760 --> 00:03:06,980: sove- sovereignty, right,
- 00:03:07,540 --> 00:03:09,700: can mean a lot of different things. Uh, we definitely hear it
- 00:03:09,800 --> 00:03:11,380: a lot more from our customers,
- 00:03:11,920 --> 00:03:13,520: uh, uh, especially here in Europe.
- 00:03:14,080 --> 00:03:16,200: Um, what-- When you hear
- 00:03:16,240 --> 00:03:18,020: that, when customers talk to you about, like, sovereignty,
- 00:03:18,460 --> 00:03:19,020: what do they mean?
- 00:03:19,800 --> 00:03:22,020: It's such a good question, and it is
- 00:03:22,080 --> 00:03:23,540: such a broad term,
- 00:03:23,900 --> 00:03:26,060: um, used to mean so many different things in
- 00:03:26,100 --> 00:03:27,060: our industry now.
- 00:03:27,700 --> 00:03:30,180: I think for Mistral specifically, because
- 00:03:30,220 --> 00:03:32,360: of the way, uh, our company started,
- 00:03:32,440 --> 00:03:33,420: which was three,
- 00:03:33,780 --> 00:03:35,780: um, three AI researchers
- 00:03:35,900 --> 00:03:36,960: in, in France,
- 00:03:37,220 --> 00:03:39,280: um, who wanted to basically
- 00:03:39,800 --> 00:03:40,960: give another option
- 00:03:41,240 --> 00:03:43,520: for foundational models, um, rather
- 00:03:43,560 --> 00:03:45,660: than just kind of having two, two options in the
- 00:03:45,720 --> 00:03:47,680: US to choose from. They wanted something that was
- 00:03:48,180 --> 00:03:49,000: Europe-bred.
- 00:03:49,320 --> 00:03:51,500: Uh, they wanted something that was compute
- 00:03:51,560 --> 00:03:54,400: in Europe, developed in Europe by European scientists
- 00:03:54,740 --> 00:03:56,280: that can be deployed in Europe,
- 00:03:56,540 --> 00:03:59,240: um, and, and fits the regulatory environment.
- 00:03:59,940 --> 00:04:01,940: Um, so the, the geography
- 00:04:02,060 --> 00:04:03,160: piece is a big
- 00:04:03,520 --> 00:04:05,700: part of the Mistral story for sovereignty.
- 00:04:06,320 --> 00:04:07,140: But, um,
- 00:04:08,060 --> 00:04:10,100: then you can take it one step further and look
- 00:04:10,140 --> 00:04:12,720: at the way that a company like Mistral allows
- 00:04:12,920 --> 00:04:14,980: the kind of keys of the system
- 00:04:15,020 --> 00:04:16,600: to be handed to the enterprise.
- 00:04:17,140 --> 00:04:19,399: Uh, and the way we do that is we allow
- 00:04:19,500 --> 00:04:21,959: for, um, an enterprise to decide how
- 00:04:22,000 --> 00:04:23,640: and where the model is deployed.
- 00:04:24,080 --> 00:04:25,560: So every single one of our models,
- 00:04:25,980 --> 00:04:28,060: audio, OCR included, but also our,
- 00:04:28,180 --> 00:04:30,800: our larger reasoning models, can be deployed on-prem
- 00:04:30,860 --> 00:04:31,840: in a private cloud.
- 00:04:32,140 --> 00:04:34,560: They can be accessed via the API, whatever fits
- 00:04:34,620 --> 00:04:35,660: the needs of,
- 00:04:35,960 --> 00:04:37,120: uh, of the enterprise,
- 00:04:37,460 --> 00:04:39,560: wherever you want your data to stay and you
- 00:04:39,600 --> 00:04:40,940: want your compute to stay.
- 00:04:41,700 --> 00:04:43,720: Um, and by doing that, we create
- 00:04:43,740 --> 00:04:45,760: an environment where the enterprise
- 00:04:45,820 --> 00:04:47,120: has control over,
- 00:04:47,680 --> 00:04:49,680: like I said, where the model is deployed,
- 00:04:50,020 --> 00:04:51,900: who has access to that model,
- 00:04:52,520 --> 00:04:53,060: governance,
- 00:04:54,040 --> 00:04:56,300: what models are used for what tasks.
- 00:04:56,720 --> 00:04:58,940: This is something that I think we're seeing more and more these
- 00:04:59,000 --> 00:05:00,120: days. Just using the
- 00:05:00,460 --> 00:05:02,840: biggest and best model is not always
- 00:05:02,880 --> 00:05:03,740: the right approach.
- 00:05:04,380 --> 00:05:06,620: Um, and when you, when you hand the keys
- 00:05:06,680 --> 00:05:08,840: to the enterprise, they can make that decision
- 00:05:08,880 --> 00:05:09,760: for themselves.
- 00:05:10,300 --> 00:05:12,560: What data the models have access to,
- 00:05:12,700 --> 00:05:14,600: the governance of the entire system,
- 00:05:14,960 --> 00:05:17,100: um, and, and cost. Cost especially as
- 00:05:17,140 --> 00:05:17,780: you're scaling.
- 00:05:18,360 --> 00:05:20,620: So, uh, giving, giving kind of the onus
- 00:05:20,760 --> 00:05:22,760: of that system back to the enterprise, that is
- 00:05:22,820 --> 00:05:24,240: how Mistral sees,
- 00:05:24,500 --> 00:05:25,440: sees sovereignty. Yeah.
- 00:05:25,760 --> 00:05:27,580: No, absolutely. And, and I can imagine-- I think the,
- 00:05:28,320 --> 00:05:30,400: the, the, the geographical component, I think it
- 00:05:30,440 --> 00:05:31,340: is super important.
- 00:05:31,660 --> 00:05:33,360: Uh, I think it is. I think
- 00:05:33,640 --> 00:05:34,480: we saw in the clouds,
- 00:05:34,780 --> 00:05:37,040: uh, in the clouds too many, I would argue. Uh,
- 00:05:37,580 --> 00:05:39,580: cl- hyperscalers be US only, and I think
- 00:05:39,620 --> 00:05:41,560: it, it really, really matters, so, uh, I really,
- 00:05:42,300 --> 00:05:44,500: uh, I really value, appreciate, like, Mi- Mistral really
- 00:05:44,580 --> 00:05:46,580: t- taking a stake in the ground to, to, to do
- 00:05:46,620 --> 00:05:47,580: that. Um,
- 00:05:48,060 --> 00:05:50,120: to, to your point as well with, with the models, right? Like,
- 00:05:50,420 --> 00:05:51,160: a while ago, it was all about
- 00:05:51,460 --> 00:05:53,740: having the best model for everything. That's quickly
- 00:05:53,780 --> 00:05:54,820: changing. Yes.
- 00:05:55,200 --> 00:05:57,380: I hear a lot about open weights models,
- 00:05:57,740 --> 00:06:00,040: specialized models, small models,
- 00:06:00,160 --> 00:06:01,680: fine-tuning models. Again, we,
- 00:06:01,940 --> 00:06:03,040: we, we, we hear a lot there.
- 00:06:03,680 --> 00:06:04,240: Does that kind of
- 00:06:04,620 --> 00:06:06,080: tie into the whole sovereignty
- 00:06:06,560 --> 00:06:08,320: discussion as well, and what are you
- 00:06:08,620 --> 00:06:10,640: seeing and hearing from, from customers or your strategy
- 00:06:10,660 --> 00:06:11,500: at Mistral to how to
- 00:06:11,960 --> 00:06:14,080: have the best model in the best way, in the best location,
- 00:06:14,140 --> 00:06:14,860: with the best control
- 00:06:15,240 --> 00:06:16,280: for the best use case?
- 00:06:17,088 --> 00:06:17,768: 100%.
- 00:06:18,068 --> 00:06:20,148: Yeah, I think if you think about kind of all the
- 00:06:20,228 --> 00:06:21,128: tools that are at
- 00:06:21,528 --> 00:06:23,668: your disposal right now in a toolbox,
- 00:06:24,488 --> 00:06:25,528: you're not going to,
- 00:06:25,928 --> 00:06:28,268: uh, take on a, a small task
- 00:06:28,288 --> 00:06:30,768: with a sledgehammer, and that's basically what
- 00:06:30,828 --> 00:06:32,988: taking the most powerful, largest
- 00:06:33,088 --> 00:06:34,768: reasoning model out there is doing.
- 00:06:35,308 --> 00:06:37,568: Um, and it's very easy to choose
- 00:06:37,628 --> 00:06:39,668: that model because it's performing the best on the
- 00:06:39,708 --> 00:06:40,848: benchmarks right now,
- 00:06:41,108 --> 00:06:43,128: um, or it's being spoken about the most.
- 00:06:43,588 --> 00:06:45,428: But, uh, when you're thinking about
- 00:06:45,688 --> 00:06:47,888: how AI is really deployed in these
- 00:06:47,948 --> 00:06:48,947: organizations,
- 00:06:49,828 --> 00:06:52,428: the, the workflows are much closer
- 00:06:52,508 --> 00:06:53,868: to kind of, um...
- 00:06:54,648 --> 00:06:56,908: The, the, they're-- the, the need for determinism
- 00:06:57,008 --> 00:06:59,048: is, is much greater, and they're much closer to kind
- 00:06:59,088 --> 00:06:59,968: of automation
- 00:07:00,288 --> 00:07:02,708: as they're spinning up AI in their organization.
- 00:07:03,348 --> 00:07:05,368: Uh, and in order to get there, you need
- 00:07:05,688 --> 00:07:07,908: either a specialized model or, as you said,
- 00:07:08,008 --> 00:07:09,088: a fine-tuned model.
- 00:07:09,588 --> 00:07:11,748: Um, and there are a variety of reasons why you would choose
- 00:07:11,808 --> 00:07:13,868: one or the other, but it allows you to
- 00:07:13,928 --> 00:07:15,988: have greater control over your system
- 00:07:16,428 --> 00:07:18,748: to go in with a more fine-tuned instrument
- 00:07:18,908 --> 00:07:20,308: for a specific task
- 00:07:20,788 --> 00:07:23,208: to achieve a greater level of determinism.
- 00:07:23,428 --> 00:07:25,488: Again, AI, non-deterministic, but
- 00:07:25,548 --> 00:07:27,488: you can-- there are levers you can pull
- 00:07:27,908 --> 00:07:30,468: to make that determinism a little more reliable.
- 00:07:30,988 --> 00:07:31,728: Um, and then
- 00:07:32,108 --> 00:07:34,188: to also have greater regulatory control.
- 00:07:34,288 --> 00:07:35,428: Mm. So for example,
- 00:07:35,708 --> 00:07:38,008: you know, if you have a transcription model running,
- 00:07:38,568 --> 00:07:39,308: um, uh,
- 00:07:39,708 --> 00:07:42,048: and you're choosing a cascaded audio agent
- 00:07:42,148 --> 00:07:44,268: system, which is a transcription model, an
- 00:07:44,368 --> 00:07:46,388: LLM, and then a text-to-speech model,
- 00:07:47,248 --> 00:07:48,728: that system is actually
- 00:07:48,988 --> 00:07:51,128: much easier for auditability and
- 00:07:51,168 --> 00:07:53,268: much easier for regulatory control than a
- 00:07:53,328 --> 00:07:54,848: singular speech-to-speech model.
- 00:07:55,428 --> 00:07:56,068: Um, so
- 00:07:56,588 --> 00:07:59,088: I, I think what we are trying to do with Mistral
- 00:07:59,288 --> 00:08:01,508: is provide the tools, provide
- 00:08:01,548 --> 00:08:03,728: them in an open source capacity or
- 00:08:03,908 --> 00:08:05,788: deliver on-prem capabilities,
- 00:08:06,208 --> 00:08:08,748: and then give the enterprise the choice
- 00:08:08,788 --> 00:08:09,748: to look at their data,
- 00:08:10,408 --> 00:08:11,488: look at the task,
- 00:08:11,888 --> 00:08:12,688: look at the needs,
- 00:08:13,048 --> 00:08:15,008: and choose the right model for the right task. Yep.
- 00:08:15,388 --> 00:08:16,408: No, makes sense. And
- 00:08:16,968 --> 00:08:18,308: control, to your point, comes from
- 00:08:18,888 --> 00:08:20,648: where you run it, how, how you run it.
- 00:08:21,308 --> 00:08:23,308: Um, but then within the model itself,
- 00:08:23,388 --> 00:08:25,448: there's, there's the training of the model, right?- Yes... the, the
- 00:08:25,508 --> 00:08:27,888: weights of the model. Like, what, what's your strategy
- 00:08:27,948 --> 00:08:29,328: there? Are, are you seeing
- 00:08:29,808 --> 00:08:31,268: customers that also want to truly own
- 00:08:31,868 --> 00:08:33,908: the way to training themselves? Do you work with those
- 00:08:33,948 --> 00:08:34,828: customers to, like--
- 00:08:35,168 --> 00:08:37,348: The ownership is not just the model where it runs, but also
- 00:08:37,388 --> 00:08:39,428: the, what's, how the model's actually trained and the weights
- 00:08:39,448 --> 00:08:40,168: of the models themselves.
- 00:08:40,508 --> 00:08:42,008: Hundred percent. Yeah. Hundred percent.
- 00:08:42,448 --> 00:08:43,208: So, um,
- 00:08:43,528 --> 00:08:45,748: and this is where, where fine-tuning really- Yeah... comes into
- 00:08:45,788 --> 00:08:48,148: play. Think there, there are a variety
- 00:08:48,168 --> 00:08:50,568: of reasons why you would want to fine-tune a model.
- 00:08:50,868 --> 00:08:53,128: Um, one could be something as simple
- 00:08:53,248 --> 00:08:54,768: as proprietary language.
- 00:08:55,128 --> 00:08:57,188: Um, it could also be as complex as you have
- 00:08:57,228 --> 00:08:59,248: a proprietary coding language that
- 00:08:59,308 --> 00:09:00,108: you use, and
- 00:09:00,528 --> 00:09:02,528: you want a coding model to be able to work
- 00:09:02,588 --> 00:09:04,628: in that language, but there's no published
- 00:09:04,648 --> 00:09:06,068: doc, so it would never show up
- 00:09:06,388 --> 00:09:08,468: in organic training data for a coding model.
- 00:09:09,168 --> 00:09:11,508: Um, this is a, a use case we've deployed
- 00:09:11,548 --> 00:09:12,128: recently.
- 00:09:12,428 --> 00:09:13,808: Uh, and, and the--
- 00:09:14,108 --> 00:09:16,108: there are, uh, you know, like I said, a
- 00:09:16,168 --> 00:09:17,848: variety of reasons why you would do this.
- 00:09:18,248 --> 00:09:18,968: You can achieve
- 00:09:19,228 --> 00:09:21,268: huge gains in accuracy
- 00:09:21,328 --> 00:09:23,508: and productivity by fine-tuning a model.
- 00:09:24,168 --> 00:09:26,168: But what I will say is that in order
- 00:09:26,228 --> 00:09:28,508: to get there, you have to have really good data. Yep.
- 00:09:28,628 --> 00:09:30,688: Um, and, and that is kind of the, the
- 00:09:30,768 --> 00:09:32,368: stumbling block that we see
- 00:09:32,688 --> 00:09:34,748: first, um, when, when we, we
- 00:09:34,788 --> 00:09:37,008: get a request to use, uh, our fine-tuning
- 00:09:37,048 --> 00:09:37,968: tool called Forge.
- 00:09:38,568 --> 00:09:40,568: Um, the, the first question we ask
- 00:09:40,628 --> 00:09:42,648: is- Where's your data?... where's your data? [laughs] Yeah.
- 00:09:43,308 --> 00:09:45,308: Yeah. Where's your data, and what does it look like?
- 00:09:45,568 --> 00:09:46,388: Because, uh,
- 00:09:46,648 --> 00:09:47,948: you know, you're-- you might be--
- 00:09:48,768 --> 00:09:50,948: you might, you might just be thinking about
- 00:09:50,988 --> 00:09:52,268: the quantity of data,
- 00:09:52,628 --> 00:09:54,708: um, but you might not be thinking about how clean or
- 00:09:54,748 --> 00:09:56,848: hy-hygienic that data is, and you're
- 00:09:56,908 --> 00:09:59,068: encoding that data and those, those
- 00:09:59,108 --> 00:10:01,168: learnings into a model- Right... that is then gonna
- 00:10:01,188 --> 00:10:02,388: run on your system. So,
- 00:10:02,948 --> 00:10:05,468: um, I would, would say the kind of Calibra layer
- 00:10:05,848 --> 00:10:07,868: is really important- Yep... when thinking about
- 00:10:07,968 --> 00:10:10,108: something-- approaching something like fine-tuning.
- 00:10:10,428 --> 00:10:10,988: Yeah, and there's a,
- 00:10:11,488 --> 00:10:13,588: there's a training/fine-tuning component of
- 00:10:13,608 --> 00:10:15,648: which data is absolutely critical, right?
- 00:10:15,728 --> 00:10:17,768: But there's also-- we talked a lot, uh, today about, like, the
- 00:10:17,808 --> 00:10:18,988: context at inference time.
- 00:10:19,488 --> 00:10:21,928: Yes. How do you actually instruct the agents to behave correctly?
- 00:10:21,988 --> 00:10:23,988: And there's training, there's data that
- 00:10:24,028 --> 00:10:24,588: you provide
- 00:10:25,008 --> 00:10:25,568: while training,
- 00:10:26,128 --> 00:10:27,908: uh, but there's training, there's data you provide
- 00:10:28,288 --> 00:10:30,428: during inference a-a-as context. And I think,
- 00:10:31,168 --> 00:10:32,868: um, sovereignty even there
- 00:10:33,468 --> 00:10:34,428: plays a role.
- 00:10:34,748 --> 00:10:36,768: I would argue if you think about your intellec-intellectual
- 00:10:36,847 --> 00:10:37,428: property or
- 00:10:37,907 --> 00:10:38,828: proprietary knowledge,
- 00:10:39,088 --> 00:10:40,488: you're providing that to the
- 00:10:40,908 --> 00:10:43,128: models, and depending on where those models run and
- 00:10:43,188 --> 00:10:44,148: who, who runs those models
- 00:10:44,688 --> 00:10:45,748: is being seen. And so
- 00:10:46,488 --> 00:10:47,508: how do you think about this kind of
- 00:10:47,888 --> 00:10:48,828: fine-tuning training
- 00:10:49,128 --> 00:10:51,068: side versus the kind of inference
- 00:10:51,368 --> 00:10:51,888: context
- 00:10:52,528 --> 00:10:54,548: and then sovereignty on that, on that side of
- 00:10:54,588 --> 00:10:56,688: the equation? I can give you a very
- 00:10:56,728 --> 00:10:58,568: easy answer to this one. But, um,
- 00:10:58,928 --> 00:11:00,968: because of the way that we build our models and because
- 00:11:01,028 --> 00:11:03,048: we, we allow both fine-tuning
- 00:11:03,148 --> 00:11:04,788: and, and the models to be run on-prem,
- 00:11:05,428 --> 00:11:07,568: if a, an enterprise wants their data
- 00:11:07,628 --> 00:11:09,728: to stay completely within a closed system,
- 00:11:10,228 --> 00:11:12,448: we can do that, and we can fine-tune a model within
- 00:11:12,508 --> 00:11:13,488: that closed system.
- 00:11:14,108 --> 00:11:16,608: Um, I think this is particularly
- 00:11:16,708 --> 00:11:18,728: relevant when you're moving beyond,
- 00:11:18,988 --> 00:11:21,488: uh, kind of industry or domain terminology
- 00:11:21,548 --> 00:11:23,668: and you're looking at, uh, proprietary
- 00:11:23,748 --> 00:11:25,608: knowledge that is your edge.
- 00:11:25,948 --> 00:11:27,988: Um, and it's- Yep... something that you mentioned in your keynote,
- 00:11:28,168 --> 00:11:30,048: uh, that our data
- 00:11:30,548 --> 00:11:31,768: is, is an advantage
- 00:11:32,128 --> 00:11:34,068: in this industry and, um,
- 00:11:34,508 --> 00:11:36,548: and what fine-tuning
- 00:11:36,608 --> 00:11:38,928: allows you to do is basically operationalize
- 00:11:39,088 --> 00:11:41,348: that advantage. Mm. Um, and with
- 00:11:41,388 --> 00:11:42,428: Mistral and, and
- 00:11:42,808 --> 00:11:45,048: with, with the way that we are built, you can keep
- 00:11:45,108 --> 00:11:46,788: that within your own system,
- 00:11:47,128 --> 00:11:48,468: maintain that advantage,
- 00:11:48,888 --> 00:11:49,808: but take advantage of it.
- 00:11:50,068 --> 00:11:51,888: Yep. A-any, any thought on,
- 00:11:52,668 --> 00:11:53,908: um, when
- 00:11:54,188 --> 00:11:56,288: fine-tuning makes more sense versus when
- 00:11:56,568 --> 00:11:58,708: just context or inference m-makes more
- 00:11:58,748 --> 00:12:00,884: sense? A tough one
- 00:12:01,044 --> 00:12:02,184: without, uh, without
- 00:12:02,484 --> 00:12:04,584: anchoring it in a, in a real use case.
- 00:12:04,624 --> 00:12:05,564: Sure, yeah. But, um,
- 00:12:06,924 --> 00:12:07,884: uh, yeah, I, I would say
- 00:12:08,244 --> 00:12:10,564: the, the jump to fine-tuning is not always
- 00:12:10,684 --> 00:12:11,244: the right one.
- 00:12:11,564 --> 00:12:13,364: Um, and there, there are ways that you can
- 00:12:13,904 --> 00:12:14,504: build a,
- 00:12:15,224 --> 00:12:17,424: a system around a more generalist
- 00:12:17,484 --> 00:12:19,804: model that allows you to take those wins
- 00:12:19,844 --> 00:12:22,124: that you might anticipate getting with a fine-tuned
- 00:12:22,184 --> 00:12:24,264: model. Um, but like I said,
- 00:12:24,344 --> 00:12:26,084: I think what, what is so powerful,
- 00:12:26,504 --> 00:12:28,804: uh... what is so powerful about fine-tuning
- 00:12:28,844 --> 00:12:30,924: and why I'm such a huge proponent of it and,
- 00:12:31,064 --> 00:12:32,724: and why I think it's kind of one of the next
- 00:12:33,184 --> 00:12:35,224: most exciting frontiers for AI,
- 00:12:35,804 --> 00:12:37,844: is that y- your data is something
- 00:12:37,884 --> 00:12:40,364: that you already are, are curating.
- 00:12:40,504 --> 00:12:42,844: Mm. Um, and if you're working with a company like Libra,
- 00:12:42,964 --> 00:12:45,284: you're curating that to a very high level.
- 00:12:45,804 --> 00:12:48,024: So being able to take that and apply it to
- 00:12:48,084 --> 00:12:50,024: the accuracy and operational,
- 00:12:50,404 --> 00:12:51,444: um, you know,
- 00:12:51,724 --> 00:12:53,524: success of an AI system,
- 00:12:54,204 --> 00:12:56,444: I'm, I'm a, I'm a fan. Yeah. [laughs] Uh,
- 00:12:56,964 --> 00:12:57,724: great. Um,
- 00:12:58,084 --> 00:12:59,584: maybe specifically for the audience here, which
- 00:12:59,864 --> 00:13:01,884: mostly data people, right? We, we, we
- 00:13:01,944 --> 00:13:04,204: care deeply about providing, building that best
- 00:13:04,263 --> 00:13:05,944: data foundation to make AI work.
- 00:13:06,704 --> 00:13:08,924: But at least in my experience, I often see the AI
- 00:13:09,004 --> 00:13:11,444: team could some- somewhat be disconnected
- 00:13:11,544 --> 00:13:12,564: from the, from the data
- 00:13:13,104 --> 00:13:15,184: team. Y- I'm assuming you, you work typically with
- 00:13:15,204 --> 00:13:15,964: the AI teams.
- 00:13:16,704 --> 00:13:17,904: Um, what
- 00:13:18,184 --> 00:13:19,164: can we do as
- 00:13:19,724 --> 00:13:21,764: calling ourselves data people, like, to help
- 00:13:21,824 --> 00:13:24,364: bring those team closer together? Any recommendations
- 00:13:24,424 --> 00:13:25,964: for, for the room of how to
- 00:13:26,664 --> 00:13:29,224: improve there? Yeah, it's a great question.
- 00:13:29,524 --> 00:13:30,484: I think, uh,
- 00:13:33,204 --> 00:13:35,664: I think there, there are a lot of ways
- 00:13:35,744 --> 00:13:36,404: that, uh,
- 00:13:36,664 --> 00:13:38,684: the people who are designing the AI systems
- 00:13:38,884 --> 00:13:40,904: are not thinking about,
- 00:13:41,204 --> 00:13:42,924: uh, data as,
- 00:13:43,184 --> 00:13:45,504: um, as
- 00:13:45,544 --> 00:13:47,584: an advantage. They're more thinking of it just
- 00:13:47,664 --> 00:13:48,644: as an input. Right.
- 00:13:49,104 --> 00:13:51,164: Um- The value, the intelligence
- 00:13:51,204 --> 00:13:53,564: in a model, not necessarily the data. Exactly.
- 00:13:53,604 --> 00:13:55,804: Yeah. Exactly. And the, the data is more a,
- 00:13:55,924 --> 00:13:58,044: a piece of the puzzle- Right... a building brick rather than
- 00:13:58,104 --> 00:14:00,144: something that can actually really advance the
- 00:14:00,184 --> 00:14:02,304: system or open up completely new
- 00:14:02,384 --> 00:14:03,204: opportunities.
- 00:14:03,764 --> 00:14:05,804: Um, I, I will take, again,
- 00:14:06,524 --> 00:14:08,664: my, my brain is, is on Voxoral. My brain
- 00:14:08,703 --> 00:14:10,824: is on audio. Um, so in the case
- 00:14:11,044 --> 00:14:11,824: of, uh,
- 00:14:12,084 --> 00:14:12,704: something like,
- 00:14:13,044 --> 00:14:14,364: um, a transcription model
- 00:14:14,784 --> 00:14:16,324: where you might have something like,
- 00:14:16,644 --> 00:14:18,704: uh, context biasing or keyword
- 00:14:18,784 --> 00:14:19,444: biasing,
- 00:14:20,004 --> 00:14:22,364: you have the ability to give a transcription
- 00:14:22,404 --> 00:14:24,444: model extra context. Uh, so
- 00:14:24,484 --> 00:14:26,524: it is a way to improve the overall
- 00:14:26,624 --> 00:14:27,824: accuracy of the model,
- 00:14:28,404 --> 00:14:30,684: um, to give it context on, on what it's listening
- 00:14:30,744 --> 00:14:32,204: to, whether that's key terms or just,
- 00:14:32,504 --> 00:14:34,564: you know, a Wikipedia page of, of a company.
- 00:14:35,184 --> 00:14:36,984: Uh, when,
- 00:14:37,304 --> 00:14:39,384: when the people who are building the AI systems are
- 00:14:39,424 --> 00:14:41,804: looking at that, they're looking at that context
- 00:14:42,184 --> 00:14:44,244: in, in the realm of a
- 00:14:44,344 --> 00:14:45,544: specific use case.
- 00:14:45,964 --> 00:14:48,004: What we would love is for
- 00:14:48,064 --> 00:14:50,104: a data team to look at that context and
- 00:14:50,164 --> 00:14:52,164: think, "What are all the possibilities here?"
- 00:14:52,204 --> 00:14:54,244: Right. "What could we put into context for this
- 00:14:54,284 --> 00:14:55,164: transcription model
- 00:14:55,604 --> 00:14:57,424: that might allow us to open up,"
- 00:14:57,704 --> 00:14:59,784: like I said, "a whole new world of audio
- 00:14:59,904 --> 00:15:00,644: use cases
- 00:15:00,964 --> 00:15:03,144: that were previously inaccessible because you couldn't
- 00:15:03,184 --> 00:15:04,544: get there with the accuracy?"
- 00:15:04,804 --> 00:15:05,664: Yeah. Um, so
- 00:15:06,044 --> 00:15:07,264: again, not just thinking about
- 00:15:07,664 --> 00:15:09,664: data as an input, but thinking about
- 00:15:09,704 --> 00:15:11,464: data as kind of a foundational
- 00:15:12,604 --> 00:15:14,184: avenue for, for new AI.
- 00:15:14,484 --> 00:15:15,384: Yeah, yeah, yeah. No,
- 00:15:15,824 --> 00:15:17,944: absolutely. And, and part of it is I'm, I'm assuming is all
- 00:15:18,444 --> 00:15:19,964: just language terminology.
- 00:15:20,624 --> 00:15:22,884: Uh, it's all, we're all people. It's all about communication,
- 00:15:23,164 --> 00:15:24,804: right? Uh, understanding each other's
- 00:15:25,304 --> 00:15:26,524: perspectives, and I think that, that
- 00:15:26,844 --> 00:15:28,164: could go a, a long way.
- 00:15:28,764 --> 00:15:30,984: Um, maybe on, on the, the governance
- 00:15:31,064 --> 00:15:32,744: side- Mm... right, of, of AI.
- 00:15:33,224 --> 00:15:33,944: We saw, uh,
- 00:15:34,284 --> 00:15:36,184: Weir Group and, and Teresa talk about our
- 00:15:36,524 --> 00:15:37,404: AI, AI command center.
- 00:15:38,264 --> 00:15:40,364: What are you seeing with, with the customers that you work with,
- 00:15:40,424 --> 00:15:41,124: some of the biggest
- 00:15:41,524 --> 00:15:44,104: bottlenecks from a governance compliance
- 00:15:44,204 --> 00:15:46,184: EU AI Act, which I think you're working deeply,
- 00:15:46,464 --> 00:15:47,164: deeply with, right?
- 00:15:47,684 --> 00:15:49,804: What are some of the, the, the bottlenecks you're seeing there,
- 00:15:49,864 --> 00:15:51,864: and, and are these real blockers, or
- 00:15:51,924 --> 00:15:53,024: do we find ways around them?
- 00:15:54,084 --> 00:15:56,204: Yeah. So the, um, regulatory
- 00:15:56,304 --> 00:15:58,484: piece is really interesting. Uh,
- 00:15:58,724 --> 00:16:00,864: the, the landscape in the EU
- 00:16:00,924 --> 00:16:03,104: and in the UK is so different than,
- 00:16:03,284 --> 00:16:04,644: than anywhere else in the world.
- 00:16:04,904 --> 00:16:07,004: Um, and I think it is very
- 00:16:07,084 --> 00:16:09,364: easy to sort of demonize regulation
- 00:16:09,424 --> 00:16:11,964: in this context. But, uh, we,
- 00:16:12,004 --> 00:16:14,284: we do see regulation
- 00:16:14,624 --> 00:16:16,624: as a potential
- 00:16:16,684 --> 00:16:17,204: enabler,
- 00:16:17,684 --> 00:16:20,084: um, for innovation and for AI adoption
- 00:16:20,624 --> 00:16:22,904: because what it allows is greater
- 00:16:22,984 --> 00:16:25,144: confidence on the part of the stakeholder
- 00:16:25,204 --> 00:16:26,884: that their data is being cared for,
- 00:16:27,184 --> 00:16:29,324: um, that you are adhering
- 00:16:29,384 --> 00:16:31,764: to rules that make their system safe
- 00:16:31,844 --> 00:16:34,164: and make the customer's use of their system safe.
- 00:16:34,884 --> 00:16:35,784: The challenge,
- 00:16:36,164 --> 00:16:38,464: um, the challenge-- couple of challenges.
- 00:16:38,544 --> 00:16:39,564: [laughs] One of them
- 00:16:40,224 --> 00:16:42,304: is understanding how
- 00:16:42,364 --> 00:16:44,464: each one of the geographies treats
- 00:16:44,504 --> 00:16:45,864: that regulation differently-
- 00:16:46,124 --> 00:16:48,464: Mm... um, especially when data needs to cross-
- 00:16:48,704 --> 00:16:49,544: Right... geographies.
- 00:16:49,984 --> 00:16:51,344: That can get quite complicated.
- 00:16:52,064 --> 00:16:52,684: Another one
- 00:16:52,944 --> 00:16:53,744: is, uh,
- 00:16:54,004 --> 00:16:54,964: a cohesive
- 00:16:55,264 --> 00:16:56,464: regulatory body.
- 00:16:56,744 --> 00:16:58,804: Um, so for example, GDPR, there's
- 00:16:58,844 --> 00:17:01,164: not just one interpretation. I think there's 27
- 00:17:01,204 --> 00:17:02,684: interpretations of GDPR,
- 00:17:03,084 --> 00:17:05,204: and that gets complicated when you're looking at it from
- 00:17:05,304 --> 00:17:07,704: a adherence point of view. Yeah.
- 00:17:07,724 --> 00:17:08,724: And then the last one
- 00:17:08,984 --> 00:17:11,144: is, um, predictability in these
- 00:17:11,184 --> 00:17:12,204: systems. So
- 00:17:12,544 --> 00:17:14,924: if you have a predictable regulatory
- 00:17:15,064 --> 00:17:17,364: system, there isn't a fear on the behalf
- 00:17:17,404 --> 00:17:19,844: of the enterprise that they might need to kind of scramble
- 00:17:19,904 --> 00:17:21,964: and, and find a way to,
- 00:17:22,124 --> 00:17:22,904: to, um,
- 00:17:23,424 --> 00:17:24,484: become compliant.
- 00:17:25,124 --> 00:17:27,204: Uh, when you don't have
- 00:17:27,244 --> 00:17:29,264: that predictability, that fear might actually
- 00:17:29,304 --> 00:17:30,144: block adoption.
- 00:17:30,404 --> 00:17:32,444: Um, and it becomes very complicated when
- 00:17:32,464 --> 00:17:34,584: you're looking specifically on the data perspective
- 00:17:34,664 --> 00:17:36,784: because it might change where that data needs to be
- 00:17:36,824 --> 00:17:37,344: housed,
- 00:17:37,744 --> 00:17:39,544: where the compute happens, where,
- 00:17:39,824 --> 00:17:41,904: where the, the AI system sits
- 00:17:41,944 --> 00:17:43,424: from a geographic perspective,
- 00:17:43,984 --> 00:17:46,004: um, or where it sits in the cloud.
- 00:17:46,264 --> 00:17:46,964: Uh, so I think
- 00:17:47,304 --> 00:17:49,584: if we can get to a point where the regulation
- 00:17:49,704 --> 00:17:51,944: is clear, cohesive, and predictable,
- 00:17:52,524 --> 00:17:53,888: um- That the
- 00:17:54,168 --> 00:17:56,208: overall confidence, both on the behalf of- Right... the
- 00:17:56,268 --> 00:17:58,388: model builders and the enterprises raises
- 00:17:58,448 --> 00:17:59,068: and, and we,
- 00:17:59,328 --> 00:18:01,168: we see adoption explode. Yeah.
- 00:18:01,648 --> 00:18:02,528: I, I, I agree. I think
- 00:18:02,828 --> 00:18:03,908: done well, it could be a
- 00:18:04,228 --> 00:18:06,268: force of good. Huge. Done poorly,
- 00:18:06,388 --> 00:18:08,708: could, uh, explode complexity-
- 00:18:08,708 --> 00:18:10,708: [laughs]... pretty dramatically. And, and you brought
- 00:18:10,748 --> 00:18:11,728: all these dimensions.
- 00:18:12,088 --> 00:18:13,088: Yes. Interpretation,
- 00:18:13,368 --> 00:18:14,248: national, uh,
- 00:18:14,688 --> 00:18:15,348: differences,
- 00:18:15,968 --> 00:18:17,988: uh, lack of predictability. I think these are
- 00:18:18,028 --> 00:18:19,988: real challenges. So, so let's hope we
- 00:18:20,588 --> 00:18:22,668: find the right way forward, and, uh- I
- 00:18:22,708 --> 00:18:23,868: have faith. Yeah, exactly.
- 00:18:24,128 --> 00:18:26,168: Um, wanna be mindful of time. It's been a
- 00:18:26,188 --> 00:18:26,808: long day already.
- 00:18:27,148 --> 00:18:29,168: Um, although I'd love to continue the chat. But
- 00:18:29,208 --> 00:18:31,168: maybe to close things off, any
- 00:18:31,708 --> 00:18:33,708: recommendation here for the team? You,
- 00:18:33,888 --> 00:18:35,448: you live in this world every day,
- 00:18:36,048 --> 00:18:36,948: very, very deep.
- 00:18:37,588 --> 00:18:39,628: Um, what do you want the team here
- 00:18:39,688 --> 00:18:41,568: or the, the people here to take away that
- 00:18:41,988 --> 00:18:44,088: could be valuable? I think,
- 00:18:44,268 --> 00:18:46,308: uh, maybe, maybe two things
- 00:18:46,328 --> 00:18:47,268: that I've, I've said,
- 00:18:47,988 --> 00:18:49,868: um, already. But one,
- 00:18:50,568 --> 00:18:52,928: choosing the right model for the right task is,
- 00:18:53,068 --> 00:18:55,248: I think, the easiest way to make a win
- 00:18:55,288 --> 00:18:55,868: when you're,
- 00:18:56,148 --> 00:18:58,168: um, when you're implementing AI in a new
- 00:18:58,248 --> 00:19:00,308: system. Uh, and the
- 00:19:00,408 --> 00:19:00,988: second,
- 00:19:01,348 --> 00:19:03,388: um, with, with data
- 00:19:03,688 --> 00:19:05,008: and, uh,
- 00:19:05,708 --> 00:19:06,628: data governance,
- 00:19:08,928 --> 00:19:10,568: when you're thinking about AI,
- 00:19:10,988 --> 00:19:13,568: uh, in a transformative way for an enterprise,
- 00:19:13,688 --> 00:19:15,768: what is often the
- 00:19:15,828 --> 00:19:17,148: last thought is,
- 00:19:17,608 --> 00:19:18,508: is the data piece.
- 00:19:19,168 --> 00:19:20,088: Um, and
- 00:19:20,448 --> 00:19:22,448: I would encourage everyone here
- 00:19:22,528 --> 00:19:24,108: to, to think about, um,
- 00:19:24,368 --> 00:19:26,388: data and, and its role
- 00:19:26,468 --> 00:19:29,008: in this AI transformation, that that ownership
- 00:19:29,468 --> 00:19:31,608: being, uh, transformative for,
- 00:19:31,728 --> 00:19:33,408: for the amount of, of
- 00:19:34,388 --> 00:19:36,568: benefit, the, the win, the accuracy,
- 00:19:36,968 --> 00:19:38,908: the KPIs, the, the ROI,
- 00:19:39,688 --> 00:19:42,188: that, that data piece is likely
- 00:19:42,248 --> 00:19:44,408: what I've seen, likely the biggest mover.
- 00:19:44,808 --> 00:19:46,848: Um, so take ownership early and
- 00:19:46,968 --> 00:19:49,168: often in the implementation of AI,
- 00:19:49,288 --> 00:19:51,268: um, and, and choose the right model. Great.
- 00:19:51,788 --> 00:19:53,788: Well, I- I'll take that as a take away. Data
- 00:19:53,828 --> 00:19:55,288: is the biggest mover for AI.
- 00:19:55,688 --> 00:19:57,748: All yours. Let's close, let's close with that. Uh, thank you so
- 00:19:57,788 --> 00:19:59,808: much, Kayla, to join us here and, uh, for the
- 00:19:59,848 --> 00:20:00,708: insights that you provided.
- 00:20:00,968 --> 00:20:08,548: Absolutely. Thank you.
- 00:20:08,548 --> 00:20:08,968: [audience applauding]
Sovereignty spotlight
Sovereignty is now a top priority for enterprise AI, covering both where models run and who controls them. Felix Van de Maele, Collibra co-founder and CEO, and Kayla Nahi, Product Lead at Mistral AI, discuss what European organizations are asking for, how regulation shapes their choices and why many are customizing open-weight models.
Video: Ontology spotlight
Transcript
- 00:00:07,620 --> 00:00:08,420: When we opened today,
- 00:00:08,680 --> 00:00:09,520: I asked, uh,
- 00:00:09,820 --> 00:00:12,060: which of the five definitions of customer
- 00:00:12,580 --> 00:00:13,740: is the real definition.
- 00:00:15,100 --> 00:00:17,420: Everything you've heard about context relies
- 00:00:17,480 --> 00:00:19,280: on that question being answered.
- 00:00:20,220 --> 00:00:22,300: Now, this is not a question that can be
- 00:00:22,400 --> 00:00:23,820: answered by a large
- 00:00:24,120 --> 00:00:24,820: LLM model.
- 00:00:25,500 --> 00:00:27,300: This truly relies on
- 00:00:27,880 --> 00:00:29,760: figuring out what things actually mean,
- 00:00:30,020 --> 00:00:32,119: and are they written down so that both
- 00:00:32,180 --> 00:00:33,720: humans and machines
- 00:00:34,300 --> 00:00:36,200: can refer to and access the same answer.
- 00:00:37,660 --> 00:00:39,180: This is what we mean by ontology.
- 00:00:39,440 --> 00:00:40,860: And ontology, as we heard,
- 00:00:41,240 --> 00:00:43,340: used to be something that only specialized team-
- 00:00:43,600 --> 00:00:45,760: teams did, but now it's becoming
- 00:00:46,160 --> 00:00:46,720: mainstream.
- 00:00:47,440 --> 00:00:49,100: It is truly the AI voca--
- 00:00:49,380 --> 00:00:51,580: the shared vocabulary that your AI agents
- 00:00:51,660 --> 00:00:53,820: need. So we will now
- 00:00:53,880 --> 00:00:55,400: have a discussion with, uh,
- 00:00:56,460 --> 00:00:58,580: a, a few special guests. I'd like
- 00:00:58,640 --> 00:01:00,840: to welcome Jessica Talisman, CEO
- 00:01:00,880 --> 00:01:01,700: of Contextually.
- 00:01:02,500 --> 00:01:04,819: Jessica has been doing this, uh, building
- 00:01:04,860 --> 00:01:06,920: ontologies with, uh, various enterprises
- 00:01:06,960 --> 00:01:08,640: for a very long time. Welcome, Jessica.
- 00:01:08,980 --> 00:01:11,680: Thank you. And, um [audience applauding]
- 00:01:11,740 --> 00:01:12,580: yes, round of applause.
- 00:01:13,060 --> 00:01:14,900: And I'd also like to welcome Steve Borsus,
- 00:01:15,320 --> 00:01:17,400: a Senior Consultant of Business Architecture
- 00:01:17,440 --> 00:01:18,160: at Vanguard.
- 00:01:18,600 --> 00:01:19,620: Welcome, Steve. Thank you.
- 00:01:21,660 --> 00:01:22,600: Is that a seat? Yeah.
- 00:01:24,260 --> 00:01:25,940: All right. So Steve, I know,
- 00:01:26,220 --> 00:01:28,460: um, Vanguard has been working on,
- 00:01:28,560 --> 00:01:30,600: uh, the ontology project, and,
- 00:01:30,680 --> 00:01:32,420: uh, you're, you're leading that. So
- 00:01:33,740 --> 00:01:35,820: tell us a little bit about why is this a good
- 00:01:35,880 --> 00:01:37,920: time for Vanguard to, uh,
- 00:01:38,060 --> 00:01:40,300: invest and work on the ontology project.
- 00:01:41,240 --> 00:01:42,180: Yeah. Thanks, Ron.
- 00:01:43,420 --> 00:01:45,540: We've been on a data product journey
- 00:01:45,580 --> 00:01:46,340: for quite a while,
- 00:01:47,040 --> 00:01:48,040: and, um,
- 00:01:48,960 --> 00:01:51,040: one of the things that hasn't come along quite
- 00:01:51,100 --> 00:01:51,780: as fast,
- 00:01:52,600 --> 00:01:54,760: uh, is the meaning behind that data.
- 00:01:54,980 --> 00:01:57,140: So with the advent of, uh,
- 00:01:57,560 --> 00:01:59,820: GenAI being introduced and
- 00:01:59,900 --> 00:02:01,480: so prevalent in all of our workflows,
- 00:02:01,880 --> 00:02:04,180: it becomes really important, especially in a regulated
- 00:02:04,240 --> 00:02:06,360: industry, to, to nail down
- 00:02:06,400 --> 00:02:08,400: the meaning and provide
- 00:02:08,479 --> 00:02:10,139: grounding for these agents.
- 00:02:10,800 --> 00:02:13,040: So this isn't so much a data problem
- 00:02:13,140 --> 00:02:15,560: we're trying to solve, it's a knowledge management problem.
- 00:02:17,720 --> 00:02:18,600: Jessica, um,
- 00:02:19,060 --> 00:02:21,280: indeed, as Steve is saying, it's a knowledge management
- 00:02:21,320 --> 00:02:23,500: problem. Uh, what do you see across
- 00:02:23,560 --> 00:02:25,560: regulated industries when you, when you work in
- 00:02:25,600 --> 00:02:27,600: this area? When did well-defined meaning go
- 00:02:27,660 --> 00:02:29,880: from like something that's ni- nice to have
- 00:02:29,940 --> 00:02:31,840: to now being urgent? And, and
- 00:02:32,900 --> 00:02:33,640: w- when does it break?
- 00:02:34,060 --> 00:02:36,080: Oh, gosh, I'm so excited that we're at this
- 00:02:36,120 --> 00:02:37,480: moment because I was working on
- 00:02:37,800 --> 00:02:40,060: the context and ontology problem well before
- 00:02:40,200 --> 00:02:41,860: AI. So, um,
- 00:02:42,680 --> 00:02:45,160: y- you know, when we approach things as a data-only
- 00:02:45,200 --> 00:02:47,220: problem, we run into or c-
- 00:02:47,280 --> 00:02:49,320: repeat the bad habits from the past.
- 00:02:49,840 --> 00:02:52,360: Um, we can't impose or superimpose
- 00:02:52,580 --> 00:02:55,200: data governance and data management
- 00:02:55,580 --> 00:02:56,420: simply on top of
- 00:02:56,720 --> 00:02:57,920: knowledge. Context is,
- 00:02:58,320 --> 00:03:00,420: is a little bit of a, a different beast. And
- 00:03:00,500 --> 00:03:02,540: so within a regulated industry, it's
- 00:03:02,600 --> 00:03:04,940: really important to have a really good handle
- 00:03:05,040 --> 00:03:06,700: on, on where knowledge lives.
- 00:03:07,220 --> 00:03:09,360: I hear a lot of people, especially today, talking
- 00:03:09,420 --> 00:03:10,660: about the ownership problem,
- 00:03:11,080 --> 00:03:13,260: but it's a shared responsibility, and it's really
- 00:03:13,320 --> 00:03:15,580: about, about building a muscle. It's about
- 00:03:15,620 --> 00:03:17,480: building that muscle and understanding,
- 00:03:18,120 --> 00:03:18,760: um, how,
- 00:03:19,080 --> 00:03:21,080: how to build systems that are trustworthy,
- 00:03:21,220 --> 00:03:23,920: but also that have things like provenance,
- 00:03:24,080 --> 00:03:26,200: that have-- I mean, all of us are, are
- 00:03:26,240 --> 00:03:28,340: familiar who've worked in regulated industries
- 00:03:28,380 --> 00:03:29,580: with data classification.
- 00:03:29,920 --> 00:03:31,160: The same goes for knowledge.
- 00:03:31,620 --> 00:03:34,080: Um, ontologies and taxonomies can support
- 00:03:34,520 --> 00:03:37,600: really good and strong data classification
- 00:03:37,680 --> 00:03:38,620: to help with, um,
- 00:03:39,060 --> 00:03:41,040: uh, provenance, but also with, with
- 00:03:41,640 --> 00:03:42,420: things like, uh,
- 00:03:42,700 --> 00:03:43,380: you know, how we can,
- 00:03:43,780 --> 00:03:46,020: we can lend trust not only to AI systems,
- 00:03:46,120 --> 00:03:47,160: but also for humans,
- 00:03:47,620 --> 00:03:50,280: so... Makes
- 00:03:50,300 --> 00:03:51,440: sense. Yeah. Steve, uh,
- 00:03:51,820 --> 00:03:53,840: earlier you mentioned, you know, when we were chatting
- 00:03:53,900 --> 00:03:55,380: earlier, you, you spoke about
- 00:03:55,860 --> 00:03:57,940: one of-- for you, success looks like truly
- 00:03:57,980 --> 00:04:00,000: having that business meaning figured out at,
- 00:04:00,100 --> 00:04:00,760: at Vanguard.
- 00:04:01,660 --> 00:04:02,360: How do you
- 00:04:02,920 --> 00:04:05,380: go from ontology not being just a scientific
- 00:04:05,460 --> 00:04:06,420: project to
- 00:04:06,840 --> 00:04:09,000: it demonstrating real business value with
- 00:04:09,100 --> 00:04:09,620: urgency?
- 00:04:10,120 --> 00:04:12,180: Uh, how, and how did you get buy-in internally for that?
- 00:04:13,800 --> 00:04:15,340: Yeah. So we,
- 00:04:17,000 --> 00:04:18,440: we start with business outcomes.
- 00:04:19,120 --> 00:04:20,720: Uh, we don't start with ontologies,
- 00:04:21,000 --> 00:04:21,920: or taxonomies,
- 00:04:22,260 --> 00:04:23,680: or knowledge graphs.
- 00:04:24,080 --> 00:04:26,200: We're, we're, we're here to solve business problems
- 00:04:26,760 --> 00:04:29,020: for people in financial services.
- 00:04:29,400 --> 00:04:31,660: So everything nets back to
- 00:04:32,140 --> 00:04:34,020: how are you delivering value for the business.
- 00:04:34,680 --> 00:04:36,260: So let me tell you a little bit about our use case.
- 00:04:38,140 --> 00:04:39,360: We create
- 00:04:39,900 --> 00:04:41,940: financial service products, investment
- 00:04:42,000 --> 00:04:44,400: products. These products
- 00:04:44,460 --> 00:04:45,760: cover different asset types,
- 00:04:46,420 --> 00:04:47,980: uh, they're different types of products,
- 00:04:48,880 --> 00:04:50,400: uh, and they have different legal structures.
- 00:04:51,500 --> 00:04:53,480: So starting with a data foundation,
- 00:04:53,900 --> 00:04:55,960: we started to define what do we mean
- 00:04:56,180 --> 00:04:58,400: when we say we have an asset class
- 00:04:58,480 --> 00:05:00,260: for equity or fixed income?
- 00:05:01,040 --> 00:05:03,440: What do we mean when we say we have a product
- 00:05:03,520 --> 00:05:04,320: of an ETF
- 00:05:04,900 --> 00:05:06,980: or a target retirement fund? Like, what, what
- 00:05:07,020 --> 00:05:08,920: does the business definition
- 00:05:09,500 --> 00:05:10,560: that we can agree on
- 00:05:11,720 --> 00:05:13,860: and we can share across the enterprise so that we can
- 00:05:13,900 --> 00:05:15,120: define this one place
- 00:05:15,680 --> 00:05:17,760: and then share that? Because once
- 00:05:17,840 --> 00:05:19,960: people agree on shared definitions,
- 00:05:21,340 --> 00:05:23,560: then we can start to change the conversation
- 00:05:24,640 --> 00:05:25,340: to more,
- 00:05:25,700 --> 00:05:27,040: more of a value-centered
- 00:05:27,660 --> 00:05:29,040: conversation rather than,
- 00:05:29,440 --> 00:05:31,480: um, a, a governance conversation.
- 00:05:33,920 --> 00:05:36,140: So let's now go a little bit deeper on how it actually
- 00:05:36,180 --> 00:05:38,240: works. And Jessica- Yeah... you've been working
- 00:05:38,260 --> 00:05:39,700: with Vanguard on this project.
- 00:05:41,004 --> 00:05:42,484: Tell me a little bit about, you know,
- 00:05:43,204 --> 00:05:45,264: when you start working on a project like this,
- 00:05:45,544 --> 00:05:48,304: how do you factor in the existing,
- 00:05:48,904 --> 00:05:51,024: uh, data landscape, you know?
- 00:05:51,144 --> 00:05:53,364: Um, what if a company already has a mess?
- 00:05:53,464 --> 00:05:55,564: Are you encoding now that mess and- Oh, gosh...
- 00:05:55,764 --> 00:05:57,804: and, and, and putting that into ontology? What
- 00:05:57,864 --> 00:05:59,884: has to truly happen before
- 00:06:00,004 --> 00:06:01,184: you start modeling anything?
- 00:06:01,544 --> 00:06:02,184: Oh, gosh. So,
- 00:06:02,544 --> 00:06:04,744: so the one thing to dispel is
- 00:06:04,784 --> 00:06:06,824: we're not trying to have one definition to
- 00:06:06,884 --> 00:06:08,584: rule them all for the whole business.
- 00:06:09,104 --> 00:06:11,564: It's actually starting with the human factor
- 00:06:11,664 --> 00:06:13,644: of, of having these conversations
- 00:06:14,004 --> 00:06:16,204: and having people define their own
- 00:06:16,284 --> 00:06:16,984: local meaning.
- 00:06:17,484 --> 00:06:19,604: What we build is based o-based off of
- 00:06:19,704 --> 00:06:21,424: RDF, so you've probably heard that,
- 00:06:21,804 --> 00:06:23,764: um, passed around before. But what we're,
- 00:06:24,124 --> 00:06:26,724: what we're trying to do is we're trying to structure definitions,
- 00:06:27,304 --> 00:06:29,384: concepts throughout the organization,
- 00:06:29,524 --> 00:06:31,444: so the meaning of concepts
- 00:06:31,884 --> 00:06:34,464: and ac- and accounting for acronyms,
- 00:06:34,964 --> 00:06:36,604: for, um, synonyms.
- 00:06:36,884 --> 00:06:38,964: So we're actually building these rich
- 00:06:39,044 --> 00:06:41,704: vocabularies throughout the organization
- 00:06:41,784 --> 00:06:44,184: that we can then network and connect.
- 00:06:44,244 --> 00:06:45,204: We actually map,
- 00:06:45,884 --> 00:06:48,024: um, those words. So it's really about
- 00:06:48,124 --> 00:06:49,024: gaining trust,
- 00:06:49,524 --> 00:06:51,644: giving people ownership, so it's not
- 00:06:51,864 --> 00:06:53,004: one owner, but that
- 00:06:53,424 --> 00:06:54,924: multiple teams can own,
- 00:06:55,464 --> 00:06:57,524: um, the process. And then there's tool
- 00:06:57,584 --> 00:06:58,324: availability.
- 00:06:58,584 --> 00:07:00,784: That's the other thing, is making sure that everyone
- 00:07:00,844 --> 00:07:01,984: has, um,
- 00:07:02,464 --> 00:07:05,124: has access to the tools to be successful
- 00:07:05,224 --> 00:07:07,224: in this activity or this effort.
- 00:07:07,844 --> 00:07:10,084: Um, and then we build shared governance
- 00:07:10,164 --> 00:07:12,264: out of that, um, because we all have shared under-
- 00:07:12,444 --> 00:07:14,964: understanding and ownership. And it may sound kumbaya,
- 00:07:15,464 --> 00:07:17,504: but if you build a program, not a
- 00:07:17,564 --> 00:07:19,564: project, not a product locally,
- 00:07:19,644 --> 00:07:21,864: but you build it as a sustained program
- 00:07:21,924 --> 00:07:22,604: with governance,
- 00:07:23,164 --> 00:07:25,184: um, then you can reach a really high level of
- 00:07:25,264 --> 00:07:27,284: trust, which is essential for
- 00:07:27,344 --> 00:07:28,804: building this type of system.
- 00:07:29,504 --> 00:07:31,504: Got it. Yeah. Let's go back to the customer
- 00:07:31,564 --> 00:07:33,584: example. [laughs] Yes. I'm in marketing. Yes. I
- 00:07:33,624 --> 00:07:35,664: think of a customer, for example, somebody who's
- 00:07:35,724 --> 00:07:37,724: tried a product. Yep. The sales team thinks
- 00:07:37,784 --> 00:07:39,204: of a customer who's already paying.
- 00:07:39,784 --> 00:07:42,144: So let's take this example to,
- 00:07:42,224 --> 00:07:44,244: uh, how you're solving the ontology problem.
- 00:07:44,364 --> 00:07:46,844: What happens when two teams disagree on a definition?
- 00:07:46,924 --> 00:07:47,884: How do you break the tie?
- 00:07:48,504 --> 00:07:50,584: Yeah. Well, the reality is they both could be
- 00:07:50,664 --> 00:07:52,484: right within their own context.
- 00:07:52,824 --> 00:07:54,164: Right. So the
- 00:07:55,004 --> 00:07:57,344: definition of revenue inside
- 00:07:57,404 --> 00:07:59,444: marketing may differ than the definition of
- 00:07:59,504 --> 00:08:01,284: revenue inside, um,
- 00:08:02,324 --> 00:08:02,944: uh, finance,
- 00:08:04,004 --> 00:08:06,144: and those contexts are really important.
- 00:08:06,384 --> 00:08:08,424: So it's important not so much to break a
- 00:08:08,484 --> 00:08:10,784: tie, but to capture the meaning
- 00:08:11,364 --> 00:08:11,884: inside
- 00:08:12,424 --> 00:08:13,384: the, the business
- 00:08:14,044 --> 00:08:14,844: vocabulary,
- 00:08:16,104 --> 00:08:16,844: provide scope,
- 00:08:17,144 --> 00:08:19,424: provide examples, provide relationships
- 00:08:19,924 --> 00:08:22,264: so that humans and AI can understand
- 00:08:22,964 --> 00:08:24,744: the data that you're, you're talking about.
- 00:08:27,144 --> 00:08:29,044: Jessica, now let's talk a little bit about,
- 00:08:29,304 --> 00:08:30,984: um, how do you
- 00:08:31,964 --> 00:08:33,364: build an ontology? Is it
- 00:08:33,844 --> 00:08:35,784: human-led or an agent-led?
- 00:08:36,404 --> 00:08:38,404: Can LLMs- Human first. Human first?
- 00:08:38,484 --> 00:08:40,724: Human first. Yeah. Tell me more. Why? Um- Why cannot
- 00:08:40,764 --> 00:08:41,424: an agent help
- 00:08:41,704 --> 00:08:43,284: bui- uh, help me build an ontology?
- 00:08:43,584 --> 00:08:46,284: Uh, first of all, agents are exposed
- 00:08:46,364 --> 00:08:48,404: to the vast amount of training data
- 00:08:48,484 --> 00:08:50,544: that they, you know, so they come already
- 00:08:51,024 --> 00:08:52,164: imbued with knowledge.
- 00:08:52,644 --> 00:08:55,124: And so very likely, depending on your industry,
- 00:08:55,404 --> 00:08:57,464: um, what's going-- the agent's gonna
- 00:08:57,504 --> 00:08:59,784: construct a reality based on its
- 00:08:59,844 --> 00:09:00,644: training data.
- 00:09:01,184 --> 00:09:03,544: And so that same accuracy
- 00:09:03,584 --> 00:09:05,804: that you struggle with just using agents without
- 00:09:05,864 --> 00:09:08,004: any context, the same kind of
- 00:09:08,104 --> 00:09:10,264: struggle exists in having an agent
- 00:09:10,304 --> 00:09:11,504: build your ontology.
- 00:09:12,124 --> 00:09:14,384: We see this with, like, open standard ontologies,
- 00:09:14,444 --> 00:09:16,844: that there's openly available and published ontologies.
- 00:09:16,904 --> 00:09:18,124: An example at Vanguard
- 00:09:18,744 --> 00:09:20,964: is, um, the FIBO ontology, which
- 00:09:21,044 --> 00:09:23,424: was flashed on the screen earlier, which is the Financial
- 00:09:23,504 --> 00:09:26,004: Industry Business Ontology. It's standardized,
- 00:09:26,064 --> 00:09:26,664: it's open.
- 00:09:27,044 --> 00:09:29,564: Investment vehicle is one of the top classes.
- 00:09:29,964 --> 00:09:32,084: No one at Vanguard calls it investment
- 00:09:32,124 --> 00:09:34,344: vehicle. It's called investment product. So
- 00:09:34,924 --> 00:09:37,264: the, you know, the, the nuance of the language
- 00:09:37,304 --> 00:09:39,364: is important. The nuance of the definitions,
- 00:09:39,564 --> 00:09:41,584: how we structure our realities
- 00:09:41,664 --> 00:09:43,664: is like a digital thumbprint of
- 00:09:43,704 --> 00:09:44,824: your organization.
- 00:09:45,424 --> 00:09:47,564: Um, it's very hard to replicate or
- 00:09:47,644 --> 00:09:49,884: to buy an out-of-the-box reality
- 00:09:49,944 --> 00:09:51,724: that perfectly represents,
- 00:09:52,164 --> 00:09:54,324: you know, your organization or teams or
- 00:09:54,404 --> 00:09:56,604: your, your business. So it's really
- 00:09:56,644 --> 00:09:58,863: important that that's a, that's a control plane
- 00:09:58,923 --> 00:10:00,964: for your business. That's how you're able
- 00:10:01,024 --> 00:10:02,584: to really architect meaningful con-
- 00:10:02,923 --> 00:10:03,584: context.
- 00:10:03,904 --> 00:10:06,144: Right. Yeah. So the way I understand is that a company's
- 00:10:06,184 --> 00:10:08,204: knowledge is tru-truly unique. It is. And-
- 00:10:08,204 --> 00:10:10,344: I mean, it's your, it's your, it's your, I don't wanna say
- 00:10:10,424 --> 00:10:12,784: moat, but it's- Right. Yeah. It's important.
- 00:10:12,804 --> 00:10:15,004: And, and that's why you- Yeah... you need truly
- 00:10:15,044 --> 00:10:15,584: to have
- 00:10:16,004 --> 00:10:18,084: h-humans as part of the loop to- Absolutely... to
- 00:10:18,104 --> 00:10:20,124: construct it. And, and you can. You
- 00:10:20,164 --> 00:10:22,404: know, the beauty is once you have that starter
- 00:10:22,464 --> 00:10:23,124: ontology,
- 00:10:23,584 --> 00:10:25,924: because ontologies are a form of symbolic
- 00:10:26,004 --> 00:10:28,304: AI, is that the combination with generative
- 00:10:28,524 --> 00:10:30,844: AI means neurosymbolic AI. So
- 00:10:31,304 --> 00:10:33,484: you actually enable an, a lot more
- 00:10:33,524 --> 00:10:35,644: by taking that initial step in defining
- 00:10:35,684 --> 00:10:37,604: your business, your business ontology,
- 00:10:38,084 --> 00:10:40,304: because then you can trust agents
- 00:10:40,364 --> 00:10:42,864: or AI to continue to help to expand
- 00:10:42,924 --> 00:10:43,684: and flesh out
- 00:10:44,344 --> 00:10:46,384: more, more of your ontology, the deeper
- 00:10:47,044 --> 00:10:49,284: elements. So at Collibra,
- 00:10:49,404 --> 00:10:51,544: we truly believe in open standards. Steve- Yes... could you tell
- 00:10:51,564 --> 00:10:54,604: us a little bit about RDF and W3C
- 00:10:54,684 --> 00:10:56,784: over proprietary formats, and, um,
- 00:10:57,124 --> 00:10:59,124: what was your thinking there for choosing those for
- 00:10:59,164 --> 00:10:59,984: building the ontology?
- 00:11:00,684 --> 00:11:03,304: Yeah. The really important part here is interoperability,
- 00:11:03,784 --> 00:11:04,344: right? So
- 00:11:05,444 --> 00:11:07,404: business meaning is gonna survive
- 00:11:07,764 --> 00:11:09,844: platforms and tools over time.
- 00:11:10,524 --> 00:11:12,624: It's really important that we capture that in a way that
- 00:11:12,664 --> 00:11:14,684: can be moved from platform
- 00:11:14,724 --> 00:11:15,384: to platform.
- 00:11:16,424 --> 00:11:17,604: RDF's been around a long time.
- 00:11:18,484 --> 00:11:20,584: Uh, the, it supports the open
- 00:11:20,644 --> 00:11:23,044: web, and the,
- 00:11:23,084 --> 00:11:25,084: the importance of us capturing that
- 00:11:25,184 --> 00:11:27,204: in a way that can be moved throughout
- 00:11:27,284 --> 00:11:28,664: time is really important,
- 00:11:28,984 --> 00:11:31,504: and also verified and governed within the same
- 00:11:31,924 --> 00:11:33,564: f- uh, the same framework.
- 00:11:33,904 --> 00:11:34,924: RDF provides a nice
- 00:11:35,404 --> 00:11:36,004: container,
- 00:11:36,664 --> 00:11:38,784: uh, the, the framework to be able to add
- 00:11:38,844 --> 00:11:39,564: governance in,
- 00:11:40,804 --> 00:11:41,424: add meaning,
- 00:11:42,184 --> 00:11:44,524: add structure and relationships
- 00:11:45,084 --> 00:11:47,304: all inside an application profile that's
- 00:11:47,404 --> 00:11:49,584: really a great tool for a business.
- 00:11:49,844 --> 00:11:52,084: And that's the key, is application profile.
- 00:11:52,424 --> 00:11:53,544: Yes. Yeah.
- 00:11:55,004 --> 00:11:57,124: Yeah. Great. So to close,
- 00:11:57,244 --> 00:11:58,384: I'd love to ask you,
- 00:11:58,704 --> 00:12:00,904: uh, like one thing
- 00:12:00,984 --> 00:12:02,124: our audience can do,
- 00:12:02,564 --> 00:12:04,564: uh, and what they can take away. So Jessica,
- 00:12:04,804 --> 00:12:07,104: for the team who's starting from zero
- 00:12:07,164 --> 00:12:09,244: to build ontologies, what is one thing they
- 00:12:09,264 --> 00:12:09,864: could start doing?
- 00:12:10,284 --> 00:12:12,144: Oh, gosh. Start with definitions,
- 00:12:12,444 --> 00:12:13,844: um, and small.
- 00:12:14,284 --> 00:12:15,804: So it's scoped small.
- 00:12:16,184 --> 00:12:18,264: You know, if like anything else you want to, um,
- 00:12:18,464 --> 00:12:20,584: organize or have some sort of MVP
- 00:12:20,644 --> 00:12:21,624: or a POC,
- 00:12:22,324 --> 00:12:24,364: um, so you can get used to the workflow
- 00:12:24,444 --> 00:12:26,744: and understand the sort of o-orders
- 00:12:26,784 --> 00:12:28,864: of operation. It's not the same as data
- 00:12:28,944 --> 00:12:30,964: again. Dealing with knowledge, um,
- 00:12:31,104 --> 00:12:33,164: you know, as we saw before through all
- 00:12:33,244 --> 00:12:35,524: of the, uh, product launches with Collibra,
- 00:12:36,064 --> 00:12:38,064: is that context and knowledge,
- 00:12:38,284 --> 00:12:40,764: um, are nuanced. It's not just your database.
- 00:12:40,844 --> 00:12:42,664: You're gonna have to look in your SharePoint and,
- 00:12:43,204 --> 00:12:45,404: and, um, in your document stores and,
- 00:12:45,484 --> 00:12:47,604: and in your workflows. So it's throughout
- 00:12:47,664 --> 00:12:49,664: your entire organization, not just in your
- 00:12:49,704 --> 00:12:50,524: database. So,
- 00:12:51,124 --> 00:12:53,404: um, really give yourself grace
- 00:12:53,484 --> 00:12:55,484: in that and, and understanding
- 00:12:55,604 --> 00:12:57,784: and, and have that first effort
- 00:12:57,824 --> 00:12:59,664: really be about discovery and testing.
- 00:13:00,444 --> 00:13:02,704: Yeah. Great. And Steve, for the team that already
- 00:13:02,764 --> 00:13:04,324: has data disciplines in place?
- 00:13:05,284 --> 00:13:07,384: Yeah. I would suggest pick a business concept that
- 00:13:07,444 --> 00:13:09,524: matters, define it,
- 00:13:10,304 --> 00:13:12,284: scope it, provide examples,
- 00:13:12,764 --> 00:13:13,524: write it down,
- 00:13:14,124 --> 00:13:15,344: get feedback on it,
- 00:13:16,084 --> 00:13:18,604: and if you can describe that to a human and they understand,
- 00:13:19,284 --> 00:13:21,344: you'll be able to let an agent
- 00:13:21,444 --> 00:13:22,204: understand- Yes...
- 00:13:23,444 --> 00:13:25,544: the same. Great. Well, thank you both. Thank you.
- 00:13:25,584 --> 00:13:26,764: This was truly a pleasure.
- 00:13:27,324 --> 00:13:29,964: Thank you. Yeah. Thank you. [audience applauding]
- 00:13:30,024 --> 00:13:30,244: All right.
Ontology spotlight
In this fireside chat, Steve Brosius of Vanguard and ontologist Jessica Talisman join Collibra CMO Chirag Dhull to talk about what it takes to build a knowledge graph and ontology inside a large regulated firm: why Vanguard anchored this work to a use case with clear business outcomes before scaling the foundation, how definitions get settled when smart teams disagree, and where language models help with this work and where they stop.
Video: From standards to control: Securing the agentic enterprise
Transcript
- 00:00:07,200 --> 00:00:09,140: Hi, David. Hey. Nice to have you here. I'm Anna.
- 00:00:09,700 --> 00:00:12,120: I'm gonna improvise this because of Theresa's
- 00:00:12,200 --> 00:00:14,120: poor voice. Uh, I thought I'm just gonna
- 00:00:14,380 --> 00:00:16,400: jump in. For everybody that doesn't know
- 00:00:16,440 --> 00:00:18,500: me, I'm the co-founder of Traill, the company
- 00:00:18,600 --> 00:00:21,000: Collibra just has acquired and announced yesterday
- 00:00:21,500 --> 00:00:23,560: for AI governance, so I think this is pretty fitting.
- 00:00:24,300 --> 00:00:26,300: So very good to be, uh, to meet you, and thanks for
- 00:00:26,340 --> 00:00:27,720: being here, David, with us. Thank you for having me.
- 00:00:28,360 --> 00:00:30,420: Um, so maybe for everybody in the audience,
- 00:00:30,640 --> 00:00:32,800: um, can you explain to us what the AI
- 00:00:32,860 --> 00:00:35,220: you see is, and why did an underwriting
- 00:00:35,260 --> 00:00:37,640: company end up writing the standard for AI agents?
- 00:00:37,960 --> 00:00:38,480: Yeah, absolutely.
- 00:00:38,760 --> 00:00:40,760: So I think AIC was born from
- 00:00:40,820 --> 00:00:42,860: the realization that you have quite a big gap
- 00:00:42,920 --> 00:00:45,000: between people in San Francisco
- 00:00:45,040 --> 00:00:46,480: and Silicon Valley explaining to you that
- 00:00:46,760 --> 00:00:48,860: AI is taking over, that we'll be out of a job in ten
- 00:00:48,900 --> 00:00:50,000: years, and
- 00:00:50,260 --> 00:00:52,380: the reality of AI adoption within
- 00:00:52,440 --> 00:00:52,980: enterprises.
- 00:00:53,500 --> 00:00:55,780: I think as we've seen today, there's multiple presentation
- 00:00:55,860 --> 00:00:57,920: around this gap being
- 00:00:57,980 --> 00:01:00,140: not linked to the capabilities of those agents,
- 00:01:00,320 --> 00:01:01,060: but more to
- 00:01:01,660 --> 00:01:03,840: how can you actually trust them, how can you actually trust,
- 00:01:04,500 --> 00:01:06,840: uh, they're reliable, that they're safe, that they're producing
- 00:01:06,880 --> 00:01:07,660: the right information.
- 00:01:08,540 --> 00:01:10,820: And our whole thesis at AUC is that
- 00:01:10,900 --> 00:01:13,060: if you look into the history of disruptive
- 00:01:13,100 --> 00:01:13,640: technologies,
- 00:01:14,660 --> 00:01:16,960: you have actually had a consistent answer
- 00:01:17,000 --> 00:01:19,320: from the market on how to secure those technologies,
- 00:01:19,380 --> 00:01:21,080: which has always been a combination of
- 00:01:21,580 --> 00:01:23,700: a shared set of rules on how those technologies should work,
- 00:01:23,860 --> 00:01:24,400: so a standard;
- 00:01:25,420 --> 00:01:27,460: audits, so the capacity to verify that those
- 00:01:27,500 --> 00:01:28,620: rules have been applied properly;
- 00:01:29,180 --> 00:01:30,780: and insurance, which is a way to make sure
- 00:01:31,320 --> 00:01:33,660: someone has skin in the game and uphold the bar and the standard.
- 00:01:33,980 --> 00:01:35,620: And so our objective is really to replicate
- 00:01:36,040 --> 00:01:38,220: this kind of trust infrastructure for AI
- 00:01:38,280 --> 00:01:40,360: specifically, starting with a, with a standard
- 00:01:40,400 --> 00:01:41,280: which is AIC One.
- 00:01:42,060 --> 00:01:43,460: Very nice. Makes a lot of sense.
- 00:01:44,140 --> 00:01:46,240: Um, what would you say changes about risk when
- 00:01:46,320 --> 00:01:48,720: AI stops answering questions
- 00:01:48,780 --> 00:01:51,000: and starts taking action? So what we see
- 00:01:51,060 --> 00:01:52,300: right now, the shift, right, from
- 00:01:52,680 --> 00:01:54,840: everybody moving from their chatbot to
- 00:01:54,880 --> 00:01:57,080: really deploying an agent fleet,
- 00:01:57,200 --> 00:01:58,140: as Theresa just showed.
- 00:01:58,540 --> 00:01:59,120: Yeah. I think there's
- 00:01:59,700 --> 00:02:01,700: two main factors which change the nature
- 00:02:01,740 --> 00:02:03,880: of risk kind of fundamentally. The first one
- 00:02:04,000 --> 00:02:06,140: is the potential impact of the risk
- 00:02:06,180 --> 00:02:08,440: or the blast radius. If an agent can write
- 00:02:08,520 --> 00:02:09,820: into systems, actually
- 00:02:10,100 --> 00:02:11,440: failures can propagate and
- 00:02:11,760 --> 00:02:13,300: take a much more concrete and, and,
- 00:02:13,860 --> 00:02:16,020: you know, bigger impact than, than it can sort
- 00:02:16,020 --> 00:02:16,860: of read on the agent.
- 00:02:17,540 --> 00:02:18,720: And the second one is how
- 00:02:19,240 --> 00:02:21,320: hard it becomes to detect those failures.
- 00:02:21,500 --> 00:02:23,560: So you're seeing chain of actions and
- 00:02:23,620 --> 00:02:24,300: chain of events,
- 00:02:24,840 --> 00:02:26,220: and it's very hard to disentangle
- 00:02:26,780 --> 00:02:28,880: where is the failure coming from, at which stage
- 00:02:28,920 --> 00:02:30,180: in the workflow did this come from.
- 00:02:30,580 --> 00:02:32,740: And obviously, those two things compound together.
- 00:02:33,200 --> 00:02:35,280: So autonomous agent with those
- 00:02:35,340 --> 00:02:36,040: two qualities
- 00:02:36,400 --> 00:02:38,540: represent kind of a new type of risk profile
- 00:02:38,580 --> 00:02:39,680: that needs to be managed differently.
- 00:02:40,060 --> 00:02:42,060: Yeah, totally agree. And I think, uh, the whole area
- 00:02:42,140 --> 00:02:44,400: around, like, AI insurance is gonna be
- 00:02:44,480 --> 00:02:46,160: one of-- a very important one.
- 00:02:46,940 --> 00:02:49,000: You know that, uh, at Collibra, we're not just
- 00:02:49,080 --> 00:02:51,100: a, a, a company helping other,
- 00:02:51,280 --> 00:02:53,360: uh, companies- Yeah... govern the AI. We also take it
- 00:02:53,400 --> 00:02:56,120: very seriously ourselves as pract-pr-practitioners,
- 00:02:56,200 --> 00:02:58,440: right? So we actually also went, uh,
- 00:02:58,560 --> 00:03:00,700: through the assessment, uh, with you guys.
- 00:03:00,780 --> 00:03:02,780: So maybe explain to the audience
- 00:03:02,820 --> 00:03:03,700: what gets tested,
- 00:03:04,000 --> 00:03:05,820: who verifies it- Yeah... like, who's the auditor,
- 00:03:06,280 --> 00:03:08,800: and what did Maestro, as we all learned, the Collibra,
- 00:03:09,000 --> 00:03:10,660: uh, agent, uh, go through.
- 00:03:10,980 --> 00:03:12,980: And also maybe in that process, what
- 00:03:13,040 --> 00:03:13,680: surprised you
- 00:03:14,060 --> 00:03:15,360: in doing it? Yeah. So
- 00:03:16,520 --> 00:03:18,660: we basically verify three areas of AI
- 00:03:18,700 --> 00:03:20,240: risks. The first one is how
- 00:03:21,120 --> 00:03:23,400: humans interact with AI systems. So do you have the right policies
- 00:03:23,420 --> 00:03:25,500: in place to mitigate a potential failure of your
- 00:03:25,600 --> 00:03:27,380: agents? Are you equipped to deal with that?
- 00:03:28,020 --> 00:03:30,180: The second one is, how is your AI agent
- 00:03:30,240 --> 00:03:32,380: built? So really looking into the code,
- 00:03:32,520 --> 00:03:33,760: is Maestros, guardrails,
- 00:03:34,060 --> 00:03:36,060: are they configured properly? Is the filtering applied where
- 00:03:36,080 --> 00:03:38,020: it should be? And the last one is,
- 00:03:38,400 --> 00:03:40,180: because those systems are probabilistic by nature,
- 00:03:40,780 --> 00:03:41,500: you wanna make sure that
- 00:03:41,780 --> 00:03:44,220: you can build them in a certain way, and it's still gonna behave
- 00:03:44,280 --> 00:03:45,840: with some kind of variability in the outcome.
- 00:03:46,260 --> 00:03:47,720: So we designed a complete program
- 00:03:48,300 --> 00:03:50,440: of evaluations that map the risk and attacks that
- 00:03:50,480 --> 00:03:52,540: Maestro is exposed to in a normal production
- 00:03:52,580 --> 00:03:54,820: context, run them, grade them,
- 00:03:55,000 --> 00:03:56,280: and this way kind of quantified
- 00:03:56,740 --> 00:03:58,400: the areas that Maestro was exposed to.
- 00:03:59,040 --> 00:04:01,440: You know, where did it fail? What did it take for Maestro
- 00:04:01,480 --> 00:04:03,520: to fail? How well did it perform under this
- 00:04:03,580 --> 00:04:04,340: evaluation program?
- 00:04:04,960 --> 00:04:05,580: This was done,
- 00:04:05,920 --> 00:04:07,400: uh, in collaboration with Schellman,
- 00:04:07,700 --> 00:04:09,480: uh, who's the first auditor accredited,
- 00:04:09,760 --> 00:04:11,280: uh, against the AIC One, uh,
- 00:04:11,860 --> 00:04:12,720: the AIC One framework.
- 00:04:14,040 --> 00:04:15,500: I think there are c-
- 00:04:15,880 --> 00:04:18,200: probably a couple of things that surprised us on the
- 00:04:18,260 --> 00:04:20,459: Maestro agent itself. The first thing is it's,
- 00:04:20,540 --> 00:04:22,120: um, it's a sophisticated agent.
- 00:04:22,420 --> 00:04:24,800: Uh- Good to hear. It's a complicated agent.
- 00:04:25,420 --> 00:04:27,500: The risk and attack profile are
- 00:04:27,580 --> 00:04:29,700: different from, you know, a standard kind of customer
- 00:04:29,740 --> 00:04:30,460: support chatbot.
- 00:04:31,040 --> 00:04:33,420: Um, so the, the evaluation program itself
- 00:04:33,480 --> 00:04:35,500: that we kind of designed and validated with you
- 00:04:35,540 --> 00:04:36,680: guys was interesting to build.
- 00:04:37,600 --> 00:04:39,660: The second thing that was surprising is actually held
- 00:04:39,720 --> 00:04:41,800: up pretty well under the most
- 00:04:41,880 --> 00:04:44,240: adversarial attacks that we ran against Maestro.
- 00:04:44,360 --> 00:04:45,060: Uh, and
- 00:04:45,400 --> 00:04:47,020: the guardrail actually worked really well.
- 00:04:47,360 --> 00:04:49,180: Uh, it was not easy to manipulate,
- 00:04:49,600 --> 00:04:51,300: and it actually held pretty well,
- 00:04:51,660 --> 00:04:52,840: uh, its ability to do
- 00:04:53,500 --> 00:04:55,580: the right tool calls, so it did not really engage in
- 00:04:55,620 --> 00:04:56,540: unauthorized actions.
- 00:04:57,240 --> 00:04:59,360: The thing where we've seen it behave is
- 00:04:59,400 --> 00:05:00,060: really around,
- 00:05:00,360 --> 00:05:02,400: um... and that's very classic, but
- 00:05:02,420 --> 00:05:04,600: like the ability to kind of overstate its capabilities
- 00:05:04,640 --> 00:05:06,680: and say, "Okay, I actually have access to
- 00:05:06,720 --> 00:05:08,500: this data that it might not have access to,"
- 00:05:09,000 --> 00:05:09,600: or just say,
- 00:05:10,200 --> 00:05:12,480: "Look, I've looked into a hundred thirty-five
- 00:05:12,540 --> 00:05:14,400: documents," when it actually only looked into
- 00:05:14,720 --> 00:05:17,360: sixty documents or something like that. But it never re-resulted
- 00:05:17,400 --> 00:05:19,440: in a, a kind of critical failure, I
- 00:05:19,480 --> 00:05:21,620: would say. I mean, we all know that Claude moment
- 00:05:21,660 --> 00:05:23,780: when Claude tells us- Yeah... "Oh, good catch." And I'm like,
- 00:05:23,840 --> 00:05:26,020: "I don't wanna catch you." Yeah. [laughs] "I want you to be right."
- 00:05:26,140 --> 00:05:28,480: They want to please. That's for sure. [laughs]
- 00:05:28,520 --> 00:05:29,880: Very, very interesting. Um,
- 00:05:30,240 --> 00:05:32,340: yeah, I guess it turns out well that, uh,
- 00:05:32,460 --> 00:05:33,800: a governance company takes that go-
- 00:05:34,060 --> 00:05:36,080: takes, uh, governance very seriously, and I
- 00:05:36,140 --> 00:05:38,180: think there's gonna be huge market opportunity,
- 00:05:38,780 --> 00:05:40,980: um, for the standard for the assurance
- 00:05:41,020 --> 00:05:42,260: and the insurance industry.
- 00:05:42,600 --> 00:05:44,780: But I think as well what we at Collibra do
- 00:05:44,860 --> 00:05:46,840: is help our clients to
- 00:05:47,200 --> 00:05:49,200: get access to the standard, right- Yeah... and perform it
- 00:05:49,260 --> 00:05:49,940: in the platform.
- 00:05:50,280 --> 00:05:52,780: And I think this is also speaking from somebody
- 00:05:52,840 --> 00:05:54,940: who has dealt with different standards in
- 00:05:54,980 --> 00:05:55,840: their life as well,
- 00:05:56,380 --> 00:05:58,860: is a very important thing to make standards accessible
- 00:05:58,920 --> 00:06:00,000: and actually operational,
- 00:06:00,620 --> 00:06:02,640: operationalable because otherwise it's just
- 00:06:02,700 --> 00:06:03,960: a checkbox exercise
- 00:06:04,360 --> 00:06:06,760: and not something that's actually gonna be lived in practice.
- 00:06:06,920 --> 00:06:08,180: So great to have you here.
- 00:06:08,780 --> 00:06:10,450: Um- No?
- 00:06:11,930 --> 00:06:12,610: There's-- Okay.
- 00:06:12,950 --> 00:06:15,770: I'm just, uh... Keep, keep the questions coming. [laughs]
- 00:06:15,850 --> 00:06:17,889: Um, so maybe next question is: A certificate
- 00:06:17,930 --> 00:06:20,110: is a point in time. That's what I just said, right? Yeah. Like, it's
- 00:06:20,150 --> 00:06:22,250: like a snapshot of, okay, now you have
- 00:06:22,330 --> 00:06:22,850: everything
- 00:06:23,570 --> 00:06:24,110: figured out.
- 00:06:24,850 --> 00:06:27,670: Um, can
- 00:06:27,690 --> 00:06:28,990: we go to the question before?
- 00:06:29,990 --> 00:06:31,310: Yeah. So how do you
- 00:06:32,050 --> 00:06:34,370: turn a standard into a day-to-day
- 00:06:34,450 --> 00:06:36,550: oversight? So basically maybe something that we also
- 00:06:36,950 --> 00:06:39,110: touched upon with the whole runtime enforcement theme, right?
- 00:06:39,170 --> 00:06:41,090: Like, how do we make sure that what, what
- 00:06:41,450 --> 00:06:42,810: you now assured
- 00:06:43,190 --> 00:06:45,190: that Kalipa and Master is doing
- 00:06:45,270 --> 00:06:46,530: is actually also
- 00:06:47,010 --> 00:06:48,670: being kept at such a place?
- 00:06:49,230 --> 00:06:50,170: Yeah. I think that's where the
- 00:06:50,930 --> 00:06:53,010: kind of handshake or the interface between
- 00:06:53,130 --> 00:06:55,310: us and a solution like yours actually happens.
- 00:06:55,370 --> 00:06:57,630: So we see our role as defining
- 00:06:57,670 --> 00:06:59,690: the standard. So we'll define the categories of risks
- 00:06:59,710 --> 00:07:01,790: that needs to be reported against. We'll define how to
- 00:07:01,850 --> 00:07:03,990: satisfy those requirements, what evidence can
- 00:07:04,030 --> 00:07:04,590: be accepted.
- 00:07:05,270 --> 00:07:07,750: Uh, but of course, as you said, there is a need to verify
- 00:07:07,810 --> 00:07:09,890: that the security posture for
- 00:07:09,930 --> 00:07:11,970: the agent is maintained over time.
- 00:07:12,210 --> 00:07:13,610: Um, you have development teams who
- 00:07:13,990 --> 00:07:15,450: continue working on those agents.
- 00:07:15,810 --> 00:07:17,070: Risks evolve all the time. So
- 00:07:17,770 --> 00:07:19,990: the ability to kind of continuously monitor
- 00:07:20,490 --> 00:07:21,350: compliance- Mm-hmm...
- 00:07:21,710 --> 00:07:22,950: is extremely complementary to
- 00:07:23,250 --> 00:07:25,490: just frame and the requirement that need to be met.
- 00:07:26,050 --> 00:07:28,050: Um, one other way we catch
- 00:07:28,090 --> 00:07:30,150: this is we'll actually rerun our technical
- 00:07:30,210 --> 00:07:32,190: evaluation every quarter to make sure- Uh-huh...
- 00:07:33,010 --> 00:07:34,630: the model is, uh, or the
- 00:07:35,130 --> 00:07:35,830: product itself,
- 00:07:36,190 --> 00:07:38,490: uh, we catch any evolution of the product and that the underlying
- 00:07:38,530 --> 00:07:39,130: foundation models
- 00:07:39,550 --> 00:07:41,110: doesn't bear any additional risk. So I think your,
- 00:07:41,750 --> 00:07:43,090: yours is coming up in December
- 00:07:43,630 --> 00:07:45,730: sixteenth, so we'll see- All right... how you guys held up then.
- 00:07:46,330 --> 00:07:48,350: I mean, that's, uh, actually something I think about a lot when
- 00:07:48,430 --> 00:07:49,010: I think about
- 00:07:49,290 --> 00:07:51,550: the next step of like AI insurance as well, right?
- 00:07:51,590 --> 00:07:52,730: And then, and all of these like
- 00:07:53,370 --> 00:07:55,610: standards, they do need information
- 00:07:55,670 --> 00:07:56,930: that provides
- 00:07:57,370 --> 00:07:58,830: that a guardrail or like, uh,
- 00:07:59,370 --> 00:08:01,790: an operational measure is actually maintained
- 00:08:01,850 --> 00:08:02,410: over time.
- 00:08:02,850 --> 00:08:04,670: Yeah. So this will be the
- 00:08:05,310 --> 00:08:07,330: source providing this information not just to
- 00:08:07,390 --> 00:08:09,190: an assurance companies, but also
- 00:08:09,610 --> 00:08:11,690: insurance companies. Yeah. You can think of it as
- 00:08:11,750 --> 00:08:13,890: the kind of loss data- Yeah... that insurance
- 00:08:13,930 --> 00:08:16,090: companies need to be able to price and quantify
- 00:08:16,170 --> 00:08:17,190: AI risk. And the main,
- 00:08:18,070 --> 00:08:19,230: the main theory behind this is
- 00:08:19,570 --> 00:08:21,590: you can build your agents with all of the guardrails
- 00:08:21,630 --> 00:08:23,490: in the world. You'll still have residual risk
- 00:08:23,750 --> 00:08:25,770: because of the probabilistic nature of those agents, which
- 00:08:25,830 --> 00:08:27,510: actually you want. That's where you get benefit from.
- 00:08:27,990 --> 00:08:30,170: So you need to be able to quantify it, understand
- 00:08:30,830 --> 00:08:32,910: how it happens, how often does it happen, what does it
- 00:08:32,970 --> 00:08:34,070: take for this risk to happen,
- 00:08:34,550 --> 00:08:36,770: and this is what insurance companies will use to price the premium
- 00:08:36,830 --> 00:08:38,910: accordingly. So basically what that means, we will
- 00:08:38,970 --> 00:08:41,050: still have standards. We have agent-specific
- 00:08:41,110 --> 00:08:42,470: standards, but we basically
- 00:08:42,910 --> 00:08:44,990: reduce the time between the audits because I know
- 00:08:45,050 --> 00:08:47,150: for the standard audits, like a year or something,
- 00:08:47,210 --> 00:08:48,530: right, where you always have to re-audit.
- 00:08:48,850 --> 00:08:50,930: But also we need to make sure that we
- 00:08:50,990 --> 00:08:53,050: monitor it in production over time,
- 00:08:53,170 --> 00:08:54,290: so that we can actually see,
- 00:08:54,890 --> 00:08:57,110: um, we can catch up with the dynamic
- 00:08:57,150 --> 00:08:58,270: nature of AI, let's say.
- 00:08:58,550 --> 00:08:59,770: Absolutely. Cool.
- 00:09:00,470 --> 00:09:02,830: So if I am a CDO or CA--
- 00:09:03,270 --> 00:09:05,650: CIAO with agents going
- 00:09:05,730 --> 00:09:07,770: GA, um, what should I
- 00:09:07,830 --> 00:09:09,370: do differently starting Monday?
- 00:09:10,870 --> 00:09:13,350: Um, I think I have an unpleasant recommendation,
- 00:09:13,430 --> 00:09:14,110: but basically
- 00:09:14,790 --> 00:09:17,070: spend at least twenty-five percent of your time
- 00:09:17,110 --> 00:09:18,950: or resources thinking about incidents.
- 00:09:19,590 --> 00:09:21,370: Um, what incidents can happen?
- 00:09:21,930 --> 00:09:23,630: How likely are those incidents to happen?
- 00:09:24,250 --> 00:09:26,350: What does it take for this incident to happen? It's very
- 00:09:26,390 --> 00:09:27,330: different if it's, you know,
- 00:09:27,590 --> 00:09:28,190: a simple
- 00:09:28,630 --> 00:09:30,790: ask from a common user can generate an incident
- 00:09:30,890 --> 00:09:33,030: versus- Mm-hmm... if you need someone to actively
- 00:09:33,130 --> 00:09:35,010: and perform against sophisticated attacks.
- 00:09:35,750 --> 00:09:38,130: Um, what guardrail or what mitigation procedures
- 00:09:38,150 --> 00:09:39,150: do you have in place today?
- 00:09:39,730 --> 00:09:41,190: What do you need to be able to be more
- 00:09:41,650 --> 00:09:42,750: confident in this? But
- 00:09:43,310 --> 00:09:45,370: I think the difference in the
- 00:09:45,410 --> 00:09:47,430: allocation of resources and time between kind of
- 00:09:47,470 --> 00:09:49,570: building and securing today is probably a hundred to
- 00:09:49,670 --> 00:09:51,270: one. I think this should be closer to
- 00:09:51,570 --> 00:09:52,870: four to one, three to one, ideally.
- 00:09:53,130 --> 00:09:55,230: And good thing is building gets easier, right? So I
- 00:09:55,270 --> 00:09:57,410: think we can spend more time about really
- 00:09:57,470 --> 00:09:59,730: understanding, okay, what it is that we,
- 00:09:59,770 --> 00:10:02,010: that we can, can face and in terms of risks,
- 00:10:02,370 --> 00:10:04,590: and actually ties in pretty well to the agent contracts
- 00:10:04,610 --> 00:10:05,550: that researchers showed,
- 00:10:05,890 --> 00:10:08,550: where it actually makes the users think about
- 00:10:08,590 --> 00:10:10,670: what are the natural language rules that
- 00:10:10,710 --> 00:10:12,010: we wanna give this agent.
- 00:10:12,330 --> 00:10:14,370: And then the enforcement is another piece, right? But
- 00:10:14,410 --> 00:10:15,930: this is really like tying into the,
- 00:10:16,550 --> 00:10:18,790: the, the activity of thinking
- 00:10:18,870 --> 00:10:21,050: about what risks can actually emerge because
- 00:10:21,150 --> 00:10:23,350: I do think it needs to come from the person that
- 00:10:23,410 --> 00:10:25,690: developed the agent, like the business user, because
- 00:10:25,730 --> 00:10:27,990: they understand the context in which it's operating.
- 00:10:28,390 --> 00:10:30,309: But those are also n-not, not often,
- 00:10:30,830 --> 00:10:31,410: uh, the most,
- 00:10:32,110 --> 00:10:33,950: uh, risk management, uh,
- 00:10:34,250 --> 00:10:34,810: uh, people.
- 00:10:35,650 --> 00:10:37,830: All right. So we learned that we have
- 00:10:37,930 --> 00:10:38,830: to, um,
- 00:10:39,350 --> 00:10:41,130: get recertified every quarter,
- 00:10:41,430 --> 00:10:41,950: and also
- 00:10:42,430 --> 00:10:44,530: the standard changes every quarter. It's the,
- 00:10:44,550 --> 00:10:46,650: the speed of AI, right? The tagline of today's
- 00:10:46,710 --> 00:10:48,930: event. So how is it stable
- 00:10:48,970 --> 00:10:50,370: enough to audit yet, and, uh,
- 00:10:50,630 --> 00:10:53,070: yet fast enough to keep up, up with the attackers?
- 00:10:54,130 --> 00:10:56,330: I mean, technically, basically, the standard evolves
- 00:10:56,390 --> 00:10:58,410: every quarter. So we'll source incidents
- 00:10:58,470 --> 00:11:00,650: from just the real-life use of AI,
- 00:11:00,830 --> 00:11:03,130: best practices observed by our consortium,
- 00:11:03,190 --> 00:11:05,210: in which you guys are also represented, which has more
- 00:11:05,230 --> 00:11:07,590: than three hundred CSOS from, from Fortune one thousand.
- 00:11:08,290 --> 00:11:10,290: Um, and kind of bake it in additional
- 00:11:10,330 --> 00:11:12,430: requirements, additional descriptions of what
- 00:11:12,490 --> 00:11:13,730: risks should be covered and how.
- 00:11:14,530 --> 00:11:15,230: In practice,
- 00:11:15,970 --> 00:11:17,970: you need to recertify every year. So you
- 00:11:18,010 --> 00:11:20,330: would be expected to recertify against the most recent
- 00:11:20,390 --> 00:11:21,490: version of the standard-
- 00:11:22,010 --> 00:11:23,450: Mm-hmm... uh, and then recertify every quarter.
- 00:11:23,830 --> 00:11:25,430: But the idea is really to make sure that
- 00:11:25,990 --> 00:11:28,310: every time a new type of risk, behavior, attack
- 00:11:28,350 --> 00:11:29,970: vector, strike vectors comes up,
- 00:11:30,590 --> 00:11:32,670: it's surfaced, understood, codified
- 00:11:32,730 --> 00:11:34,870: in the standard, and tested for in the next, uh,
- 00:11:34,970 --> 00:11:35,890: recertification cycle.
- 00:11:36,450 --> 00:11:38,530: Okay. That's very smart. And also what I, what
- 00:11:38,650 --> 00:11:40,830: I at least from my trail time always notice is
- 00:11:40,890 --> 00:11:41,970: that especially also the,
- 00:11:42,370 --> 00:11:44,410: the importance of AI risk management and AI
- 00:11:44,450 --> 00:11:46,150: governance and also following standards
- 00:11:46,610 --> 00:11:47,690: really becomes evident
- 00:11:48,110 --> 00:11:50,110: once an incident occurred, right? Because then
- 00:11:50,170 --> 00:11:52,310: suddenly, or once an audit happened, and it doesn't even
- 00:11:52,350 --> 00:11:54,350: have to be an external audit, can be an internal audit,
- 00:11:54,410 --> 00:11:56,230: can be the financial authority, whoever,
- 00:11:56,650 --> 00:11:58,850: and then you finally see, oh, my governance program is not
- 00:11:58,910 --> 00:12:00,970: up to speed. Now it's time to think about how do
- 00:12:01,010 --> 00:12:01,570: we build it,
- 00:12:01,910 --> 00:12:03,910: and if there's a standard we can adhere to, that's even
- 00:12:03,990 --> 00:12:05,550: easier, so we can actually also
- 00:12:06,030 --> 00:12:08,030: like get it certified from an external
- 00:12:08,090 --> 00:12:10,130: party. Yeah. And I mean, you made the parallel
- 00:12:10,170 --> 00:12:12,350: with financial reporting. You wouldn't trust even
- 00:12:12,390 --> 00:12:14,550: the biggest bank out there to certify their own books.
- 00:12:14,590 --> 00:12:16,630: So- Yep... I don't see why the same thing should not
- 00:12:16,670 --> 00:12:18,290: apply for AI. Makes a lot of sense.
- 00:12:19,690 --> 00:12:21,790: Um, all right. I don't know
- 00:12:21,850 --> 00:12:24,070: if y- we have any other questions
- 00:12:24,130 --> 00:12:26,010: on the screen, but maybe there's
- 00:12:26,690 --> 00:12:28,730: a question from the audience. Otherwise,
- 00:12:28,870 --> 00:12:30,910: I would thank you a lot for your time-
- 00:12:30,930 --> 00:12:32,930: Thank you... David. It was very nice meeting you, learning more
- 00:12:33,010 --> 00:12:34,730: about the, the standard. Hopefully also
- 00:12:35,310 --> 00:12:37,630: something the audience wants to conduct within Kalipa,
- 00:12:37,910 --> 00:12:39,330: um, and, and with you together.
- 00:12:39,790 --> 00:12:41,870: So yeah. I would thank
- 00:12:41,890 --> 00:12:44,030: you very much and welcome Chirag back on the stage.
- 00:12:44,030 --> 00:12:45,850: [clapping] Thank you for having me. Thanks. Thank you.
From standards to control: Securing the agentic enterprise
AI is moving from systems that answer questions to agents that reason, use tools and take action. As autonomy grows, so does the need for clear standards and controls that define what agents can do — and ensure they stay within those boundaries.
Join David Meyer, AIUC, and Teresa Rojas, Collibra, for a fireside chat on the new requirements for security, safety and reliability in an agentic world. They’ll explore how standards such as AIUC-1 address emerging agent risks and how organizations can translate those standards into continuous oversight and agentic control at scale.Video: Enabling safe AI agent access to structured data
Transcript
- 00:00:05,780 --> 00:00:06,340: Thanks everybody.
- 00:00:06,980 --> 00:00:07,560: Nice to be here.
- 00:00:07,880 --> 00:00:09,180: Uh, I'm Tim Swan. I,
- 00:00:09,440 --> 00:00:11,720: I-- as you just said, I lead the mana-- metadata
- 00:00:11,760 --> 00:00:13,180: management team at Synchrony Financial,
- 00:00:14,040 --> 00:00:16,200: and we spend
- 00:00:16,560 --> 00:00:18,740: a lot of time thinking about how to use our metadata
- 00:00:19,480 --> 00:00:21,160: to better enable our business,
- 00:00:21,440 --> 00:00:21,960: including,
- 00:00:22,840 --> 00:00:25,140: um, enabling agents to access
- 00:00:25,200 --> 00:00:26,340: our, our structured data.
- 00:00:26,800 --> 00:00:28,000: So today we're gonna talk about
- 00:00:28,400 --> 00:00:29,720: why semantic layers are important,
- 00:00:30,280 --> 00:00:32,439: what purpose do they s-do they provide,
- 00:00:32,520 --> 00:00:34,080: what benefits do they provide.
- 00:00:34,740 --> 00:00:36,260: We'll talk a little bit about the what
- 00:00:36,580 --> 00:00:37,120: in terms of
- 00:00:37,720 --> 00:00:40,020: where does the semantic layer sit in a
- 00:00:40,080 --> 00:00:41,060: conceptual architecture.
- 00:00:41,960 --> 00:00:43,060: We'll talk about how,
- 00:00:43,480 --> 00:00:44,860: how do we generate a semantic layer,
- 00:00:45,400 --> 00:00:47,600: and we're gonna, we're gonna do this using Collibra data
- 00:00:47,640 --> 00:00:49,720: products. And lastly,
- 00:00:49,760 --> 00:00:52,180: we'll just look at a really simple text to SQL example
- 00:00:52,920 --> 00:00:53,700: and just
- 00:00:54,140 --> 00:00:56,300: kind of place the semantic layer in that type of solution
- 00:00:56,320 --> 00:00:57,400: and understand how it's used.
- 00:00:58,500 --> 00:00:59,580: So why?
- 00:00:59,960 --> 00:01:01,080: Why do we need a semantic layer?
- 00:01:01,780 --> 00:01:03,280: Well, as I think we've heard
- 00:01:03,560 --> 00:01:05,620: the word context about five hundred times today,
- 00:01:06,380 --> 00:01:07,440: if you do not have
- 00:01:08,160 --> 00:01:08,740: context
- 00:01:09,520 --> 00:01:10,300: around the meaning
- 00:01:10,820 --> 00:01:12,920: or the proper usage of your
- 00:01:12,980 --> 00:01:15,040: structured data, you're not gonna get good results from an
- 00:01:15,100 --> 00:01:16,660: AI, right? Asking an agent to
- 00:01:17,400 --> 00:01:19,600: analyze a piece of data or a set of data is not
- 00:01:19,620 --> 00:01:20,240: gonna fly,
- 00:01:21,000 --> 00:01:22,740: um, if you don't have rich context.
- 00:01:23,120 --> 00:01:25,500: So the purpose of a semantic layer is to turn
- 00:01:25,560 --> 00:01:26,320: that raw data,
- 00:01:26,700 --> 00:01:28,040: tables and column structures,
- 00:01:28,520 --> 00:01:30,180: into actual meaning.
- 00:01:30,600 --> 00:01:31,560: What do my tables mean?
- 00:01:31,940 --> 00:01:32,820: How are they dimensioned?
- 00:01:33,460 --> 00:01:34,480: What's the granularity?
- 00:01:34,980 --> 00:01:37,020: What do my columns mean? What data they contain?
- 00:01:37,680 --> 00:01:39,720: What are the business terms associated with those
- 00:01:39,780 --> 00:01:40,340: columns?
- 00:01:40,680 --> 00:01:42,440: What are synonyms for those business terms?
- 00:01:43,120 --> 00:01:44,340: All that sort of context
- 00:01:44,840 --> 00:01:46,900: that starts to enable an agent
- 00:01:46,940 --> 00:01:48,920: to act more intelligently on our data.
- 00:01:49,700 --> 00:01:52,100: Um, that can include things like how to join tables properly,
- 00:01:52,200 --> 00:01:53,880: how to make-- how to calculate metrics
- 00:01:54,360 --> 00:01:55,820: in a, in a approved way.
- 00:01:56,560 --> 00:01:57,660: And so the benefits of this,
- 00:01:58,140 --> 00:02:00,140: we talked about the hallucination tax. I think that was topic
- 00:02:00,240 --> 00:02:02,820: one. Let's try to get more
- 00:02:02,940 --> 00:02:05,040: accurate results out of our agents. Let's try
- 00:02:05,080 --> 00:02:06,400: to reduce hallucination.
- 00:02:06,920 --> 00:02:09,220: Let's try to get more repeatability, more accuracy.
- 00:02:10,300 --> 00:02:12,440: Um, I don't think there's such a thing as a hundred
- 00:02:12,500 --> 00:02:13,320: percent in this business.
- 00:02:13,740 --> 00:02:15,320: With the AI world, I think you can get
- 00:02:16,120 --> 00:02:18,540: very good results depending on the complexity of the dataset,
- 00:02:18,580 --> 00:02:20,140: depending on how good your semantics are.
- 00:02:20,780 --> 00:02:22,720: But I don't, I don't think you're gonna get a hundred percent
- 00:02:23,400 --> 00:02:25,620: repeatable over long periods of time. So
- 00:02:26,920 --> 00:02:29,140: you have to use this stuff in the right use cases. You have to have humans
- 00:02:29,200 --> 00:02:29,760: over the loop.
- 00:02:30,380 --> 00:02:30,900: Um, but
- 00:02:31,880 --> 00:02:33,940: having a good semantic layer can get you a long way
- 00:02:33,980 --> 00:02:35,340: in terms of using structured data.
- 00:02:35,660 --> 00:02:37,460: Just in terms of where does this thing sit w-
- 00:02:37,880 --> 00:02:39,700: i-in the overall scheme of things.
- 00:02:40,300 --> 00:02:42,140: Um, we've got
- 00:02:43,160 --> 00:02:45,140: a consumption layer tool of some kind at the top.
- 00:02:45,500 --> 00:02:47,600: Might be an AI agent, it might be a BI
- 00:02:47,680 --> 00:02:49,480: tool. We've got a semantic layer.
- 00:02:50,460 --> 00:02:52,460: We've got a data product that we're generating that semantic
- 00:02:52,500 --> 00:02:54,600: layer from, and at the bottom, we've got our
- 00:02:54,640 --> 00:02:55,700: data, our raw data.
- 00:02:56,860 --> 00:02:57,620: So if you think about
- 00:02:58,140 --> 00:03:00,220: this as kind of a process, if I know that I want
- 00:03:00,240 --> 00:03:02,480: to expose or have an agent work
- 00:03:02,540 --> 00:03:03,300: on a set of data,
- 00:03:03,740 --> 00:03:06,060: the first thing to do is create a data product that
- 00:03:06,180 --> 00:03:08,500: actually encompasses that data and describes
- 00:03:08,540 --> 00:03:11,140: it. And what that gives you is
- 00:03:11,240 --> 00:03:12,360: a nice governable
- 00:03:13,540 --> 00:03:15,120: owned set of data
- 00:03:15,840 --> 00:03:16,720: that you can
- 00:03:17,780 --> 00:03:19,800: work on getting AI ready, right? It's very
- 00:03:19,920 --> 00:03:22,440: daunting when you say, "Let's create an ontology for our enterprise,"
- 00:03:22,500 --> 00:03:24,740: or "Let's, let's make sure our data's AI ready."
- 00:03:25,220 --> 00:03:25,980: But when you say,
- 00:03:26,660 --> 00:03:28,660: "Let's define a, a
- 00:03:28,760 --> 00:03:30,840: small set of data that we need to fulfill a use case,
- 00:03:31,280 --> 00:03:33,340: let's encompass it in a product, let's make sure we have good
- 00:03:33,400 --> 00:03:34,100: definitions,
- 00:03:34,780 --> 00:03:36,420: we understand how to calculate metrics,
- 00:03:37,040 --> 00:03:39,200: we understand what the prope-proper use of that product
- 00:03:39,260 --> 00:03:41,200: is," that's a much more achievable goal.
- 00:03:41,600 --> 00:03:43,640: From that data product, you're gonna generate a semantic
- 00:03:43,700 --> 00:03:44,800: layer, and
- 00:03:45,380 --> 00:03:47,520: we'll talk more about what the semantic layer actually
- 00:03:47,760 --> 00:03:49,840: is and what it looks like in the next slide, but
- 00:03:49,940 --> 00:03:51,840: it is, you know, machine readable.
- 00:03:52,260 --> 00:03:54,600: Think of it as kind of a markdown file that describes
- 00:03:54,660 --> 00:03:56,780: to the, the agent or
- 00:03:56,860 --> 00:03:57,840: the BI tool
- 00:03:58,700 --> 00:04:00,240: how to operate on the data, what it means,
- 00:04:00,580 --> 00:04:02,600: how to access it, how to, how to join tables, how
- 00:04:02,620 --> 00:04:03,640: to calculate metrics.
- 00:04:04,240 --> 00:04:06,520: And then lastly, you've got the consumption layer tool
- 00:04:07,260 --> 00:04:09,340: agent querying the
- 00:04:09,380 --> 00:04:10,460: data. So it's, it's,
- 00:04:10,900 --> 00:04:12,680: it's ingesting that semantic layer.
- 00:04:13,220 --> 00:04:15,220: It's gaining context by doing that.
- 00:04:15,560 --> 00:04:16,959: And now when you ask it to go,
- 00:04:17,820 --> 00:04:19,880: you know, "Give me the top ten accounts for the last
- 00:04:19,959 --> 00:04:22,040: year," it knows how to go and do that properly
- 00:04:22,060 --> 00:04:23,240: and how to add those numbers up
- 00:04:23,740 --> 00:04:25,840: over the right time periods. The other thing I'd point
- 00:04:25,880 --> 00:04:28,200: out on this slide is that the separation
- 00:04:29,100 --> 00:04:31,120: between the semantic layer and the consumption layer tool
- 00:04:31,140 --> 00:04:32,580: is important, right? We heard about
- 00:04:33,180 --> 00:04:35,220: kind of this proliferation of different
- 00:04:36,100 --> 00:04:38,180: context across different tools, and how
- 00:04:38,220 --> 00:04:38,800: do you keep
- 00:04:39,400 --> 00:04:41,440: the semantics or the context consistent
- 00:04:41,500 --> 00:04:43,940: across Snowflake and Databricks
- 00:04:44,000 --> 00:04:45,960: and Tableau and all these different tools.
- 00:04:46,400 --> 00:04:48,440: And part of what we're trying to show
- 00:04:48,480 --> 00:04:50,620: here is that by separating
- 00:04:50,660 --> 00:04:52,760: the semantic layer from the tool using the semantic
- 00:04:52,820 --> 00:04:55,020: layer, and hopefully using a tool-neutral semantic layer,
- 00:04:55,560 --> 00:04:56,220: you can create that,
- 00:04:56,500 --> 00:04:57,960: that semantics once,
- 00:04:58,720 --> 00:05:01,060: and then share it across multiple agents,
- 00:05:01,120 --> 00:05:02,360: multiple BI tools, et cetera.
- 00:05:03,420 --> 00:05:04,040: So that's a, I think,
- 00:05:04,320 --> 00:05:05,240: another key point in here.
- 00:05:05,860 --> 00:05:07,080: So how do we generate one of these things?
- 00:05:08,000 --> 00:05:10,060: Um, we talked
- 00:05:10,100 --> 00:05:12,220: about using a data product to do this. So on the left, we've
- 00:05:12,240 --> 00:05:14,520: got a data product. On the right, we've got a semantic file.
- 00:05:15,160 --> 00:05:16,020: And the semantic
- 00:05:16,380 --> 00:05:18,740: f-format we're talking about here is called Open Semantic Interchange,
- 00:05:18,800 --> 00:05:20,020: or O-S-I, or OSI.
- 00:05:20,640 --> 00:05:22,980: There's a Apache project called the OSI
- 00:05:23,040 --> 00:05:24,220: Apache Project that is,
- 00:05:25,080 --> 00:05:26,400: um, developing this standard,
- 00:05:26,660 --> 00:05:28,100: and it's being contributed to by
- 00:05:28,940 --> 00:05:31,200: all the big players, Databricks, Snowflake, AWS,
- 00:05:31,380 --> 00:05:31,940: Collibra.
- 00:05:32,520 --> 00:05:35,020: Um, and the goal is to generate or to define
- 00:05:35,060 --> 00:05:36,680: an, an industry-wide
- 00:05:37,600 --> 00:05:39,900: tool-neutral, vendor-neutral semantic
- 00:05:39,960 --> 00:05:42,080: standard that can be used by
- 00:05:42,120 --> 00:05:44,300: all these tools, so we don't have to have a bunch of proprietary
- 00:05:44,360 --> 00:05:44,900: formats.
- 00:05:45,640 --> 00:05:47,740: Um, it's a worthy goal. [chuckles]
- 00:05:47,780 --> 00:05:49,820: It just started... It was just released for the first time earlier
- 00:05:49,860 --> 00:05:51,880: this year. It's being,
- 00:05:52,000 --> 00:05:53,580: um, enhanced as we speak.
- 00:05:53,860 --> 00:05:54,540: I think it's, um,
- 00:05:54,940 --> 00:05:56,440: it's, it's moving along nicely.
- 00:05:56,820 --> 00:05:58,920: But even in its current state, it's, it's, it's actually
- 00:05:59,000 --> 00:06:00,240: quite an effective format,
- 00:06:00,800 --> 00:06:01,660: and it gives you some,
- 00:06:01,980 --> 00:06:03,920: some durability over time in terms of,
- 00:06:04,360 --> 00:06:06,420: you know it's gonna be around, and it's not stuck on a s- a
- 00:06:06,460 --> 00:06:07,220: particular tool.
- 00:06:08,060 --> 00:06:10,110: Um- So
- 00:06:10,150 --> 00:06:12,510: the data product metadata on the left, you can see
- 00:06:12,550 --> 00:06:13,370: we've got a lot in there.
- 00:06:14,210 --> 00:06:16,290: Uh, some of it is specific to the data
- 00:06:16,330 --> 00:06:17,610: product. So it might be things like
- 00:06:18,010 --> 00:06:19,810: what is the purpose of this product? What is, is,
- 00:06:20,070 --> 00:06:21,530: is its intended use?
- 00:06:22,130 --> 00:06:24,170: What are the data sets that this product contains?
- 00:06:24,710 --> 00:06:26,750: What are the columns? What are the definitions of those
- 00:06:26,810 --> 00:06:29,090: columns? How do I join
- 00:06:29,150 --> 00:06:31,450: the data sets or tables within this, this product?
- 00:06:32,310 --> 00:06:33,650: How do I calculate metrics
- 00:06:34,550 --> 00:06:36,550: within the data that's found in this product? So if
- 00:06:36,590 --> 00:06:38,810: you have standard metric definitions
- 00:06:38,850 --> 00:06:40,930: and you want to make sure that data is being calculated properly
- 00:06:40,950 --> 00:06:42,750: within particular products, you can define it there.
- 00:06:43,670 --> 00:06:46,090: Um, you might include things like what are my indexes?
- 00:06:46,130 --> 00:06:47,770: What are my partition keys? If you have
- 00:06:48,350 --> 00:06:50,410: large data sets and you want to make sure your agent is
- 00:06:50,470 --> 00:06:51,550: generating smart SQL,
- 00:06:52,210 --> 00:06:54,270: you're going to want to include the keys that enable it to
- 00:06:54,310 --> 00:06:56,350: do that. And there's, there's a
- 00:06:56,390 --> 00:06:57,750: lot of other examples of, of
- 00:06:58,070 --> 00:06:59,410: context that you can provide.
- 00:07:00,110 --> 00:07:02,150: Um, curated queries is
- 00:07:02,190 --> 00:07:04,070: another really good example. So you can give--
- 00:07:04,630 --> 00:07:06,930: maybe you associate with a particular product, let's
- 00:07:06,950 --> 00:07:07,710: say it's an account
- 00:07:08,130 --> 00:07:09,190: profitability product.
- 00:07:10,510 --> 00:07:12,670: Here are a bunch of queries that are commonly used
- 00:07:12,730 --> 00:07:14,610: against this product. Here's what they do,
- 00:07:15,030 --> 00:07:16,410: and here's the SQL that, that
- 00:07:16,790 --> 00:07:18,830: does them. That's a really nice thing
- 00:07:18,870 --> 00:07:21,070: to hand to an agent because an a-agent really can
- 00:07:21,110 --> 00:07:23,170: kind of learn that and learn from that and use it.
- 00:07:23,230 --> 00:07:25,470: So curated queries are another really
- 00:07:25,530 --> 00:07:26,750: powerful piece of context.
- 00:07:27,470 --> 00:07:29,630: Um, so to generate this, we've got all this,
- 00:07:29,810 --> 00:07:31,610: all this metadata that's sitting in the product.
- 00:07:32,070 --> 00:07:34,230: Some of it you kind of get for free. So if you have good-- if you've
- 00:07:34,270 --> 00:07:36,790: done good data governance and you've got good column definitions,
- 00:07:37,310 --> 00:07:39,310: or if you've got your, you know, foreign keys
- 00:07:39,370 --> 00:07:40,270: captured in Collibra,
- 00:07:40,730 --> 00:07:42,950: a lot of this you kind of get for free when you include
- 00:07:42,990 --> 00:07:44,630: those tables or views in your product.
- 00:07:45,350 --> 00:07:47,670: Um, but any event, we've got all that metadata
- 00:07:47,750 --> 00:07:50,050: either directly in the product or in the underlying assets,
- 00:07:50,110 --> 00:07:51,450: the tables, columns, what have you.
- 00:07:51,970 --> 00:07:53,490: And what we did
- 00:07:53,970 --> 00:07:54,530: was we
- 00:07:55,390 --> 00:07:57,570: took a for-free workflow off
- 00:07:57,610 --> 00:08:00,370: the Collibra marketplace that Collibra had written that basically
- 00:08:00,430 --> 00:08:02,530: can start with the output port
- 00:08:02,570 --> 00:08:03,310: of a data product
- 00:08:03,910 --> 00:08:06,010: and generate an OSI semantic file. So
- 00:08:06,050 --> 00:08:08,150: it, it knows the format of OSI.
- 00:08:08,230 --> 00:08:10,250: It's a pretty simple schema that you see on the right.
- 00:08:10,650 --> 00:08:13,010: There's a section for the semantic model itself.
- 00:08:13,110 --> 00:08:14,230: What is this model for?
- 00:08:14,770 --> 00:08:15,830: Description, et cetera.
- 00:08:16,650 --> 00:08:18,870: What are the data sets in this, in this semantic
- 00:08:18,910 --> 00:08:21,150: model? So you might have N number of tables
- 00:08:21,190 --> 00:08:23,450: and, you know, Z number of columns.
- 00:08:23,850 --> 00:08:25,850: It's got relationships, so that's where you express
- 00:08:25,890 --> 00:08:26,890: the joins and how to,
- 00:08:27,250 --> 00:08:28,990: how to do valid joins in the data set.
- 00:08:29,550 --> 00:08:31,810: And it's got a description of any, any metrics
- 00:08:31,850 --> 00:08:34,110: that you want to be calculated in a standard way.
- 00:08:34,850 --> 00:08:36,610: Um, so what we did was we
- 00:08:37,370 --> 00:08:39,550: stole this workflow off the, uh, Collibra marketplace
- 00:08:39,590 --> 00:08:41,750: for free, and we just did a bunch of enhancing.
- 00:08:41,810 --> 00:08:44,150: We, we kind of tuned it to our meta model, our operating
- 00:08:44,210 --> 00:08:46,250: model, uh, and the things that we were
- 00:08:46,310 --> 00:08:48,490: seeing as helping the agents be more effective
- 00:08:48,530 --> 00:08:50,590: at, at querying using the semantic layer.
- 00:08:50,650 --> 00:08:51,830: So we did a bunch of enhancements.
- 00:08:52,390 --> 00:08:54,010: Um, but at the end of the day, it's just a text file.
- 00:08:54,570 --> 00:08:56,610: So last, last slide is just
- 00:08:56,690 --> 00:08:57,990: a simple example of a
- 00:08:58,490 --> 00:08:59,650: text to SQL chatbot.
- 00:09:00,290 --> 00:09:03,010: Um, so we've got a, a query assistant
- 00:09:03,050 --> 00:09:05,310: that a user is using to try to, you know, understand
- 00:09:05,370 --> 00:09:06,510: some data in a database,
- 00:09:07,130 --> 00:09:09,190: and they say, "Provide a list in Excel of the top
- 00:09:09,250 --> 00:09:11,410: ten accounts by transaction volume over
- 00:09:11,450 --> 00:09:12,390: the last four quarters."
- 00:09:13,250 --> 00:09:15,690: And what the agent is doing
- 00:09:15,810 --> 00:09:17,950: is it's got the semantic layer loaded
- 00:09:18,010 --> 00:09:19,810: for that, for the appropriate data set
- 00:09:20,570 --> 00:09:21,430: that's in green.
- 00:09:22,270 --> 00:09:24,590: It's probably got some guardrails, right? We talked a lot about guardrails
- 00:09:24,630 --> 00:09:25,730: today and making sure that,
- 00:09:26,390 --> 00:09:27,630: you know, you need to tell agents
- 00:09:27,950 --> 00:09:30,110: things like, "Only select columns
- 00:09:30,150 --> 00:09:30,810: that I'm giving you,
- 00:09:31,070 --> 00:09:33,470: and only calculate metrics using the metric calculations
- 00:09:33,530 --> 00:09:35,610: I'm giving you. Only do joins I tell you
- 00:09:35,650 --> 00:09:37,810: you can do." You have to tell the agent
- 00:09:37,850 --> 00:09:39,850: what to do, what not to do. So that's typically part of
- 00:09:39,870 --> 00:09:40,730: this sort of solution.
- 00:09:41,330 --> 00:09:43,510: And then you might have some kind of query execution service
- 00:09:43,570 --> 00:09:45,210: that just checks the validity of the SQL,
- 00:09:45,670 --> 00:09:47,690: maybe looks for out of bound conditions, make sure
- 00:09:47,710 --> 00:09:49,370: there aren't any deletes or updates happening,
- 00:09:49,950 --> 00:09:50,650: that sort of thing.
- 00:09:51,070 --> 00:09:52,150: Goes and gets the results.
- 00:09:52,710 --> 00:09:54,950: The LLM might do some formatting, put it into an Excel
- 00:09:55,010 --> 00:09:56,230: sheet or whatever the user asked for,
- 00:09:56,910 --> 00:09:57,810: and away we go.
- 00:09:58,790 --> 00:09:59,450: That is
- 00:10:00,610 --> 00:10:02,070: semantic layers. That's how they're
- 00:10:02,810 --> 00:10:04,830: made, how they're used, why they matter.
- 00:10:05,490 --> 00:10:07,630: Um, I encourage you guys to play
- 00:10:07,710 --> 00:10:09,750: around with this stuff if you haven't already. How many people are actually
- 00:10:09,830 --> 00:10:12,110: using semantic layers to access
- 00:10:12,170 --> 00:10:13,990: structured data with agents today?
- 00:10:15,550 --> 00:10:16,270: Okay. So
- 00:10:17,010 --> 00:10:17,750: pretty small number.
- 00:10:18,030 --> 00:10:20,110: So, um, this is pretty easy stuff to
- 00:10:20,150 --> 00:10:21,390: do. You know, we saw a lot of really
- 00:10:22,090 --> 00:10:24,470: incredible technology today and a lot of stuff on the horizon.
- 00:10:24,910 --> 00:10:26,990: This is pretty simple stuff to do. If you have a set
- 00:10:27,030 --> 00:10:29,249: of data that you want to expose to an agent and just test,
- 00:10:30,290 --> 00:10:32,250: create a data product in Collibra, it takes minutes.
- 00:10:32,710 --> 00:10:34,690: Make sure, you know, get some good metadata around it.
- 00:10:35,250 --> 00:10:37,330: Check out the OSI generator from Collibra.
- 00:10:37,350 --> 00:10:39,610: And by the way, I think I saw yesterday that
- 00:10:39,670 --> 00:10:42,230: you guys now have a no-code or a low-code version
- 00:10:42,310 --> 00:10:44,430: of the same thing. So my team spent a bunch
- 00:10:44,450 --> 00:10:46,450: of time writing code [chuckles]. You guys can probably do
- 00:10:46,490 --> 00:10:48,630: it for free with an agent and generate the same thing.
- 00:10:49,210 --> 00:10:51,230: Um, and try it out. You know, try it, try
- 00:10:51,270 --> 00:10:52,990: it with an agent. I think you can, um,
- 00:10:53,390 --> 00:10:55,070: you can learn a lot just by doing that. And
- 00:10:55,650 --> 00:10:57,690: hopefully, as some of these tools get better, we'll, we'll be able
- 00:10:57,710 --> 00:10:59,030: to get even more accurate over time.
- 00:11:01,010 --> 00:11:02,590: All right. Thank you guys very much.
Enabling safe AI agent access to structured data
Everyone's building AI agents, but how do we give agents the context they need to access structured data without hallucinations and errors? Tim Swan, Metadata Management and Data Sourcing Leader at Synchrony Financial, shows how they are leveraging Collibra data products and underlying metadata to enable semantic layers.
In this session, Tim will show you how to leverage data product metadata to generate an OSI (Open Semantic Interchange) semantic layer, how this context enables agents to write complex SQL queries, and the business benefits this drives. He'll cover what it took to make the semantic layer agent-ready and why they're building bottom-up from real use cases instead of waiting on a perfect enterprise ontology.Video: Customer fireside hosted by Julia Bardmesser
Transcript
- 00:00:05,640 --> 00:00:07,700: Hi, everyone. So I'm Julia Bartmesscher,
- 00:00:07,960 --> 00:00:09,640: and very brief introduction.
- 00:00:10,060 --> 00:00:12,540: I spent over twenty-five years in financial
- 00:00:12,640 --> 00:00:13,660: services industry,
- 00:00:14,140 --> 00:00:16,940: leading large data and technology transformations,
- 00:00:17,460 --> 00:00:19,520: and my focus has always been: how
- 00:00:19,600 --> 00:00:20,560: do you get value
- 00:00:21,040 --> 00:00:23,700: out of whatever it is, like data technology,
- 00:00:23,820 --> 00:00:24,360: cloud,
- 00:00:24,680 --> 00:00:26,080: AI, uh,
- 00:00:27,620 --> 00:00:28,780: machine learning, and so on.
- 00:00:29,360 --> 00:00:30,920: Uh, and then I decided
- 00:00:31,180 --> 00:00:33,280: to step out of corporate life, honestly,
- 00:00:33,380 --> 00:00:34,380: politics got to me,
- 00:00:35,220 --> 00:00:37,360: uh, and start my own company
- 00:00:37,400 --> 00:00:39,440: where I advise, teach, and it's
- 00:00:39,460 --> 00:00:41,320: all around how do you get value.
- 00:00:42,060 --> 00:00:44,000: Um, so we're gonna talk about
- 00:00:44,860 --> 00:00:47,400: the va-the value AI data
- 00:00:48,100 --> 00:00:48,720: and,
- 00:00:48,980 --> 00:00:51,000: uh... Actually, let me
- 00:00:51,100 --> 00:00:53,360: ask my panelists to introduce themselves first,
- 00:00:53,420 --> 00:00:54,420: and then I'll kick it off.
- 00:00:55,080 --> 00:00:57,160: Please, Drew. Thanks, Julia. Uh, hi, everybody. Drew
- 00:00:57,240 --> 00:00:57,760: Palsgrove.
- 00:00:58,260 --> 00:00:59,480: I'm part of Pfizer Digital,
- 00:00:59,800 --> 00:01:02,220: um, one of those companies that has a little bit of politics sometimes.
- 00:01:02,420 --> 00:01:04,660: Sometimes, yes. Um, [chuckles] and happy to be here today.
- 00:01:04,879 --> 00:01:07,100: Um, my team is a team within,
- 00:01:07,260 --> 00:01:09,560: uh, uh, the central part of Pfizer
- 00:01:09,800 --> 00:01:11,960: that manages platforms, including many of the ones
- 00:01:12,000 --> 00:01:12,840: we've talked about today,
- 00:01:13,300 --> 00:01:14,660: and we have lots of teammates here from
- 00:01:15,240 --> 00:01:17,460: our, uh, our partners and stakeholders who, uh,
- 00:01:17,640 --> 00:01:18,780: do the really heavy work of
- 00:01:19,280 --> 00:01:20,060: the data ecosystem.
- 00:01:20,640 --> 00:01:22,180: Chris. Uh, hey. Kristen Brown.
- 00:01:22,620 --> 00:01:24,880: I, um, am the head of our digital
- 00:01:24,980 --> 00:01:27,000: data design and engineering function
- 00:01:27,140 --> 00:01:27,840: for, um,
- 00:01:28,140 --> 00:01:29,360: the asset management arm.
- 00:01:29,980 --> 00:01:32,160: Uh, for those of you who have not heard of Macquarie,
- 00:01:32,240 --> 00:01:34,040: it's actually based in Sydney, Australia,
- 00:01:34,420 --> 00:01:35,300: um, but our
- 00:01:35,680 --> 00:01:36,400: North American
- 00:01:37,020 --> 00:01:39,160: home base is, uh, on Fifth Avenue here
- 00:01:39,220 --> 00:01:39,940: in New York City,
- 00:01:40,420 --> 00:01:41,420: um, where our,
- 00:01:41,900 --> 00:01:43,260: uh, asset management and
- 00:01:44,000 --> 00:01:46,340: capital or investment banking arms, um,
- 00:01:46,620 --> 00:01:47,500: are kind of headquartered.
- 00:01:48,140 --> 00:01:50,220: Okay. So we're gonna talk
- 00:01:50,280 --> 00:01:52,420: about what changes and what stays
- 00:01:52,460 --> 00:01:54,500: the same, uh, with-- as
- 00:01:54,540 --> 00:01:56,460: you move, uh, into the next
- 00:01:56,920 --> 00:01:58,440: iteration or-- of
- 00:01:58,780 --> 00:01:59,700: AI capabilities.
- 00:02:00,340 --> 00:02:01,200: So first,
- 00:02:01,700 --> 00:02:03,700: I wanna see some show of hands. So one of
- 00:02:03,740 --> 00:02:05,720: the thing-- one of the reactions I had when
- 00:02:06,160 --> 00:02:08,320: the whole AI craze broke out, when
- 00:02:08,360 --> 00:02:09,500: ChatGPT started,
- 00:02:10,000 --> 00:02:11,140: I had to suppress
- 00:02:11,600 --> 00:02:12,160: eye-rolling
- 00:02:13,800 --> 00:02:14,360: because,
- 00:02:14,660 --> 00:02:16,980: uh, you know, quants, I started working, I'm dating
- 00:02:17,020 --> 00:02:18,360: myself, there were quants,
- 00:02:18,940 --> 00:02:21,120: and they were doing statistical modeling, they were doing
- 00:02:21,180 --> 00:02:23,540: predictive modeling, and they were the stars
- 00:02:23,980 --> 00:02:24,980: of the organization.
- 00:02:25,340 --> 00:02:27,580: And then a little bit later, we got into data science
- 00:02:27,660 --> 00:02:29,160: craze, right? We remember that one.
- 00:02:29,980 --> 00:02:31,080: And it's also
- 00:02:31,640 --> 00:02:33,460: similar techniques on this next level,
- 00:02:34,200 --> 00:02:36,280: and at least data science acknowledged
- 00:02:36,300 --> 00:02:37,500: the fact that there is data.
- 00:02:37,900 --> 00:02:40,200: Quants, like, data was nowhere, right?
- 00:02:40,280 --> 00:02:41,380: It wasn't acknowledged.
- 00:02:42,080 --> 00:02:44,140: Uh, and now we have AI, and we have
- 00:02:44,280 --> 00:02:46,280: similar craze around AI.
- 00:02:46,600 --> 00:02:48,700: So how many of you have the same reaction when this
- 00:02:48,720 --> 00:02:49,620: whole thing started?
- 00:02:50,240 --> 00:02:51,740: Show of hands. Oh, okay.
- 00:02:52,940 --> 00:02:54,940: People are less skeptical than I am. I was
- 00:02:54,980 --> 00:02:56,240: expecting a lot more hands.
- 00:02:57,000 --> 00:02:58,260: Uh, so,
- 00:02:58,640 --> 00:03:00,780: and so we've done a lot to manage data,
- 00:03:01,080 --> 00:03:03,380: starting with the quants and then moving on to machine
- 00:03:03,420 --> 00:03:05,020: learning and data science,
- 00:03:05,420 --> 00:03:07,540: and we have these whole statistics that eighty percent
- 00:03:07,580 --> 00:03:09,420: of the time data scientists spend is
- 00:03:09,740 --> 00:03:12,340: cleaning data versus actually doing data science.
- 00:03:12,900 --> 00:03:15,160: And for some reason, they were considered to be more expensive
- 00:03:15,220 --> 00:03:17,020: people to be doing that than data people.
- 00:03:17,380 --> 00:03:18,760: Never quite figured that one out.
- 00:03:19,340 --> 00:03:20,120: Uh, so,
- 00:03:21,720 --> 00:03:23,800: but th-there are some changes. So my
- 00:03:23,860 --> 00:03:25,920: first question, Drew, is to you. When did you
- 00:03:25,980 --> 00:03:28,140: realize that the work that you were doing
- 00:03:28,800 --> 00:03:30,120: for AI--
- 00:03:30,380 --> 00:03:32,460: for data governance or
- 00:03:32,560 --> 00:03:34,300: managing data had to make
- 00:03:34,800 --> 00:03:36,940: changes for AI? What was the thing that's
- 00:03:36,980 --> 00:03:38,740: like, "Oh, we have to do things
- 00:03:39,100 --> 00:03:41,300: differently"? I think it was not
- 00:03:41,360 --> 00:03:43,580: one singular event. It was the deluge
- 00:03:43,780 --> 00:03:45,780: of, of use cases that emerged
- 00:03:45,820 --> 00:03:47,240: very quickly. I think people like you-
- 00:03:47,540 --> 00:03:49,680: Mm-hmm... saw the power of, uh, AI
- 00:03:49,720 --> 00:03:51,400: emerging pretty quickly with ChatGPT
- 00:03:51,860 --> 00:03:53,800: and said, "I wanna apply that everywhere."
- 00:03:54,100 --> 00:03:56,120: And so, you know, our legal teams
- 00:03:56,180 --> 00:03:58,240: get concerned, our compliance teams get concerned. They're like, "What
- 00:03:58,260 --> 00:03:59,060: do you mean you have
- 00:03:59,680 --> 00:04:00,820: computer systems that
- 00:04:01,340 --> 00:04:03,360: are non-deterministic," right? They're, they're sending our
- 00:04:03,400 --> 00:04:04,040: data anywhere,
- 00:04:04,640 --> 00:04:06,800: um, on the internet potentially and opening a risk,
- 00:04:07,060 --> 00:04:09,120: um, in that, in that vector too. So,
- 00:04:09,500 --> 00:04:11,180: so we had a flurry of-
- 00:04:11,440 --> 00:04:13,540: Mm... use cases that, uh, many of our teams wanted
- 00:04:13,560 --> 00:04:14,380: to develop very quickly,
- 00:04:14,860 --> 00:04:16,720: and I think that's-- that was our, that was our first
- 00:04:17,579 --> 00:04:18,120: major,
- 00:04:18,420 --> 00:04:20,680: I think, shift was understanding this is a completely different
- 00:04:21,240 --> 00:04:23,400: type of category than we had managed
- 00:04:23,420 --> 00:04:25,420: before. Yeah. And so it wa-- it, it
- 00:04:25,480 --> 00:04:26,960: came more from the risk perspective.
- 00:04:27,420 --> 00:04:28,200: I think so, yeah. Yeah.
- 00:04:28,880 --> 00:04:29,660: Kristen, what about you?
- 00:04:30,020 --> 00:04:32,180: Uh, well, I would say it, uh, was probably more
- 00:04:32,260 --> 00:04:33,980: on the, uh, hallucination side
- 00:04:34,260 --> 00:04:36,380: in that the first, uh, 10K
- 00:04:36,440 --> 00:04:37,740: that somebody tried to, uh,
- 00:04:38,400 --> 00:04:39,220: in the first
- 00:04:39,520 --> 00:04:40,420: generation of,
- 00:04:40,700 --> 00:04:41,940: uh, ChatGPT and
- 00:04:42,540 --> 00:04:43,060: then tried to
- 00:04:43,480 --> 00:04:45,520: alt-- you know, copy-paste information that
- 00:04:45,580 --> 00:04:47,580: came out, and then when you looked at the actual 10K
- 00:04:47,640 --> 00:04:49,660: report, none of the tables
- 00:04:49,700 --> 00:04:51,160: matched the data [laughs] that was,
- 00:04:51,540 --> 00:04:53,720: that came out in their summary of,
- 00:04:53,820 --> 00:04:55,440: you know, of those kinds of scenarios.
- 00:04:56,020 --> 00:04:57,160: So I think it was, um,
- 00:04:57,620 --> 00:05:00,080: you know, the, uh, excitement and promise
- 00:05:00,160 --> 00:05:01,180: of what you would get,
- 00:05:01,600 --> 00:05:02,460: you know, from the,
- 00:05:03,120 --> 00:05:04,740: you know, just throw a file in and
- 00:05:05,460 --> 00:05:06,420: magical output,
- 00:05:06,680 --> 00:05:08,100: uh, you know, does it all for you,
- 00:05:08,800 --> 00:05:10,700: uh, is when, when we started to realize,
- 00:05:11,000 --> 00:05:11,640: you know, we had to
- 00:05:11,920 --> 00:05:12,980: think much broader,
- 00:05:13,460 --> 00:05:14,300: um, about what,
- 00:05:14,620 --> 00:05:16,280: you know, context means, what,
- 00:05:16,860 --> 00:05:17,420: you, you know, the,
- 00:05:18,180 --> 00:05:20,180: the reaches of data governance, um,
- 00:05:20,300 --> 00:05:22,400: in its traditional sense needed to extend.
- 00:05:22,740 --> 00:05:25,120: Well, sometimes magic misfires. Yeah.
- 00:05:25,160 --> 00:05:26,200: Right? Uh,
- 00:05:26,460 --> 00:05:28,460: so having said that, so we've, again,
- 00:05:28,540 --> 00:05:30,660: we've been managing data for different, for
- 00:05:30,920 --> 00:05:32,040: BI, for
- 00:05:32,440 --> 00:05:33,940: data science, for,
- 00:05:34,580 --> 00:05:36,660: uh, just operational reporting,
- 00:05:36,740 --> 00:05:38,780: financial reporting for really long time, and
- 00:05:38,820 --> 00:05:40,820: we've created the whole data governance, data
- 00:05:40,880 --> 00:05:41,860: management structures
- 00:05:42,340 --> 00:05:43,940: for regulatory purposes and then
- 00:05:44,320 --> 00:05:45,540: moving on to defense.
- 00:05:45,920 --> 00:05:48,060: So what are the things, Kristen, even with, uh,
- 00:05:48,220 --> 00:05:49,100: that stay
- 00:05:50,000 --> 00:05:52,340: basically are still very much relevant
- 00:05:52,920 --> 00:05:55,640: as you move into using AI capabilities
- 00:05:55,700 --> 00:05:56,520: to bring the value?
- 00:05:57,220 --> 00:05:57,820: Yeah. Um,
- 00:05:58,320 --> 00:06:00,500: so not to be a, you know, paid actor on this,
- 00:06:00,620 --> 00:06:02,840: but, uh- [chuckles]... when you talk about semantics,
- 00:06:02,900 --> 00:06:04,900: right? It's funny that they had Tim go right before us, but
- 00:06:04,940 --> 00:06:06,320: that, that is true 'cause,
- 00:06:07,000 --> 00:06:09,048: um- Everybody's been in the
- 00:06:09,108 --> 00:06:11,228: mode of, you know, data mesh, data products.
- 00:06:11,448 --> 00:06:13,468: Um, we call our semantic layer,
- 00:06:13,668 --> 00:06:14,288: you know, we've
- 00:06:14,788 --> 00:06:16,988: separate them into sets of things called core marts
- 00:06:17,028 --> 00:06:18,328: and consumer marts, and then,
- 00:06:18,888 --> 00:06:20,908: you know, do it in the same kind of
- 00:06:20,948 --> 00:06:22,308: definition that, that you have there.
- 00:06:22,828 --> 00:06:23,708: So th- that's, that,
- 00:06:24,048 --> 00:06:26,208: that doesn't change, right? All about the metadata,
- 00:06:26,288 --> 00:06:27,468: metadata collection.
- 00:06:28,068 --> 00:06:30,068: Um, you, you have to stay
- 00:06:30,228 --> 00:06:32,248: vigilant and, and continue on in,
- 00:06:32,368 --> 00:06:34,248: in, in that regard of, of things.
- 00:06:34,908 --> 00:06:35,588: Um, and
- 00:06:35,988 --> 00:06:38,388: still all the same principles, right, around having
- 00:06:38,488 --> 00:06:40,108: proper structured data products,
- 00:06:40,407 --> 00:06:41,208: data cataloging,
- 00:06:41,708 --> 00:06:43,748: right? That, that's not gonna change, and you're gonna have
- 00:06:43,788 --> 00:06:45,188: to continue to, um,
- 00:06:45,568 --> 00:06:46,448: broaden out
- 00:06:46,728 --> 00:06:48,968: what goes in the catalog more so than anything.
- 00:06:49,508 --> 00:06:51,568: Yeah. I think if anything, AI actually
- 00:06:51,628 --> 00:06:52,448: makes metadata
- 00:06:53,348 --> 00:06:55,648: much more important and easier to explain
- 00:06:55,688 --> 00:06:57,888: why you need it, because that was always the case.
- 00:06:58,248 --> 00:06:58,768: Trying, like,
- 00:06:59,348 --> 00:07:00,168: everybody agrees
- 00:07:00,648 --> 00:07:02,448: descriptions are really, really important,
- 00:07:02,848 --> 00:07:04,888: but when it gets to it, not a lot of people wanna
- 00:07:04,948 --> 00:07:06,328: do them. So
- 00:07:06,688 --> 00:07:07,468: it's one of those... So
- 00:07:07,808 --> 00:07:09,988: Drew, what, what, on your side, what do you see
- 00:07:10,068 --> 00:07:12,428: that was important and stayed important
- 00:07:12,527 --> 00:07:13,968: or is staying important for
- 00:07:14,328 --> 00:07:15,808: where we are right now? Yeah. I think the
- 00:07:16,068 --> 00:07:16,708: overall
- 00:07:16,988 --> 00:07:18,028: muscle of,
- 00:07:18,608 --> 00:07:19,608: uh, managing risk
- 00:07:19,868 --> 00:07:20,728: and the,
- 00:07:21,248 --> 00:07:22,008: the process of
- 00:07:22,448 --> 00:07:24,188: governance in the context of a use case-
- 00:07:24,548 --> 00:07:26,648: Mm-hmm... is really important. It looks much differently today,
- 00:07:27,128 --> 00:07:28,888: but the, but the muscle of, um,
- 00:07:29,388 --> 00:07:30,488: of allowing innovation
- 00:07:30,748 --> 00:07:31,768: in secure,
- 00:07:32,328 --> 00:07:34,328: controlled environments with some degree of
- 00:07:34,348 --> 00:07:34,988: flexibility
- 00:07:35,468 --> 00:07:37,288: was really important for us to, to move forward.
- 00:07:37,748 --> 00:07:38,668: Yeah. So that
- 00:07:39,128 --> 00:07:41,548: forever balance, right? Yeah. How much you control
- 00:07:41,588 --> 00:07:43,768: and how much you let people to innovate, and where
- 00:07:43,828 --> 00:07:44,828: do you do, because you don't wanna
- 00:07:45,328 --> 00:07:46,148: stop everything.
- 00:07:46,648 --> 00:07:48,668: Like CISO, that's always a problem with CISOs as
- 00:07:48,708 --> 00:07:49,908: well. They want nothing
- 00:07:51,188 --> 00:07:53,468: happening, and of course, for the business to happen,
- 00:07:53,568 --> 00:07:54,808: something must be happening.
- 00:07:55,308 --> 00:07:56,648: So that's always the balance.
- 00:07:57,048 --> 00:07:58,188: So Drew, let me take
- 00:07:58,488 --> 00:08:00,528: next question. So okay, so this is
- 00:08:00,568 --> 00:08:02,568: continues to be important. What are the new
- 00:08:02,628 --> 00:08:04,828: things that you haven't had to worry
- 00:08:04,868 --> 00:08:05,668: about before,
- 00:08:06,128 --> 00:08:08,328: and now that you have to bring into
- 00:08:08,448 --> 00:08:10,788: your data management, data governance
- 00:08:10,848 --> 00:08:12,408: set of tools and practices?
- 00:08:13,108 --> 00:08:13,788: What I love about--
- 00:08:14,188 --> 00:08:16,487: What I love that's new is the demand
- 00:08:16,668 --> 00:08:18,928: for better data governance, just the whole idea,
- 00:08:19,048 --> 00:08:19,768: right? So-
- 00:08:20,068 --> 00:08:22,308: You're popular now. I am popular now. Yeah.
- 00:08:22,328 --> 00:08:23,648: [chuckles] I'm very proud to say that, yeah.
- 00:08:24,128 --> 00:08:25,788: Um, so I, I think in the, in the past,
- 00:08:26,448 --> 00:08:27,228: there was a... It's a,
- 00:08:27,528 --> 00:08:29,288: you know, I view it as a tax to do
- 00:08:29,908 --> 00:08:32,228: data governance well, in a sense. Mm-hmm. And
- 00:08:32,408 --> 00:08:34,408: most teams weren't willing to pay that tax because
- 00:08:34,428 --> 00:08:36,588: they didn't have a value driver for it. Mm-hmm. I
- 00:08:36,608 --> 00:08:38,508: think what's exciting now is that we now have...
- 00:08:38,808 --> 00:08:39,748: Did my, the microphone go?
- 00:08:40,488 --> 00:08:42,528: Um, what, what, what we now have is
- 00:08:42,568 --> 00:08:44,688: a value driver that's very clear, that
- 00:08:44,728 --> 00:08:45,348: if you do,
- 00:08:45,948 --> 00:08:48,108: um, governance well, uh, I think Tim was showing
- 00:08:48,168 --> 00:08:50,168: the, the, the stairsteps there up
- 00:08:50,228 --> 00:08:52,488: to the, the stack of the, uh, AI agent
- 00:08:52,528 --> 00:08:53,648: and the BI tools at the top,
- 00:08:54,008 --> 00:08:56,028: then you can get a lot more value out of that work
- 00:08:56,068 --> 00:08:56,628: that you've done
- 00:08:57,048 --> 00:08:59,128: to steward your data, to make sure it's high quality,
- 00:08:59,548 --> 00:09:01,308: and to provide that context around it.
- 00:09:01,668 --> 00:09:03,348: Becomes much more meaningful and useful.
- 00:09:03,648 --> 00:09:05,888: Yeah. Okay. So basically, the AI
- 00:09:05,948 --> 00:09:08,008: gives a reason to
- 00:09:08,068 --> 00:09:10,088: do all of this. It does, yeah. And you don't have to talk
- 00:09:10,108 --> 00:09:11,568: people into it. Compel. Compel. And you don't have to...
- 00:09:11,988 --> 00:09:14,028: Yeah, and you don't have to compel them with policy
- 00:09:14,048 --> 00:09:16,068: and audit, which is why I've always been
- 00:09:16,148 --> 00:09:18,348: always great friends with audits teams
- 00:09:18,408 --> 00:09:19,688: in CTN and Deutsche Bank,
- 00:09:19,968 --> 00:09:21,728: because that's how you got people to comply.
- 00:09:22,408 --> 00:09:24,488: Yeah. Yeah. Kristen, what about you? I, I think the
- 00:09:24,828 --> 00:09:26,828: most exciting part for this is, um,
- 00:09:27,648 --> 00:09:28,968: we've been able to leverage
- 00:09:29,308 --> 00:09:31,468: graph technologies more than you normally
- 00:09:31,528 --> 00:09:32,468: could, right? Yeah.
- 00:09:32,908 --> 00:09:33,468: It's, um,
- 00:09:34,008 --> 00:09:36,048: in that concept of, of context, right?
- 00:09:36,108 --> 00:09:36,628: And building
- 00:09:37,128 --> 00:09:38,368: ontologies, right? Like it's,
- 00:09:38,648 --> 00:09:40,248: it's, uh... I try to tell people, uh,
- 00:09:41,028 --> 00:09:43,128: I worked for Google at, for eight years, and I
- 00:09:43,168 --> 00:09:45,208: got to, you know, do lots of things inside of
- 00:09:45,268 --> 00:09:46,088: search. Um,
- 00:09:46,388 --> 00:09:48,568: so it's kind of cool to actually finally have
- 00:09:48,628 --> 00:09:49,308: real world-
- 00:09:49,688 --> 00:09:52,348: Mm-hmm... proper commercial [chuckles] capabilities
- 00:09:52,428 --> 00:09:54,648: other than how to serve more ads
- 00:09:54,768 --> 00:09:56,068: to humans and, you know,
- 00:09:56,768 --> 00:09:58,888: give, give the, the, the pool to
- 00:09:58,928 --> 00:10:00,748: retailers, right? It's, uh, it's a different
- 00:10:01,028 --> 00:10:02,348: way of thinking about search, and,
- 00:10:02,968 --> 00:10:05,028: and that's really what you're doing, right? Is you're building your own
- 00:10:05,088 --> 00:10:08,128: giant knowledge repository/search
- 00:10:08,168 --> 00:10:09,388: engine, right? Mm-hmm. That
- 00:10:09,768 --> 00:10:11,348: gives the ultimate layer of,
- 00:10:12,008 --> 00:10:12,568: of, um,
- 00:10:13,168 --> 00:10:15,348: uh, articulated context, right? Across
- 00:10:15,448 --> 00:10:17,448: so that everybody's on the same playing
- 00:10:17,548 --> 00:10:18,208: field and, uh,
- 00:10:18,828 --> 00:10:20,148: you-- we used to have pockets of,
- 00:10:20,408 --> 00:10:22,428: you know, this investment team or that investment
- 00:10:22,468 --> 00:10:23,188: team would have these,
- 00:10:23,448 --> 00:10:25,648: you know, what we called golden nuggets of information
- 00:10:25,687 --> 00:10:27,708: that we've... Now you can surface all of
- 00:10:27,768 --> 00:10:28,367: that and
- 00:10:28,708 --> 00:10:30,008: the, um, amount of,
- 00:10:30,728 --> 00:10:31,668: you know, deal time,
- 00:10:32,028 --> 00:10:34,028: you can get faster and all those things. Um,
- 00:10:34,168 --> 00:10:36,368: it's just kinda exciting to be in this space
- 00:10:36,428 --> 00:10:37,328: right now. Yeah.
- 00:10:37,708 --> 00:10:38,868: So I remember when knowledge gra-
- 00:10:39,328 --> 00:10:40,488: graphs came out, and
- 00:10:40,928 --> 00:10:42,108: all of the data... Like, I'm,
- 00:10:42,388 --> 00:10:44,188: can count myself as a data geek,
- 00:10:45,008 --> 00:10:47,048: and you're so excited about it, and you keep
- 00:10:47,088 --> 00:10:49,348: thinking, "How can we use it in real life?"
- 00:10:49,388 --> 00:10:51,068: And then, you know, when you're in a
- 00:10:51,568 --> 00:10:53,768: CDO position, there is a data geek, and then
- 00:10:53,808 --> 00:10:55,928: there is you have to explain why you're doing this to
- 00:10:55,988 --> 00:10:57,548: business and what's the
- 00:10:57,868 --> 00:11:00,068: value, and you could-- I could never get
- 00:11:00,108 --> 00:11:01,108: there with knowledge graphs,
- 00:11:01,388 --> 00:11:02,188: and now we can.
- 00:11:03,108 --> 00:11:04,448: Now we can. So that's
- 00:11:04,888 --> 00:11:06,988: one of the things. So for, actually, for both of you.
- 00:11:07,088 --> 00:11:09,288: For you, that governance is now very visible
- 00:11:09,388 --> 00:11:11,388: value, and now we can finally
- 00:11:11,808 --> 00:11:13,908: get to do knowledge graphs. [chuckles] And we don't...
- 00:11:14,108 --> 00:11:16,208: And it's easy to explain why, because, you
- 00:11:16,268 --> 00:11:18,048: know, you get AI to hallucinate
- 00:11:18,488 --> 00:11:19,408: and, like, yeah,
- 00:11:20,288 --> 00:11:21,528: that things will help with that.
- 00:11:21,988 --> 00:11:23,988: All right. So let me ask you a different
- 00:11:24,048 --> 00:11:25,648: question. So
- 00:11:26,108 --> 00:11:26,728: what do you need,
- 00:11:27,228 --> 00:11:29,528: what would, did you need or you're thinking
- 00:11:29,948 --> 00:11:30,908: of stopping?
- 00:11:31,288 --> 00:11:31,848: So again,
- 00:11:32,108 --> 00:11:34,168: we have created a lot of practices. We
- 00:11:34,208 --> 00:11:34,768: have created
- 00:11:35,308 --> 00:11:37,668: a lot of processes around data governance
- 00:11:37,708 --> 00:11:38,748: and data management.
- 00:11:39,748 --> 00:11:41,928: Which things are no longer useful
- 00:11:42,588 --> 00:11:44,788: or don't work well in the age of
- 00:11:44,848 --> 00:11:46,248: AI when you do this for
- 00:11:46,788 --> 00:11:48,488: GenAI? Um,
- 00:11:48,848 --> 00:11:50,828: yeah, the one area that, um,
- 00:11:51,228 --> 00:11:53,388: actually the guys from DZ Labs have been
- 00:11:53,428 --> 00:11:55,568: phenomenal at helping us and advising on,
- 00:11:56,288 --> 00:11:58,408: um, is in the unstructured data, right?
- 00:11:58,468 --> 00:12:00,248: So what you would traditionally think of,
- 00:12:00,528 --> 00:12:02,288: you know, how you store, record,
- 00:12:02,788 --> 00:12:03,648: structure your,
- 00:12:04,388 --> 00:12:04,948: you know, good old
- 00:12:05,388 --> 00:12:07,568: everybody in the Microsoft hierarchy folder.
- 00:12:07,728 --> 00:12:09,828: Like, it's, like, stuck in your brain that this is how
- 00:12:09,848 --> 00:12:11,168: it goes. Um,
- 00:12:11,668 --> 00:12:12,668: is, is
- 00:12:12,928 --> 00:12:14,068: getting out of that mold,
- 00:12:14,348 --> 00:12:16,448: right? That it doesn't have to necessarily be
- 00:12:16,608 --> 00:12:17,388: so, you know-
- 00:12:18,148 --> 00:12:20,168: It needs to be organized, but not in the way that you
- 00:12:20,188 --> 00:12:21,188: would traditionally think,
- 00:12:21,468 --> 00:12:23,328: um, to, you know, kind of free up your data.
- 00:12:23,868 --> 00:12:25,967: So w-we spent an exorbitant amount of
- 00:12:26,088 --> 00:12:26,928: time and money on,
- 00:12:27,288 --> 00:12:28,148: you know, data room-
- 00:12:28,768 --> 00:12:29,308: Mm-hmm... times
- 00:12:29,628 --> 00:12:31,628: X, times Y, and those kinds of scenarios.
- 00:12:32,208 --> 00:12:33,128: So we've been able to,
- 00:12:33,688 --> 00:12:35,868: you know, kind of look outside the box and,
- 00:12:35,908 --> 00:12:36,928: and not have to
- 00:12:37,608 --> 00:12:39,708: r-rely on those kind of traditional kind of
- 00:12:40,148 --> 00:12:41,788: structures in the unstructured world,
- 00:12:42,428 --> 00:12:43,408: um, I think has been the,
- 00:12:43,668 --> 00:12:45,968: the, the biggest thing that we've
- 00:12:46,028 --> 00:12:48,028: had to make a hard pivot as you think
- 00:12:48,088 --> 00:12:49,788: about the full corpus of your data.
- 00:12:50,308 --> 00:12:52,328: Well, so basically, for the unstructured data, and
- 00:12:52,388 --> 00:12:54,608: somebody today at lunch mentioned that we should define
- 00:12:55,148 --> 00:12:56,748: what unstructured data means, because
- 00:12:57,188 --> 00:12:59,128: we, we can all define the things differently.
- 00:12:59,488 --> 00:13:01,688: So the data that's not in a tabular format, that's
- 00:13:01,728 --> 00:13:03,928: not stored in the database, I'm gonna do really
- 00:13:03,968 --> 00:13:05,068: simple. So
- 00:13:05,788 --> 00:13:07,928: it's kind of interesting because, again, records management
- 00:13:08,028 --> 00:13:09,008: has been there forever.
- 00:13:09,708 --> 00:13:11,848: Information security for unstructured
- 00:13:11,888 --> 00:13:13,888: data has kind of sort of tried
- 00:13:13,928 --> 00:13:14,808: to be there forever.
- 00:13:15,448 --> 00:13:17,228: But managing it for,
- 00:13:18,008 --> 00:13:20,208: uh, value for AI,
- 00:13:20,348 --> 00:13:21,228: that hasn't been.
- 00:13:21,688 --> 00:13:22,667: So to me, it's actually...
- 00:13:23,048 --> 00:13:24,088: it's interesting because for me,
- 00:13:24,348 --> 00:13:26,308: it's more the new thing that you have to do,
- 00:13:26,848 --> 00:13:29,008: not so much... You still, you don't stop records
- 00:13:29,068 --> 00:13:31,408: management. Right. Right. But you need to do this differently,
- 00:13:31,848 --> 00:13:34,028: because now it's a totally different usage
- 00:13:34,128 --> 00:13:35,128: pattern of this.
- 00:13:35,908 --> 00:13:37,988: What about you, Drew? What did you have to or what
- 00:13:38,008 --> 00:13:39,168: are you thinking of stopping?
- 00:13:39,608 --> 00:13:41,668: Well, we've been slowing something down for
- 00:13:41,688 --> 00:13:43,048: a long time, and I, and I think it's
- 00:13:43,588 --> 00:13:44,548: accelerating in its
- 00:13:44,828 --> 00:13:47,088: decrease, which is the idea of centralization.
- 00:13:47,508 --> 00:13:49,508: So the idea that you have a curated set of data
- 00:13:49,568 --> 00:13:50,088: experts
- 00:13:50,588 --> 00:13:52,668: that are able to get all the answers- Mm-hmm... that you go to, to
- 00:13:52,728 --> 00:13:53,568: answer your questions.
- 00:13:54,008 --> 00:13:55,808: A, a big word internally at Pfizer now is,
- 00:13:56,188 --> 00:13:58,368: is federated, and I think it's very true- Mm-hmm... is that we can,
- 00:13:58,868 --> 00:14:00,868: we can take what used to be, you know, with the right
- 00:14:00,948 --> 00:14:02,448: context and with the trusted data,
- 00:14:02,928 --> 00:14:05,388: we can take what used to h- what used to be centralized
- 00:14:05,428 --> 00:14:06,048: functions
- 00:14:06,508 --> 00:14:07,888: and skills, and now
- 00:14:08,308 --> 00:14:10,248: we've decreased the complexity, and we're able to,
- 00:14:10,808 --> 00:14:11,728: to push that
- 00:14:12,028 --> 00:14:13,608: ability out to teams that can,
- 00:14:13,928 --> 00:14:15,948: and people that can now innovate much faster-
- 00:14:16,248 --> 00:14:18,108: Mm-hmm... than if they had to go through a central process.
- 00:14:18,568 --> 00:14:20,688: So that, that's a trend that's been going on for a long
- 00:14:20,768 --> 00:14:23,028: time. I really, I really see a lot of acceleration
- 00:14:23,108 --> 00:14:23,788: in that right now. Mm-hmm.
- 00:14:24,628 --> 00:14:26,067: I've always been a fan of
- 00:14:26,328 --> 00:14:27,048: hub and spoke.
- 00:14:28,488 --> 00:14:30,508: So, you know, when you overly... And
- 00:14:30,548 --> 00:14:32,628: I've been part of centralized organization, and
- 00:14:32,688 --> 00:14:34,568: I've been part of federated organization,
- 00:14:34,908 --> 00:14:36,688: and actually, I started my career in data
- 00:14:37,328 --> 00:14:39,648: being on the out, like, not in the center,
- 00:14:39,788 --> 00:14:40,908: in the federated stuff,
- 00:14:41,488 --> 00:14:43,628: uh, spot, and tried really hard not
- 00:14:43,668 --> 00:14:44,548: to get to the center.
- 00:14:45,028 --> 00:14:47,148: Like, i-including misnaming the
- 00:14:47,208 --> 00:14:47,748: database,
- 00:14:48,168 --> 00:14:49,788: just so they didn't pull us to the center.
- 00:14:50,488 --> 00:14:52,528: Uh, and of course, for my sins, my next bunch
- 00:14:52,568 --> 00:14:54,708: of jobs were all been on the enterprise side
- 00:14:54,728 --> 00:14:56,388: and giving people like, "Yeah, you have to come
- 00:14:56,848 --> 00:14:58,908: to the middle." But I always like hub
- 00:14:58,928 --> 00:14:59,488: and spoke,
- 00:14:59,968 --> 00:15:00,928: because you can do a
- 00:15:01,328 --> 00:15:02,828: s-- you need to do standardization,
- 00:15:03,168 --> 00:15:04,408: right? And that's the hub.
- 00:15:04,948 --> 00:15:05,488: But then
- 00:15:05,948 --> 00:15:08,048: you can let people innovate and do what
- 00:15:08,328 --> 00:15:09,548: business areas need to do
- 00:15:09,888 --> 00:15:11,088: in the spokes. What do you think?
- 00:15:11,728 --> 00:15:13,268: Yeah, no, no, I agree. I... and, uh,
- 00:15:13,848 --> 00:15:15,908: yeah, federated data mesh, right? That
- 00:15:15,968 --> 00:15:17,348: is really the- Yeah... the, it is
- 00:15:17,688 --> 00:15:19,708: getting the data closest to the teams
- 00:15:19,748 --> 00:15:21,148: that need it. Um,
- 00:15:21,448 --> 00:15:23,268: but honoring, you know, all things you need for
- 00:15:23,528 --> 00:15:25,608: mastering of data, you know- Yeah... that kind of scenario.
- 00:15:25,668 --> 00:15:27,688: No, I agree. It, it actually has lent itself
- 00:15:27,748 --> 00:15:29,508: to, to, um,
- 00:15:30,088 --> 00:15:31,468: be more flexible, right? Where
- 00:15:32,248 --> 00:15:33,348: for a while there, right, it was,
- 00:15:33,628 --> 00:15:35,888: went from the, the swamp, the data swamp,
- 00:15:35,968 --> 00:15:38,128: to the data lake, to the super warehouse,
- 00:15:38,208 --> 00:15:40,208: right? To the... Right. So it's, uh,
- 00:15:40,348 --> 00:15:42,368: it's like the full circle of things. But no, it's, yeah,
- 00:15:42,408 --> 00:15:43,968: it's getting to where you actually can,
- 00:15:44,568 --> 00:15:45,228: you know, have the
- 00:15:45,508 --> 00:15:46,468: right data sets and
- 00:15:47,048 --> 00:15:47,588: components
- 00:15:47,868 --> 00:15:48,628: as close to the,
- 00:15:48,908 --> 00:15:50,888: you know, consumption layer as possible.
- 00:15:51,428 --> 00:15:53,528: Yeah. Data swamp is a nice way of saying it. I
- 00:15:53,588 --> 00:15:54,708: used to call it data sewer.
- 00:15:54,968 --> 00:15:57,068: [laughs] So,
- 00:15:57,728 --> 00:15:58,588: uh, question.
- 00:15:58,848 --> 00:16:00,388: Uh, not question, but advice. So
- 00:16:00,688 --> 00:16:02,328: if people in the audience, and
- 00:16:02,708 --> 00:16:03,968: again, talking to people,
- 00:16:04,288 --> 00:16:05,728: people on different levels,
- 00:16:06,168 --> 00:16:08,228: uh, of not even maturity, but just
- 00:16:08,268 --> 00:16:09,508: approaching where they are.
- 00:16:10,008 --> 00:16:10,808: So if they
- 00:16:11,328 --> 00:16:12,128: are starting
- 00:16:12,488 --> 00:16:14,688: on ev-evolving their data management,
- 00:16:14,748 --> 00:16:16,228: data governance discipline,
- 00:16:17,028 --> 00:16:18,408: uh, towards AI,
- 00:16:18,808 --> 00:16:20,808: what would be the one thing you would advise them
- 00:16:20,848 --> 00:16:23,048: to do? One thing,
- 00:16:23,188 --> 00:16:25,448: uh, one thing that's pretty hard to boil down, but I think recognizing
- 00:16:25,488 --> 00:16:26,628: we're all on a learning curve
- 00:16:27,208 --> 00:16:29,228: is probably the one overall thing. And,
- 00:16:29,248 --> 00:16:30,608: and part of that learning curve
- 00:16:31,008 --> 00:16:33,448: is we're, we're all at different levels of understanding literacy.
- 00:16:33,508 --> 00:16:34,108: I, I love the,
- 00:16:34,688 --> 00:16:36,988: the, um, the comment that Felix made earlier about,
- 00:16:37,568 --> 00:16:38,387: you know, pointing
- 00:16:38,668 --> 00:16:40,688: AI at data gets you very low accuracy. I
- 00:16:40,708 --> 00:16:42,628: think, I think it was Anthropic's study that said 20,
- 00:16:43,108 --> 00:16:43,788: 22%, right?
- 00:16:44,108 --> 00:16:46,188: So having that
- 00:16:46,388 --> 00:16:47,048: literacy,
- 00:16:47,328 --> 00:16:48,088: um, ha-having your,
- 00:16:48,548 --> 00:16:50,568: your peers, your stakeholders, and the community
- 00:16:50,608 --> 00:16:51,908: that needs that data to understand
- 00:16:52,308 --> 00:16:54,348: a little bit more about the world we live in right now,
- 00:16:54,888 --> 00:16:56,088: uh, really important. And then
- 00:16:56,548 --> 00:16:57,168: a lot of the,
- 00:16:57,468 --> 00:16:58,528: uh, the foundational
- 00:16:59,428 --> 00:17:00,508: muscles that I mentioned earlier-
- 00:17:00,808 --> 00:17:02,468: Mm-hmm... uh, focusing on, on them,
- 00:17:03,108 --> 00:17:04,668: uh, enough, but also giving that,
- 00:17:04,988 --> 00:17:06,708: finding that freedom to experiment and to fail,
- 00:17:07,168 --> 00:17:09,628: right? Mm-hmm. So, so there's a lot of, a lot of cultural investment
- 00:17:09,668 --> 00:17:11,688: that I think, uh, really pays off that it
- 00:17:11,728 --> 00:17:12,748: may be invisible,
- 00:17:13,028 --> 00:17:14,588: um, in most, most programs. Mm-hmm.
- 00:17:15,388 --> 00:17:16,848: Okay. And I would just say,
- 00:17:17,128 --> 00:17:19,328: um, to Tom's point, is
- 00:17:19,968 --> 00:17:22,448: get your teams to spend more time in that ownership
- 00:17:22,568 --> 00:17:24,508: space, 'cause that definitely is something that
- 00:17:24,808 --> 00:17:27,068: I ca- I've yet to s- know it hasn't been solved
- 00:17:27,108 --> 00:17:28,748: in the financial services industry.
- 00:17:29,048 --> 00:17:31,208: Mm-hmm. I don't know that it's solved in the healthcare,
- 00:17:31,268 --> 00:17:34,028: right? Like, a-a-any of... as I talk cross-industry
- 00:17:34,088 --> 00:17:35,728: and to lots of peers,
- 00:17:36,288 --> 00:17:37,488: you know, in all different markets,
- 00:17:38,148 --> 00:17:40,148: um, is that would be my advice is, right,
- 00:17:40,228 --> 00:17:40,928: the more you can
- 00:17:41,328 --> 00:17:43,508: look in that s- white space, right, for what works
- 00:17:43,568 --> 00:17:45,648: for your company and how to solve for that. Because,
- 00:17:46,428 --> 00:17:48,468: um, the stewarding component of
- 00:17:48,528 --> 00:17:50,788: this is just gonna get more complex and-
- 00:17:50,888 --> 00:17:51,588: Mm-hmm... um,
- 00:17:52,088 --> 00:17:54,208: you know, take a hard lens, 'cause what it'll come
- 00:17:54,248 --> 00:17:56,028: down to is, in the end, it's [chuckles]
- 00:17:56,388 --> 00:17:58,388: what's been auth-authorized, right, to
- 00:17:58,488 --> 00:17:59,608: access. And, um,
- 00:18:00,348 --> 00:18:02,388: there, there's a l- you'll have a lot of friction that
- 00:18:02,508 --> 00:18:04,348: starts forming, right, as, as people,
- 00:18:04,648 --> 00:18:05,448: you know, agents-
- 00:18:06,188 --> 00:18:08,268: Yeah... ge-go to the hundred, to the thousands,
- 00:18:08,868 --> 00:18:09,868: um, and try to get,
- 00:18:10,148 --> 00:18:12,108: you know, unbounded access. And if you
- 00:18:12,388 --> 00:18:13,088: don't have that right,
- 00:18:13,428 --> 00:18:15,048: you're gonna- Yeah... have a world of hurt.
- 00:18:15,488 --> 00:18:17,468: Well, yeah. Who owns the outcome
- 00:18:18,088 --> 00:18:19,108: of the agent
- 00:18:19,508 --> 00:18:21,848: acting on the business process? And I've had some arguments
- 00:18:21,908 --> 00:18:22,828: with people and say, "Well,
- 00:18:23,248 --> 00:18:25,328: it's s- it's a program, so IT owns
- 00:18:25,388 --> 00:18:27,648: it." But that's not true for programs either,
- 00:18:27,728 --> 00:18:30,028: because there is always a business owner of the system.
- 00:18:30,388 --> 00:18:31,708: And I think for the agent that
- 00:18:32,108 --> 00:18:33,708: acts in the business process,
- 00:18:34,088 --> 00:18:36,448: the owner of the business process owns the agent.
- 00:18:37,128 --> 00:18:39,348: But it's easy for me to say from the stage,
- 00:18:40,188 --> 00:18:41,348: make it [laughs]
- 00:18:41,688 --> 00:18:44,148: all right, but it's much harder
- 00:18:44,348 --> 00:18:46,388: to make it happen in the organization.
- 00:18:46,848 --> 00:18:49,168: Because it's new set of accountabilities
- 00:18:49,468 --> 00:18:51,488: and a new set of uncertainty, and also,
- 00:18:52,508 --> 00:18:54,688: there is also unknown risk at
- 00:18:54,728 --> 00:18:55,268: this point.
- 00:18:56,368 --> 00:18:58,408: Eh, so I think you're absolutely right. I
- 00:18:58,468 --> 00:19:00,588: think thinking through the ownerships,
- 00:19:00,648 --> 00:19:02,268: especially as you're moving into agents,
- 00:19:02,588 --> 00:19:03,788: who owns the outcome,
- 00:19:04,688 --> 00:19:06,848: and who is responsible for setting
- 00:19:06,908 --> 00:19:08,808: limits on those outcomes, and a- so on.
- 00:19:09,268 --> 00:19:10,048: So yeah. Well,
- 00:19:10,908 --> 00:19:12,948: uh, we don't have to... it's, uh, we, we only
- 00:19:12,988 --> 00:19:15,208: have 20 minutes, and we have, like, 10 seconds left.
- 00:19:15,228 --> 00:19:17,328: [laughs] So no time for questions, but
- 00:19:17,388 --> 00:19:18,688: all of us are going to be,
- 00:19:18,968 --> 00:19:20,048: after the next session,
- 00:19:20,768 --> 00:19:22,628: in there. So stop by and, uh,
- 00:19:23,148 --> 00:19:25,268: ask questions, and we can have a conversation. Thank you.
Customer fireside hosted by Julia Bardmesser
As AI adoption accelerates, governance is more important than ever. In this candid discussion hosted by Julia Bardmesser, Christin Brown (Macquarie Asset Management) and Drew Palsgrove (Pfizer) explore the key strategies for successfully governing and scaling enterprise AI.
Video: OpenAI: From pilot to platform - The enterprise AI adoption journey
Transcript
- 00:00:06,080 --> 00:00:08,100: Thank you guys so much for being here. Before we get started, just
- 00:00:08,119 --> 00:00:10,140: thanks to Chirag, Lisa, Felix, the whole
- 00:00:10,220 --> 00:00:11,300: Fabel team for having us
- 00:00:11,640 --> 00:00:13,760: participate. It's been such a great event and such
- 00:00:13,800 --> 00:00:14,580: a great conversation
- 00:00:15,220 --> 00:00:17,540: all day. Um, quickly intro myself.
- 00:00:17,620 --> 00:00:19,880: My name's Reid Sheppard. I'm part of the go-to-market
- 00:00:19,940 --> 00:00:20,900: team at OpenAI.
- 00:00:21,440 --> 00:00:22,060: Uh, my colleague
- 00:00:22,380 --> 00:00:24,520: Smit Shah is part of our Applied AI
- 00:00:24,580 --> 00:00:25,360: Architecture team.
- 00:00:25,980 --> 00:00:27,180: Uh, so we spend all day,
- 00:00:27,500 --> 00:00:29,640: every day helping customers deploy high
- 00:00:29,700 --> 00:00:31,200: impact, high scope,
- 00:00:31,700 --> 00:00:33,380: uh, enterprise deployments.
- 00:00:33,940 --> 00:00:35,740: And so our session today
- 00:00:36,220 --> 00:00:36,960: focuses on
- 00:00:37,760 --> 00:00:38,840: how organizations,
- 00:00:39,200 --> 00:00:41,460: like the strategies and the investments they're making,
- 00:00:41,580 --> 00:00:42,740: both from an organizational
- 00:00:43,280 --> 00:00:44,220: structure perspective
- 00:00:44,720 --> 00:00:46,880: and also from a context layer
- 00:00:47,220 --> 00:00:47,780: standpoint,
- 00:00:48,120 --> 00:00:49,140: to make sure that those,
- 00:00:49,620 --> 00:00:50,620: uh, AI deployments
- 00:00:51,380 --> 00:00:52,760: output highly accurate,
- 00:00:53,220 --> 00:00:54,240: uh, highly trustworthy
- 00:00:54,780 --> 00:00:56,920: information that is respecting the governance
- 00:00:57,040 --> 00:00:59,140: apparatuses that your organizations have.
- 00:00:59,780 --> 00:01:01,860: Um, I do realize that we also have the
- 00:01:01,920 --> 00:01:04,140: time-honored and prestigious role
- 00:01:04,200 --> 00:01:06,560: of being the last thing standing between this audience
- 00:01:06,600 --> 00:01:07,240: and happy hour.
- 00:01:08,000 --> 00:01:10,460: Um, so we will try to make this crisp
- 00:01:10,640 --> 00:01:12,800: and, uh, share some learnings, uh, that we've
- 00:01:12,880 --> 00:01:14,360: seen at organizations
- 00:01:14,740 --> 00:01:16,100: along the way. Um, and
- 00:01:16,740 --> 00:01:18,840: I-- we're gonna pay spec-specific attention
- 00:01:18,880 --> 00:01:20,880: to the role that we see data and governance
- 00:01:20,980 --> 00:01:22,240: teams playing
- 00:01:22,740 --> 00:01:24,920: in making sure that these enterprise
- 00:01:25,080 --> 00:01:25,880: AI rollouts,
- 00:01:26,420 --> 00:01:28,940: uh, maintain high quality, they get adoption,
- 00:01:28,980 --> 00:01:31,100: they're earning trust, and they're delivering the value,
- 00:01:31,520 --> 00:01:33,060: um, that organizations expect
- 00:01:33,320 --> 00:01:34,660: from, from these types of rollouts.
- 00:01:35,120 --> 00:01:36,380: So, um, before
- 00:01:36,800 --> 00:01:38,840: we talk about some of the strategies and
- 00:01:39,000 --> 00:01:40,880: the investments, I think it's important to ground,
- 00:01:41,520 --> 00:01:43,720: um, ourselves in like the different types
- 00:01:43,820 --> 00:01:46,140: of buckets that enterprises are deploying
- 00:01:46,820 --> 00:01:48,080: AI solutions.
- 00:01:48,640 --> 00:01:49,560: So, um,
- 00:01:50,620 --> 00:01:52,620: individual productivity, team productivity,
- 00:01:52,780 --> 00:01:53,740: workflow automation,
- 00:01:54,180 --> 00:01:56,200: enterprise innovation. Individual productivity,
- 00:01:56,500 --> 00:01:58,860: uh, I think this is pretty well understood
- 00:01:58,900 --> 00:02:00,900: at the enterprise and the individual level in terms
- 00:02:00,960 --> 00:02:01,980: of what this means.
- 00:02:02,440 --> 00:02:03,580: Uh, not to say that this is
- 00:02:04,200 --> 00:02:06,200: static, right? This is dynamic. It's changing. I
- 00:02:06,240 --> 00:02:08,259: think when AI first came out, it was
- 00:02:08,300 --> 00:02:10,160: like, "Hey, chat, help me brainstorm,
- 00:02:10,759 --> 00:02:11,980: uh, something. Help me,
- 00:02:12,480 --> 00:02:13,340: uh, draft,
- 00:02:13,640 --> 00:02:15,640: uh, an e-- or summarize a document or
- 00:02:15,700 --> 00:02:16,460: draft an email."
- 00:02:17,100 --> 00:02:19,260: Um, more recently it's become like, "Hey,
- 00:02:19,900 --> 00:02:22,000: chat, Codex," whatever your agentic coding
- 00:02:22,040 --> 00:02:24,180: tool is, "pull context across a
- 00:02:24,220 --> 00:02:25,260: lot of different systems,
- 00:02:25,620 --> 00:02:27,820: uh, and work towards some sort of end goal," whether it's,
- 00:02:28,180 --> 00:02:30,500: uh, building a presentation or, uh, preparing
- 00:02:30,560 --> 00:02:31,540: a data analysis
- 00:02:31,860 --> 00:02:33,400: from an inter-internal perspective.
- 00:02:33,840 --> 00:02:35,500: And now in the last couple of weeks with
- 00:02:35,920 --> 00:02:36,620: always-on
- 00:02:37,060 --> 00:02:37,720: persistent
- 00:02:38,020 --> 00:02:39,800: AI, we can now not act-- They
- 00:02:40,200 --> 00:02:41,040: learn how you work.
- 00:02:41,480 --> 00:02:43,560: They don't actually have to be prompted, and then it can
- 00:02:43,800 --> 00:02:46,020: surface items that need-- require
- 00:02:46,040 --> 00:02:46,780: your attention and,
- 00:02:47,260 --> 00:02:49,480: um, you know, help you work more efficiently. So
- 00:02:49,860 --> 00:02:50,860: this is dynamic, but I think
- 00:02:51,200 --> 00:02:52,740: well understood in the context of like
- 00:02:53,480 --> 00:02:55,680: how enterprises deploy these types
- 00:02:55,780 --> 00:02:57,480: of AI solutions and how they measure,
- 00:02:58,060 --> 00:02:58,840: uh, its impact.
- 00:03:00,500 --> 00:03:01,820: So team productivity,
- 00:03:02,520 --> 00:03:03,180: uh, is really
- 00:03:03,640 --> 00:03:04,440: the lens of like
- 00:03:04,840 --> 00:03:06,480: when an organization identifies
- 00:03:06,760 --> 00:03:08,460: a high leverage,
- 00:03:08,800 --> 00:03:10,320: very time-consuming task,
- 00:03:10,900 --> 00:03:12,120: um, where
- 00:03:12,400 --> 00:03:13,980: having access to consistent,
- 00:03:14,300 --> 00:03:14,860: reliable,
- 00:03:15,320 --> 00:03:17,080: uh, repeatable AI outputs
- 00:03:17,480 --> 00:03:19,580: can provide a lot of time back to the business
- 00:03:19,620 --> 00:03:20,260: and help folks,
- 00:03:20,680 --> 00:03:21,980: uh, spend their energy on more
- 00:03:22,440 --> 00:03:24,720: higher leverage, higher business development-driving,
- 00:03:24,980 --> 00:03:26,100: uh, types of opportunities.
- 00:03:26,520 --> 00:03:28,620: And so a great example of this is a customer of ours,
- 00:03:28,740 --> 00:03:29,440: Morgan Stanley.
- 00:03:29,980 --> 00:03:32,140: And so what Morgan Stanley identified
- 00:03:32,340 --> 00:03:33,940: is that they had customers,
- 00:03:34,380 --> 00:03:36,400: or they had financial advisors rather, who were
- 00:03:36,460 --> 00:03:37,680: spending a ton of time
- 00:03:38,300 --> 00:03:39,320: preparing for
- 00:03:39,700 --> 00:03:40,600: client conversations.
- 00:03:41,120 --> 00:03:43,240: And they would get com-- they would get questions from
- 00:03:43,280 --> 00:03:45,640: their clients like, "If there's a shock
- 00:03:46,160 --> 00:03:47,680: in the energy supply,
- 00:03:48,080 --> 00:03:49,380: how is that going to impact
- 00:03:49,780 --> 00:03:51,220: growth and inflation across,
- 00:03:51,660 --> 00:03:52,680: uh, a variety of different
- 00:03:52,940 --> 00:03:53,680: market segments,
- 00:03:54,180 --> 00:03:56,360: and how can I adjust my portfolio
- 00:03:56,420 --> 00:03:58,600: accordingly?" Or a question like,
- 00:03:59,100 --> 00:04:01,260: uh, "AI spending is great for stocks,
- 00:04:01,380 --> 00:04:02,640: but how does that affect,
- 00:04:03,020 --> 00:04:03,920: uh, the bond market?"
- 00:04:04,280 --> 00:04:06,560: And so these are like nuanced, detailed questions
- 00:04:06,620 --> 00:04:07,260: that require,
- 00:04:07,780 --> 00:04:08,520: uh, a lot of
- 00:04:09,020 --> 00:04:10,480: business context and
- 00:04:11,120 --> 00:04:12,920: items from a bunch of different documents.
- 00:04:13,440 --> 00:04:15,200: And so what Morgan Stanley did was they
- 00:04:15,620 --> 00:04:17,640: stored a corpus of documents, uh, a hundred
- 00:04:17,720 --> 00:04:18,540: thousand documents,
- 00:04:19,000 --> 00:04:21,000: uh, in a, a very organized
- 00:04:21,019 --> 00:04:21,740: and regimented
- 00:04:22,140 --> 00:04:22,880: context layer
- 00:04:23,260 --> 00:04:24,900: that their financial advisors could
- 00:04:25,280 --> 00:04:27,640: query, uh, and they would get back specific,
- 00:04:28,080 --> 00:04:30,280: um, synthesized insights that they could share,
- 00:04:30,860 --> 00:04:32,440: uh, directly with their clients. So
- 00:04:32,900 --> 00:04:34,320: saved, uh, a ton of time.
- 00:04:34,780 --> 00:04:36,960: Uh, and again, this is kind of like an example of,
- 00:04:37,420 --> 00:04:39,680: uh, deploying AI at
- 00:04:39,740 --> 00:04:41,980: the, uh, team, team productivity level.
- 00:04:42,600 --> 00:04:44,900: Um, so we've talked a little bit about individual productivity.
- 00:04:44,980 --> 00:04:45,680: We've talked about
- 00:04:46,280 --> 00:04:47,100: team productivity.
- 00:04:47,700 --> 00:04:49,860: What do workflow automation and enterprise innovation
- 00:04:49,900 --> 00:04:51,960: mean? So workflow automation is kind of the
- 00:04:52,000 --> 00:04:53,040: next logical step.
- 00:04:53,380 --> 00:04:54,720: Once you've gotten to the point of,
- 00:04:55,240 --> 00:04:57,620: hey, it's not just individual users getting
- 00:04:57,700 --> 00:05:00,000: a little bit of efficiency from using a chatbot,
- 00:05:00,620 --> 00:05:01,640: uh, you've moved into,
- 00:05:01,920 --> 00:05:02,540: hey, we've got
- 00:05:02,900 --> 00:05:04,300: something that the entire team can use.
- 00:05:04,840 --> 00:05:06,920: Now we're gonna talk about workflow automation, which is we're
- 00:05:06,940 --> 00:05:08,740: gonna stitch those individual pieces together
- 00:05:09,280 --> 00:05:10,480: into something that lets you automate
- 00:05:11,120 --> 00:05:13,620: a big chunk or entire end-to-end workflows.
- 00:05:14,200 --> 00:05:16,400: Uh, and then enterprise innovation is taking it
- 00:05:16,480 --> 00:05:17,560: even one step beyond that,
- 00:05:18,080 --> 00:05:19,700: uh, giving your, your organization
- 00:05:20,180 --> 00:05:21,380: new capabilities that you
- 00:05:21,800 --> 00:05:24,020: didn't have at all prior to your organization's
- 00:05:24,100 --> 00:05:25,000: use of AI.
- 00:05:25,580 --> 00:05:27,680: Uh, so it's a good segue into another one of our
- 00:05:27,720 --> 00:05:29,360: customers, uh, BNY.
- 00:05:30,060 --> 00:05:32,080: Uh, what they did, you know, they, they had similar
- 00:05:32,260 --> 00:05:34,020: use cases to Morgan Stanley in terms of,
- 00:05:34,480 --> 00:05:36,540: hey, they set up a, an, a corpus of knowledge
- 00:05:36,600 --> 00:05:38,200: that users could query. They set up,
- 00:05:38,800 --> 00:05:41,160: uh, individual workflows that were streamlined
- 00:05:41,200 --> 00:05:41,980: with the help of AI.
- 00:05:42,400 --> 00:05:43,060: But then they said,
- 00:05:43,540 --> 00:05:45,100: "Actually, we wanna scale this even
- 00:05:45,420 --> 00:05:46,340: more than we already have."
- 00:05:46,720 --> 00:05:48,300: And what they did is they built a platform
- 00:05:48,820 --> 00:05:49,360: that allowed
- 00:05:49,700 --> 00:05:51,800: individual teams, individual users to create
- 00:05:51,840 --> 00:05:53,060: their own agents, and not just
- 00:05:53,520 --> 00:05:55,460: create their own agents, but they, they realized,
- 00:05:55,940 --> 00:05:57,900: hey, there are common workflows across teams,
- 00:05:58,320 --> 00:06:00,500: or there are pieces of workflows that are common across
- 00:06:00,560 --> 00:06:02,620: teams. So if one team builds-
- 00:06:03,380 --> 00:06:05,580: An agent that will streamline a single workflow,
- 00:06:05,960 --> 00:06:08,200: we wanna be able to scale that across other teams. So
- 00:06:08,500 --> 00:06:09,620: we want them to be able to reuse
- 00:06:10,120 --> 00:06:10,820: the pieces
- 00:06:11,080 --> 00:06:12,780: of those agents. So they built this platform.
- 00:06:13,140 --> 00:06:15,260: People can see what agents have been built across their
- 00:06:15,340 --> 00:06:16,500: team, across other teams,
- 00:06:17,020 --> 00:06:19,140: and then they could take bits and pieces of those agents
- 00:06:19,180 --> 00:06:20,020: that are already been built
- 00:06:20,640 --> 00:06:22,740: and remix those, rebuild those into
- 00:06:22,760 --> 00:06:23,820: something that's more specific
- 00:06:24,200 --> 00:06:26,220: for their team or that individual's workflows.
- 00:06:29,260 --> 00:06:31,360: So shifting gears a bit. So now that we're
- 00:06:31,680 --> 00:06:32,940: aligned on, like, the different
- 00:06:33,200 --> 00:06:34,240: scope in terms of,
- 00:06:34,640 --> 00:06:37,160: like, the ambition of different enterprise AI deployments
- 00:06:37,200 --> 00:06:38,220: across the organization,
- 00:06:38,860 --> 00:06:40,920: uh, wanna narrow down on two things that we
- 00:06:40,980 --> 00:06:42,940: see organizations do well repeatably
- 00:06:43,400 --> 00:06:43,920: that enable
- 00:06:44,380 --> 00:06:45,120: high quality,
- 00:06:45,440 --> 00:06:46,620: high leverage, high impact,
- 00:06:46,900 --> 00:06:47,840: uh, enterprise solutions.
- 00:06:48,280 --> 00:06:50,120: So one is organizational structure.
- 00:06:50,520 --> 00:06:52,820: So what we mean when we say this is thinking really carefully
- 00:06:52,880 --> 00:06:54,260: about how teams
- 00:06:54,860 --> 00:06:55,520: brainstorm,
- 00:06:55,840 --> 00:06:56,540: identify,
- 00:06:56,960 --> 00:06:59,040: prioritize, and actually execute and
- 00:06:59,120 --> 00:07:00,700: implement AI solutions.
- 00:07:01,080 --> 00:07:02,100: Um, there's a couple different
- 00:07:02,480 --> 00:07:04,500: formations that we, we see
- 00:07:04,540 --> 00:07:06,000: taking place across the enterprise, and
- 00:07:06,340 --> 00:07:07,940: we'll get into what those look like. But I think
- 00:07:08,520 --> 00:07:10,660: the key point here is, like, making sure that you think
- 00:07:10,700 --> 00:07:13,080: through what this looks like for your organization
- 00:07:13,120 --> 00:07:14,500: and iterate to make sure that,
- 00:07:15,120 --> 00:07:17,140: um, you're optimizing, like, how teams
- 00:07:17,180 --> 00:07:19,380: are organized around deploying these capabilities,
- 00:07:19,980 --> 00:07:22,100: um, is something important, something worth spending
- 00:07:22,140 --> 00:07:24,240: time on. Uh, and then the next, I, I
- 00:07:24,280 --> 00:07:25,080: guess we've been talking about it
- 00:07:25,340 --> 00:07:26,120: all day, is
- 00:07:26,520 --> 00:07:27,320: context. So
- 00:07:27,780 --> 00:07:28,380: making sure
- 00:07:28,780 --> 00:07:31,100: that there's a system in place,
- 00:07:31,360 --> 00:07:33,360: um, there's scaffolding, there's engineering,
- 00:07:33,400 --> 00:07:34,240: and there's process
- 00:07:34,720 --> 00:07:35,660: around how to take
- 00:07:35,980 --> 00:07:37,740: enterprise data assets and your
- 00:07:38,600 --> 00:07:40,480: unstructured and unstructured data
- 00:07:40,860 --> 00:07:42,960: and make sure that that is organized in a way that's
- 00:07:43,020 --> 00:07:45,280: available to AI agents so they're surfacing
- 00:07:45,720 --> 00:07:47,880: the right information, uh, for the right questions
- 00:07:47,920 --> 00:07:48,440: in a way that's
- 00:07:48,820 --> 00:07:49,940: valuable and gonna actually
- 00:07:50,380 --> 00:07:50,940: add, um--
- 00:07:51,500 --> 00:07:52,340: gonna impact,
- 00:07:52,620 --> 00:07:54,800: uh, positively the, the business folks
- 00:07:54,840 --> 00:07:57,180: who are, who are querying it, and that's how you get adoption
- 00:07:57,220 --> 00:07:59,520: and, you know, makes all, all of this effort worthwhile.
- 00:08:00,040 --> 00:08:00,680: Um, so,
- 00:08:01,120 --> 00:08:03,160: you know, obviously we're talking to data governance folks
- 00:08:03,200 --> 00:08:03,820: for the most part,
- 00:08:04,480 --> 00:08:06,520: um, and we wanna talk data governance, but also
- 00:08:06,560 --> 00:08:08,740: we're t-- why do I have organizational structure
- 00:08:08,800 --> 00:08:11,240: up here? How successful
- 00:08:11,280 --> 00:08:11,900: an organization
- 00:08:12,480 --> 00:08:14,500: is in their adoption of AI is going to
- 00:08:14,560 --> 00:08:15,680: come down, to some extent,
- 00:08:16,100 --> 00:08:17,500: to how you've structured your teams.
- 00:08:17,960 --> 00:08:19,280: And what we've seen working with
- 00:08:19,600 --> 00:08:20,900: all of our customers here at OpenAI,
- 00:08:22,060 --> 00:08:24,120: for the most part, our customers fall into one of these
- 00:08:24,160 --> 00:08:26,320: three buckets in terms of how they've structured
- 00:08:26,700 --> 00:08:28,860: their organization or at least the part of the organization
- 00:08:28,900 --> 00:08:30,900: that owns AI adoption and AI
- 00:08:30,960 --> 00:08:31,900: solutions internally.
- 00:08:32,640 --> 00:08:33,180: Um, so
- 00:08:33,800 --> 00:08:35,900: we've, we've seen a lot of organizations use a
- 00:08:35,960 --> 00:08:37,040: central center of excellence,
- 00:08:37,560 --> 00:08:39,580: where there's this AI center of excellence that
- 00:08:39,680 --> 00:08:40,680: operates org-wide,
- 00:08:41,220 --> 00:08:41,840: and they have
- 00:08:42,440 --> 00:08:44,560: executive sponsors on each team, but
- 00:08:44,580 --> 00:08:46,680: really it's that one central center of excellence that
- 00:08:46,780 --> 00:08:48,800: owns what are we gonna build and how are we gonna
- 00:08:48,840 --> 00:08:50,820: roll it out. We've also seen,
- 00:08:51,100 --> 00:08:53,120: um... I'm gonna actually skip over number two and go to number
- 00:08:53,180 --> 00:08:55,200: three. We've seen a lot of companies do
- 00:08:55,260 --> 00:08:57,320: this, like, team-specific or line of
- 00:08:57,360 --> 00:08:58,220: business-specific
- 00:08:58,700 --> 00:08:59,700: AI teams. So they--
- 00:09:00,280 --> 00:09:02,320: It's almost like a center of excellence embedded within
- 00:09:02,400 --> 00:09:03,380: each line of business
- 00:09:03,780 --> 00:09:06,100: as opposed to one org-wide center of excellence.
- 00:09:06,220 --> 00:09:08,320: And then, uh, going back to number two, we
- 00:09:08,380 --> 00:09:10,380: have kind of the hybrid approach where there is
- 00:09:10,420 --> 00:09:12,820: that, still that central AI center of excellence,
- 00:09:13,260 --> 00:09:15,540: but there are AI champions that are embedded
- 00:09:15,600 --> 00:09:17,660: within each line of business.
- 00:09:18,240 --> 00:09:18,900: Uh, there isn't
- 00:09:19,560 --> 00:09:21,200: one right way to do this.
- 00:09:21,560 --> 00:09:23,640: Uh, I know that's, that's a bit of a cop-out answer, so if you're
- 00:09:23,660 --> 00:09:25,720: looking for a little more guidance in, in
- 00:09:25,760 --> 00:09:27,760: terms of, hey, what is gonna work well for me,
- 00:09:28,420 --> 00:09:29,980: really what's important is to think about
- 00:09:30,700 --> 00:09:31,500: how is change
- 00:09:31,800 --> 00:09:33,540: typically done in your org. So
- 00:09:34,080 --> 00:09:35,040: are you more of a
- 00:09:35,540 --> 00:09:37,180: top-down driven change
- 00:09:37,740 --> 00:09:38,600: type of organization?
- 00:09:39,440 --> 00:09:41,580: A central center of excellence is probably gonna work
- 00:09:41,620 --> 00:09:43,740: better. Are you more of a bottoms-up adoption
- 00:09:43,820 --> 00:09:45,000: type of, of organization?
- 00:09:45,420 --> 00:09:47,420: The domain-specific teams are probably gonna
- 00:09:47,440 --> 00:09:48,280: work a little bit better.
- 00:09:48,660 --> 00:09:50,540: But again, it, it's really getting a sense of
- 00:09:51,000 --> 00:09:53,020: how do our teams work best and how
- 00:09:53,120 --> 00:09:55,560: is change most effectively
- 00:09:56,000 --> 00:09:58,000: driven in our organization that's gonna drive
- 00:09:58,300 --> 00:09:59,160: how you set your teams up.
- 00:10:00,320 --> 00:10:01,580: So now
- 00:10:01,900 --> 00:10:02,520: focusing on
- 00:10:02,860 --> 00:10:04,880: the context piece of this enterprise investment.
- 00:10:04,940 --> 00:10:06,500: So it's funny, I was having a conversation with
- 00:10:07,040 --> 00:10:09,020: Felix earlier today at lunch, and he was saying,
- 00:10:09,720 --> 00:10:11,720: um, like, is OpenAI gonna have a product
- 00:10:11,860 --> 00:10:13,660: that, like, helps enterprises,
- 00:10:14,120 --> 00:10:16,200: uh, build this context layer and manage this
- 00:10:16,240 --> 00:10:16,800: context layer?
- 00:10:17,120 --> 00:10:19,020: And I think the answer to that six months ago might have been
- 00:10:19,600 --> 00:10:21,660: yes, and then we found out, like, this is way too
- 00:10:21,720 --> 00:10:24,360: hard because every enterprise and every organization's
- 00:10:24,380 --> 00:10:25,040: data is so
- 00:10:25,520 --> 00:10:26,060: bespoke.
- 00:10:26,520 --> 00:10:28,540: So this isn't a product. This is more of
- 00:10:28,580 --> 00:10:29,500: just a framework,
- 00:10:29,920 --> 00:10:32,020: um, that we apply and meld to
- 00:10:32,060 --> 00:10:33,460: different organizations based off,
- 00:10:33,780 --> 00:10:35,100: you know, what is gonna work for them.
- 00:10:35,940 --> 00:10:37,040: Um, but the six context
- 00:10:37,440 --> 00:10:38,960: layers you see here listed
- 00:10:39,580 --> 00:10:41,720: are i-- or is the paradigm that
- 00:10:41,780 --> 00:10:44,000: OpenAI used to build our own
- 00:10:44,060 --> 00:10:45,020: internal data agent,
- 00:10:45,360 --> 00:10:47,520: Kepler, which serves queries
- 00:10:47,680 --> 00:10:49,000: all across the business around,
- 00:10:49,520 --> 00:10:51,680: uh, how folks are using the product,
- 00:10:52,000 --> 00:10:54,020: what revenue is for different products. Um,
- 00:10:54,260 --> 00:10:55,080: I use it every day,
- 00:10:55,540 --> 00:10:57,720: um, trying to understand what's going on, uh, in my
- 00:10:57,760 --> 00:10:58,700: sphere of the world. So,
- 00:10:59,300 --> 00:11:01,320: um, real quickly, we'll, we'll run through kind
- 00:11:01,360 --> 00:11:03,460: of what these mean, these context layers mean for
- 00:11:03,500 --> 00:11:04,100: OpenAI,
- 00:11:04,700 --> 00:11:05,220: um, and then
- 00:11:05,500 --> 00:11:07,920: we'll double back to the underlined portions
- 00:11:07,980 --> 00:11:08,940: of this slide, which is where,
- 00:11:09,540 --> 00:11:11,900: uh, I see opportunity and where I see data and governance
- 00:11:12,000 --> 00:11:14,060: folks really, um, having a hand in
- 00:11:14,080 --> 00:11:14,720: making sure that
- 00:11:15,160 --> 00:11:17,220: this part of, of enterprise
- 00:11:17,300 --> 00:11:18,620: context is well curated.
- 00:11:19,220 --> 00:11:20,560: Um, so we've seen, I feel like,
- 00:11:20,960 --> 00:11:23,180: um, similar slides to this earlier, uh,
- 00:11:23,320 --> 00:11:24,600: i-in the day, but,
- 00:11:25,220 --> 00:11:27,260: um, the, the different context layers that
- 00:11:27,300 --> 00:11:29,360: OpenAI thinks about when constructing this
- 00:11:29,420 --> 00:11:30,260: data agent is
- 00:11:30,760 --> 00:11:31,820: table usage. So it's like,
- 00:11:32,260 --> 00:11:33,680: where is the data living?
- 00:11:34,120 --> 00:11:36,240: Uh, where is the data that's relevant to serve
- 00:11:36,380 --> 00:11:37,220: specific queries,
- 00:11:37,640 --> 00:11:39,720: um, for different lines of business and different personas
- 00:11:40,060 --> 00:11:40,740: who are going to be
- 00:11:41,220 --> 00:11:41,940: asking questions?
- 00:11:42,540 --> 00:11:44,000: Um, expert annotations.
- 00:11:44,380 --> 00:11:46,400: This is metadata. So, like, what do these columns
- 00:11:46,500 --> 00:11:47,420: mean? Um,
- 00:11:48,180 --> 00:11:49,820: making sure that the AI agent has a really
- 00:11:50,300 --> 00:11:51,660: deep understanding of, like,
- 00:11:52,260 --> 00:11:53,680: whe-where to actually
- 00:11:53,980 --> 00:11:55,900: pull data from these different columns and
- 00:11:56,320 --> 00:11:57,540: why it's, it's doing so.
- 00:11:58,260 --> 00:11:59,800: Um, code enrichment is
- 00:12:00,120 --> 00:12:01,580: making available to the data,
- 00:12:02,140 --> 00:12:04,340: uh, or making available to the agent the context
- 00:12:04,380 --> 00:12:05,220: of how this data
- 00:12:05,640 --> 00:12:07,620: was compiled. So if data is compiled,
- 00:12:08,080 --> 00:12:10,320: uh, in a certain table on a daily, weekly,
- 00:12:10,420 --> 00:12:11,180: monthly basis,
- 00:12:11,540 --> 00:12:13,600: uh, it's important for the AI agent to know that and be
- 00:12:13,640 --> 00:12:14,380: able to caveat
- 00:12:14,920 --> 00:12:17,000: an answer if someone's like, "Hey, what is,
- 00:12:17,460 --> 00:12:19,720: uh, the latest happenings in the last hour
- 00:12:19,800 --> 00:12:21,940: of sales for a retail company?"
- 00:12:22,040 --> 00:12:23,440: For, or for example,
- 00:12:24,040 --> 00:12:26,108: um- The, the AI needs to be able
- 00:12:26,128 --> 00:12:28,248: to qualify that answer that it retrieves with, "Hey,"
- 00:12:28,348 --> 00:12:30,068: like, "this table was only updated,
- 00:12:30,488 --> 00:12:31,928: uh, within the last week," and so
- 00:12:32,328 --> 00:12:33,928: not able to, like, accommodate
- 00:12:34,208 --> 00:12:35,908: specifically, like, the exact ask.
- 00:12:36,548 --> 00:12:38,608: Um, institutional knowledge, so this means
- 00:12:38,788 --> 00:12:40,388: unstructured data, so we're talking about
- 00:12:40,947 --> 00:12:43,048: Google Drive, Notion, Slack,
- 00:12:43,548 --> 00:12:44,128: SharePoint,
- 00:12:44,668 --> 00:12:45,948: uh, and making sure that,
- 00:12:46,748 --> 00:12:47,948: um, making sure that
- 00:12:48,388 --> 00:12:50,828: the AI agent has the context
- 00:12:50,888 --> 00:12:51,828: of how to operate
- 00:12:52,228 --> 00:12:53,328: within your business and
- 00:12:53,588 --> 00:12:55,528: how the business logic applies
- 00:12:55,828 --> 00:12:57,628: to the different tables and structured data
- 00:12:58,048 --> 00:12:58,708: that it's pulling.
- 00:12:59,828 --> 00:13:02,048: Um, memory you could think of as kind of like
- 00:13:02,248 --> 00:13:03,408: a, a constitution
- 00:13:03,688 --> 00:13:04,408: or a, um,
- 00:13:05,148 --> 00:13:07,468: a system prompt. Uh, it's really the result
- 00:13:07,588 --> 00:13:08,608: of, like, trial and error
- 00:13:09,088 --> 00:13:11,208: on an agent querying information and coming
- 00:13:11,248 --> 00:13:11,768: back with,
- 00:13:12,308 --> 00:13:13,528: um, specific results.
- 00:13:13,888 --> 00:13:15,248: Like, for example, if, um,
- 00:13:15,748 --> 00:13:16,588: you wanted to...
- 00:13:17,088 --> 00:13:19,108: If there were, like, analysts querying about, like,
- 00:13:19,188 --> 00:13:21,528: what the margin means for a particular line of business,
- 00:13:21,888 --> 00:13:23,628: like they could be refer-referring to
- 00:13:24,208 --> 00:13:26,308: operating margin, contribution margin, net
- 00:13:26,348 --> 00:13:27,928: interest margin, trading margin.
- 00:13:28,368 --> 00:13:30,428: And so only after understanding,
- 00:13:30,508 --> 00:13:32,668: like, what that means for particular
- 00:13:32,748 --> 00:13:34,388: personas within your organization
- 00:13:34,748 --> 00:13:36,868: can you then, in a markdown file or a system
- 00:13:36,968 --> 00:13:38,968: prompt, uh, instruct the AI
- 00:13:39,308 --> 00:13:41,708: to be able to make sure it's serving the right information
- 00:13:41,748 --> 00:13:42,468: to the right user.
- 00:13:43,208 --> 00:13:44,308: And then [chuckles] all that being said,
- 00:13:44,828 --> 00:13:46,948: um, there's the runtime context. And so this
- 00:13:46,988 --> 00:13:49,088: was maybe the single biggest needle
- 00:13:49,168 --> 00:13:49,948: mover in terms of
- 00:13:50,468 --> 00:13:52,368: how OpenAI delivered a,
- 00:13:52,928 --> 00:13:55,368: a lot more improved accuracy and results
- 00:13:55,428 --> 00:13:56,848: with its internal data AI ag-
- 00:13:57,128 --> 00:13:57,668: AI agent.
- 00:13:58,168 --> 00:14:00,288: They took all this context and moved it from pointing
- 00:14:00,588 --> 00:14:01,168: at a model
- 00:14:01,688 --> 00:14:02,908: to actually an agentic
- 00:14:03,428 --> 00:14:05,488: coding tool that has the pow- has the
- 00:14:05,528 --> 00:14:06,548: power to write code,
- 00:14:06,848 --> 00:14:07,828: it has its own computer,
- 00:14:08,148 --> 00:14:09,008: it can call tools,
- 00:14:09,348 --> 00:14:11,388: it can reason about the information that it pulls,
- 00:14:11,428 --> 00:14:12,828: and it can plan its next steps
- 00:14:13,388 --> 00:14:14,008: towards a goal.
- 00:14:14,508 --> 00:14:15,368: Um, if
- 00:14:15,768 --> 00:14:17,868: you do nothing else within your organizations, just
- 00:14:17,908 --> 00:14:18,788: making that switch,
- 00:14:19,268 --> 00:14:21,308: uh, is a huge needle mover in terms
- 00:14:21,368 --> 00:14:22,488: of, like, the quality of
- 00:14:22,788 --> 00:14:24,848: the outputs that you're gonna get from an AI agent.
- 00:14:25,228 --> 00:14:26,448: And yeah, and doubling back to
- 00:14:26,828 --> 00:14:28,988: where data and governance folks fit into
- 00:14:29,048 --> 00:14:31,088: this, I think expert annotations, we've been
- 00:14:31,128 --> 00:14:32,368: talking about it all day,
- 00:14:32,828 --> 00:14:34,908: um, this is a great place where folks
- 00:14:34,968 --> 00:14:37,048: can make sure the AI agent has context
- 00:14:37,108 --> 00:14:39,568: around what data means a-across
- 00:14:39,608 --> 00:14:41,228: different tables. It's a good place to,
- 00:14:41,768 --> 00:14:43,968: uh, put metadata catalogs and ontologies
- 00:14:44,028 --> 00:14:46,048: like, um, permissions to make sure that
- 00:14:46,088 --> 00:14:48,108: the right data is being served to the right folks.
- 00:14:48,588 --> 00:14:50,728: Uh, same goes for institutional knowledge. So
- 00:14:51,188 --> 00:14:52,708: if you have a framework that shows
- 00:14:53,088 --> 00:14:54,228: certain documents are
- 00:14:54,528 --> 00:14:56,628: internal only, certain documents are external
- 00:14:56,728 --> 00:14:57,248: o-only,
- 00:14:57,668 --> 00:14:59,668: you can provide an agent at this
- 00:14:59,728 --> 00:15:00,908: layer context around
- 00:15:01,328 --> 00:15:02,588: how it can categorize,
- 00:15:03,048 --> 00:15:03,688: um, different,
- 00:15:04,168 --> 00:15:06,568: different types of documents in, in that respect.
- 00:15:07,008 --> 00:15:07,848: And if there's,
- 00:15:08,628 --> 00:15:10,228: if there's a, uh, contradictory
- 00:15:10,548 --> 00:15:11,328: or an unclear
- 00:15:11,688 --> 00:15:12,988: document where it's not sure,
- 00:15:13,448 --> 00:15:15,508: um, this is the place where you would instruct the agent to
- 00:15:15,568 --> 00:15:16,708: say, like, "Okay,
- 00:15:16,988 --> 00:15:19,408: based on what I know about how this business operates,
- 00:15:19,688 --> 00:15:20,888: here's the qualification
- 00:15:21,248 --> 00:15:22,668: that I would offer around
- 00:15:23,148 --> 00:15:24,268: why, um,
- 00:15:24,608 --> 00:15:26,948: this m-might be internal or might be external
- 00:15:27,408 --> 00:15:29,668: facing." And so ano-another framework
- 00:15:29,708 --> 00:15:30,228: to think about
- 00:15:30,788 --> 00:15:32,908: this, this data agent
- 00:15:33,128 --> 00:15:34,708: is offline and
- 00:15:35,048 --> 00:15:35,948: online processing.
- 00:15:36,388 --> 00:15:38,388: So everything I said, the, the tables,
- 00:15:38,968 --> 00:15:40,628: um, the metadata about the tables,
- 00:15:41,088 --> 00:15:41,828: uh, how the datas are,
- 00:15:42,088 --> 00:15:43,108: how the data is compiled,
- 00:15:43,728 --> 00:15:44,588: unstructured data,
- 00:15:45,308 --> 00:15:46,068: and, um,
- 00:15:46,328 --> 00:15:47,588: memories or system prompts,
- 00:15:48,068 --> 00:15:50,108: all of this can be processed in some sort of
- 00:15:50,568 --> 00:15:51,188: offline
- 00:15:51,608 --> 00:15:52,308: cadence. So
- 00:15:52,768 --> 00:15:53,368: whether these
- 00:15:53,628 --> 00:15:55,028: layers are updated
- 00:15:55,448 --> 00:15:56,788: daily, hourly,
- 00:15:57,088 --> 00:15:58,308: weekly, monthly,
- 00:15:58,888 --> 00:16:00,928: uh, it's important to make sure that there's a system in
- 00:16:00,988 --> 00:16:02,348: place to make sure that those are
- 00:16:02,668 --> 00:16:03,548: not happening at
- 00:16:03,828 --> 00:16:05,788: runtime. Uh, at runtime,
- 00:16:06,228 --> 00:16:07,708: the agent only needs to
- 00:16:08,068 --> 00:16:09,968: query what the difference is between
- 00:16:10,388 --> 00:16:12,488: when the user is asking a question
- 00:16:12,988 --> 00:16:14,248: to when the last time
- 00:16:14,648 --> 00:16:15,408: all of this data
- 00:16:16,008 --> 00:16:16,608: was updated.
- 00:16:17,488 --> 00:16:19,908: Uh, and that is key in terms of driving accuracy,
- 00:16:20,608 --> 00:16:22,628: in terms of lowering latency, and making sure that
- 00:16:22,648 --> 00:16:23,348: the results of
- 00:16:24,068 --> 00:16:24,988: internal agents are
- 00:16:25,288 --> 00:16:26,548: remaining highly consistent.
- 00:16:27,048 --> 00:16:27,568: Structuring,
- 00:16:28,088 --> 00:16:29,968: uh, your context layers around,
- 00:16:30,588 --> 00:16:32,928: um, h- whatever's gonna work most importantly
- 00:16:32,968 --> 00:16:34,288: for your organization and keeping
- 00:16:34,708 --> 00:16:35,768: this framework in mind,
- 00:16:36,268 --> 00:16:38,568: uh, is how you get an agent to respond differently
- 00:16:38,608 --> 00:16:39,868: to different types of questions
- 00:16:40,308 --> 00:16:41,428: from folks. So you can have...
- 00:16:41,828 --> 00:16:43,408: Like, when I ask my data agent,
- 00:16:43,768 --> 00:16:44,307: "What's the
- 00:16:44,588 --> 00:16:46,908: profit and loss for my business unit?" It knows
- 00:16:47,368 --> 00:16:49,428: I'm not Sarah Friar, I'm not the CFO of
- 00:16:49,808 --> 00:16:50,448: OpenAI,
- 00:16:50,848 --> 00:16:53,048: uh, I don't have access to the tables and permissions
- 00:16:53,128 --> 00:16:55,148: that she has access to, and it's gonna give
- 00:16:55,208 --> 00:16:57,128: me a, a completely different answer,
- 00:16:57,568 --> 00:16:58,128: um, than
- 00:16:58,648 --> 00:17:00,488: someone in a different line of business or, or,
- 00:17:00,948 --> 00:17:02,408: uh, at a different seniority level.
- 00:17:03,068 --> 00:17:05,248: This slide is, is really just a PSA
- 00:17:05,328 --> 00:17:07,588: for evals, uh, and data governance
- 00:17:07,628 --> 00:17:09,328: folks' role, uh, within evals.
- 00:17:09,728 --> 00:17:11,148: And so I think evals, like,
- 00:17:11,948 --> 00:17:13,388: how, how many folks have heard of evals?
- 00:17:13,688 --> 00:17:15,028: Like most?
- 00:17:15,528 --> 00:17:17,628: Half? Yeah. I mean, it's a very buzzy
- 00:17:17,728 --> 00:17:18,828: term. I, I think it's
- 00:17:19,208 --> 00:17:21,228: really just a fancy word for trial and error.
- 00:17:21,808 --> 00:17:22,828: Um, and so
- 00:17:23,268 --> 00:17:23,848: the idea
- 00:17:24,208 --> 00:17:26,208: of an eval is before you deploy
- 00:17:26,248 --> 00:17:27,428: an A-AI agent,
- 00:17:28,288 --> 00:17:29,328: make sure that you define
- 00:17:29,768 --> 00:17:31,848: what success looks like for your organization.
- 00:17:31,908 --> 00:17:33,568: Like, what should an answer look like?
- 00:17:33,948 --> 00:17:35,428: Subject matter experts can do this,
- 00:17:35,708 --> 00:17:37,128: line of business users can do this,
- 00:17:37,548 --> 00:17:39,148: and data and governance folks can
- 00:17:39,468 --> 00:17:40,628: define, like,
- 00:17:41,008 --> 00:17:43,428: are folks seeing the right data? Are the permissions
- 00:17:43,508 --> 00:17:45,088: being respected by the agent? And
- 00:17:45,448 --> 00:17:46,028: is it pulling
- 00:17:46,488 --> 00:17:47,348: the right information?
- 00:17:47,748 --> 00:17:50,008: And the way to run an eval is you define that success
- 00:17:50,168 --> 00:17:52,248: upfront, um, you run the question,
- 00:17:52,348 --> 00:17:53,648: and then if there's a gap
- 00:17:53,908 --> 00:17:55,128: between what
- 00:17:55,668 --> 00:17:56,468: is a good outcome
- 00:17:57,008 --> 00:17:58,548: and what actually happens,
- 00:17:58,968 --> 00:18:01,188: you can go back and look at the trace or,
- 00:18:01,288 --> 00:18:03,328: or the spans of what the AI agent
- 00:18:03,368 --> 00:18:04,928: did, what tools did it call,
- 00:18:05,348 --> 00:18:06,408: what SQL did it run,
- 00:18:06,868 --> 00:18:08,208: what tables did it look at,
- 00:18:08,608 --> 00:18:09,828: uh, what metadata did it
- 00:18:10,088 --> 00:18:11,788: look at about the different tables, and
- 00:18:12,308 --> 00:18:14,368: diagnose the failure. And this is where you can
- 00:18:14,408 --> 00:18:15,188: iterate and make,
- 00:18:15,628 --> 00:18:17,768: you know, adjustments to system prompts. And you really
- 00:18:17,788 --> 00:18:18,808: don't have to be technical,
- 00:18:19,308 --> 00:18:21,408: uh, to be able to do this in today's world
- 00:18:21,448 --> 00:18:23,708: and in today's world of evals and some of the platforms
- 00:18:23,748 --> 00:18:24,268: that are out there.
- 00:18:24,788 --> 00:18:26,048: Um, you can just adjust
- 00:18:26,388 --> 00:18:27,028: kind of how,
- 00:18:27,528 --> 00:18:29,988: uh, your memory markdown file might be indexed.
- 00:18:30,068 --> 00:18:31,928: You can adjust, uh, the system prompt or
- 00:18:32,328 --> 00:18:34,348: how, uh, this metadata is organized,
- 00:18:34,728 --> 00:18:35,548: rerun the result,
- 00:18:35,928 --> 00:18:38,168: and that, that's how you kind of iterate and continue
- 00:18:38,208 --> 00:18:39,288: to improve quality,
- 00:18:39,728 --> 00:18:40,368: uh, as you
- 00:18:40,708 --> 00:18:42,708: roll out AI agents, uh, across more and
- 00:18:42,768 --> 00:18:43,528: more, uh,
- 00:18:43,868 --> 00:18:45,428: lines of business throughout the organization.
- 00:18:45,788 --> 00:18:47,848: So wrapping up, we have four minutes
- 00:18:47,928 --> 00:18:49,188: left. I think we're gonna be able to do it.
- 00:18:49,768 --> 00:18:51,808: Um, we have three... We're just gonna
- 00:18:51,828 --> 00:18:52,848: leave you with three kind of
- 00:18:53,248 --> 00:18:55,248: high-level takeaways and three kind of low-level
- 00:18:55,328 --> 00:18:56,368: takeaways that you guys can,
- 00:18:57,008 --> 00:18:59,148: uh, take back to your or- own organizations
- 00:18:59,188 --> 00:19:00,068: and help make sure that-
- 00:19:00,572 --> 00:19:01,392: Enterprise AI
- 00:19:01,872 --> 00:19:03,332: rollouts are, are going smoothly.
- 00:19:03,772 --> 00:19:04,492: So the first is
- 00:19:05,172 --> 00:19:05,812: related to the
- 00:19:06,352 --> 00:19:08,512: organizational structure that Smit was talking about.
- 00:19:08,992 --> 00:19:11,092: Uh, it's important that you have business sponsoring
- 00:19:11,152 --> 00:19:13,372: or owning the outcome. I think we had some conversation
- 00:19:13,412 --> 00:19:15,532: about that in the previous con-- in
- 00:19:15,572 --> 00:19:16,292: the previous talk.
- 00:19:16,792 --> 00:19:17,812: Um, if
- 00:19:18,172 --> 00:19:19,912: these kinds of AI solutions are
- 00:19:20,272 --> 00:19:22,812: pushed out by IT and engineering,
- 00:19:23,352 --> 00:19:24,972: uh, oftentimes folks are gonna be like,
- 00:19:25,452 --> 00:19:27,892: "This initiative is coming from someone else's
- 00:19:28,032 --> 00:19:28,552: North Star.
- 00:19:28,812 --> 00:19:30,132: This isn't my point of view." They're
- 00:19:30,472 --> 00:19:32,612: gonna have forty or fifty different things to focus
- 00:19:32,692 --> 00:19:35,012: on, and it's just not gonna get the AI adoption.
- 00:19:35,092 --> 00:19:35,832: So making sure that
- 00:19:36,392 --> 00:19:38,692: lines of business are brought into the process early
- 00:19:38,792 --> 00:19:40,372: on when these things are being rolled out
- 00:19:40,672 --> 00:19:42,052: is, uh, is important.
- 00:19:43,132 --> 00:19:43,972: The second is
- 00:19:44,412 --> 00:19:46,492: can't overemphasize how important
- 00:19:47,192 --> 00:19:49,312: thinking about and putting a process and a framework
- 00:19:49,352 --> 00:19:50,592: in place around how to curate
- 00:19:51,192 --> 00:19:52,372: enterprise context is
- 00:19:52,812 --> 00:19:54,892: to the value of these AI
- 00:19:54,932 --> 00:19:55,692: deployments. So
- 00:19:56,172 --> 00:19:57,492: I showed you the six layers that
- 00:19:57,892 --> 00:19:58,572: OpenAI,
- 00:19:58,932 --> 00:20:01,072: um, thinks about and, and puts design
- 00:20:01,132 --> 00:20:03,252: around to curate its own data agent.
- 00:20:03,772 --> 00:20:05,792: Um, that's just a framework. That's
- 00:20:05,852 --> 00:20:06,672: just, uh, you know,
- 00:20:07,292 --> 00:20:08,432: kind of an inspiration.
- 00:20:08,852 --> 00:20:10,892: I think the important thing is that you think through what
- 00:20:10,932 --> 00:20:12,012: are the context layers
- 00:20:12,352 --> 00:20:14,472: that are relevant at, at your own organizations,
- 00:20:14,832 --> 00:20:15,532: and think through
- 00:20:15,852 --> 00:20:17,932: what's gonna go into, to curating those, and obviously,
- 00:20:18,032 --> 00:20:19,412: OpenAI has resources to kinda
- 00:20:19,952 --> 00:20:20,972: help you, um,
- 00:20:21,292 --> 00:20:22,232: help you sift through that.
- 00:20:23,192 --> 00:20:25,232: And then the last is, uh, a throwback to the
- 00:20:25,532 --> 00:20:26,192: previous slide
- 00:20:26,452 --> 00:20:28,512: I, I re-referenced. Just
- 00:20:28,592 --> 00:20:30,732: I think it's important to evaluate
- 00:20:30,792 --> 00:20:32,772: and continually improve as you guys,
- 00:20:33,292 --> 00:20:35,392: um, run these AI agents. Make sure you're
- 00:20:35,432 --> 00:20:37,852: sampling production data, you're understanding
- 00:20:38,312 --> 00:20:39,112: how, uh,
- 00:20:39,732 --> 00:20:40,812: how lines of business are
- 00:20:41,412 --> 00:20:43,152: using the agent, what the results are,
- 00:20:43,652 --> 00:20:45,672: uh, and you're iterating and making sure that
- 00:20:45,692 --> 00:20:47,272: you're catching edge cases to,
- 00:20:47,912 --> 00:20:49,952: uh, maintain trust and maintain the quality
- 00:20:50,012 --> 00:20:52,152: of, of the outputs of your AI systems. Let's
- 00:20:52,172 --> 00:20:54,472: bring it home with, uh, with the actual tactical takeaways.
- 00:20:54,512 --> 00:20:55,292: How do we make this real?
- 00:20:56,192 --> 00:20:58,292: Uh, what does Codex's runtime mean? It means
- 00:20:59,252 --> 00:21:01,252: an agent, you're not just passing queries to
- 00:21:01,292 --> 00:21:01,812: an LLM.
- 00:21:02,472 --> 00:21:04,752: Uh, you're using Codex, you're using the, the Codex
- 00:21:04,792 --> 00:21:05,712: harness as
- 00:21:06,032 --> 00:21:07,072: the actual runtime. So
- 00:21:07,632 --> 00:21:09,972: the agents we've built internally at OpenAI, the agents
- 00:21:10,011 --> 00:21:11,432: that a lot of our customers have built,
- 00:21:12,132 --> 00:21:14,132: uh, they're using the Codex harness, whe-whether
- 00:21:14,172 --> 00:21:16,232: that's via Agents API or whether they're running things
- 00:21:16,272 --> 00:21:17,072: directly in Codex.
- 00:21:17,832 --> 00:21:18,532: Uh, you're--
- 00:21:18,792 --> 00:21:20,532: you have a system that is deciding
- 00:21:21,032 --> 00:21:23,092: what needs to go to the LLM, what context
- 00:21:23,132 --> 00:21:25,072: needs to be pulled in when sending a query
- 00:21:25,532 --> 00:21:26,932: to the LLM, as opposed to
- 00:21:27,652 --> 00:21:29,692: just sending queries to an LLM and
- 00:21:29,712 --> 00:21:31,572: expecting good, high-quality output.
- 00:21:32,772 --> 00:21:35,092: Uh, fewer tools. Again, this is about limiting
- 00:21:35,232 --> 00:21:36,952: what you send to the LLM.
- 00:21:37,532 --> 00:21:39,612: Uh, you're making a decision on
- 00:21:39,652 --> 00:21:41,652: not just, "Hey, these are all the tools I have. This
- 00:21:41,692 --> 00:21:43,192: is all the data I have available. I'm gonna
- 00:21:43,852 --> 00:21:46,092: give the agent access to all of it for every query."
- 00:21:46,832 --> 00:21:48,052: You're gon-- You're not gonna get good output.
- 00:21:48,592 --> 00:21:49,132: You need to,
- 00:21:49,492 --> 00:21:51,072: as part of that system, decide
- 00:21:51,572 --> 00:21:53,792: what the agent is and isn't going to have access
- 00:21:53,832 --> 00:21:55,912: to, and what, uh, systems it is and
- 00:21:55,972 --> 00:21:57,412: isn't going to query live
- 00:21:57,732 --> 00:21:58,912: when the user asks a question.
- 00:21:59,732 --> 00:22:01,532: Uh, and then guide goals, not paths. What does that mean?
- 00:22:02,652 --> 00:22:04,792: When you think about traditional software development, you have
- 00:22:04,812 --> 00:22:06,912: kind of your, your test cases and your unit tests,
- 00:22:06,952 --> 00:22:09,052: and it's a little bit different when we're talking about
- 00:22:09,132 --> 00:22:10,712: AI. It's non-deterministic,
- 00:22:11,272 --> 00:22:13,552: uh, and you may not be able to
- 00:22:13,612 --> 00:22:15,752: account for every single type of query that a user
- 00:22:15,772 --> 00:22:16,832: throws at the agent. So
- 00:22:17,412 --> 00:22:18,992: you wanna, instead of thinking about,
- 00:22:19,412 --> 00:22:21,432: "Hey, here's how you write this SQL query. Here's
- 00:22:21,472 --> 00:22:22,012: how you look up
- 00:22:22,392 --> 00:22:23,512: this particular data point,"
- 00:22:23,932 --> 00:22:24,752: you wanna instead
- 00:22:25,112 --> 00:22:27,352: give that definition of, "This
- 00:22:27,412 --> 00:22:29,452: is the data you have access to. These are the systems
- 00:22:29,492 --> 00:22:30,232: you have access to.
- 00:22:30,652 --> 00:22:31,672: This is what it all means.
- 00:22:32,112 --> 00:22:34,312: These are the types of problems the users might
- 00:22:34,332 --> 00:22:35,172: be needing to solve,"
- 00:22:35,612 --> 00:22:37,632: and then let the agent figure out what is the best
- 00:22:37,672 --> 00:22:38,552: way to get that. What
- 00:22:38,932 --> 00:22:40,992: queries do I need to run? What systems do I need to connect to?
OpenAI: From pilot to platform - The enterprise AI adoption journey
A practical maturity map, with real examples of customers and use cases at each stage, followed by a connected art-of-the-possible demo.















