In 2021, Collibra quietly released Assessments, a privacy capability for data privacy stewards.
Collibra Assessments enables data privacy stewards to design, publish and archive assessment templates. These templates can be used to conduct assessments, which can be submitted for review and approval or rejection.
The Collibra Assessments also offer packaged templates — including those for DPIA and PIA — that you can use “as is” or can be easily customized to accommodate privacy-related needs and requirements beyond the scope of data privacy.
Throughout the year, new features to Assessments, including an out-of-the-box Assessments workflow, copy draft Assessments, Assessment as PDF, a Governance toggle, URL support , Export/Import templates, and an Assessment tab (Assessment Review).
But … we have not seen adoption of Assessments.
As part of our yearly investment review, a usage study revealed that customers have not (yet) adopted Assessments and therefore we see little usage within our customer base.
Over the last 90 days we saw that only 46 assessment templates have been published as new or updated templates and 79 assessments have been submitted by 24 unique users over 12 unique customers. Nearly all of these assessment templates can be positioned within a privacy context.
When we dig a little deeper into the usage data, we saw that the drop-off rates of users visiting the Assessments landing page is quite high (85%), and users who did effectively manage to launch an assessment (after finding the right template) only submitted one 23% of the time.
These numbers are quite disappointing and trigger a range of questions for our team, including:
- Was Assessments the wrong feature to build?
- Do customers even know that Assessment can be used outside of privacy? (Assessments was released as a platform feature available to all customers.)
- Have we lacked to properly enable our customers to get familiar with the feature?
Thinking out of the box
As part of our Collibra for Collibra initiative — where we drink our own champagne and really want to be our very first customer of our product — we’ve built out a governance framework around launching new products, features, and services to our customers.
We call it NPI.
The key reason to leverage the internal Collibra team for this business process governance was the ability to create asynchronous governance check-points (gates) with Assessments at its core. These gates – business case review, product legal review, product naming review, infosec review, beta readiness, release readiness, pricing reviews, support readiness and finally launch readiness – present opportunities for the Product Managers or Product Marketing Managers to align with their key go-to-market stakeholders on standard lists of checkpoints to validate whether everyone agrees that all has been done and is in place to be able to proceed with a successful launch in the market.
Since we launched the NPI process, we have seen great adoption and participation of all stakeholders in the process.
Let’s zoom in on our own usage data for assessments!
Over the last 90 days at Collibra, 6 unique users published and or updated 29 Assessment templates. Compared to our collective production customers, that is almost 40% of all template updates on CDIC.
But even more impressive is the 151 assessments submitted successfully by 43 unique Collibrians (that’s what Collibra employees call themselves). This is almost twice as much as the collective production customer base!
So why do we see this great adoption within Collibra?
When we zoom into the usage data, surprisingly we see the same pattern as with our customers if we focus on the Assessment app and pages: a high drop-off rate from the Assessment Landing page and a low success rate for users that effectively manage to start an assessment.
However when investigating the usage paths towards “submitting an assessment,” we noticed that most of the successfully submitted assessments actually were triggered from our NPI cockpit through our bespoke,easy-to-use workflows. (108 compared with 43)
The usage pattern is actually completely different with a much higher drop-off percentage from the NPI cockpit to starting an assessment workflow. (We hypothesise that this is because the cockpit is a starting point for different user persona and different actions, like browsing to an Assessment type library, to a NPI Tableau dashboard, etc.)
However, most users (71%) that start the workflow from within the NPI context successfully manage to submit the assessment.
4 success factors for Assessments @ Collibra 4 Collibra
Embed Assessments in their business context
The NPI cockpit (which basically is a configured dashboard) leverages Assessments without the need for the business users of the NPI process to know they’re using Assessments.
Our NPI process stakeholders know that to release and launch a new product, service, pricing, set of features, etc., they need to pass several gates. Passing a gate means conducting an assessment. Through the assessment, a set of gate specific questions need to be answered. And, finally, a review meeting to collectively review and approve the assessment.
Assessment Review as a first-class asset
The out-of-the-box asset type “Assessment Review” is too bare-bones to build a governance structure and process around it.
So for NPI, we have extended the Assessment Review asset type with custom attributes and relations, like NPI stage summary, NPI preview stage, NPI pricing priority, etc.
We also extended the operating model with a new asset type “Assessment Type”, which is essential for the automation of the workflow process, building diagrams that enable the product managers to easily understand where they are in the process, and essential functionality for building dynamic dashboards in Tableau.
Finally, we’ve extended Collibra Assessments with the ability to create Follow-up Actions (NPI Actions) to allow stakeholders to understand what still needs to be done before an Assessment Review can be approved.
To allow our key users to easily find back their submitted assessments, we’ve built Assessment type based libraries highlighting key characteristics and easy links from the tile to the assessment page of the Assessment Reviews. We also made sure that roles and responsibilities (RACI based) are correctly set for each assessment type :
- Owner (accountable for this assessment type),
- Business Steward (responsible for approving the assessment) and
- Business User (responsible for conducting and submitting the assessment)
Leverage Insights Data Access to build leadership dashboard
A dynamic and actionable NPI Pulse dashboard helps our Product Leadership and executive team understand where we are on our journey from idea to launched product, service, or feature set.
The NPI Pulse dashboard leverages Insights Data Access to export the Collibra data onto the Collibra Lake and, through TriFacta, the specific NPI assessment data is collected and transformed into a specific data set for the Tableau dashboard to consume.
The NPI Pulse dashboard allows our leadership and key stakeholders to follow up on the progress and allows them to easily understand where new product initiatives get stuck and where their help and assistance is needed.
Assessment Workflow Wrapper
Finally, we made it easy for our process stakeholders to start specific assessments in their business context with simple and straightforward workflows. The Assessment Type workflow wrapper for the NPI process allows our stakeholders to easily launch assessments from a central dashboard.
The workflow not only starts the specific assessment process, but also takes care of linking the resulting Assessment Review asset with the appropriate Assessment Type and NPI Initiative and storing the Assessment Review asset in the correct library.
It also takes care of the approval process, where the business stewards defined at the library will get the workflow task to review and approve the assessment review.
Assessments: A business friendly way to govern critical data assets, business processes, and compliance challenges
Collibra Assessment has been built respecting the Agile principle of releasing an MVP fast, and learning from customer usage …
However, as this idea highlights, maybe Assessments was released too early and with a too-minimal MVP.
But it is clear that the NPI use case for Collibra by Collibra shows that Assessment has great potential for data governance platforms because
- It sparks asynchronous alignment with multiple business stakeholders
- It allows business users to easily build these assessments without the need for deep technical skills or the need to write code (#nocode)
- It easily creates higher platform adoption
- Although its primary use case is Privacy, it is a free platform capability that can be used for many different use cases like business process governance, ethical data governance, etc.
Even today assessments can already be successfully leveraged when you:
- Put Assessments in your business context!
- Identify your stakeholders, build their own libraries, and set clear ownership and responsibilities.
- Leverage Insights to build great dashboards on top of your governance processes and achieve transparency across the entire process for all stakeholders
- Assessments AND workflows make it easier to collaborate with your colleagues around cross functional challenges to help your organization align faster and achieve desired and collectively supported outcomes.
At Data Citizens 2022 in San Diego, Simla Sivanandan and Tom Dejonghe presented this in more detail during a fully booked hands-on workshop session.
In addition, the attendees got a dedicated demo instance which allowed them to explore and conduct assessments in the business context of requesting, designing, building and deploying data products. This instance will stay online for the next 30 days to allow all participants to continue exploring, copying and providing feedback via a dedicated feedback assessment!
Also our product leader Wouter Mertens for Privacy and Protect briefly introduced his plans for Assessments in the coming years and highlighted that enterprise-grade permissioning and authorization is coming soon. And later in 2023 the team will work on solid REST APIs to be used in workflows and/or integrations. Also a serious uplift of User Experience is planned, including user experience improvements for the Assessment landing page, Template Editor and enhanced Asset Picker, etc.