Trustworthy AI Has a Toolkit Problem, Landmark Analysis of 938 Tools Reveals

Artificial intelligence has produced no shortage of ethical soul-searching. Since 2016, more than 80 sets of AI ethics principles have been published by governments, companies, and academic institutions, converging with striking consistency on a familiar canon: transparency, fairness, safety, privacy, accountability, and beneficence. Yet a growing chorus of critics has charged that these documents are abstract, “toothless,” and in the words of one European expert group member, “deliberately vague” — lofty declarations that leave developers guessing about what to actually do. Now, a new open-access study published in AI & Society by Michael Papademas and colleagues at the National Centre for Scientific Research “Demokritos” and Panteion University in Athens offers the most systematic empirical look yet at whether the promised machinery of trustworthy AI actually exists — and the answer is a nuanced, sometimes uncomfortable, portrait of an ecosystem that fixes what is easy and postpones what is hard.

The team turned to an unusual source of evidence: the OECD’s catalogue of AI ethics tools and trust or quality mark schemes. As of 17 July 2025, that repository listed 938 tools, of which 24 were formal certification or quality marks intended to label AI systems as trustworthy. Rather than sampling, the researchers analyzed the full set, using a descriptive and comparative approach that mapped each tool and framework against the ethical objectives it supports, the type of intervention it represents — technical, educational, or procedural — the stage of the AI lifecycle it addresses, the stakeholders it targets, and the skills it demands. The authors are candid about the limits of this method: they adopted the OECD’s own classifications without independent recoding, and their findings describe the curated repository rather than the entire global landscape. Even so, the patterns that emerge are striking enough to matter.

The first finding concerns which ethical principles get built and which get talked about. Transparency emerged as the single most widely supported objective in the catalogue, followed closely by fairness and robustness. This is the good news: the AI community has invested heavily in bias-mitigation toolkits, model documentation standards, and stability testing, translating the most-cited principles into software libraries, checklists, and assessment frameworks. But the distribution is sharply non-uniform. Explainability — the capacity to interpret and comprehend why an AI system made a particular decision — is addressed far less frequently than transparency, likely because producing genuinely human-interpretable explanations for complex models remains a formidable technical challenge, and because the boundary between “transparency” and “explainability” is itself contested. Digital security tools, which defend against adversarial attacks and data breaches, are similarly scarce. And environmental sustainability is almost an afterthought: very few tools explicitly target the energy consumption or carbon footprint of AI systems.

This asymmetry has a telling structure. Tools proliferate where problems are quantifiable, where regulatory and public pressure is strongest, and where solutions can be generalized into reusable libraries. Measuring demographic bias in a dataset is tractable; explaining the internal reasoning of a deep neural network in a way that satisfies a regulator, a doctor, and a defendant is not. The authors argue that the implementation ecosystem does not simply execute pre-given moral commitments — it filters them through organizational feasibility, technical legibility, and institutional incentives. Values that can be measured, audited, and formalized get infrastructure; values that resist formalization, such as sustainability or genuine interpretability, remain rhetorical. The result, they warn, is a risk of achieving only a “verisimilitude of trustworthiness” — an appearance of ethical completeness that is incomplete at its core.

The second major finding concerns the types of interventions being deployed. Technical tools — software libraries, algorithms, and evaluation platforms — dominate the catalogue, concentrated overwhelmingly on transparency, robustness, and fairness. Procedural tools, such as governance checklists, risk-management frameworks, and documentation templates, come next, and they too cluster around transparency, fairness, and privacy. Educational tools — training programs, courses, and best-practice guides designed to build ethical competence among practitioners — are the least common category of all, and the few that exist focus on broad principles while neglecting explainability, data governance, digital security, and sustainability. This points to what the researchers call a significant capacity-building gap: organizations are buying technical fixes and adopting process documents, but they are not systematically educating the people who must apply them.

The authors connect this gap to a deeper philosophical worry. A governance ecosystem centered on tools and procedures risks externalizing moral responsibility into artifacts of compliance — the implicit assumption that ethical adequacy can be achieved through the correct use of instruments alone. But trustworthiness, they argue, is not an inherent property of a technical system; it arises from socio-technical relationships among designers, institutions, affected communities, and the normative assumptions embedded in practice. If developers and managers lack ethical literacy, checklists may be implemented perfunctorily or audited workarounds found — the phenomenon critics have labeled ethicswashing or ethics theater. Citing research arguing that AI development often proceeds in an “ethically empty milieu,” the study suggests that cultivating judgment, reflexivity, and practical wisdom remains structurally undervalued, and that future certification schemes could mandate or incentivize accredited ethics training for AI teams rather than treating training as optional.

Perhaps the most consequential finding concerns timing. When the researchers examined where trust and quality mark frameworks intervene in the AI lifecycle, they found a heavy skew toward the late stages: the “Verify & Validate” and “Operate & Monitor” phases are by far the most frequently addressed, while the early “Plan & Design” and “Collect & Process Data” stages receive comparatively little attention. In other words, most certification schemes assume the AI system is already built and then audit its behavior — a final compliance check rather than a design discipline. The authors liken this to the history of software security, which long relied on penetration testing after development until “Secure by Design” philosophies took hold. If ethical flaws — biased training data, an unsafe architecture, opaque model logic — are embedded early and only discovered at validation, fixing them may be too late or prohibitively expensive.

This temporal imbalance, the study argues, reflects a particular moral chronology in which ethical reflection is displaced downstream, where it is weaker, costlier, and less transformative. The “Ethics by Design” literature has long maintained that values are not appended to systems at the end of development; they are already inscribed in problem formulation, data selection, and assumptions about users and harms. The researchers recommend that next-generation governance frameworks incorporate design-phase requirements: algorithmic impact assessments conducted before model development, participatory design with affected stakeholders, bias-aware data collection strategies, and documentation of how ethical considerations shaped design decisions — not merely how the final model was evaluated. The hard part, they acknowledge, is creating concrete standards for design practices that can actually be verified.

The third axis of asymmetry concerns audience. The most frequently targeted users of trust mark frameworks are data scientists, developers, and business leaders, with many schemes also addressing all employees of an organization — an internal, corporate-compliance orientation. Policymakers, regulators, the broader public sector, and the general public are markedly under-targeted. This skew means current trust marks function largely as industry-led self-governance: companies assessing their own adherence to ethical standards, with little external oversight or public empowerment. The authors warn that when the power to define what counts as a trustworthy system remains concentrated among those closest to production and deployment, the harms, dependencies, and exclusions experienced by affected communities — often invisible from a technical or managerial standpoint — go underrepresented. They call for co-regulation, independent audits, and public–private partnerships that bring regulators and civil society into the definition and verification of trustworthiness, noting that the required skills listed in the catalogue — programming, data management, IT competencies — reinforce the technical gatekeeping.

The study closes with a set of recommendations that read as a roadmap for the field: broaden the ethical objectives of tools and certifications to include environmental sustainability, explainability, and other neglected principles; embed ethics earlier in the lifecycle through design-stage assessments; expand multi-stakeholder participation to include policymakers, end-user representatives, and interdisciplinary experts; invest in ethics education as a core organizational competence; and strengthen enforcement by linking voluntary marks to regulatory or contractual requirements with independent audit capabilities, updated through regular structured reviews. The overall diagnosis is one of significant progress coupled with significant imbalance. The AI community has demonstrably moved beyond principles on paper — bias audits, transparency documentation, and certification schemes now exist at scale. But the ecosystem, the authors conclude, selectively stabilizes the values that are easiest to formalize while marginalizing the rest. The future of AI governance, they suggest, will depend not on producing more principles, but on transforming the institutional, epistemic, and design conditions under which any principle can become materially operative — and on ensuring that the AI systems increasingly woven into society earn the trust of the people they affect.

Subject of Research: Empirical analysis of trustworthy AI tools and trust mark frameworks using the OECD catalogue

Article Title: A critical analysis of trustworthy AI tools, mark frameworks, and the implementation chasms

Article References: Papademas, M., Karpouzis, K., Ziouvelou, X., & Karkaletsis, V. (2026). A critical analysis of trustworthy AI tools, mark frameworks, and the implementation chasms. AI & SOCIETY. https://doi.org/10.1007/s00146-026-03364-4

Image Credits: AI Generated

DOI: 10.1007/s00146-026-03364-4

Keywords: trustworthy AI, AI ethics, OECD catalogue, AI governance, explainability, algorithmic fairness, ethics by design, AI certification, ethicswashing, AI lifecycle, sustainability, AI auditing