AI Product Safety Culture
This study examines the product safety culture of AI companies, evaluating whether foundation model developers and medical AI firms meaningfully prioritize safety, risk awareness, and responsible deployment.
What is this project about?
This research investigates the product safety culture of companies developing advanced AI systems, including:
- Foundation model developers building large-scale general-purpose AI systems
- Medical AI companies building clinical software, agents, or decision-support tools on top of foundation models
The study assesses the degree to which these organizations:
- Explicitly prioritize safety in leadership messaging and internal governance
- Allocate resources to risk assessment, validation, and post-deployment monitoring
- Implement structured processes for red-teaming, bias detection, and harm mitigation
- Encourage internal reporting of safety concerns without retaliation
- Incorporate domain expertise (e.g., clinicians) into development workflows
- Measure and reward safety performance alongside speed and innovation
Using surveys, structured interviews, document analysis, and publicly available disclosures, we evaluate whether safety is embedded in decision-making processes — or subordinated to competitive pressures such as speed-to-market and scaling.
The study is particularly focused on AI products that influence healthcare delivery, where failure modes may directly affect patient outcomes.
Why is this important?
Technical accuracy alone does not determine safety. Organizational culture determines:
- What risks are identified
- Which risks are tolerated
- How aggressively systems are validated
- Whether safety concerns are surfaced — or suppressed
In healthcare, poorly governed AI systems can:
- Amplify diagnostic errors
- Introduce bias into triage or treatment decisions
- Create automation bias among clinicians
- Scale small design flaws across thousands of patients
If safety is not structurally embedded in AI companies — especially those building systems intended for medical contexts — patient harm may occur not because the technology is inherently unsafe, but because the organizational incentives surrounding it are misaligned.
Healthcare has learned repeatedly that culture drives safety outcomes. The same principle applies to AI development.
Understanding product safety culture is therefore essential to preventing AI from exacerbating the patient safety crisis.
Where can it be applied?
1. Healthcare Procurement and Vendor Evaluation
Hospitals and health systems can use findings to evaluate not only what an AI tool does, but how safely it was built. Product safety culture metrics could become part of vendor selection criteria.
2. Regulatory and Policy Development
Insights may inform regulators developing oversight frameworks for clinical AI systems, emphasizing organizational safety practices in addition to algorithmic validation.
3. Investor and Board-Level Governance
Investors and boards can incorporate safety culture indicators into due diligence processes, reducing long-term liability and reputational risk.
4. Industry Benchmarks for Responsible AI Development
The research can contribute to standardized benchmarks for AI product safety culture, encouraging companies to embed risk management into their operational DNA.
5. Strengthening Patient Safety in the AI Era
Ultimately, ensuring that AI improves healthcare requires more than high-performing models. It requires companies whose internal incentives, governance structures, and leadership priorities consistently place patient safety above speed, scale, or valuation.
This study helps illuminate whether that foundation is currently strong — and where it must be strengthened.
What are this results?
This study is currently being conducted. Please sign up to our newsletter to be allerted when results are availbile!
Research You Can Rely On
Did you know that many research findings are manipulated—or even outright false? Some estimates suggest that up to 90% of published research may be unreliable. Meanwhile, more than $167 billion in taxpayer money is spent annually on research and development.
At BRITE Institute, we believe research should do more than just look credible. It should be credible. That’s why we go above and beyond typical standards with rigorous practices that ensure honesty, transparency, and accuracy at every step. Below are just some of the ways we safeguard the integrity of our work:
Honest Statistics
BRITE Institute never p-hacks or manipulates data to achieve a desired outcome. If a paper relies on complex statistical analyses, we use an external statistician to ensure objectivity and validity.
Open and Transparent
BRITE Institute prioritizes transparency at every stage of the research process. Whenever possible, we publish our full data sets and use open access publishing.
Impact Centered
BRITE Institute does not publish for the sake of publishing. Our research is built with end-users in mind—whether it’s policy-makers, engineers, or community leaders—ensuring that findings are not only trustworthy but also actionable.