Badiyaa is a small news site that uses AI to help produce journalism at a pace a team of our size could not otherwise manage. This paper sets out, in more detail than our Editorial Policy, how six published frameworks for human oversight and responsible AI shape the way we work, where we think we meet them, and where we do not yet. We are publishing it early, and openly, because the questions it raises are bigger than one newsroom. We would like help answering them.
Why publish a working paper?
Most discussion of AI in newsrooms happens at one of two extremes: high-level principles that are hard to put into practice, or vendor case studies that are hard to verify. Small publishers sit in between. We have to make concrete decisions every day about what a machine may do, what a person must do, and how either can be checked.
So this is a practitioner’s account, not a research finding. It describes a live system, names the frameworks it borrows from, and lists open questions. It will change as we learn, and every version will be kept with a dated changelog.
The system in brief
Every story moves through three phases. In Plan, people choose what to cover and the angle to take. In Produce, AI systems carry out research synthesis, draft an original article in a defined house voice, source licensed imagery and run automated integrity checks. In Review, editors refine, approve and, after publication, correct. A governance layer drawn from Google’s publisher guidance and our own policy applies throughout. The full description, with diagrams, is on our Editorial Policy page.
The rest of this paper takes each framework in turn: what it says, how we have mapped it to that system, and what remains unresolved.
1. Human agency and oversight: the EU High-Level Expert Group on AI (2019)
What it says. The Ethics Guidelines for Trustworthy AI, published by the European Commission’s High-Level Expert Group on AI in April 2019, list human agency and oversight as the first of seven requirements. They describe three governance mechanisms: human-in-the-loop, where a person can intervene in every decision cycle; human-on-the-loop, where a person oversees the system’s design and operation and can intervene; and human-in-command, where people govern the overall activity of the system, including when and how to use it at all.
How we map it. High-risk stories (politics, health, finance, crime and allegations about named people) are human-in-the-loop: nothing publishes without an editor’s approval. Other stories are human-on-the-loop: they may publish automatically after passing every gate, and every one is then reviewed by an editor, who can correct or withdraw it. The decision to run the pipeline at all, its rules and its daily volume are human-in-command.
Open questions. On-the-loop oversight only works if reviewers have the time and attention to catch what matters. What is the right ratio of reviewers to stories, and how should attention be measured rather than assumed? At what point does post-publication review become a formality?
2. Types and levels of automation: Parasuraman, Sheridan and Wickens (2000)
What it says. In a widely cited paper in IEEE Transactions on Systems, Man, and Cybernetics, Parasuraman, Sheridan and Wickens proposed that automation should be analysed separately across four stages of information processing: information acquisition, information analysis, decision selection and action implementation. Each stage can be automated to a different degree, on a scale that runs from fully manual to fully autonomous, building on the ten levels described by Sheridan and Verplank in 1978. Their central argument is that the right level differs by stage and should be chosen by weighing human performance consequences such as workload, situation awareness, complacency and skill degradation.
| Stage | Level of automation at Badiyaa | Human role |
|---|---|---|
| Information acquisition | High. Signal monitoring and research gathering are largely automated. | Sets the sources, markets and topics that are in scope. |
| Information analysis | High. Drafting, fact-consistency and originality checks are automated. | Defines the rules and thresholds; audits samples. |
| Decision selection | Low. The system ranks candidates; people choose. | Approves the daily agenda and every angle. |
| Action implementation | Mixed. Low for high-risk stories; higher for other stories, which can publish after passing all gates. | Approves high-risk stories; reviews, corrects or withdraws all others. |
Open questions. The model warns specifically about automation bias: people trusting a well-performing system more than they should. Our reviewers see drafts that have already passed several automated checks. How do we stop a green light from becoming a reason to look less carefully?
3. Human-centred AI: Shneiderman (2020)
What it says. Ben Shneiderman’s human-centred AI framework, set out in the International Journal of Human–Computer Interaction in 2020 and expanded in his 2022 book, rejects the idea that more automation must mean less human control. It describes a two-dimensional space in which systems can and should aim for high automation and high human control at the same time, and it recommends practical measures for reliability, safety and trustworthiness, including audit trails, independent oversight and clear accountability.
How we map it. We use automation for the work machines do well at scale, and keep human control where judgment matters: what to cover, what to publish, and what to correct. Every story leaves a record of the checks it passed or failed and the reasons it was held, so decisions can be reviewed after the fact.
Open questions. Shneiderman places weight on independent oversight. A small publisher cannot easily fund an external auditor. Could a shared, peer-review style arrangement between small newsrooms serve the same purpose?
4. NIST AI Risk Management Framework 1.0 (2023)
What it says. The US National Institute of Standards and Technology published its AI Risk Management Framework in January 2023, organised around four functions: Govern, Map, Measure and Manage. In July 2024 it added a Generative AI Profile (NIST AI 600-1) that addresses risks particular to generative systems, including confabulation and information integrity. The framework is voluntary and designed to be adapted to organisations of any size.
| Function | What it covers | Badiyaa practice |
|---|---|---|
| Govern | Policies, roles, accountability and culture. | A public Editorial Policy, named editorial roles, and a clear statement that the publisher is responsible for everything it publishes. |
| Map | Understanding context and identifying risks. | Every story is classified by topic risk before production; high-risk topics follow a stricter path. |
| Measure | Assessing and tracking risks. | Automated gates for originality, claim-level fact consistency and headline integrity, with results recorded per story. |
| Manage | Prioritising and acting on risks. | Revise-once-then-hold rules, pre-publication approval, corrections, and regular revision of the rules themselves. |
Open questions. The Generative AI Profile treats confabulation, a model stating something false with confidence, as a core risk. Claim-level checking against research catches many errors, but not errors that are also present in the research. What independent checks are proportionate for a small newsroom?
5. The Paris Charter on AI and Journalism (2023)
What it says. Published in November 2023 by Reporters Without Borders and partner organisations, following a commission chaired by Nobel Peace Prize laureate Maria Ressa, the Charter sets out ten principles for the use of AI in journalism. Among them: journalism ethics should guide technological choices; human agency must remain central to editorial decisions; media outlets remain accountable for the content they publish; and the use of AI should be transparent to the public.
How we map it. Editorial decisions sit with people. We accept full responsibility for everything we publish, however it was produced. Every article discloses AI assistance in its visible text and links to our policy.
Where we fall short. The Charter also calls for AI systems used in journalism to undergo prior, independent evaluation. We have not had our pipeline independently evaluated. We would welcome researchers who want to do so.
6. The EU AI Act, Article 50
What it says. Article 50 of the EU Artificial Intelligence Act sets transparency obligations that apply from 2 August 2026. Among them, deployers who publish AI-generated or manipulated text to inform the public on matters of public interest must disclose that the text was artificially generated. The obligation does not apply where the content has undergone human review or editorial control and a natural or legal person holds editorial responsibility for its publication.
How we map it. Our Editorial Desk holds editorial responsibility for every article, and our review process is designed to be substantive rather than nominal. We disclose AI assistance on every article in any case, because we think readers should know, whether or not the law requires it.
Open questions. What counts as genuine human review at volume? If one editor reviews many stories a day, is that editorial control in the sense the Act intends? Clearer, shared benchmarks would help publishers and regulators alike.
What we will measure next
A framework is only as good as the evidence that it works. Over the coming months we intend to track, and publish in a future version of this paper:
- the share of stories held by each automated gate, and why;
- the time editors spend reviewing each story, by risk category;
- corrections per hundred stories, and how quickly they are made;
- how often post-publication review leads to a change.
We do not have these numbers yet. We would rather say so than publish figures we cannot stand behind.
An invitation
We are looking for collaborators: researchers in journalism, human–computer interaction and AI governance; editors and engineers at other newsrooms; and anyone who can tell us where this approach is wrong. We are particularly interested in independent evaluation of our process, in shared benchmarks for meaningful human review, and in critique of any part of this paper.
Comments, corrections and proposals are welcome at badiyaavideo@gmail.com. Substantive contributions will be acknowledged by name, with permission, in the next version.
References
- High-Level Expert Group on Artificial Intelligence (2019). Ethics Guidelines for Trustworthy AI. European Commission. digital-strategy.ec.europa.eu
- Parasuraman, R., Sheridan, T. B. and Wickens, C. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics, Part A, 30(3), 286–297. doi.org/10.1109/3468.844354
- Sheridan, T. B. and Verplank, W. L. (1978). Human and Computer Control of Undersea Teleoperators. MIT Man-Machine Systems Laboratory.
- Shneiderman, B. (2020). Human-centered artificial intelligence: Reliable, safe and trustworthy. International Journal of Human–Computer Interaction, 36(6), 495–504. doi.org/10.1080/10447318.2020.1741118
- Shneiderman, B. (2022). Human-Centered AI. Oxford University Press.
- National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1. doi.org/10.6028/NIST.AI.100-1
- National Institute of Standards and Technology (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1. doi.org/10.6028/NIST.AI.600-1
- Reporters Without Borders et al. (2023). Paris Charter on AI and Journalism. rsf.org
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 50. eur-lex.europa.eu
- Google Search Central. Creating helpful, reliable, people-first content. developers.google.com