Methodology
How we choose sources, grade evidence, use AI, check for retractions and correct mistakes.
Source-selection criteria
We include a source when all of these hold:
- It addresses a question practitioners who consult and coach actually bring, starting with how to show and talk about the impact of their work.
- It is a peer-reviewed study, a systematic review or meta-analysis, or clearly identified grey literature (for example, a dissertation or an independent evaluation report).
- Its methods are described well enough to grade the design against the question it answers.
- We can read at least its abstract, and we can say exactly where in the source each reported figure appears.
We prefer open-access versions, systematic reviews and designs with comparison groups, but we also include weaker designs when they are the best available evidence, graded accordingly. We keep a record of sources we considered and rejected, with the reason, so our selection can be checked.
Citation standards
- Sources are cited in APA 7th edition style, with a DOI wherever one exists.
- Every number we report names its location in the source: a page, table or section.
- When we read an author’s accepted manuscript rather than the published version, we say so, and page numbers refer to that manuscript.
- When we could read only the abstract, the entry says “Abstract only” and reports nothing beyond the abstract.
- We summarize in our own words, quote only briefly with attribution, and never reproduce paywalled text, figures or tables.
Evidence-grading scheme
Status: PROVISIONAL. Adapted from Center for Evidence-Based Management (CEBMa), Overview of Methodological Appropriateness (May 2025).
| Grade | What it means | What it does not mean |
|---|---|---|
| High | The design is well suited to this question. Findings are a strong basis for practice. | It does not mean the result is certain, or that it applies in your setting. |
| Moderate | The design is reasonably suited, with some alternative explanations left open. | It does not mean "probably true." |
| Limited | The design leaves important alternative explanations open. Treat the finding as a lead, not a conclusion. | It does not mean "wrong." |
| Low | The design is poorly suited to this question. | It is not evidence that the intervention fails. |
| Illustrative · not evidence of effect | Case studies and practitioner reports. Useful for ideas and context. (consultants.coach addition) | It is not evidence that anything works. |
Effect questions: how designs map to grades
- Before/after without a comparison group: Limited
- Controlled study at one point in time: Limited
- Controlled before/after: Moderate
- Cross-sectional survey: Low
- Randomized controlled trial: High
- Qualitative study: not applicable for effect questions
- Case study and practitioner report: Illustrative
Our rules beyond CEBMa
- Case studies and practitioner reports are graded Illustrative for every question type.
- Downgrade one level only when the source itself reports at least one of these serious flaws - significant publication bias, attrition above 30 percent, or a sample too small for the claim. Reporting errors and internal inconsistencies are flagged in the commentary, not downgraded.
- Systematic reviews and meta-analyses take the grade of their predominant included designs and are tagged Synthesis.
- Correlational designs never get causal wording, whatever their grade.
Changes to this scheme
- 2026-10-24: Rule 1 (serious flaws) given defined criteria after the Phase 4 re-test.
Editorial voice
- Every entry separates three things: what the study found, what its authors conclude, and our own commentary.
- We report effect sizes, uncertainty, sample, design and limitations whenever the source provides them.
- We never use causal language for correlational designs.
- We do not use isolated or flattering statistics (“vanity statistics”) to promote coaching or consulting.
- We write in plain language for experienced practitioners and say clearly when the evidence is thin.
How AI was used
consultants.coach uses AI assistance, in the form of a large language model, under human editorial control. The AI assistant searches for candidate sources, reads open-access text and abstracts, drafts entries, records where each figure appears in the source, re-checks every figure against the source, and runs automated retraction lookups. AI assistance was also used to build the site’s code.
AI can misread a table or miss context. That is why no entry is published until a human editor has checked it against the source and confirmed it in writing. The AI assistant cannot mark an entry as verified, and the site’s build refuses to publish any entry that a human has not verified or that has changed since it was verified.
The approach does not depend on any particular AI product. The same rules apply whichever assistant is used: sources are read before they are summarized, every figure names its location, and only a human editor can approve publication.
Verification and retraction checks
- Sources are supplied or approved by the editor.
- An entry is drafted from the source, with the location of every figure (status: draft).
- Every figure is re-checked against the source (status: tool-checked).
- The editor audits the entry against the source and confirms it (status: human-verified).
Before publication, each source is checked for retractions, corrections and expressions of concern on the publisher’s page and in Crossref, which carries the Retraction Watch database. The editor confirms the result. Entries are re-checked at least every 12 months, and again whenever we learn of a concern.
Any edit to a verified entry automatically sends it back for a fresh human check.
Corrections
We acknowledge every reported error within 24 hours and correct it, or explain why no change is needed, within 7 days. Every correction is recorded in the entry’s change log and on the corrections page. See contact to report an error.