RepoLanternHow we check every guide
Measured, not assumed. Every guide is checked against the code it describes, and we publish how well the checks work.
- 12/12
- golden guides reproduce exactly, with no false alarms
- 94.3%
- of planted names caught in a blind round
- 83.3%
- caught in a second, harder blind round
The grounding score
Each guide gets a score from 0 to 100 for how well it sticks to what is really in the code: whether the files, functions and classes it names exist, whether its citations resolve, and whether its required sections are present. It checks names, paths and citations, not whether every explanation is right, and the guide says so.
Names we couldn't find
When a guide mentions something that isn't in the repository, RepoLantern rewrites that section once with the list of unverified names. The rewrite is kept only if it has fewer of them. Whatever remains is listed at the end of the guide, so you know exactly what to double-check.
How well the checks work
We test the checker itself, the way you would test any safety system:
- Golden guides: 12 real guides across 5 repositories and 4 languages, with hand-verified facts. All 12 reproduce exactly, with no false alarms.
- Planted mistakes: invented names planted into real guides are all caught.
- Blind tests: in a blind round written by a separate reviewer, 100 of 106 planted names were caught (94.3%). In a second, harder round across all five repositories, 189 of 227 were caught (83.3%), and none of 376 real names was flagged as invented.
- Prompt injection: in our live tests, repositories containing instructions aimed at AI tools were documented without the model obeying any of them, and the guide lists that text for you.
These checks run on every change and before every release, and a regression stops the release.
Answers to your questions
Answers in the question panel are checked the same way: paths and names in an answer are looked up in the code, and anything unverified is shown with the answer.