The Check Passed. The Reviewer Didn’t.

Abstract

Every R package bound for CRAN runs a gauntlet of automated checks, and passing them feels like the finish line. It is only the first gate. A human reviewer still reads the package against a rulebook the automated tooling never enforces, one that lives scattered across policy pages, a style book, a community cookbook, and plain memory. This is the story of how my own package cleared every automated check and still came back with a list of six fixes, and how checktor turns that hidden rulebook into a check you can run before a human ever sees your work.

Most R packages people install come from one place, CRAN, the archive that acts as the app store for the R language. Getting a package in is a rite of passage, and the gate everyone talks about is a program called R CMD check. It takes the package tarball you have built, installs it, runs your examples and tests, inspects your documentation, and prints a tidy verdict of errors, warnings, and notes. When it comes back with three zeros, it feels like crossing the finish line.

Last year I crossed that line with a small package of mine called peeky. Clean check, no errors, no warnings, no notes. I wrote my submission note, uploaded the package, and closed the laptop feeling done.

I was not done. A few days later an email arrived from a CRAN reviewer. Not a rejection, exactly, more of a polite list of six things to fix before the package could go in. And here is the part that stuck with me. Not one of the six was something R CMD check had flagged. Every item was a rule a person applies by hand, out of a rulebook that is real, enforced, and almost entirely absent from the automated test I had just aced.

For the record, peeky was written in 2024, first submitted at the start of 2025, resubmitted in June 2026, and is on CRAN as I write this. The package was never really the problem. The list was short and the fixes were small. What ran long was finding the time to make them.

That gap, between a clean check and a clean submission, has driven a real community effort to make CRAN submission smoother, and checktor is one more push in that direction. This is the story of the gap, and what it takes to close it before a human ever has to.

Two gates, not one

A diagram of a package passing through an automated gate that stamps a green check, then meeting a second gate held by a human reviewer with an amber checklist.

Two gates stand between a package and CRAN. The first is an automated check that stamps a green pass; the second is a human reviewer holding a checklist of flagged items.

A diagram of a package passing through an automated gate that stamps a green check, then meeting a second gate held by a human reviewer with an amber checklist.

Two gates stand between a package and CRAN. The first is an automated check that stamps a green pass; the second is a human reviewer holding a checklist of flagged items.

It helps to picture two gates rather than one. The first gate is mechanical. It asks a narrow, answerable question. Does this thing work? Does it build, do the examples run, does the documentation match the code? A machine can answer that, and R CMD check does, quickly and without opinion.

The second gate is human. It asks a different question. Does this thing belong? Does it follow the conventions of a shared library that tens of thousands of people rely on? Are the names written the way names are written here, does each function say what it gives back, does the package stay out of folders that aren’t its own? Those are questions of judgment and convention, and they are the reviewer’s job. You can ace the first gate and stall at the second, because they are not testing the same thing.

The six things

Here is the actual list that came back for peeky, translated out of jargon. The original is public, and every item on it has a write-up and a fix in the CRAN Cookbook.

Software and package names in the DESCRIPTION needed single quotes, so ‘Quarto’ and ‘Shiny’ read as the proper nouns they are.
The methods the package leaned on needed a reference with a DOI or ISBN.
Two help pages said what a function did but never what it gave back.
A few help pages carried examples for internal helpers that were never part of the public interface.
Runnable examples were hidden behind a do-not-run marker meant for code that genuinely cannot run.
One function wrote into the user’s own directories by default, which CRAN’s policy forbids.

Read that list again with one thing in mind. None of it makes the package crash. Nothing there is an error, a warning, or even a note. They are conventions, courtesies, and one hard policy, the etiquette of a shared commons. And a reviewer reads by hand, one pass at a time. Something that slips past one read can surface on the next, so a package the machine called clean can make the round trip, submit, wait, read, fix, resubmit, more than once.

The rulebook nobody runs

The frustrating part is that none of this is secret. Every one of those six rules is written down somewhere. The trouble is the somewhere.

Five document sources on the left labeled CRAN Repository Policy, R Packages book, CRAN Cookbook, extrachecks, and reviewer memory, with arrows converging into a single checktor tool on the right.

Five separate sources of CRAN rules feed into a single tool: the CRAN Repository Policy, the R Packages book, the CRAN Cookbook, the extrachecks list, and reviewer memory, all converging into checktor.

Five document sources on the left labeled CRAN Repository Policy, R Packages book, CRAN Cookbook, extrachecks, and reviewer memory, with arrows converging into a single checktor tool on the right.

The single-quote rule lives in the CRAN Repository Policy. The advice about examples and what a function returns is spread through the R Packages book. Common rejections and their fixes are collected in the CRAN Cookbook. A running community list, Davis Vaughan’s extrachecks, catalogs exactly the kind of check “that CRAN does that are not checked for by devtools::check().” And a good deal of it lives nowhere but in the memory of people who have been through this before.

None of this has gone unanswered. There has been a real, ongoing effort to make CRAN submission less painful. usethis’s use_release_issue() opens a to-do list on your project, the release helpers in devtools (release() and submit_cran()) walk you through the final steps, and there is even a GitHub Action that checks and submits a package straight from continuous integration. But most of these hand you a checklist to tick off by hand, or automate the upload rather than the reading a reviewer does. The rules themselves were everywhere, and runnable nowhere.

One command, not a checklist

checktor makes that rulebook runnable. It takes the scattered rules and turns them into a single command you run on your own package, before a reviewer ever opens it.

It leans into a medical metaphor, which makes it read the way the work feels. You run a checktor() examination and get a diagnosis across the categories a reviewer cares about, your code, your description, your documentation, your file layout, and CRAN policy. You ask for a prescribe()d set of fixes. You keep a health_report() on file. And a one-line checkup() returns a plain yes-or-no, so your automated pipeline can refuse to move forward while anything is still wrong.

A diagram of a single command, checktor(), fanning out to a diagnosis across five categories, prescribe() for fixes, health_report() for a report, and checkup() returning TRUE or FALSE for CI.

One call to checktor() hands back everything you need: a diagnosis across five categories, plus prescribe(), health_report(), and checkup().

A diagram of a single command, checktor(), fanning out to a diagnosis across five categories, prescribe() for fixes, health_report() for a report, and checkup() returning TRUE or FALSE for CI.

One call to checktor() hands back everything you need: a diagnosis across five categories, plus prescribe(), health_report(), and checkup().

Run it against peeky today and it catches nearly the whole list from earlier on its own, the unquoted names, the help pages missing a return value, the examples for internal helpers, the misused do-not-run markers, the function reaching into a user’s folders. The reviewer found them by reading carefully. checktor finds them by reading deterministically, the same way every time, in about a second.

Reading, not searching

One design choice underneath checktor is worth slowing down for, because it is the difference between a tool you trust and one you learn to ignore.

Say you want to catch a hardcoded random seed, a set.seed(42) left in the body of a function, which quietly makes “random” results identical every run. The obvious way to find it is to search the text of the file for set.seed. That works right up until it doesn’t:

set.seed(42)          # a real command
x <- "set.seed(42)"   # the very same letters, but inside a string
# In a help page, we might even write about set.seed(42) in a sentence.

A plain text search sees the identical characters in all three places and cannot tell a command from a quotation. It cries wolf, you stop believing it, and a tool you stop believing is worse than no tool at all.

Two ways to inspect one line of code. A text search highlights the letters set.seed everywhere they appear, including inside a string, producing a false alarm. Reading the structure turns the line into a small tree where only the real function call is flagged.

checktor does not search the text. It reads the structure, the same way R itself does before running anything. Ask R to parse that line and it does not see a string of characters, it sees a shape, a function call named set.seed with the number 42 inside it. The words in quotes are a different kind of thing entirely, a piece of text, and text is never mistaken for a command. It is the difference between scanning a page for a word and actually reading the grammar of the sentence it sits in. One gets fooled by context; the other understands it.

That is why checktor can afford to be strict. Because it reads code the way the language reads it, it can flag the real thing and stay quiet about the lookalikes, and a tool that only speaks up when it is right is a tool you will actually run.

Could Rust do this better?

There are certainly faster ways. A wave of Rust-based R tooling, most of it built on a tree-sitter grammar rather than R’s own parser, runs circles around the R-native tools, Posit’s Air formatter and Etienne Bacher’s jarl linter among them.

Compiled code is not where checktor went, though, at least not yet. Reading structure instead of text is already the idea underneath it, and underneath lintr before it. Handing the reading back to R itself is slower and R-only, but never wrong about R, where a separate grammar is a second definition of the language that can drift from the one R actually runs. If those grammars become canonical, moving the heavy lifting there would be an easy trade. For now, the safest thing to trust about R code is R.

Green checks are for machines

The check can tell you whether a package works. Whether it belongs takes a person, and a person’s attention does not scale the way a checklist does. That attention, the patience to sit with something unfamiliar and work out what it is trying to do, is the scarcest thing in this whole system, and we are asking more of it every year. Anyone, or anything, can produce a plausible package in an afternoon now, and the queue of work waiting to be read only gets longer.

The tempting reply is to answer that flood in kind, to point a language model at your package and ask whether it looks right. But these conventions are not a question that wants a guess. They are settled, and already written as code. A model is expensive to test and quick to invent a rule, or wave a real one through, where a check that runs the rules is right every time. The winning move is not a cleverer guesser. It is taking the code that already knows the rules and making it easy to find and easy to run, with a model on top if you like, to call it and explain what it found.

Spending that attention on quote marks and missing return values is a poor trade. Those are the parts a machine settles in a second, and every one a reviewer has to raise by hand is attention taken from the real question of whether your idea holds together.

checktor has a modest aim. Clear the mechanical rules R CMD check leaves behind, so that when a person reads your package, what they bring is the one thing no tool can, their judgment about the work itself. Run it before you hit submit, and leave their attention for the parts that are worth it.

Installation

checktor is on CRAN:

install.packages("checktor")

You can also install the development version from GitHub:

# install.packages("pak")
pak::pak("coatless-rpkg/checktor")

For the tour of every diagnostic, see the release notes and the project page. And if you want the companion story to this one, peeky has its own write-up in There Are No Secrets in Shinylive.