How to debug your package on Win-Builder before sending to CRAN?

TL;DR

To debug your code on Win-Builder, you can create unit tests that save outputs inside the “testthat/” directory. Save all meaningful, intermediate outputs that you want to open or run locally after download. Compress these files in a zip file using utils::zip() inside this same directory. Then, on the Win-Builder page dedicated to your package, you will be able to download it and explore it locally.

Note that some files format are not downloadable from the server, this is why I recommend a zip file.

Saving files inside the “testthat/” directory is a practice to avoid for a submission. You are supposed to let a clean state after you leave tests. However, here, you’re doing it for debugging. If you want to keep debugging code for future use, but let the state clean once everything is set up for CRAN, use a parameter in your tests.
For instance, here is a part of my unit test code for {gitdown}:

# debug if problems with CRAN to retrieve outputs
debug <- FALSE

repo_input <- tempfile(pattern = "git2r-")
repo <- fake_repo(repo_input)

# Classic use
with_dir(repo, {
  res <- git_down(author = "Cervan",
                  pattern = c("Issue" = "#[[:digit:]]+"),
                  open = FALSE
  )
})

test_that("git_down function",{
  expect_match(res, regexp = ".html")
  expect_true(file.exists(file.path(dirname(res), "section-issue.html")))
})

# clean dir
if (debug) {
  dir.create("res_one")
  file.copy(dirname(res), "res_one", recursive = TRUE, overwrite = TRUE)
  # Zip it for download
  utils::zip(zipfile = "res_one.zip", files = "res_one")
}

You’re not really ready for CRAN submission, until you are…

I decided to submit {gitdown}, a package that builds a bookdown report of commit messages arranged according to a pattern to CRAN. I thought it was ready to go:

• There are working reproducible examples for each exported function
• Code coverage is high, preventing many mistakes for future maintainance:
• Checks pass locally
• Checks pass on GitHub Actions for every OS:
• Not to say, there is also documentation with vignettes and Readme. It is not mandatory for CRAN, but it is needed if you want people to be able to use your package.

I followed all steps of the ThinkR guide: “Preparing your package for a CRAN submission”. This includes checks using {rhub}, a package that allows to check packages on several (recent, old, weird) platforms, using rhub::check_for_cran().

Here comes the final checking step: Win-builder. This is a service provided by Uwe Ligges (CRAN maintainer for Windows binary packages), which is “intended for useRs who do not have Windows available for checking and building Windows binary packages”. This is a good taste of checks that will happen on CRAN servers as not all checks are included in the automated rcmdcheck::rcmdcheck(args = "--as-cran") function. That’s where my problems come in.

On this specific Windows platform, one of my unit test failed and I could not reproduce it, neither on my local Windows installation, nor on both Windows configurations on GitHub Actions: {os: windows-latest, r: 'release'} and {os: windows-latest, r: 'devel'}. The output of one of my functions was different for this specific platform.

I was not ready to submit to CRAN…

Use log file to test remote outputs locally

On the Win-Builder server, you can get the log of the checks and some other files.

As you would do locally, you look at the checks. This allows to track the source of the error and then be able to guess its source.
In my case, a unit test failed. This is related to knitting a Rmarkdown file into HTML, but the test says that the output HTML file does not exists.

Maybe my tests were not verbose enough. Maybe I could test some more. But how can I get the Rmd file that could not be knitted properly?

With Win-Builder, you not only get the check log file, you can retrieve examples code and outputs, and the content of the “tests/” directory after running the checks, if you click on “examples_and_tests” above.
I used this possibility to add code inside my unit tests in order to get all the intermediate steps and files I needed to try to debug locally.

Not all files are readable and downloadable for Win-builder

I was interested in the Rmd file that conducted to the failing HTML compilation. However, some files extensions are not readable on the server, nor are they downloadable.

You can read and download:

• “.R” files
• “.txt” files
• “.html” files
• “.zip” files
• “.css” files
• “.js” files

You can not read nor download:

• “.Rmd” files
• “.rds” files
• “.yml” files

Hence, you better zip everything you need in a zip file, like intermediate R object, files. Then, you are able to download and explore them locally.

Remember to create these code parts conditionally so that all those files useful for debugging are not created when you set everything up for sending to CRAN. For instance, here is a part of my unit test code for {gitdown} with condition for debug:

# debug if problems with CRAN to retrieve outputs
debug <- FALSE

repo_input <- tempfile(pattern = "git2r-")
repo <- fake_repo(repo_input)

# Classic use
with_dir(repo, {
  res <- git_down(author = "Cervan",
                  pattern = c("Issue" = "#[[:digit:]]+"),
                  open = FALSE
  )
})

test_that("git_down function",{
  expect_match(res, regexp = ".html")
  expect_true(file.exists(file.path(dirname(res), "section-issue.html")))
})

# clean dir
if (debug) {
  dir.create("res_one")
  file.copy(dirname(res), "res_one", recursive = TRUE, overwrite = TRUE)
  # Zip it for download
  utils::zip(zipfile = "res_one.zip", files = "res_one")
}

What about my rmarkdown, pandoc and knit error on Win-builder ?

Well, it appears that, a potentially different version of pandoc was not able to separate text from code included inside backticks…

Here is the difference of markdown code, as seen by git, between before and after modification:

Here the HTML output before the modification of the markdown code. You can see the backticks after Message content: not used to start a code chunk.

And after the modification, with an extra empty line, the code block after Message content: is good.

Continue to share your work, it’s for the good of all

I hope this blog post can help other R package developers reduce their pain when they make their wonderful packages available to everyone!

Note that I was able to detect my problem only thanks to unit tests I put in the package. These tests ensure that outputs are the same whatever the OS, R and packages versions used. Do not minimize the power of unit tests.

Remember that there is a ThinkR guide with all steps “Prepare your package for a CRAN submission”. Also, {usethis}, {devtools} and {rhub} are your friends on this path before the final step: devtools::release().