A Roadmap to (Computational) Reproducibility

Once data are collected…

Concept Reason
Do not use a GUI, use a script. A script can be rerun.
Use an IDE that supports projects, ensuring that references are relative to the root working directory. The script can be ported to another machine.
Do not overwrite variables, generate a new, derived variable. Allows you to trace errors.
Use version control. Track changes over time for workflow transparency.
Maintain Documentation. Keep a record of tacit knowledge.
Use a language and approach that supports literate programming.1 Make your code intelligible to others.
Manage your code to: a) separate exploratory and confirmatory analyses (use notebooks, like RMarkdown); b) maximize reuse (use scripts for frequently reused code); and c) limit accidentally overwriting objects (functional programming). Maintain clear delineations between instructions and outputs (functional programming), minimize mistakes / errors (code reuse), separate hypothesis generation from hypothesis testing (notebookes).
Use open file formats and when possible ones that have formal standards. Maximize persistence in time and portability across software and hardware.
Document your: a) Operating System and version; b) Program and version; c) External packages and version. Provide a replicable computational framework.
Find a way to account for dependencies - these may be language specific, operating specific, hardware specific.2 Provide a “package” that is a replicable framework.


1
“Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do.”

2
How to Account for Dependencies:

  • renv() for language specific
  • containers for OS
  • emulators for hardware
LS0tCnRpdGxlOiAiUm9hZG1hcCB0byBSZXByb2R1Y2liaWxpdHkiCnBhZ2V0aXRsZTogIlJvYWRtYXAgdG8gUmVwcm9kdWNpYmlsaXR5IgpvdXRwdXQ6CiAgaHRtbF9kb2N1bWVudDoKICAgIGNvZGVfZm9sZGluZzogc2hvdyAjIGFsbG93cyB0b2dnbGluZyBvZiBzaG93aW5nIGFuZCBoaWRpbmcgY29kZS4gUmVtb3ZlIGlmIG5vdCB1c2luZyBjb2RlLgogICAgY29kZV9kb3dubG9hZDogdHJ1ZSAjIGFsbG93cyB0aGUgdXNlciB0byBkb3dubG9hZCB0aGUgc291cmNlIC5SbWQgZmlsZS4gUmVtb3ZlIGlmIG5vdCB1c2luZyBjb2RlLgogICAgaW5jbHVkZXM6CiAgICAgIGFmdGVyX2JvZHk6IGZvb3Rlci5odG1sICMgaW5jbHVkZSBhIGN1c3RvbSBmb290ZXIuCiAgICB0b2M6IHRydWUKICAgIHRvY19kZXB0aDogMwogICAgdG9jX2Zsb2F0OgogICAgICBjb2xsYXBzZWQ6IGZhbHNlCiAgICAgIHNtb290aF9zY3JvbGw6IGZhbHNlCmRhdGU6ICIyMDI1LTAyLTI4IgotLS0KCiMjIEEgUm9hZG1hcCB0byAoQ29tcHV0YXRpb25hbCkgUmVwcm9kdWNpYmlsaXR5CgpPbmNlIGRhdGEgYXJlIGNvbGxlY3RlZC4uLgoKOjo6bWQtdGFibGUKfCBDb25jZXB0IHwgUmVhc29uIHwKfCA6LS0tIHwgOi0tLSB8CnwgRG8gbm90IHVzZSBhIEdVSSwgdXNlIGEgc2NyaXB0LiB8IEEgc2NyaXB0IGNhbiBiZSByZXJ1bi4gfAp8IFVzZSBhbiBJREUgdGhhdCBzdXBwb3J0cyBwcm9qZWN0cywgZW5zdXJpbmcgdGhhdCByZWZlcmVuY2VzIGFyZSByZWxhdGl2ZSB0byB0aGUgcm9vdCB3b3JraW5nIGRpcmVjdG9yeS4gfCBUaGUgc2NyaXB0IGNhbiBiZSBwb3J0ZWQgdG8gYW5vdGhlciBtYWNoaW5lLiB8CnwgRG8gbm90IG92ZXJ3cml0ZSB2YXJpYWJsZXMsIGdlbmVyYXRlIGEgbmV3LCBkZXJpdmVkIHZhcmlhYmxlLiB8IEFsbG93cyB5b3UgdG8gdHJhY2UgZXJyb3JzLiB8CnwgVXNlIHZlcnNpb24gY29udHJvbC4gfCBUcmFjayBjaGFuZ2VzIG92ZXIgdGltZSBmb3Igd29ya2Zsb3cgdHJhbnNwYXJlbmN5LiB8CnwgTWFpbnRhaW4gRG9jdW1lbnRhdGlvbi4gfCBLZWVwIGEgcmVjb3JkIG9mIHRhY2l0IGtub3dsZWRnZS4gfAp8IFVzZSBhIGxhbmd1YWdlIGFuZCBhcHByb2FjaCB0aGF0IHN1cHBvcnRzIGxpdGVyYXRlIHByb2dyYW1taW5nLjxzdXA+MTwvc3VwPiB8IE1ha2UgeW91ciBjb2RlIGludGVsbGlnaWJsZSB0byBvdGhlcnMuIHwKfCBNYW5hZ2UgeW91ciBjb2RlIHRvOiBhKSBzZXBhcmF0ZSBleHBsb3JhdG9yeSBhbmQgY29uZmlybWF0b3J5IGFuYWx5c2VzICh1c2Ugbm90ZWJvb2tzLCBsaWtlIFJNYXJrZG93bik7IGIpIG1heGltaXplIHJldXNlICh1c2Ugc2NyaXB0cyBmb3IgZnJlcXVlbnRseSByZXVzZWQgY29kZSk7IGFuZCBjKSBsaW1pdCBhY2NpZGVudGFsbHkgb3ZlcndyaXRpbmcgb2JqZWN0cyAoZnVuY3Rpb25hbCBwcm9ncmFtbWluZykuIHwgTWFpbnRhaW4gY2xlYXIgZGVsaW5lYXRpb25zIGJldHdlZW4gaW5zdHJ1Y3Rpb25zIGFuZCBvdXRwdXRzIChmdW5jdGlvbmFsIHByb2dyYW1taW5nKSwgbWluaW1pemUgbWlzdGFrZXMgLyBlcnJvcnMgKGNvZGUgcmV1c2UpLCBzZXBhcmF0ZSBoeXBvdGhlc2lzIGdlbmVyYXRpb24gZnJvbSBoeXBvdGhlc2lzIHRlc3RpbmcgKG5vdGVib29rZXMpLiB8CnwgVXNlIG9wZW4gZmlsZSBmb3JtYXRzIGFuZCB3aGVuIHBvc3NpYmxlIG9uZXMgdGhhdCBoYXZlIGZvcm1hbCBzdGFuZGFyZHMuIHwgTWF4aW1pemUgcGVyc2lzdGVuY2UgaW4gdGltZSBhbmQgcG9ydGFiaWxpdHkgYWNyb3NzIHNvZnR3YXJlIGFuZCBoYXJkd2FyZS4gfAp8IERvY3VtZW50IHlvdXI6IGEpIE9wZXJhdGluZyBTeXN0ZW0gYW5kIHZlcnNpb247IGIpIFByb2dyYW0gYW5kIHZlcnNpb247IGMpIEV4dGVybmFsIHBhY2thZ2VzIGFuZCB2ZXJzaW9uLiB8IFByb3ZpZGUgYSByZXBsaWNhYmxlIGNvbXB1dGF0aW9uYWwgZnJhbWV3b3JrLiB8CnwgRmluZCBhIHdheSB0byBhY2NvdW50IGZvciBkZXBlbmRlbmNpZXMgLSB0aGVzZSBtYXkgYmUgbGFuZ3VhZ2Ugc3BlY2lmaWMsIG9wZXJhdGluZyBzcGVjaWZpYywgaGFyZHdhcmUgc3BlY2lmaWMuPHN1cD4yPC9zdXA+IHwgUHJvdmlkZSBhICJwYWNrYWdlIiB0aGF0IGlzIGEgcmVwbGljYWJsZSBmcmFtZXdvcmsuIHwKOjo6Cgo8YnIgLz4KPGhyIC8+CjxiciAvPgoKPHN1cD4xPC9zdXA+ICAKIkluc3RlYWQgb2YgaW1hZ2luaW5nIHRoYXQgb3VyIG1haW4gdGFzayBpcyB0byBpbnN0cnVjdCBhIGNvbXB1dGVyIHdoYXQgdG8gZG8sIGxldCB1cyBjb25jZW50cmF0ZSByYXRoZXIgb24gZXhwbGFpbmluZyB0byBodW1hbiBiZWluZ3Mgd2hhdCB3ZSB3YW50IGEgY29tcHV0ZXIgdG8gZG8uIgoKPHN1cD4yPC9zdXA+ICAKSG93IHRvIEFjY291bnQgZm9yIERlcGVuZGVuY2llczoKCiogcmVudigpIGZvciBsYW5ndWFnZSBzcGVjaWZpYwoqIGNvbnRhaW5lcnMgZm9yIE9TCiogZW11bGF0b3JzIGZvciBoYXJkd2FyZQoKCg==