Day 1 · Acquisition, preprocessing, and the GLM

Day 1Session 1.4Wager0:30 hLecture

Reproducible analysis: coding and data-management practices; CANlab tools install check

Lays out what a reproducible, interpretable, well-documented analysis looks like and why it matters: analysis errors are common, and results that others can rerun and build on are more trusted and more useful. Offers concrete recommendations for design, analysis, and sharing, then walks through version control with GitHub and installing the CANlab toolboxes.

Take-aways

  • Write analyses as scripts that rerun end to end, with sanity checks and controls built in, and document every variable.
  • Errors are the norm, not the exception; reuse tested code and get more eyes on it.
  • Use GitHub to keep toolboxes current and to track your own code; clone the CANlab repositories and run canlab_toolbox_setup.

Key terms

  • reproducibility
  • code book
  • positive and negative controls
  • pre-registration
  • prototype to function to toolbox
  • date-stamped reports
  • version control
  • GitHub Desktop
  • CanlabCore
  • canlab_toolbox_setup
Color mesh surface over the Boulder Flatirons
Color mesh surface over the Boulder Flatirons. Lecture 1.4 slides (Wager)

Outline

What the session covers

01Goals: the perfect analysis

  • Reproducibility: you or someone else can rerun the whole analysis, figures and statistics included, with one command.
  • Interpretability and documentation: simple, commented code with standard variable names and a code book explaining every variable and the study context.
  • Code reuse: modular code; common operations use heavily vetted, tested functions.
  • Sanity checks built in: data inspection, assumption checks, and positive and negative controls (effects that should and should not appear).
  • History and sharing: track what changed when analysis options changed; share the data and code needed to reproduce results.

02Why bother

  • Errors are very common; minimize them with code that is reused and tested, clearly laid out, and seen by many eyes.
  • Replication record: about one third of 100 psychology studies replicated; Amgen reproduced 6 of 53 landmark cancer studies; 2 of 18 microarray studies fully reproduced.
  • Nick Barnes (Climate Code Foundation): most professional software is not very good either, so publish your code anyway.
  • Shared, understandable results are trusted and can be reused; publication is a starting point, not the endpoint.
  • Cumulative science: utility is value; building on your own and others' code and results moves the field forward.

03Perfection versus pragmatics

  • The perfect analysis is infinitely slow; the goal is the best analysis you can actually do.
  • Design: include positive controls; pre-register if you are testing a falsifiable hypothesis.
  • Analysis: reuse existing tools; move from GUI to scripts and develop in GitHub; progress from prototype to function to toolbox.
  • Generate time- and date-stamped reports so results are tied to a specific code state.
  • Sharing: share code and models via OpenfMRI and NeuroVault.

04Version control with GitHub

  • Version control lets you share openly or privately, reuse code, integrate changes from multiple contributors, track history, and revert.
  • Install GitHub Desktop, create an account, find a repository, and use Code > Open with GitHub Desktop to clone it.
  • GitHub Desktop shows recent changes and history and pulls the latest version to synchronize your copy.
  • If you develop code: commit changes, open pull requests, or merge branches; everything is also available from command-line git.
  • GIFT is also distributed on GitHub.

05CANlab toolboxes install check

  • CANlab tools (github.com/canlab, canlab.github.io) provide object-oriented data handling, machine learning, batch second-level workflows, mediation, robust regression, HRF modeling, and visualization.
  • Core idea: simple commands on data objects; images are vectorized, which suits pattern-based and machine-learning analyses.
  • Steps: make a local repository folder, clone CanlabCore, Neuroimaging_Pattern_Masks, MediationToolbox, and RobustToolbox, open MATLAB in that folder, and run canlab_toolbox_setup.
  • Workaround: clone individual toolboxes and add them to the path with subfolders.
  • Resources: Principles of fMRI book, canlab.github.io tutorials and walkthroughs, and the fMRI methods chapters (Handbooks 2007 to 2017).
Axial brain slices in a false-color map — Lecture 1.3 slides (Wager)