Discussion questions#
A selling point for adopting a permutation test to evaluate the empirical probability of your null hypothesis is that it does not assume a shape (or, more accurately, a particular distribution) of your test statistic under the null hypothesis. But there is an assumption built into the permutation test that may not be valid in some cases. What is this assumption and how might it not be valid in specific contexts?
Permutation testing is the gold standard for testing your observed effects against a null distribution. Bootstrapping is the gold standard to measure the confidence of a set of estimated effects. Cross-validation is the gold standard to measure the generalizability of a model. Each of these approaches involves the specification of various hyperparameters: p (permutations), m (resamples), or k (folds). In a perfect world we could maximize the benefit from all three, but the interaction of these hyperparameters in some cases increases the computational demand exponentially. Are there ways to have our cake and eat it too? Are there cases when we can be lax on some parameters rather than others?