How to Science

Notes on methodology and statistics, including literature reviews and R code. A work in progress — spotted a mistake or disagree with something? Open an issue or get in touch.

Methodology

Survey Design

Response OptionsHow to choose the number, polarity, and labeling of scale response options Response OrderHow the left-to-right or top-to-bottom ordering of scale options affects survey responses Clarification FeaturesWhether tooltips and inline definitions help respondents interpret survey questions correctly Data Quality ChecksTechniques for detecting inattentive respondents using attention, comprehension, and commitment checks

Scale Development

Item DevelopmentGuidelines for writing and pilot-testing items that validly cover a construct's full range Reliability and ValidityHow to assess whether a scale measures consistently and captures what it claims to measure

Contingent Valuation

WTPMethods and pitfalls for eliciting willingness-to-pay using contingent valuation surveys

AI in Research

LLM BasicsHow large language models work.Synthetic Survey RespondentsUsing LLMs as synthetic survey respondents (silicon sampling), and when it holds up LLM DeterminismWhy language models give different answers to identical prompts, and how to make their output reproducible

Translation

TRAPDThe five-step Translation, Review, Adjudication, Pretesting, and Documentation framework for translating survey instruments.Translation ValidationMethods for checking whether texts were translated correctly.Machine TranslationWhen machine translation can stand in for human survey-translation protocols, used with care

Data Analysis

Regression

CollinearityWhy correlated predictors widen standard errors and why model-building thinking matters more than fixes How Regression Predictions WorkWhat a regression model's predictions actually are, illustrated with R's predict() function, and how confidence intervals differ from prediction intervals Controlling for CovariatesHow to compute covariate-adjusted marginal means using emmeans and marginaleffects in R

Factor Analysis

Statistical Inference

Confidence Interval OverlapWhy eyeballing whether two confidence intervals overlap is an unreliable test of whether two groups differ significantly.

Group Differences

Family-Wise ErrorWhat family-wise error is, how the claim being tested determines whether it applies, and when correction is worth the loss in power.The Omnibus F-TestWhat it is, when to use it, and when to use something else.Pairwise ComparisonsEstimating and testing pairwise differences between group means with emmeans Sequential AnalysesBayesian sequential testing: tracking Bayes factors as data accumulate

Representativeness

Non Response NPSHow non-probability samples behave and when they can approximate population estimates Survey-Weighted EstimationHow sampling design and weights shape population estimates from survey data

Meta-Analysis

BasicsA basic introduction to meta-analysis.Dependent Effect SizesCorrecting standard errors for non-independent effect sizes, from known covariance matrices to robust variance estimation Multilevel Meta-AnalysisDecomposing within-study and between-study variance when studies report multiple effect sizes

Qualitative Data Analysis

Text Categorization with LLMsClassifying open-ended text into categories with LLMs, and validating the output Measuring Rater AgreementQuantifying how consistently raters or LLMs agree, and choosing the right metric

Data SimulationCommon simulation techniques to test out statistical methods and build intuitions.String SimilarityComparing strings with edit-distance, n-gram, and embedding-based similarity metrics

Materials

Scales

Animal Attitude ScaleMeasures attitudes toward the use and treatment of animals BFI-2-XSUltra-brief 15-item measure of the Big Five personality traits Domain-Specific Risk-Taking (DOSPERT) ScaleRisk-taking tendencies across financial, health, social, and other domains New Ecological Paradigm ScaleEnvironmental attitudes about humanity's relationship with nature Oxford Utilitarianism ScaleMeasures impartial beneficence and instrumental harm utilitarian tendencies The Optimism–Pessimism Short Scale–2Brief 8-item measure of dispositional optimism and pessimism TIPITen-item personality inventory for the Big Five traits Speciesism ScaleMeasures attitudes toward different species and species-based discrimination

About

AboutWhat this site is and the approach behind it.Formatting FeaturesA reference for the citation, callout, code, and math features available when writing pages.