
Bayesian modelling of cross-temporal decoding generalisation (2D temporal)
2026-10-30
Source:vignettes/temporal_2d.qmd
Bayesian modelling of cross-temporal decoding generalisation
Cross-temporal generalisation analysis investigates how neural representations evolve over time by training a classifier at one time point and evaluating its performance at other time points. This produces a two-dimensional matrix in which each cell represents decoding performance for a particular combination of training and testing times.
Decoding performance along the diagonal reflects classification at matching training and testing times, whereas off-diagonal generalisation provides information about how decoding patterns transfer across time. Broad generalisation patterns may suggest temporally persistent or overlapping discriminative information, while more restricted patterns may indicate time-specific changes.
In this vignette, we demonstrate how to analyse cross-temporal decoding matrices using Bayesian generalised additive multilevel models (BGAMMs), as introduced in Nalborczyk & Bürkner (2025, Appendix A; DOI: 10.1101/2025.08.29.672336). We illustrate how to model decoding performance as a smooth function of training and testing times, account for repeated observations across participants, and identify regions showing posterior evidence for above-chance decoding.
Importing and visualising the data
We use the timegen_data dataset included in neurogam, which contains cross-temporal decoding performance measured using the area under the receiver operating characteristic curve (AUC). Each observation corresponds to decoding performance at a particular combination of training and testing times. For binary classification, an AUC of 0.5 corresponds to chance-level discrimination, whereas higher values indicate better-than-chance classification.
We first average AUC values across participants at each training–testing time combination to visualise the overall temporal generalisation pattern. The diagonal indicates matching training and testing times, while the dashed horizontal and vertical lines indicate stimulus onset (0 s). This visualisation is descriptive: the statistical model introduced below retains the participant-level observations rather than fitting the group-averaged matrix alone.
library(neurogam)
library(ggplot2)
library(scico)
library(dplyr)
# retrieving some EEG data
data(timegen_data)
head(timegen_data)
#> train_time test_time auc participant
#> 1 -0.2 -0.2 0.5639503 1
#> 2 -0.2 -0.2 0.5967508 2
#> 3 -0.2 -0.2 0.4648438 3
#> 4 -0.2 -0.2 0.4935377 4
#> 5 -0.2 -0.2 0.5228509 5
#> 6 -0.2 -0.2 0.4771995 6
# plotting the cross-temporal generalisation matrix
timegen_data |>
summarise(auc = mean(auc), .by = c(train_time, test_time) ) |>
ggplot(aes(x = train_time, y = test_time, fill = auc) ) +
geom_tile() +
geom_vline(xintercept = 0, linetype = 2, linewidth = 0.5) +
geom_hline(yintercept = 0, linetype = 2, linewidth = 0.5) +
geom_abline(slope = 1, intercept = 0, linetype = 2, linewidth = 0.5) +
scale_x_continuous(expand = c(0, 0) ) +
scale_y_continuous(expand = c(0, 0) ) +
coord_fixed() +
theme_bw() +
scico::scale_fill_scico(palette = "vik", midpoint = 0.5, name = "AUC") +
labs(x = "Training time (s)", y = "Testing time (s)")
Fitting a two-dimensional BGAMM
To model cross-temporal decoding performance, we use a BGAMM with a two-dimensional tensor-product smooth over training and testing times. The smooth term t2(train_time, test_time) allows the model to capture potentially nonlinear variation in decoding accuracy across the temporal generalisation matrix. We additionally include a participant-specific random intercept to account for between-participant differences in overall decoding performance, without introducing participant-specific smooths.
Compared with one-dimensional temporal models, two-dimensional BGAMMs can be considerably more computationally demanding because they estimate a smooth surface rather than a single time course. We therefore recommend beginning with a relatively simple model, using moderate basis dimensions (k), before considering more complex random-effects structures.
The example below uses two MCMC chains with within-chain parallelisation, implemented through brms and the cmdstanr backend. Computational requirements depend on the number of observations, basis dimensions, model structure, and available hardware.
For a broader introduction to generalised additive models and tensor-product smooths, see Pedersen et al. (2019).
library(brms)
# fitting a BGAMM with two temporal dimensions
timegen_gam <- brm(
formula = auc ~ t2(train_time, test_time, bs = "tp", k = 25) + (1 | participant),
data = timegen_data,
warmup = 1000,
iter = 3000,
chains = 2,
cores = 2,
backend = "cmdstanr",
threads = threading(threads = 4),
stan_model_args = list(stanc_options = list("O1") )
)Identifying regions of above-chance decoding
After fitting the BGAMM, we can use testing_through_time() to summarise posterior predictions across the training–testing time grid and identify contiguous regions showing evidence for above-chance decoding. Rather than relying exclusively on the theoretical chance level of AUC = 0.5, we can define a small region of practical equivalence (ROPE) around chance to account for minor deviations in decoding performance.
For this example, we derive an empirical threshold from the pre-stimulus portion of the decoding matrix, restricting the calculation to cells where both training and testing times precede stimulus onset. We calculate the 95th percentile of baseline AUC values and use its excess above 0.5 as the ROPE half-width, provided that this excess is positive.
With odds_mode = "directional_rope", positive evidence is quantified using the posterior odds that decoding performance exceeds the upper ROPE boundary. We use a posterior-odds threshold of 10 and restrict cluster detection to the positive direction (threshold_type = "above"). The example below illustrates the complete analysis. Because fitting this model can require several hours, the vignette uses a previously fitted model to produce the displayed results.
# we compute the smallest effect size of interest (SESOI) as the baseline
# (i.e., before stimulus onset) upper 95% quantile of AUC values
baseline_upper_quantile <- timegen_data %>%
filter(train_time < 0 & test_time < 0) %>%
summarise(quantile(x = auc, probs = 0.95) ) %>%
pull()
# fitting the BGAMM to identify clusters
results <- testing_through_time(
# cross-temporal generalisation data
data = timegen_data,
# participant column
participant_id = "participant",
# AUC column
outcome_id = "auc",
# testing against chance level
predictor_id = NA,
# defining training and testing times
time_id = c("train_time", "test_time"),
# basis dimension
kvalue = 25,
# only includes a per-participant varying intercept (no varying smooth)
multilevel = "summary",
use_se = FALSE,
varying_smooth = FALSE,
# defining chance level for AUC values
# chance_level = 0.5,
# defining chance level + SESOI (upper AUC values during baseline)
chance_level = baseline_upper_quantile,
# threshold on posterior odds
threshold = 10,
# testing whether AUC is above chance
threshold_type = "above",
# number of warmup iterations
warmup = 1000,
# total number of iterations (per chain)
iter = 3000,
# number of MCMCs
chains = 2,
# number of parallel cores
cores = 2,
# using with-chain parallelisation (needs 2x4 parallel cores)
threads = brms::threading(threads = 4),
# increasing max_treedepth
stan_control = list(max_treedepth = 15),
backend = "cmdstanr",
file = "cache/timegen_results"
)Summarising and visualising the results
The returned object can be inspected using the dedicated print(), summary(), and plot() methods.
The printed results provide information about the detected clusters, whereas the graphical representation allows us to examine their distribution across training and testing times in relation to the estimated decoding surface. These clusters identify connected regions in which the posterior evidence exceeds the specified threshold. Together, these methods provide a framework for characterising both the estimated temporal generalisation pattern and the strength of posterior evidence for decoding performance beyond a specified reference level.
# print a summary table
print(results)
#>
#> ==== Time-resolved GAM results ================================
#> Analysis: Temporal generalization (2D)
#> Model coordinates: time1, time2
#> Cluster coordinates: time1, time2
#> Contrast: one_sample
#> Posterior odds mode: directional_rope
#> Odds threshold: 10
#> Clusters found: 2
#>
#> Cluster table:
#>
#> sign id n_points time1_min time1_max time2_min time2_max
#> positive 1 314 0.06 0.76 0.08 0.40
#> positive 2 9 0.92 1.00 0.10 0.14
#> =================================================================
# for available colour palettes, see scico::scico_palette_show()
plot(
x = results,
palette = "vik", midpoint = 0.5,
axes_labels = c("Training time (s)", "Testing time (s)")
)