About the sample mean

The experimental activity considered in this post consists of acquiring a dataset of measurements of some quantity at discrete instants of time that, without loss of generality, can be assumed equally spaced by an interval .

The measurand, whose physical nature is immaterial for the present discussion, is taken to be constant and equal to . Observations are affected by additive noise modeled as a random variable with zero mean and constant variance .

The resulting sequence can therefore be written as

where the explicit time dependence has been omitted for the sake of clarity.

If measurement errors are furthermore assumed to be independent, a basic probabilistic model consists of independent identically distributed (IID) random variables , whose realizations are . No assumption has yet been made about the probability distribution of .

A central result of estimation theory is that the unknown quantity can be estimated using the sample mean, defined as

Since is itself a random variable, the available dataset yields the numerical estimate

The expectation of is

This holds for every value of , meaning that the estimator is unbiased.

The variance of is

showing the decrease of uncertainty as the number of observations increases. This result ultimately relies on the fact that the variance of a sum is equal to the sum of the variances if the added terms are independent (or at least pairwise uncorrelated).

Moreover, provided that is finite, the central limit theorem implies that, for sufficiently large , the distribution of becomes approximately Gaussian even when the original noise distribution is not.

All these results are mathematically elegant, but they also hide an important practical issue: in real experimental settings, collecting a “large” number of genuinely independent observations may be much more difficult than it first appears.

Under the additional assumption of Gaussian noise, the sample mean enjoys several important optimality properties. In particular, it coincides with:

• the least-squares (LS) solution,
• the maximum-likelihood (ML) estimator,
• the minimum mean-square error (MMSE) estimator.

From an LS perspective, the problem of estimating a constant quantity can also be interpreted as a particularly simple regression model. Since data are acquired sequentially, the acquisition index (or time itself) may be formally regarded as a predictor variable, while the observations play the role of the response.

In regression terms, computing the sample mean corresponds to fitting an intercept-only model, or equivalently a linear model with zero slope. Importantly, the LS solution does not require Gaussian assumptions. In fact, the regression coefficients are obtained simply by minimizing a quadratic cost function, which leads to a linear system of equations when its gradient is set to zero. This procedure produces a straight line that always passes through the centroid of the observed data. When the slope is explicitly constrained to zero, the LS solution coincides with the sample mean. Probabilistic assumptions on instead determine the statistical properties and optimality characteristics of the estimator. In particular, under Gaussian assumptions, the sample mean is also the MMSE estimator.

Before leaving this section, one further observation is worth making: the sample mean admits a recursive formulation. If denotes the estimate computed from the first observations, then the arrival of a new observation produces the updated estimate

This recursive expression is algebraically equivalent to the batch computation of the sample mean in Equation 1, but it highlights a key idea: the estimate can be updated sequentially using the previous estimate and each newly acquired observation. As increases, the weight assigned to new data points progressively decreases.

The Kalman filter in a nutshell

When Rudolf E. Kálmán (1930–2016) introduced his recursive filtering framework in the early 1960s, the problem was not merely statistical. Modern dynamical systems — aircraft, missiles, spacecraft — were generating streams of noisy observations that needed to be processed sequentially and in real time. Batch estimation techniques already existed, but recomputing the entire estimate from scratch as new data arrived was computationally unattractive, especially on the constrained onboard computers of that era.

Kalman’s key insight was that, under linear Gaussian assumptions, the estimate could instead be updated recursively: each newly acquired observation could be combined with the current state estimate and its uncertainty to produce an improved estimate, without storing or reprocessing the entire measurement history. Equally important, if not revolutionary, was the adoption of a state-space approach to estimation, in which both system dynamics and the measurement process could be modeled explicitly and treated within a unified probabilistic framework.

Over the decades, Kalman’s original linear framework has been extended in many directions, including nonlinear filtering approaches such as the Extended Kalman Filter (EKF), the Unscented Kalman Filter (UKF), and particle filters. Despite their substantial algorithmic differences, all these approaches share the same Bayesian genealogical roots: sequentially updating probabilistic beliefs as new information becomes available.

Linear Kalman filter

In its simplest linear form, the Kalman filter (KF) can be regarded as a recursive MMSE estimator for dynamical systems observed through noisy measurements.

The KF alternates between two complementary operations: a prediction step based on the dynamical model, and a correction step driven by incoming measurements, as schematically illustrated in Figure 1.

For a linear discrete-time dynamical system observed through noisy measurements,

where the state vector is , the state transition matrix is square , the measurement vector is , and is an measurement matrix.

The process noise and the measurement noise are modeled as zero-mean Gaussian random vectors with covariance matrices

The covariance matrices and are associated with the assumption that the process and measurement noise components are temporally uncorrelated. Because the noise processes are Gaussian, uncorrelated components are also statistically independent. and fully characterize the second-order statistical structure of the process and measurement noise.

In Kalman filtering notation, the symbols and denote the predicted (a priori) and updated (a posteriori) state estimates at time step , respectively. The corresponding estimation errors are

with covariance matrices

Is a physical quantity ever really constant?

In the previous section, the KF state was assumed perfectly constant, corresponding to the choice . Let us now relax this assumption and consider the more realistic situation in which the state may slowly evolve over time.

The simplest stochastic model for a slowly varying quantity consists of describing the scalar state as a random walk driven by continuous-time white noise with strength :

where

If observations are acquired every seconds, the corresponding discrete-time model becomes

where represents the stochastic increment accumulated over the sampling interval . The variance of this discrete-time process noise is therefore

This simple relation already contains an important physical idea: uncertainty accumulates over time. The longer the interval between consecutive observations, the larger the uncertainty injected during the prediction step.

For the scalar random-walk model,

the KF prediction step is particularly simple. Since and ,

The Kalman gain is therefore

After the measurement update,

Substituting the expression of ,

and therefore

This recursion is the scalar Riccati equation for the random-walk model.

The most interesting behavior emerges when the recursion reaches steady state:

then

Unlike the previous case with , the covariance no longer collapses toward zero because uncertainty is continuously reinjected into the dynamics.

Solving this quadratic equation gives

As soon as , the steady-state covariance remains strictly positive. Consequently, the steady-state Kalman gain is also strictly positive:

The KF therefore never stops listening to new measurements.

Simulations

The following simulation compares two KFs observing the same random-walk process. The first filter uses the correct stochastic model (), so that uncertainty is continuously reinjected during prediction. The second incorrectly assumes a perfectly constant state (), producing the sleepy behavior discussed below.

% ********************************
% Random-walk process simulation
% ********************************

dt    = 1;          % sampling interval, s
T     = 50;         % observation time, s

q     = 1;          % continuous-time noise strength, [a.u.]^2/s
std_q = sqrt(q*dt); % discrete-time process noise std, [a.u.]
std_v = 0.1;        % measurement noise standard deviation, [a.u.]

% ***************
% Path generation
% ***************

rng(1234)

N  = T/dt + 1;
xi = zeros(N,1);

for i = 2:N
    xi(i) = xi(i-1) + sqrt(q*dt)*randn;
end

x = xi;

z_meas = x + randn(N,1)*std_v;

% ******
% Models
% ******

F = 1;
H = 1;

R = std_v^2;

Q_true = std_q^2;
Q_sleepy = 0;

P_0 = Q_true;

% **************
% Initialization
% **************

x_true = zeros(N, 1);
P_true = zeros(N, 1);
K_true = zeros(N, 1);

x_sleepy = zeros(N, 1);
P_sleepy = zeros(N, 1);
K_sleepy = zeros(N, 1);

x_pred_true = 0;
P_pred_true = P_0;

x_pred_sleepy = randn;
P_pred_sleepy = P_0;

for i = 1:N

    % *******************
    % compute Kalman gain
    % *******************

    K_true(i)   = P_pred_true*H/(H*P_pred_true*H + R);
    K_sleepy(i) = P_pred_sleepy*H/(H*P_pred_sleepy*H + R);
  
    % ******
    % update
    % ******

    x_true(i) = x_pred_true + ...
                K_true(i)*(z_meas(i) - H*x_pred_true);

    P_true(i) = (1 - K_true(i)*H)*P_pred_true;

    x_sleepy(i) = x_pred_sleepy + ...
                K_sleepy(i)*(z_meas(i) - H*x_pred_sleepy);

    P_sleepy(i) = (1 - K_sleepy(i)*H)*P_pred_sleepy;

    % *************
    % project-ahead
    % *************

    x_pred_true = F*x_true(i);
    P_pred_true = F*P_true(i)*F + Q_true;

    x_pred_sleepy = F*x_sleepy(i);
    P_pred_sleepy = F*P_sleepy(i)*F + Q_sleepy;
    
end

% ************************
% State estimate evolution
% ************************

plot((0:N-1), x, 'k', 'LineWidth', 2)

hold on

plot((0:N-1), x_true, 'b.', ...
     (0:N-1), x_sleepy, 'r.', ...
     'LineWidth', 1, ...
     'MarkerSize', 18)

legend("process", ...
       "matched KF", ...
       "sleepy KF", ...
       "Location", "northwest")

xlabel('Steps')
ylabel('State [a.u.]')

set(gca, 'FontSize', 14, 'LineWidth', 2)

% ***********************
% Kalman gain evolution
% ***********************

figure

plot((0:N-1), K_true, 'b', ...
     (0:N-1), K_sleepy, 'r', ...
     'LineWidth', 2)

legend("matched KF", ...
       "sleepy KF")

ylim([0 1.25])

xlabel('Steps')
ylabel('Kalman gain')

set(gca, 'FontSize', 14, 'LineWidth', 2)

Figure 2 compares the state estimates produced by the two KFs. While the KF using the correct process model continues to track the evolving random walk, the KF with progressively loses responsiveness.

The consequences become evident in Figure 3. When , the Kalman gain progressively collapses toward zero and the KF gradually stops incorporating new information. By contrast, when , the gain converges to a finite steady-state value, allowing the KF to remain responsive to the evolving dynamics. In other words, the KF never becomes completely certain about the system.

To better visualize the effect of the process noise variance , a second simulation is performed using three different process models obtained by perturbing a nominal value of Q by one order of magnitude in both directions. More precisely, the three KFs use

with , while the measurement noise standard deviation is kept fixed at .

The first KF intentionally underestimates the process uncertainty and behaves as a sleepy filter, characterized by long memory and slow response to incoming measurements. The second KF uses the correct stochastic model and represents a balanced compromise between prediction and measurement update. The third KF largely overestimates the process uncertainty and behaves as an awake filter, rapidly incorporating incoming measurements into the state estimate.

Figure 4 compares the resulting state estimates.

The effect of the different values of becomes even more evident in the corresponding Kalman gains shown in Figure 5.

The process noise variance effectively acts as a tuning knob. Small values of produce KFs with slow response, while large values of produce highly responsive filters that rapidly follow incoming measurements — sometimes too closely.

This behavior is simply another manifestation of the classical trade-off between promptness of response and noise sensitivity encountered in virtually every filtering problem. In practice, mildly overestimating is often less dangerous than underestimating it: an awake filter may become noisy, but a sleepy filter may eventually stop reacting altogether.

Conclusion

Throughout this post, we explored how different assumptions about the measurand lead to different KF behaviors: from the recursive averaging of a perfectly static constant () to a drifting random walk with independent increments (). We saw how the process noise covariance acts as a tuning knob balancing promptness of response and noise sensitivity.

Many physical quantities, however, possess a more structured temporal behavior than a simple random walk. In the next post, we will move one step further by modeling the measurand as a first-order Gauss–Markov process: we will see how the KF manages systems with intrinsic temporal memory.

The physical system

Suppose we have a low-pass RC filter composed of a resistor in series with a capacitor , with the output voltage available across the capacitor plates. If the output terminals are connected to, e.g., an oscilloscope, a noise-like waveform can be observed on the screen, even in the absence of any externally applied input voltage (Figure 1).

Where does this voltage come from? It is well known that thermal noise is generated by the erratic motion of free electrons in the metallic conductor from which the resistor is made. If the resistor is at absolute temperature , the classical Nyquist–Johnson model states that, to an excellent approximation, this thermal noise can be represented as a zero-mean, stationary, Gaussian white process, with flat power spectral density (PSD)

where is Boltzmann’s constant ( J/K).

Using standard circuit theory, the output noise voltage—the one we actually observe—is still Gaussian and stationary, but no longer white. It is colored by the RC low-pass filter, and its PSD is

The variance of the output voltage, equal to its mean-square value since the mean is zero, is therefore

It can also be shown using the Wiener–Khinchin theorem that the autocorrelation function of the output voltage is the inverse Fourier transform of the PSD:

Here, is the time constant of the exponential memory kernel that characterizes the first-order Gauss–Markov process .

So far, everything looks clear. But this standard analytical treatment—familiar from systems theory and signal processing—hides an important fact from view. The approach, rooted in Wiener’s theory, is extremely effective for solving linear ordinary differential equations with constant coefficients and stochastic forcing, almost without giving the impression that one is solving a stochastic differential equation at all.

The crucial step is the formal introduction of white noise: a stationary random process with infinite bandwidth, formally infinite pointwise variance, and impulsive autocorrelation:

where is Dirac’s delta.

Since ideal white noise has infinite bandwidth and infinite power, it should be regarded more as a mathematical idealization than as a literal physical signal—even when it provides an excellent model for phenomena such as thermal noise over the chosen frequency range of interest.

This is where the central question of the post arises: how can we simulate the simple physical system described above in discrete time? And how can we write, in MATLAB, a simulation code that allows us to verify empirically whether the statistical properties of the output are those predicted by the theory of Gauss–Markov processes?

The pieces of the puzzle

MATLAB code

At this stage, the conceptual picture is now in place: the input noise can be modeled as a discrete-time white Gaussian sequence, provided that we remember what this really means.

We are not sampling ideal continuous-time white noise, but generating the Nyquist-rate representation of a band-limited white process whose effective bandwidth is placed beyond the frequency range relevant for the RC filter. This is the correct interpretation of a standard command such as randn() in MATLAB: it produces independent Gaussian samples with finite variance and impulsive autocorrelation in the discrete-time sense.

That variance, however, must be scaled consistently with the sampling interval so that the equivalent continuous-time PSD matches the physical Nyquist–Johnson thermal-noise level. The following code implements a single realization of the RC output voltage using the Tustin discretization discussed above. The input noise is calibrated directly in physical units, so that the output variance emerges naturally from the model and converges to the theoretical value of (Equation 3), rather than being imposed afterward by artificial normalization.

clear
clc
close all

rng(1234)           % Reproducibility

% Physical constants and RC parameters

kB = 1.380649e-23;  % Boltzmann constant [J/K]
T  = 300;           % Absolute temperature [K]
R  = 10e3;          % Resistance [Ohm]
C  = 100e-9;        % Capacitance [F]

tau = R*C;          % Theoretical time constant [s]
sigma2_y = kB*T/C;  % Theoretical output variance [V^2]

% Sampling settings

oversampling = 50;
Ts = tau/oversampling; % Sampling interval [s]
Fs = 1/Ts;             % Sampling frequency [Hz]
BN = Fs/2;             % Effective white-noise bandwidth [Hz]

N = 2^14;              % Number of samples

% Tustin discretization

alpha = 2*R*C/Ts;
b0    = 1/(1 + alpha);
b1    = b0;
a1    = (1 - alpha)/(1 + alpha);

% Input white-noise calibration

N0       = 2*kB*T*R;
sigma2_x = 2*BN*N0;
sigma_x  = sqrt(sigma2_x);

% Generate calibrated discrete-time white Gaussian noise

x = sigma_x*randn(N, 1);

% Filter input noise through the discrete-time RC filter

y = zeros(N, 1);

for n = 2:N
    y(n) = b0*x(n) + b1*x(n-1) - a1*y(n-1);
end

% Empirical variance check

fprintf('Theoretical output variance = %.4e V^2\n', sigma2_y)
fprintf('Empirical output variance   = %.4e V^2\n', var(y))

% Time-domain visualization

t = (0:N-1)*Ts;

figure
plot(t(1:5000).*1e3, y(1:5000), 'b', 'LineWidth', 1)

xlabel('Time [ms]')
ylabel('Output voltage [V]')

grid on
set(gca, "FontSize", 14, "LineWidth", 1.5)

A single realization of the simulated RC output voltage is shown in Figure 2. The waveform no longer resembles ideal white noise. The RC filter introduces temporal correlation, producing the smooth random fluctuations characteristic of a first-order Gauss–Markov process. The signal remains random, but it now carries memory: nearby samples are no longer statistically independent.

This is exactly the stochastic structure predicted by the continuous-time model discussed above.

Why autocorrelation estimation is always hungry for data

The final piece of the puzzle is not the simulation itself, but its statistical verification. Generating a discrete-time realization of the RC output voltage is easy. Demonstrating that its statistical properties actually agree with the theory of the Gauss–Markov process is much less trivial. In particular, the key quantity to recover is the exponential autocorrelation law in Equation 4, from which both the stationary variance and the characteristic time constant (RC) can be estimated.

A single realization is never enough. Even when the model is perfectly correct, empirical autocorrelation estimates are noisy, strongly variable at large lags, and notoriously hungry for data. Second-order statistics require long observation windows, and the apparent quality of a simulation often depends more on record length than on the model itself.

To make this visible, a Monte Carlo strategy is particularly useful. Rather than generating many completely independent short simulations, it is often more convenient to generate one very long stationary sequence, discard the initial transient needed to reach equilibrium, and then divide the remaining portion into many consecutive segments of equal length.

In MATLAB, this can be done naturally by reshaping the vector into a matrix:

• rows represent time evolution within each realization
  
• columns represent approximately independent replications

This provides a practical ensemble over which autocorrelation functions can be estimated and compared.

% The physical constants, sampling settings, and Tustin coefficients
% are the same as in the single-realization example above.

rng(1234)           % Reproducibility

% Monte Carlo settings

nRep       = 200;             % Number of Monte Carlo repetitions
nPerRep    = 2^14;            % Samples per repetition
nTransient = 20*oversampling; % Initial transient to discard
Ntot       = nTransient + nRep*nPerRep;

% Input white-noise calibration

N0       = 2*kB*T*R;
sigma2_x = 2*BN*N0;
sigma_x  = sqrt(sigma2_x);

% Generate calibrated discrete-time white Gaussian noise

x = sigma_x*randn(Ntot, 1);

% Filter input noise through the discrete-time RC filter

y = zeros(Ntot, 1);

for n = 2:Ntot
    y(n) = b0*x(n) + b1*x(n-1) - a1*y(n-1);
end

% Remove transient and reshape into Monte Carlo replications

y_stat = y(nTransient+1:end);

fprintf('Theoretical output variance = %.4e V^2\n', sigma2_y)
fprintf('Empirical output variance   = %.4e V^2\n', var(y_stat))

Y = reshape(y_stat, nPerRep, nRep);

% Autocorrelation estimation

maxLag = round(8*tau/Ts);
lags   = (0:maxLag)';
tLag   = lags*Ts;

acfMat = zeros(maxLag+1, nRep);

for r = 1:nRep
    yr = Y(:, r);
    yr = yr - mean(yr);

    % The biased estimator is used to obtain a smoother and more
    % stable estimate of the exponentially decaying autocorrelation.
    acfFull = xcorr(yr, maxLag, 'biased');

    acfMat(:, r) = acfFull(maxLag+1:end);
end

acfMean = mean(acfMat, 2);
acfLow  = prctile(acfMat, 2.5, 2);
acfHigh = prctile(acfMat, 97.5, 2);

acfTheory = sigma2_y*exp(-tLag/tau);

% Plot autocorrelation estimates

figure
hold on

fill([tLag(2:end); flipud(tLag(2:end))]/tau, ...
     [acfLow(2:end); flipud(acfHigh(2:end))], ...
     [0.85 0.85 0.85], ...
     'EdgeColor', 'none')

plot(tLag/tau, acfMean, 'b', 'LineWidth', 2)
plot(tLag(1:10:end)/tau, acfTheory(1:10:end), ...
     'k.', 'MarkerSize', 18)

xlabel('Lag / RC')
ylabel('Autocorrelation [V^2]')

legend('95% Monte Carlo band', ...
       'Monte Carlo mean', ...
       'Theory', ...
       'Location', 'northeast')

grid off
set(gca, "FontSize", 14, "LineWidth", 1.5)

% Estimate time constant from semilog regression

fitMaxLag = round(3*tau/Ts);
fitIdx    = (1:fitMaxLag)';   % Exclude lag zero

tauHat = zeros(nRep, 1);

for r = 1:nRep
    acf_r = acfMat(:, r);

    yFit = acf_r(fitIdx + 1);

    % Keep only positive values before taking logs
    valid = yFit > 0;

    xReg = tLag(fitIdx(valid));
    zReg = log(yFit(valid));

    p = polyfit(xReg, zReg, 1);

    tauHat(r) = -1/p(1);
end

% Summary of estimated time constants

tauMean = mean(tauHat);
tauCI   = prctile(tauHat, [2.5 97.5]);

fprintf('Theoretical tau = %.4e s\n', tau)
fprintf('Estimated tau   = %.4e s\n', tauMean)
fprintf('95%% Monte Carlo CI = [%.4e, %.4e] s\n', ...
        tauCI(1), tauCI(2))

Two situations are especially instructive:

1. Experimentally realistic regime
   Longer records allow the expected exponential law to emerge clearly, with confidence intervals narrow enough to support reliable parameter estimation.

2. Data-poor regime
   Short records produce unstable autocorrelation estimates, large uncertainty bands, and poor estimates of the decay constant.

The decay constant can be estimated by exploiting the exponential form of the theoretical autocorrelation. Taking logarithms,

so that the time constant can be recovered by linear regression on the semilog scale.

From repeated Monte Carlo replications, one obtains not only a point estimate of (RC), but also an empirical confidence interval for the estimate itself. This final step closes the loop between theory and simulation: not merely generating a noise-like waveform, but verifying quantitatively that the stochastic process observed numerically is statistically consistent with the physical model from which it originated.

And this is precisely where simulation stops being coding and becomes measurement.

The Monte Carlo results for the experimentally realistic regime are shown in Figure 3.

The agreement between the empirical mean autocorrelation and the theoretical exponential law of Equation 4 is excellent. The shaded region represents the empirical 95% confidence band obtained from repeated stationary segments of the same long realization. Even with a correctly specified model, the uncertainty remains substantial, especially at larger lags where autocorrelation estimates become progressively less stable.

In the present simulation, the theoretical time constant was

while the Monte Carlo estimate obtained from semilog regression gave

with empirical 95% confidence interval

The mean estimate is accurate, while the relatively wide confidence interval confirms how demanding autocorrelation-based parameter estimation can be, even when the underlying model is perfectly correct.

The contrast becomes even clearer in the data-poor regime shown in Figure 4.

Repeating exactly the same procedure with much shorter records ( rather than ) produced

with empirical 95% confidence interval

The mean estimate remains reasonable, but the uncertainty increases substantially. Although the physical time constant of the system is only ms, reliable estimation of that constant from autocorrelation requires observation windows hundreds or even thousands of times longer:

The process forgets quickly; the statistic does not.

Concluding remarks

At first sight, simulating thermal noise in an RC filter looks like a very simple problem. A resistor generates white noise, the filter shapes it, and a few lines of MATLAB code seem enough to reproduce the expected stochastic output. The real difficulty appears only when one asks a slightly more careful question: what exactly is being simulated?

Ideal continuous-time white noise, with infinite bandwidth and impulsive autocorrelation, cannot be sampled in any literal sense. Trying to do so leads immediately to paradoxes involving aliasing, infinite power, and meaningless pointwise variance. The resolution is not mathematical trickery, but physical modeling. Real systems always place bandwidth somewhere.

If thermal noise is assumed to be effectively white only over a sufficiently large frequency range—larger than the bandwidth of the RC filter itself—then discrete-time simulation becomes perfectly well defined. A Gaussian white sequence generated numerically is not the impossible sampling of an idealized process, but the Nyquist-rate representation of a band-limited stochastic process. The filter then completes the picture: the discrete-time implementation is not merely a numerical approximation, but a way of preserving the structure of the underlying Gauss–Markov process.

And finally, statistics reminds us that even a correct model is not enough. Recovering exponential autocorrelation from finite data requires long records, careful estimation, and a healthy respect for the fact that second-order statistics are always hungry for data.

In the end, the most useful question is not

how do we simulate white noise?

but rather

where does physics place bandwidth?

Everything else—including randn, Tustin discretization, and Monte Carlo verification—comes only after that.

Final note

Give an engineer a noisy oscilloscope trace, and they will immediately start looking for hidden periodicities.

“Wait… is that mains hum?
A badly placed notch?
Some forgotten pole?
Why does that look suspiciously like 50 Hz?
And does the time base actually support that conclusion?”

At that point, the waveform stops being “just noise” and becomes an investigation. Perhaps that is exactly how it should be. For an engineer, noise is rarely just noise. It is usually a signal that is trying—sometimes very stubbornly—to reveal the physical system that generated it. And yes, looking at the thumbnail of this post, I am still not entirely convinced that the oscilloscope trace is innocent.

Problem statement

Consider the simplest possible problem in Newtonian kinematics: the motion of a point along a straight line. Its position, velocity and acceleration are denoted by and , respectively. The equations of motion are elementary:

Here and throughout the post, the dot notation denotes differentiation with respect to time.

If the acceleration is known as a deterministic function of time, the solution is trivial: integrate once to obtain the velocity, and integrate again to obtain the position, starting from known initial conditions.

In many practical problems, however, acceleration is unknown. Examples abound:

• a vehicle subject to unpredictable maneuvers,
• a moving object affected by small, unmodeled disturbances,
• a target whose motion can only be described approximately.

In all these cases, acceleration must be modeled, rather than specified explicitly. This raises the apparently simple question of how the acceleration should be represented.

At first sight, the answer seems obvious: treat acceleration as a stochastic process, that is, specify the statistical rules governing its time evolution. More precisely, acceleration can often be described as the superposition of deterministic and stochastic components. For instance, a known nominal motion—such as constant velocity or constant acceleration—may be affected by unpredictable perturbations. In this post, the focus is on the stochastic component alone, in order to isolate and understand its effect on the statistical properties of position (and velocity).

Even when the equations of motion are trivial, the modeling choices for the input have profound consequences — especially when we recall that, while the motion of the point naturally occurs in continuous time, in practice we study it in discrete-time:

• measurement data are sampled,
• tracking algorithms operate on sequences rather than continuous trajectories,
• computer simulations are run with a fixed time step.

When the input to the system is deterministic, this passage is straightforward. However, when the input is stochastic, the transition from continuous time to discrete time can become surprisingly subtle.

This post is about understanding what it really means to discretize Equation 1 when is modeled as a stochastic process. The goal of this post is to show that, while the physical problem is simple, the associated modeling choices are not trivial.

The motion model

Before introducing any stochastic description, it is useful to clarify what is given and what is unknown in the motion model itself.

The kinematic structure of the problem is completely known. The relationships between position, velocity, and acceleration are fixed by Newtonian kinematics, as shown in Equation 1. From a systems point of view, is not treated as a state variable, but as an external input to an otherwise deterministic system.

Because velocity and position are obtained by integrating acceleration, any assumption made on the acceleration propagates through the system. Small differences in how acceleration is modeled can result in large differences in the statistical properties of position and velocity, once the model is discretized.

The state of the system described by Equation 1 is defined as:

The continuous-time kinematic model is:

with:

Consider sampling the motion at times , with sampling interval . For any input (deterministic or stochastic), the exact solution at the sampling instants is:

where the state transition matrix is

and the input vector can be written:

This is a standard result in systems theory: discretization affects the deterministic state transition through the matrix exponential , while the input contributes through an integral that “accumulates” its effect over the sampling interval.

It is worth noting that the discrete-time representation in Equation 5, with as in Equation 6 and as in Equation 7, is exact at the sampling times and holds regardless of how acceleration is modeled. What is still unspecified is the statistical characterization of , which depends entirely on the modeling assumptions made for .

Different assumptions on the time evolution of acceleration (for instance, whether it is memoryless or correlated) lead to different statistical properties of , and therefore to different statistical properties of the state at the sampling times. In particular, the position component is obtained by integrating acceleration twice (via velocity). As a result, the statistical behavior of position is highly sensitive to how acceleration is modeled. In many applications, position is the central quantity of interest: we care about predicting where the point will be, often more than how fast it will be moving.

The discrete-time input covariance

A specific stochastic model is now introduced: acceleration is regarded as a wide-sense stationary stochastic process with zero mean, , and autocorrelation function:

In the constant-velocity kinematic model, a zero-mean acceleration describes random deviations around a nominal constant velocity, with no preferred direction of motion.

The covariance matrix of is defined as

where the last equality requires that (this follows from the linearity of both the expectation operator and the integral).

Substituting the expression for in Equation 7 into Equation 9 yields:

Under the stationarity assumption, the expectation depends only on , and we obtain:

This expression makes the central point explicit: the discrete-time covariance is determined by the autocorrelation structure of acceleration over the sampling interval.

For the one-dimensional constant-velocity kinematic model, with and as in Equation 4 and Equation 6 respectively, we can write:

As a result, the discrete-time covariance matrix can be written explicitly as

Written component-wise, this expression highlights how each entry of is obtained by weighting the autocorrelation function of acceleration with deterministic kernels:

At this point, no assumption has yet been made on the specific form of . The structure of is entirely determined by the kinematics; the autocorrelation function of acceleration determines its numerical values.

White noise acceleration as a limiting model

We now consider a specific, and widely used, stochastic model for acceleration: white noise acceleration. Formally, acceleration is modeled as a zero-mean stochastic process with autocorrelation

where denotes the Dirac delta and the noise strength is a constant. This expression should not be interpreted literally as a function: white noise does not exist as a time function in the usual sense. Rather, it is a mathematical idealization that captures the idea of rapidly varying, uncorrelated fluctuations. What matters for our purposes is that the autocorrelation is impulsive, and that all second-order properties of the process are encoded in the single parameter .

The parameter has a clear physical dimension. Acceleration has units of length over time squared, , the Dirac delta has units of inverse time, and the autocorrelation has units . It follows that This dimensional analysis makes explicit that is the (constant) power spectral density (PSD) of the white noise acceleration.

Substituting Equation 15 into Equation 14, each double integral defining collapses to a single integral:

Evaluating the integral yields the well-known result:

This matrix summarizes, in discrete time, the cumulative effect of continuous-time white acceleration over one sampling interval.

It is worth stressing that the discrete-time covariance depends not only on the noise PSD, but also explicitly on the sampling interval . Changing without recomputing therefore amounts to changing the underlying model, not merely its numerical implementation.

Concluding remarks

The goal of this post was not to provide an exhaustive discussion of stochastic acceleration models, nor to explore the interpretation of white noise as the limit of rapidly fluctuating correlated processes.

Rather, the focus was on understanding how a simple kinematic model, when driven by a stochastic input, can be consistently carried from continuous time to discrete time, and how the statistical properties of that input propagate into the discrete-time state increments.

Within this framework, white acceleration was introduced as a convenient and widely used specialization that leads to a closed-form expression for the discrete-time covariance . It should be regarded as one modeling choice among others, not as a physical description of acceleration itself.

What matters, ultimately, is not the specific choice of model, but the clarity with which its assumptions are stated and their consequences understood. Even for the simplest possible motion problem, careful modeling at this level can make a substantial difference.

Why multivariate thinking is important

In many experimental settings, statistical analysis still proceeds one response variable (outcome) at a time. Group differences are assessed through a sequence of tests applied to single outcomes, implicitly assuming that each carries independent and self-contained information. This approach is widely regarded as inadequate when variables are numerous and correlated, as is often the case in empirical data drawn from natural, social, or cultural processes.

Multivariate statistics addresses this limitation by shifting the focus from individual outcomes to configurations of outcomes. Rather than asking whether a single response differs across groups, the central question becomes whether groups occupy distinct regions of a multidimensional space, and how such separation relates to the variability observed within each group.

From this perspective, multivariate methods are less about hypothesis testing in the narrow sense, and more about geometry. Concepts such as centroids, dispersion, and variance–covariance structure characterize the data before any formal test is performed. Understanding what a given method assesses therefore requires understanding the space in which the data are represented.

This post focuses on one widely used approach to multivariate group comparison: Multivariate Analysis of Variance (MANOVA). The aim here is to clarify what the method actually measures, under which conditions it is informative, and how it can be used responsibly in exploratory analyses, either directly or after Principal Component Analysis (PCA).

A pedagogical example for multivariate reasoning

To ground the discussion, it is useful to start from a simple and well-understood example. The iris dataset is a canonical choice in this respect, not because it is representative of real-world complexity, but because it allows key geometric ideas to be illustrated without unnecessary distractions.

The dataset consists of 150 observations from three biological species (Iris setosa, Iris versicolor, Iris virginica). Each observation is described by four continuous measurements: sepal length, sepal width, petal length, and petal width.

While iris is often used to demonstrate classification performance, it serves a different purpose here: providing a controlled environment in which the relationship between data representation, group separation, and variability can be examined explicitly.

Principal component analysis as a geometric transformation

Principal component analysis (PCA) is often introduced as a dimensionality reduction technique. While this description is not incorrect, it can be misleading if taken at face value. Conceptually, PCA is best understood as a change of coordinates: the original variables are replaced by a new set of orthogonal axes that capture maximal variance.

In the context of multivariate group comparison, PCA can play a crucial preparatory role. By projecting the data into a lower-dimensional space that preserves most of the variance, it becomes easier to visualize group structure and to reason about distances and dispersion.

Importantly, PCA does not create separation between groups. Any apparent separation observed in principal component space reflects structure already present in the data. PCA merely provides a coordinate system in which that structure can be more easily inspected.

A first look at principal component space

As shown in Figure 1, the three species occupy partially overlapping yet distinct regions of principal component space.

Explained variance and dimensionality

Principal components are ordered by decreasing variance. Reporting the proportion of variance explained is useful to justify why a low-dimensional projection (typically the first two components) is an informative geometric summary.

In this example, the first two components capture a substantial fraction of total variability, supporting their use for visualization and for subsequent multivariate comparisons in component space.

Table 1 reports the proportion of variance explained by each component.

Explained variance and cumulative variance for the iris PCA.

Component	Proportion of variance explained	Cumulative
PC1	0.730	0.730
PC2	0.229	0.958
PC3	0.037	0.995
PC4	0.005	1.000

The scree plot in Figure 2 provides an alternative view of the information reported in Table 1.

MANOVA: assumptions, scope, and interpretation

Multivariate Analysis of Variance (MANOVA) extends the logic of univariate ANOVA to situations in which each observational unit is described by multiple response variables. Rather than testing group differences on individual outcomes one at a time, MANOVA evaluates whether groups differ in their multivariate location, taking into account the joint distribution of the responses.

MANOVA in principal component space

With the conceptual and methodological groundwork in place, the combination of PCA and MANOVA can now be operationalized. In this setting, PCA defines a low-dimensional space that captures the dominant structure of the data. MANOVA is then applied to the corresponding component scores to quantify group separation within that space.

Operationally, this amounts to replacing the original response variables with a selected subset of principal component scores. These scores serve as multivariate responses in the MANOVA model, while group membership remains unchanged. The logic of the test is identical to that of a conventional MANOVA; what changes is the space in which multivariate separation is assessed.

This approach offers several practical advantages. By working with a reduced number of orthogonal components, the dimensionality of the response space is controlled explicitly, multicollinearity is eliminated by construction, and the ratio between the number of observations and response variables is improved. At the same time, because principal components are ordered by explained variance, the analysis can be restricted to those dimensions that capture the most informative aspects of the data.

In what follows, this procedure is illustrated on the iris dataset by applying MANOVA to the first two principal component scores. This example is not intended to optimize classification or prediction, but to demonstrate how multivariate separation can be measured and interpreted once the data have been embedded in principal component space.

Conclusions

This post argued that PCA and MANOVA address complementary stages of the same analytical problem. PCA defines a low-dimensional geometric representation in which multivariate structure becomes visible, while MANOVA quantifies group separation by comparing between-group and within-group variability within that space.

Using the iris dataset as a pedagogical example, the analysis showed that applying MANOVA in principal component space does not alter the underlying statistical machinery—the model is still fitted via manova()—but changes the geometry on which separation is assessed. In this sense, PCA-based MANOVA should not be viewed as a different test, but as the same test applied to a different, and often more interpretable, representation of the data. Seen in this light, PCA and MANOVA are best understood not as a sequence of techniques, but as complementary tools for reasoning geometrically about multivariate structure.

A central conceptual distinction was emphasized throughout: principal component loadings explain variance, whereas group discrimination must be assessed on the distribution of scores across groups. Confusing these two roles leads to common misinterpretations, particularly when PCA is used as a preparatory step for group comparison.

More broadly, the discussion highlighted the value of multivariate methods as tools for geometric reasoning rather than as mere extensions of univariate testing. By shifting attention from individual variables to configurations, centroids, and dispersion, PCA and MANOVA encourage a more structural understanding of group differences.

A natural next step is to revisit the same geometric questions using distance-based approaches, leading to Permutational MANOVA (PERMANOVA). This extension replaces model-based assumptions with permutation-based inference, offering a complementary perspective on multivariate separation that can be explored in a follow-up post.

Principal Component Analysis

The PC vectors are computed by doing the eigenvector-eigenvalue analysis of the covariance matrix :

where the centering matrix , of size , is defined as:

Here, is the identity matrix, and is a column vector of ones of length . The matrix (symmetric and idempotent) acts as a projection operator that removes the mean from each column of , thereby centering the data.

The loading vectors, denoted by , are expressed in terms of the eigenvectors and eigenvalues of :

Since is symmetric and positive semi-definite, its eigenvectors form an orthonormal basis and the associated eigenvalues are real and non-negative.

Let us suppose that the eigenvalues are sorted in decreasing order. The first PC is then given by the first loading vector , which corresponds to the direction of maximum variance in the data. Each subsequent PC defines a new axis that is orthogonal to the previous ones and captures the largest remaining variance.

The set spans the reduced feature space onto which the data are projected.

The associated score vectors are obtained by projecting the original data onto each loading vector:

These form the columns of the score matrix , where each row represents a transformed observation in the low-dimensional space. This is the same matrix introduced earlier in the general model formulation.

PCA identifies the loading vectors by maximizing the variance of the projected data along each component, subject to orthogonality constraints. This ensures that each PC captures the greatest possible amount of variance not already explained by previous components. Implicitly, the PCA model assumes that the residuals in are uncorrelated, homoscedastic, and normally distributed — although in practice, PCA is often used as a purely geometric transformation without relying on strict distributional assumptions.

Partial Least-Squares Discriminant Analysis

Let us suppose that the class labels for each observational unit are known. In the binary case, the labels can be collected in a column vector ; in the multiclass setting, they are typically represented by a matrix , where is the number of distinct classes and each row is a one-hot encoded vector.

As in PCA, we seek a set of latent components by projecting the original data onto a lower-dimensional subspace. However, unlike PCA, PLS-DA chooses the projection directions in such a way that the covariance between the projected data and the class labels is maximized.

The matrix of cross-covariance is defined as:

and the optimal projection direction (the first loading vector) is found as the leading eigenvector of . This ensures that the first latent component captures as much class-discriminative information as possible in a single direction.

The algorithm proceeds iteratively. Once the first loading vector has been determined, the corresponding score vector is computed, and a linear regression of on is performed to obtain , the weight vector in label space. The outer product yields a rank-one approximation of the class label structure.

In order to extract additional components, the information already captured by the current component is removed from both and through a deflation step. This is achieved by subtracting the contribution explained by ) and , resulting in residual matrices:

The process is then repeated on the residuals to obtain the next components. Each new latent component is orthogonal to the previous ones in , and captures additional class-discriminative information, if available.

Unlike PCA, PLS-DA explicitly exploits the structure of to guide dimensionality reduction, making it a supervised projection method. It is particularly well suited to settings where the number of features is large and class separation is expected to be encoded in specific linear combinations of variables. Similar to PCA, the error structure in PLS-DA follows the same assumptions — namely, centered, uncorrelated, and homoscedastic residuals — although in practice, the method is often used with minimal reference to distributional properties.

PLS-DA can be further extended by introducing sparsity constraints on the loading vectors, yielding sparse PLS-DA (sPLS-DA). In this variant, only a subset of the original variables — those deemed most informative for class separation — are retained in the projection. The number of retained features per component is controlled by the parameter , and the optimization is regularized accordingly.

This formulation enables both class separation and embedded feature selection, which is particularly useful in high-dimensional datasets such as omics or text.

Geometrically, sPLS-DA defines a projection of the data into a low-dimensional latent space, just like PCA, but the directions are chosen to maximize class separability rather than data variance. This distinction is crucial: while PCA may preserve structure unrelated to the class labels, PLS-DA aligns the representation with the supervised task.

As will become evident in the following sections, this shift in optimization criterion can make PLS-DA a powerful tool for supervised dimension reduction. However, it also introduces specific vulnerabilities — particularly in high-dimensional settings with limited sample size — where the method may overfit to noise or to accidental separability. Careful validation is therefore essential to assess the generalization performance of PLS-DA models.

The Great Library Heist

To illustrate the use of dimensionality-reduction and classification techniques in text data, I consider a corpus known as the “Great Library Heist”, originally featured by Silge and Robinson in Text Mining with R (https://www.tidytextmining.com), where it is used to demonstrate the potential of Latent Dirichlet Allocation — one of the most common algorithms for topic modeling.

Topic modeling is a method for unsupervised classification of documents, in this case applied to the chapters of four well-known English novels from Project Gutenberg:

• Great Expectations by Charles Dickens
• Pride and Prejudice by Jane Austen
• The War of the Worlds by H. G. Wells
• Twenty Thousand Leagues Under the Sea by Jules Verne

library(tidyverse)   # Core data wrangling and piping (includes dplyr, tidyr, etc.)
library(ggplot2)     # Grammar of graphics for plots (explicit, though included in tidyverse)
library(MASS)        # For multivariate data simulation (e.g., mvrnorm)
library(tidymodels)  # Unified framework for modeling and machine learning workflows
library(textrecipes) # Text preprocessing steps within tidymodels recipes
library(stringr)     # Consistent string handling (regex, manipulation)
library(mixOmics)    # Multivariate methods including sPLS-DA, PLS, PCA
library(yardstick)   # Performance metrics and confusion matrices
library(patchwork)   # Combine multiple ggplot2 plots into one layout

Following the general strategy introduced by Silge and Robinson, each chapter of the four books in the “Great Library Heist” corpus is treated as a separate document.

In author’s own words: > When examining a statistical method, it can be useful to try it on a very simple case where we know the “right answer”. For example, a set of documents that definitely relate to four separate topics is collected, then topic modeling is performed to see whether the algorithm can correctly distinguish the four groups. This lets us double-check that the method is useful, and gain a sense of how and when it can go wrong.

The goal of the analysis is to construct a term-document matrix based on term frequency (TF) scores, which will serve as the input for both dimensionality reduction and classification. In this classical bag-of-words approach, each document is represented as an unordered collection of words — ignoring grammar, syntax, and word order. What matters is the frequency of each term across documents.

I intend to do the same with sPLS-DA. In doing so, I take inspiration from Saccenti and Tenori, who applied sPLS-DA to the stylometric analysis of the Divina Commedia.

load("books.rda")

df_books <- books %>%
    
  # Identify chaper for each book
  group_by(title) %>%
  mutate(chapter = cumsum(str_detect(text, regex("^chapter\\s", ignore_case = TRUE)))) %>%
  ungroup() %>%
  filter(chapter > 0) %>%
  
  # Prepare label and document name 
  mutate(label = str_squish(title)) %>%
  unite(document, title, chapter, sep = "_") %>%
  
  # All text for each chapter, in each book, is united in a single field
  group_by(label, document) %>%
  summarise(text = str_c(text, collapse = " "), .groups = "drop") %>%
  
  # Remove headings like "CHAPTER I", "CHAPTER TWENTY-ONE", etc.
  mutate(text = str_remove(text, regex("^CHAPTER\\s+.*?\\s+", ignore_case = TRUE))) %>%

  # To get chapters in order
  mutate(chapter_num = as.integer(str_extract(document, "\\d+$"))) %>%
  arrange(label, chapter_num) %>%
  dplyr::select(label, document, text)

set.seed(123)

# First split 80% train_val vs 20% test
split_1   <- initial_split(df_books, prop = 0.80, strata = label)
train_val <- training(split_1)
test_set  <- testing(split_1)

# Second split 95% train vs 5% val (part of training)
split_2   <- initial_split(train_val, prop = 0.95, strata = label)
train_set <- training(split_2)
val_set   <- testing(split_2)

# Check
df_check <- bind_rows(
  train = count(train_set, label),
  validation = count(val_set, label),
  test = count(test_set, label),
  .id = "split"
)
n_chapters <- sum(df_check$n)

# Feature engineering via textrecipes::recipe
text_rec <- recipe(label ~ text, data = train_set) %>%
  step_tokenize(text) %>%
  step_stopwords(text) %>%
  step_tokenfilter(text, max_tokens = 500) %>%
  step_tf(text)

# Prep/bake the recipe
rec_prep <- prep(text_rec)

X_train  <- bake(rec_prep, new_data = NULL) %>% dplyr::select(-label) %>% as.matrix()
y_train  <- bake(rec_prep, new_data = NULL) %>% pull(label)

X_test <- bake(rec_prep, new_data = test_set) %>% dplyr::select(-label) %>% as.matrix()
y_test <- bake(rec_prep, new_data = test_set) %>% pull(label)

Model tuning with adaptive feature selection

In the tune.splsda() step, I explicitly include max_tokens (i.e., the full vocabulary size) in the grid of keepX. This allows the tuning procedure to consider not only sparse sPLS‑DA models, but also the special case of full PLS‑DA with no feature selection. Crucially, tune.splsda() returns a separate optimal keepX for each component, enabling each latent dimension to adaptively select the number of features most useful for class separation.

Because the procedure runs efficiently, I can afford a relatively fine grid for robust comparison. A typical choice might be:

set.seed(1234)
tune_result <- tune.splsda(
  X_train, y_train,
  ncomp = 3,
  test.keepX = c(25, 50, 75, 100, 125, 150, 200, 250, 500),
  validation = "Mfold",
  folds = 5,
  dist = "max.dist",
  progressBar = FALSE
)

sPLS-DA model fit

# Tuning keepX for sPLS-DA
set.seed(1234)
tune_result <- tune.splsda(
  X_train, y_train,
  ncomp = 3,
  test.keepX = c(25, 50, 75, 100, 125, 150, 200, 250, 500),
  validation = "Mfold",
  folds = 5,
  dist = "max.dist",
  progressBar = FALSE
)

# Final fit with best keepX
best_keepX <- tune_result$choice.keepX
splsda_fit <- splsda(X_train, y_train, ncomp = 3, keepX = best_keepX)

# Predict on training set
pred_train <- predict(splsda_fit, X_train, comp = 1:3)

# Predict on test set
pred_test <- predict(splsda_fit, X_test, comp = 1:3)

Model evaluation with yardstick

# --- 1. PREDICTIONS TEST SET

# Tibble for accuracy calculation
df_test_proj             <- as.data.frame(pred_test$variates)
df_test_proj$.truth      <- factor(y_test)
df_test_proj$.pred_class <- factor(pred_test$class$max.dist[, 3], levels = levels(df_test_proj$.truth))

# Accuracy calculated on the test set
acc_test <- accuracy(df_test_proj, truth = .truth, estimate = .pred_class)$.estimate

# --- 2. PREDICTIONS TRAINING SET

# Tibble for accuarcy calculation
df_train_proj <- as.data.frame(pred_train$variates)
df_train_proj$.truth <- factor(y_train)
df_train_proj$.pred_class <- factor(pred_train$class$max.dist[, 3], levels = levels(df_train_proj$.truth))

# Accuracy calculated on the training set
acc_train <- accuracy(df_train_proj, truth = .truth, estimate = .pred_class)$.estimate

# --- 3. COMPLETE METRICS TABLE

# Test set
metrics_test <- metrics(df_test_proj, truth = .truth, estimate = .pred_class)

# Training set
metrics_train <- metrics(df_train_proj, truth = .truth, estimate = .pred_class)

# --- 4. LATENT SPACE VISUALIZATION USING TRUE LABELS

p_12 <- ggplot(df_test_proj, aes(x = dim1, y = dim2, color = .truth)) +
  geom_point(size = 2, alpha = 0.85) +
  labs(title = "sPLS-DA: Test set projected by true labels",
       x = "Component 1", y = "Component 2") +
  theme_minimal(base_size = 13) +
  theme(legend.title = element_blank())

p_13 <- ggplot(df_test_proj, aes(x = dim1, y = dim3, color = .truth)) +
  geom_point(size = 2, alpha = 0.85) +
  labs(title = "sPLS-DA: Test set projected by true labels",
       x = "Component 1", y = "Component 3") +
  theme_minimal(base_size = 13) +
  theme(legend.title = element_blank())

p_23 <- ggplot(df_test_proj, aes(x = dim2, y = dim3, color = .truth)) +
  geom_point(size = 2, alpha = 0.85) +
  labs(title = "sPLS-DA: Test set projected by true labels",
       x = "Component 2", y = "Component 3") +
  theme_minimal(base_size = 13) +
  theme(legend.title = element_blank())

# --- 5. IMPORTANT WORD VISUALIZATION (the visualization will be produced later using ggplot)

# Component 1
# p_1 <- plotLoadings(splsda_fit,
#                     comp = 1,
#                     method = "mean",
#                     contrib = "max",
#                     ndisplay = 20,
#                     title = "Words most contributing to Component 1")

# Component 2
# p_2 <- plotLoadings(splsda_fit,
#                     comp = 2,
#                     method = "mean",
#                     contrib = "max",
#                     ndisplay = 20,
#                     title = "Words most contributing to Component 2")

# Component 3
# p_3 <- plotLoadings(splsda_fit,
#                     comp = 3,
#                     method = "mean",
#                     contrib = "max",
#                     ndisplay = 20,
#                     title = "Words most contributing to Component 3")

# --- 6. CONFUSION MATRIX 

pred_labels <- pred_test$class$max.dist[, 3]  
test_coords <- as.data.frame(pred_test$variates)

df_test_proj <- test_coords %>%
  mutate(
    .truth = factor(y_test),
    .pred_class = factor(pred_labels, levels = levels(factor(y_test)))
  )

confmat <- conf_mat(df_test_proj, truth = .truth, estimate = .pred_class)

As shown in Figure 2, the model achieves an accuracy of 97.6% on the test set. Since the training accuracy is 97.9%, this small difference indicates that the model generalizes well, with no signs of overfitting.

The tuning procedure selected keepX = c(200, 200, 500) as the optimal number of features to retain on each of the three components. This means that the first two components focus on relatively sparse subsets of the vocabulary (200 terms each), while the third component includes the entire trimmed vocabulary of 500 terms. This adaptive behavior reflects the role of each component in capturing residual discriminative structure: early components rely on compact feature sets, while later ones may require broader lexical support to separate harder-to-distinguish classes.

Since the method assumes that classes can be linearly separated in the latent space, it becomes relatively straightforward to assign an importance score to each feature, reflecting its contribution to the separation along each component. This enables direct interpretability of the model in terms of the most discriminative words associated with each class. The function plotLoadings() from the mixOmics package conveniently extracts and visualizes these features. As shown in Figure 3, the method assigns a class-specific contribution to each selected term, allowing us to interpret each component in terms of discriminative words. It is important to note that terms are not necessarily exclusive to a single novel — many occur in multiple chapters across books. The color in the barplots reflects the class (i.e., book) for which the term contributes most strongly to class separation along that component, not exclusivity of occurrence. Component 1 highlights features specific to Great Expectations (e.g., “pip”, “havisham’s”). Component 2 reflects opposing contributions from Pride and Prejudice and Twenty Thousand Leagues Under the Sea — with “jane”, “elizabeth”, and “darcy” linked to the former, and “captain”, “nemo”, and “nautilus” to the latter. Component 3 is almost entirely dominated by lexical signals from The War of the Worlds (e.g., “martians”, “martian”, “people”).

Several of the terms highlighted by sPLS-DA (see Figure 3) also appear among the top-ranked topic terms extracted via LDA, indicating that lexical cues effective for supervised classification often coincide with those that define unsupervised thematic structure.

Quantitatively, both models perform remarkably well on the Great Library Heist dataset. The original LDA model (trained on the full corpus) misclassified only 2 out of 193 documents, despite being entirely unsupervised (accuracy: 99%). The sPLS-DA classifier, trained on just 80% of the corpus, reached 97.6% on a held-out test set. When retrained on the full corpus (with no sparsity constraint), sPLS-DA misclassified 3 documents (accuracy: 98.4%).

This post illustrated how sparse PLS-DA — a linear, supervised method — can be used effectively for document classification. Along the way, I compared it with topic modeling via LDA, observed strong performance on real-world text data, and provided a complete, reproducible R workflow. Despite its simplicity, this approach proved to be both interpretable and surprisingly effective.

References

Ruiz-Perez D, Guan H, Madhivanan P, Mathee K, and Narasimhan G, “So you think you can PLS-DA?” BMC Bioinformatics 2020, vol 21, (suppl 1):2.

Saccenti E, Tenori L, “Stylometric investigation of Dante’s Divina Commedia by means of multivariate data analysis techniques” Internat J Comput Linguistics Res 2012, vol 3, n 2, pp 35-48.

A four-state Markov chain encoding second-order binary dependencies

Now consider a more structured chain where the state at time depends on both and . This can be seen as a natural extension of the two-state MC to capture memory over two steps. This corresponds to a second-order Markov chain on binary values . It can be encoded as a first-order Markov chain with 4 states (four-state MC), each representing a pair ():

state	pair
	(0,0)	0
	(0,1)	1
	(1,0)	0
	(1,1)	1

The transition matrix then takes the form:

Each row corresponds to a transition determined by the current pair (), and the choice of the next value determines the new state Figure 2.

Autocorrelation function

To compute the ACF, we first define a projection function , which recovers the binary value encoded by state :

Then, the expectation of is:

and the variance becomes:

To evaluate the autocorrelation at lag , we compute the joint moment:

so that the ACF is:

This method relies only on the stationary distribution and the -step transition matrix . Unlike the two-state MC discussed above, here does not admit a closed-form expression and must be computed numerically.

The following R function implements the numerical computation of the ACF for the four-state MC described above. It constructs the transition matrix, solves for the stationary distribution, and computes the projected ACF based on the binary encoding of the states.

Numerical computation of ACF

compute_acf_four_state <- function(p11, p10, p01, p00, max_lag = 10) {
    
  # Transition matrix
  P <- matrix(c(p11, 1 - p11, 0, 0,
                0, 0, p10, 1 - p10,
                p01, 1 - p01, 0, 0,
                0, 0, p00, 1 - p00), nrow = 4, byrow = TRUE)

  # Solve for stationary distribution π such that π %*% P = π
  A <- t(P) - diag(4)
  A <- rbind(A, rep(1, 4))             # Add normalization row
  b <- c(rep(0, 4), 1)
  pi <- solve(t(A) %*% A, t(A) %*% b)  # Least-squares solution

  # Projection function f(i) = 1 for states 1 and 3, else 0
  f <- c(0, 1, 0, 1)

  EX   <- sum(f * pi)
  VarX <- EX * (1 - EX)

  rho <- numeric(max_lag)
  P_power <- diag(4)

  for (r in 1:max_lag) {
    P_power <- P_power %*% P
    EXX <- 0
    for (i in 1:4) {
      for (j in 1:4) {
        EXX <- EXX + f[i] * f[j] * pi[i] * P_power[i, j]
      }
    }
    rho[r] <- (EXX - EX^2) / VarX
  }

  return(list(
    pi   = round(pi, 5),
    EX   = round(EX, 5),
    VarX = round(VarX, 5),
    rho   = round(rho, 5)
  ))
}

Figure 3 shows the resulting autocorrelation pattern, which reflects the alternating behavior encoded in the four-state structure and highlights the persistence introduced by second-order dependencies.

Estimating the standard error of a sample proportion

When working with binary time series, a fundamental question is how to estimate the standard error (SE) of the sample proportion

In the case of independent Bernoulli trials, the variance of the sample mean is:

and the corresponding i.i.d. formula for the standard error is:

However, when the data are generated from a dependent process such as a Markov chain (either order 1 or 2), this formula underestimates or overestimates the true SE, because the observations are no longer independent.

With the introduction of the , the expression of becomes:

The can be computed for both the two-state and four-state MCs, for example when estimating the proportion of time spent in each state — and its standard error — from an observed binary sequence of length .

• In the two-state MC, define the transition probabilities as follows:

With this parametrization, the lag- autocorrelation of the process is:

This expression fully determines the ACF, which decays geometrically. The corresponding becomes:

In the two-state case, the process is i.i.d. only when the transition matrix is both symmetric and memoryless — that is, when transitions occur with equal probability in both directions. In this case, the autocorrelations vanish and .

• In the four-state MC, the autocorrelations must be computed numerically (as done above), and plugged into the expression for above. As shown in Figure 3, the ACF decays fast enough to justify using the simplified formula above.

In both cases, accounting for autocorrelation is essential to avoid underestimating uncertainty in sample-based inference.

This example illustrates how two Markov models generating binary sequences with identical marginal distributions can nevertheless lead to different effective sample sizes, due to distinct autocorrelation structures that significantly impact inference.

In short, correlation matters—even when proportions look the same. These insights remind us that dependencies in data—often subtle—can have major consequences for statistical inference. Understanding and quantifying them is essential, even in the simplest of models.

What is a statistical model?

In classical statistics — for example, for methods introduced early in the training of sophomore students, such as simple linear regression and ANOVA (ANalysis Of VAriance) — the model is typically written in a very structured form. Let us suppose that is the response variable (outcome) and are explanatory variables (predictors), collected into the vector ; the model is the equation that we suppose relate together the predictors to the outcome, for any of a number of observational units, with an important element noticed in the formulation of the model-the random error term, :

where are the parameters to be estimated. This decomposition highlights two essential ideas:

• There is an underlying systematic component (),
• Plus an error component (), which captures variability not explained by the model.

The classical statistical inferential procedures—whether hypothesis testing or interval estimation—are fundamentally designed around the behavior of this error term. It is also important to realize that we can never observe the error term, but we can infer its behavior from the analysis of the model residual — the difference between observed and predicted values.

In classical parametric inference, crucial assumptions are made about the behavior of the error terms in the general model Equation 1.

To ensure analytical tractability and valid inference, it is common to summarize these assumptions using the acronym LINE — Linearity, Independence, Normality, and Equality of variance:

1. Linearity: The model is linear in the parameters ; that is, the expected value of the outcome variable can be expressed as a linear combination of the parameters, possibly through transformations of the predictors .

2. Independence: The error terms are independent across observations. In particular, for time series data, consecutive errors are assumed to be uncorrelated. In cross-sectional data, failure of independence often reflects issues in experimental design, such as inadequate random assignment or hidden confounding. Detecting such problems can be challenging and typically requires careful planning at the data collection stage.

3. Normality: The error terms are normally distributed with mean zero.

4. Equality of variance (Homoscedasticity): The error terms have constant variance across the range of the predictors.

These assumptions provide the theoretical foundation for estimating parameters, constructing confidence intervals, and performing hypothesis tests in a coherent analytical framework. Understanding how residuals behave is not only crucial in classical inference, but also serves as a bridge toward modern predictive approaches, as discussed in the note below.

This careful structure is what allows classical statistics to calibrate confidence intervals, compute p-values, and estimate statistical power, all based on theoretically derived sampling distributions. However, when these assumptions are violated, the inferential reliability may break down — often in unpredictable ways. Modern approaches like bootstrap resampling offer alternatives that reduce dependency on strong theoretical assumptions, and interestingly, align closely with resampling strategies widely used in ML (such as cross-validation for hyperparameter tuning). In a sense, bootstrapping can be seen as a bridge: a classical method moving closer to the data-driven, flexible spirit that characterizes much of modern ML practice.

Key Concepts

From Hypothesis Testing to Predictive Modeling

Let us now illustrate the modeling perspective starting with a classical example: the independent-samples t-test. First, the neded libraries are uploaded.

# Load required libraries
library(tidyverse)
library(tidymodels)
library(ggplot2)
library(broom)
library(kableExtra)

Simulating Data for Classical Inference

# Simulate two groups with slightly different means
set.seed(123)
group1 <- rnorm(64, mean = 5.0, sd = 1.0)
group2 <- rnorm(64, mean = 5.5, sd = 1.0)

# Combine data into a single data frame
df <- data.frame(
  group = factor(c(rep("A", 64), rep("B", 64))),
  outcome = c(group1, group2)
)

We simulate two groups, randomly drawn from two normally distributed populations, with means and , and standard deviations . Each group contains observations. In the spirit of a real experiment, the population parameters are unknown — the only information we have comes from the observed data. Our inferential task is to test whether a difference exists between the group means, under the standard null and alternative hypotheses:

In this example, we simulate 64 observations per group — a total of 128 — to achieve 80% statistical power for detecting a medium effect size () with a two-sided t-test at the conventional 5% significance level. This decision is not arbitrary: in a well-planned experiment, the sample size should be defined before data collection, based on acceptable error rates, either of Type I or Type II. Surprisingly, many — even published — experimental studies skip this crucial planning step, treating sample size as an afterthought rather than a design parameter.

This consideration can be visualized by examining how statistical power varies with the sample size and the effect size. The plot in Figure 1 shows the power curves for three typical values of Cohen’s effect size (, , ), representing small, medium, and large effects. The dashed line marks the conventional threshold of 80% power.

As the plot shows, detecting a medium effect with 80% reliability requires at least 64 observations per group. Informal rules of thumb such as “30 is enough” may drastically underestimate the sample size needed for meaningful results.

# Perform an independent-samples t-test
t_test_res <- t.test(outcome ~ group, data = df, var.equal = TRUE)

# Fit a linear model with group as a predictor
lm_model <- lm(outcome ~ group, data = df)

# Apply ANOVA to the linear model
anova_res <- anova(lm_model)

→ t-test
──────────────

    Two Sample t-test

data:  outcome by group
t = -2.8597, df = 126, p-value = 0.004964
alternative hypothesis: true difference in means between group A and group B is not equal to 0
95 percent confidence interval:
 -0.7629029 -0.1388665
sample estimates:
mean in group A mean in group B 
       5.038478        5.489362 

→ ANOVA
──────────────

Analysis of Variance Table

Response: outcome
           Df  Sum Sq Mean Sq F value   Pr(>F)   
group       1   6.506  6.5055  8.1781 0.004964 **
Residuals 126 100.231  0.7955                    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

→ Linear Model (summary)
──────────────

Call:
lm(formula = outcome ~ group, data = df)

Residuals:
     Min       1Q   Median       3Q      Max 
-2.29853 -0.59548 -0.05597  0.53407  2.19797 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   5.0385     0.1115   45.19  < 2e-16 ***
groupB        0.4509     0.1577    2.86  0.00496 ** 
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.8919 on 126 degrees of freedom
Multiple R-squared:  0.06095,   Adjusted R-squared:  0.0535 
F-statistic: 8.178 on 1 and 126 DF,  p-value: 0.004964

From Inference to Prediction: A First Machine Learning Example

We now shift from inference to prediction. In supervised ML, model validation is built into the workflow — often using separate training and testing sets — to evaluate how well the model generalizes.

Before diving deeper, it’s worth highlighting that although ML is often associated with “algorithms,” the core idea remains modeling. Algorithms are tools to find models — representations of patterns in the data — much as in statistics. The shift lies in the greater flexibility and adaptability expected of models built via ML.

Let’s now construct a simple classification example using simulated data. To keep the comparison with the previous example meaningful, we simulate two equally sized groups — 64 observations each — and generate two continuous predictors for classification. As with the t-test setting, the balanced design is not strictly necessary, but it remains highly advisable. Severe class imbalance can distort statistical inference and degrade the performance of classifiers, especially when probabilistic calibration is involved.

Wherever possible, experimental design should strive for balanced or near-balanced group sizes. In statistical inference, imbalance may reduce the precision of estimates or distort p-values. In ML, it can lead to biased classifiers, especially when the outcome classes are highly skewed. While techniques exist to handle imbalance — such as weighting, oversampling, or downsampling, it is good experimental practice to minimize it at the design stage whenever possible.

Simulating Data for Binary Classification

# Simulate data
set.seed(123)
n <- 100
x1 <- c(rnorm(n, 2.5, 1), rnorm(n, 3.75, 1))
x2 <- c(rnorm(n, 2.5, 1), rnorm(n, 3.75, 1))
class <- factor(c(rep(0, n), rep(1, n)), levels = c("0", "1"))

df_ml <- tibble(x1 = x1, x2 = x2, class = class)

The two classes are linearly separable, by design — each group was generated from a distinct bivariate Gaussian distribution, Figure 2. This separation facilitates learning and allows us to explore classification behavior in a clean, controlled setting.

# Split and annotate
set.seed(123)
split <- initial_split(df_ml, prop = 0.75, strata = class)
train_df <- training(split) %>% mutate(set = "Train")
test_df  <- testing(split)  %>% mutate(set = "Test")

df_all   <- bind_rows(train_df, test_df)

# Fit logistic regression on training set
model_ml <- logistic_reg() %>%
  set_engine("glm") %>%
  fit(class ~ x1 + x2, data = train_df)

Visualizing the Decision Boundary

In classical statistical testing, the choice of an operational point is often driven by conventions: a significance level of 0.05 and a power of 0.80 are typically adopted without extensive reflection. In ML, however, this decision is more explicitly tied to the ROC curve. The model provides estimated probabilities, and it is up to the practitioner — in consultation with stakeholders — to choose a classification threshold that reflects real-world priorities: minimizing false positives, false negatives, or some balanced trade-off. In the example discussed here, the boundary decision is plotted as a line over the point distributions of the two classes, as shown in Figure 3.

Evaluating Performance: ROC Curve and AUC

We now evaluate model performance using the ROC curve, introduced earlier in the ML-based key concepts callout.

Code

# Predict and evaluate
test_df <- test_df %>%
  bind_cols(predict(model_ml, new_data = test_df, type = "prob"))

roc_data <- roc_curve(test_df, truth = class, .pred_0)
auc_val  <- roc_auc(test_df, truth = class, .pred_0)

# Predict on training set (add probabilities + class prediction)
train_df <- train_df %>%
  bind_cols(predict(model_ml, new_data = ., type = "prob")) %>%
  mutate(pred_class = if_else(.pred_1 >= 0.5, "1", "0") %>% factor(levels = c("0", "1")))

# Ensure predicted class exists on test set
test_df <- test_df %>%
  mutate(pred_class = if_else(.pred_1 >= 0.5, "1", "0") %>% factor(levels = c("0", "1")))

This curve, shown in Figure 4, summarizes the trade-off between sensitivity and specificity across all possible thresholds, and provides a basis for threshold-independent comparison between classifiers.

The ROC curve provides a global view of model discrimination across all possible thresholds, but model decisions are ultimately made at specific thresholds. This is where the confusion matrix (Figure 5) and the classification metrics table come into play: they reveal how a fixed cutoff — here 0.5 — translates into concrete outcomes in terms of correct and incorrect predictions. Such threshold-dependent evaluations are essential when aligning models with real-world decision-making contexts.

To complement the confusion matrix, we report key performance metrics in Table 1. These include accuracy, sensitivity (recall for the positive class), and specificity (true negative rate), computed separately on the training and test sets.

Code

# Compute each metric separately, then combine
metrics_train <- tibble(
  set = "Train",
  accuracy = yardstick::accuracy(train_df, truth = class, estimate = pred_class) %>% pull(.estimate),
  sensitivity = yardstick::sens(train_df, truth = class, estimate = pred_class) %>% pull(.estimate),
  specificity = yardstick::spec(train_df, truth = class, estimate = pred_class) %>% pull(.estimate)
)

metrics_test <- tibble(
  set = "Test",
  accuracy = yardstick::accuracy(test_df, truth = class, estimate = pred_class) %>% pull(.estimate),
  sensitivity = yardstick::sens(test_df, truth = class, estimate = pred_class) %>% pull(.estimate),
  specificity = yardstick::spec(test_df, truth = class, estimate = pred_class) %>% pull(.estimate)
)

This summary highlights how well the model generalizes and reinforces the idea that threshold selection — while often fixed by convention in statistical testing — plays a pivotal role in applied ML.

set	accuracy	sensitivity	specificity
Train	0.76	0.787	0.733
Test	0.74	0.680	0.800

Concluding remarks

The performance table above reveals not just how well the model generalizes from training to test data, but also highlights a deeper contrast between traditional statistical inference and machine learning practice. In classical statistics, the choice of a decision threshold — say, a significance level — is often fixed by convention, serving as a gatekeeper for hypothesis testing. In ML, by contrast, the threshold ( in our case) is not sacred: it can and should be adapted depending on context, cost, and stakeholder priorities.

The ROC curve offers a global view of the model’s discriminative ability across all thresholds, while the confusion matrix and the table of metrics make concrete the implications of a specific threshold. Together, they illustrate that prediction is not just about accuracy. It’s about decisions, trade-offs, and the flexibility to adapt models to real-world needs. And that’s precisely where statistical reasoning and ML begin to meet — not in opposition, but in dialogue.

The tension between theoretical rigor and empirical adaptability will continue to shape the evolving relationship between statistics and ML. In future explorations, we will examine how models behave when ideal conditions break down, and how different strategies — from regularization to robust modeling — attempt to meet the timeless challenge of separating signal from noise.

Further Reading

If you’d like to dig deeper into the ideas behind inference, prediction, and modern data modeling, here are some recommended readings — ranging from foundational texts to more advanced perspectives:

All of Statistics — Larry Wasserman (2004)
A concise and mathematically grounded guide to statistical reasoning, ideal for those transitioning into data science.

Statistics — Freedman, Pisani, Purves (2007)
A clear and accessible introduction to the logic and language of statistics.

Applied Predictive Modeling — Kuhn & Johnson (2013)
A hands-on resource focused on practical machine learning with a statistical backbone.

Deep Learning — Goodfellow, Bengio, Courville (2016)
The reference text for deep learning, covering theory and implementation.

Computer Age Statistical Inference — Efron & Hastie (2016)
A beautifully written journey through the convergence of statistics and algorithmic thinking.

Full MNIST Classification Pipeline in R

The MNIST (Modified National Institute of Standards and Technology) dataset is a widely used benchmark for testing machine learning algorithms, particularly in the field of deep learning. It consists of 70,000 handwritten digit images, each labeled with the corresponding digit (0-9). The images are 28x28 pixels in grayscale, making them a relatively simple task for modern machine learning models, though still an effective tool for evaluating model performance.

The following R code snippet presents the entire pipeline for the MNIST classification task using Keras for deep learning. This pipeline includes:

• Data Preparation: Loading and preprocessing the MNIST dataset, including normalization and reshaping of image data to fit the model.
• Model Definition: Constructing a deep neural network (DNN) using Keras, with layers for dense processing, dropout regularization, and softmax output.
• Model Training: Compiling and fitting the model to the training data, while using a validation split for model evaluation.
• Performance Evaluation: Timing the training process and calculating the overall duration.
• Exporting Data: Saving both the training and test datasets, as well as predictions, for further analysis or use in external systems.

The implementation also ensures reproducibility by setting a fixed random seed across the entire workflow, and leverages the tidyverse for data manipulation and visualization, alongside the Keras library for deep learning.

This pipeline provides a streamlined approach to model development, training, and evaluation while integrating seamlessly with Python-based environments (such as Google Colab) for GPU-accelerated training.

Minimal setup

This section outlines the essential libraries for the project and specifies the base directory path at which Google Drive is mounted, enabling file access within both R and Colab-based Python environments.

library(tensorflow)
library(keras)
library(reticulate)

library(tidyverse)
library(yardstick)
library(readr)

# Define Google Drive base path
gdrive_base <- "~/Library/CloudStorage/GoogleDrive-<your-mail>@gmail.com/My Drive/<your-project-folder>"

Load and preprocess the MNIST dataset

The dataset is reshaped to match the input requirements of a fully connected (dense) neural network architecture. Additionally, class labels are one-hot encoded to facilitate categorical classification during model training.

mnist <- dataset_mnist()

x_train <- mnist$train$x / 255
x_test  <- mnist$test$x / 255
y_train <- mnist$train$y
y_test  <- mnist$test$y

# Reshape for dense model (flatten 28x28 images)
x_train <- array_reshape(x_train, c(nrow(x_train), 784))
x_test  <- array_reshape(x_test, c(nrow(x_test), 784))

# One-hot encode labels
y_train_cat <- to_categorical(y_train, 10)
y_test_cat  <- to_categorical(y_test, 10)

Define model

The model is defined using the Keras API as a sequential architecture composed of fully connected layers. Following its construction, the model is compiled by specifying the loss function, optimization algorithm, and evaluation metrics, thereby preparing it for training.

Dropout is used to help the model generalize better and avoid overfitting. During training, it randomly turns off a portion of the neurons in the network. This forces the model to learn more robust features instead of relying too heavily on any one part of the network.

tf$keras$optimizers$legacy$Adam() is used to maintain the original behavior of the Adam optimizer, which can be especially valuable on M1/M2 MacBook architectures where newer versions might encounter compatibility issues. Newer versions changed how optimizers work, so the legacy version ensures compatibility with older code and stable training results.

model <- keras_model_sequential(list(
  layer_dense(units = 128, activation = "relu", input_shape = 784),
  layer_dropout(rate = 0.4),
  layer_dense(units = 64, activation = "relu"),
  layer_dropout(rate = 0.3),
  layer_dense(units = 10, activation = "softmax")
))

model %>% compile(
  loss = "categorical_crossentropy",
  optimizer = "adam", #tf$keras$optimizers$legacy$Adam(),
  metrics = "accuracy"
)

Training and time the training

Setting a seed ensures reproducibility of training results, while timing the training process helps assess the efficiency of the model’s learning and optimization.

The validation split refers to the portion of the training data set aside during training to evaluate model performance on unseen data, helping to detect overfitting (and tune hyperparameters, though not considered in this post).

The number of epochs is set to 10. For the purposes of this post, I will not address whether the training process adequately accounts for the issue of overfitting and generalization.

# Set seed for reproducibility across R, TensorFlow, NumPy
set.seed(123)
tensorflow::set_random_seed(123)

# Time the training
start_time <- Sys.time()

model %>% fit(
  x_train, y_train_cat,
  epochs = 10,
  batch_size = 256,
  validation_split = 0.2,
  verbose = FALSE
)

end_time <- Sys.time()

duration <- c()
x <- end_time-start_time
duration[1] <- as.numeric(regmatches(x,regexpr("\\d*\\.?\\d+",x)))

Export training and testing sets to Drive

The training and testing sets saved to Drive will be used later in a Colab notebook for GPU-based training of the same model defined in Keras above.

# Convert data to data frames with labels
colnames(x_train) <- paste0("pixel_", seq_len(ncol(x_train)))
colnames(x_test)  <- paste0("pixel_", seq_len(ncol(x_test)))

train_df <- as_tibble(cbind(label = y_train, x_train))
test_df  <- as_tibble(cbind(label = y_test, x_test))

write_csv(train_df, file.path(gdrive_base, "data", "mnist_train.csv"))
write_csv(test_df, file.path(gdrive_base, "data", "mnist_test.csv"))

Predict in R and export

After making predictions in R, we can export the results and visualize the confusion matrix to evaluate the model’s performance. We also compute and export the accuracy, along with the training time, to track the model’s efficiency.

# Only needed if you want to compare R vs Colab predictions.

preds_r <- model %>% predict(x_test, verbose = FALSE) %>% k_argmax()
preds_df <- tibble(
  actual = y_test,
  predicted = as.array(preds_r)
)

# Convert to factors
preds_df <- preds_df %>%
  mutate(
    actual = factor(actual),
    predicted = factor(predicted, levels = levels(actual))
  )

acc <- c()
acc[1] <- accuracy(preds_df, truth = actual, estimate = predicted)$.estimate
conf_mat(preds_df, truth = actual, estimate = predicted) %>% 
    autoplot(type = "heatmap") +
    ggtitle("R-based DNN confusion matrix for the MNIST dataset")

write_csv(preds_df, file.path(gdrive_base, "data", "predictions_r.csv"))

(Later) Compare with Colab predictions

The predictions made in R can be compared with those generated in Colab to assess the consistency and performance of the model across different environments.

1. Open the notebook in Google Colab.
2. At the top menu bar, click “Runtime”
3. Select “Change runtime type”
4. In the “Hardware accelerator” dropdown menu, choose:

• GPU to enable GPU support
• TPU if you want to experiment with TPU acceleration (more advanced)
• None to use CPU only

5. Click “Save”
6. Colab will restart the kernel and launch an environment with the selected hardware.

# After Colab finishes and saves 'predictions_colab.csv':
preds_colab <- read_csv(file.path(gdrive_base, "data", "predictions_colab.csv"))

# Convert to factors
preds_colab <- preds_colab %>%
  mutate(
    actual = factor(actual),
    predicted = factor(predicted, levels = levels(actual))
  )

acc[4] <- accuracy(preds_colab, truth = actual, estimate = predicted)$.estimate
conf_mat(preds_colab, truth = actual, estimate = predicted) %>% 
    autoplot(type = "heatmap") +
    ggtitle("Python-based DNN confusion matrix for the MNIST dataset")

# Optionally print Colab training time (if saved)
duration[4] <- read_lines(file.path(gdrive_base, "data", "colab_duration.txt"))

Comparative analysis

After running the model for 10 epochs—with all other elements of the pipeline held constant—I repeated the classification task using the DNN trained for 50 and 100 epochs. The results, in terms of accuracy and training time, are presented in the table below. Although not strictly necessary — or even correct — I used three decimal places to highlight subtle differences in accuracy observed in the experiments. Moreover, I chose to round the training time to the nearest integer.

Even with matching seeds, data splits, architectures, and all the same training settings in R and Python, small differences in accuracy still appeared (compare the confusion matrices above). These are likely due to subtle differences in weight initialization, batch shuffling, and dropout masking between the two frameworks. Such variations are common in deep learning and typically don’t affect overall performance trends.

Why MacBook CPUs (M1/M2) can beat Colab GPUs for MNIST

• Model + Dataset are too small for GPU to shine: MNIST has only 60,000 grayscale 28×28 images, and the DNN models used here have a relatively limited number of parameters to learn (109386). With only 100k parameters, each training step is very fast on modern CPUs and the overhead of using a GPU — loading the data, launching kernels, moving data around — starts to outweigh the benefits of parallelism.

• Apple Silicon (M1/M2) is extremely optimized: M1/M2 chips have Unified Memory (RAM + GPU share memory), fast memory access, and dedicated matrix multiplication engines (AMX). TensorFlow and PyTorch now have native Metal backends or use Accelerate.framework under the hood on macOS, enabling them to be blazingly fast even without explicit GPU use.

• Colab GPUs aren’t top-tier GPUs: Colab’s free tier provides K80 or T4 GPUs, which are relatively old or mid-tier and often shared among users. Their performance can fluctuate depending on server load, whereas a MacBook runs everything locally without such interruptions.

• Overhead on the GPU pipeline: the GPU has to load the data, transfer it to GPU memory, perform the computation, and then transfer the results back. This overhead adds latency unless the batch size and model complexity are large enough to justify the transfer.

MNIST Classification Training in Python

In the snippet below, I present an example of a *.ipynb notebook that facilitates the sharing of input and output files with a local R project via a Google Drive synced folder.

# Mount Google Drive
from google.colab import drive
drive.mount('/content/drive')

# Imports
import pandas as pd
import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.utils import to_categorical
from sklearn.metrics import accuracy_score, confusion_matrix
import time

# Set seed
np.random.seed(123)
tf.random.set_seed(123)

# Load training and testing data from Drive
gdrive_path = "/content/drive/MyDrive/parallel_processing_colab/data"
train_df = pd.read_csv(f"{gdrive_path}/mnist_train.csv")
test_df = pd.read_csv(f"{gdrive_path}/mnist_test.csv")

x_train = train_df.drop(columns = ["label"]).values.astype("float32")
y_train = to_categorical(train_df["label"].values, num_classes = 10)

x_test = test_df.drop(columns = ["label"]).values.astype("float32")
y_test = test_df["label"].values

# Define the same model as in R
model = Sequential([
    Dense(128, activation = 'relu', input_shape = (784, )),
    Dropout(0.4),
    Dense(64, activation = 'relu'),
    Dropout(0.3),
    Dense(10, activation = 'softmax')
])

model.compile(
    loss = 'categorical_crossentropy',
    optimizer = "adam",
    metrics = ['accuracy']
)

# Time the training
start = time.time()

model.fit(
    x_train, y_train,
    epochs = 10,
    batch_size = 128,
    validation_split = 0.2,
    verbose = 2
)

end = time.time()
duration = end - start
print("Training time (seconds):", duration)

# Save Colab training time to Google Drive
with open('/content/drive/MyDrive/parallel_processing_colab/data/colab_duration.txt', 'w') as f:
    f.write(str(duration))
    
# Predict on test set and export predictions
preds = model.predict(x_test)
pred_labels = np.argmax(preds, axis = 1)

# Save predictions to Drive
preds_df = pd.DataFrame({
    "actual": y_test,
    "predicted": pred_labels
})
preds_df.to_csv(f"{gdrive_path}/predictions_colab.csv", index = False)

# Evaluate metrics (optional here)
print("Accuracy:", accuracy_score(y_test, pred_labels))
print("Confusion matrix:\n", confusion_matrix(y_test, pred_labels))

Summary

When working with deep learning tasks, timing considerations between R and Python (especially in Colab) can significantly impact performance.

1. Hardware acceleration

• R: limited GPU support unless utilizing packages like tensorflow, often relying on the CPU for training, leading to longer processing times for complex models.
• Python (Colab): seamless GPU support accelerates training dramatically, reducing training times from hours to minutes.

2. Execution times

• R: training on a CPU can be slow for deep learning tasks. Execution time can be measured using system.time().
• Python (Colab): with GPU enabled, training times are drastically reduced. Timing can be monitored using Python’s time module.

3. Colab overheads

• Initialization time: setting up Colab (mounting Drive, loading data, GPU setup) adds some overhead, especially when starting a new session.
• Session limitations: free GPU access has time limits and potential disconnects, requiring reinitialization.
• File access delays: reading and writing to Google Drive in Colab can slow down the workflow compared to local file systems.
• Cold start for GPU: initializing GPU resources adds delay when starting a session.

4. Hybrid workflow: a hybrid R-Python workflow maximizes the strengths of both ecosystems. R handles data preprocessing, feature engineering, evaluation, and visualization, while Python in Colab handles heavy-duty model training with GPU support. This combination offers a flexible and efficient solution, especially for large models and datasets.

5. Total execution time: while Colab’s GPU accelerates training, overheads like initialization and session limitations can add to the overall time. For simpler models or smaller datasets, running everything locally may be more efficient, but for larger models, the speedup from Colab’s GPU outweighs the overheads.

What I’ve shown here is how to leverage Colab’s GPU support for deep learning while staying within the comfort of R, taking full advantage of the tidyverse ecosystem for analysis and visualization. In a future post, I’ll circle back to this topic—probably with a hands-on example using LSTM for time series forecasting, where Colab’s GPU support will really get a chance to shine compared to single-CPU runs.

In summary, the hybrid R + Python approach offers a balance of computational power (via Colab’s GPUs) and flexibility (using R for preprocessing and evaluation), though the Colab overheads should be considered when designing the workflow.

Parallel icons created by juicy_fish - Flaticon

Overview

R supports three popular parallel processing methods that work well on M1/M2 MacBooks. Programming hints on how to set up/activate and deactivate/reset each of the three methods are provided in the table below.

The table below summarizes the key pros and cons of each method.

# Data handling and wrangling
library(tidyverse)

# For the Ames Housing dataset
library(modeldata)

# Modeling and resampling
library(tidymodels)
tidymodels_prefer()

data(ames, package = "modeldata")
ames <- ames %>% mutate(Sale_Price = log10(Sale_Price))

set.seed(502)
ames_split <- initial_split(ames, prop = 0.80, strata = Sale_Price) 
ames_train <- training(ames_split)
ames_test  <- testing(ames_split)

set.seed(1004)
ames_folds <- vfold_cv(ames_train, v = 10, strata = Sale_Price) 

ames_rec <- 
  recipe(Sale_Price ~ Neighborhood + Gr_Liv_Area + Year_Built + Bldg_Type,
         data = ames_train) %>%
  step_log(Gr_Liv_Area, base = 10) %>% 
  step_dummy(all_nominal_predictors())

rf_model <- rand_forest(trees = 1000) %>% 
    set_engine("ranger") %>% 
    set_mode("regression")

rf_wflow <- workflow() %>% 
    add_recipe(ames_rec) %>% 
    add_model(rf_model)

keep_pred <- control_resamples(save_pred = TRUE, save_workflow = TRUE)

set.seed(1003) 

start_time <- Sys.time() 

rf_res <- rf_wflow %>% 
    fit_resamples(resamples = ames_folds, control = keep_pred) 

end_time <- Sys.time() 

metrics <- collect_metrics(rf_res) 
performance <- list()
performance[[1]] <- metrics %>%
    filter(.metric %in% c("rmse", "rsq")) %>%
               pull(mean)

duration <- c()
x <- end_time-start_time
duration[1] <- as.numeric(regmatches(x,regexpr("\\d*\\.?\\d+",x)))

library(doParallel) 

set.seed(1003) 

cl <- makeCluster(4) 
registerDoParallel(cl) 

start_time <- Sys.time() 
rf_res <- rf_wflow %>% 
    fit_resamples(resamples = ames_folds, control = keep_pred) 
end_time <- Sys.time()

stopCluster(cl)

metrics <- collect_metrics(rf_res)  
performance[[2]] <- metrics %>%
    filter(.metric %in% c("rmse", "rsq")) %>%
               pull(mean)

x <- end_time - start_time
duration[2] <- as.numeric(regmatches(x,regexpr("\\d*\\.?\\d+",x)))

set.seed(1003) 

library(doMC) 
registerDoMC(cores = 4) 

start_time <- Sys.time() 

rf_res <- rf_wflow %>% 
    fit_resamples(resamples = ames_folds, control = keep_pred) 

end_time <- Sys.time()

registerDoSEQ()

metrics <- collect_metrics(rf_res)  
performance[[3]] <- metrics %>%
    filter(.metric %in% c("rmse", "rsq")) %>%
               pull(mean)

x <- end_time - start_time
duration[3] <- as.numeric(regmatches(x,regexpr("\\d*\\.?\\d+",x)))

library(furrr) 
library(future) 

set.seed(1003) 

plan(multisession, workers = 4) 

start_time <- Sys.time() 

rf_res <- rf_wflow %>% 
    fit_resamples(resamples = ames_folds, control = keep_pred) 

end_time <- Sys.time()

plan(sequential)

metrics <- collect_metrics(rf_res)  
performance[[4]] <- metrics %>%
    filter(.metric %in% c("rmse", "rsq")) %>%
               pull(mean)

x <- end_time - start_time
duration[4] <- as.numeric(regmatches(x,regexpr("\\d*\\.?\\d+",x)))

Recommended core use on M1/M2 Macs

Use 4 cores to target the performance cores only. This choice offers a good balance between performance and system responsiveness.

Summary

• Use doMC for lightweight, fast, Mac-only workflows.
• Use doParallel if you need Windows compatibility or more control.
• Use furrr + future for a modern, tidyverse-friendly setup and future scalability.

Parallel icons created by juicy_fish - Flaticon

Load the libraries and the dataset

For demonstration purposes, the historical end-of-day stock price data for Apple Inc. are considered (period: 01/01/2022-06/30/2024). The quantmod package in R can be used to download this data from Yahoo Finance.

library(tidyverse)
library(quantmod)
library(keras)

tensorflow::set_random_seed(1234)

# Define the stock symbol for Apple Inc. and specifiy the date range
stock_symbol <- "AAPL"
start_date   <- as.Date("2021-12-31")
end_date     <- as.Date("2024-06-30")
# Download data from Yahoo Finance using quantmod
getSymbols(stock_symbol, src = "yahoo", 
           from = start_date, to = end_date) -> a
# Set data as dataframe
apple_data <- data.frame(date = as.Date(index(AAPL)), 
                         close = as.numeric(Cl(AAPL)))
# Compute log-returns from end-of-day stock price, adjusting for date
log_return <- diff(log(apple_data$close))
date <- apple_data %>%
    pull(date)
date <- date[-1]
apple_data <- data.frame(date = date, log_return = log_return)

Data preparation

Transform data to stationary

Usually, this is done by getting the difference between two consecutive values in the series. This transformation, commonly known as differencing, is considered in traditional time series models, such as ARIMA and GARCH. For a short discussion about non-stationarity in time series and statistical tests that can be used to detect whether a time series is stationary or not, see my previous post (here). There are different approaches in the literature regarding whether LSTM networks can be applied to stationary or non-stationary time series (here). For the example discussed in this post, I decided to analyze the log-returns of the end-of-day stock prices:

The time series of log-returns plotted in Figure 4 is to be regarded as being stationary.

get_scaling_factors <- function(data) {
  out <- c(mean = mean(data), sd = sd(data))
  return(out)
}

reverse_data <- function(data, scaling_factors) {
  temp <- (data*scaling_factors[2]) + scaling_factors[1]
  out <- as.matrix(temp)
  return(out)
}

# Data processing using differencing with diff() and normalization with scale() 
scaling_factors <- get_scaling_factors(apple_data$log_return)
log_return      <- scale(apple_data$log_return)
log_return      <- data.frame(log_return)

# Define the train and test split
split_data <- function(series, lag) {
    lag <- lag+1
    data_raw <- as.matrix(series) # convert to matrix (series is a data frame)
    data <- array(dim = c(0, lag, ncol(data_raw)))
    # Create all possible sequences of length lag
    for (i in 1:(nrow(data_raw)-lag)) {
        data <- rbind(data, data_raw[i:(i+lag-1), ])
    }
    train_set_size <- round(0.8*nrow(data))
    test_set_size  <- nrow(data)-train_set_size
    x_train <- data[1:train_set_size, 1:(lag-1)]
    y_train <- data[1:train_set_size, lag]
    x_test <- data[(train_set_size+1):nrow(data), 1:(lag-1)]
    y_test <- data[(train_set_size+1):nrow(data), lag]
    return(list(x_train = x_train, y_train = y_train,
                x_test = x_test, y_test = y_test))
}

# Divide data into train and test
lag <- 7 # Choose sequence length
split_data <- split_data(log_return, lag)
x_train <- split_data$x_train
y_train <- split_data$y_train
x_test <- split_data$x_test
y_test <- split_data$y_test
n_train <- dim(x_train)[1]
n_test <- dim(x_test)[1]

Model construction

It is important to consider that the input to every LSTM layer must be three-dimensional. The three dimensions of the input are:

• Samples This dimension indicates how many samples are included in each batch; in this example, we have 494 samples for the training data and 123 for the testing data.

• Time Steps This dimension indicates how many points of observation are included in each sample; in this example, we have 7 points.

• Features This dimension indicates how many features are in each observation at each time step; for a univariate case, like in this example, we have just one feature.

input_dim <- 1

# Reshape the training and test data to have a 3D array
x_train <- array_reshape(x_train, c(n_train, lag, input_dim))
x_test  <- array_reshape(x_test, c(n_test, lag, input_dim))

# Decide structure
hidden_dim <- 32
num_layers <- 2
output_dim <- 1

# Define the LSTM model using Keras
model <- keras_model_sequential() %>%
  layer_lstm(units = hidden_dim, input_shape = c(lag, input_dim), 
             return_sequences = TRUE) %>%
  layer_lstm(units = hidden_dim) %>%
  layer_dense(units = output_dim) 
model

Model: "sequential"
________________________________________________________________________________
 Layer (type)                       Output Shape                    Param #     
================================================================================
 lstm_1 (LSTM)                      (None, 7, 32)                   4352        
 lstm (LSTM)                        (None, 32)                      8320        
 dense (Dense)                      (None, 1)                       33          
================================================================================
Total params: 12705 (49.63 KB)
Trainable params: 12705 (49.63 KB)
Non-trainable params: 0 (0.00 Byte)
________________________________________________________________________________

Compile and fit the model

The model is compiled in-place using compile(), which specifies the loss function (mean squared error) and the optimizer (Adam), with learning rate assigned over each update. The model is then fit using the training data and validated on the testing data. The training history is stored in the history object.

Code

num_epochs <- 125
batch      <- 16

model %>%
  compile(loss = "mean_squared_error", 
          optimizer = optimizer_adam(learning_rate = 0.01))

# Train the model on the training data
history <- model %>% fit(x_train, y_train, 
                         epochs = num_epochs, 
                         batch_size = batch, 
                         validation_data = list(x_test, y_test),
                         verbose = FALSE,
                         shuffle = FALSE)

The loss over the training epochs is shown in Figure 5. The validation mean squared error is 1.2078.

Extract predictions

Figure 6 shows the results of the forecasting exercise using LSTM.

Code

# Extract predictions from the estimated model
y_train_pred <- model %>% predict(x_train, verbose = FALSE)
y_test_pred  <- model %>% predict(x_test, verbose = FALSE)

# Rescale the predictions and original values
y_train_pred_orig <- reverse_data(y_train_pred, scaling_factors)
y_train_orig      <- reverse_data(y_train, scaling_factors)
y_test_pred_orig  <- reverse_data(y_test_pred, scaling_factors)
y_test_orig       <- reverse_data(y_test, scaling_factors)

# Shift the predicted values to start from where the training data predictions end
y_test_pred_orig_shifted <- c(rep(NA, n_train), y_test_pred_orig[, 1])
y_train_pred_orig        <- c(y_train_pred_orig, rep(NA, n_test))

Forecasting icons created by zero_wing - Flaticon

Fourier analysis

The Fourier series of is defined as a trigonometric series of the form:

where the Fourier series coefficients are defined by the integrals:

Since is odd-symmetric, is null (i.e., the sine-wave tone burst has no DC component) and . The harmonic coefficients are given by:

where () and . To make the notation lighter, in the following. The spectrum of the sine-wave tone burst requires the calculation of the Fourier integrals:

Applying the Werner formulas, the integrand can be written:

Therefore:

and

Using the duty-ratio factor, we have:

When (i.e., ), we get for all , but . Recall that the th harmonic component in the spectrum of corresponds to the fundamental frequency of the sine-wave, - as noted above, if , then . For any value of different from 1, the maximum value of , still occurs when , however other lines are present in the spectrum. These lines are at the (analog) frequencies .

Examples

A sine-wave tone burst with duty-ratio factor () was simulated in MATLAB. The sampling frequency chosen for the simulation was . The Fast Fourier Transform (FFT) algorithm was applied for spectrum computation. The guidelines discussed in one previous post of mine (here) were followed to prevent spectral leakage and amplitude ambiguity. Superimposed on the values computed by the FFT (solid black points) is the envelope obtained by interpolating the discrete spectrum (see above), Figure 2.

The MATLAB code was run again in a scenario where the frequency of the sine wave moved up to , with a corresponding adjustment of the sampling frequency (). The results of the FFT-based spectrum analysis are reported in Figure 3.

The FFT-based spectrum can be also interpreted as the outcome of the following cascaded frequency-domain signal-processing operations:

• frequency sampling the spectrum - this is explained by the time periodization of the rectangular function with period .

• frequency shifting the frequency-sampled spectrum - this is explained by the amplitude modulation impressed on the periodized rectangular function when the latter is multiplied by the sine wave at frequency .

Check individual words

Let us suppose that the text to be spell checked is composed of a char array, with each element of the array being a single word expressed in the specified language (Italian, here). A custom dictionary can be set in the dict parameter when functions hunspell_* are invoked.

library(hunspell)

# check individual words
words   <- c("amore", "ammore", "prof", "professsore")
correct <- hunspell_check(words, dict = dictionary("it_IT"))
print(correct)

[1]  TRUE FALSE  TRUE FALSE

# incorrect words
incorrect.words <- words[!correct]
# find suggestions for incorrect words
suggested.words <- hunspell_suggest(incorrect.words, dict = dictionary("it_IT"))
suggested.words <- unlist(lapply(suggested.words, function(x) x[1]))

Note that the outcome of the function hunspell_suggest() is a list. An array of corrected words can then be constructed for the corresponding misspelled words. Clearly, some words are not the best choice, and the first word is usually the best alternative. The quality of data can be improved by careful exploration of the list.

An example

Let us suppose that we want to analyze the lyrics of the song titled “Il mio canto libero” (here). One single occurrence of the words “emozioni”, “nuda” and “pianto” was intentionally misspelled, namely, “emozzioni”, “nudda”, “painto” were transcripted, respectively.

To get the list of words used in the lyrics, I first split the text into individual words. To do so, I use unnest_tokens() from the tidytext package.

library(tidyverse)
library(stringi)
library(tidytext)

# data upload
lyrics <- stri_read_lines("Il mio canto libero.txt")
df_pre <- data.frame(text = lyrics) %>%
  mutate(id = row_number()) %>%
  select(id, text)
# tokenization
tokenized_text_pre <- df_pre %>%
  unnest_tokens(output = word, token = "words", input = text) %>%
  count(word, sort = TRUE)
# check spelling
correct <- hunspell_check(tokenized_text_pre$word, dict = dictionary("it_IT"))
# incorrect words
incorrect.words <- tokenized_text_pre$word[!correct]
# find suggestions for incorrect words
suggested.words <- hunspell_suggest(incorrect.words, dict = dictionary("it_IT"))
suggested.words <- unlist(lapply(suggested.words, function(x) x[1]))
# list of incorrect and suggested words
word.list <- as.data.frame(cbind(word = incorrect.words, `best alternative` = suggested.words))

The misspelled words in the song lyrics can be replaced with the suggested alternative using the code below.

incorrect.whole.words <- paste0("\\b", word.list$word, "\\b")
lyrics <- stri_replace_all_regex(lyrics, incorrect.whole.words, suggested.words, 
                                 vectorize_all = FALSE)

The count of the words that were misspelled is correctly updated after spell checking and replacement.

The thumbnail image is credited to Text mining icons created by Freepik - Flaticon

Exemplary dataset

housetasks, available in the package factoextra, is a data frame that contains the frequency of execution of 13 house tasks performed by the couple in four different ways: a) the wife only; b) alternatively; c) the husband only; d) jointly.

data(housetasks)
housetasks %>%
  kable(
    align = "rrrr",
    col.names = c("Wife", "Alternating", "Husband", "Jointly"),
    caption = "House tasks contingency table",
  ) %>%
  kable_classic()

# Convert the data frame to a table
dt <- as.table(as.matrix(housetasks))
# Graph
balloonplot(t(dt), main ="House tasks contingency table", xlab = "", ylab = "",
            label = FALSE, show.margins = FALSE)

Implementation

The function CA() from the package FactoMineR is used. It performs the Singular Value Decomposition of the standardized residuals, see my previous post here. The first step of the analysis was done there in order to evaluate whether there was a significant dependency between the rows and columns using the chi-square statistic.

res.ca <- CA(housetasks, graph = FALSE)
print(res.ca)

**Results of the Correspondence Analysis (CA)**
The row variable has  13  categories; the column variable has 4 categories
The chi square of independence between the two variables is equal to 1944.456 (p-value =  0 ).
*The results are available in the following objects:

   name              description                   
1  "$eig"            "eigenvalues"                 
2  "$col"            "results for the columns"     
3  "$col$coord"      "coord. for the columns"      
4  "$col$cos2"       "cos2 for the columns"        
5  "$col$contrib"    "contributions of the columns"
6  "$row"            "results for the rows"        
7  "$row$coord"      "coord. for the rows"         
8  "$row$cos2"       "cos2 for the rows"           
9  "$row$contrib"    "contributions of the rows"   
10 "$call"           "summary called parameters"   
11 "$call$marge.col" "weights of the columns"      
12 "$call$marge.row" "weights of the rows"         

Visualization and interpretation

The following functions available the factoextra package can be used to help in the SCA interpretation and visualization: