From Pixels to Physics: A Deep Dive into Signal Processing for 2D Video-Based Biomechanics

The evolution of AI pose estimation models has opened the door for a new level of deep and insightful human motion analysis. The raw output of pose estimation models, a time-series of (x, y, z) coordinates for anatomical landmarks, is only the first step. The true engineering challenge lies in transforming this inherently noisy, discrete data stream into a clean, continuous signal from which reliable kinematic derivatives (velocity, acceleration, jerk, etc.) can be calculated.

For any platform claiming to deliver accurate biomechanical analysis, a robust and principled signal processing pipeline is not a feature; it is the fundamental prerequisite for data integrity. This article explores the common pitfalls and the advanced methodologies required to bridge the gap from raw pixels to verifiable physics.

1. The Nature of the Input Signal: Discrete, Jittery, and Imperfect

A pose estimation model's output is not a perfect representation of an athlete's motion. It is a series of discrete, high-frequency measurements, each subject to sources of error:

• Model Jitter: Frame-to-frame variations in the model's precise localization of a joint, even in a static pose.
• Sensor & Environmental Noise: Sub-pixel flicker from lighting, minute camera vibrations, and lens distortion.
• Gross Outliers: Momentary model failures, such as when a limb is occluded, causing a landmark to briefly "jump" to a completely erroneous location.
• Compression Artifacts: Lossy video compression can subtly alter pixel data, causing the model's perception of a landmark to shift.

This results in a time-series signal that is a composite of the true biomechanical motion, high-frequency noise, and sporadic, large-magnitude outliers.

2. The Fallacy of Naive Differentiation

The most direct method to compute velocity is to take the first derivative of the position data (v = dP/dt). However, applying this to a raw, noisy signal is a critical error. The process of differentiation is mathematically equivalent to a high-pass filter; it inherently amplifies high-frequency components.

Consider a signal P(t) = P_true(t) + Noise(t). The derivative is v(t) = dP_true/dt + d(Noise)/dt. Since the noise is characterized by rapid, high-frequency oscillations, its derivative d(Noise)/dt will be a series of large-magnitude spikes, overwhelming the true velocity signal. This results in physically implausible peak velocity and acceleration values, rendering the analysis useless.

3. Compounded Error: The Peril of Long-Chain Derivative Calculations

The problem of noise amplification does not stop at the first derivative. It snowballs. Each successive derivative calculation re-amplifies the noise from the previous step, leading to an exponential degradation of signal quality.

• Position contains the original noise.
• Velocity (1st derivative) contains amplified noise.
• Acceleration (2nd derivative) contains noise that has been amplified twice.
• Jerk (3rd derivative), a critical measure of movement smoothness, contains noise amplified three times, often rendering it completely unusable if calculated from an unfiltered source.

This issue is even more acute in angular kinematics. Calculating a metric like shoulder external rotation velocity is a long-chain process. It requires first calculating the angle itself from three separate, noisy position vectors (e.g., hip, shoulder, elbow). The resulting angle time-series is therefore a composite of the noise from all three source signals. Taking the derivative of this already-compromised angle signal adds another layer of noise amplification. The final output can be dominated by artifacts rather than the athlete's actual motion, a classic "garbage in, garbage out" scenario.

This mathematical reality dictates an architectural necessity: the signal must be filtered to remove high-frequency noise at its source before any differentiation is performed.

4. Pre-processing: Handling Outliers and Gaps with Statistical Algorithms

Before we can even consider smoothing the signal's shape, we must first address the most egregious errors: gross outliers and data gaps. A sophisticated smoothing filter is designed to handle noise, but its least-squares fitting process can be heavily skewed by a single, wildly inaccurate data point. Therefore, a statistical pre-processing step is essential for data conditioning.

• Median Filtering for Outlier Rejection: The median filter is a non-linear digital filter that is exceptionally robust against "shot noise" or impulse outliers. It operates by sliding a window over the data and replacing the center point with the median of all values within the window. Its power lies in its ability to effectively ignore outliers, providing a much more stable representation. Applying a median filter as a first pass is a powerful technique to "zap" these outliers before the more nuanced smoothing takes place.
• Statistical Outlier Detection (Z-score): An alternative approach is to use a rolling window to calculate the Z-score for each data point (Z = (x - μ) / σ). Any point exceeding a defined threshold can be flagged as an outlier and subsequently replaced using interpolation.
• Handling Data Gaps (Interpolation): It is common for a landmark to be occluded for several frames, resulting in NaN values. These gaps must be filled before filtering. For biomechanical data, a cubic spline interpolation is often superior to linear methods. It fits a piecewise cubic polynomial to the known data points, ensuring that the interpolated segment is smooth and continuous in its first and second derivatives, which is crucial for subsequent kinematic calculations.

5. A Survey of Smoothing Methodologies and the Risk of Signal Attenuation

Once the data has been conditioned, we can proceed with a smoothing filter to address the remaining high-frequency jitter. The goal is to achieve selective attenuation—aggressively reducing the amplitude (attenuating) of the high-frequency noise while leaving the frequency band of the true motion signal untouched.

• Moving Average Filter: Simple to implement, but it is a poor choice for selective attenuation. It aggressively blunts the sharp peaks of athletic motion, causing significant signal attenuation of the very data we aim to measure. It also introduces phase lag, misaligning key events. It is generally unsuitable for high-performance kinematic analysis.
• Kalman Filter: A powerful recursive filter that excels in real-time tracking. However, its complexity can be overkill for post-processing analysis where the entire signal is known.
• Savitzky-Golay (SavGol) Filter: This is our preferred method. It is a finite impulse response (FIR) filter that operates by fitting a low-degree polynomial to a small, sliding window of the data using a linear least-squares method. Its key advantage is its ability to reduce noise while minimizing attenuation of the underlying signal's peaks far better than a moving average.

6. The Savitzky-Golay Filter: A Detailed Examination

The effectiveness of a SavGol filter is controlled by two key parameters: the window_length and the polyorder.

window_length: This integer defines the number of data points included in the local regression. A larger window increases the amount of smoothing but reduces the filter's responsiveness to rapid changes. It must be an odd number.

polyorder: This defines the degree of the polynomial used for the least-squares fit. The choice of order is critical for modeling the underlying physics:

• polyorder = 1 (Linear): y(t) = a₀ + a₁t
• polyorder = 2 (Quadratic): y(t) = a₀ + a₁t + a₂t² (Optimal for preserving peaks and valleys)
• polyorder = 3 (Cubic): y(t) = a₀ + a₁t + a₂t² + a₃t³ (Effective for modeling S-curve transitions)

A mathematical constraint dictates that polyorder must be less than window_length. The art of tuning involves balancing these parameters. A small window (e.g., 5 or 7) requires a lower polyorder (e.g., 2 or 3) to achieve stable smoothing.

7. Our Architectural Choice: A Multi-Stage "Condition and Smooth" Pipeline

Based on these principles, we have architected our engine around a multi-stage pipeline that explicitly rejects the flawed model of differentiating first and cleaning later.

Our process is as follows:

1. Ingest Raw Data: The (x, y, z) time-series for each relevant anatomical landmark is loaded.
2. Condition the Signal: A statistical pre-processing pass is executed. A median filter is applied to reject gross impulse outliers, and any data gaps from occlusions are filled using cubic spline interpolation. This produces a complete and statistically stable signal.
3. Contextual Smoothing: The system identifies the athletic motion being analyzed. Based on its known physical characteristics, a pre-tuned set of SavGol parameters (window_length, polyorder) is selected and applied to each coordinate vector (x(t), y(t), z(t)) of the conditioned data.
4. Iterative Differentiation: The first derivative is calculated from the smoothed position vectors to yield a clean velocity signal. The second derivative is taken from this clean velocity signal to yield a clean acceleration signal, and so on.

This architecture ensures that distinct types of data imperfection are handled by specialized tools in the correct sequence. It allows us to calculate metrics like peak hand speed with high confidence, knowing that the value represents the athlete's true performance, not a digital artifact. This rigorous application of classical signal processing theory is the bedrock upon which any claim of biomechanical accuracy must be built.