Correlation analysis plays a central role in exploratory data analysis. It helps clarify how variables move together, points to potential relationships worth modeling, and surfaces data issues that may otherwise go unnoticed. When used thoughtfully, correlation analysis provides early insight into structure and signal before more complex methods are applied.
This article focuses on two widely used correlation measures, Pearson’s correlation and Spearman’s correlation. Each captures a different type of relationship and relies on different assumptions. Understanding when and how to use each method is essential for sound statistical practice.
Understanding correlation coefficients
A correlation coefficient summarizes the strength and direction of the association between two variables in a single number that ranges from -1 to 1. The sign gives the direction. Positive values mean the variables tend to increase together, and negative values mean one tends to decrease as the other increases. The distance from zero gives the strength. A value near zero, however, does not necessarily mean the variables are unrelated. It means only that the specific pattern the coefficient looks for is weak, whether that pattern is a straight line for Pearson or a consistent ranking for Spearman. A strong but curved relationship can still produce a coefficient close to zero. Throughout, correlation measures association rather than causation, a distinction that should remain front of mind.
Figure 1 makes these ideas concrete. Drag the control to see how the sign and size of the coefficient change as the cloud of points tilts and tightens, then switch on the curved pattern to see a clear relationship that the coefficient reports as almost nothing.
Anatomy of a correlation
It also helps to separate the coefficient from its square. Squaring Pearson’s r gives r², the proportion of the variation in one variable that a straight-line fit accounts for. An r of 0.5 corresponds to r² = 0.25, or a quarter of the variation explained, so it represents considerably less than half of a perfect relationship. Read the familiar labels weak, moderate, and strong as a guide, since they are rough conventions that shift from field to field.
Pearson’s correlation
Pearson’s correlation, written r, measures how strong a linear relationship is between two continuous variables. It asks how closely the observations hug a straight line. Near 1 they rise together tightly. Near -1 one falls as the other rises, just as tightly. Near zero a straight line tells you almost nothing. What r summarizes, in short, is how well one straight line describes the pair.
Key assumptions
Several assumptions should be reasonably satisfied before relying on Pearson’s correlation:
• Continuous variables
Both variables should be measured on an interval or ratio scale, such as height, weight, or test scores.
• Linear relationship
The association should follow a roughly linear pattern. Scatterplots are a practical and reliable way to assess this visually.
• Absence of influential outliers
Pearson’s correlation is sensitive to extreme values. A small number of outliers can substantially inflate or deflate the coefficient.
• Normality matters only when you test
Computing r requires no distributional assumption. The usual significance test and confidence interval for r do assume the two variables are approximately bivariate normal, so with strong skew or small samples those p-values and intervals deserve caution.
Pearson’s correlation performs poorly when relationships are nonlinear. For example, variables related through a curve that changes direction may show a Pearson coefficient near zero, and even a curve that rises throughout will hold the Pearson coefficient below what Spearman reports for the same points. Outlier detection and visual inspection should always accompany computation of the coefficient.
Spearman’s correlation
Spearman’s correlation provides a nonparametric alternative that measures monotonic relationships rather than strictly linear ones. A monotonic relationship exists when one variable consistently increases or decreases as the other changes, even if the rate of change varies. Spearman’s correlation is typically written rs or ρ (the Greek letter rho). It operates on ranked values rather than raw measurements, making it more robust to skewed distributions and outliers.
Key assumptions
Spearman’s correlation relies on fewer and weaker assumptions than Pearson’s:
• Monotonic relationship
The variables should move in a consistent direction overall. Scatterplots remain useful for assessing this pattern.
• Ordinal or continuous data
Because the method uses ranks, it applies naturally to ordinal data and remains valid for continuous data.
• Paired observations
Each observation must represent a matched pair across the two variables being compared.
Spearman’s correlation cannot capture relationships that change direction across the range of the data. For example, U-shaped or inverted-U-shaped relationships may yield low correlations despite meaningful structure. When many observations share the same value, these ties reduce the information in the ranks and affect the coefficient. As with Pearson’s correlation, plotting the data is essential for interpretation.
Because Pearson and Spearman ask different questions, they can agree or disagree, and the size of the gap between them is itself informative. Figure 2 compares the two on the same data, one relationship at a time.
Two rulers: Pearson and Spearman
As a rough guide, similar values suggest an approximately linear relationship with little outlier influence, while a Spearman coefficient noticeably larger than Pearson points to a curved but monotonic pattern, or to outliers pulling Pearson down. Neither number on its own tells you which situation you are in, which is exactly why plotting the data matters.
Choosing between Pearson and Spearman
The choice between the two coefficients follows from the shape of the data and the question being asked. Neither method outranks the other. The table below collects the properties described in the two preceding sections so they can be compared side by side.
Two coefficients, two questions
| Property | Pearson’s r | Spearman’s ρ |
|---|---|---|
| Question it answers | How closely do the points follow a straight line? | Do the variables move in a consistent order, whatever the shape? |
| Computed from | The measured values themselves. | The ranks of the values, so only their order is used. |
| Suits data that are | Continuous, on an interval or ratio scale. | Continuous or ordinal, including ordered categories. |
| Reaches 1.0 when | The points lie exactly on a rising straight line. | The order matches exactly, even along a curve. |
| Effect of an outlier | Substantial. One extreme point can move the value a long way. | Smaller, because an extreme value becomes only the next rank. Reduced, though still present. |
| Main blind spot | Curved patterns, which it can report as close to zero. | Patterns that change direction, and heavy ties, which drain information from the ranks. |
Reading a coefficient with care
Whichever coefficient is used, a single number summarizes a great deal. The same value can arise from very different data. A lone outlier can manufacture a correlation or hide one, a narrow measurement range can flatten a real relationship, and a trend seen in the pooled data can reverse inside subgroups. Figure 3 walks through all three.
One coefficient, three pictures
Sample size deserves separate attention, because, for any given nonzero association, a larger sample drives the p-value down without making the coefficient itself any larger. A significance test asks whether an association is distinguishable from zero. Whether it is large enough to act on is a separate question. In a large sample even a very small correlation clears that bar, so a significant result can still describe a relationship too weak to be of any practical use. Reporting the coefficient itself, and a confidence interval alongside it where the analysis supports one, keeps attention on the size of the association rather than on whether it passed a threshold.
Two conditions matter once a whole correlation matrix is on screen. Four variables produce six distinct pairs, and larger matrices grow quickly, so scanning many pairs and reporting the largest raises the chance that a value stands out through luck alone. A matrix is best treated as a way to generate questions that a focused analysis can then answer. The standard test also assumes observations are independent of one another, an assumption broken by repeated measurements on the same subject or by observations clustered within families, clinics or schools. Where the data have that structure, the coefficient still describes the sample, but the accompanying p-value understates the uncertainty.
The practical response is a habit the following example puts to work. Plot the data before trusting a coefficient, and keep in mind that even a strong, well-estimated correlation is not by itself evidence that one variable causes another. The worked example computes and compares Pearson and Spearman correlations on real cardiometabolic measurements, beginning, as always, by looking at the data.
Example: Correlation analysis of cardiometabolic variables
This notebook builds correlograms (correlation heatmaps) for a small set of clinically meaningful variables.
Goals in this workflow:
- Get a quick, visual sense of which variables move together
- Compare Pearson (linear association) vs Spearman (rank / monotonic association)
- Sanity-check the data by inspecting distributions and outliers before trusting correlation values
A correlogram is an exploratory tool with no causal content. It helps prioritize hypotheses, potential confounders, and downstream modeling choices.
Data sources
The dataset is a processed version of a public cardiovascular screening dataset:
- Kaggle – Cardiovascular Disease dataset by Aidan (Kaggle user colewelkins)
All records are anonymized. The file carries derived columns for age in years and body mass index alongside the recorded systolic and diastolic blood pressure, which are the variables this notebook uses. Dataset licensing and attribution, together with the terms covering this article and its code, are set out in the Data and License note at the end of the article.
Import dependencies
import numpy as np
import pandas as pd
import plotly.graph_objects as go
from plotly.subplots import make_subplotsLoad the dataset
This section reads the (preprocessed) dataset into a pandas DataFrame and standardizes a few column names for readability in plots.
Practical notes: If the file path is relative, the CSV is expected to be located in the same directory as this notebook.
# Create pandas dataframe from the records
df = pd.read_csv("cardio_data_processed.csv")
# Rename the columns
df.rename(columns={
"age_years": "Age",
"bmi": "Body Mass Index",
"ap_lo": "Diastolic Blood Pressure",
"ap_hi": "Systolic Blood Pressure"
}, inplace=True)Choose variables and apply basic sample filtering
A correlogram is only as useful as the variables included. Here we define a small list of variables of interest that capture:
- Demographics (Age)
- Adiposity / body size (body mass index, BMI)
- Blood pressure physiology (systolic/diastolic)
We also apply a simple filter to remove implausible / extreme values (example: very large BMI), which can dominate correlations and distort color scales.
Filtering is intentionally minimal here. For a more formal analysis, consider a documented quality control (QC) step (range checks, missingness review, and unit validation).
variables_of_interest = ["Age", "Body Mass Index", "Diastolic Blood Pressure", "Systolic Blood Pressure"]
# Filter the dataframe to include only subjects with a BMI of <= 80
df = df[(df["Body Mass Index"] <= 80)]Why filter extreme BMI values?
Outliers can compress the heatmap color scale and can inflate or deflate correlation estimates. A simple upper bound is a pragmatic safeguard for exploratory plots.
If the analysis requires keeping all observations, consider robust approaches instead (Spearman correlation, winsorization, or robust regression diagnostics).
Explore distributions with histograms
Before interpreting correlations, it helps to understand the shape and spread of each variable.
Histograms help identify:
- Skewness (e.g., long right tails)
- Potential data-entry issues / outliers
- Whether a variable behaves more like continuous vs ordinal
These checks are especially important because Pearson’s correlation is sensitive to outliers and assumes roughly linear relationships.
The resulting histograms appear in Figure 4.
# Number of variables and grid layout
n_vars = len(variables_of_interest)
n_cols = 2
n_rows = int(np.ceil(n_vars / n_cols))
# Initialize subplot figure with bold subplot titles
fig = make_subplots(
rows=n_rows, cols=n_cols,
subplot_titles=[f"<b>{var}</b>" for var in variables_of_interest]
)
# Add histogram for each variable
for i, var in enumerate(variables_of_interest):
row = i // n_cols + 1
col = i % n_cols + 1
# Drop missing values for clean histograms
values = df[var].dropna()
# Add histogram trace
fig.add_trace(go.Histogram(
x=values,
nbinsx=20,
name=var,
showlegend=False,
marker=dict(line=dict(width=0.5, color='black')),
hoverlabel=dict(font=dict(color='black')),
hovertemplate=f"<span style='color:black'><b>{var}</b><br>Value: %{{x}}<br>Count: %{{y}}</span><extra></extra>"
), row=row, col=col)
# Update layout for aesthetics and bold black titles
fig.update_layout(
title_text="<b>Distribution of Selected Variables</b>",
title_x=0.5,
height=400 * n_rows,
width=900,
template="plotly_white",
font=dict(color='black'),
title_font=dict(color='black', size=20, family="Arial",),
)
# Update axes to set tick and label colors to black
fig.update_xaxes(title_font=dict(color='black', family="Arial", size=14), tickfont=dict(color='black'))
fig.update_yaxes(title_font=dict(color='black', family="Arial", size=14), tickfont=dict(color='black'))
# Save plot as HTML file
fig.write_html("histograms.html")
# Show figure
fig.show()Distribution of each selected variable
Compute correlation matrices (Pearson vs Spearman)
Two correlation definitions are computed:
- Pearson: measures linear association between two continuous variables
- Spearman: Pearson’s correlation applied to the ranked values, capturing monotonic relationships and reducing sensitivity to outliers
Interpreting Pearson vs Spearman together:
- Similar values: association is likely approximately linear (or at least monotonic without extreme outlier influence)
- Spearman >> Pearson: relationship may be monotonic but nonlinear, or Pearson may be dampened by outliers
- Pearson >> Spearman: most often an influential point inflating Pearson, the case shown in Figure 3, and occasionally a strong linear trend confined to a subset
Correlation is computed pairwise on the rows available for each variable pair (pandas default behavior). If missingness is substantial, missing-data handling should be explicit.
corr_matrix_pearson = df[variables_of_interest].corr(method='pearson')
corr_matrix_spearman = df[variables_of_interest].corr(method='spearman')
# Generate a mask for the upper triangle
mask = np.triu(np.ones_like(corr_matrix_pearson, dtype=bool))
# Mask the correlation matrices
masked_corr_pearson = corr_matrix_pearson.mask(mask)
masked_corr_spearman = corr_matrix_spearman.mask(mask)
# Prepare data for plotting
labels = variables_of_interest
z_pearson = masked_corr_pearson.values
z_spearman = masked_corr_spearman.values
annotations_pearson = np.where(masked_corr_pearson.isna(), '', masked_corr_pearson.round(2).astype(str))
annotations_spearman = np.where(masked_corr_spearman.isna(), '', masked_corr_spearman.round(2).astype(str))Plotting helpers
To keep the plotting code tidy, helper functions are defined to:
- Create a Plotly heatmap from a correlation matrix
- Overlay numeric annotations so the figure is readable without hovering
A consistent plotting function makes it easier to reuse this notebook for other variable sets (for example, adding lab values or medication indicators).
def create_heatmap(z_data, annotations, showscale=False, colorbar_title=None):
return go.Heatmap(
z=z_data,
x=labels,
y=labels,
colorscale='RdBu_r',
zmin=-1,
zmax=1,
showscale=showscale,
colorbar=dict(
title={"text": colorbar_title, "font": dict(size=16, color="black", family="Arial")},
x=1.05,
y=0.5,
len=0.7,
) if showscale else None,
text=annotations,
hoverinfo='text',
hovertext=annotations,
texttemplate="%{text}",
textfont=dict(size=16, color="black")
)
def update_layout(fig):
fig.update_layout(
width=1000,
height=450,
font=dict(size=16),
title_font_size=22,
margin=dict(t=50),
annotations=[dict(font=dict(size=18, color="black"))],
)
fig.update_yaxes(autorange='reversed', showgrid=False, zeroline=False)
fig.update_xaxes(showgrid=False, zeroline=False)
fig = make_subplots(
rows=1,
cols=2,
subplot_titles=('<b>Pearson Correlation Matrix</b>', '<b>Spearman Correlation Matrix</b>'),
shared_yaxes=True,
horizontal_spacing=0.1
)Plot correlograms
This final section renders the correlograms side-by-side.
How to read the plot:
- Values range from -1 (strong negative association) to +1 (strong positive association)
- Look for blocks of high correlation that suggest redundancy (potential multicollinearity)
The two correlograms appear in Figure 5.
fig.add_trace(create_heatmap(z_pearson, annotations_pearson), row=1, col=1)
fig.add_trace(create_heatmap(z_spearman, annotations_spearman, showscale=True, colorbar_title='Correlation Coefficient'), row=1, col=2)
update_layout(fig)
# Save the figure as an HTML file
fig.write_html("correlation_matrices.html")
# Show the figure
fig.show()Pearson and Spearman correlograms side by side
Reading Figure 5, the two blood pressure measures stand apart from every other pair, at about 0.73 by Pearson and 0.74 by Spearman. Every remaining pair sits far lower, roughly between 0.10 and 0.28. Pearson and Spearman also agree closely across the whole matrix, the pattern expected when relationships are approximately linear and no small group of extreme points is dominating the estimate, so for these variables there is little to choose between the two coefficients. That agreement holds despite the rounding flagged in Figure 4. Tied readings share an averaged rank, which drains information from the ranks Spearman works on, but here it does not pull Spearman away from Pearson.
The blood pressure pair is the one to handle carefully. Systolic and diastolic readings describe the same underlying physiology, so a high coefficient there points to redundancy rather than to a discovery. Deciding whether to enter both into a model is the kind of question a correlogram is meant to raise. The weaker coefficients deserve equally careful reading. A value near 0.1 means the straight-line association is slight in this sample, which leaves plenty of room for the variables to be related some other way, and no value in this matrix, at either end of the range, speaks to cause.
At this point, the notebook provides a quick correlation overview for the selected variables.
If this correlogram is being used to inform modeling:
- Treat highly correlated features as a signal to check multicollinearity and redundancy
- Prefer Spearman when relationships are monotonic but nonlinear, or when outliers remain
- Use domain knowledge to decide whether correlated variables represent the same physiological construct or distinct mechanisms
Key Takeaways
A coefficient measures how two variables move together. No value of it, however large, shows that one variable causes the other. A third variable may drive both.
The sign gives the direction and the distance from zero gives the strength. Squaring Pearson’s r gives the share of variation a straight-line fit accounts for, so r = 0.5 accounts for a quarter of it, well short of half.
A coefficient close to zero rules out only the pattern it looks for. A strong curved relationship can sit behind a Pearson value of almost zero, which is why plotting comes first.
Similar Pearson and Spearman values point to an approximately linear pattern. A Spearman value well above Pearson points to curvature or to outliers pulling Pearson down.
An influential point, a narrow measurement range, or subgroups mixed together can each produce the same coefficient from different data. A plot is what distinguishes them.
Labels such as weak, moderate and strong are field-dependent conventions. Statistical significance offers no firmer ground on importance, since it reports detectability, which in a large sample extends to even a very small coefficient.
Data & License
Cardiovascular Disease dataset (about 70,000 patient records) published on Kaggle by Aidan (user “colewelkins”). Source: kaggle.com/datasets/colewelkins/cardiovascular-disease.License: Open Database License (ODbL) v1.0 for the database and Database Contents License (DbCL) v1.0 for its contents; used with attribution. ODbL 1.0 · DbCL 1.0 This dataset appears to be a re-upload; the license shown is the one declared on Kaggle, and its upstream chain of rights is unverified.
© 2025 Philip Sarajlic. All rights reserved for the article’s original text and figures; the third-party data remains under the license shown above.
Code examples in this article are licensed under the Common Public Attribution License Version 1.0 (CPAL-1.0), an OSI-approved copyleft license based on the Mozilla Public License 1.1. Initial Developer: Philip Sarajlic.
Attribution required by CPAL Exhibit B: © 2025 Philip Sarajlic · “Based on code by Philip Sarajlic” · philipsarajlic.com · no graphic image. This attribution must be displayed in Larger Works.
Modifications must be released in Source Code form under CPAL-1.0. Making the code usable by anyone other than you over a network is External Deployment under the license and is treated as distribution, so the Source Code must be made available to those users.
Full text: opensource.org/license/cpal-1-0 (SPDX identifier CPAL-1.0)
