Exploratory data analysis (EDA)
Detective Alex Yu
[email protected]
What isn't EDA
EDA does not mean lack of planning or messy planning.
“I don't know what I am doing; just ask as many questions as possible in the survey; I don't need a wellconceptualized research question or a well-planned research design. Just explore.”
EDA is not opposed to confirmatory data factor (CDA) e.g. check assumptions, residual analysis, model diagnosis.
What is EDA?
Pattern-seeking Skepticism (detective spirit) Abductive reasoning John Tukey (not Turkey): Explore the data in as many ways as possible until a plausible story of the data emerges.
Elements of EDA
Velleman & Hoaglin (1981):
Residual analysis
Re-expression (data transformation)
Resistant
Display (revelation, data visualization)
Residual
Data = fit + residual
Data = model + error
Residual is a modern concept. In the past many scientists ignored it. They reported the “fit” only
Johannanes Kepler
Gregor Mendel
Random residual plot
No systematic pattern
Normal distribution
Strange residual patterns
Fitness data
Residuals are not normally distributed.
Explore another model!
Strange residual patterns
Non-random, systematic
Check the data!
Robust residual
Robust regression in SAS The residual plot tags the influential points (less severe) and outliers (more severe).
Re-expression or transformation
Parametric tests require certain assumptions e.g. normality, homogeneity of variances, linearity...etc. When your data structure cannot meet the requirements, you need a transformer (ask Autobots, not Deceptions)!
Transformers!
Normalize the distribution: log transformation or inverse probability
Stabilize the variance: square root transformation: y* = sqrt(y) Linearize the trend = log transformation (but sometime it is better to leave it alone and do a nonlinear fit, will be discussed next)
Skewed distribution
The distributions of publication of scientific studies and patents are skewed. A few countries (e.g. US, Japan) have the most. Log transformation can normalize them.
JMP
Create the transformed variable while doing analysis. Faster, but will not store the new variable. You cannot preview the distribution.
JMP
Create a permanent new variable for reanalysis later.
Before and after
Regression with transformed variables makes much more sense!
Example from JMP
Corn.jmp
DV: yield
IV: nitrate
Skewed distributions
Both DV and IV distributions are skewed. What regression result would you expect?
Remove outliers? •
•
Three observations are located outside the boundary of the 99% density ellipse (the majority of the data) Only one is considered an outlier.
Remove outliers?
Removing the two observations at the lower left will not make things better. They fall along the nonlinear path.
Transform yield only
Remove the outlier at the far right.
It didn't look any better. 9.5
Log[ yield]
9 8.5 8 7.5 7 0
10
20
30
40
50
nitrate
60
70
80
90
Transform nitrate only
The regression model looks linear.
It is acceptable, but the underlying pattern is really nonlinear.
Interactive nonlinear fit
Linear model is too simplistic and underfit
Overfit and complicated model
Smooth things out: Almost right • Lambda: Smoothing parameter • Not a bad model, but the data points at the lower left are neglected.
General Ambrose says:
Polynominal (nonlinear) fit
Quadratic = 2 turns
Cubic = 3 turns
Quartic = 4 turns
Quintic = 5 turns, take the lower left into account, but too complicated (too many turns)
Fit spline
Like Graph Builder, in Fit Spline you can control the curve interactively. It shows you the Rsquare (variance explained), too. It still does not take the lower left data into account.
Kernel Smoother
Local smoother: take localized variations and patterns into account. Interactive, too But the line still does not go towards the data points at the lower left.
Fit nonlinear
MM has the lowest AICc and it takes the data points at the lower left into account. Should we take it? MM is a specific model of enzyme kinetics in biochemistry.
Custom formula for data transformation
Custom transformation
You need prior research to support it. You cannot makeup a transformation or an equation. It is a linear model, it might distort the real pattern (nonlinear).
Fit special
It works! Now the line passes through all data points! Yeah!
I am the best transformer!
Resistance
Resistance is not the same as robustness. Resistance: Immune to outliers Robustness: immune to parametric assumption violations Use median, trimean, winsorized mean, trimmed mean to countermeasure outliers, but it is less important today (will be explained next).
Data visualization: Revelation
Data visualization is the primary tool of EDA. Without “seeing” the data pattern,...
how can you know whether the residuals are random or not.
how can you spot the skewed distribution, nonlinear relationship, and decide whether transformation is needed?
how can you detect outliers and decide whether you need resistance or robust procedures?
DV will be explained in detail in the next unit.
Data visualization
One of the great inventions of graphical techniques by John Tukey is the boxplot.
It is resistant against extreme cases (use the median)
It can easily spot outliers.
It can check distributional assumption using a quick 5-point summary.
Classical EDA
Some classical EDA techniques are less important because today many new procedures...
do not require parametric assumptions or are robust against the violations (e.g. decision tree, generalized regression).
Are immune against outliers (e.g. decision tree, two-step clustering).
Can handle strange data structure or perform transformation during the process (e.g. artificial neural networks).
EDA and data mining
Same:
Data mining is an extension of EDA: it inherits the exploratory spirit; don't start with a preconceived hypothesis.
Both heavily rely on data visualization.
Difference:
DM: Machine learning and resampling
DM: More robust
DM: can get the conclusion with CDA
Assignment 6.1
Download the World Bank data set from the Unit 6 folder.
Use 2005 patents by residents to predict 2007 GNP per person employed. Make a regression model using log transformation and another one using log10 transformation. Which one is better? Copy and paste the graphs into a Word document, and explain your answer.
Assignment 6.2
Open the sample data set “US demographics” from JMP. Use college degrees to predict alcohol consumption.
Use Fit Y by X or Fit nonlinear to find the relationship between the two variables. You can try different transformation methods, too. What is the underlying relationship between college degrees and alcohol consumption? Copy an paste the graphs into the same document. Explain you answer and upload the file to Sakai.
Assignment 6.3
Transform yourself into a Pink Volkswagen or a GMC truck.
