How to handle outliers so they don't skew your data

Key quotes:

• "A data is called as skewed when curve appears distorted or skewed either to the left or to the right, in a statistical distribution."
• "In a normal distribution, the graph appears symmetry meaning that there are about as many data values on the left side of the median as on the right side."
• "So, the distribution which is right skewed have a long tail that extends to the right or positive side of the x axis, same as the below plot."
• "Below is one real life example You can clearly see that it is a right skewed data with its tail in the +ve side of the distribution."
• "So in skewed data, the tail region may act as an outlier for the statistical model and we know that outliers adversely affect the model’s performance especially regression-based models."
• "A log transformation can help to fit a very skewed distribution into a Gaussian one."

Transcript: In this video I'm going to describe and discuss the method that is used by SPSS for the purposes of detecting an outlier and I'll point out in the first instance that there are two ways that SPSS goes about that based on something called the interquartile range rule and the multiplier it uses in that context I'm also going to discuss it in the context of whether it's really appropriate and if I had to choose which one I would use so the what I've done is I've created three variables one of which does not actually include an outlier but two of which might include an outlier now I'm going to use the utility and SPSS to help me identify whether there are outliers in these three variables to do that going to analyze descriptive statistics and explore and I got a variable in there already because I did this analysis so I'm just going to look at the first variable in the first instance you don't have to click on any buttons SPSS has as a default the result that will be produced in this case if you press ok so I'm going to do that and... See more →

Transcript: When you perform some specific statistical analysis and there are some criteria that should be met in order to get a really reliable result so for example in the independent sample t-test or away in one way another you should be sure that your data is normally distributed or that there are no outliers in your data or extreme outliers in this video I'm going to show you how to detect and deal with outliers so the procedure for you to detect outliers is as follows you just have to click and analyze hover your mouse over on descriptive statistics and click on Explorer now again let me just change this to display variable names so it's shorter and we want to check out Liars in these two scales competence in the argument so we select both and add it to our dependent list in this case we are not going to add a factor list a factor list in this case might be sex or gender so but we are not going to use it now additionally and we don't want a stick for this example so we are just going to display plots when I click here notice... See more →

Transcript: In this video we want to identify outliersin a set of data. If you are not sure what an outliers is, hereis what they are. An outlier is an extremely high or extremelylow value in the data set. Now in addition to just being something extremelyhigh or something low, you want to make sure that it satisfies the following criteria. If you want to find an outlier it must begreater than Q3 + 1.5(Interquartile Range) or it must be lower than Q1 - 1.5(InterquartileRange) This is making sure that it really is an extremelyhigh value or extremely low value. You can see though that you need to computea few different things like Q3 and Q1 and the Interquartile Range if we are going toproperly identify one of these outliers. So lets look at some data, and see how thisworks. In my data I have a chart of how many phonecalls were received on any given day. So I have 10 phone calls on the first day,12 phone calls on the second day, and so on and so forth. If I'm going to compute things like Q1 andQ3 and the Interquartile Range, its probably a good idea to take all of... See more →

Transcript: Now let's look at some continuous variables using histograms and plots for a basic histogram you call function hist with a variable of interest in this case interest rate you can use the arguments main and xlab for nicer labels the frequencies for the variable of interest are shown on the y-axis here you can see that all loans had an interest rate over 5% and very few loans had an interest rate higher than 20% let's have a look at the histogram of annual income we notice that we get a strange results here with seemingly just one big bar stirring the histogram in hist underscore income and using dollar sign breaks we get information on the location of the histogram breaks in order to get a clear idea on the data structure you can change the number of breaks using the breaks argument such that you get a more intuitive plot this can be done by choosing a number that seems more appropriate or use a rule of thumb such as a square root of the number of survey shion's in the data set this results in a much longer vector vector of breaks however the result still... See more →

Key quotes:

• "The variance is computed by finding the difference between every data point and the mean, squaring them, summing them up and then taking the average of those numbers."
• "Let’s look at an example that illustrates the difference between variance and standard deviation: Imagine a data set that contains centimeter values between 1 and 15, which results in a mean of 8."
• "Squaring the difference between each data point and the mean and averaging the squares renders a variance of 18.67 (squared centimeters), while the standard deviation is 4.3 centimeters."
• "In a normal distribution, approximately 34% of the data points are lying between the mean and one standard deviation above or below the mean."
• "Since a normal distribution is symmetrical, 68% of the data points fall between one standard deviation above and one standard deviation below the mean."
• "Note that a perfect normal distribution would have a skewness of zero because the mean equals the median."

Key quotes:

• "It measures the lack of symmetry in data distribution.It differentiates extreme values in one versus the other tail."
• "There are two types of Skewness: Positive and Negative Positive Skewness means when the tail on the right side of the distribution is longer or fatter."
• "If the peak of the distributed data was right of the average value, that would mean a negative skew."
• "High kurtosis in a data set is an indicator that data has heavy tails or outliers."
• "Investigate!Low kurtosis in a data set is an indicator that data has light tails or lack of outliers."
• "It means that the extreme values of the distribution are similar to that of a normal distribution characteristic."

Key quotes:

• "Different features in the data set may have values in different ranges."
• "In such a situation, applying statistical measures across this data set may not give desired result."
• "Data transformation predominantly deals with normalizing also known as scaling data , handling skewness and aggregation of attributes."
• "Min-Max normalization: It is simple way of scaling values in a column."
• "Here is the formula Converting it into R can be pretty simple as follows Let’s apply this normalization technique to year attribute of our data set."
• "let us calculate the normalized values manually as well as using scale() function."

Key quotes:

• "We will also look into the outlier detection and treatment techniques while seeing their impact on different types of machine learning models."
• "Many machine learning models, like linear & logistic regression, are easily impacted by the outliers in the training data."
• "This can become an issue if that outlier is an error of some type, or if we want our model to generalize well and not care for extreme values."
• "These extreme values need not necessarily impact the model performance or accuracy, but when they do they are called “Influential” points."
• "With multiple predictors, extreme values may be particularly high or low for one or more predictors (univariate analysis — analysis of one variable at a time) or may be “unusual” combinations of predictor values (multivariate analysis) In the following figure, all the points on the right-hand side of the orange line are leverage points."
• "It represents the number of standard deviations an observation is away from the mean: Here, we normally define outliers as points whose modulus of z-score is greater than a threshold value."