Note: This article was originally published on Oct 6th, 2015 and updated on Sept 13th, 2017
Mastering machine learning algorithms isn’t a myth at all. Most of the beginners start by learning regression. It is simple to learn and use, but does that solve our purpose? Of course not! Because you can do so much more than just Regression!
Think of machine learning algorithms as an armoury packed with axes, sword, blades, bow, dagger, etc. You have various tools, but you ought to learn to use them at the right time. As an analogy, think of ‘Regression’ as a sword capable of slicing and dicing data efficiently, but incapable of dealing with highly complex data. On the contrary, ‘Support Vector Machines’ is like a sharp knife – it works on smaller datasets, but on the complex ones, it can be much stronger and powerful in building machine learning models.
By now, I hope you’ve now mastered Random Forest, Naive Bayes Algorithm and Ensemble Modeling. If not, I’d suggest you take out a few minutes and read about them as well. In this article, I shall guide you through the basics to advanced knowledge of a crucial machine learning algorithm, support vector machines.
You can learn about Support Vector Machines in course format here (it’s free!):
If you’re a beginner looking to start your data science journey, you’ve come to the right place! Check out the below comprehensive courses, curated by industry experts, that we have created just for you:
“Support Vector Machine” (SVM) is a supervised machine learning algorithm which can be used for both classification or regression challenges. However, it is mostly used in classification problems. In the SVM algorithm, we plot each data item as a point in n-dimensional space (where n is number of features you have) with the value of each feature being the value of a particular coordinate. Then, we perform classification by finding the hyper-plane that differentiates the two classes very well (look at the below snapshot).
Support Vectors are simply the co-ordinates of individual observation. The SVM classifier is a frontier which best segregates the two classes (hyper-plane/ line).
You can look at support vector machines and a few examples of its working here.
Above, we got accustomed to the process of segregating the two classes with a hyper-plane. Now the burning question is “How can we identify the right hyper-plane?”. Don’t worry, it’s not as hard as you think!
Let’s understand:
Here, maximizing the distances between nearest data point (either class) and hyper-plane will help us to decide the right hyper-plane. This distance is called as Margin. Let’s look at the below snapshot:
Above, you can see that the margin for hyper-plane C is high as compared to both A and B. Hence, we name the right hyper-plane as C. Another lightning reason for selecting the hyper-plane with higher margin is robustness. If we select a hyper-plane having low margin then there is high chance of miss-classification.
Some of you may have selected the hyper-plane B as it has higher margin compared to A. But, here is the catch, SVM selects the hyper-plane which classifies the classes accurately prior to maximizing margin. Here, hyper-plane B has a classification error and A has classified all correctly. Therefore, the right hyper-plane is A.
In the SVM classifier, it is easy to have a linear hyper-plane between these two classes. But, another burning question which arises is, should we need to add this feature manually to have a hyper-plane. No, the SVM algorithm has a technique called the kernel trick. The SVM kernel is a function that takes low dimensional input space and transforms it to a higher dimensional space i.e. it converts not separable problem to separable problem. It is mostly useful in non-linear separation problem. Simply put, it does some extremely complex data transformations, then finds out the process to separate the data based on the labels or outputs you’ve defined.
When we look at the hyper-plane in original input space it looks like a circle:
Now, let’s look at the methods to apply SVM classifier algorithm in a data science challenge.
In Python, scikit-learn is a widely used library for implementing machine learning algorithms. SVM is also available in the scikit-learn library and we follow the same structure for using it(Import library, object creation, fitting model and prediction).
Now, let us have a look at a real-life problem statement and dataset to understand how to apply SVM for classification
Dream Housing Finance company deals in all home loans. They have a presence across all urban, semi-urban and rural areas. A customer first applies for a home loan, after that the company validates the customer’s eligibility for a loan.
Company wants to automate the loan eligibility process (real-time) based on customer details provided while filling an online application form. These details are Gender, Marital Status, Education, Number of Dependents, Income, Loan Amount, Credit History and others. To automate this process, they have given a problem to identify the customers’ segments, those are eligible for loan amount so that they can specifically target these customers. Here they have provided a partial data set.
Use the coding window below to predict the loan eligibility on the test set. Try changing the hyperparameters for the linear SVM to improve the accuracy.
The e1071 package in R is used to create Support Vector Machines with ease. It has helper functions as well as code for the Naive Bayes Classifier. The creation of a support vector machine in R and Python follow similar approaches, let’s take a look now at the following code:
#Import Library require(e1071) #Contains the SVM Train <- read.csv(file.choose()) Test <- read.csv(file.choose()) # there are various options associated with SVM training; like changing kernel, gamma and C value. # create model model <- svm(Target~Predictor1+Predictor2+Predictor3,data=Train,kernel='linear',gamma=0.2,cost=100) #Predict Output preds <- predict(model,Test) table(preds)
Tuning the parameters’ values for machine learning algorithms effectively improves model performance. Let’s look at the list of parameters available with SVM.
sklearn.svm.SVC(C=1.0, kernel='rbf', degree=3, gamma=0.0, coef0=0.0, shrinking=True, probability=False,tol=0.001, cache_size=200, class_weight=None, verbose=False, max_iter=-1, random_state=None)
I am going to discuss about some important parameters having higher impact on model performance, “kernel”, “gamma” and “C”.
kernel: We have already discussed about it. Here, we have various options available with kernel like, “linear”, “rbf”,”poly” and others (default value is “rbf”). Here “rbf” and “poly” are useful for non-linear hyper-plane. Let’s look at the example, where we’ve used linear kernel on two feature of iris data set to classify their class.
Example: Have a linear SVM kernel
import numpy as np import matplotlib.pyplot as plt from sklearn import svm, datasets
# import some data to play with iris = datasets.load_iris() X = iris.data[:, :2] # we only take the first two features. We could # avoid this ugly slicing by using a two-dim dataset y = iris.target
# we create an instance of SVM and fit out data. We do not scale our # data since we want to plot the support vectors C = 1.0 # SVM regularization parameter svc = svm.SVC(kernel='linear', C=1,gamma=0).fit(X, y)
# create a mesh to plot in x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1 y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1 h = (x_max / x_min)/100 xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
plt.subplot(1, 1, 1) Z = svc.predict(np.c_[xx.ravel(), yy.ravel()]) Z = Z.reshape(xx.shape) plt.contourf(xx, yy, Z, cmap=plt.cm.Paired, alpha=0.8)
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Paired) plt.xlabel('Sepal length') plt.ylabel('Sepal width') plt.xlim(xx.min(), xx.max()) plt.title('SVC with linear kernel') plt.show()
Example: Use SVM rbf kernel
Change the kernel type to rbf in below line and look at the impact.
svc = svm.SVC(kernel='rbf', C=1,gamma=0).fit(X, y)
I would suggest you go for linear SVM kernel if you have a large number of features (>1000) because it is more likely that the data is linearly separable in high dimensional space. Also, you can use RBF but do not forget to cross-validate for its parameters to avoid over-fitting.
gamma: Kernel coefficient for ‘rbf’, ‘poly’ and ‘sigmoid’. Higher the value of gamma, will try to exact fit the as per training data set i.e. generalization error and cause over-fitting problem.
Example: Let’s difference if we have gamma different gamma values like 0, 10 or 100.
svc = svm.SVC(kernel='rbf', C=1,gamma=0).fit(X, y)
C: Penalty parameter C of the error term. It also controls the trade-off between smooth decision boundaries and classifying the training points correctly.
We should always look at the cross-validation score to have effective combination of these parameters and avoid over-fitting.
In R, SVMs can be tuned in a similar fashion as they are in Python. Mentioned below are the respective parameters for e1071 package:
Find right additional feature to have a hyper-plane for segregating the classes in below snapshot:
Answer the variable name in the comments section below. I’ll shall then reveal the answer.
In this article, we looked at the machine learning algorithm, Support Vector Machine in detail. I discussed its concept of working, process of implementation in python, the tricks to make the model efficient by tuning its parameters, Pros and Cons, and finally a problem to solve. I would suggest you to use SVM and analyse the power of this model by tuning the parameters. I also want to hear your experience with SVM, how have you tuned parameters to avoid over-fitting and reduce the training time?
Did you find this article helpful? Please share your opinions/thoughts in the comments section below.
Lorem ipsum dolor sit amet, consectetur adipiscing elit,
10 Nov 23 • 08:00pm
hi, gr8 articles..explaining the nuances of SVM...hope u can reproduce the same with R.....it would be gr8 help to all R junkies like me
NEW VARIABLE (Z) = SQRT(X) + SQRT (Y)
Given problem Data points looks like y=x^2+c. So i guess z=x^2-y OR z=y-x^2.
i think x coodinates must increase after sqrt
Kernel
I mean kernel will add the new feature automatically.
Nicely Explained . The hyperplane to separate the classes for the above problem can be imagined as 3-D Parabola. z=ax^2 + by^2 + c
Thanks a lot for this great hands-on article!
Really impressive content. Simple and effective. It could be more efficient if you can describe each of the parameters and practical application where you faced non-trivial problem examples.
kernel
How does the python code look like if we are using LSSVM instead of SVM?
Polynomial kernel function?! for exzmple : Z= A(x^2) + B(y^2) + Cx + Dy + E
Hi Sunil. Great Article. However, there's an issue in the code you've provided. When i compiled the code, i got the following error: Name error: name 'h' is not defined. I've faced this error at line 16, which is: "xx, yy = np.meshgrid(np.arange(x_min, x_min, h), ...). Could you look into it and let me know how to fix it?
great explanation :) I think new variable Z should be x^2 + y.
Nice Articlel
The solution is analogue to scenario-5 if you replace y by y-k
Your SVM explanation and kernel definition is very simple, and easy to understand. Kudos for that effort.
Most intuitive explanation of multidimensional svm I have seen. Thank you!
what is 'h' in the code of SVM . xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
z = (x^2 - y) z > 0, red circles
very neat explaination to SVC. For the proposed problem, my answers are: (1) z = a* x^2 + b y + c, a parabola. (2) z = a (x-0)^2 + b (y- y0)^2 - R^2, a circle or an ellipse enclosing red stars.
Great article.. I think the below formula would give a new variable that help to separate the points in hyper plane z = y - |x|
THANKS FOR EASY EXPLANATION
Useful article for Machine learners.. Why can't you discuss about effect of kernel functions.
The explanation is really impressive. Can you also provide some information about how to determine the theoretical limits for the parameter's optimal accuracy.
how can we use SVM for regression? can someone please explain..
hi please if you have an idiea about how it work for regression can you help me ?
That was a really good explanation! thanks a lot. I read many explanations about SVM but this one help me to understand the basics which I really needed it.
please give us the answer
This is very useful for understanding easily.
just substitude x with |x|
Same goes with Diana. This really help me a lot to figure out things from basic. I hope you would also share any computation example using R provided with simple dataset, so that anyone can practice with their own after referring to your article. I have a question, if i have time-series dataset containing mixed linear and nonlinear data, (for example oxygen saturation data ; SaO2), by using svm to do classification for diseased vs health subjects, do i have to separate those data into linear and non-linear fisrt, or can svm just performed the analysis without considering the differences between the linearity of those data? Thanks a lot!
z
Could you please explain how SVM works for multiple classes? How would it work for 9 classes? I used a function called multisvm here: http://www.mathworks.com/matlabcentral/fileexchange/39352-multi-class-svm but I'm not sure how it's working behind the scenes. Everything I've read online is rather confusing.
NEW VARIABLE (Z) = SQRT(X) + SQRT (Y)
Thank you so much!! That is really good explanation! I read many explanations about SVM but this one help me to understand the basics which I really needed it. keep it up!!
Thanks for the great article. There are even cool shirts for anyone who became SVM fan ;) http://www.redbubble.com/de/people/perceptron/works/24728522-support-vector-machines?grid_pos=2&p=t-shirt&style=mens
great explanation!! Thanks for posting it.
I think this is |X|
It is very nicely written and understandable. Thanks a lot...
z=ax^2 + by^2
nice explanations with scenarios and margin values
wow!!! excellent explanation.. only now i understood the concepts clearly thanks a lot..
(Z) = SQRT(X) + SQRT (Y)
thanks, and well done for the good article
it's magnific your explanation
Great Explanation..Thanks..
simple and refreshed the core concepts in just 5 mins! kudos Mr.Sunil
Best starters material for SVM, really appreciate the simple and comprehensive writing style. Expecting more such articles from you
Z= square(x)
Hey Sunil, Nice job of explaining it concisely and intuitively! Easy to follow and covers many aspects in a short space. Thanks!
Very well written - concise, clear, well-organized. Thank you.
Excellent explanation..Can you please also tell what are the parameter values one should start with - like C, gamma ..Also, again a very basic question.. Can we say that lesser the % of support vectors (count of SVs/total records) better my model/richer my data is- assuming the datasize to be the same.. Waiting for more on parameter tuning..Really appreciate the knowledge shared..
Hi could you please explain why SVM perform well on small dataset?
Another nice kernel for the problem stated in the article is the radial basis kernel.
[…] 资源:阅读这篇文章来理解SVM support vector machines。 […]
wow excellent
very appreciating for explaining
Nice tutorial. The new feature to separate data would be something like z = y - x^2 as most dots following the parabola will have lower z than stars.
Very intuitive explanation. Thank you! Good to add SVM for Regression of Continuous variables.
this is so simple method that anyone can get easily thnx for that but also explain the 4 senario of svm.
Great article for understanding of SVM: But, When and Why do we use the SVM algorithm can anyone make that help me understand because until this thing is clear there may not be use of this article. Thanks in advance.
It is one of best explanation of machine learning technique that i have seen! and new variable: i think Z=|x| and new Axis are Z and Y
higher degree polynomial will separate the points in the problem,
I guess the required feature is z = x^2 / y^2 For the red points, z will be close to 1 but for the blue points z values will be significantly more than 1
amazing article no doubt! It makes me clear all the concept and deep points regarding SVM. many thanks.
The best explanation ever! Thank you!
z = x^2+y^2
[…] [1] Naive Bayes and Text Classification. [2]Naive Bayes by Example. [3] Andrew Ng explanation of Naive Bayes video 1 and video 2 [4] Please explain SVM like I am 5 years old. [5] Understanding Support Vector Machines from examples. […]
new variable = ABS(Y)
Man, I was looking for definition of SVM for my diploma, but I got interested in explanation part of this article. Keep up good work!
we can use 'poly' kernel with degree=2
Hi.. Very well written, great article !:). Thanks so much share knowledge on SVM.
z=y-x^2
Wonderful, easy to understand explanation.
Thanks a lot for your explanations, they were really helpful and easy to understand
It would be a parabola z = a*x^2 + b*y^2 + c*x + d*y + e
Very good explanation, helpful
valuable explanation!!
Very helpfull
|X|
thank u sir ,it is easy to understand
z = x^2 + y
It may be z=x^2+y
y=x^2
z=ax^2 + by^2 + c
Nice. new variable is z = abs(x). Then replace x coordinates with z coordinates
z = |x|
PARABOLA
I think the boundaryf between two type of snapshot could be a curve (of a part of circle). So I prefer kernel Z=sqrt(X^2+(Y-c)^2)
Thanks a lot. I like how you define a problem and then solve it. It makes things clear.
z=x-y^2