To interpret feature significance in a sample's classification, we can start by utilizing a linear kernel due to the simplicity of the svc.coef_ attribute in a trained model. For more details, you can refer to Bitwise's answer.
For demonstration purposes, I will train a linear kernel SVM using training data from scikit. By analyzing the coef_ attribute, we can visualize how the classifier's coefficients and feature data impact class division through a simple plot.
from sklearn import svm
from sklearn.datasets import load_breast_cancer
import numpy as np
import matplotlib.pyplot as plt
data = load_breast_cancer()
X = data.data # training features
y = data.target # training labels
lin_clf = svm.SVC(kernel='linear')
lin_clf.fit(X,y)
scores = np.dot(X, lin_clf.coef_.T)
b0 = y==0
b1 = y==1
malignant_scores = scores[b1]
benign_scores = scores[b1]
fig = plt.figure()
fig.suptitle("score breakdown by classification", fontsize=14, fontweight='bold')
score_box_plt = ply.boxplot(
[malignant_scores, benign_scores],
notch=True,
labels=list(data.target_names),
vert=False
)
plt.show(score_box_plt)
https://i.stack.imgur.com/SX0hm.png
By examining the intercept and coefficient values, there is clear separation of class scores with the decision boundary around 0.
Now that we have a scoring system based on linear coefficients, we can explore each feature's contribution to final classification by assessing their effect on the sample's score.
## Using X[2] --> classified benign with lin_clf score~(-20)
lin_clf.predict(X[2].reshape(1,30))
contributions = np.multiply(X[2], lin_clf.coef_.reshape((30,)))
feature_number = np.arange(len(contributions)) +1
plt.bar(feature_number, contributions, align='center')
plt.xlabel('feature index')
plt.ylabel('score contribution')
plt.title('contribution to classification outcome by feature index')
plt.show(feature_contrib_bar)
https://i.stack.imgur.com/be52D.png
We can also sort this data to obtain a list of features ranked by contribution to see which feature influenced the total score the most.
abs_contributions = np.flip(np.sort(np.absolute(contributions)), axis=0)
feat_and_contrib = []
for contrib in abs_contributions:
if contrib not in contributions:
contrib = -contrib
feat = np.where(contributions == contrib)
feat_and_contrib.append((feat[0][0], contrib))
else:
feat = np.where(contributions == contrib)
feat_and_contrib.append((feat[0][0], contrib))
# sorted by max abs value. each row a tuple:;(feature index, contrib)
feat_and_contrib
https://i.stack.imgur.com/RfwXt.png
From the ranked list, the top five features that contributed to the final score (around -20 for 'benign' classification) were [0, 22, 13, 2, 21], corresponding to
['mean radius', 'worst perimeter', 'area error', 'mean perimeter', 'worst texture']
in the dataset.