TF/04_Support_Vector_Machines/02_Working_with_Linear_SVMs03_Reduction_to_Linear_Regression
2017-05-26 11:36
483 查看
02 Working with Linear SVMs
We introduce a linear SVM on a binary set, which will be a subset of the Iris data. We know for I. setosa, that petal width and sepal length are completely separable. We will create a linear SVM to predict I. setosa based on two features: petal width and sepal length.It is worth noting that due to the small data set and the randomness of separating into train/test sets, that it may appear that a few points can end up on the wrong side of the line. This is because they are in the test set, and this will result in a lower test accuracy.
Model
We will aim to maximize the margin width, 2/||A||, or minimize ||A||. We allow for a soft margin by having an error term in the loss function which is the max(0, 1-pred*actual).Graph of Linear SVM
Here is a plot of the linear SVM separator of I. setosa based on petal width and sepal length.The accuracy is below, plotted over each iteration.
An important observation is that while we achieve the linear separator rather quickly (100% accuracy), the loss function continues to decrease. This is because we are trying to optimize for the maximal linear separator between the two classes.
Ideally, we would have enough data to do a cross validation technique, or even to separate the data into train and test sets before optimization.
#02_linear_svm.py # Linear Support Vector Machine: Soft Margin # ---------------------------------- # # This function shows how to use TensorFlow to # create a soft margin SVM # # We will use the iris data, specifically: # x1 = Sepal Length # x2 = Petal Width # Class 1 : I. setosa # Class -1: not I. setosa # # We know here that x and y are linearly seperable # for I. setosa classification. import matplotlib.pyplot as plt import numpy as np import tensorflow as tf from sklearn import datasets from tensorflow.python.framework import ops ops.reset_default_graph() # Create graph sess = tf.Session() # Load the data # iris.data = [(Sepal Length, Sepal Width, Petal Length, Petal Width)] iris = datasets.load_iris() x_vals = np.array([[x[0], x[3]] for x in iris.data]) y_vals = np.array([1 if y == 0 else -1 for y in iris.target]) # Split data into train/test sets train_indices = np.random.choice(len(x_vals), round(len(x_vals)*0.8), replace=False) test_indices = np.array(list(set(range(len(x_vals))) - set(train_indices))) x_vals_train = x_vals[train_indices] x_vals_test = x_vals[test_indices] y_vals_train = y_vals[train_indices] y_vals_test = y_vals[test_indices] # Declare batch size batch_size = 100 # Initialize placeholders x_data = tf.placeholder(shape=[None, 2], dtype=tf.float32) y_target = tf.placeholder(shape=[None, 1], dtype=tf.float32) # Create variables for linear regression A = tf.Variable(tf.random_normal(shape=[2, 1])) b = tf.Variable(tf.random_normal(shape=[1, 1])) # Declare model operations model_output = tf.subtract(tf.matmul(x_data, A), b) # Declare vector L2 'norm' function squared l2_norm = tf.reduce_sum(tf.square(A)) # Declare loss function # Loss = max(0, 1-pred*actual) + alpha * L2_norm(A)^2 # L2 regularization parameter, alpha alpha = tf.constant([0.01]) # Margin term in loss classification_term = tf.reduce_mean(tf.maximum(0., tf.subtract(1., tf.multiply(model_output, y_target)))) # Put terms together loss = tf.add(classification_term, tf.multiply(alpha, l2_norm)) # Declare prediction function prediction = tf.sign(model_output) accuracy = tf.reduce_mean(tf.cast(tf.equal(prediction, y_target), tf.float32)) # Declare optimizer my_opt = tf.train.GradientDescentOptimizer(0.01) train_step = my_opt.minimize(loss) # Initialize variables init = tf.global_variables_initializer() sess.run(init) # Training loop loss_vec = [] train_accuracy = [] test_accuracy = [] for i in range(500): rand_index = np.random.choice(len(x_vals_train), size=batch_size) rand_x = x_vals_train[rand_index] rand_y = np.transpose([y_vals_train[rand_index]]) sess.run(train_step, feed_dict={x_data: rand_x, y_target: rand_y}) temp_loss = sess.run(loss, feed_dict={x_data: rand_x, y_target: rand_y}) loss_vec.append(temp_loss) train_acc_temp = sess.run(accuracy, feed_dict={ x_data: x_vals_train, y_target: np.transpose([y_vals_train])}) train_accuracy.append(train_acc_temp) test_acc_temp = sess.run(accuracy, feed_dict={ x_data: x_vals_test, y_target: np.transpose([y_vals_test])}) test_accuracy.append(test_acc_temp) if (i + 1) % 100 == 0: print('Step #{} A = {}, b = {}'.format( str(i+1), str(sess.run(A)), str(sess.run(b)) )) print('Loss = ' + str(temp_loss)) # Extract coefficients [[a1], [a2]] = sess.run(A) [[b]] = sess.run(b) slope = -a2/a1 y_intercept = b/a1 # Extract x1 and x2 vals x1_vals = [d[1] for d in x_vals] # Get best fit line best_fit = [] for i in x1_vals: best_fit.append(slope*i+y_intercept) # Separate I. setosa setosa_x = [d[1] for i, d in enumerate(x_vals) if y_vals[i] == 1] setosa_y = [d[0] for i, d in enumerate(x_vals) if y_vals[i] == 1] not_setosa_x = [d[1] for i, d in enumerate(x_vals) if y_vals[i] == -1] not_setosa_y = [d[0] for i, d in enumerate(x_vals) if y_vals[i] == -1] # Plot data and line plt.plot(setosa_x, setosa_y, 'o', label='I. setosa') plt.plot(not_setosa_x, not_setosa_y, 'x', label='Non-setosa') plt.plot(x1_vals, best_fit, 'r-', label='Linear Separator', linewidth=3) plt.ylim([0, 10]) plt.legend(loc='lower right') plt.title('Sepal Length vs Pedal Width') plt.xlabel('Pedal Width') plt.ylabel('Sepal Length') plt.show() # Plot train/test accuracies plt.plot(train_accuracy, 'k-', label='Training Accuracy') plt.plot(test_accuracy, 'r--', label='Test Accuracy') plt.title('Train and Test Set Accuracies') plt.xlabel('Generation') plt.ylabel('Accuracy') plt.legend(loc='lower right') plt.show() # Plot loss over time plt.plot(loss_vec, 'k-') plt.title('Loss per Generation') plt.xlabel('Generation') plt.ylabel('Loss') plt.show()
Step #100 A = [[-0.47445503] [-0.44861239]], b = [[-2.6099627]] Loss = [ 0.45341077] Step #200 A = [[-0.41855416] [-0.73976141]], b = [[-2.66076303]] Loss = [ 0.27603617] Step #300 A = [[-0.35709888] [-1.00628901]], b = [[-2.71116281]] Loss = [ 0.25569785] Step #400 A = [[-0.3331852 ] [-1.24517143]], b = [[-2.7465632]] Loss = [ 0.18576032] Step #500 A = [[-0.31063056] [-1.41434669]], b = [[-2.77026415]] Loss = [ 0.16047686]
03 SVM Reduction to Linear Regression
Instead of optimizing the maximal linear separator, we change the loss function to maximize the amount of data points we can make fit in our margin. This will give us a linear regression estimation.# 03_support_vector_regression.py # SVM Regression #---------------------------------- # # This function shows how to use TensorFlow to # solve support vector regression. We are going # to find the line that has the maximum margin # which INCLUDES as many points as possible # # We will use the iris data, specifically: # y = Sepal Length # x = Pedal Width import matplotlib.pyplot as plt import numpy as np import tensorflow as tf from sklearn import datasets from tensorflow.python.framework import ops ops.reset_default_graph() # Create graph #修改位置 config = tf.ConfigProto(allow_soft_placement= True, log_device_placement= True) sess = tf.Session(config = config) # Create graph # sess = tf.Session() # Load the data # iris.data = [(Sepal Length, Sepal Width, Petal Length, Petal Width)] iris = datasets.load_iris() x_vals = np.array([x[3] for x in iris.data]) y_vals = np.array([y[0] for y in iris.data]) # Split data into train/test sets train_indices = np.random.choice(len(x_vals), round(len(x_vals)*0.8), replace=False) test_indices = np.array(list(set(range(len(x_vals))) - set(train_indices))) x_vals_train = x_vals[train_indices] x_vals_test = x_vals[test_indices] y_vals_train = y_vals[train_indices] y_vals_test = y_vals[test_indices] # Declare batch size batch_size = 50 # Initialize placeholders x_data = tf.placeholder(shape=[None, 1], dtype=tf.float32) y_target = tf.placeholder(shape=[None, 1], dtype=tf.float32) # Create variables for linear regression A = tf.Variable(tf.random_normal(shape=[1,1])) b = tf.Variable(tf.random_normal(shape=[1,1])) # Declare model operations model_output = tf.add(tf.matmul(x_data, A), b) # Declare loss function # = max(0, abs(target - predicted) + epsilon) # 1/2 margin width parameter = epsilon epsilon = tf.constant([0.5]) # Margin term in loss loss = tf.reduce_mean(tf.maximum(0., tf.subtract(tf.abs(tf.subtract(model_output, y_target)), epsilon))) # Declare optimizer my_opt = tf.train.GradientDescentOptimizer(0.075) train_step = my_opt.minimize(loss) # Initialize variables init = tf.global_variables_initializer() sess.run(init) # Training loop train_loss = [] test_loss = [] for i in range(200): rand_index = np.random.choice(len(x_vals_train), size=batch_size) rand_x = np.transpose([x_vals_train[rand_index]]) rand_y = np.transpose([y_vals_train[rand_index]]) sess.run(train_step, feed_dict={x_data: rand_x, y_target: rand_y}) temp_train_loss = sess.run(loss, feed_dict={x_data: np.transpose([x_vals_train]), y_target: np.transpose([y_vals_train])}) train_loss.append(temp_train_loss) temp_test_loss = sess.run(loss, feed_dict={x_data: np.transpose([x_vals_test]), y_target: np.transpose([y_vals_test])}) test_loss.append(temp_test_loss) if (i+1)%50==0: print('-----------') print('Generation: ' + str(i+1)) print('A = ' + str(sess.run(A)) + ' b = ' + str(sess.run(b))) print('Train Loss = ' + str(temp_train_loss)) print('Test Loss = ' + str(temp_test_loss)) # Extract Coefficients [[slope]] = sess.run(A) [[y_intercept]] = sess.run(b) [width] = sess.run(epsilon) # Get best fit line best_fit = [] best_fit_upper = [] best_fit_lower = [] for i in x_vals: best_fit.append(slope*i+y_intercept) best_fit_upper.append(slope*i+y_intercept+width) best_fit_lower.append(slope*i+y_intercept-width) # Plot fit with data plt.plot(x_vals, y_vals, 'o', label='Data Points') plt.plot(x_vals, best_fit, 'r-', label='SVM Regression Line', linewidth=3) plt.plot(x_vals, best_fit_upper, 'r--', linewidth=2) plt.plot(x_vals, best_fit_lower, 'r--', linewidth=2) plt.ylim([0, 10]) plt.legend(loc='lower right') plt.title('Sepal Length vs Pedal Width') plt.xlabel('Pedal Width') plt.ylabel('Sepal Length') plt.show() # Plot loss over time plt.plot(train_loss, 'k-', label='Train Set Loss') plt.plot(test_loss, 'r--', label='Test Set Loss') plt.title('L2 Loss per Generation') plt.xlabel('Generation') plt.ylabel('L2 Loss') plt.legend(loc='upper right') plt.show()
----------- Generation: 50 A = [[ 2.13920832]] b = [[ 2.75946236]] Train Loss = 0.573591 Test Loss = 0.494945 ----------- Generation: 100 A = [[ 1.57715869]] b = [[ 3.75246167]] Train Loss = 0.231339 Test Loss = 0.199602 ----------- Generation: 150 A = [[ 1.19150889]] b = [[ 4.31496096]] Train Loss = 0.0996593 Test Loss = 0.118922 ----------- Generation: 200 A = [[ 1.02305913]] b = [[ 4.54146004]] Train Loss = 0.0792556 Test Loss = 0.110936
相关文章推荐
- TF/04_Support_Vector_Machines/04_Working_with_Kernels
- Linear to physical address translation with support for page attributes
- Linear Regression with TensorFlow
- 解决 The 'InnoDB' feature is disabled; you need MySQL built with 'InnoDB' to have it working
- Support Vector Machine Algorithm For Regression(SVR)算法详细解析
- 机器学习之多变量线性回归(Linear Regression with multiple variables)
- this client is too old to work with working? SVN上遇到的这个问题的原因
- Linear regression with one variable算法实例讲解(绘制图像,cost_Function ,Gradient Desent, 拟合曲线, 轮廓图绘制)_矩阵操作
- SVN:This client is too old to work with working copy…解决办法
- WEEK3-Quick guide to linear regression
- 带正则项的线性回归Regression (linear regression with regulation)
- Machine Learning week 2 quiz: Linear Regression with Multiple Variables
- support vector regression与 kernel ridge regression
- 【Machine Learning】单参数线性回归 Linear Regression with one variable
- An introduction to Linear Regression
- 3. Linear Regression with Multiple Variables
- linear regression code with python
- Standford 机器学习—第二讲 Linear Regression with multiple variables(多变量线性回归)
- Gradle Goodness: Add Incremental Build Support to Custom Tasks with Annotations
- Stanford公开课机器学习---week2-1.多变量线性回归 (Linear Regression with multiple variable)