A Tour of Machine Learning Algorithms
2015-01-13 15:56
537 查看
After we understand the type of
machine learning problem we are working with, we can think about the type of data to collect and the types of machine learning algorithms we can try. In this post we take a tour of the most popular machine learning algorithms. It is useful to tour the main
algorithms to get a general idea of what methods are available.
There are so many algorithms available. The difficulty is that there are classes of method and there are extensions to methods and it quickly becomes very difficult to determine what constitutes a canonical algorithm. In this post I want to give you two
ways to think about and categorize the algorithms you may come across in the field.
The first is a grouping of algorithms by the learning style. The second is a grouping of algorithms by similarity in form or function (like grouping similar animals together). Both approaches are useful.
learning styles that an algorithm can adopt.
There are only a few main learning styles or learning models that an algorithm can have and we’ll go through them here with a few examples of algorithms and problem types that they suit. This taxonomy or way of organizing machine learning algorithms is useful
because it forces you to think about the the roles of the input data and the model preparation process and select one that is the most appropriate for your problem in order to get the best result.
Supervised Learning: Input data is called training data and has a known label or result such as spam/not-spam or a stock price at a time. A model is prepared through a training process where it is required to make predictions and is corrected
when those predictions are wrong. The training process continues until the model achieves a desired level of accuracy on the training data. Example problems are classification and regression. Example algorithms are Logistic Regression and the Back Propagation
Neural Network.
Unsupervised Learning: Input data is not labelled and does not have a known result. A model is prepared by deducing structures present in the input data. Example problems are association rule learning and clustering. Example algorithms
are the Apriori algorithm and k-means.
Semi-Supervised Learning: Input data is a mixture of labelled and unlabelled examples. There is a desired prediction problem but the model must learn the structures to organize the data as well as make predictions. Example problems are
classification and regression. Example algorithms are extensions to other flexible methods that make assumptions about how to model the unlabelled data.
Reinforcement Learning: Input data is provided as stimulus to a model from an environment to which the model must respond and react. Feedback is provided not from of a teaching process as in supervised learning, but as punishments and rewards
in the environment. Example problems are systems and robot control. Example algorithms are Q-learning and Temporal difference learning.
When crunching data to model business decisions, you are most typically using supervised and unsupervised learning methods. A hot topic at the moment is semi-supervised learning methods in areas such as image classification where there are large datasets
with very few labelled examples. Reinforcement learning is more likely to turn up in robotic control and other control systems development.
just as easily fit into multiple categories like Learning Vector Quantization that is both a neural network inspired method and an instance-based method. There are also categories that have the same name that describes the problem and the class of algorithm
such as Regression and Clustering. As such, you will see variations on the way algorithms are grouped depending on the source you check. Like machine learning algorithms themselves, there is no perfect model, just a good enough model.
In this section I list many of the popular machine leaning algorithms grouped the way I think is the most intuitive. It is not exhaustive in either the groups or the algorithms, but I think it is representative and will be useful to you to get an idea of
the lay of the land. If you know of an algorithm or a group of algorithms not listed, put it in the comments and share it with us. Let’s dive in.
learning. This may be confusing because we can use regression to refer to the class of problem and the class of algorithm. Really, regression is a process. Some example algorithms are:
Ordinary Least Squares
Logistic Regression
Stepwise Regression
Multivariate Adaptive Regression Splines (MARS)
Locally Estimated Scatterplot Smoothing (LOESS)
measure in order to find the best match and make a prediction. For this reason, instance-based methods are also called winner-take all methods and memory-based learning. Focus is put on representation of the stored instances and similarity measures used between
instances.
k-Nearest Neighbour (kNN)
Learning Vector Quantization (LVQ)
Self-Organizing Map (SOM)
and generally simple modifications made to other methods.
Ridge Regression
Least Absolute Shrinkage and Selection Operator (LASSO)
Elastic Net
regression problems.
Classification and Regression Tree (CART)
Iterative Dichotomiser 3 (ID3)
C4.5
Chi-squared Automatic Interaction Detection (CHAID)
Decision Stump
Random Forest
Multivariate Adaptive Regression Splines (MARS)
Gradient Boosting Machines (GBM)
Naive Bayes
Averaged One-Dependence Estimators (AODE)
Bayesian Belief Network (BBN)
or regression problems are easier to model.
Support Vector Machines (SVM)
Radial Basis Function (RBF)
Linear Discriminant Analysis (LDA)
in the data to best organize the data into groups of maximum commonality.
k-Means
Expectation Maximisation (EM)
by an organisation.
Apriori algorithm
Eclat algorithm
comprised of hundreds of algorithms and variations for all manner of problem types. Some of the classically popular methods include (I have separated Deep Learning from this category):
Perceptron
Back-Propagation
Hopfield Network
Self-Organizing Map (SOM)
Learning Vector Quantization (LVQ)
learning problems where large datasets contain very little labelled data.
Restricted Boltzmann Machine (RBM)
Deep Belief Networks (DBN)
Convolutional Network
Stacked Auto-encoders
or to simplify data which can then be used in a supervized learning method.
Principal Component Analysis (PCA)
Partial Least Squares Regression (PLS)
Sammon Mapping
Multidimensional Scaling (MDS)
Projection Pursuit
to combine them. This is a very powerful class of techniques and as such is very popular.
Boosting
Bootstrapped Aggregation (Bagging)
AdaBoost
Stacked Generalization (blending)
Gradient Boosting Machines (GBM)
Random Forest
Example of an ensemble of lines of best fit. Weak members are grey, the combined prediction is red. Specifically shows Temperature/Ozone data and a plot of a model prepared with LOESS method.
Image is licensed public domain and is attributed to Wikipedia.
The resources for this post are as you would expect, other great lists of machine learning algorithms. Try not to feel overwhelmed. It is useful to know about many algorithms, but it is also useful to be effective and have a deep knowledge of just a few
key methods.
List of Machine Learning Algorithms: On Wikipedia. Although extensive, I do not find this list or the organization of the algorithms particularly useful.
Machine Learning Algorithms Category: Also on Wikipedia, a slightly more useful than Wikipedias great list above. It organizes algorithms alphabetically.
CRAN Task View: Machine Learning & Statistical Learning: A list of all the packages and all the algorithms supported by each machine learning package in R. Gives you
a grounded feeling of what’s out there and what people are using for analysis day-to-day.
Top 10 Algorithms in Data Mining:
Published article and now a book (Affiliate Link) on the most popular algorithms for data mining. Another grounded and less overwheling take on
methods that you could go off and learn deeply.
I hope you have found this tour useful. Leave a comment if you know of a better way to think about organizing algorithms or if you know of any other great lists of machine learning algorithms.
UPDATE: Continue the
discussion on HackerNews and
reddit.
The field of machine learning can be confusing. There are so many algorithms and problems. Terms that are thrown around. Terms that you are expected to know.
You need to know the lay of the land before diving in.
You need the 28-page PDF guide:
Machine Learning Foundations
In this guide you will discover what machine learning is as well as the high-level concepts you need to know to get started.
Learn more about him.
View all posts by Jason Brownlee
→
http://machinelearningmastery.com/a-tour-of-machine-learning-algorithms/
machine learning problem we are working with, we can think about the type of data to collect and the types of machine learning algorithms we can try. In this post we take a tour of the most popular machine learning algorithms. It is useful to tour the main
algorithms to get a general idea of what methods are available.
There are so many algorithms available. The difficulty is that there are classes of method and there are extensions to methods and it quickly becomes very difficult to determine what constitutes a canonical algorithm. In this post I want to give you two
ways to think about and categorize the algorithms you may come across in the field.
The first is a grouping of algorithms by the learning style. The second is a grouping of algorithms by similarity in form or function (like grouping similar animals together). Both approaches are useful.
Learning Style
There are different ways an algorithm can model a problem based on its interaction with the experience or environment or whatever we want to call the input data. It is popular in machine learning and artificial intelligence text books to first consider thelearning styles that an algorithm can adopt.
There are only a few main learning styles or learning models that an algorithm can have and we’ll go through them here with a few examples of algorithms and problem types that they suit. This taxonomy or way of organizing machine learning algorithms is useful
because it forces you to think about the the roles of the input data and the model preparation process and select one that is the most appropriate for your problem in order to get the best result.
Supervised Learning: Input data is called training data and has a known label or result such as spam/not-spam or a stock price at a time. A model is prepared through a training process where it is required to make predictions and is corrected
when those predictions are wrong. The training process continues until the model achieves a desired level of accuracy on the training data. Example problems are classification and regression. Example algorithms are Logistic Regression and the Back Propagation
Neural Network.
Unsupervised Learning: Input data is not labelled and does not have a known result. A model is prepared by deducing structures present in the input data. Example problems are association rule learning and clustering. Example algorithms
are the Apriori algorithm and k-means.
Semi-Supervised Learning: Input data is a mixture of labelled and unlabelled examples. There is a desired prediction problem but the model must learn the structures to organize the data as well as make predictions. Example problems are
classification and regression. Example algorithms are extensions to other flexible methods that make assumptions about how to model the unlabelled data.
Reinforcement Learning: Input data is provided as stimulus to a model from an environment to which the model must respond and react. Feedback is provided not from of a teaching process as in supervised learning, but as punishments and rewards
in the environment. Example problems are systems and robot control. Example algorithms are Q-learning and Temporal difference learning.
When crunching data to model business decisions, you are most typically using supervised and unsupervised learning methods. A hot topic at the moment is semi-supervised learning methods in areas such as image classification where there are large datasets
with very few labelled examples. Reinforcement learning is more likely to turn up in robotic control and other control systems development.
Algorithm Similarity
Algorithms are universally presented in groups by similarity in terms of function or form. For example, tree based methods, and neural network inspired methods. This is a useful grouping method, but it is not perfect. There are still algorithms that couldjust as easily fit into multiple categories like Learning Vector Quantization that is both a neural network inspired method and an instance-based method. There are also categories that have the same name that describes the problem and the class of algorithm
such as Regression and Clustering. As such, you will see variations on the way algorithms are grouped depending on the source you check. Like machine learning algorithms themselves, there is no perfect model, just a good enough model.
In this section I list many of the popular machine leaning algorithms grouped the way I think is the most intuitive. It is not exhaustive in either the groups or the algorithms, but I think it is representative and will be useful to you to get an idea of
the lay of the land. If you know of an algorithm or a group of algorithms not listed, put it in the comments and share it with us. Let’s dive in.
Regression
Regression is concerned with modelling the relationship between variables that is iteratively refined using a measure of error in the predictions made by the model. Regression methods are a work horse of statistics and have been cooped into statistical machinelearning. This may be confusing because we can use regression to refer to the class of problem and the class of algorithm. Really, regression is a process. Some example algorithms are:
Ordinary Least Squares
Logistic Regression
Stepwise Regression
Multivariate Adaptive Regression Splines (MARS)
Locally Estimated Scatterplot Smoothing (LOESS)
Instance-based Methods
Instance based learning model a decision problem with instances or examples of training data that are deemed important or required to the model. Such methods typically build up a database of example data and compare new data to the database using a similaritymeasure in order to find the best match and make a prediction. For this reason, instance-based methods are also called winner-take all methods and memory-based learning. Focus is put on representation of the stored instances and similarity measures used between
instances.
k-Nearest Neighbour (kNN)
Learning Vector Quantization (LVQ)
Self-Organizing Map (SOM)
Regularization Methods
An extension made to another method (typically regression methods) that penalizes models based on their complexity, favoring simpler models that are also better at generalizing. I have listed Regularization methods here because they are popular, powerfuland generally simple modifications made to other methods.
Ridge Regression
Least Absolute Shrinkage and Selection Operator (LASSO)
Elastic Net
Decision Tree Learning
Decision tree methods construct a model of decisions made based on actual values of attributes in the data. Decisions fork in tree structures until a prediction decision is made for a given record. Decision trees are trained on data for classification andregression problems.
Classification and Regression Tree (CART)
Iterative Dichotomiser 3 (ID3)
C4.5
Chi-squared Automatic Interaction Detection (CHAID)
Decision Stump
Random Forest
Multivariate Adaptive Regression Splines (MARS)
Gradient Boosting Machines (GBM)
Bayesian
Bayesian methods are those that are explicitly apply Bayes’ Theorem for problems such as classification and regression.Naive Bayes
Averaged One-Dependence Estimators (AODE)
Bayesian Belief Network (BBN)
Kernel Methods
Kernel Methods are best known for the popular method Support Vector Machines which is really a constellation of methods in and of itself. Kernel Methods are concerned with mapping input data into a higher dimensional vector space where some classificationor regression problems are easier to model.
Support Vector Machines (SVM)
Radial Basis Function (RBF)
Linear Discriminant Analysis (LDA)
Clustering Methods
Clustering, like regression describes the class of problem and the class of methods. Clustering methods are typically organized by the modelling approaches such as centroid-based and hierarchal. All methods are concerned with using the inherent structuresin the data to best organize the data into groups of maximum commonality.
k-Means
Expectation Maximisation (EM)
Association Rule Learning
Association rule learning are methods that extract rules that best explain observed relationships between variables in data. These rules can discover important and commercially useful associations in large multidimensional datasets that can be exploitedby an organisation.
Apriori algorithm
Eclat algorithm
Artificial Neural Networks
Artificial Neural Networks are models that are inspired by the structure and/or function of biological neural networks. They are a class of pattern matching that are commonly used for regression and classification problems but are really an enormous subfieldcomprised of hundreds of algorithms and variations for all manner of problem types. Some of the classically popular methods include (I have separated Deep Learning from this category):
Perceptron
Back-Propagation
Hopfield Network
Self-Organizing Map (SOM)
Learning Vector Quantization (LVQ)
Deep Learning
Deep Learning methods are a modern update to Artificial Neural Networks that exploit abundant cheap computation. They are concerned with building much larger and more complex neural networks, and as commented above, many methods are concerned with semi-supervisedlearning problems where large datasets contain very little labelled data.
Restricted Boltzmann Machine (RBM)
Deep Belief Networks (DBN)
Convolutional Network
Stacked Auto-encoders
Dimensionality Reduction
Like clustering methods, Dimensionality Reduction seek and exploit the inherent structure in the data, but in this case in an unsupervised manner or order to summarise or describe data using less information. This can be useful to visualize dimensional dataor to simplify data which can then be used in a supervized learning method.
Principal Component Analysis (PCA)
Partial Least Squares Regression (PLS)
Sammon Mapping
Multidimensional Scaling (MDS)
Projection Pursuit
Ensemble Methods
Ensemble methods are models composed of multiple weaker models that are independently trained and whose predictions are combined in some way to make the overall prediction. Much effort is put into what types of weak learners to combine and the ways in whichto combine them. This is a very powerful class of techniques and as such is very popular.
Boosting
Bootstrapped Aggregation (Bagging)
AdaBoost
Stacked Generalization (blending)
Gradient Boosting Machines (GBM)
Random Forest
Example of an ensemble of lines of best fit. Weak members are grey, the combined prediction is red. Specifically shows Temperature/Ozone data and a plot of a model prepared with LOESS method.
Image is licensed public domain and is attributed to Wikipedia.
Resources
This tour of machine learning algorithms was intended to give you an overview of what is out there and some tools to relate algorithms that you may come across to each other.The resources for this post are as you would expect, other great lists of machine learning algorithms. Try not to feel overwhelmed. It is useful to know about many algorithms, but it is also useful to be effective and have a deep knowledge of just a few
key methods.
List of Machine Learning Algorithms: On Wikipedia. Although extensive, I do not find this list or the organization of the algorithms particularly useful.
Machine Learning Algorithms Category: Also on Wikipedia, a slightly more useful than Wikipedias great list above. It organizes algorithms alphabetically.
CRAN Task View: Machine Learning & Statistical Learning: A list of all the packages and all the algorithms supported by each machine learning package in R. Gives you
a grounded feeling of what’s out there and what people are using for analysis day-to-day.
Top 10 Algorithms in Data Mining:
Published article and now a book (Affiliate Link) on the most popular algorithms for data mining. Another grounded and less overwheling take on
methods that you could go off and learn deeply.
I hope you have found this tour useful. Leave a comment if you know of a better way to think about organizing algorithms or if you know of any other great lists of machine learning algorithms.
UPDATE: Continue the
discussion on HackerNews and
reddit.
Take
The Next Step
The field of machine learning can be confusing. There are so many algorithms and problems. Terms that are thrown around. Terms that you are expected to know.You need to know the lay of the land before diving in.
You need the 28-page PDF guide:
Machine Learning Foundations
In this guide you will discover what machine learning is as well as the high-level concepts you need to know to get started.
About Jason Brownlee
Editor and Chief at MachineLearningMastery.com. Dr Brownlee is a husband, father, professional programmer and a machine learning enthusiast.Learn more about him.
View all posts by Jason Brownlee
→
http://machinelearningmastery.com/a-tour-of-machine-learning-algorithms/
相关文章推荐
- 机器学习算法之旅A Tour of Machine Learning Algorithms
- A Tour of Machine Learning Algorithms
- A Tour of Machine Learning Algorithms
- 机器学习文献(一)Systematic Analysis of Machine Learning Algorithms on EEG Data for Brain State Intelligence
- The Implementation of baseline algorithms of Machine Learning
- An overview of machine learning
- 4. 机器学习基石-When can Machine Learn? - Feasible of Learning
- Machine Learning Algorithms Study Notes(5)—Reinforcement Learning
- Machine learning of neural representations of suicide and emotion concepts identifies suicidal youth
- type of machine learning
- Teaching Your Computer To Play Super Mario Bros. – A Fork of the Google DeepMind Atari Machine Learning Project
- How to Study Machine Learning Algorithms
- 论文读书笔记-Supervised machine learning:a review of classification techniques
- ON THE EVOLUTION OF MACHINE LEARNING: FROM LINEAR MODELS TO NEURAL NETWORKS
- my list of cool machine learning books
- #Paper Reading# Lifelong Machine Learning Systems: Beyond Learning Algorithms
- state of the art result for machine learning problems
- Resource of computer vision, pattern recognition, machine learning etc.
- books of machine learning
- Machine Learning Algorithms Study Notes(4)—无监督学习(unsupervised learning)