Why R?<\/strong> Well, the quick and easy question for this is that I do all my plotting in R (mostly because I think ggplot2<\/a> looks very pretty). I decided to explore Random Forests in R and to assess what are its advantages and shortcomings. I am planning to compare Random Forests in R against the python implementation in scikit-learn<\/a>. Do expect a post about this in the near future!<\/p>\n The data:\u00a0<\/strong>to keep things simple, I decided to use the\u00a0Edgar Anderson’s Iris Data set. You can have a look at it by inspecting the contents of\u00a0iris\u00a0<\/em>in R. This data set contains observations for four features (sepal length and width, and petal length and width – all in cm) of 150 flowers, equally split between three different iris species. This data set is fairly canon in classification and data analysis. Let us take a look at it, shall we:<\/p>\n As you can observe, there seems to be some\u00a0separation in regards to the different features and our three species\u00a0of irises [note: this set is not very representative of a real world data set and results should be taken with a grain of salt].<\/p>\n Training and Validation sets:\u00a0<\/strong>great care needs to be taken to ensure clear separation between training and validation sets. I tend to\u00a0save the cases for which I am actually interested in performing predictions as a second validation set (Validation 2). Then I split the remaining data evenly into Training and Validation 1.<\/p>\n Let us split our data set then, shall we?<\/p>\n Before we can move on, here are some things to consider:<\/p>\n 1- The size of your data set usually imposes a hard limit on how many features you can consider. This occurs due to the curse of dimensionality<\/a>, i.e. your data becomes sparser and sparser as you increase the number of features considered, which usually leads to overfitting. While there is no rule of thumb relating to how many features vs. \u00a0the number of observations you should use, I try to keep e^Nf < No (Nf = number of features, No = number of observations) to minimise overfitting [this is not always possible and it does not ensure that we won’t overfit]. In this case, our training set has 75 observations, which suggests that using four features (e^4 ~ 54.6) is not entirely absurd. Obviously, this depends on your data, so we will cover some further overfitting checks later on.<\/p>\n 2- An important thing to consider when assembling training sets is the proportion of negatives vs. positives in your data. Think of an extreme scenario where you have many, many more observations for one class vs. the others. How will this affect classification? This would make it more likely for the classifier to predict the dominant class when given new values. I mentioned before that the iris<\/em> set is quite nice to play with. It comes with exactly 50 observations for each species of irises. What happens if you have a data set with a much higher number of observations for a particular class? You can bypass any imbalance regarding the representation of each class by carefully constructing your training set in order not to favour any particular class. In this case, our randomly selected set has 21 observations for species setosa<\/em>\u00a0and 27 observations for each of species\u00a0versicolor\u00a0<\/em>and\u00a0virginica<\/em>, so we are good to go.<\/p>\n 3- Another common occurrence that is not represented by the iris<\/em> data set is missing values (NAs) for observations. There are many ways of dealing with missing values, including assigning the median or the mode for that particular feature to the missing observation or even disregarding some observations entirely, depending on how many observations you have. There are even ways to use random forests to estimate a good value to assign to the missing observations, but for the sake of brevity, this will not be covered here.<\/p>\n Right, data sets prepared and no missing values, it is time to fire our random forests algorithm. I am using the \u00a0randomForest\u00a0<\/em>package<\/a>. You can click the link for additional documentation. Here is the\u00a0example usage code:<\/p>\n Note some important parameters:<\/p>\n -The first parameter specifies our formula: Species ~ . (we want to predict Species using each of the remaining columns of data). We can quickly look at the results of our classifier for our training set by printing the contents of rf_classifier<\/em>:<\/p>\n As you can see, it lists the call used to build the classifier, the number of trees (100), the variables at each split (2), and it outputs a very useful confusion matrix and OOB estimate of error rate. This estimate is calculated by counting however many points in the training set were misclassified (2 versicolor and 2 virginica observations = 4) and dividing this number by the total number of observations (4\/75 ~= 5.33%).<\/p>\n The OOB estimate of error rate is a useful measure to discriminate between different random forest classifiers. We could, for instance, vary the number of trees or the number of variables to be considered, and select the combination that produces the smallest value for this error rate. For more complicated data sets, i.e. when a higher number of features is present, a good idea is to use cross-validation to perform feature selection using the OOB error rate (see rfcv from randomForest for more details).<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/www.blopig.com\/blog\/wp-content\/uploads\/2017\/04\/Blogpost1.png?resize=625%2C625&ssl=1\")
<\/a><\/p>\n# Set random seed to make results reproducible:\r\nset.seed(17)\r\n# Calculate the size of each of the data sets:\r\ndata_set_size <- floor(nrow(iris)\/2)\r\n# Generate a random sample of \"data_set_size\" indexes\r\nindexes <- sample(1:nrow(iris), size = data_set_size)\r\n\r\n# Assign the data to the correct sets\r\ntraining <- iris[indexes,]\r\nvalidation1 <- iris[-indexes,]\r\n\r\n<\/pre>\n
#import the package\r\nlibrary(randomForest)\r\n# Perform training:\r\nrf_classifier = randomForest(Species ~ ., data=training, ntree=100, mtry=2, importance=TRUE)\r\n<\/pre>\n
\n–ntree<\/em> defines the number of trees to be generated. It is typical to test a range of values for this parameter (i.e. 100,200,300,400,500) and choose the one that minimises the OOB estimate of error rate.
\n–mtry<\/em> is the number of features used in the construction of each tree. These features are selected at random, which is where the “random” in “random forests” comes from. The default value for this parameter, when performing classification, is sqrt(number of features).
\n–importance<\/em>\u00a0enables the algorithm to calculate variable importance.<\/p>\n> rf_classifier\r\n\r\nCall:\r\n randomForest(formula = Species ~ ., data = training,ntree=100,mtry=2, importance = TRUE) \r\n Type of random forest: classification\r\n Number of trees: 100\r\nNo. of variables tried at each split: 2\r\n\r\n OOB estimate of error rate: 5.33%\r\nConfusion matrix:\r\n setosa versicolor virginica class.error\r\nsetosa 21 0 0 0.00000000\r\nversicolor 0 25 2 0.07407407\r\nvirginica 0 2 25 0.07407407\r\n\r\n\r\n<\/pre>\n
![\"\"](\"https:\/\/i0.wp.com\/www.blopig.com\/blog\/wp-content\/uploads\/2017\/04\/Blogpost2.png?resize=625%2C313&ssl=1\")
<\/a><\/p>\n![\"\"](\"https:\/\/i0.wp.com\/www.blopig.com\/blog\/wp-content\/uploads\/2017\/04\/Blogpost3.png?resize=625%2C625&ssl=1\")
<\/a><\/p>\n