By Jordi de la Torre on June 4, 2018
Error sources are noise, bias and variance. Model ensembles are a very effective way of reducing prediction errors. Bagging and Boosting are two ways of combining classifiers. They are able to convert a weak classifier into a very powerful one, just averaging multiple individual weak predictors.
The combination of multiple predictors decreases variance, increasing stability. Bagging uses a base learner algorithm (fe., classification trees), ie. a weak learner, for creating a pool of N weak predictors. Every predictor is generated by a different sample genereted by random sampling with replacement from the original dataset. In bagging every sample has the same probability of being chosen from the original dataset. The nature of the algorithm makes Bagging an easy paralelizable algorithm.
Boosting is very similar to Bagging with a significant difference: boosting uses a weighting strategy in sampling and in the combination of predictors. In Boosting the predictors generation process is sequential. Boosting increases the weights of misclassified data to emphasize the most difficult cases. Boosting also assigns weights to predictors giving more importance to best predictors. Due to this weighting strategy with predictors, Boosting is able not only to reduce variance but also to reduce bias. In counterpart, Boosting can overfit training data, problem that is not present in bagging predictors.