Tree-Based Architectures & Ensembles: Decision Trees, Random Forests, XGBoost, LightGBM & CatBoost
Impurity splitting criteria (Gini, Entropy), bagging variance reduction, and sequential gradient boosting algorithms.
Executive Summary
Master decision tree split criteria (Gini vs Entropy), Random Forest bagging, and gradient boosted powerhouses like XGBoost, LightGBM, and CatBoost.
Key Takeaways
- ✓Decision Trees — covered in depth with practical examples, formulas, and code.
- ✓Bagging: Random Forest — covered in depth with practical examples, formulas, and code.
- ✓Gradient Boosting Frameworks — covered in depth with practical examples, formulas, and code.
Tree-Based Models & Ensemble Frameworks
#1. Decision Trees
Splits feature space recursively based on split criteria:
Key hyperparameters: max_depth, min_samples_split, min_samples_leaf.
#2. Bagging: Random Forest
Ensemble of independent decision trees trained on bootstrap samples with random feature selection. Reduces model variance without increasing bias.
#3. Gradient Boosting Frameworks
Sequential trees fitting negative gradients of the loss function:
| Model | Split Strategy | Key Strength |
|---|---|---|
| XGBoost | Pre-sorted exact / approx splits | Robust, accurate, handles missing data |
| LightGBM | Histogram leaf-wise (best-first) | 10x faster training on massive datasets |
| CatBoost | Symmetric balanced trees | Superior out-of-the-box categorical feature handling |
import xgboost as xgb
model = xgb.XGBClassifier(n_estimators=200, learning_rate=0.05, max_depth=6).fit(X_train, y_train)Govindarajan Selvaraj
ML Engineer
Govindarajan Selvaraj is part of the Junglans Solutions engineering team, specializing in model architectures. Junglans builds a 20-product ecosystem of local-first enterprise software — AI developer tools, encrypted communication, and data infrastructure with zero cloud telemetry.
Meet the full Junglans engineering team ↗This article is part of the Junglans Research knowledge base, produced alongside the engineering teams that build our production AI tools. Explore related product documentation and research:
- Junglans ML Visualizer ↗
Interactive 22-algorithm machine learning sandbox — see the concepts in action.
- JunglasNCode ↗
Line-by-line code execution and call stack visualizer for algorithm practice.
- All Junglans Research Articles ↗
More engineering and AI deep-dives from the Junglans team.
Related Research & Articles
ROC-AUC Curves, PR-AUC, and Multi-Class Evaluation Metrics
Explore threshold-agnostic ROC curves, probability ranking, Precision-Recall AUC for rare classes, and multi-class macro, micro, and weighted averaging techniques.
Regression & Computer Vision Metrics: MAE, MSE, RMSE, R², IoU, Dice & mAP
A practical guide to continuous error metrics (MAE, MSE, RMSE, R²) and computer vision spatial metrics (Intersection over Union, Dice Coefficient, mAP).
Linear Models, Regularization (Ridge/Lasso/ElasticNet), KNN Distance & Naive Bayes
Explore linear hyperplanes, sigmoid logistic log-loss, L1 Lasso feature selection, L2 Ridge shrinkage, KNN distance metrics, and Naive Bayes independence assumptions.