Tree-Based Architectures & Ensembles: Decision Trees, Random Forests, XGBoost, LightGBM & CatBoost
Impurity splitting criteria (Gini, Entropy), bagging variance reduction, and sequential gradient boosting algorithms.
⚡ Executive Summary
Master decision tree split criteria (Gini vs Entropy), Random Forest bagging, and gradient boosted powerhouses like XGBoost, LightGBM, and CatBoost.
Key Takeaways
- ✓Decision Trees — covered in depth with practical examples, formulas, and code.
- ✓Bagging: Random Forest — covered in depth with practical examples, formulas, and code.
- ✓Gradient Boosting Frameworks — covered in depth with practical examples, formulas, and code.
Tree-Based Models & Ensemble Frameworks
#1. Decision Trees
Splits feature space recursively based on split criteria:
Key hyperparameters: max_depth, min_samples_split, min_samples_leaf.
#2. Bagging: Random Forest
Ensemble of independent decision trees trained on bootstrap samples with random feature selection. Reduces model variance without increasing bias.
#3. Gradient Boosting Frameworks
Sequential trees fitting negative gradients of the loss function:
| Model | Split Strategy | Key Strength |
|---|---|---|
| XGBoost | Pre-sorted exact / approx splits | Robust, accurate, handles missing data |
| LightGBM | Histogram leaf-wise (best-first) | 10x faster training on massive datasets |
| CatBoost | Symmetric balanced trees | Superior out-of-the-box categorical feature handling |
import xgboost as xgb
model = xgb.XGBClassifier(n_estimators=200, learning_rate=0.05, max_depth=6).fit(X_train, y_train)Govindarajan Selvaraj
ML Engineer
Govindarajan Selvaraj is part of the Junglans Solutions engineering team, specializing in model architectures. Junglans builds a 20-product ecosystem of local-first enterprise software — AI developer tools, encrypted communication, and data infrastructure with zero cloud telemetry.
Meet the full Junglans engineering team ↗This article is part of the Junglans Research knowledge base, produced alongside the engineering teams that build our production AI tools. Explore related product documentation and research:
- Junglans ML Visualizer ↗
Interactive 22-algorithm machine learning sandbox — see the concepts in action.
- JunglasNCode ↗
Line-by-line code execution and call stack visualizer for algorithm practice.
- All Junglans Research Articles ↗
More engineering and AI deep-dives from the Junglans team.
Related Research & Articles
ROC-AUC Curves, PR-AUC, and Multi-Class Evaluation Metrics
Explore threshold-agnostic ROC curves, probability ranking, Precision-Recall AUC for rare classes, and multi-class macro, micro, and weighted averaging techniques.
Regression & Computer Vision Metrics: MAE, MSE, RMSE, R², IoU, Dice & mAP
A practical guide to continuous error metrics (MAE, MSE, RMSE, R²) and computer vision spatial metrics (Intersection over Union, Dice Coefficient, mAP).
Linear Models, Regularization (Ridge/Lasso/ElasticNet), KNN Distance & Naive Bayes
Explore linear hyperplanes, sigmoid logistic log-loss, L1 Lasso feature selection, L2 Ridge shrinkage, KNN distance metrics, and Naive Bayes independence assumptions.