Model Architectures9 min read• Published July 29, 2026

Linear Models, Regularization (Ridge/Lasso/ElasticNet), KNN Distance & Naive Bayes

Foundational machine learning models: OLS linear regression, logistic classification, L1/L2 penalties, KNN lazy learning, and Naive Bayes.

GS
Govindarajan Selvaraj
ML Engineer

⚡ Executive Summary

Explore linear hyperplanes, sigmoid logistic log-loss, L1 Lasso feature selection, L2 Ridge shrinkage, KNN distance metrics, and Naive Bayes independence assumptions.

Key Takeaways

  • ✓Linear Models — covered in depth with practical examples, formulas, and code.
  • ✓Distance-Based: KNN — covered in depth with practical examples, formulas, and code.
  • ✓Probabilistic: Naive Bayes — covered in depth with practical examples, formulas, and code.

Classical Machine Learning — Linear, Distance & Probabilistic Models

text
ML Model Families
├── Linear Models          → Linear Regression, Logistic Regression, Ridge, Lasso, ElasticNet
├── Distance-Based        → K-Nearest Neighbors (KNN)
└── Probabilistic          → Naive Bayes (Gaussian, Multinomial, Bernoulli)

#1. Linear Models

1.1 Linear Regression

MATHEMATICAL FORMULA ŷ = w₁ x₁ + w₂ x₂ + ... + wₙ xₙ + b

Learns weights by minimizing MSE via Normal Equation or Gradient Descent.

1.2 Logistic Regression

MATHEMATICAL FORMULA P(y=1) = (1) / (1 + e^-(w · x + b))

Classification model minimizing cross-entropy log-loss.

1.3 Regularization

  • Ridge (L2): Loss = MSE + α ∑ wᵢ² (shrinks weights smoothly).
  • Lasso (L1): Loss = MSE + α ∑ |wᵢ| (forces uninformative weights to exact zero).
  • ElasticNet: Combines L1 and L2 penalties.

    • #2. Distance-Based: KNN

    Lazy learner predicting majority class or average of K closest training neighbors. Requires feature standardization.

    #3. Probabilistic: Naive Bayes

    Applies Bayes' Theorem assuming feature conditional independence:

    MATHEMATICAL FORMULA P(class mid features) ∝ P(class) × ∏ P(featureᵢ mid class)

    Variants: GaussianNB, MultinomialNB, BernoulliNB. Extremely fast text baseline.

    Tags:#Linear Models#Logistic Regression#Ridge#Lasso#KNN#Naive Bayes
    ABOUT THE AUTHOR
    GS

    Govindarajan Selvaraj

    ML Engineer

    Govindarajan Selvaraj is part of the Junglans Solutions engineering team, specializing in model architectures. Junglans builds a 20-product ecosystem of local-first enterprise software — AI developer tools, encrypted communication, and data infrastructure with zero cloud telemetry.

    Meet the full Junglans engineering team ↗
    RELATED RESOURCES & REFERENCES

    This article is part of the Junglans Research knowledge base, produced alongside the engineering teams that build our production AI tools. Explore related product documentation and research:

    Related Research & Articles

    Machine Learning Foundations

    ROC-AUC Curves, PR-AUC, and Multi-Class Evaluation Metrics

    Explore threshold-agnostic ROC curves, probability ranking, Precision-Recall AUC for rare classes, and multi-class macro, micro, and weighted averaging techniques.

    Read Article ↗
    Machine Learning Foundations

    Regression & Computer Vision Metrics: MAE, MSE, RMSE, R², IoU, Dice & mAP

    A practical guide to continuous error metrics (MAE, MSE, RMSE, R²) and computer vision spatial metrics (Intersection over Union, Dice Coefficient, mAP).

    Read Article ↗
    Model Architectures

    Tree-Based Architectures & Ensembles: Decision Trees, Random Forests, XGBoost, LightGBM & CatBoost

    Master decision tree split criteria (Gini vs Entropy), Random Forest bagging, and gradient boosted powerhouses like XGBoost, LightGBM, and CatBoost.

    Read Article ↗