Voting Classifier

Hard Voting and Soft Voting Ensembles

1. What Is a Voting Classifier?

A Voting Classifier combines the predictions of multiple heterogeneous classifiers — classifiers of different types (logistic regression, SVM, random forest, gradient boosting, etc.) — by aggregating their outputs through voting.

The key distinction from Bagging/Random Forest: Voting classifiers combine different algorithms, each trained on the same full dataset, while bagging combines many instances of the same algorithm trained on different bootstrap samples.

2. Hard Voting — Majority Rule

2.1 The Mechanics

Each classifier votes for one class. The class receiving the most votes wins:

Hard vote:  ŷ = argmax_c  Σₜ 𝟙[ĥₜ(x) = c]   (count votes for each class)

Example (K=2, B=5 classifiers):
  Classifier 1: class A
  Classifier 2: class A
  Classifier 3: class B
  Classifier 4: class A
  Classifier 5: class B

  Votes: A=3, B=2
  Prediction: A  (majority)

For binary classification, ties are broken by the first class (sklearn behavior).

When hard voting fails: If all classifiers make the same mistake — which is likely when they are highly correlated — the majority vote fails just as spectacularly as any single classifier. Hard voting's error rate is only better than individual classifiers when errors are uncorrelated.

2.2 Condorcet's Jury Theorem

The theoretical justification for hard voting comes from Condorcet's Jury Theorem (1785) — a result from political philosophy applied to ensemble learning.

Theorem: If each of B voters independently makes the correct decision with probability p > 0.5, then the probability that the majority makes the correct decision approaches 1 as B → ∞.

For an ensemble of B independent classifiers each with accuracy p:

P(majority correct) = Σ_{k=⌈B/2⌉}^{B} C(B,k) · p^k · (1-p)^{B-k}

Examples:

The theorem requires:

1. p > 0.5 — each classifier must be better than random
2. Independence — classifiers must make independent errors

Condition 2 is the hard part. Classifiers trained on the same data are not independent — they tend to fail on the same hard examples.

2.3 Mathematical Bound on Hard Vote Error

For B classifiers with pairwise error correlation ρ and individual error rate ε:

E[Majority vote error] ≤ ε²·B·ρ + ε·(1 − ε)·(1 − ρ)·B / (B-1)

As B → ∞ (many diverse classifiers):

Majority error → ε · ρ / (1 − ε + ε·ρ)  ≈ ε · ρ  (for small ε)

The error rate of the majority vote is approximately ε·ρ — the product of the individual error rate and the correlation. If ρ = 0.1 (nearly independent classifiers with ε = 0.3), the ensemble error approaches ~0.03 — 10× better.

If ρ = 1.0 (all classifiers make identical errors), the ensemble error is exactly ε — no improvement from voting.

3. Soft Voting — Probability Averaging

3.1 The Mechanics

Each classifier outputs a probability vector over K classes. The ensemble averages these probability vectors and predicts the class with the highest average probability:

P̂(y=c | x) = (1/B) Σₜ P̂_t(y=c | x)
ŷ = argmax_c P̂(y=c | x)

With weights:

P̂(y=c | x) = Σₜ wₜ · P̂_t(y=c | x)  /  Σₜ wₜ

Example (K=3 classes, B=3 classifiers):

Classifier 1: P̂ = [0.7, 0.2, 0.1]  → hard vote: class A (confident)
Classifier 2: P̂ = [0.4, 0.35, 0.25] → hard vote: class A (barely)
Classifier 3: P̂ = [0.3, 0.5, 0.2]  → hard vote: class B (moderate)

Hard voting result: A (2 votes) vs B (1 vote) → Predict A

Soft voting:
  Average: [(0.7+0.4+0.3)/3, (0.2+0.35+0.5)/3, (0.1+0.25+0.2)/3]
           = [0.467, 0.35, 0.183]
  → Predict A  (same outcome here, but with more information used)

Case where they differ:
Classifier 1: P̂ = [0.51, 0.49, 0.0]  → hard vote: A (barely)
Classifier 2: P̂ = [0.49, 0.51, 0.0]  → hard vote: B (barely)
Classifier 3: P̂ = [0.49, 0.51, 0.0]  → hard vote: B (barely)

Hard voting: B (2 votes) wins
Soft voting: [(0.51+0.49+0.49)/3, (0.49+0.51+0.51)/3] = [0.497, 0.503]
             → B still wins, but notice: all three are very uncertain — soft voting
                preserves this uncertainty in the output probability

3.2 Why Soft Voting Almost Always Outperforms Hard Voting

Hard voting discards all probability information and converts each classifier's output to a binary vote. This throws away:

• Confidence information: A classifier that predicts "class A with 0.99 probability" counts the same as one that predicts "class A with 0.51 probability"
• Near-miss information: A classifier that nearly voted B but voted A instead provides a signal that soft voting uses but hard voting ignores

Formal argument: Let f₁*, f₂*, ..., fₙ* be the optimal decision functions for B classifiers. The soft vote:

F_soft(x) = (1/B) Σₜ fₜ*(x)

Has minimum Bayes risk among all linear combinations of {fₜ*} — hard voting is a suboptimal nonlinear transformation of the same information.

The information loss from hard voting:

Hard: h_t(x) = argmax_c P̂_t(y=c|x)  → 1 bit per classifier
Soft: P̂_t(y=c|x)                     → K-1 real numbers per classifier

Soft voting uses K-1 times more information per classifier. Whenever classifiers are uncertain (probabilities spread across classes), this additional information is valuable.

Empirical rule: Soft voting consistently outperforms hard voting on multi-class problems. For binary problems, the gap is smaller but still present. The gap is largest when classifiers have similar accuracy but different confidence profiles.

3.3 Calibration Requirement

Soft voting requires that classifier probability estimates be calibrated — the stated probability 0.7 should actually mean "70% of the time this is the correct class."

Why calibration matters for soft voting:

If Classifier A always outputs probabilities near 0 or 1 (overconfident) and Classifier B outputs moderate probabilities (well-calibrated), simple averaging will give A's predictions more influence than warranted.

Overconfident: P̂_A = [0.98, 0.02] → dominates the average
Well-calibrated: P̂_B = [0.55, 0.45] → modest contribution

Average: [(0.98+0.55)/2, (0.02+0.45)/2] = [0.765, 0.235]
→ A's overconfidence distorts the ensemble

Calibration check:

from sklearn.calibration import calibration_curve
import matplotlib.pyplot as plt

for clf, name in base_classifiers:
    proba = clf.predict_proba(X_val)[:, 1]
    frac_pos, mean_pred = calibration_curve(y_val, proba, n_bins=10)
    plt.plot(mean_pred, frac_pos, label=name)

plt.plot([0,1],[0,1],'k--', label='Perfect')
plt.legend(); plt.title('Calibration of Base Classifiers')

When to calibrate:

• SVMs: Always calibrate (Platt scaling) — SVM scores are not probabilities by default
• Random Forest: Usually well-calibrated, but check for imbalanced datasets
• Gradient Boosting: Usually well-calibrated for log-loss objective
• Naive Bayes: Often overconfident — calibrate with isotonic regression
• Neural networks: Often overconfident — calibrate with temperature scaling

4. Weighted Voting

4.1 Fixed Weights

Assign higher weight to more accurate classifiers:

from sklearn.ensemble import VotingClassifier

# Weight classifiers by their validation accuracy
clf_a = LogisticRegression()        # val_acc = 0.82 → weight = 2
clf_b = RandomForestClassifier()    # val_acc = 0.85 → weight = 3
clf_c = GradientBoostingClassifier()# val_acc = 0.87 → weight = 4

voting = VotingClassifier(
    estimators=[('lr', clf_a), ('rf', clf_b), ('gbt', clf_c)],
    voting='soft',
    weights=[2, 3, 4]
)

Weights are applied to probability averaging in soft voting:

P̂(y=c|x) = (Σₜ wₜ · P̂_t(y=c|x)) / Σₜ wₜ

How to set weights: Proportional to validation accuracy, log-odds accuracy, or AUC. Don't use training accuracy (overfit).

4.2 Optimal Weights via Optimization

For soft voting, the optimal weights minimize a loss function on validation data:

from scipy.optimize import minimize
import numpy as np

# Get probability predictions from each classifier
proba_preds = [clf.predict_proba(X_val) for clf in base_clfs]
# Each proba_preds[t] has shape (n_val, K)

def ensemble_loss(weights):
    weights = np.array(weights)
    weights = np.maximum(weights, 0)        # Non-negative
    weights /= weights.sum()                 # Normalize
    avg_proba = sum(w * p for w, p in zip(weights, proba_preds))
    # Cross-entropy loss
    loss = -np.mean(np.log(avg_proba[np.arange(len(y_val)), y_val] + 1e-10))
    return loss

result = minimize(
    ensemble_loss,
    x0=np.ones(len(base_clfs)) / len(base_clfs),  # Start with uniform weights
    method='Nelder-Mead',
    options={'xatol': 1e-5, 'fatol': 1e-5, 'maxiter': 1000}
)
optimal_weights = result.x / result.x.sum()

Caution: Optimize weights on a held-out validation set, not the training set. Optimizing on training data will overfit the weights.

5. Diversity — The Secret Ingredient

5.1 Why Diversity Matters

The fundamental theorem of voting ensembles: improvement is proportional to diversity.

Two classifiers that always agree produce the same error as either one individually — there is nothing to be gained from combining identical predictors. Two classifiers that disagree frequently (but each is individually accurate) produce a much better ensemble — their disagreements cancel out, leaving only their agreements (which are mostly correct).

Perfect correlation (ρ=1):   Ensemble error = Individual error
Zero correlation (ρ=0):      Ensemble error ≈ Individual error / B
Negative correlation (ρ<0):  Ensemble error < Individual error / B  (rare but possible)

5.2 Sources of Diversity

Algorithm diversity: The primary source in voting classifiers. Different algorithms have different inductive biases — they make mistakes in different places.

Logistic Regression:  Wrong on nonlinear boundaries
Random Forest:        Wrong on extrapolation, rare patterns
SVM (RBF):           Wrong at boundary edge cases in kernel space
Gradient Boosting:    Wrong on noisy examples (overfit tendency)
Naive Bayes:          Wrong when features are correlated

Each algorithm is wrong in a different way — the ensemble is only wrong where all are wrong simultaneously.

Hyperparameter diversity: Same algorithm, different settings:

# Multiple gradient boosting models with different depth
gb1 = GradientBoostingClassifier(max_depth=3, n_estimators=100)
gb2 = GradientBoostingClassifier(max_depth=5, n_estimators=200)
gb3 = GradientBoostingClassifier(max_depth=7, n_estimators=50)

Feature diversity: Different feature subsets for different classifiers:

# Classifier A uses features 0-4, Classifier B uses features 3-9
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer

clf_a = Pipeline([
    ('select', FunctionTransformer(lambda X: X[:, :5])),
    ('clf', RandomForestClassifier())
])

Training data diversity: Bootstrap samples (this is Bagging — a special case of voting with identical base algorithms and different data).

5.3 Measuring Diversity

Several metrics quantify ensemble diversity:

Q-statistic (for two classifiers):

Q = (N¹¹N⁰⁰ − N¹⁰N⁰¹) / (N¹¹N⁰⁰ + N¹⁰N⁰¹)

Where Nᵃᵇ = number of samples where classifier 1 predicts a and classifier 2 predicts b.

• Q near 0: classifiers are diverse
• Q near 1: classifiers agree often (similar, less diverse)
• Q near -1: classifiers complement each other (anti-correlated — ideal)

import numpy as np

def q_statistic(pred_a, pred_b, y_true):
    n11 = ((pred_a == y_true) & (pred_b == y_true)).sum()   # Both correct
    n00 = ((pred_a != y_true) & (pred_b != y_true)).sum()   # Both wrong
    n10 = ((pred_a == y_true) & (pred_b != y_true)).sum()   # A correct, B wrong
    n01 = ((pred_a != y_true) & (pred_b == y_true)).sum()   # A wrong, B correct
    return (n11*n00 - n10*n01) / (n11*n00 + n10*n01 + 1e-10)

# Lower Q → more diverse → better ensemble

Disagreement measure:

Disagreement(a, b) = P(ĥ_a(x) ≠ ĥ_b(x))

Higher disagreement → more diverse. Compute pairwise over all (B choose 2) classifier pairs:

from itertools import combinations

disagreements = []
for (i, clf_a), (j, clf_b) in combinations(enumerate(clfs), 2):
    pred_a = clf_a.predict(X_val)
    pred_b = clf_b.predict(X_val)
    disagreements.append((pred_a != pred_b).mean())

mean_disagreement = np.mean(disagreements)
print(f"Mean pairwise disagreement: {mean_disagreement:.3f}")
# Higher is better for ensemble potential

6. Building a Voting Ensemble — Strategy

Step 1: Identify diverse, individually strong classifiers

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.svm import SVC
from sklearn.naive_bayes import GaussianNB
from sklearn.model_selection import cross_val_score

candidates = [
    ('lr',   LogisticRegression(max_iter=1000)),
    ('rf',   RandomForestClassifier(n_estimators=300, n_jobs=-1)),
    ('gbt',  GradientBoostingClassifier(n_estimators=200)),
    ('svm',  SVC(probability=True)),
    ('nb',   GaussianNB()),
]

# Screen: select classifiers above a performance threshold
threshold = 0.80   # Min AUC
selected = []
for name, clf in candidates:
    score = cross_val_score(clf, X_train, y_train, cv=5, scoring='roc_auc').mean()
    print(f"{name}: AUC = {score:.4f}")
    if score >= threshold:
        selected.append((name, clf))

Step 2: Measure pairwise diversity

Use Q-statistic or disagreement on validation data. Avoid including classifiers that agree too much with existing ensemble members.

Step 3: Calibrate probabilities

from sklearn.calibration import CalibratedClassifierCV

calibrated_selected = [
    (name, CalibratedClassifierCV(clf, method='isotonic', cv=5))
    for name, clf in selected
]

Step 4: Optimize weights

Use the weight optimization from Section 4.2 on a held-out validation set.

Step 5: Evaluate on test set

Never use the test set until the final evaluation.

7. VotingClassifier in sklearn — Full API

from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.svm import SVC

estimators = [
    ('lr',  LogisticRegression(max_iter=1000)),
    ('rf',  RandomForestClassifier(n_estimators=300, n_jobs=-1)),
    ('gbt', GradientBoostingClassifier(n_estimators=200)),
    ('svm', SVC(probability=True))
]

# Hard voting
clf_hard = VotingClassifier(
    estimators=estimators,
    voting='hard',
    n_jobs=-1
)

# Soft voting
clf_soft = VotingClassifier(
    estimators=estimators,
    voting='soft',
    weights=[1, 2, 3, 1],    # Optional: weight each classifier
    n_jobs=-1
)

# Fit
clf_soft.fit(X_train, y_train)

# Predict
y_pred  = clf_soft.predict(X_test)
y_proba = clf_soft.predict_proba(X_test)   # Only available for soft voting

# Access individual classifiers
clf_soft.estimators_[0]   # Fitted LogisticRegression
clf_soft.named_estimators_['rf'].feature_importances_   # Access RF importance

Important: hard voting requires no predict_proba — useful when one of your classifiers doesn't output probabilities and you still want to ensemble it.

8. Multi-Class Voting

Both hard and soft voting work naturally for K > 2 classes:

Hard voting: Each classifier votes for one of K classes. Predict the class with the most votes. With B classifiers and K classes, majority vote requires only B/K + 1 votes (not B/2 + 1).

Soft voting: Average K-dimensional probability vectors. Requires each classifier to output probabilities for all K classes — this is the standard behavior of predict_proba in sklearn.

# Multi-class soft voting
from sklearn.datasets import load_iris
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier

X, y = load_iris(return_X_y=True)   # 3 classes

clf = VotingClassifier(
    estimators=[
        ('lr', LogisticRegression(multi_class='multinomial')),
        ('rf', RandomForestClassifier(n_estimators=100))
    ],
    voting='soft'
)
clf.fit(X_train, y_train)

# predict_proba returns (n_samples, K) probability matrix
y_proba = clf.predict_proba(X_test)

9. Probability Calibration for Soft Voting

Not all classifiers produce calibrated probabilities. Calibrate before using soft voting:

from sklearn.calibration import CalibratedClassifierCV
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier

# SVM: must calibrate (no native probabilities)
svm_calibrated = CalibratedClassifierCV(SVC(), method='platt', cv=5)

# Naive Bayes: often needs calibration (overconfident)
from sklearn.naive_bayes import GaussianNB
nb_calibrated = CalibratedClassifierCV(GaussianNB(), method='isotonic', cv=5)

# GBT with log-loss: usually calibrated, can still improve
gbt = GradientBoostingClassifier(loss='log_loss')

estimators = [
    ('svm', svm_calibrated),
    ('nb',  nb_calibrated),
    ('gbt', gbt)
]

clf = VotingClassifier(estimators, voting='soft')
clf.fit(X_train, y_train)

10. Hyperparameters — Complete Reference

from sklearn.ensemble import VotingClassifier

VotingClassifier(
    estimators,         # List of (name, estimator) tuples — required
    voting='hard',      # 'hard' (majority label) or 'soft' (avg probability)
    weights=None,       # List of weights for each classifier (default: equal)
    n_jobs=None,        # Parallel fitting (-1 = all cores)
    flatten_transform=True,  # If True, transform() flattens probability arrays
    verbose=False
)

No n_estimators: Unlike Bagging or Random Forest, there's no "number of trees" — you manually define each classifier in estimators.

Disabling classifiers:

# Temporarily disable one classifier without refitting
clf.set_params(lr='drop')   # Drop logistic regression from the ensemble

11. The Bias-Variance Profile

Voting classifiers have a distinctive profile:

If base classifiers are high-bias (underfitting):   Voting doesn't help — still high bias
If base classifiers are high-variance (overfitting): Soft voting reduces variance
If base classifiers are diverse and accurate:        Soft voting provides meaningful boost

The improvement from soft voting over the best individual classifier is approximately:

ΔError ≈ −(1/2) · mean_{j≠k} Cov(P̂_j(x), P̂_k(x)) / Var(P̂_avg(x))

Higher diversity (lower covariance between classifiers) → larger improvement.

Practical expectation: A well-constructed voting ensemble of 5 diverse classifiers typically improves AUC by 1–3% over the best individual classifier. This is less spectacular than XGBoost vs. Random Forest, but requires no additional training — it's free performance from combining existing models.

12. Assumptions

13. Advantages

✅ Exploits Algorithm Diversity

Different algorithm families make systematically different errors — combining them hedges against any single algorithm's failure modes.

✅ Simple to Implement

VotingClassifier in sklearn requires minimal configuration. No training beyond fitting each base classifier.

✅ Works with Any Combination of Classifiers

Can mix tree-based models, linear models, kernel methods, neural networks — no restriction on base learner types.

✅ Soft Voting Preserves Probability Information

The averaged probability vector is more informative than any individual prediction — useful for decision thresholds and downstream calibration.

✅ Hard Voting Doesn't Require predict_proba

If one classifier in the ensemble doesn't support probability output, hard voting is still possible.

✅ Parallel Training

All base classifiers are trained independently — n_jobs=-1 parallelizes fitting.

✅ Interpretable Ensemble Structure

Each component classifier can be examined individually — not a black box in the same way as stacking.

✅ Marginal Improvement at Low Cost

A 1–3% accuracy improvement from combining already-trained classifiers has near-zero marginal cost. High return on marginal effort.

14. Drawbacks & Limitations

❌ Modest Performance Improvement

A voting ensemble of 5 strong classifiers typically improves on the best single classifier by 1–3% AUC. Gradient boosting hyperparameter tuning often achieves more improvement.

❌ Requires Calibrated Probabilities for Soft Voting

If base classifiers are miscalibrated (especially SVMs), soft voting is distorted. Hard voting avoids this but sacrifices information.

❌ Error Correlation Problem

If all classifiers are trained on the same features and data, they will correlate significantly. The ensemble is only as good as its weakest-correlated pair.

❌ No Adaptation to Hard Examples

Unlike boosting (which focuses on hard examples) or stacking (which learns optimal combination), voting treats all examples equally. There's no mechanism to give more weight to classifiers that are better on the specific types of hard examples.

❌ Adding Weak Classifiers Hurts

Adding a classifier worse than the current ensemble's average can reduce performance (in soft voting, its poor probability estimates pollute the average). Hard voting is more robust — adding a weak classifier just adds noise.

❌ Memory: Stores All Base Classifiers

Each fitted classifier is stored in memory. For large models (big random forests, large neural networks), the memory footprint multiplies with the number of classifiers.

15. Voting vs. Stacking vs. Blending vs. Bagging

16. Practical Tips & Gotchas

The Golden Rule: Diversity Over Individual Accuracy

# WRONG: Two highly correlated models gain little from voting
clf_rf1 = RandomForestClassifier(n_estimators=100)
clf_rf2 = RandomForestClassifier(n_estimators=200)  # Very similar to rf1
voting_wrong = VotingClassifier([('rf1', clf_rf1), ('rf2', clf_rf2)], voting='soft')
# These two will agree ~90% of the time — barely better than either alone

# RIGHT: Diverse algorithms with different biases
clf_lr  = LogisticRegression()      # Linear boundary
clf_rf  = RandomForestClassifier()  # Non-linear, tree-based
clf_gbt = GradientBoostingClassifier()  # Non-linear, boosted
voting_right = VotingClassifier([('lr', clf_lr), ('rf', clf_rf), ('gbt', clf_gbt)])
# These three will disagree on meaningful cases — larger benefit

Choose Soft over Hard Voting (Almost Always)

# Compare both on validation set
for voting_type in ['hard', 'soft']:
    vc = VotingClassifier(estimators, voting=voting_type)
    score = cross_val_score(vc, X, y, cv=5, scoring='roc_auc').mean()
    print(f"{voting_type}: AUC = {score:.4f}")
# Soft almost always wins; only use hard if a classifier lacks predict_proba

Calibrate Before Soft Voting

from sklearn.calibration import CalibratedClassifierCV

calibrated_estimators = [
    (name, CalibratedClassifierCV(clf, method='isotonic', cv=5))
    for name, clf in estimators
]
voting = VotingClassifier(calibrated_estimators, voting='soft')

Full Pipeline

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Each classifier may need different preprocessing
lr_pipe  = Pipeline([('scaler', StandardScaler()), ('clf', LogisticRegression())])
svm_pipe = Pipeline([('scaler', StandardScaler()), ('clf', SVC(probability=True))])
rf       = RandomForestClassifier()  # No scaling needed

voting = VotingClassifier(
    estimators=[('lr', lr_pipe), ('svm', svm_pipe), ('rf', rf)],
    voting='soft',
    n_jobs=-1
)
voting.fit(X_train, y_train)

Adding Classifiers Incrementally

# Can't incrementally add to VotingClassifier directly
# Rebuild with additional classifier
from copy import deepcopy

existing = [('lr', lr_clf), ('rf', rf_clf)]
new_clf = GradientBoostingClassifier()
new_clf.fit(X_train, y_train)

extended = existing + [('gbt', new_clf)]
new_voting = VotingClassifier(extended, voting='soft')
new_voting.fit(X_train, y_train)  # Refits all classifiers

17. When to Use It

Use VotingClassifier when:

• You have multiple well-trained classifiers of different types and want free marginal improvement
• You're in a competition setting where 1–2% accuracy gain matters
• Interpretability is still required — each component can be examined individually
• Hard voting is needed because one classifier lacks predict_proba
• You want a simple, transparent ensemble without the complexity of stacking

Use Stacking instead when:

• You want to learn the optimal combination rather than use a fixed rule
• Higher accuracy improvement justifies the added complexity and CV infrastructure

Use Bagging/Random Forest instead when:

• All base learners are the same algorithm (especially trees)
• You want free OOB evaluation and feature importance

Do NOT use VotingClassifier when:

• All base classifiers are nearly identical (same algorithm, similar hyperparameters) — no benefit from voting
• Base classifiers are severely miscalibrated and you can't fix this — soft voting will be distorted

Summary

┌──────────────────────────────────────────────────────────────────────┐
│               VOTING CLASSIFIER AT A GLANCE                         │
├──────────────────────────────────────────────────────────────────────┤
│  HARD VOTE    Majority label — 1 vote per classifier                │
│  SOFT VOTE    Average probabilities — uses confidence info          │
│  DIVERSITY    Improvement ∝ diversity (lower ρ → bigger gain)       │
│  CONDORCET    If p>0.5 and independent, majority error → 0          │
│  CALIBRATION  Soft voting requires calibrated probabilities         │
│  WEIGHTS      Can weight by accuracy; optimal via minimize(loss)    │
│  STRENGTH     Simple, diverse, transparent, any classifier type     │
│  WEAKNESS     Modest gain, calibration sensitive, error correlation  │
│  HARD vs SOFT Soft wins unless classifiers lack predict_proba       │
│  BEST FOR     Combining existing diverse classifiers, competitions  │
└──────────────────────────────────────────────────────────────────────┘

The Voting Classifier is the ensemble version of "ask a diverse panel of experts." It requires no sophisticated machinery — just collect strong, diverse predictors and let their disagreements cancel. Condorcet showed in 1785 that independent voters above 50% accuracy produce a perfect majority decision as their number grows. The challenge in 2024 is the same as in 1785: independence. Classifiers trained on the same data are not independent — they share correlated errors. The skill in building a voting ensemble is the skill of engineering diversity: choosing algorithms with fundamentally different inductive biases, calibrating their probabilities to the same scale, and combining them in a way that lets their disagreements work in your favor.