Classification Problems and Common Algorithms Explained | 分类问题与常用算法解析

📚 Classification Problems and Common Algorithms Explained | 分类问题与常用算法解析

Classification is a core task in machine learning where an algorithm learns to assign input data to predefined categories. It is used in applications ranging from spam detection to medical diagnosis, and understanding its core methods is essential for any computer science student.

分类是机器学习中的核心任务，算法学习将输入数据分配到预定义类别中。从垃圾邮件检测到医疗诊断，分类应用广泛，理解其核心方法对任何计算机科学学生都至关重要。

1. What Is Classification? | 什么是分类？

Classification is a supervised learning technique. The model is trained on a labeled dataset, where each example has a known class label. The goal is to learn a mapping from input features to output categories, allowing predictions on unseen data.

分类是一种监督学习技术。模型在带有标签的数据集上训练，每个样本都有已知的类别标签。目标是学习从输入特征到输出类别的映射，从而对未见数据进行预测。

Formally, given input features \(X\) and target labels \(Y\), the classifier learns a function \(f: X \to Y\). Common examples include binary classification (two classes) and multiclass classification (more than two classes).

形式化地说，给定输入特征 \(X\) 和目标标签 \(Y\)，分类器学习一个函数 \(f: X \to Y\)。常见示例包括二分类（两个类别）和多分类（两个以上类别）。

2. Types of Classification Problems | 分类问题的类型

Binary classification involves only two possible outcomes, such as yes/no, true/false, or spam/not spam. Multiclass classification assigns one label from a set of three or more categories, such as classifying handwritten digits from 0 to 9.

二分类只涉及两种可能结果，例如是/否、真/假或垃圾/非垃圾。多分类从三个或更多类别中分配一个标签，例如对手写数字 0 到 9 进行分类。

Multilabel classification is different: each instance can belong to multiple classes simultaneously. For example, a document might be tagged with both “politics” and “economics”. In contrast, standard classification assumes each instance has exactly one label.

多标签分类不同：每个实例可以同时属于多个类别。例如，一篇文章可以同时标记为“政治”和“经济”。相比之下，标准分类假设每个实例只有一个标签。

3. Training and Testing | 训练与测试

In supervised classification, the labeled dataset is usually split into a training set and a test set. The training set is used to fit the model, while the test set evaluates its performance on new data. A common split ratio is 70% training and 30% testing.

在监督分类中，带标签的数据集通常分为训练集和测试集。训练集用于拟合模型，测试集用于评估模型在新数据上的表现。常见的分割比例为 70% 训练、30% 测试。

Cross-validation is often used to reduce variance. In k-fold cross-validation, the data is divided into k folds; the model is trained on k-1 folds and validated on the remaining fold, repeating this process k times. This gives a more reliable estimate of generalization performance.

交叉验证常用于减少方差。在 k 折交叉验证中，数据被分成 k 份；模型在 k-1 份上训练，在剩余一份上验证，重复 k 次。这提供了对泛化性能更可靠的估计。

4. Decision Trees | 决策树

A decision tree is a flowchart-like structure where each internal node tests an attribute, each branch represents an outcome, and each leaf node holds a class label. It is intuitive and easy to interpret, making it a popular baseline algorithm.

决策树是一种类似流程图的结构，其中每个内部节点测试一个属性，每条分支代表一个结果，每个叶节点保存一个类别标签。它直观且易于解释，是常用的基线算法。

Decision trees are built using attribute selection measures such as information gain or Gini impurity. Information gain is based on entropy, which measures the impurity of a set of samples. The attribute with the highest information gain is chosen as the splitting attribute.

决策树使用属性选择度量构建，如信息增益或基尼不纯度。信息增益基于熵，熵衡量样本集合的不纯度。选择具有最高信息增益的属性作为分裂属性。

Advantages: simple to understand, requires little data preprocessing, handles both numerical and categorical data.
优点：易于理解，几乎不需要数据预处理，能处理数值型和类别型数据。
Disadvantages: prone to overfitting, can be unstable with small changes in data.
缺点：容易过拟合，数据微小变化可能导致树结构不稳定。

5. k-Nearest Neighbors (k-NN) | k 近邻算法

k-Nearest Neighbors is a lazy learning algorithm that stores the entire training dataset and makes predictions based on the distance between a new sample and all training samples. The class of the sample is determined by a majority vote among its k nearest neighbors.

k 近邻算法是一种惰性学习算法，它存储整个训练数据集，并根据新样本与所有训练样本之间的距离进行预测。样本的类别由其 k 个最近邻居的多数投票决定。

Distance metrics commonly used include Euclidean distance and Manhattan distance. For continuous features, Euclidean distance is the standard choice. The parameter k is critical: a small k leads to high variance, while a large k may oversmooth the decision boundary.

常用的距离度量包括欧氏距离和曼哈顿距离。对于连续特征，欧氏距离是标准选择。参数 k 至关重要：k 过小导致高方差，k 过大可能使决策边界过于平滑。

Formula for Euclidean distance between two points p and q with n features:

两个具有 n 个特征的点 p 和 q 之间的欧氏距离公式：

d(p, q) = √( Σᵢ₌₁ⁿ (pᵢ – qᵢ)² )

Advantages: simple to implement, effective for small datasets, no training phase.
优点：实现简单，在小型数据集上有效，无需训练阶段。
Disadvantages: slow at prediction time, sensitive to irrelevant features, needs feature scaling.
缺点：预测时速度慢，对无关特征敏感，需要特征缩放。

6. Logistic Regression | 逻辑回归

Despite its name, logistic regression is a classification algorithm, not regression. It models the probability that an instance belongs to a particular class using a logistic (sigmoid) function.

尽管名字叫回归，逻辑回归是一种分类算法，而不是回归算法。它使用逻辑（Sigmoid）函数建模实例属于某个特定类别的概率。

The sigmoid function maps any real-valued input to a value between 0 and 1. If the output probability is greater than or equal to 0.5, the instance is classified as the positive class; otherwise, it is classified as the negative class.

Sigmoid 函数将任意实数输入映射到 0 到 1 之间的值。如果输出概率大于或等于 0.5，则将该实例分类为正类；否则分类为负类。

σ(z) = 1 / (1 + e⁻ᶻ), where z = w·x + b

Logistic regression is a linear classifier, meaning the decision boundary is a hyperplane. It is widely used because it is fast, requires little computational resources, and provides calibrated probabilities.

逻辑回归是一种线性分类器，意味着决策边界是一个超平面。它被广泛使用，因为速度快、计算资源需求少，并提供校准的概率输出。

7. Support Vector Machines (SVM) | 支持向量机

Support Vector Machine is a powerful classification algorithm that finds the optimal hyperplane that maximizes the margin between different classes. The margin is the distance between the hyperplane and the nearest support vectors from each class.

支持向量机是一种强大的分类算法，它找到最大化不同类别之间间隔的最优超平面。间隔是超平面与每个类别最近的支持向量之间的距离。

For non-linearly separable data, SVM uses the kernel trick. A kernel function maps the original feature space into a higher-dimensional space where a linear separator can be found. Common kernels include polynomial, radial basis function (RBF), and sigmoid.

对于非线性可分数据，SVM 使用核技巧。核函数将原始特征空间映射到更高维空间，从而可以找到线性分隔超平面。常用核包括多项式核、径向基函数（RBF）核和 Sigmoid 核。

Advantages: effective in high-dimensional spaces, memory efficient because only support vectors are stored.
优点：在高维空间中有效，内存效率高，因为只存储支持向量。
Disadvantages: not directly suitable for large datasets, may perform poorly with overlapping classes.
缺点：不太适合大规模数据集，在类别重叠时可能表现不佳。

8. Naïve Bayes Classifier | 朴素贝叶斯分类器

Naïve Bayes is a probabilistic classifier based on Bayes’ theorem, with the “naïve” assumption that features are conditionally independent given the class label. This simplification makes the model computationally efficient even for high-dimensional data.

朴素贝叶斯是一种基于贝叶斯定理的概率分类器，其“朴素”假设是给定类别标签时特征之间条件独立。这种简化使模型即使在高维数据上也能高效计算。

Bayes’ theorem is expressed as:

贝叶斯定理表示为：

P(C|X) = P(X|C) × P(C) / P(X)

In practice, the denominator P(X) is constant for a given instance, so classification is based on maximizing P(X|C) × P(C). Different variants exist, including Gaussian Naïve Bayes, Multinomial Naïve Bayes, and Bernoulli Naïve Bayes, depending on the type of features.

在实际计算中，分母 P(X) 对给定实例是常数，因此分类基于最大化 P(X|C) × P(C)。根据特征类型存在不同变体，包括高斯朴素贝叶斯、多项式朴素贝叶斯和伯努利朴素贝叶斯。

9. Ensemble Methods: Random Forest | 集成方法：随机森林

Random Forest is an ensemble learning method that constructs many decision trees during training and outputs the class that is the mode of the classes of individual trees. It reduces overfitting by averaging the results of diverse trees built on random subsets of the data and features.

随机森林是一种集成学习方法，训练时构建多棵决策树，并输出各棵树类别的众数作为最终结果。它通过在数据和特征的随机子集上构建多样化的树并取平均结果来减少过拟合。

Each tree in a random forest is trained on a bootstrap sample (sampling with replacement). Additionally, at each split, only a random subset of features is considered. This double randomness ensures that trees are decorrelated, improving robustness.

随机森林中的每棵树都在自助样本（有放回抽样）上训练。此外，每次分裂时只考虑特征的随机子集。这种双重随机性确保树之间相关性低，从而提高鲁棒性。

Advantages: high accuracy, handles large datasets, provides feature importance scores.
优点：准确率高，能处理大规模数据集，提供特征重要性分数。
Disadvantages: less interpretable than a single tree, more computationally expensive.
缺点：比单棵决策树更难解释，计算开销更大。

10. Model Evaluation Metrics | 模型评估指标

Accuracy is the simplest metric: the ratio of correctly predicted instances to the total number of instances. However, it is not reliable for imbalanced datasets. Precision, recall, and F1-score provide more detailed insight.

准确率是最简单的指标：正确预测的样本数与总样本数的比值。然而，在类别不平衡的数据集上它并不可靠。精确率、召回率和 F1 分数提供了更详细的洞察。

Precision measures the proportion of positive predictions that are actually correct. Recall measures the proportion of actual positive instances that are correctly identified. The F1-score is the harmonic mean of precision and recall.

精确率衡量被预测为正类的样本中实际为正类的比例。召回率衡量实际正类样本中被正确识别的比例。F1 分数是精确率和召回率的调和平均数。

Metric	Formula	Use Case
Accuracy	(TP+TN)/(TP+TN+FP+FN)	Balanced classes
Precision	TP/(TP+FP)	Spam detection
Recall	TP/(TP+FN)	Medical screening
F1-score	2×P×R/(P+R)	Imbalanced data

The confusion matrix lists true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). It forms the basis for all these metrics and helps diagnose the types of errors a classifier makes.

混淆矩阵列出真正例（TP）、真负例（TN）、假正例（FP）和假负例（FN）。它是所有上述指标的基础，有助于诊断分类器产生的错误类型。

11. Overfitting and Underfitting | 过拟合与欠拟合

Overfitting occurs when a model learns the training data too well, including its noise, and fails to generalize to new data. Symptoms include high training accuracy but low test accuracy. Techniques such as pruning (for decision trees), regularization, and cross-validation help mitigate overfitting.

过拟合发生在模型对训练数据学习得过于完美，包括其中的噪声，从而无法泛化到新数据。症状包括训练准确率高但测试准确率低。剪枝（针对决策树）、正则化和交叉验证等技术有助于缓解过拟合。

Underfitting happens when a model is too simple to capture the underlying structure of the data. It results in poor performance on both training and test sets. Increasing model complexity or adding more relevant features are common remedies.

欠拟合发生在模型过于简单，无法捕捉数据的潜在结构时。它导致训练集和测试集上的表现都差。增加模型复杂度或添加更多相关特征是常见的修正方法。

A strong classifier should achieve a balance between bias and variance. Bias is the error due to overly simplistic assumptions, while variance is the error due to sensitivity to small fluctuations in the training set.

强分类器应在偏差和方差之间取得平衡。偏差是由于过于简单的假设引起的误差，而方差是由于对训练集微小波动的敏感性引起的误差。

12. Choosing the Right Algorithm | 选择合适的算法

There is no single best classifier for all problems. The choice depends on dataset size, number of features, data type, interpretability needs, and computational resources. Linear models like logistic regression work well for linearly separable data, while tree-based methods handle non-linear interactions better.

不存在对所有问题都最好的单一分类器。选择取决于数据集大小、特征数量、数据类型、可解释性需求和计算资源。线性模型如逻辑回归在数据线性可分时表现良好，而基于树的方法能更好地处理非线性交互。

A practical approach is to start with a simple baseline like logistic regression or a decision tree, then try more complex models like SVM or random forest. Use cross-validation to compare their performance on validation data and choose the model with the best trade-off between accuracy, speed, and explainability.

实用方法是从简单基线开始，如逻辑回归或决策树，然后尝试更复杂的模型，如 SVM 或随机森林。使用交叉验证比较它们在验证数据上的表现，选择在准确率、速度和可解释性之间最佳平衡的模型。

Published by TutorHao | Computer Science Revision Series | aleveler.com

更多咨询请联系16621398022（同微信）

Classification Problems and Common Algorithms Explained | 分类问题与常用算法解析