📚 Character Encoding Principles and Common Encoding Formats | 字符编码原理与常用编码格式
Character encoding is the process of representing characters, symbols, and control codes as numeric values that a computer can store and process. Every text file, webpage, or database record depends on a consistent encoding system to display meaningful information.
字符编码是将字符、符号和控制代码表示为计算机可以存储和处理的数值的过程。每一个文本文件、网页或数据库记录都依赖于一致的编码系统才能显示有意义的信息。
1. What Is Character Encoding? | 什么是字符编码？
A character encoding scheme defines a mapping between a set of characters and their binary representations. The basic unit of this mapping is a code point, which is an integer assigned to a character.
字符编码方案定义了字符集合与其二进制表示之间的映射关系。这种映射的基本单位是码点，即分配给某个字符的整数。
For example, in the ASCII scheme, the uppercase letter ‘A’ is assigned the decimal value 65, which is stored as binary 01000001. Without this shared mapping, two devices would interpret the same byte differently.
例如，在ASCII方案中，大写字母’A’被分配十进制值65，以二进制01000001存储。如果没有这种共享映射，两个设备会对同一个字节作出不同解释。
2. Binary and Character Mapping | 二进制与字符映射
Computers ultimately work only with bits: 0 and 1. A single byte consists of 8 bits, giving 256 possible values (2⁸). Early encodings therefore used one byte per character, allowing each byte to represent one symbol.
计算机最终只处理位：0和1。一个字节由8位组成，共有256种可能值（2⁸）。因此早期编码采用每字符一个字节，每个字节可以表示一个符号。
The mapping can be represented as a table. For instance, ‘0’ maps to decimal 48, ‘A’ to 65, and ‘a’ to 97. These values are not arbitrary: they were chosen to simplify sorting and case conversion.
这种映射可以用表格表示。例如，’0’对应十进制48，’A’对65，’a’对97。这些值并非任意选择，而是为了简化排序和大小写转换。
‘A’ → 65 → 01000001
Understanding this direct byte-to-character relationship is essential for explaining how later encoding systems evolved.
理解这种直接的字节到字符关系，是解释后来编码系统如何演进的关键。
3. ASCII: American Standard Code for Information Interchange | ASCII：美国信息交换标准代码
ASCII was developed in the 1960s and uses 7 bits to represent 128 characters. It includes uppercase and lowercase English letters, digits 0–9, punctuation marks, and 33 non-printable control characters such as newline and carriage return.
ASCII开发于20世纪60年代，使用7位表示128个字符。它包括英文大小写字母、数字0–9、标点符号，以及33个不可打印控制字符，例如换行和回车。
- Printable characters: 95
- Control characters: 33
- Storage size: 1 byte (with leading 0)
- 可打印字符：95个
- 控制字符：33个
- 存储大小：1字节（最高位为0）
Because only 7 bits are needed, the most significant bit of an ASCII byte is always 0. This was later exploited for extended character sets.
由于只需要7位，ASCII字节的最高位总是0。这一点后来被用于扩展字符集。
4. Extended ASCII and Its Limitations | 扩展ASCII及其局限性
To represent more characters, extended ASCII uses all 8 bits of a byte, allowing 128 additional code points (128–255). Different countries used these extra positions for accented letters, mathematical symbols, or box-drawing characters.
为了表示更多字符，扩展ASCII使用一个字节的全部8位，额外允许128个码点（128–255）。不同国家使用这些额外位置表示重音字母、数学符号或制表符。
However, there was no universal agreement on what values 128–255 should mean. For example, code point 233 represented ‘é’ in the Latin-1 encoding but a different character in the Cyrillic encoding. This caused the “mojibake” problem when files moved between systems.
然而，对于128–255的值应该表示什么，没有统一标准。例如，码点233在Latin-1编码中表示’é’，但在西里尔编码中却表示不同字符。这导致文件在不同系统间传输时出现“乱码”问题。
The fundamental limitation of extended ASCII is that it can never cover all the world’s writing systems. Even a single byte cannot hold enough information for Chinese, Arabic, and emoji simultaneously.
扩展ASCII的根本局限在于它永远不能覆盖世界上所有的书写系统。单独一个字节无法同时容纳中文、阿拉伯文和表情符号所需的信息。
5. Unicode: A Universal Character Set | Unicode：通用字符集
Unicode solves the fragmentation by assigning every character a unique code point, regardless of platform, program, or language. The Unicode standard currently defines over 1.1 million code points, covering 150+ scripts.
Unicode通过为每个字符分配唯一的码点来解决碎片化问题，无论平台、程序或语言如何。Unicode标准目前定义了超过110万个码点，涵盖150多种文字。
Unicode code points are usually written in hexadecimal with the prefix “U+”. For instance, ‘A’ is U+0041, ‘中’ is U+4E2D, and ‘😀’ is U+1F600.
Unicode码点通常用十六进制加前缀“U+”书写。例如，’A’是U+0041，’中’是U+4E2D，’😀’是U+1F600。
It is important to distinguish between a code point and its encoded bytes. Unicode itself is an abstract mapping; actual byte storage is handled by encoding forms such as UTF-8.
重要的是区分码点与其编码字节。Unicode本身是抽象映射；实际字节存储由UTF-8等编码形式处理。
6. UTF-8: Variable-Length Encoding | UTF-8：变长编码
UTF-8 is the most widely used character encoding on the web. It encodes each Unicode code point into 1 to 4 bytes using a clever pattern based on binary prefixes.
UTF-8是网络上使用最广泛的字符编码。它利用基于二进制前缀的巧妙模式，将每个Unicode码点编码为1到4个字节。
The table below shows the general structure:
下表显示了一般结构：
| Code point range | Byte 1 | Byte 2 | Byte 3 | Byte 4 |
| U+0000 – U+007F | 0xxxxxxx | – | – | – |
| U+0080 – U+07FF | 110xxxxx | 10xxxxxx | – | – |
| U+0800 – U+FFFF | 1110xxxx | 10xxxxxx | 10xxxxxx | – |
| U+10000 – U+10FFFF | 11110xxx | 10xxxxxx | 10xxxxxx | 10xxxxxx |
ASCII characters remain identical to one-byte UTF-8 sequences, making UTF-8 backward-compatible with ASCII. Non-ASCII characters such as ‘中’ require three bytes: E4 B8 AD in hexadecimal.
ASCII字符与单字节UTF-8序列完全相同，因此UTF-8与ASCII向后兼容。像“中”这样的非ASCII字符需要三个字节：十六进制E4 B8 AD。
7. UTF-16 and UTF-32 | UTF-16与UTF-32
UTF-16 encodes most common characters in exactly 2 bytes, and others in 2 code units (4 bytes). UTF-32 uses exactly 4 bytes for every character, making indexing simple but memory-heavy.
UTF-16将大多数字符编码为2字节，其他字符用2个编码单元（4字节）。UTF-32对每个字符固定使用4字节，索引简单但占用内存大。
For Chinese characters in the Basic Multilingual Plane, UTF-16 requires only 2 bytes, whereas UTF-8 requires 3 bytes. This makes UTF-16 more space-efficient for East Asian texts.
对于基本多语言平面内的汉字，UTF-16只需2字节，而UTF-8需要3字节。这使得UTF-16在处理东亚文本时更节省空间。
However, UTF-16 is less compatible with ASCII-based tooling and introduces byte-order ambiguity. Most modern web pages prefer UTF-8 because it is self-synchronizing and byte-oriented.
然而，UTF-16与基于ASCII的工具兼容性较差，并引入字节序歧义。大多数现代网页更偏好UTF-8，因为它是自同步的且面向字节。
8. Comparison of Common Encoding Formats | 常见编码格式对比
The following comparison summarises the key properties:
以下对比总结了主要属性：
| Format | ‘A’ encoding | ‘中’ encoding | Length for ‘中’ |
| ASCII | 0x41 | Not supported | – |
| UTF-8 | 0x41 | E4 B8 AD | 3 bytes |
| UTF-16 | 00 41 | 4E 2D | 2 bytes |
| UTF-32 | 00 00 00 41 | 00 00 4E 2D | 4 bytes |
Notice that UTF-16 and UTF-32 may swap byte order depending on the system. This leads us to the concept of endianness.
注意UTF-16和UTF-32可能根据系统交换字节顺序。这引出了字节序的概念。
9. Endianness and Byte Order Mark | 字节序与字节序标记
When a multi-byte character is stored in memory, the byte order can be either big-endian (most significant byte first) or little-endian (least significant byte first). For UTF-16, a Byte Order Mark (BOM) is often placed at the beginning of a file.
当多字节字符存储在内存中时，字节顺序可以是大端序（最高有效字节在前）或小端序（最低有效字节在前）。对于UTF-16，文件开头常放置字节序标记（BOM）。
The BOM is the character U+FEFF. If a reader sees the bytes FE FF, it knows the file is big-endian; if it sees FF FE, it knows the file is little-endian.
BOM是字符U+FEFF。如果读取器看到字节FE FF，则知道文件是大端序；如果看到FF FE，则知道是小端序。
UTF-8 does not require a BOM because its individual bytes are read in a strict sequence. However, some programs insert EF BB BF to signal that the file is UTF-8, which can cause unexpected text at the start of a file.
UTF-8不需要BOM，因为它的每个字节按严格顺序读取。但有些程序会插入EF BB BF来标示文件为UTF-8，这可能导致文件开头出现意外文本。
10. Character Encoding in Web and Programming | 网页与编程中的字符编码
In HTML, the meta tag <meta charset="UTF-8"> tells the browser to interpret the document using UTF-8. Modern HTML5 requires this declaration to avoid encoding ambiguities.
在HTML中，<meta charset="UTF-8">标签告诉浏览器使用UTF-8解释文档。现代HTML5要求此声明以避免编码歧义。
In programming languages, strings are often stored in memory as sequences of Unicode code points. Python 3 uses Unicode for all strings, while many C and C++ programs explicitly specify UTF-8 for external file I/O.
在编程语言中，字符串在内存中通常存储为Unicode码点序列。Python 3对所有字符串使用Unicode，而许多C和C++程序在外部文件I/O中显式指定UTF-8。
Web clients and servers communicate using HTTP headers such as Content-Type: text/html; charset=UTF-8. A mismatch between the declared charset and the actual bytes produces garbled text in browsers.
Web客户端和服务器通过HTTP头通信，例如Content-Type: text/html; charset=UTF-8。声明的字符集与实际字节不匹配会导致浏览器中显示乱码。
11. Common Pitfalls and Best Practices | 常见陷阱与最佳实践
One common failure occurs when an application assumes that one byte equals one character. This is true for ASCII but false for UTF-8 with accented or non-Latin characters. Always use library functions designed for Unicode.
一个常见错误是应用程序假设一个字节等于一个字符。这在ASCII中成立，但在包含重音或非拉丁字符的UTF-8中不成立。应始终使用为Unicode设计的库函数。
- Always specify an encoding when opening a text file.
- Prefer UTF-8 for new systems and web content.
- Do not manually count bytes in UTF-8 strings.
- Normalize Unicode forms if exact comparison is required.
- 打开文本文件时始终指定编码。
- 新系统和网页内容优先使用UTF-8。
- 不要手动统计UTF-8字符串中的字节数。
- 若需精确比较，使用Unicode规范化形式。
Another important issue is the distinction between collation order and code point order. Sorting Chinese characters by Unicode code point does not match dictionary order, so locale-aware collation libraries are necessary.
另一个重要问题是排序顺序与码点顺序的区别。按Unicode码点对汉字排序并不符合字典顺序，因此需要使用考虑区域设置的排序库。
12. Conclusion | 结论
Character encoding is the invisible foundation of all digital text. From ASCII’s 128 symbols to Unicode’s global coverage, every system must agree on how bytes map to characters.
字符编码是所有数字文本的隐形基础。从ASCII的128个符号到Unicode的全球覆盖，每个系统都必须在字节到字符的映射上达成一致。
For examination purposes, you should be able to compare ASCII, extended ASCII, UTF-8, UTF-16, and UTF-32, calculate the byte size of a given string, and explain the role of BOM and endianness in text files.
就考试而言，你应当能够比较ASCII、扩展ASCII、UTF-8、UTF-16和UTF-32，计算给定字符串的字节大小，并解释BOM和字节序在文本文件中的作用。
Mastering these concepts not only answers exam questions correctly but also prevents real-world encoding bugs before they happen.
掌握这些概念不仅能在考试中正确作答，还能在实际开发中提前预防编码错误。
Published by TutorHao | Computer Science Revision Series | aleveler.com
更多咨询请联系16621398022（同微信）
屏轩国际教育cambridge primary/secondary checkpoint, cat4, ukiset,ukcat,igcse,alevel,PAT,STEP,MAT, ibdp,ap,ssat,sat,sat2课程辅导，国外大学本科硕士研究生博士课程论文辅导