Kilcullen Science and Engineering

Artefacts in .Txt Files and Mojibake

Infographic explaining character encoding, ASCII, ANSI and UTF-8, showing how incorrect decoding produces garbled text, known as Mojibake. That's 127 characters, so you're within the limit.
Image generated by ChatGPT

If you saved a file in Notepad and then double-clicked it, WordPad would likely have opened it, as this is the default application for opening such files. You may have noticed what looked like corruption in the file. For instance, the copyright symbol, ©, appears as "©". So how does this happen?
ASCII (American Standard Code for Information Interchange), developed in 1963, is a 7-bit character encoding system used for representing alphanumeric and other characters. It built on the earlier 5-bit Baudot code, originally used for teleprinters (also generically known as teletype machines, after the manufacturer, the Teletype Corporation). Teleprinters are also used in connection with the Telex network, an international teleprinter communications system.
Since ASCII uses only 7 bits to represent each character, it can encode 128 different values, numbered from 0 to 127. However, only 95 of these represent printable characters; the remainder are control characters. The copyright symbol (©) isn't included.
An 8-bit system allows 256 different values, and various extended ASCII character encodings were developed to make use of this additional capacity. Windows-1252 is one such encoding, providing characters such as the copyright symbol. However, a more versatile character encoding system called UTF-8 is now widely used.
In UTF-8, the basic ASCII characters are represented by a single byte, while additional characters require between two and four bytes in total. The copyright symbol, for example, is represented by two bytes: C2 and A9 (in hexadecimal).
Unfortunately, although Notepad is an older application than WordPad, it has been updated over successive versions of Windows. Modern versions of Notepad use UTF-8 as their default encoding and can correctly interpret UTF-8 files without a Byte Order Mark (BOM). WordPad, however, has less reliable UTF-8 handling in certain circumstances. If it interprets the two bytes representing the copyright symbol as Windows-1252 characters instead, they appear as two separate characters: "©". This is an example of character encoding misinterpretation, sometimes called mojibake.

No comments:

Post a Comment

×

Posts by Category