UNICODE CHARACTER DATABASE

Summary

This document describes the format and content of the Unicode Character Database (UCD)

Status

This file and the files described herein are part of the Unicode Character Database and are governed by the UCD Terms of Use given below.

The References provide related information that is useful in understanding this document.

Warning: the information in this file does not completely describe the use and interpretation of Unicode character properties and behavior. It must be used in conjunction with the data in the other files in the Unicode Character Database, and relies on the notation and definitions supplied in The Unicode Standard. All chapter references are to Version 4.0.0 of the standard unless otherwise indicated.

Contents

• Introduction
• Conformance
• UCD File Format
• UCD Files
• Properties
• Property Values
  • General Category Values
  • Bidi Class Values
  • Character Decomposition Mapping
  • Canonical Combining Classes
  • Decompositions and Normalization
  • Case Mappings
• Other UCD Files
• Derived Extracted Properties
• Property Invariants
• References
• Modification History

Introduction

The Unicode Character Database (UCD) is a set of files that define the Unicode character properties and internal mappings. This document describes the properties and files that are part of The Unicode Standard, Version 4.0 [U4.0]. The main changes in this version are:

• The four documentation files (UnicodeCharacterDatabase.html, UnicodeData.html, DerivedProperties.html, and PropList.html) have been merged together.
• There is an additional index by property instead of by file.
• A number of additional properties have been added as a part of Unicode 4.0.

This documentation file does not link directly to other files in the UCD. This is because the files need to be exactly the same in the specific update directory (e.g. http://www.unicode.org/Public/4.0-Update/), and when copied to the "latest" directory (http://www.unicode.org/Public/UNIDATA/).

Conformance

For information on the meaning and application of the terms normative, informative, and provisional, see "Chapter 3, Character Properties" in the Unicode Standard, Version 4.0.

UCD File Format

Files in the UCD use the following format, unless otherwise specified.

• Each line of data consists of fields separated by semicolons. The fields are numbered starting with zero. Code points are expressed as hexadecimal numbers with four to six digits. They are written without "U+". Within a sequence of code points, spaces are used for separation. Leading and trailing spaces within a field are not significant.

• The first field (0) of each line in the Unicode Character Database files represents a code point or range. The remaining fields (1..n) are properties associated with that code point.

• A range of code points is specified by the form "X..Y". Each code point from X to Y has the associated properties. For example:

  0000..007F; Basic Latin
  0080..00FF; Latin-1 Supplement
  
  1680      ; White_Space # Zs OGHAM SPACE MARK
  2000..200A; White_Space # Zs [11] EN QUAD..HAIR SPACE

• For backwards compatibility, in the file UnicodeData.txt a range is specified not by the form "X..Y", but by their start and end characters. In such cases, the names of characters in the range are algorithmically derivable. Surrogate code points and private use characters have no names. See U4.0 for more information.
• Hash marks ("#") are used to indicate comments: all characters from the hash mark to the end of the line are comments, and disregarded when parsing data. In many files, the comments on data lines use a common format.

  00BC..00BE ; numeric # No [3] VULGAR FRACTION ONE QUARTER..VULGAR FRACTION THREE QUARTERS

• The first part of the comment is generally the UCD general category. The symbol "L&" indicates characters of type Lu, Ll, or Lt. This is the same as the LC property in PropertyValueAliases. The code point ranges are calculated so that they all have the same General Category (or LC). While this results in more ranges than are strictly necessary, it makes the contents of the ranges clearer. The second part of the comment (in square brackets), indicates the number of items in a range, if there is one. The third part is the name of the character in field zero: if it is a range, then the character names for the ends of the range are separated by "..".

  However, the comments are purely informational, and may change format or be omitted in the future. They should not be parsed for content.

• In the following table, NF* refers to one of NFD, NFC, NFKC, or NFKD.
• The Unihan data format differs from the standard format, and is described in the header of the file. The header also describes which properties are informative, which are normative, and which are provisional.

• In some cases, segments of the file are distinguished by a line starting with an "@" sign.

• The files are either Latin-1 or UTF-8. Unless otherwise noted, non-ASCII characters only appear in comments.

UCD Files

The following table describes the format and meaning of each property data file in the UCD. The first column lists the files and the properties for which they contain data. The second column indicates the type of property value: String, Numeric, Enumeration (non-binary), Binary. The third column indicates the status (Normative vs. Informative), and the fourth column provides a description of the data.

The files with a small number of properties are listed first, followed by the files with a large number of properties: DerivedCoreProperties.txt, DerivedNormalizationProperties.txt, Proplist.txt, and UnicodeData.txt. For UnicodeData, the field numbers are supplied in the description. In a number of cases, fields in a data file only contribute to a UCD property; for example, the name field in UnicodeData.txt does not provide all the values for the Name property; Jamo.txt must be used as well.

None of these properties should be used without consulting the relevant discussions in the Unicode Standard.

Where a data file does not explicitly list property values for all code points, the code points are given default property values. These default property values are documented in the data files, with the exception of UnicodeData.txt. For that case the default property values are listed below in parentheses after the property name, with (=) indicating the code point itself.  The default property values are also documented in any corresponding extracted data file.

b = NFKC(Fold(a));
c = NFKC(Fold(b));
if (c != b) add mapping from a to c

Notes

1. Closure: XID_Start and XID_Continue are defined by adding or removing certain special characters as per UAX #15, Annex 7. They do not remove the non-NFKD nor the non-NFKC characters; if that is desired it needs to be a separate filter. They merely ensure that:

if isIdentifer(string)
then isIdentifier(NFKC(string))
and isIdentifier(NFKD(string))

2. Cf: The general category Cf characters are not included in ID_Continue nor in XID_Continue; they should continue identifiers, but be filtered out of the result.

For more information on identifiers, see Chapter 5, Implementation Guidelines, and UAX #15, Annex 7.

3. Stabilized properties are those that have not been found to be particularly useful in practice, and are no longer actively maintained, nor are they extended as new characters are added.

Properties

The following table lists the properties in the UCD. They are roughly organized into groups based on the usage of the property (this grouping is purely for convenience, and has no other implications). The link on each property leads to description in the file index. The contributory properties (those of the form Other_XXX) are sets of exceptions used to generate properties in DerivedCoreProperties.txt. They are not intended for general use, such as in APIs that return property values.

Property Values

The following gives a summary of property values for certain properties. Other property values are documented in other locations; for example, the Linebreak property values are documented in UAX #14.

General Category Values

The values in this field are abbreviations for the following values. For more information, see the Unicode Standard.

Note: The Unicode Standard does not assign information to control characters (except for certain cases). Implementations will generally also assign categories to certain control characters, notably CR and LF, according to platform conventions. See Section 5.8 "Newline Guidelines" for more information.

Note: The term "L&" is used to stand for Uppercase, Lowercase or Titlecase letters (Lu, Ll, or Lt) in comments. The LC value in PropertyValueAliases.txt also stands for Uppercase, Lowercase or Titlecase letters.

Bidi Class Values

Please refer to Chapter 3 for an explanation of the algorithm for Bidirectional Behavior and an explanation of the significance of these categories. An up-to-date version can be found on UAX #9: The Bidirectional Algorithm.

Character Decomposition Mapping

The tags supplied with certain decomposition mappings generally indicate formatting information. Where no such tag is given, the mapping is canonical. Conversely, the presence of a formatting tag also indicates that the mapping is a compatibility mapping and not a canonical mapping. In the absence of other formatting information in a compatibility mapping, the tag is used to distinguish it from canonical mappings.

In some instances a canonical mapping or a compatibility mapping may consist of a single character. For a canonical mapping, this indicates that the character is a canonical equivalent of another single character. For a compatibility mapping, this indicates that the character is a compatibility equivalent of another single character. The compatibility formatting tags used are:

Reminder: There is a difference between decomposition and decomposition mapping. The decomposition mappings are defined in the UnicodeData, while the decomposition (also termed "full decomposition") is defined in Chapter 3 to use those mappings recursively.

• The canonical decomposition is formed by recursively applying the canonical mappings, then applying the canonical reordering algorithm.
• The compatibility decomposition is formed by recursively applying the canonical and compatibility mappings, then applying the canonical reordering algorithm.

Canonical Combining Class Values

Note: some of the combining classes in this list do not currently have members but are specified here for completeness.

Decompositions and Normalization

Decomposition is specified in Chapter 3. UAX #15: Unicode Normalization Forms specifies the interaction between decomposition and normalization. That report specifies how the decompositions defined in UnicodeData.txt are used to derive normalized forms of Unicode text.

Note that as of the 2.1.9 update of the Unicode Character Database, the decompositions in the UnicodeData.txt file can be used to recursively derive the full decomposition in canonical order, without the need to separately apply canonical reordering. However, canonical reordering of combining character sequences must still be applied in decomposition when normalizing source text which contains any combining marks.

The QuickCheck property values are as follows:

For more information, see UAX #15 Annex 8.

Case Mappings

There are a number of complications to case mappings that occur once the repertoire of characters is expanded beyond ASCII. For more information, see Chapter 3 in Unicode 4.0.

For compatibility with existing parsers, UnicodeData.txt only contains case mappings for characters where they are one-to-one mappings; it also omits information about context-sensitive case mappings. Information about these special cases can be found in a separate data file, SpecialCasing.txt.

Unihan Tags

The following is a summary of the data tags in the Unihan.txt file. Only a few of these correspond to Unicode normative or informative properties: the rest are provisional. For more information on the meaning of these tags, see the header of the data file.

Other UCD Files

The following files in the Unicode Character Database are not used directly for Unicode properties.  For more information about these files, see the referenced technical report(s), files, or section of Unicode Standard.

Derived Extracted Properties

The following files contain other properties of the UCD that are simply separated out, and listed in range format. These files are provided purely as a reformatting of existing data, with a certain exceptions listed below. They are all contained in a subdirectory called extracted.

Bidi Note: The BidiMirrored property and the BidiMirroring property are different. The former is a normative property that indicates whether characters are mirrored in a right-to-left context in the Unicode Bidirectional Algorithm. The latter is an informative mapping of BidiMirrored characters, where possible, to characters that normally have the corresponding mirrored glyph.

Property Invariants

Values in the UCD are subject to correction as errors are found; however, some characteristics of the properties and files are considered invariants. Applications may wish to take these invariants into account when choosing how to implement character properties. The most important invariants are described in Unicode Policies. The following lists some additional invariants and more detail on some of the invariants in Unicode Policies.

UnicodeData Fields

• The number of fields in UnicodeData.txt is fixed.
  • Any additional information about character properties to be added in the future will appear in separate data files, rather than being added as an additional field or by subdivision or reinterpretation of existing fields.
• The order of the fields is also fixed.

Combining Classes

• Combining classes are limited to the values 0 to 255.
  • In practice, there are far fewer than 256 values used. Implementations may take advantage of this fact for compression, since only the ordering of the non-zero values matters for the Canonical Reordering Algorithm. It is possible for up to 256 values to be used in the future; however, UTC decisions in the future may restrict the number of values to 128, since this has implementation advantages. [Signed bytes can be used without widening to ints in Java, for example.]
• All characters other than those of General Category M* have the combining class 0.
  • Currently, all characters other than those of General Category Mn have the value 0. However, some characters of General Category Me or Mc may be given non-zero values in the future.
  • The precise values above the value 0 are not invariant--only the relative ordering of values is considered fixed. For example, it is not guaranteed in future versions that the class of U+05B4 will be precisely 14.

Decimal Digits

• In Unicode 4.0 and thereafter, the General_Category value Decimal_Number (Nd), and the Numeric_Type value Decimal (de) are defined to be co-extensive, that is, the set of character having Nd will always be the same as the set of characters having de.

References

Modification History

This section provides a summary of the changes between update versions of the Unicode Standard. The modifications prior to Unicode 4.0 only listed changes in UnicodeData.txt. From 4.0 onward, the consolidated modifications include the changes in other files.

Unicode 4.0

• UnicodeData.txt
  • Decimal Digits
    • Numeric_Type=decimal digit now aligned with General_Category=Nd
  • Modifier letters*
    • The general category of 02B9..02BA, 02C6..02CF changed to general category Lm.
• Other Files
  • New Properties and Values
    • Hangul_Syllable_Type, Unicode_Radical_Stroke
    • CJK numeric values added.
    • PropertyValueAliases adds block names
    • UCD fallback props more precisely defined, for code points not explicitly in data files
    • Added script value for Braille
    • New Linebreak properties: NL, WJ
  • Khmer
    • Two Khmer characters are deprecated; four others strongly discouraged.
  • Special Casing
    • Fixed for Turkish, Lithuanian
  • Default Ignorables
    • Hangul Filler characters
    • Soft-Hyphen, CGJ, ZWS
    • Arabic End of Ayah and Syriac Abbreviation Mark no longer DI (their shaping classes are also fixed.)
  • Grapheme_Extend
    • Removes halfwidth katakana marks, most Mc (except as needed for canonical equivalence)
  • Stabilized Properties
    • The Hyphen property is now stabilized.

Unicode 3.2

Modifications made for Version 3.2.0 of UnicodeData.txt include:

• Addition of 1016 new entries, to cover new characters encoded in Unicode 3.2.
• Updated ISO 6429 names for control functions to match the currently published version of that standard.
• Changed general category for Mongolian free variation selectors (U+180B..U+180D) from Cf to Mn.
• Changed general category for U+0B83 TAMIL SIGN VISARGA (aytham) from Mc to Lo.
• Changed general category for U+06DD ARABIC END OF AYAH from Me to Cf.
• Changed general category for U+17D7 KHMER SIGN LEK TOO from Po to Lm.
• Changed general category for U+17DC KHMER SIGN AVAKRAHASANYA from Po to Lo.
• Changed canonical decomposition for U+F951 from 96FB to 964B (see Corrigendum #3: U+F951 Normalization).

Unicode 3.1.1

Modifications made for Version 3.1.1 of UnicodeData.txt include:

• Modification of ISO 10646 annotation regarding Greek tonos, affecting entries for U+0301 and U+030D.

Unicode 3.1

Modifications made for Version 3.1.0 of UnicodeData.txt include:

• Addition of 2237 new entries, to cover new characters and new ranges of unified Han characters encoded in Unicode 3.1.
• Changed General Category value of 16EE..16F0 (Runic golden numbers) from No to Nl.

Unicode 3.0.1

Modifications made for Version 3.0.1 of UnicodeData.txt include:

• Added 5- and 6-digit representation of code points past U+FFFF.
• Added Private Use range definitions for Planes 15 and 16.
• Minor additions for the 10646 comment field.

Unicode 3.0.0

Modifications made for Version 3.0.0 of UnicodeData.txt include many new characters and a number of property changes. These are summarized in Appendix D of The Unicode Standard, Version 3.0.

Unicode 2.1.9

Modifications made for Version 2.1.9 of UnicodeData.txt include:

• Corrected combining class for U+05AE HEBREW ACCENT ZINOR.
• Corrected combining class for U+20E1 COMBINING LEFT RIGHT ARROW ABOVE
• Corrected combining class for U+0F35 and U+0F37 to 220.
• Corrected combining class for U+0F71 to 129.
• Added a decomposition for U+0F0C TIBETAN MARK DELIMITER TSHEG BSTAR.
• Added  decompositions for several Greek symbol letters: U+03D0..U+03D2, U+03D5, U+03D6, U+03F0..U+03F2.
• Removed  decompositions from the conjoining jamo block: U+1100..U+11F8.
• Changes to decomposition mappings for some Tibetan vowels for consistency in normalization. (U+0F71, U+0F73, U+0F77, U+0F79, U+0F81)
• Updated the decomposition mappings for several Vietnamese characters with two diacritics (U+1EAC, U+1EAD, U+1EB6, U+1EB7, U+1EC6, U+1EC7, U+1ED8, U+1ED9), so that the recursive decomposition can be generated directly in canonically reordered form (not a normative change).
• Updated the decomposition mappings for several Arabic compatibility characters involving shadda (U+FC5E..U+FC62, U+FCF2..U+FCF4), and two Latin characters (U+1E1C, U+1E1D), so that the decompositions are generated directly in canonically reordered form (not a normative change).
• Changed BIDI category for: U+00A0 NO-BREAK SPACE, U+2007 FIGURE SPACE, U+2028 LINE SEPARATOR.
• Changed BIDI category for extenders of General Category Lm: U+3005, U+3021..U+3035, U+FF9E, U+FF9F.
• Changed General Category and BIDI category for the Greek numeral signs: U+0374, U+0375.
• Corrected General Category for U+FFE8 HALFWIDTH FORMS LIGHT VERTICAL.
• Added Unicode 1.0 names for many Tibetan characters (informative).

Unicode 2.1.8

Modifications made for Version 2.1.8 of UnicodeData.txt include:

• Added combining class 240 for U+0345 COMBINING GREEK YPOGEGRAMMENI so that decompositions involving iota subscript are derivable directly in canonically reordered form; this also has a bearing on simplification of casing of polytonic Greek.
• Changes in decompositions related to Greek tonos. These result from the clarification that monotonic Greek "tonos" should be equated with U+0301 COMBINING ACUTE, rather than with U+030D COMBINING VERTICAL LINE ABOVE. (All Greek characters in the Greek block involving "tonos"; some Greek characters in the polytonic Greek in the 1FXX block.)
• Changed decompositions involving dialytika tonos. (U+0390, U+03B0)
• Changed ternary decompositions to binary. (U+0CCB, U+FB2C, U+FB2D) These changes simplify normalization.
• Removed canonical decomposition for Latin Candrabindu. (U+0310)
• Corrected error in canonical decomposition for U+1FF4.
• Added compatibility decompositions to clarify collation tables. (U+2100, U+2101, U+2105, U+2106, U+1E9A)
• A series of general category changes to assist the convergence of the Unicode definition of identifier with ISO TR 10176:
  • So > Lo: U+0950, U+0AD0, U+0F00, U+0F88..U+0F8B
  • Po > Lo: U+0E2F, U+0EAF, U+3006
  • Lm > Sk: U+309B, U+309C
  • Po > Pc: U+30FB, U+FF65
  • Ps/Pe > Mn: U+0F3E, U+0F3F
• A series of bidi property changes for consistency.
  • L > ET: U+09F2, U+09F3
  • ON > L: U+3007
  • L > ON: U+0F3A..U+0F3D, U+037E, U+0387
• Add case mapping: U+01A6 <-> U+0280
• Updated symmetric swapping value for guillemets: U+00AB, U+00BB, U+2039, U+203A.
• Changes to combining class values. Most Indic fixed position class non-spacing marks were changed to combining class 0. This fixes some inconsistencies in how canonical reordering would apply to Indic scripts, including Tibetan. Indic interacting top/bottom fixed position classes were merged into single (non-zero) classes as part of this change. Tibetan subjoined consonants are changed from combining class 6 to combining class 0. Thai pinthu (U+0E3A) moved to combining class 9. Moved two Devanagari stress marks into generic above and below combining classes (U+0951, U+0952).
• Corrected placement of semicolon near symmetric swapping field. (U+FA0E, etc., scattered positions to U+FA29)

Version 2.1.7

This version was for internal change tracking only, and never publicly released.

Version 2.1.6

This version was for internal change tracking only, and never publicly released.

Unicode 2.1.5

Modifications made for Version 2.1.5 of UnicodeData.txt include:

• Changed decomposition for U+FF9E and U+FF9F so that correct collation weighting will automatically result from the canonical equivalences.
• Removed canonical decompositions for U+04D4, U+04D5, U+04D8, U+04D9, U+04E0, U+04E1, U+04E8, U+04E9 (the implication being that no canonical equivalence is claimed between these 8 characters and similar Latin letters), and updated 4 canonical decompositions for U+04DB, U+04DC, U+04EA, U+04EB to reflect the implied difference in the base character.
• Added Pi, and Pf categories and assigned the relevant quotation marks to those categories, based on the Unicode Technical Corrigendum on Quotation Characters.
• Updating of many bidi properties, following the advice of the ad hoc committee on bidi, and to make the bidi properties of compatibility characters more consistent.
• Changed category of several Tibetan characters: U+0F3E, U+0F3F, U+0F88..U+0F8B to make them non-combining, reflecting the combined opinion of Tibetan experts.
• Added case mapping for U+03F2.
• Corrected case mapping for U+0275.
• Added titlecase mappings for U+03D0, U+03D1, U+03D5, U+03D6, U+03F0.. U+03F2.
• Corrected compatibility label for U+2121.
• Add specific entries for all the CJK compatibility ideographs, U+F900..U+FA2D, so the canonical decomposition for each (the URO character it is equivalent to) can be carried in the database.

Version 2.1.4

This version was for internal change tracking only, and never publicly released.

Version 2.1.3

This version was for internal change tracking only, and never publicly released.

Unicode 2.1.2

Modifications made in updating UnicodeData.txt to Version 2.1.2 for the Unicode Standard, Version 2.1 (from Version 2.0) include:

• Added two characters (U+20AC and U+FFFC).
• Amended bidi properties for U+0026, U+002E, U+0040, U+2007.
• Corrected case mappings for U+018E, U+019F, U+01DD, U+0258, U+0275, U+03C2, U+1E9B.
• Changed combining order class for U+0F71.
• Corrected canonical decompositions for U+0F73, U+1FBE.
• Changed decomposition for U+FB1F from compatibility to canonical.
• Added compatibility decompositions for U+FBE8, U+FBE9, U+FBF9..U+FBFB.
• Corrected compatibility decompositions for U+2469, U+246A, U+3358.

Version 2.1.1

This version was for internal change tracking only, and never publicly released.

Unicode 2.0.0

The modifications made in updating UnicodeData.txt for the Unicode Standard, Version 2.0 include:

• Fixed decompositions with TONOS to use correct NSM: 030D.
• Removed old Hangul Syllables; mapping to new characters are in a separate table.
• Marked compatibility decompositions with additional tags.
• Changed old tag names for clarity.
• Revision of decompositions to use first-level decomposition, instead of maximal decomposition.
• Correction of all known errors in decompositions from earlier versions.
• Added control code names (as old Unicode names).
• Added Hangul Jamo decompositions.
• Added Number category to match properties list in book.
• Fixed categories of Koranic Arabic marks.
• Fixed categories of precomposed characters to match decomposition where possible.
• Added Hebrew cantillation marks and the Tibetan script.
• Added place holders for ranges such as CJK Ideographic Area and the Private Use Area.
• Added categories Me, Sk, Pc, Nl, Cs, Cf, and rectified a number of mistakes in the database.

UCD Terms of Use

Disclaimer

The Unicode Character Database is provided as is by Unicode, Inc. No claims are made as to fitness for any particular purpose. No warranties of any kind are expressed or implied. The recipient agrees to determine applicability of information provided. If this file has been purchased on magnetic or optical media from Unicode, Inc., the sole remedy for any claim will be exchange of defective media within 90 days of receipt.

This disclaimer is applicable for all other data files accompanying the Unicode Character Database, some of which have been compiled by the Unicode Consortium, and some of which have been supplied by other sources.

Limitations on Rights to Redistribute This Data

Recipient is granted the right to make copies in any form for internal distribution and to freely use the information supplied in the creation of products supporting the Unicode^TM Standard. The files in the Unicode Character Database can be redistributed to third parties or other organizations (whether for profit or not) as long as this notice and the disclaimer notice are retained. Information can be extracted from these files and used in documentation or programs, as long as there is an accompanying notice indicating the source.

The file Unihan.txt contains older and inconsistent Terms of Use. That language is overridden by these terms.