What are the most common non-BMP Unicode characters in actual use? on arvau

Stack Overflow / What are the most common non-BMP Unicode characters in actual use?

126 votesby stackoverflow.com/users/527702written 2011-04-06last updated 2013-05-30

In your experience which Unicode characters, codepoints, ranges outside the BMP (Basic Multilingual Plane) are the most common so far? These are the ones which require 4 bytes in UTF-8 or surrogates in UTF-16.

I would've expected the answer to be Chinese and Japanese characters used in names but not included in the most widespread CJK multibyte character sets, but on the project I do most work on, the English Wiktionary, we have found that the Gothic alphabet is far more common so far.

UPDATE

I've written a couple of software tools to scan entire Wikipedias for non-BMP characters and found to my surprise that even in the Japanese Wikipedia Gothic alphabet is the most common. This is also true in the Chinese Wikipedia but it also had many Chinese characters being used up to 50 or 70 times, including "", "", and "".

Stack Overflow / What are the most common non-BMP Unicode characters in actual use?

74 votesby stackoverflow.com/users/471272written 2011-04-07last updated 2022-02-08

Excellent question!

The answer is the mathematical letters. This past December I did a scan of the entire PubMed Open Access corpus, and came up with these figures for astral characters in it.

The first number in the figures below is how many copies of each given code point I found in the entire corpus. First, though, to give you a notion on the relative frequencies, here are the top ten trans-ASCII code points in that corpus:

 2663710 U+002013 â¹ââº  GC=Pd    EN DASH
 1065594 U+0000A0 â¹Â âº  GC=Zs    NO-BREAK SPACE
 1009762 U+0000B1 â¹Â±âº  GC=Sm    PLUS-MINUS SIGN
  784139 U+002212 â¹ââº  GC=Sm    MINUS SIGN
  602377 U+002003 â¹ââº  GC=Zs    EM SPACE
  528576 U+0003BC â¹Î¼âº  GC=Ll    GREEK SMALL LETTER MU
  519669 U+0003B2 â¹Î²âº  GC=Ll    GREEK SMALL LETTER BETA
  512312 U+0003B1 â¹Î±âº  GC=Ll    GREEK SMALL LETTER ALPHA
  491842 U+00200A â¹ââº  GC=Zs    HAIR SPACE
  462505 U+0000B0 â¹Â°âº  GC=So    DEGREE SIGN

And here now are the trans-BMP code points, in order of decending frequency:

     544 U+01D49E â¹âº  GC=Lu    MATHEMATICAL SCRIPT CAPITAL C
     450 U+01D4AF â¹âº  GC=Lu    MATHEMATICAL SCRIPT CAPITAL T
     385 U+01D4AE â¹âº  GC=Lu    MATHEMATICAL SCRIPT CAPITAL S
     292 U+01D49F â¹âº  GC=Lu    MATHEMATICAL SCRIPT CAPITAL D
     285 U+01D4B3 â¹âº  GC=Lu    MATHEMATICAL SCRIPT CAPITAL X
     262 U+01D4A9 â¹âº  GC=Lu    MATHEMATICAL SCRIPT CAPITAL N
     258 U+01D4AB â¹âº  GC=Lu    MATHEMATICAL SCRIPT CAPITAL P
     254 U+01D4A2 â¹âº  GC=Lu    MATHEMATICAL SCRIPT CAPITAL G
     185 U+01D49C â¹âº  GC=Lu    MATHEMATICAL SCRIPT CAPITAL A
     178 U+01D53C â¹âº  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL E
     137 U+01D4AA â¹âº  GC=Lu    MATHEMATICAL SCRIPT CAPITAL O
      56 U+01D4A5 â¹âº  GC=Lu    MATHEMATICAL SCRIPT CAPITAL J
      48 U+01D4A6 â¹âº  GC=Lu    MATHEMATICAL SCRIPT CAPITAL K
      44 U+01D4B1 â¹âº  GC=Lu    MATHEMATICAL SCRIPT CAPITAL V
      43 U+01D4B2 â¹âº  GC=Lu    MATHEMATICAL SCRIPT CAPITAL W
      42 U+01D4B4 â¹âº  GC=Lu    MATHEMATICAL SCRIPT CAPITAL Y
      41 U+01D4B5 â¹âº  GC=Lu    MATHEMATICAL SCRIPT CAPITAL Z
      35 U+01D4B0 â¹âº  GC=Lu    MATHEMATICAL SCRIPT CAPITAL U
      30 U+01D4AC â¹âº  GC=Lu    MATHEMATICAL SCRIPT CAPITAL Q
      23 U+01D54A â¹âº  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL S
      21 U+01D539 â¹âº  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL B
      19 U+01D5A7 â¹âº  GC=Lu    MATHEMATICAL SANS-SERIF CAPITAL H
      18 U+01D517 â¹âº  GC=Lu    MATHEMATICAL FRAKTUR CAPITAL T
      15 U+01D4C3 â¹âº  GC=Ll    MATHEMATICAL SCRIPT SMALL N
      14 U+01D535 â¹âº  GC=Ll    MATHEMATICAL FRAKTUR SMALL X
      13 U+01D4BF â¹âº  GC=Ll    MATHEMATICAL SCRIPT SMALL J
      11 U+01D540 â¹âº  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL I
       9 U+01D465 â¹âº  GC=Ll    MATHEMATICAL ITALIC SMALL X
       9 U+01D4CE â¹âº  GC=Ll    MATHEMATICAL SCRIPT SMALL Y
       9 U+01D538 â¹âº  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL A
       8 U+01D4C2 â¹âº  GC=Ll    MATHEMATICAL SCRIPT SMALL M
       8 U+01D54D â¹âº  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL V
       7 U+01D4B6 â¹âº  GC=Ll    MATHEMATICAL SCRIPT SMALL A
       7 U+01D4BE â¹âº  GC=Ll    MATHEMATICAL SCRIPT SMALL I
       7 U+01D4CC â¹âº  GC=Ll    MATHEMATICAL SCRIPT SMALL W
       7 U+01D516 â¹âº  GC=Lu    MATHEMATICAL FRAKTUR CAPITAL S
       7 U+01D4BE â¹âº  GC=Ll    MATHEMATICAL SCRIPT SMALL I
       7 U+01D4CC â¹âº  GC=Ll    MATHEMATICAL SCRIPT SMALL W
       7 U+01D516 â¹âº  GC=Lu    MATHEMATICAL FRAKTUR CAPITAL S
       4 U+01D4CF â¹âº  GC=Ll    MATHEMATICAL SCRIPT SMALL Z
       4 U+01D53B â¹âº  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL D
       4 U+01D54B â¹âº  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL T
       3 U+01D4BB â¹âº  GC=Ll    MATHEMATICAL SCRIPT SMALL F
       3 U+01D4CA â¹âº  GC=Ll    MATHEMATICAL SCRIPT SMALL U
       3 U+01D507 â¹âº  GC=Lu    MATHEMATICAL FRAKTUR CAPITAL D
       3 U+01D542 â¹âº  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL K
       3 U+01D546 â¹âº  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL O
       2 U+01D4BD â¹âº  GC=Ll    MATHEMATICAL SCRIPT SMALL H
       2 U+01D4C5 â¹âº  GC=Ll    MATHEMATICAL SCRIPT SMALL P
       2 U+01D505 â¹âº  GC=Lu    MATHEMATICAL FRAKTUR CAPITAL B
       2 U+01D50E â¹âº  GC=Lu    MATHEMATICAL FRAKTUR CAPITAL K
       2 U+01D541 â¹âº  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL J
       2 U+01D543 â¹âº  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL L
       2 U+100002 â¹âº  GC=Co    &lt;private use character&gt;
       1 U+01D4B8 â¹âº  GC=Ll    MATHEMATICAL SCRIPT SMALL C
       1 U+01D4C1 â¹âº  GC=Ll    MATHEMATICAL SCRIPT SMALL L
       1 U+01D53D â¹âº  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL F
       1 U+01D53E â¹âº  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL G
       1 U+01D54C â¹âº  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL U
       1 U+01D6A4 â¹âº  GC=Ll    MATHEMATICAL ITALIC SMALL DOTLESS I
       1 U+01D7D9 â¹âº  GC=Nd    MATHEMATICAL DOUBLE-STRUCK DIGIT ONE

I really wish I knew what they were using U+100002 to do. :(

If those aren't showing up in your browser, you should install George Dourosâs Symbola font or another mirror for dowload. It also has all the fun Unicode 6.0.0 code points in it, too.

Stack Overflow / What are the most common non-BMP Unicode characters in actual use?

9 votesby stackoverflow.com/users/178761written 2011-04-06

For me, the Mathematical Alphanumeric Symbols that are used for math typesetting with OpenType fonts such as Cambria Math.

Stack Overflow / What are the most common non-BMP Unicode characters in actual use?

116 votesby stackoverflow.com/users/773754written 2013-08-28last updated 2017-09-13

Emoji are now the most common non-BMP characters by far. , otherwise known as U+1F602 FACE WITH TEARS OF JOY, is the most common one on Twitter's public stream. It occurs more frequently than the tilde!