Tokenization.html 13 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252
  1. <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
  2. <html>
  3. <!-- Copyright (C) 1987-2017 Free Software Foundation, Inc.
  4. Permission is granted to copy, distribute and/or modify this document
  5. under the terms of the GNU Free Documentation License, Version 1.3 or
  6. any later version published by the Free Software Foundation. A copy of
  7. the license is included in the
  8. section entitled "GNU Free Documentation License".
  9. This manual contains no Invariant Sections. The Front-Cover Texts are
  10. (a) (see below), and the Back-Cover Texts are (b) (see below).
  11. (a) The FSF's Front-Cover Text is:
  12. A GNU Manual
  13. (b) The FSF's Back-Cover Text is:
  14. You have freedom to copy and modify this GNU Manual, like GNU
  15. software. Copies published by the Free Software Foundation raise
  16. funds for GNU development. -->
  17. <!-- Created by GNU Texinfo 5.2, http://www.gnu.org/software/texinfo/ -->
  18. <head>
  19. <title>The C Preprocessor: Tokenization</title>
  20. <meta name="description" content="The C Preprocessor: Tokenization">
  21. <meta name="keywords" content="The C Preprocessor: Tokenization">
  22. <meta name="resource-type" content="document">
  23. <meta name="distribution" content="global">
  24. <meta name="Generator" content="makeinfo">
  25. <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  26. <link href="index.html#Top" rel="start" title="Top">
  27. <link href="Index-of-Directives.html#Index-of-Directives" rel="index" title="Index of Directives">
  28. <link href="index.html#SEC_Contents" rel="contents" title="Table of Contents">
  29. <link href="Overview.html#Overview" rel="up" title="Overview">
  30. <link href="The-preprocessing-language.html#The-preprocessing-language" rel="next" title="The preprocessing language">
  31. <link href="Initial-processing.html#Initial-processing" rel="prev" title="Initial processing">
  32. <style type="text/css">
  33. <!--
  34. a.summary-letter {text-decoration: none}
  35. blockquote.smallquotation {font-size: smaller}
  36. div.display {margin-left: 3.2em}
  37. div.example {margin-left: 3.2em}
  38. div.indentedblock {margin-left: 3.2em}
  39. div.lisp {margin-left: 3.2em}
  40. div.smalldisplay {margin-left: 3.2em}
  41. div.smallexample {margin-left: 3.2em}
  42. div.smallindentedblock {margin-left: 3.2em; font-size: smaller}
  43. div.smalllisp {margin-left: 3.2em}
  44. kbd {font-style:oblique}
  45. pre.display {font-family: inherit}
  46. pre.format {font-family: inherit}
  47. pre.menu-comment {font-family: serif}
  48. pre.menu-preformatted {font-family: serif}
  49. pre.smalldisplay {font-family: inherit; font-size: smaller}
  50. pre.smallexample {font-size: smaller}
  51. pre.smallformat {font-family: inherit; font-size: smaller}
  52. pre.smalllisp {font-size: smaller}
  53. span.nocodebreak {white-space:nowrap}
  54. span.nolinebreak {white-space:nowrap}
  55. span.roman {font-family:serif; font-weight:normal}
  56. span.sansserif {font-family:sans-serif; font-weight:normal}
  57. ul.no-bullet {list-style: none}
  58. -->
  59. </style>
  60. </head>
  61. <body lang="en" bgcolor="#FFFFFF" text="#000000" link="#0000FF" vlink="#800080" alink="#FF0000">
  62. <a name="Tokenization"></a>
  63. <div class="header">
  64. <p>
  65. Next: <a href="The-preprocessing-language.html#The-preprocessing-language" accesskey="n" rel="next">The preprocessing language</a>, Previous: <a href="Initial-processing.html#Initial-processing" accesskey="p" rel="prev">Initial processing</a>, Up: <a href="Overview.html#Overview" accesskey="u" rel="up">Overview</a> &nbsp; [<a href="index.html#SEC_Contents" title="Table of contents" rel="contents">Contents</a>][<a href="Index-of-Directives.html#Index-of-Directives" title="Index" rel="index">Index</a>]</p>
  66. </div>
  67. <hr>
  68. <a name="Tokenization-1"></a>
  69. <h3 class="section">1.3 Tokenization</h3>
  70. <a name="index-tokens"></a>
  71. <a name="index-preprocessing-tokens"></a>
  72. <p>After the textual transformations are finished, the input file is
  73. converted into a sequence of <em>preprocessing tokens</em>. These mostly
  74. correspond to the syntactic tokens used by the C compiler, but there are
  75. a few differences. White space separates tokens; it is not itself a
  76. token of any kind. Tokens do not have to be separated by white space,
  77. but it is often necessary to avoid ambiguities.
  78. </p>
  79. <p>When faced with a sequence of characters that has more than one possible
  80. tokenization, the preprocessor is greedy. It always makes each token,
  81. starting from the left, as big as possible before moving on to the next
  82. token. For instance, <code>a+++++b</code> is interpreted as
  83. <code>a&nbsp;++&nbsp;++&nbsp;+&nbsp;b<!-- /@w --></code>, not as <code>a&nbsp;++&nbsp;+&nbsp;++&nbsp;b<!-- /@w --></code>, even though the
  84. latter tokenization could be part of a valid C program and the former
  85. could not.
  86. </p>
  87. <p>Once the input file is broken into tokens, the token boundaries never
  88. change, except when the &lsquo;<samp>##</samp>&rsquo; preprocessing operator is used to paste
  89. tokens together. See <a href="Concatenation.html#Concatenation">Concatenation</a>. For example,
  90. </p>
  91. <div class="smallexample">
  92. <pre class="smallexample">#define foo() bar
  93. foo()baz
  94. &rarr; bar baz
  95. <em>not</em>
  96. &rarr; barbaz
  97. </pre></div>
  98. <p>The compiler does not re-tokenize the preprocessor&rsquo;s output. Each
  99. preprocessing token becomes one compiler token.
  100. </p>
  101. <a name="index-identifiers"></a>
  102. <p>Preprocessing tokens fall into five broad classes: identifiers,
  103. preprocessing numbers, string literals, punctuators, and other. An
  104. <em>identifier</em> is the same as an identifier in C: any sequence of
  105. letters, digits, or underscores, which begins with a letter or
  106. underscore. Keywords of C have no significance to the preprocessor;
  107. they are ordinary identifiers. You can define a macro whose name is a
  108. keyword, for instance. The only identifier which can be considered a
  109. preprocessing keyword is <code>defined</code>. See <a href="Defined.html#Defined">Defined</a>.
  110. </p>
  111. <p>This is mostly true of other languages which use the C preprocessor.
  112. However, a few of the keywords of C++ are significant even in the
  113. preprocessor. See <a href="C_002b_002b-Named-Operators.html#C_002b_002b-Named-Operators">C++ Named Operators</a>.
  114. </p>
  115. <p>In the 1999 C standard, identifiers may contain letters which are not
  116. part of the &ldquo;basic source character set&rdquo;, at the implementation&rsquo;s
  117. discretion (such as accented Latin letters, Greek letters, or Chinese
  118. ideograms). This may be done with an extended character set, or the
  119. &lsquo;<samp>\u</samp>&rsquo; and &lsquo;<samp>\U</samp>&rsquo; escape sequences. GCC only accepts such
  120. characters in the &lsquo;<samp>\u</samp>&rsquo; and &lsquo;<samp>\U</samp>&rsquo; forms.
  121. </p>
  122. <p>As an extension, GCC treats &lsquo;<samp>$</samp>&rsquo; as a letter. This is for
  123. compatibility with some systems, such as VMS, where &lsquo;<samp>$</samp>&rsquo; is commonly
  124. used in system-defined function and object names. &lsquo;<samp>$</samp>&rsquo; is not a
  125. letter in strictly conforming mode, or if you specify the <samp>-$</samp>
  126. option. See <a href="Invocation.html#Invocation">Invocation</a>.
  127. </p>
  128. <a name="index-numbers"></a>
  129. <a name="index-preprocessing-numbers"></a>
  130. <p>A <em>preprocessing number</em> has a rather bizarre definition. The
  131. category includes all the normal integer and floating point constants
  132. one expects of C, but also a number of other things one might not
  133. initially recognize as a number. Formally, preprocessing numbers begin
  134. with an optional period, a required decimal digit, and then continue
  135. with any sequence of letters, digits, underscores, periods, and
  136. exponents. Exponents are the two-character sequences &lsquo;<samp>e+</samp>&rsquo;,
  137. &lsquo;<samp>e-</samp>&rsquo;, &lsquo;<samp>E+</samp>&rsquo;, &lsquo;<samp>E-</samp>&rsquo;, &lsquo;<samp>p+</samp>&rsquo;, &lsquo;<samp>p-</samp>&rsquo;, &lsquo;<samp>P+</samp>&rsquo;, and
  138. &lsquo;<samp>P-</samp>&rsquo;. (The exponents that begin with &lsquo;<samp>p</samp>&rsquo; or &lsquo;<samp>P</samp>&rsquo; are
  139. used for hexadecimal floating-point constants.)
  140. </p>
  141. <p>The purpose of this unusual definition is to isolate the preprocessor
  142. from the full complexity of numeric constants. It does not have to
  143. distinguish between lexically valid and invalid floating-point numbers,
  144. which is complicated. The definition also permits you to split an
  145. identifier at any position and get exactly two tokens, which can then be
  146. pasted back together with the &lsquo;<samp>##</samp>&rsquo; operator.
  147. </p>
  148. <p>It&rsquo;s possible for preprocessing numbers to cause programs to be
  149. misinterpreted. For example, <code>0xE+12</code> is a preprocessing number
  150. which does not translate to any valid numeric constant, therefore a
  151. syntax error. It does not mean <code>0xE&nbsp;+&nbsp;12<!-- /@w --></code>, which is what you
  152. might have intended.
  153. </p>
  154. <a name="index-string-literals"></a>
  155. <a name="index-string-constants"></a>
  156. <a name="index-character-constants"></a>
  157. <a name="index-header-file-names"></a>
  158. <p><em>String literals</em> are string constants, character constants, and
  159. header file names (the argument of &lsquo;<samp>#include</samp>&rsquo;).<a name="DOCF2" href="#FOOT2"><sup>2</sup></a> String constants and character
  160. constants are straightforward: <tt>&quot;&hellip;&quot;</tt> or <tt>'&hellip;'</tt>. In
  161. either case embedded quotes should be escaped with a backslash:
  162. <tt>'\''</tt> is the character constant for &lsquo;<samp>'</samp>&rsquo;. There is no limit on
  163. the length of a character constant, but the value of a character
  164. constant that contains more than one character is
  165. implementation-defined. See <a href="Implementation-Details.html#Implementation-Details">Implementation Details</a>.
  166. </p>
  167. <p>Header file names either look like string constants, <tt>&quot;&hellip;&quot;</tt>, or are
  168. written with angle brackets instead, <tt>&lt;&hellip;&gt;</tt>. In either case,
  169. backslash is an ordinary character. There is no way to escape the
  170. closing quote or angle bracket. The preprocessor looks for the header
  171. file in different places depending on which form you use. See <a href="Include-Operation.html#Include-Operation">Include Operation</a>.
  172. </p>
  173. <p>No string literal may extend past the end of a line. You may use continued
  174. lines instead, or string constant concatenation.
  175. </p>
  176. <a name="index-punctuators"></a>
  177. <a name="index-digraphs"></a>
  178. <a name="index-alternative-tokens"></a>
  179. <p><em>Punctuators</em> are all the usual bits of punctuation which are
  180. meaningful to C and C++. All but three of the punctuation characters in
  181. ASCII are C punctuators. The exceptions are &lsquo;<samp>@</samp>&rsquo;, &lsquo;<samp>$</samp>&rsquo;, and
  182. &lsquo;<samp>`</samp>&rsquo;. In addition, all the two- and three-character operators are
  183. punctuators. There are also six <em>digraphs</em>, which the C++ standard
  184. calls <em>alternative tokens</em>, which are merely alternate ways to spell
  185. other punctuators. This is a second attempt to work around missing
  186. punctuation in obsolete systems. It has no negative side effects,
  187. unlike trigraphs, but does not cover as much ground. The digraphs and
  188. their corresponding normal punctuators are:
  189. </p>
  190. <div class="smallexample">
  191. <pre class="smallexample">Digraph: &lt;% %&gt; &lt;: :&gt; %: %:%:
  192. Punctuator: { } [ ] # ##
  193. </pre></div>
  194. <a name="index-other-tokens"></a>
  195. <p>Any other single character is considered &ldquo;other&rdquo;. It is passed on to
  196. the preprocessor&rsquo;s output unmolested. The C compiler will almost
  197. certainly reject source code containing &ldquo;other&rdquo; tokens. In ASCII, the
  198. only other characters are &lsquo;<samp>@</samp>&rsquo;, &lsquo;<samp>$</samp>&rsquo;, &lsquo;<samp>`</samp>&rsquo;, and control
  199. characters other than NUL (all bits zero). (Note that &lsquo;<samp>$</samp>&rsquo; is
  200. normally considered a letter.) All characters with the high bit set
  201. (numeric range 0x7F&ndash;0xFF) are also &ldquo;other&rdquo; in the present
  202. implementation. This will change when proper support for international
  203. character sets is added to GCC.
  204. </p>
  205. <p>NUL is a special case because of the high probability that its
  206. appearance is accidental, and because it may be invisible to the user
  207. (many terminals do not display NUL at all). Within comments, NULs are
  208. silently ignored, just as any other character would be. In running
  209. text, NUL is considered white space. For example, these two directives
  210. have the same meaning.
  211. </p>
  212. <div class="smallexample">
  213. <pre class="smallexample">#define X^@1
  214. #define X 1
  215. </pre></div>
  216. <p>(where &lsquo;<samp>^@</samp>&rsquo; is ASCII NUL). Within string or character constants,
  217. NULs are preserved. In the latter two cases the preprocessor emits a
  218. warning message.
  219. </p>
  220. <div class="footnote">
  221. <hr>
  222. <h4 class="footnotes-heading">Footnotes</h4>
  223. <h3><a name="FOOT2" href="#DOCF2">(2)</a></h3>
  224. <p>The C
  225. standard uses the term <em>string literal</em> to refer only to what we are
  226. calling <em>string constants</em>.</p>
  227. </div>
  228. <hr>
  229. <div class="header">
  230. <p>
  231. Next: <a href="The-preprocessing-language.html#The-preprocessing-language" accesskey="n" rel="next">The preprocessing language</a>, Previous: <a href="Initial-processing.html#Initial-processing" accesskey="p" rel="prev">Initial processing</a>, Up: <a href="Overview.html#Overview" accesskey="u" rel="up">Overview</a> &nbsp; [<a href="index.html#SEC_Contents" title="Table of contents" rel="contents">Contents</a>][<a href="Index-of-Directives.html#Index-of-Directives" title="Index" rel="index">Index</a>]</p>
  232. </div>
  233. </body>
  234. </html>