<erl>
out(A) ->
       {ssi, "TAB.inc", "%%",[{"internals", "choosen"}]}.
</erl>


<div id="entry">

  <h1>Internals</h1>

  <h2>Introduction</h2>

  <p>I'll try to describe some of the internal workings of Yaws in this page.
  The page is thus mostly interesting for people interested in either hacking Yaws
  or simply wanting to get a better understanding.
  </p>


  <p>I'll describe how Yaws pages get compiled, the process structure
  and other things which can make it easier to understand the code. This page
  is ment to be read by programmers that wish to either work on Yaws or
  just get a better understanding.
  </p>


  <h2> JIT Compiling a .yaws page</h2>

  <p>
    When the client GETs a a page that has a .yaws suffix. The Yaws  server
    will read that page from the hard disk and divide it in parts
    that consist of HTML code and Erlang  code. Each chunk of Erlang  code
    will be compiled into a module. The chunk of Erlang  code must contain
    a function <tt>out/1</tt> If it doesn't the Yaws  server will insert a
    proper error message into the generated HTML output.

  </p>

  <p>When the Yaws server ships a .yaws page it will process it chunk by chunk
  through the .yaws file. If it is HTML code, the server will ship that
  as is, whereas if it is Erlang  code, the Yaws  server will invoke the
  <tt>out/1</tt> function in that code and insert the output of that <tt>out/1</tt>
  function into the stream
  of HTML that is being shipped to the client.
  </p>

  <p>Yaws  will cache the result of the compilation
  and the next time a client requests the same .yaws page Yaws  will
  be able to invoke the already compiled modules directly.
  </p>


  <p>This is best explained by an example:</p>

  <p>Say that a file consists of 400 bytes, we have "foo.yaws"
  and it looks like:</p>

  <p>
    <img src="compile_layout.png" />
  </p>

  <p>When a client request the file "foo.yaws", the webserver will
  look in its cache for the file, (more on that later). For the sake of
  argument, we assume the file is not in the cache.

  </p>
  <p>The file will be processes by the code in <tt>yaws_compile.erl</tt>
  and the result will be a structure that looks like:</p>

  <div class="box">
    <verbatim>

    [CodeSpec]
    CodeSpec = Data |  Code | Error
    Data = {data, NumChars}
    Code = {mod, LineNo, YawsFile, NumSkipChars, Mod, Func}
    Err = {error, NumSkipChars, E}

    </verbatim>
  </div>


  <p>In the particular case of our "foo.yaws" file above, the JIT
  compiler will return:
  </p>

  <div class="box">
  <verbatim>

    [{mod, 1, "/foo.yaws", 100, m1, out},
     {data, 200},
     {mod, 30, "/foo.yaws", 100, m2, out}
    ]

  </verbatim>
  </div>

  <p>
    This structure gets stored in the cache and will continue
    to be associated to the file "foo.yaws".
  </p>
  <p>When the server "ships" a .yaws page, it needs the <tt>CodeSpec</tt>
  structure to do it. If the structure is not in the cache, the page
  gets JIT compiled and inserted into the cache.
  </p>
  <p>To ship the above <tt>CodeSpec</tt> structure, the server
  performs the following steps:</p>
  <ol>
    <li>Create the Arg structure which is a #arg{} record, this
    structure is wellknown to all yaws programmers since it's the
    main mechanism to pass data from the server to the .yaws
    page.</li>
    <li>Item (1) Invoke <tt>m1:out(Arg)</tt></li>
    <li>Look at the return value from <tt>m1:out(Arg)</tt> and
    perform whatever is requested. This typically involves generating
    some dynamic ehtml code, generate headers or whatever.
    </li>
    <li>Finally jump ahead 100 bytes in the file as a result of
    processing the first <tt>CodeSpec</tt> item.</li>

    <li>Item (2) Next <tt>CodeSpec</tt> is just plain data from the file,
    thus we read 200 bytes from the file (or rather from the cache
    since the data will be there) and ship to the client.</li>

    <li>Item (3) Yet another {mod structure which is handled
    the same way as Item (1) above except that the erlang module
    is <tt>m2</tt> instead of <tt>m1</tt></li>
  </ol>

  <p>Another thing that is worth mentioning is that yaws will
  not ship (write on the socket) data until all content is generated.
  This is questionable
  and different from what i.e. PHP does. This makes it possible to
  generate headers after content has been generated.
  </p>



  <h2>Process structure</h2>

  <p>Before describing the process structure, I need to describe
  the two most important datastructures in Yaws. The <tt>#gconf{}</tt>
  and the <tt>#sconf{}</tt> records.
  </p>
  <p><b>Note:</b> To retrieve information from these records, yaws:gconf_*/1
  and yaws:sconf_*/1 (e.g. yaws:gconf_id/1 or yaws:sconf_docroot/1) should
  be used in preference to a direct access to reduce the dependence of your
  code on it.
  </p>

  <h3>The <tt>#gconf{}</tt> record</h3>
  <p>This record is used to hold all global state, i.e. state and configuration
  data which is valid for all Virtual servers.
  The record looks like:
  </p>
  <div class="box">
    <verbatim>

 %% global conf
 -record(gconf,{
           yaws_dir,                       % topdir of Yaws installation
           trace,                          % false | {true,http} | {true,traffic}
           flags = ?GC_DEF,                % boolean flags
           logdir,
           ebin_dir = [],
           src_dir  = [],
           runmods  = [],                  % runmods for entire server
           keepalive_timeout    = 30000,
           keepalive_maxuses    = nolimit, % nolimit or non negative integer
           max_num_cached_files = 400,
           max_num_cached_bytes = 1000000, % 1 MEG
           max_size_cached_file = 8000,
           max_connections      = nolimit, % max number of TCP connections

           %% Override default connection handler processes spawn options for
           %% performance/memory tuning.
           %% [] | [{fullsweep_after,Number}, {min_heap_size, Size}]
           %% other options such as monitor, link are ignored.
           process_options = [],

           large_file_chunk_size = 10240,
           mnesia_dir            = [],
           log_wrap_size         = 1000000,  % wrap logs after 1M
           cache_refresh_secs    = 30,       % seconds  (auto zero when debug)
           include_dir           = [],       % list of inc dirs for .yaws files
           phpexe = "/usr/bin/php-cgi",      % cgi capable php executable

           yaws,                % server string
           id = "default",      % string identifying this instance of yaws

           enable_soap = false, % start yaws_soap_srv iff true

           %% a list of
           %% {{Mod, Func}, WsdlFile, Prefix} | {{Mod, Func}, WsdlFile}
           %% automatically setup in yaws_soap_srv init.
           soap_srv_mods = [],

           acceptor_pool_size = 8,             % size of acceptor proc pool

           mime_types_info,                    % undefined | #mime_types_info{}
           nslookup_pref = [inet],             % [inet | inet6]
           ysession_mod = yaws_session_server, % storage module for ysession
           ysession_cookiegen,                 % ysession cookie generation module
           ysession_idle_timeout = 2*60*1000,  % default 2 minutes
           ysession_long_timeout = 60*60*1000, % default 1 hour

           sni = disable % disable | enable | strict
          }).

    </verbatim>
  </div>

  <p>The structure is derived from the /etc/yaws/yaws.conf file and is passed
  around all through the functions in the server.
  </p>

  <h3> The <tt>#sconf{}</tt> record</h3>
  <p>The next important datastructure is the <tt>#sconf{}</tt> record. It
  is used to describe a single virtual server.
  <p>Each:
  </p>
  <p>
  <verbatim>
    <server>
      .....
    </server>
  </verbatim>
  </p>
  <p>In the /etc/yaws/yaws.conf file corresponds to one <tt>#sconf{}</tt>
  record. We have: </p>

  <div class="box">
    <verbatim>

 %% server conf
 -record(sconf, {
           port = 8000,                  % which port is this server listening to
           flags = ?SC_DEF,
           redirect_map=[],              % a list of
                                         % {Prefix, #url{}, append|noappend}
                                         % #url{} can be partially populated

           rhost,                        % forced redirect host (+ optional port)
           rmethod,                      % forced redirect method
           docroot,                      % path to the docs
           xtra_docroots = [],           % if we have additional pseudo docroots
           listen = [{127,0,0,1}],       % bind to this IP, {0,0,0,0} is possible
           servername = "localhost",     % servername is what Host: header is
           serveralias = [],             % Alternate names for this vhost
           yaws,                         % server string for this vhost
           ets,                          % local store for this server
           ssl,                          % undefined | #ssl{}
           authdirs = [],                % [{docroot, [#auth{}]}]
           partial_post_size = 10240,

           %% An item in the appmods list  can be either of the
           %% following, this is all due to backwards compat issues.
           %% 1.  an atom - this is the equivalent to {atom, atom}
           %% 2 . A two tuple {Path, Mod}
           %% 3 A three tuple {Path, Mod, [ExcludeDir ....]}
           appmods = [],

           expires = [],
           errormod_401 = yaws_outmod,   % the default 401 error module
           errormod_404 = yaws_outmod,   % the default 404 error module
           errormod_crash = yaws_outmod, % use the same module for crashes
           arg_rewrite_mod = yaws,
           logger_mod = yaws_log,        % access/auth logging module
           opaque = [],                  % useful in embedded mode
           start_mod,                    % user provided module to be started
           allowed_scripts = [yaws,php,cgi,fcgi],
           tilde_allowed_scripts = [],
           index_files = ["index.yaws", "index.html", "index.php"],
           revproxy = [],
           soptions = [{listen_opts, [{backlog, 1024}]}],
           extra_cgi_vars = [],
           stats,                        % raw traffic statistics
           fcgi_app_server,              % FastCGI application server {host,port}
           php_handler = {cgi, "/usr/bin/php-cgi"},
           shaper,
           deflate_options,              % undefined | #deflate{}
           mime_types_info,              % undefined | #mime_types_info{}
                                         % if undefined, global config is used
           dispatch_mod                  % custom dispatch module
          }).

    </verbatim>
  </div>

  <p>Both of these two structures are defined in "yaws.hrl"</p>

  <p>Now we're ready to describe the process structure. We have:</p>

  <img src="process_tree.png" />

  <p>Thus, all the different "servers" defined in the configuration
  file are clumped together in groups. For HTTP (i.e. not HTTPS) servers
  there can be multiple virtual servers per IP address. Each group is
  defined by the pair <tt>{IpAddr, Port}</tt> and they all need to
  have different server names.</p>
  <p>The client will send the server name in the "Host:" header and that
  header is used to pick a <tt>#sconf{}</tt> record out of the list
  of virtual servers for a specific <tt>{Ip,Port}</tt> pair.
  </p>

  <p>SSL servers are different, we cannot read the headers before we
  decide which virtual server to choose because the certificate is connected
  to a server name. Thus, there can only be one HTTPS server per
  <tt>{Ip,Port}</tt> pair.




</div>


<erl>
out(A) -> {ssi, "END2",[],[]}.
</erl>

