OUTLINE

The main job of the compiler is to translate Mercury into C, although it can also translate (subsets of) Mercury to some other languages (Goedel, the bytecode of a debugger currently under development, and in the future the Aditi Relational Language).

The top-level of the compiler is in the file mercury_compile.m. The basic design is that compilation is broken into the following stages:

1. parsing (source files -> HLDS)
2. semantic analysis and error checking (HLDS -> annotated HLDS)
3. high-level transformations (annotated HLDS -> annotated HLDS)
4. code generation (annotated HLDS -> LLDS)
5. low-level optimizations (LLDS -> LLDS)
6. output C code (LLDS -> C)

Note that in reality the separation is not quite as simple as that. Although parsing is listed as step 1 and semantic analysis is listed as step 2, the last stage of parsing actually includes some semantic checks. And although optimization is listed as steps 3 and 5, it also occurs in steps 2, 4, and 6. For example, elimination of assignments to dead variables is done in mode analysis; middle-recursion optimization and the use of static constants for ground terms is done in code generation; and a few low-level optimizations are done in llds_out.m as we are spitting out the C code.

DETAILED DESIGN

The action is co-ordinated from mercury_compile.m.

0. Option handling

The command-line options are defined in the module options.m. mercury_compile.m calls library/getopt.m, passing the predicates defined in options.m as arguments, to parse them. It then invokes handle_options.m to postprocess the option set. The results are stored in the io__state, using the type globals defined in globals.m.

1. Parsing

• lexical analysis (library/lexer.m)
• stage 1 parsing - convert strings to terms.
  library/parser.m contains the code to do this, while library/term.m and library/varset.m contain the term and varset data structures that result, and predicates for manipulating them.
• stage 2 parsing - convert terms to `items' (declarations, clauses, etc.)
  The result of this stage is a parse tree that has a one-to-one correspondence with the source code. The parse tree data structure definition is in prog_data.m, while the code to create it is in prog_io.m and its submodules prog_io_dcg.m (which handles clauses using Definite Clause Grammar notation), prog_io_goal.m (which handles goals), prog_io_pragma.m (which handles pragma declarations), prog_io_typeclass.m (which handles typeclass and instance declarations) and prog_io_util.m (which defines predicates and types needed by the other prog_io*.m modules. The data structure for insts is stored in its own module, inst.m.

  The modules prog_out.m and mercury_to_mercury.m contain predicates for printing the parse tree. prog_util.m contains some utility predicates for manipulating the parse tree.

• imports and exports are handled at this point (modules.m)
  modules.m has the code to write out `.int', `.int2', `.int3', `.d' and `.dep' files.
• module qualification of types, insts and modes
  module_qual.m -
  Adds module qualifiers to all types insts and modes, checking that a given type, inst or mode exists and that there is only possible match. This is done here because it must be done before the `.int' and `.int2' interface files are written. This also checks whether imports are really needed in the interface.
  Notes on module qualification:
  • all types, typeclasses, insts and modes occuring in pred, func, type, typeclass and mode declarations are module qualified by module_qual.m.
  • all types, insts and modes occuring in lambda expressions and explicit type qualifications are module qualified in make_hlds.m.
  • constructors occuring in predicate and function mode declarations are module qualified during type checking.
  • predicate and function calls and constructors within goals are module qualified during mode analysis.
  • predicate and function names in typeclass instance declarations are qualified in check_typeclass.m (after mode analysis).
• reading and writing of optimization interfaces (intermod.m).
  <module>.opt contains clauses for exported preds suitable for inlining or higher-order specialization. The .opt file for the current module is written after type-checking. .opt files for imported modules are read here.
• expansion of equivalence types (equiv_type.m)
  This is really part of type-checking, but is done on the item_list rather than on the HLDS because it turned out to be much easier to implement that way.
• conversion to superhomogeneous form and into HLDS
  make_hlds.m transforms the code into superhomogeneous form, and at the same time converts the parse tree into the HLDS. It converts `pragma import' and `pragma c_code' declarations into clauses with HLDS `pragma_c_code' instructions for bodies. make_hlds.m also calls make_tags.m which chooses the data representation for each discriminated union type by assigning tags to each functor.

The result at this stage is the High Level Data Structure, which is defined in four files:

1. hlds_data.m defines the parts of the HLDS concerned with function symbols, types, insts, modes and determinisms;
2. hlds_goal.m defines the part of the HLDS concerned with the structure of goals, including the annotations on goals;
3. hlds_pred.m defines the part of the HLDS concerning predicates and procedures;
4. hlds_module.m defines the top-level parts of the HLDS, including the type module_info.

2. Semantic analysis and error checking

Any pass which can report errors or warnings must be part of this stage, so that the compiler does the right thing for options such as `--halt-at-warn' (which turns warnings into errors) and `--error-check-only' (which makes the compiler only compile up to this stage).

implicit quantification

quantification.m handles implicit quantification and computes the set of non-local variables for each sub-goal

type checking

• typecheck.m handles type checking, overloading resolution & module name resolution, and almost fully qualifies all predicate and functor names. It sets the map(var, type) field in the pred_info. However, typecheck.m doesn't figure out the pred_id for function calls or calls to overloaded predicates; that can't be done in a single pass of typechecking, and so it is done later on (in post_typecheck.m, for preds, and in modecheck_unify.m, for function calls). Typeclass constraints are checked here, and any redundant constraints that are eliminated are recorded (as constraint_proofs) in the pred_info for future reference. When it has finished, typecheck.m calls clause_to_proc.m to make duplicate copies of the clauses for each different mode of a predicate; all later stages work on procedures, not predicates.
• type_util.m contains utility predicates dealing with types that are used in a variety of different places within the compiler
• post_typecheck.m may also be considered to logically be a part of typechecking, but it is actually called from purity analysis (see below). It contains the stuff related to type checking that can't be done in the main type checking pass.

purity analysis

purity.m is responsible for purity checking, as well as defining the purity type and a few public operations on it. It also calls post_typecheck.m to complete the handling of predicate overloading for cases which typecheck.m is unable to handle, to check for unbound type variables, and to copy the clauses to the proc_infos in preparation for mode analysis.

mode analysis

• modes.m is the main mode analysis module. It checks that the code is mode-correct, reordering it if necessary, and annotates each goal with a delta-instmap that specifies the changes in instantiatedness of each variable over that goal.
• modecheck_unify.m is the sub-module which analyses unification goals. It also converts higher-order pred terms into lambda expressions and module qualifies data constructors.
• modecheck_call.m is the sub-module which analyses calls. It also converts function calls into predicate calls.

  The following sub-modules are used:

  mode_info.m

  (the main data structure for mode analysis)

  delay_info.m

  (a sub-component of the mode_info data structure used for storing the information for scheduling: which goals are currently delayed, what variables they are delayed on, etc.)

  instmap.m

  Defines the instmap and instmap_delta ADTs which store information on what instantiations a set of variables may be bound to.

  inst_match.m

  This contains the code for examining insts and checking whether they match.

  inst_util.m

  This contains the code for creating new insts from old ones: unifying them, merging them and so on.

  mode_errors.m

  This module contains all the code to print error messages for mode errors

• mode_util.m contains miscellaneous useful predicates dealing with modes (many of these are used by lots of later stages of the compiler)
• mode_debug.m contains utility code for tracing the actions of the mode checker.

indexing and determinism analysis

• switch_detection.m transforms into switches those disjunctions in which several disjuncts test the same variable against different function symbols.
• cse_detection.m looks for disjunctions in which each disjunct tests the same variable against the same function symbols, and hoists any such unifications out of the disjunction. If cse_detection.m modifies the code, it will re-run mode analysis and switch detection.
• det_analysis.m annotates each goal with its determinism; it inserts cuts in the form of "some" goals wherever the determinisms and delta instantiations of the goals involved make it necessary. Any errors found during determinism analysis are reported by det_report.m. Det_util.m contains utility predicates used in several modules.

checking of unique modes (unique_modes.m)

unique_modes.m checks that non-backtrackable unique modes were not used in a context which might require backtracking. Note that what unique_modes.m does is quite similar to what modes.m does, and unique_modes calls lots of predicates defined in modes.m to do it.

checking typeclass instances (check_typeclass.m)

check_typeclass.m checks that, each instance declaration, that the types, modes and determinism of each predicate/function that is a method of the class is correct (ie. that it matches the typeclass declaration). This pass is performed at the end of semantic analysis because it needs mode and determinism information. In this pass, pred_ids and proc_ids are assigned to the methods for each instance. In addition, while checking that the superclasses of a class are satisfied by the instance declaration, a set of constraint_proofs are built up for the superclass constraints. These are used by polymorphism.m when generating the base_typeclass_info for the instance.

simplification (simplify.m)

simplify.m finds and exploits opportunities for simplifying the internal form of the program, both to optimize the code and to massage the code into a form the code generator will accept. It also warns the programmer about any constructs that are so simple that they should not have been included in the program in the first place. (That's why this pass needs to be part of semantic analysis: because it can report warnings.) simplify.m calls common.m which looks for (a) construction unifications that construct a term that is the same as one that already exists, or (b) repeated calls to a predicate with the same inputs, and replaces them with assignment unifications. simplify.m also attempts to partially evaluate calls to builtin procedures if the inputs are all constants (see const_prop.m).

3. High-level transformations

The first pass of this stage does tabling transformations (table_gen.m). This involves the insertion of several calls to tabling predicates defined in mercury_builtin.m and the addition of some scaffolding structure.

The next two passes of this stage are code simplifications.

• introduction of type_info arguments for polymorphic predicates, introduction of typeclass_info arguments for typeclass-constrained predicates and transformation of complicated unifications into predicate calls (polymorphism.m)
• removal of lambda expressions (lambda.m)
  

  lambda.m converts lambda expressions into higher-order predicate terms referring to freshly introduced separate predicates. This pass needs to come after unique_modes.m to ensure that the modes we give to the introduced predicates are correct. It also needs to come after polymorphism.m since polymorphism.m doesn't handle higher-order predicate constants.

To improve efficiency, the above two passes are actually combined into one - polymorphism.m calls calls lambda__transform_lambda directly.

The next pass is termination analysis. The various modules involved are:

• termination.m is the control module. It sets the argument size and termination properties of builtin and compiler generated procedures, invokes term_pass1.m and term_pass2.m and writes .trans_opt files and error messages as appropriate.
• term_pass1.m analyzes the argument size properties of user-defined procedures,
• term_pass2.m analyzes the termination properties of user-defined procedures.
• term_traversal.m contains code common to the two passes.
• term_errors.m defines the various kinds of termination errors and prints the messages appropriate for each.
• term_util.m defines the main types used in termination analysis and contains utility predicates.

Most of the remaining HLDS-to-HLDS transformations are optimizations:

• specialization of higher-order and polymorphic predicates where the value of the higher-order/type_info/typeclass_info arguments are known (higher_order.m)
• inlining (i.e. unfolding) of simple procedures (inlining.m)
• pushing constraints as far left as possible (constraint.m); this does not yet work.
• deforestation and partial evaluation (deforest.m). This optimizes multiple traversals of data structures within a conjunction, and avoids creating intermediate data structures. It also performs loop unrolling where the clause used is known at compile time. deforest.m makes use of the following sub-modules (`pd_' stands for "partial deduction"):
  • pd_cost.m contains some predicates to estimate the improvement caused by deforest.m.
  • pd_debug.m produces debugging output.
  • pd_info.m contains a state type for deforestation.
  • pd_term.m contains predicates to check that the deforestation algorithm terminates.
  • pd_util.m contains various utility predicates.
• issue warnings about unused arguments from predicates, and create specialized versions without them (unused_args.m); type_infos are often unused
• elimination of dead procedures (dead_proc_elim.m). Inlining, higher-order specialization and the elimination of unused args can make procedures dead even the user doesn't, and automatically constructed unification and comparison predicates are often dead as well.
• elimination of useless assignments, assignments that merely introduce another name for an already existing variable (excess.m).
• reducing the number of variables that have to be saved across procedure calls (saved_vars.m). We do this by putting the code that generates the value of a variable just before the use of that variable, duplicating the variable and the code that produces it if necessary, provided the cost of doing so is smaller than the cost of saving and restoring the variable would be.

The module transform.m contains stuff that is supposed to be useful for high-level optimizations (but which is not yet used).

Eventually we plan to make Mercury the programming language of the Aditi deductive database system. When this happens, we will need to be able to apply the magic set transformation, which is defined for predicates whose definitions are disjunctive normal form. The module dnf.m translates definitions into DNF, introducing auxiliary predicates as necessary.

4. Code generation

pre-passes to annotate the HLDS

Before code generation there are a few more passes which annotate the HLDS with information used for code generation:

choosing registers for procedure arguments (arg_info.m)

Currently uses one of two simple algorithms, but we may add other algorithms later.

annotation of goals with liveness information (liveness.m)

This records the birth and death of each variable in the HLDS goal_info.

allocation of stack slots

This is done by live_vars.m, which works out which variables need to be saved on the stack when, and then uses graph_colour.m to determine a good allocation of variables to stack slots.

migration of builtins following branched structures

This transformation, which is performed by follow_code.m, improves the results of follow_vars.

allocating the follow vars (follow_vars.m)

Traverses backwards over the HLDS, annotating some goals with information about what locations variables will be needed in next. This allows us to generate more efficient code by putting variables in the right spot directly. This module is not called from mercury_compile.m; it is called from store_alloc.m.

allocating the store map (store_alloc.m)

Annotates each branched goal with variable location information so that we can generate correct code by putting variables in the same spot at the end of each branch.

code generation

For code generation itself, the main module is code_gen.m. It handles conjunctions and negations, but calls sub-modules to do most of the other work:

• ite_gen.m (if-then-elses)
• call_gen.m (predicate calls and also calls to out-of-line unification procedures)
• disj_gen.m (disjunctions)
• unify_gen.m (unifications)
• switch_gen.m (switches), which has sub-modules
  • dense_switch.m
  • lookup_switch.m
  • string_switch.m
  • tag_switch.m
• pragma_c_gen.m (embedded C code)

It also calls middle_rec.m to do middle recursion optimization.

The code generation modules make use of

code_info.m

The main data structure for the code generator

code_exprn.m

This defines the exprn_info type, which is a sub-component of the code_info data structure which holds the information about the contents of registers and the values/locations of variables.

exprn_aux.m

Various preds which use exprn_info

code_util.m

Some miscellaneous preds used for code generation

code_aux.m

Some miscellaneous preds which, unlike those in code_util, use code_info

continuation_info.m

For accurate garbage collection, collects information about each live value after calls, and saves information about procedures.

code generation for `pragma export' declarations (export.m)

This is handled seperately from the other parts of code generation. mercury_compile.m calls the procedures `export__produce_header_file' and `export__get_pragma_exported_procs' to produce C code fragments which declare/define the C functions which are the interface stubs for procedures exported to C.

The result of code generation is the Low Level Data Structure (llds.m). The code for each procedure is generated as a tree of code fragments which is then flattened (tree.m).

5. Low-level optimization

The various LLDS-to-LLDS optimizations are invoked from optimize.m. They are:

• optimization of jumps to jumps (jumpopt.m)
• elimination of duplicate code sequences (dupelim.m)
• optimization of stack frame allocation/deallocation (frameopt.m)
• filling branch delay slots (delay_slot.m)
• dead code and dead label removal (labelopt.m)
• peephole optimization (peephole.m)
• value numbering
  This is done by value_number.m, which has the following sub-modules:

  vn_block.m

  Traverse an extended basic block, building up tables showing the actions that must be taken, and the current and desired contents of locations.

  vn_cost.m

  Computes the cost of instruction sequences. Value numbering should never replace an instruction sequence with a more expensive sequence. Unfortunately, computing costs accurately is very difficult.

  vn_debug.m

  Predicates to dump data structures used in value numbering.

  vn_filter.m

  Module to eliminate useless temporaries introduced by value numbering. Not generating them in the first place would be better, but would be quite difficult.

  vn_flush.m

  Given the tables built up by vn_block and a list of nodes computed by vn_order, generate code to assign the required values to each temporary and live location in what is hopefully the fastest and most compact way.

  vn_order.m

  Given tables built up by vn_block showing the actions that must be taken, and the current and desired contents of locations, find out which shared subexpressions should have temporaries allocated to them and in what order these temporaries and the live locations should be assigned to. This module uses the module atsort.m to perform an approximate topological sort on the nodes of the location dependency graph it operations on (since the graph may have cycles, a precise topological sort may not exist).

  vn_table.m

  Abstract data type showing the current and desired contents of locations.

  vn_temploc.m

  Abstract data type to keep track of the availability of registers and temporaries.

  vn_type.m

  This module defines the types used by the other modules of the value numbering optimization.

  vn_util.m

  Utility predicates.

  vn_verify.m

  Sanity checks to make sure that (a) the optimized code computes the same values as the original code, and (b) the optimized code does not dereference tagged pointers until the tag is known. (Violations of (b) usually cause unaligned accesses, which cause bus errors on many machines.)

  Several of these modules (and also frameopt, above) use livemap.m, which finds the set of locations live at each label.

Depending on which optimization flags are enabled, optimize.m may invoke many of these passes multiple times.

Some of the low-level optimization passes use basic_block.m, which defines predicates for converting sequences of instructions to basic block format and back, as well as opt_util.m, which contains miscellaneous predicates for LLDS-to-LLDS optimization.

6. Output C code

• base_type_info.m generates the base_type_info structures that list the unification, index and compare predicates associated with each declared type constructor. These are added to the LLDS.
• base_type_layout.m generates the base_type_layout structures that give information on how to interpret values of a given type. It also creates base_type_functors structures that give information on the functors of a given type. The base_type_layout and base_type_functors structures of each declared type constructor are added to the LLDS.
• base_typeclass_info.m generates the base_typeclass_info structures that list the methods of a class for each instance declaration. These are added to the LLDS.
• stack_layout.m generates the stack_layout structures for accurate garbage collection. Tables are created from the data collected in continuation_info.m.
• llds_common.m extracts static terms from the main body of the LLDS, and puts them at the front. If a static term originally appeared several times, it will now appear as a single static term with multiple references to it.
• Final generation of C code is done in llds_out.m.

BYTECODE

The Mercury compiler can translate Mercury programs into bytecode for interpretation by the Mercury debugger currently under development. The generation of bytecode happens after semantic checks have been completed.

• bytecode.m defines the internal representation of bytecodes, and contains the predicates to emit them in two forms. The raw bytecode form is emitted into <filename>.bytecode for interpretation, while a human-readable form is emitted into <filename>.bytedebug for visual inspection.
• bytecode_gen.m contains the predicates that translate HLDS into bytecode.

MISCELLANEOUS

det_util:

This module contains utility predicates needed by the parts of the semantic analyzer and optimizer concerned with determinism.

special_pred.m, unify_proc.m:

These modules contain stuff for handling the special compiler-generated predicates which are generated for each type: unify/2, compare/3, and index/1 (used in the implementation of compare/3).

dependency_graph.m:

This contains predicates to compute the call graph for a module, and to print it out to a file. (The call graph file is used by the profiler.) The call graph may eventually also be used by det_analysis.m, inlining.m, and other parts of the compiler which could benefit from traversing the predicates in a module in a bottom-up or top-down fashion with respect to the call graph.

passes_aux.m

Contains code to write progress messages, and higher-order code to traverse all the predicates defined in the current module and do something with each one.

opt_debug.m:

Utility routines for debugging the LLDS-to-LLDS optimizations.

error_util.m:

Utility routines for printing nicely formatted error messages.

CURRENTLY USELESS

The following modules do not serve any function at the moment. Some of them are obsolete; other are work-in-progress. (For some of them its hard to say which!)

lco.m:

This finds predicates whose implementations would benefit from last call optimization modulo constructor application. It does not apply the optimization and will not until the mode system is capable of expressing definite aliasing.

mercury_to_goedel.m:

This converts from item_list to Goedel source code. It works for simple programs, but doesn't handle various Mercury constructs such as lambda expressions, higher-order predicates, and functor overloading.

mercury_to_c.m:

The very incomplete beginnings of an alternate code generator. When finished, it will convert HLDS to high-level C code (without going via LLDS).