Since mlds_to_il.m no-longer exists I regenerated these lists. I sorted
them by number of lines excluding comment and blank lines. I'm not sure
what had been used previously.
tools/speedtest:
As above.
With almost 6000 lines, mercury_to_mercury.m was one of the biggest modules
of compiler, but it was far from cohesive. This diff carves seven new modules
out of it, each of which is much more cohesive. The stuff remaining in
mercury_to_mercury.m is still not as cohesive as one would like, but it is
now small enough that moving its individually-cohesive parts into modules
of their own would be overkill.
Three consequences of the old mercury_to_mercury.m's lack of cohesion
were that
- the order of predicate declarations often did not match the order of
their implementation;
- related predicates were not grouped together;
- even when they were grouped together, the order of those groups
was often random.
This diff fixes all three of these problems for all eight successor modules
of mercury_to_mercury.m: the seven new modules, and the new
mercury_to_mercury.m itself.
In some cases, this diff adds or improves the documentation of the predicates
in mercury_to_mercury.m's successor modules. In some other cases, it just
documents the lack of documentation :-(. In yet other cases, it removes
"documentation" that says nothing that isn't obvious from the predicate's name.
There are some algorithmic changes, but they are all trivial.
compiler/parse_tree_out.m:
New module containing the code to print out the top levels of parse trees,
including most sorts of items.
compiler/parse_tree_out_clause.m:
New module containing the code to print out clauses and goals.
compiler/parse_tree_out_pragma.m:
New module containing the code to print out pragmas.
compiler/parse_tree_out_pred_decl.m:
New module containing the code to print out predicate, function and
mode declarations. It is separate from parse_tree_out.m because a
significant number of compiler modules need only its functionality,
and not parse_tree_out.m's functionality.
compiler/parse_tree_out_inst.m:
New module containing the code to print out insts and modes.
compiler/parse_tree_out_term.m:
New module containing the code to print out variables and terms.
compiler/parse_tree_out_info.m:
New module containing the infrastructure of both mercury_to_mercury.m
and the other new modules.
compiler/parse_tree.m:
Include the new modules.
compiler/notes/compiler_design.html:
Document the new modules.
compiler/Mercury.options:
Transfer an option from mercury_to_mercury.m to the successor module
that needs it.
compiler/*.m:
Import one of the new modules either as well as, or instead of,
mercury_to_mercury.m. In most cases, we need to import only one
or two of mercury_to_mercury.m's successor modules; nowhere do we
need to import all eight.
Clean up some code in termination.m around a call to one of the
new modules.
tools/speedtest:
Replace mercury_to_mercury.m on the list of the ten largest modules
of the compiler.
compiler/polymorphism.m:
Remove a field in poly_infos that wasn't being used. (It contained
the pred_info.)
Put the remaining fields into a more logical order.
Remove unused setter predicates for read-only fields.
For some setter predicates of the poly_info, the value being written
is frequently the same as the old value in the written-to field.
Make these setter predicates not do the update if it is unnecessary.
Provide a predicate to set the rtti_varmaps at the same time as the
varset and vartypes, since these are usually updated together, and
this way we can avoid generating the intermediate poly_info.
If we know in advance what the name of a new type_info variable should be,
create it directly with that name; don't created with a different name,
just to rename it immediately after.
Give some predicates and variables more meaningful names.
tools/speedtest:
Add an option, -m (for mix), that executes each under-test version
of the compiler once before going back to repeat. This minimizes
the probability that automatic speed throttling of the CPU (due
e.g. to overheating) will systematically skew the timing results.
If the executions of the different versions being tested is mixed
together, any skew from throttling will have effects that are
mostly random, not systematic.
Estimated hours taken: 0.5
Branches: main
tools/speedtest:
Allow the caller to specify what directory the tested command should be
executed in.
Convert to four-space indentation.
Wrap all shell variable references.
Estimated hours taken: 2
Branches: main
compiler/mercury_compile_llds_back_end.m:
Add progress messages that say when the compiler is converting the
information it gathered for the construction of layout structures
into the actual layout structures. In some cases (such as zm_enums.m),
these could take a very long time.
compiler/stack_layout.m:
Impose limits on how aggressive we want the compression of layout
structures to be. This is because in the absence of those limits,
the compression of the layout structures for zm_enums.m can literally
take an hour.
compiler/options.m:
Implement a new option controlling the limit to be applied by
stack_layout.m.
doc/user_guide.texi.m:
Document the new option.
tools/speedtest:
More changes to allow this diff to be tested for its performance
effects. With the default limit, the compiler compiles the 16 largest
modules in a debug grade 10% faster, but generates .c files that total
about 8.4% bigger. Both effects should be smaller for smaller modules;
small modules won't see any change at all.
Estimated hours taken: 1
Branches: main
This change improves the worst-case performance of the compiler when asked to
compile programs in debug grades. In the usual case, as measured by speedtest
when compiling for the debug grade, the result is a very small improvement
(0.3%), which is in the noise.
compiler/continuation_info.m:
When generating the information for label layout structures,
do not gather information about the instantiation states of variables.
There is no point in doing so, since the debugger ignores this
information, and in some cases (zm_enums.m from g12 in particular),
the groundness checks we used to do are a nontrivial bottleneck,
because (a) there are lots of insts, and (b) they can be huge.
Replace some nondet code with more efficient det code.
compiler/llds.m:
Change the representation of variables in label layout structures
to ignore insts.
compiler/llds_out_instr.m:
compiler/trace_gen.m:
Conform to the change in llds.m.
tools/speedtest:
Generalize the options handling to allow the specification of any grade
for the test.
compiler/implicit_parallelism.m
The implicit parallelism transformation emits a warning if it cannot match
feedback data to the program being compiled. With the default
--halt-at-warn this aborts compilation which is impractical since the user
cannot easily control the compiler's ability to honour the feedback data.
For example, the internal representation of the program may be different in
for profiling builds compared to release builds, even with similar
compilation options.
Therefore this warning is now informational and it does not cause
compilation to abort.
tools/speedtest:
Add a new command line option -1. This causes the speedtest script to run
the compiler against a single module only (typecheck.m). This is useful
for generating representative Deep.data files for automatic
parallelisation.
Estimated hours taken: 12
Branches: main
Change the way the typechecker iterates over the predicates of the program.
We used to do it by looking up each predicate in the module_info,
typechecking it, and putting it back into the module_info. We now do it
by converting the predicate table into a list, iterating over the list
transforming each pred_info in it, converting the updated list back to
a predicate table.
The original intention of this change was to allow different predicates
to be typechecked in parallel by removing a synchronization bottleneck:
the typechecking of a predicate now doesn't have to wait for the typechecking
of the previous predicate to generate the updated version of the module_info.
However, it turned out that the change is good for sequential execution
as well, improving the time on tools/speedtest from 11.33 seconds to 11.08
seconds, a speedup of 2.2%. On tools/speedtest -l, which tests the compilation
of more modules, the speedup is even better: 3.1% (from 32.63 to 31.60s).
compiler/typecheck.m:
Implement the above change.
compiler/hlds_module.m:
compiler/pred_table.m:
Add a new operation, setting the list of valid pred_ids, now needed by
typecheck.m, to both modules.
Make the names of the predicates for accessing the predicate table
more expressive, and make them conform to our naming conventions.
compiler/*.m:
Trivial changes to conform to the change in hlds_module.m.
library/assoc_list.m:
Add new predicates used by the new version of typecheck.m
(at some time in its development).
NEWS:
Mention the new predicates.
library/list.m:
Improve documentation that is now copied to assoc_list.m.
tools/speedtest:
Make the test command more easily configurable.
Estimated hours taken: 0.2
Branches: main
tools/speedtest:
Make the default test the compilation of the six biggest modules in the
compiler, since this is now needed for reasonable runtimes.
Make the compilation grade explicit, since different machines have
different defaults.
tools/make_arena:
Make the six biggest modules available in the arena.
Estimated hours taken: 6
Branches: main
Add a capability for measuring statistics about the stack frame usage
of a program.
configure.in:
Find out the symbolic names of the max values for the 16 and 32 bit
integer types.
runtime/mercury_conf.h.in:
Include MR_{UINT,INT}_LEAST{16,32}_MAX among the macros whose values
are determined by autoconfiguration.
runtime/mercury_conf_param.h:
Document MR_STACK_FRAME_STATS as a macro whose definition causes the
program to collect statistics on stack frame sizes.
Fix an obsolete reference: PARALLEL was renamed MR_THREAD_SAFE a long
time ago.
Document the incompatibility between MR_THREAD_SAFE and the statistics
collection flags.
runtime/mercury_stacks.h:
If MR_STACK_FRAME_STATS is defined, then record statistics every time
we create a stack frame.
runtime/mercury_stacks.c:
Define the global variables and functions needed for stack frame
statistics.
runtime/mercury_heap_profile.h:
runtime/mercury_dword.h:
Move the macros for managing 64-bit counters from
mercury_heap_profile.h to a new header file, mercury_dword.h,
since mercury_stacks.h now needs such counters too.
Rewrite the macros to make fewer assumptions, using MR_int_least64_t
and MR_int_least32_t instead of "long long" and "int".
Add expression-like forms of some of the macros for use in
mercury_stacks.h.
Rename the type MR_dword as MR_Dword.
runtime/mercury_heap_profile.c:
#include the new header file.
runtime/mercury_prof.c:
Conform to the change from MR_dword to MR_Dword.
runtime/Mmakefile:
Add the new header file to the list of header files.
runtime/mercury_wrapper.c:
If MR_STACK_FRAME_STATS is defined, initialize the stack frame stats
before execution starts and write out the statistics when execution
ends.
tools/speedtest:
Add an option that when set, causes the script to report stack frame
stats for each variant being tested.
Estimated hours taken: 0.1
Branches: main
tools/speedtest:
When preserving deep profiling data files, use a naming scheme that
allows later invocations of speedtest to distinguish them from the
saved compilers.
Estimated hours taken: 500
Branches: main
Implement deep profiling; merge the changes on the deep2 branch back
onto the trunk.
The main documentation on the general architecture of the deep profiler
is the deep profiling paper.
doc/user_guide.texi:
Document how to use the deep profiler.
deep_profiler:
deep_profiler/Mmakefile:
A new directory holding the deep profiler and its mmakefile.
Mmakefile:
Add targets for the new directory.
Add support for removing inappropriate files from directories.
deep_profiler/interface.m:
The deep profiler consists of two programs: mdprof_cgi.m, which acts
as a CGI "script", and mdprof_server.m, which implements the server
process that the CGI script talks to. Interface.m defines the
interface between them.
script/mdprof.in:
A shell script template. ../configure uses it to generate mdprof,
which is a wrapper around mdprof_cgi that tells it how to find
mdprof_server.
deep_profiler/mdprof_cgi.m:
The CGI "script" program.
deep_profiler/mdprof_server.m:
The top level predicates of the server.
deep_profiler/profile.m:
The main data structures of the server and their operations.
deep_profiler/read_profile.m:
Code for reading in profiling data files.
deep_profiler/startup.m:
Code for post-processing the information in profiling data files,
propagating costs from procedures to their ancestors and performing
various kinds of summaries.
deep_profiler/server.m:
Code for responding to requests from the CGI script.
deep_profiler/cliques.m:
Code to find cliques in graphs.
deep_profiler/array_util.m:
deep_profiler/util.m:
Utility predicates.
deep_profiler/dense_bitset.m:
An implementation of (part of) the set ADT with dense bit vectors.
deep_profiler/measurements.m:
Operations on profiling measurements.
deep_profiler/timeout.m:
An implementation of a timeout facility.
deep_profiler/conf.m:
Functions that depend on autoconfigured settings.
configure.in:
Find out what command to use to find the name of the local host.
Install deep profiling versions of the standard library along with the
other profiling versions.
runtime/mercury_conf.h.in:
Add some macros for deep_profiler/conf.m to use.
library/profiling_builtin.m:
runtime/mercury_deep_call_port_body.h:
runtime/mercury_deep_leave_port_body.h:
runtime/mercury_deep_redo_port_body.h:
A new library module that implements deep profiling primitives.
Some of these primitives have many versions, whose common code is
factor is factored out in three new include files in the runtime.
compiler/deep_profiling.m:
New module to perform the program transformations described in the
paper.
compiler/notes/compiler_design.html:
Document the new compiler module.
compiler/mercury_compiler.m:
Invoke the new module in deep profiling grades. Allow global static
data to be generated by deep_profiling.m.
compiler/options.m:
Add options to turn on deep profiling and (for benchmarking purposes)
control its implementation.
Add an optiooption disable tailcall optimization in the LLDS backend,
to help benchmarking deep profiling.
compiler/jumpopt.m:
compiler/optimize.m:
Obey the option to disable tailcalls.
compiler/handle_options.m:
Handle the implications of deep profiling.
compiler/modules.m:
In deep profiling grades, automatically import profiling_builtin.m.
compiler/prog_util.m:
doc/Makefile:
library/library.m:
Handle the new builtin module.
compiler/export.m:
In deep profiling grades, wrap deep profiling code around exported
procedures to handle the "unscheduled call" aspects of callbacks to
Mercury from the foreign language.
compiler/higher_order.m:
profiler/demangle.m:
util/demangle.c:
When creating a name for a higher-order-specialized predicate, include
the mode number in the name.
compiler/add_trail_ops.m:
compiler/type_util.m:
Move c_pointer_type from add_trail_ops to type_util, so it can also be
used by deep_profiling.m.
compiler/hlds_goal.m:
Add a new goal feature that marks a tail call, for use by
deep_profiling.m.
compiler/hlds_pred.m:
Add a new field to proc_info structures for use by deep_profiling.m.
Add a mechanism for getting proc_ids for procedure clones.
Remove next_proc_id, an obsolete and unused predicate.
compiler/hlds_data.m:
Add a new cons_id to refer to the proc_static structure of a procedure.
compiler/bytecode_gen.m:
compiler/code_util.m:
compiler/dependency_graph.m:
compiler/hlds_out.m:
compiler/mercury_to_mercury.m:
compiler/ml_unify_gen.m:
compiler/opt_debug.m:
compiler/prog_rep.m:
compiler/rl_exprn.m:
compiler/switch_util.m:
compiler/unify_gen.m:
Trivial changes to handle the new cons_id, goal feature and/or
proc_info argument.
compiler/rtti.m:
Add a utility predicate for extracting pred_id and proc_id from an
rtti_proc_label, for use by hlds_out.m
compiler/layout.m:
compiler/layout_out.m:
compiler/llds.m:
compiler/llds_common.m:
Add support for proc_static and call_site_static structures.
compiler/layout_out.m:
compiler/llds_out.m:
Add code for the output of proc_static structures.
compiler/code_util.m:
Make code_util__make_proc_label_from_rtti a function, and export it.
util/mkinit.c:
compiler/llds_out.m:
compiler/layout.m:
compiler/modules.m:
Add support for a fourth per-module C function, for writing out
proc_static structures (and the call_site_static structures they
contains).
Since proc_static structures can be referred to from LLDS code (and not
just from other static structures and compiler-generated C code),
reorganize the declarations of static structures slightly.
Change the schema for the name of the first per-module C function
slightly, to make it the addition of the fourth function easier.
The scheme now is:
mercury__<modulename>__init
mercury__<modulename>__init_type_tables
mercury__<modulename>__init_debugger
mercury__<modulename>__write_out_proc_statics
Improve formatting of the generated C code.
library/*.m:
runtime/mercury.c:
runtime/mercury_context.c:
runtime/mercury_engine.c:
runtime/mercury_ho_call.c:
runtime/mercury_tabling.c:
runtime/mercury_trace_base.c:
runtime/mercury_wrapper.c:
trace/mercrury_trace.[ch]:
trace/mercrury_trace_declarative.c:
trace/mercrury_trace_external.c:
trace/mercrury_trace_internal.c:
Conform to the new scheme for initialization functions for hand-written
modules.
compiler/mercury_compile.m:
library/benchmarking.m:
runtime/mercury_conf_param.h:
runtime/mercury.h:
runtime/mercury_engine.c:
runtime/mercury_goto.c:
runtime/mercury_grade.h:
runtime/mercury_ho_call.c:
runtime/mercury_label.[ch]:
runtime/mercury_prof.[ch]:
Add an MR_MPROF_ prefix in front of the C macros used to control the
old profiler.
compiler/handle_options.m:
runtime/mercury_grade.h:
scripts/canonical_grade.sh-subr:
scripts/init_grade_options.sh-subr:
scripts/parse_grade_options.sh-subr:
Make deep profiling completely separate from the old profiling system,
by making the deep profiling grade independent of MR_MPROF_PROFILE_TIME
and the compiler option --profile-time.
library/array.m:
library/builtin.m:
library/std_util.m:
runtime/mercury_hand_unify_body.h:
runtime/mercury_hand_compare_body.h:
In deep profiling grades, wrap the deep profiling call, exit, fail
and redo codes around the bodies of hand-written unification
and comparison procedures.
Make the reporting of array bounds violations switchable between
making them fatal errors, as we currently, and reporting them by
throwing an exception. Throwing an exception makes debugging code
using arrays easier, but since exceptions aren't (yet) propagated
across engine boundaries, we keep the old behaviour as the default;
the new behaviour is for implementors.
runtime/mercury_deep_profiling_hand.h:
New file that defines macros for use in Mercury predicates whose
definition is in hand-written C code.
library/exception.m:
runtime/mercury_exception_catch_body.h:
runtime/mercury_stacks.h:
In deep profiling grades, wrap the deep profiling call, exit, fail
and redo codes around the bodies of the various modes of builtin_catch.
Provide a function that C code can use to throw exceptions.
library/benchmarking.m:
library/exception.m:
library/gc.m:
library/std_util.m:
runtime/mercury_context.[ch]:
runtime/mercury_engine.[ch]:
runtime/mercury_debug.c:
runtime/mercury_deep_copy.c:
runtime/mercury_overflow.h:
runtime/mercury_regs.h:
runtime/mercury_stacks.h:
runtime/mercury_thread.c:
runtime/mercury_wrapper.c:
Add prefixes to the names of the fields in the engine and context
structures, to make code using them easier to understand and modify.
runtime/mercury_deep_profiling.[ch]:
New module containing support functions for deep profiling and
functions for writing out a deep profiling data file at the end of
execution.
runtime/mercury_debug.[ch]:
Add support for debugging deep profiling.
Add support for watching the value at a given address.
Make the buffered/unbuffered nature of debugging output controllable
via the -du option.
Print register contents only if -dr is specified.
runtime/mercury_goto.h:
runtime/mercury_std.h:
Use the macros in mercury_std.h instead of defining local variants.
runtime/mercury_goto.h:
runtime/mercury_stack_layout.h:
runtime/mercury_stack_trace.c:
runtime/mercury_tabling.c:
trace/mercury_trace.c:
trace/mercury_trace_declarative.c:
trace/mercury_trace_external.c:
trace/mercury_trace_vars.c:
Standardize some of the macro names with those used in the debugger
paper.
runtime/mercury_heap.h:
Add support for memory profiling with the deep profiler.
runtime/mercury_prof.[ch]:
runtime/mercury_prof_time.[ch]:
Move the functionality that both the old profiler and the deep profiler
need into the new module mercury_prof_time. Leave mercury_prof
containing stuff that is only relevant to the old profiler.
runtime/mercury_prof.[ch]:
runtime/mercury_strerror.[ch]:
Move the definition of strerror from mercury_prof to its own file.
runtime/mercury_wrapper.[ch]:
Add support for deep profiling.
Add suppory for controlling whether debugging output is buffered or
not.
Add support for watching the value at a given address.
runtime/Mmakefile:
Mention all the added files.
scripts/mgnuc.in:
Add an option for turning on deep profiling.
Add options for controlling the details of deep profiling. These
are not documented because they are intended only for benchmarking
the deep profiler itself, for the paper; they are not for general use.
tools/bootcheck:
Compile the deep_profiler directory as well as the other directories
containing Mercury code.
Turn off the creation of deep profiling data files during bootcheck,
since all but one of these in each directory will be overwritten
anyway.
Add support for turning on --keep-objs by default in a workspace.
tools/speedtest:
Preserve any deep profiling data files created by the tests.
trace/mercury_trace.c:
Trap attempts to perform retries in deep profiling grades, since they
would lead to core dumps otherwise.
util/Mmakefile:
Avoid compile-time warnings when compiling getopt.
tests/*/Mmakefile:
tests/*/*/Mmakefile:
In deep profiling grades, switch off the tests that test features
that don't work with deep profiling, either by design or because
the combination hasn't been implemented yet.
Estimated hours taken: 2
Delete the trial directory, since its contents cannot be compiled anymore.
trial/*:
Delete these files.
tools/make_arena:
A new shell script that creates a directory "arena" that contains
some compiler source files and the interface files you need to compile
them. You can now do speed tests in this directory.
tools/speedtest:
Do speed tests in the arena directory, not the trial directory.
Estimated hours taken: 5
Improvements to the infrastructure for debugging code generator changes.
tools/binary:
If either stage2.ok or stage2.bad is missing object files, then do not
complain: instead, recreate the missing object files.
Fix a bug: copy the library's .pic_o files together with its .o files.
Fix a bug: make sure that we link *all* the possible relevant .init
files from the runtime to the stage2 directory.
If the search narrows down to one file, and the script is trying to
find out which part of the file is in error, then consider all the
parts that are identical between the stage2.ok and stage2.bad to be
known good from the start. This can reduce the number of bootchecks
needed by one or two.
tools/binary:
tools/binary_step:
Allow the test to be the successful compilation of the stage 3 library
directory. (In almost all cases, bad stage 2 compilers fail while
compiling the stage 3 library. By not compiling the stage 3 compiler
if the compilation of the stage 3 library succeeds, we can save a lot
of time.)
If the search narrows down to one file, and the script is trying to
find out which part of the file is in error, watch out for errors that
prevent the stage 2 executable from being built. If such an error
occurs, then stop right then and there. In such cases, there is no
point in further binary search, since each further invocation of
binary_step will just indicate that the source files in stage2.bad and
stage2.ok are not compatible (e.g. because they do not use the same
mapping from static term numbers to static term contents.)
tools/binary_step:
Reduce the time for declaring a program to be in an infinite loop,
since the slowest machine we may want to use is faster now.
tools/makebatch:
Fix some glaring bugs: e.g. test uses -lt, not < for comparisons.
Add an option, --stop-at-failure, that stops makebatch if a bootcheck
fails and thus preserves the stage[23] directories involved in the
failure. This allows one to verify that a change works in many grades
without sacrificing the ability to debug any problems.
Add another option --c-files, that gathers the C files created
in each bootcheck. This allows the C files to be compared, e.g.
for efficiency problems.
Estimated hours taken: 1
tools/speedtest:
Add two new options, one which tests speed when the program is run
under the debugger, and one which records the size of the executable.
Also fix an old bug: there is no option -N.
Estimated hours taken: 2
Rename `mc' as `mmc'.
Note: we do not change the names of variables such as MCFLAGS, just the
`mc' executable.
configure.in:
Look for mmc not mc. If you can't find mmc to bootstrap, try for
mc.
bindist/Mmakefile:
bindist/bindist.configure.in:
doc/Mmakefile:
doc/user_guide.texi:
scripts/Mmake.vars.in:
scripts/Mmakefile:
scripts/msl.in:
tools/expand_params:
tools/optstages:
tools/speedtest:
tools/test_mercury:
Change references to mc into mmc.
Estimated hours taken: 0.1
tools/speedtest:
Since most of the time spent in compiling tree234 is spent in
an a presumably atypical part of mode analysis, use the make_hlds.m
as the file to compile when testing the speed of the compiler.
trial/make_hlds.m:
The current version of make_hlds.m, frozen for use in speed trials.
Estimated hours taken: 0.1
speedtest:
Add a workaround for the Suns, where "echo -n" works the proper
(Berkeley) way only when echo is /usr/ucb echo. Don't insist on
the .param file being present.
Estimated hours taken: 1
dotime:
A csh script that uses csh's time primitive to measure the time
taken by the command. This is useful because sh's time primitive
sucks.
speedtest:
A script to measure the speed of a batch of versions of the compiler
by using each version to execute some compilation command.
