Discussion of these changes can be found on the Mercury developers
mailing list archives from June 2018.
COPYING.LIB:
Add a special linking exception to the LGPL.
*:
Update references to COPYING.LIB.
Clean up some minor errors that have accumulated in copyright
messages.
runtime/*.[ch]:
trace/*.[chyl]:
As above. In some places, improve comments, e.g. by expanding contractions
such as "we've". Add #ifndef guards against double inclusion around
the trace/*.h files that did not already have them.
tools/*:
Make the corresponding changes in shell scripts that generate .[ch] files
in the runtime.
tests/*:
Conform to a slight change in the text of a message.
Move all .h and .c files to four-space indentation without tabs,
if they weren't there already.
Use the same vim line for all .h and .c files.
Align all backslashes at the ends of lines in macro definitions.
Align close comment signs.
In some places, fix inconsistent indentation.
Fix a bunch of comments. Add XXXs to a few of them.
Branches: main
Implement a new form of memory profiling, which tells the user what memory
is being retained during a program run. This is done by allocating an extra
word before each cell, which is used to "attribute" the cell to an
allocation site. The attribution, or "allocation id", is an address to an
MR_AllocSiteInfo structure generated by the Mercury compiler, giving the
procedure, filename and line number of the allocation, and the type
constructor and arity of the cell that it allocates.
The user must manually instrument the program with calls to
`benchmarking.report_memory_attribution', which forces a GC and summarises
the live objects on the heap using the attributions. The mprof tool is
extended with a new mode to parse and present that data.
Objects which are unattributed (e.g. by hand-written C code which hasn't
been updated) are still accounted for, but show up in profiles as "unknown".
Currently this profiling mode only works in conjunction with the Boehm
garbage collector, though in principle it can work with any memory allocator
for which we can access a list of the live objects. Since term size
profiling relies on the same technique of using an extra word per memory
cell, the two profiling modes are incompatible.
The output from `mprof -s' looks like this:
------ [1] some label ------
cells words cumul procedure / type (location)
14150 38872 total
* 1949/ 13.8% 4872/ 12.5% 12.5% <predicate `parser.parse_rest/7' mode 0>
975/ 6.9% 1950/ 5.0% list.list/1 (parser.m:502)
487/ 3.4% 1948/ 5.0% term.term/1 (parser.m:501)
487/ 3.4% 974/ 2.5% term.const/0 (parser.m:501)
* 1424/ 10.1% 4272/ 11.0% 23.5% <predicate `parser.parse_simple_term_2/6' mode 0>
708/ 5.0% 2832/ 7.3% term.term/1 (parser.m:643)
708/ 5.0% 1416/ 3.6% term.const/0 (parser.m:643)
...
boehm_gc/alloc.c:
boehm_gc/include/gc.h:
boehm_gc/misc.c:
boehm_gc/reclaim.c:
Add a callback function to be called for every live object after a GC.
Add a function to write out the GC_size_map array.
compiler/layout.m:
Define the alloc_site_info type which is equivalent to the
MR_AllocSiteInfo C structure.
Add alloc_site_array as a kind of "layout" array.
compiler/llds.m:
Add allocation sites to `cfile' structure.
Replace TypeMsg argument (which was also for profiling) on `incr_hp'
instructions by an allocation site identifier.
Add a new foreign_proc_component for allocation site ids.
compiler/code_info.m:
compiler/global_data.m:
compiler/proc_gen.m:
Keep the set of allocation sites in the code_info and global_data
structures.
compiler/unify_gen.m:
Add allocation sites to LLDS allocation instructions.
compiler/layout_out.m:
compiler/llds_out_file.m:
compiler/llds_out_instr.m:
Output MR_AllocSiteInfo arrays in generated C files.
Output code to register the MR_AllocSiteInfo array with the Mercury
runtime.
Output allocation site ids for memory allocation instructions.
compiler/llds_out_util.m:
Add allocation sites to llds_out_info.
compiler/pragma_c_gen.m:
compiler/ml_foreign_proc_gen.m:
Generate a macro MR_ALLOC_ID which resolves to an allocation site
structure, for every foreign_proc whose C code contains the string
"MR_ALLOC_ID". This is to be used by hand-written C code which
allocates memory.
MR_PROC_LABELs are retained for backwards compatibility. Though
they were introduced for profiling, they seem to have been co-opted
for printf-debugging since then.
compiler/ml_global_data.m:
Add allocation site structures to the MLDS global data.
compiler/mlds.m:
compiler/ml_unify_gen.m:
Add allocation site id to `new_object' instruction.
compiler/mlds_to_c.m:
Output allocation site arrays and allocation ids in high-level C code.
Output a call to register the allocation site array with the Mercury
runtime.
Delete an unused predicate.
compiler/exprn_aux.m:
compiler/jumpopt.m:
compiler/livemap.m:
compiler/mercury_compile_llds_back_end.m:
compiler/middle_rec.m:
compiler/ml_accurate_gc.m:
compiler/ml_elim_nested.m:
compiler/ml_optimize.m:
compiler/ml_util.m:
compiler/mlds_to_cs.m:
compiler/mlds_to_gcc.m:
compiler/mlds_to_il.m:
compiler/mlds_to_java.m:
compiler/mlds_to_managed.m:
compiler/opt_debug.m:
compiler/opt_util.m:
compiler/use_local_vars.m:
compiler/var_locn.m:
Conform to changes.
compiler/pickle.m:
compiler/prog_event.m:
compiler/timestamp.m:
Conform to changes in memory allocation macros.
library/benchmarking.m:
Add the `report_memory_attribution' instrumentation predicates.
Conform to changes to MR_memprof_record.
library/array.m:
library/bit_buffer.m:
library/bitmap.m:
library/construct.m:
library/deconstruct.m:
library/dir.m:
library/io.m:
library/mutvar.m:
library/store.m:
library/string.m:
library/thread.semaphore.m:
library/version_array.m:
Use attributed memory allocation throughout the standard library so
that objects don't show up in the memory profile as "unknown".
Replace MR_PROC_LABEL by MR_ALLOC_ID.
mdbcomp/program_representation.m:
mdbcomp/rtti_access.m:
Replace MR_PROC_LABEL by MR_ALLOC_ID.
profiler/Mercury.options:
profiler/globals.m:
profiler/mercury_profile.m:
profiler/options.m:
profiler/output.m:
profiler/snapshots.m:
Add a new mode to `mprof' to parse and present the data from
`Prof.Snapshots' files.
Add options for the new profiling mode.
profiler/process_file.m:
Fix a typo.
runtime/mercury_conf_param.h:
#define MR_MPROF_PROFILE_MEMORY_ATTRIBUTION if memory profiling
is enabled and we are using Boehm GC.
runtime/mercury.h:
Make MR_new_object take an allocation id argument.
Conform to changes in memory allocation macros.
runtime/mercury_memory.c:
runtime/mercury_memory.h:
runtime/mercury_types.h:
Define MR_AllocSiteInfo.
Add memory allocation functions and macros which take into the
account the additional word necessary for the new profiling mode.
These should be used in preferences to the raw memory allocation
functions wherever possible so that objects do not show up in the
profile as "unknown".
Add analogues of realloc/free which take into account the offset
introduced by the attribution word.
Add function versions of the MR_new_object macros, which can't be
written in standard C. They are only used when necessary.
Add built-in allocation site ids, to be used in the runtime and
other hand-written code when context-specific ids are unavailable.
runtime/mercury_heap.h:
Make MR_tag_offset_incr_hp_msg and MR_tag_offset_incr_hp_atomic_msg
allocate an extra word when memory attribution is desired, and store
the allocation id there.
Similarly for MR_create{1,2,3}_msg.
Replace proclabel arguments in allocation macros by alloc_id
arguments.
Replace MR_hp_alloc_atomic by MR_hp_alloc_atomic_msg. It was only
used for boxing floats.
Conform to change to MR_new_object macro.
runtime/mercury_bootstrap.h:
Delete obsolete macro hp_alloc_atomic.
runtime/mercury_heap_profile.c:
runtime/mercury_heap_profile.h:
Add the code to summarise the live objects on the Boehm GC heap and
writes out the data to `Prof.Snapshots', for display by mprof.
Don't store the procedure name in MR_memprof_record: the procedure
address is enough and faster to compare.
runtime/mercury_prof.c:
Finish and close the `Prof.Snapshots' file when the program
terminates.
Conform to changes in MR_memprof_record.
runtime/mercury_misc.h:
Add a macro to expand to the name of the allocation sites array
in LLDS grades.
runtime/mercury_bitmap.c:
runtime/mercury_bitmap.h:
Pass allocation id through bitmap allocation functions.
Delete unused function MR_string_to_bitmap.
runtime/mercury_string.h:
Add MR_make_aligned_string_copy_msg.
Make string allocation macros take allocation id arguments.
runtime/mercury.c:
runtime/mercury_array_macros.h:
runtime/mercury_context.c:
runtime/mercury_deconstruct.c:
runtime/mercury_deconstruct_macros.h:
runtime/mercury_dlist.c:
runtime/mercury_engine.c:
runtime/mercury_float.h:
runtime/mercury_hash_table.c:
runtime/mercury_ho_call.c:
runtime/mercury_label.c:
runtime/mercury_prof_mem.c:
runtime/mercury_stacks.c:
runtime/mercury_stm.c:
runtime/mercury_string.c:
runtime/mercury_thread.c:
runtime/mercury_trace_base.c:
runtime/mercury_trail.c:
runtime/mercury_type_desc.c:
runtime/mercury_type_info.c:
runtime/mercury_wsdeque.c:
Use attributed memory allocation throughout the runtime so that
objects don't show up in the profile as "unknown".
runtime/mercury_memory_zones.c:
Attribute memory zones to the Mercury runtime.
runtime/mercury_tabling.c:
runtime/mercury_tabling.h:
Use attributed memory allocation macros for tabling structures.
Delete unused MR_table_realloc_* and MR_table_copy_bytes macros.
runtime/mercury_deep_copy_body.h:
Try to retain the original attribution word when copying values.
runtime/mercury_ml_expand_body.h:
Conform to changes in memory allocation macros.
runtime/mercury_tags.h:
Replace proclabel arguments by alloc_id arguments in allocation macros.
runtime/mercury_wrapper.c:
If memory attribution is enabled, tell Boehm GC that pointers may be
displaced by an extra word.
trace/mercury_trace.c:
trace/mercury_trace_tables.c:
Conform to changes in memory allocation macros.
extras/net/tcp.m:
extras/solver_types/library/any_array.m:
extras/trailed_update/tr_array.m:
Conform to changes in memory allocation macros.
doc/user_guide.texi:
Document the new profiling mode.
doc/reference_manual.texi:
Update a commented out example.
Estimated hours taken: 0.5
Branches: main
runtime/*.c:
Convert all remaining C source files to four-space indentation.
Fix some deviations from our style guide.
Estimated hours taken: 5
Branches: main
Upgrade to version 7.0 alpha 6 (CVS 2006-08-14) of the Boehm garbage
collector. The main reason is that thread-local allocation is not
supported on Solaris in version 6.x of the collector.
boehm_gc/*:
Merge in changes from the vendor branch.
.README.in:
Update Boehm GC version number and copyright notice.
Mmake.common.in:
configure.in:
runtime/RESERVED_MACRO_NAMES:
scripts/mgnuc.in:
Don't define `THREAD_LOCAL_ALLOC' nor `GC_REDIRECT_TO_LOCAL' as
thread-local allocation is automatically enabled in gc 7.0.
Consistently use `GC_WIN32_THREADS' and `GC_LINUX_THREADS' as the
non-prefixed versions of those symbols are deprecated.
Don't define `NO_SIGNALS' as it is no longer necessary.
Add librt to the list of thread libraries on Solaris. This is
needed for the POSIX semaphore functions now used on Solaris.
runtime/mercury_prof_mem.c:
runtime/mercury_prof_mem.h:
Update comments mentioning `NO_SIGNALS'.
compiler/compile_target_code.m:
Make `--inline-alloc' do nothing as inline allocation is currently
broken in gc 7.0.
runtime/mercury.h:
runtime/mercury_heap.h:
Include `gc_inline.h' instead of `gc_inl.h' as the latter no longer
exists.
extras/concurrency/semaphore.m:
library/par_builtin.m:
robdd/bryant.c:
Replace instances of `GC_PTR' by `void *' as the former no longer
exists.
runtime/mercury_wrapper.c:
Remove references to `GC_quiet' which no longer exists.
Replace uses of `GC_quiet' for the MPS collector by a new variable
`MR_mps_quiet'.
tools/bootcheck:
Copy libatomic_ops-related files and directories when copying
boehm_gc.
Estimated hours taken: 2
Branches: main
runtime/mercury_prof_mem.h:
runtime/mercury_prof_mem.c:
Align memory allocated with MR_prof_malloc() to
double-word boundaries, rather than word boundaries.
This is needed to avoid bus errors on Solaris,
now that we use `long long' for MR_Dword.
Estimated hours taken: 0.25
runtime/mercury_prof_mem.h:
runtime/mercury_prof_mem.c:
Improve the documentation about the use of -DNO_SIGNALS and
the use of malloc/GC_malloc in the profiling signal handler.
boehm_gc/Makefile:
scripts/mgnuc.in:
Add pointers to the documentation in runtime/mercury_prof_mem.h.
Estimated hours taken: 4
Add MR_ prefixes to the types used when generating C code.
This means types such as Word, String, Bool, Float become MR_Word,
MR_String, MR_Bool, MR_Float. Also define MR_Box for both the LLDS and
MLDS backends so we can use it uniformly.
This is very important in environments where String or Bool have already
been used as system types (for example, managed C++). And besides, we
should do it anyway as part of the grand namespace cleanup.
I have fixed all of the uses of the non-prefixed types in the runtime
and trace directories. I haven't done it for the library and compiler
directories yet (no promises that I will do it in future either). But
if you see a non-prefixed type in code from now on, please consider it a
bug and fix it.
mercury_bootstrap.h contains #defines to map the non-prefixed types into
the prefixed ones. Like many of the other namespace cleaning backwards
compatibility macros, this can be turned off with
MR_NO_BACKWARDS_COMPAT.
This shouldn't break any code, but this kind of change affects so many
things that of course there could be problems lurking in there somewhere.
If you start getting errors from the C compiler after this change is
installed, you will want to make sure you at least have the runtime
system updated so that you are getting the backwards compatibility
definitions in mercury_bootstrap.h. Then if you continue to have
problems you can bug me about it.
compiler/export.m:
compiler/llds_out.m:
compiler/mlds_to_c.m:
Use MR_Word, MR_Float, MR_Bool, etc when generating C.
doc/reference_manual.texi:
Update the reference manual to talk about MR_Word, MR_String,
MR_Char, etc.
runtime/mercury_bootstrap.h:
Add bootstrapping typedefs.
runtime/*:
trace/*:
Change Word, Float, Bool, Code, String, etc to
MR_Word, MR_Float, MR_Bool, MR_Code, MR_String.
Estimated hours taken: 0.25
runtime/mercury_dlist.c:
runtime/mercury_prof_mem.c:
Add `#include "mercury_imp.h"' at the start,
to avoid compilation errors from gcc due to
"global register variable follows a function definition".
Estimated hours taken: 8
Reorganize the routines for allocating and deallocating memory:
runtime/mercury_std.h:
runtime/mercury_misc.c:
runtime/mercury_memory.h:
runtime/mercury_memory.c:
- Put the routines in the proper place.
Previously the declarations and definitions of the memory
allocation/deallocation routines were spread amoungst
all four of these files; I moved the ones in mercury_std.h
and mercury_misc.c so that they are now all defined
in mercury_memory.{h,c}
- Avoid unnecessary duplication
The following routines did exactly the same thing,
modulo bugs(!):
allocate_bytes() and newmem()
deallocate_bytes() and oldmem()
make() and allocate_object()
make_many() and allocate_array()
- Use appropriate names.
I added `MR_' prefixes, and ensured that macros that are not
function-like macros use all uppercase. I also used a more
consistent naming scheme.
Previously the names used were
(1) checked_malloc, checked_realloc
(2a) allocate_bytes, deallocate_bytes, reallocate_bytes,
allocate_object, allocate_array, resize_array
(2b) newmem, oldmem, resizemem,
make, make_many, resize_many
The new names are
(1) MR_malloc, MR_free, MR_realloc,
MR_NEW, MR_NEW_ARRAY, MR_RESIZE_ARRAY
(2) MR_GC_malloc, MR_GC_free, MR_GC_realloc,
MR_GC_NEW, MR_GC_NEW_ARRAY, MR_GC_RESIZE_ARRAY
runtime/*.[ch]:
trace/*.[ch]:
library/array.m:
library/benchmarking.m:
library/io.m:
library/std_util.m:
extras/odbc/odbc.m:
extras/aditi/aditi.m:
Use the new names.
Estimated hours taken: 0.25
runtime/mercury_prof_mem.h:
runtime/mercury_prof_mem.c:
Change MR_prof_malloc() to ensure that the memory that it
allocates will be Word-aligned. This change is needed
as a result of my previous change to mercury_heap_profile.c
which made it allocate strings using MR_prof_malloc().
Estimated hours taken: 40 (+ unknown time by Zoltan)
Add support for memory profiling.
(A significant part of this change is actuallly Zoltan's work. Zoltan
did the changes to the compiler and a first go at the changes to the
runtime and library. I rewrote much of Zoltan's changes to the runtime
and library, added support for the new options/grades, added code to
interface with mprof, did the changes to the profiler, and wrote the
documentation.)
[TODO: add test cases.]
NEWS:
Mention support for memory profiling.
runtime/mercury_heap_profile.h:
runtime/mercury_heap_profile.c:
New files. These contain code to record heap profiling information.
runtime/mercury_heap.h:
Add new macros incr_hp_msg(), tag_incr_hp_msg(),
incr_hp_atomic_msg(), and tag_incr_hp_atomic_msg().
These are like the non-`msg' versions, except that if
PROFILE_MEMORY is defined, they also call MR_record_allocation()
from mercury_heap_profile.h to record heap profiling information.
Also, fix up the indentation in lots of places.
runtime/mercury_prof.h:
runtime/mercury_prof.c:
Added code to dump out memory profiling information to files
`Prof.MemoryWords' and `Prof.MemoryCells' (for use by mprof).
Change the format of the `Prof.Counts' file so that the
first line says what it is counting, the units, and a scale
factor. Prof.MemoryWords and Prof.MemoryCells can thus have
exactly the same format as Prof.Counts.
Also cleaned up the interface to mercury_prof.c a bit, and did
various other minor cleanups -- indentation changes, changes to
use MR_ prefixes, additional comments, etc.
runtime/mercury_prof_mem.h:
runtime/mercury_prof_mem.c:
Rename prof_malloc() as MR_prof_malloc().
Rename prof_make() as MR_PROF_NEW() and add MR_PROF_NEW_ARRAY().
runtime/mercury_wrapper.h:
Minor modifications to reflect the new interface to mercury_prof.c.
runtime/mercury_wrapper.c:
runtime/mercury_label.c:
Rename the old `-p' (primary cache size) option as `-C'.
Add a new `-p' option to disable profiling.
runtime/Mmakefile:
Add mercury_heap_profile.[ch].
Put the list of files in alphabetical order.
Delete some obsolete stuff for supporting `.mod' files.
Mention that libmer_dll.h and libmer_globals.h are
produced by Makefile.DLLs.
runtime/mercury_imp.h:
Mention that libmer_dll.h is produced by Makefile.DLLs.
runtime/mercury_dummy.c:
Change a comment to refer to libmer_dll.h rather than
libmer_globals.h.
compiler/llds.m:
Add a new field to `create' and `incr_hp' instructions
holding the name of the type, for heap profiling.
compiler/unify_gen.m:
Initialize the new field of `create' instructions with
the appropriate type name.
compiler/llds_out.m:
Output incr_hp_msg() / tag_incr_hp_msg() instead of
incr_hp() / tag_incr_hp().
compiler/*.m:
Minor changes to most files in the compiler back-end to
accomodate the new field in `incr_hp' and `create' instructions.
library/io.m:
Add `io__report_full_memory_stats'.
library/benchmarking.m:
Add `report_full_memory_stats'. This uses the information saved
by runtime/mercury_heap_profile.{c,h} to print out a report
of memory usage by procedures and by types.
Also modify `report_stats' to print out some of that information.
compiler/mercury_compile.m:
If `--statistics' is enabled, call io__report_full_memory_stats
at the end of main/2. This will print out full memory statistics,
if the compiler was compiled with memory profiling enabled.
compiler/options.m:
compiler/handle_options.m:
runtime/mercury_grade.h:
scripts/ml.in:
scripts/mgnuc.in:
scripts/init_grade_options.sh-subr:
scripts/parse_grade_options.sh-subr:
Add new option `--memory-profiling' and new grade `.memprof'.
Add `--time-profiling' as a new synonym for `--profiling'.
Also add `--profile-memory' for more fine-grained control:
`--memory-profiling' implies both `--profile-memory' and
`--profile-calls'.
scripts/mprof_merge_runs:
Update to handle the new format of Prof.Counts and to
also merge Prof.MemoryWords and Prof.MemoryCells.
profiler/options.m:
profiler/mercury_profile.m:
Add new options `--profile memory-words' (`-m'),
`--profile memory-cells' (`-M') and `--profile time' (`-t').
Thes options make the profiler select a different count file,
Prof.MemoryWords or Prof.MemoryCells instead of Prof.Counts.
specific to time profiling.
profiler/read.m:
profiler/process_file.m:
profiler/prof_info.m:
profiler/generate_output.m:
Update to handle the new format of the counts file.
When reading the counts file, look at the first line of
the file to determine what is being profiled.
profiler/globals.m:
Add a new global variable `what_to_profile' that records
what is being profiled.
profiler/output.m:
Change the headings to reflect what is being profiled.
doc/user_guide.texi:
Document memory profiling.
Document new options.
doc/user_guide.texi:
compiler/options.m:
Comment out the documentation for `.proftime'/`--profile-time',
since doing time and call profiling seperately doesn't work,
because the code addresses change when you recompile with a
different grade. Ditto for `.profmem'/`--profile-memory'.
Also comment out the documentation for
`.profcalls'/`--profile-calls', since it is redundant --
`.memprof' produces the same information and more.
configure.in:
Build a `.memprof' grade. (Hmm, should we do this only
if `--enable-all-grades' is specified?)
Don't ever build a `.profcalls' grade.
Estimated hours taken: 1
runtime/*.h:
runtime/*.c:
runtime/mercury_conf.h.in:
Remove old .h files.
Update #includes to refer to mercury_*.h
Update #ifdef MODULE_H to be #ifdef MERCURY_MODULE_H
Estimated hours taken: 2
Cleaned up runtime directory.
runtime/*.c:
- Renamed all .c files as mercury_*.c
Some have been renamed to make their purpose clearer.
call.mod -> mercury_ho_call.c
runtime/*.h:
- Moved contents of .h files to mercury_*.h
- *.h now contain #include mercury_*.h. They be removed later.
- Updated references to conf.h -> mercury_conf.h
runtime/conf.h.in:
- Renamed conf.h.in as mercury_conf.h.in.
Didn't leave a forwarding header for this one, as conf.h was
never part of the repository anyway.
runtime/Mmakefile:
- Convert lists to one-per-line lists.
- Add mercury_accurate_gc.h to HDRS.
- Remove all .mod files
- Make sure runtime.init uses the ORIG_CS not MOD_CS.
- Fix the rules for "clean_o" and "clean_mod_c", which used
wildcards like "*.o" to remove files. The one that removed
all .c files corresponding with *.mod, instead of using MOD_CS
was particularly vicious.
- Cope with the file renamings.
configure.in:
- Cope with the file renamings.
