GCL 2.6.2 tests

GCL 2.6.2 RELEASE NOTES

The GCL team is happy to announce the release of version 2.6.2, the latest achievement in the 'stable' series. While strictly speaking a bug-fix only release, 2.6.2 incorporates several major improvements over the last stable release, 2.5.3. Some highlights:

The development of a 'lisp compiler torture tester' by GCL developer Paul Dietz which repeatedly compiles randomly generated forms of specifiable length to test the compiler for correctness.
The application of several significant corrections to the GCL lisp compiler to remove every known instance of miscompilation uncovered by this tester. To our knowledge, GCL is alone with CLISP in passing this torture test for runs of effectively indefinite length.
Major performance improvements were applied to the lisp compiler to enable it to complete random tests of great length in a reasonable amount of time.
Corrections to the GCL core files to enable very large image sizes in 64 bits, in which more than a billion cons cells can be allocated. Current 64bit options include amd64, ia64, and alpha running most flavors of GNU/Linux.
Corrections to the heap scaling behavior of the garbage collector, resulting in significant performance gains in many instances.
Support for the latest gcc and binutils versions on all platforms but mingw
The elimination of many instances of unnecessary internal garbage generation bringing the associated performance gains
Native support for execstack protected linux kernels, such as on Fedora core systems
Native support for FreeBSD, OpenBSD, and MacOSX
Static function pointer support to stabilize dynamic library usage on Itanium systems
Transparent readline initialization when compiled in
Support for profiling via gprof
Automatic disabling of SGC (stratified garbage collection) if the image is executed on a kernel not supporting fault address recovery
Remove a memory leak associated with heavy bignum usage via the introduction of SGC contiguous pages
Several significant internal bug fixes, epecially in the mingw port.
Alter the build process to perform a full self compile with full function proclamation at build time.
GCL now compiles Axiom from scratch and carries it to all supported platforms with the current exception of mingw
GCL's ANSI build now in use for its first end-user application -- maxima (current cvs)
New 64bit platform support -- amd64, with full native object relocation

The full changelog can be found in the source tree in the file 'debian/changelog'.

The GCL team has subjected this release to a wide variety of tests and benchmarks. While all such results are necessarily incomplete, one can nevertheless usefully summarize the approximate state of affairs as follows:

GCL is about as portable as CLISP
The GCL lisp compiler is about as robust/correct as that of CLISP, at least as measured by the random tester, which at present only covers a mostly integer subset of lisp.
GCL is about as fast as CMUCL
GCL plays a major role in carrying the primary large open source lisp end user applications to a wide variety of systems
GCL is still the least ANSI compliant of the freely available lisp systems, though a modest level of compliance has been achieved in this release. Much greater compliance has been achieved in the 2.7.x (cvs unstable) series yet to be officially released.

The specific test results are arranged in the following table. Some terms need defining:

BFD	the method of relocating compiled lisp object modules into the running executable using the BFD library
custreloc	the method of relocating compiled lisp object modules into the running executable using the native GCL code. This method as well as the BFD method preserve the module loading across image saving and re-execution
dlopen	the method of dynamically linking in compiled lisp object modules into the existing session only via the system dynamic linker loader, ld.so.
SGC	Stratified Garbage Collection -- an optional accelerated generational garbage collection algorithm employing read-only memory
CLtL1	Common Lisp, the Language vol I, referring to the book of the same name by Steele defining a widely used lisp language standard prior to the ANSI standardization process in 1994.
ANSI	the work in progress image build attempting to eventually extend traditional GCL into full ANSI complaince
Ansi tests	the results of the work in progress ansi compliance test suite written by GCL developer Paul Dietz presented as the number of failures divided by the total number of tests run
Random tests	the results of the random 'compiler torture tester' presented as the number of tests/the size of the random forms/the number of variables passed to the random function

In the table below, green denotes a pass, yellow denotes an as yet unimplemented option, and red indicates failure. Blank cells indicate tests that have not been run.

System	CPU	Self Build	Preferred Linking	SGC	ANSI tests	Random tests	Axiom CVS/CLtL1 (6)	pc-nqthm CLtL1
Debian GNU/Linux (sid)	i386		bfd or custreloc		303/ 10697	50000/10000/8 500000/1000/8		(setq si::multiply-stacks 16)
Debian GNU/Linux (sid)	sparc		bfd or custreloc		303/ 10697
Debian GNU/Linux (sid)	powerpc		bfd		303/ 10697
Debian GNU/Linux (sid)	amd64		bfd		303/ 10697
Debian GNU/Linux (sid)	arm		bfd		303/ 10697
Debian GNU/Linux (sid)	m68k		bfd		303/ 10697
Debian GNU/Linux (sid)	s390		bfd		303/ 10697
Debian GNU/Linux (sid)	ia64		dlopen		303/ 10697		(1)
Debian GNU/Linux (sid)	hppa	-O0	dlopen		303/ 10697		(1)
Debian GNU/Linux (sid)	mips		dlopen		303/ 10697		(1)
Debian GNU/Linux (sid)	mipsel		dlopen		303/ 10697		(1)
Debian GNU/Linux (sid)	alpha		dlopen		303/ 10697		(1)
Fedora FC1	i386		bfd or custreloc		303/ 10697	12000/1000/8
Solaris	sparc		bfd or custreloc		303/ 10697	4000/1000/8 (4)
Windows MINGW(a)	i386		custreloc		303/ 10697	57000/1000/8	(2)
MacOSX	powerpc		bfd	(3)	303/ 10697
OpenBSD	i386		bfd		303/ 10697
FreeBSD	i386		custreloc		303/ 10697

Notes:

(1) dlopen builds use file descriptors for each object load. The step in the Axiom build process which regenerates its databases consumes more than the conventional maximum of 1024 file descriptors available by default on most UNIX systems.
(2) An AXIOMsys executable can be produced, and is basically functional, but experiences sporadic errors of a type as yet unknown.
(3) This is known to work on at least some versions of the OS, but others report a hang (infinite loop) when enabling SGC. It is possible that this is due to a mprotect bug in older versions of the Darwin system shared libraries. 'compatibility version of user 6.0.0' appears to work.
(4) On this machine, the underlying gcc was old (3.0) and segfaulted outside of GCL when attempting to compile its produced C code after a few thousand iterations.

(a) The preferred build environment for Mingw/Windows is gcc 3.3.1, binutils 2.14.90, and the latest msys release.

The following table presents the results of the popular gabriel benchmarks of three freely available lisp systems, GCL, CLISP and CMUCL. Times are presented as multiples of the time GCL took in completing the tests. Green indicates tests on which GCL is the fastest, while red indicates tests on which GCLwas not the fastest. The benchmark code can be found in ftp://ftp.ma.utexas.edu/gcl/gabriel.tgz. The number of test iterations has been increased by a factor of 400 to overcome granularity issues on modern machines. The '(print (time ...))' statements around each test iteration were removed, again due to granularity and relative i/o load. Likewise the special init.lsp file conventionally used to preallocate GCL memory in such cases was removed as it is now mostly obsolete. Finally the tests were modified slightly to place the optimization declamations at the top of each file being compiled as suggested by a CMUCL expert.

As with any benchmark, results can vary somewhat with the details of the executing machine. With lisp in particular, the ratios of the cache sizes, cpu speed, and memory bandwidths can impact such tests significantly. We present the results for two popular configurations below. While the precise details of the differences are as yet known, it is speculated that the first result is more dominated by in-cache cpu performance, while the latter is more dominated by memory access efficiency.

Dual Intel Xeon 2.4Ghz, 512 Mb, Linux 2.4.20

Athlon XP 3000+ (2.1Ghz), 512 Mb, Linux 2.4.26

Benchmark	GCL 2.6.2	CMUCL 18e-9	CLISP 2.33-2
BOYER	1.000	2.200	9.869
BROWSE	1.000	2.240	NA
CTAK	1.000	0.230	1.890
DDERIV	1.000	2.148	2.909
DERIV	1.000	2.083	3.640
DESTRU-MOD	1.000	2.043	9.880
DESTRU	1.000	1.168	5.743
DIV2	1.000	2.222	3.911
FFT-MOD	1.000	1.585	206.057
FFT	1.000	1.544	176.088
FPRINT	1.000	2.136	3.742
FREAD	1.000	1.746	2.111
FRPOLY	1.000	1.524	5.112
PUZZLE-MOD	1.000	10.824	41.618
PUZZLE	1.000	11.324	37.671
STAK	1.000	1.536	9.836
TAK-MOD	1.000	1.465	15.053
TAK	1.000	1.486	14.629
TAKL	1.000	1.419	14.965
TAKR	1.000	1.933	12.327
TPRINT	1.000	0.937	1.263
TRAVERSE	1.000	0.875	8.378
TRIANG-MOD	1.000	7.067	26.814
TRIANG	1.000	1.281	18.565
GEOMETRIC AVERAGE	1.00	1.86	10.33
MEDIAN	1.00	1.67	9.87

Benchmark	GCL 2.6.2	CMUCL 18e	CLISP 2.33
BOYER	1.000	0.892	6.316
BROWSE	1.000	0.965	NA
CTAK	1.000	0.435	3.489
DDERIV	1.000	0.822	1.579
DERIV	1.000	0.651	1.639
DESTRU-MOD	1.000	0.812	4.779
DESTRU	1.000	0.550	3.239
DIV2	1.000	0.599	1.525
FFT-MOD	1.000	2.655	337.207
FFT	1.000	1.923	251.026
FPRINT	1.000	2.322	3.508
FREAD	1.000	1.890	1.900
FRPOLY	1.000	1.013	3.606
PUZZLE-MOD	1.000	5.976	20.350
PUZZLE	1.000	5.472	19.387
STAK	1.000	1.655	8.064
TAK-MOD	1.000	1.382	14.775
TAK	1.000	1.399	14.514
TAKL	1.000	1.281	12.877
TAKR	1.000	1.735	15.500
TPRINT	1.000	2.008	1.674
TRAVERSE	1.000	0.770	8.013
TRIANG-MOD	1.000	6.639	25.182
TRIANG	1.000	1.186	16.948
GEOMETRIC AVERAGE	1.00	1.40	8.46
MEDIAN	1.00	1.33	8.01

Many improvements are planned for the 2.7.x development series time permitting, the most important of which is to complete the task of building an ANSI compliant GCL image.