Among other challenges, this regular expression is designed to demonstrate
that thread prioritization is finicky: Given the string 'aaaaaa' to match,
the first four threads will try to grab the second 'a', the third thread
(the one that matched the '(a??)' group) having scheduled the same
instruction pointer to the '(a+)' group that the second -- higher-priority
-- thread will try to advance to only after processing the '(a??)' group's
SPLIT. The second thread must override the third thread in that case,
essentially stopping the latter.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
If we want to match greedy or reluctant regular expressions, we have
to make sure that certain threads are split off with a higher priority
than others. We will use the ThreadQueues' natural order as priority
order: high to low.
To support splitting into different-priority threads, let's introduce
a second SPLIT opcode: SPLIT_JMP. The latter prefers to jump while the
former prefers to execute the opcode directly after the SPLIT opcode.
There is a subtle challenge here, though: let's assume that there are
two current threads and the higher-priority one wants to jump where
the lower-priority one is already. In the PikeVM implementation
before this change, queueImmediately() would see that there is
already a thread queued for that program counter and *not* queue the
higher-priority one.
Example: when matching the pattern '(a?)(a??)(a?)' against the string
'aa', after the first character, the first (high priority) thread
will have matched the first group while the second thread matched the
second group. In the following step, therefore, the first thread will
want to SPLIT_JMP to match the final 'a' to the third group but the
second thread already queued that program counter.
The proposed solution is to introduce a third thread queue: 'queued'.
When queuing threads to be executed after reading the next character
from the string to match, they are not directly queued into 'next' but
into 'queued'. Every thread requiring immediate execution (i.e. before
reading the next character) will be queued into 'current'. Whenever
'current' is drained, the next thread from 'queued' that has not been
queued to 'current' yet will be executed.
That way, we can guarantee that 1) no lower-priority thread can override
a higher-priority thread and 2) infinite loop are prevented.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Instead of having an opcode 'CHAR', let's have the opcodes that fall
within the range of a char *be* the opcode 'match this character'.
While at it, break the ranges of the different types of opcodes apart
into ranges so that related operations are clustered.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We still do not parse the regular expression patterns, but we can at
least test that the hardcoded 'a(bb)+a' works as expected.
This class will be extended as we support more and more features.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Based on the just-implemented PikeVM, let's test it with a specific
regular expression. At this point, no parsing is implemented but instead
an explicit program executing a(bb)?a is hardcoded.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
So far, these are humble beginnings indeed. Based on the descriptions of
http://swtch.com/%7Ersc/regexp/regexp2.html
I started implementing a Thompson NFA / Pike VM.
The idea being that eventually, regular expressions are to be compiled
into special-purpose bytecode for the Pike VM that executes a varying
number of threads in lock-step over each character of the text to match.
The thread count is bounded by the length of the program: two different
threads with identical instruction pointer at the same character-to-match
would yield exactly the same outcome (and therefore, we can execute just
one such thread instead of possibly many).
To allow for matching groups, each thread carries a state with it, saving
the group offsets acquired so far.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This makes both the Pattern and the Matcher class abstract so that more
specialized patterns than the trivial patterns we support so far can be
implemented as convenient subclasses of the respective abstract base
classes.
To ease development, we work on copies in test/regex/ in the 'regex'
package. That way, it can be developed in Eclipse (because it does not
interfere with Oracle JRE's java.util.regex.* classes).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
For quick access, the sezpoz library stores lists in
META-INF/annotations/ of classes that have been annotated in a
special way.
To support the use case where the annotations actually changed since
sezpoz stored said lists, sezpoz then creates proxy instances for the
annotations to provide some backwards compatibility: as long as there
are default values for any newly-introduced annotation values,
everything is groovy.
Therefore, let's make sure that proxy instances inherit the
annotations' default values.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When the class whose field is to be inspected has no annotations at all,
at least my javac here (1.6.0_51 on MacOSX) does not produce any class
addendum.
Therefore, let's verify that the addendum is not null before proceeding.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This adds an extra class path element to the VM running the unit tests,
writes files with identical file names into both directories and then
verifies that SystemClassLoader#getResources can find them.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This change reuses the existing insertion sort (which was previously what
Arrays.sort() executed) in a full intro sort pipeline.
The implementation is based on the Musser paper on intro sort (Musser,
David R. "Introspective sorting and selection algorithms." Softw., Pract.
Exper. 27.8 (1997): 983-993.) and Wikipedia's current description of the
heap sort: http://en.wikipedia.org/wiki/Heapsort.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We do not really support regular expressions yet, but we do support
trivial patterns including ones with escaped characters. Let's make sure
that that works as advertised.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The OpenJDK library wants to track and run the shutdown hooks itself
rather than let the VM do it, so we need to tell it when we're
exiting.
Also, in machine.cpp we need to use only the modifiers specified in
the InnerClasses attribute for inner classes rather than OR them with
the flags given at the top level of the class file.
This fixes a couple of tests in the Scala test suite
(run/reflection-modulemirror-toplevel-badpath.scala and
run/reflection-constructormirror-nested-good.scala).
To execute tests on a remote host (for instance, because you're cross-compiling),
simply do:
make remote-test=true remote-test-host=<host_to_test_on> test
You can set several variables to control the functionality of remote-test.
See them below, along with their default values:
remote-test-host = localhost # host to ssh to
remote-test-port = 22
remote-test-user = ${USER} # user to execute tests as
remote-test-dir = /tmp/avian-test-${USER} # dir to rsync build output to
In order to calculate the initial stack map of GC roots for an
exception handler, we do a logical "and" of maps across all the
instructions contained in the try block for that handler. This is
complicated by the presence of jsr/ret instructions, though, because
instructions in a subroutine may have multiple maps associated with
them corresponding to all the paths from which execution might flow to
them.
The bug in this case was that we were using an uninitialized map in
our calculation, resulting in a map with no GC roots at all. By the
time the map was initialized, the damage had already been done. The
solution is to treat an uninitialized map as if it has roots at all
positions so that it has no effect on the calculation until it has
been initialized with real data.
My earlier attempt (fa5d76b) missed an important detail, and somehow I
forgot to test the 32-bit OpenJDK build which made that omission
obvious. Here's the fix.
Java requires that NaNs be converted to zero and that numbers at or
beyond the limits of integer representation be clamped to the largest
or smallest value that can be represented, respectively.
I get this error when compiling with "make openjdk=...." on both x86_64 and
arm:
compiling test classes
test/Arrays.java:90: error: reference to equals is ambiguous, both method
equals(float[],float[]) in Arrays and method equals(Object[],Object[]) in
Arrays match
expect(java.util.Arrays.equals(null, null));
test/Arrays.java:95: error: reference to hashCode is ambiguous, both method
hashCode(double[]) in Arrays and method hashCode(Object[]) in Arrays match
java.util.Arrays.hashCode(null);
The attached patch fixes this.
When we skip a single-precision register to ensure a double-precision
load is aligned, we need to remember that in case we see another
single-precision argument later on, which we must backfill into that
register we skipped according to the ABI.
We were assuming the array element size was always the native word
size, which is not correct in general for primitive arrays, and this
led to wasted space at best and memory corruption at worst.
The ArrayList(Collection) constructor was allocating two arrays
instead of one due to an off-by-one error in ArrayList.grow. This
commit fixes that and makes grow and shrink more robust.
We were not properly converting dots to slashes internally for package names
and we did not properly handle Method.getAnnotations and
Method.getAnnotation(Class<T>) on methods without any annotations.
Added some tests to cover these cases.
In the tails=true build, the calling method cannot always be
determined due to stack frames being optimized away, so we must be
prepared for LogRecord.getSourceMethodName to return null.
This was causing 8-byte SSE-to-SSE moves involving registers
xmm8-xmm15 to be misencoded on x86_64, leading to incorrect code
generation in methods with lots of local variables of type double.
Unlike the interpreter, the JIT compiler tries to resolve all the
symbols referenced by a method when compiling that method. However,
this can backfire if a symbol cannot be resolved: we end up throwing
an e.g. NoClassDefFoundError for code which may never be executed.
This is particularly troublesome for code which supports multiple
APIs, choosing one at runtime.
The solution is to defer to stub code for symbols which can't be
resolved at JIT compile time. Such a stub will try again at runtime
to resolve the needed symbol and throw an appropriate error if it
still can't be found.
Thread.yield is not enough to ensure that the tracing thread does not
starve the test thread on some QEMU VMs, so we use wait/notifyAll to
make sure both threads have opportunities to run and the test actually
finishes.
The VM uses Integer and Long instances internally to wrap the results
of dynamic method invocations, but Method.invoke should use the
correct, specific type for the primitive (e.g. Character for char).
My previous attempt at this was incomplete; it did not address
Java->native->Java->native call sequences, nor did it address
continuations. This commit takes care of both.
The stack mapping code was broken for cases of stack slots being
reused to hold primitives or addresses within subroutines after
previously being used to hold object references. We now bitwise "and"
the stack map upon return from the subroutine with the map as it
existed prior to calling the subroutine, which has the effect of
clearing map locations previously marked as GC roots where
appropriate.
This test covers the case where a local stack slot is first used to
store an object reference and later to store a subroutine return
address. Unfortunately, this confuses the VM's stack mapping code;
I'll be working on a fix for that next.
The new test requires generating bytecode from scratch, since there's
no reliable way to get javac to generate the code we want. Since we
already had primitive bytecode construction code in Proxy.java, I
factored it out so we can reuse it in Subroutine.java.
When loading a class which extends another class that contained a
field of primitive array type using defineClass in a bootimage=true
build, the VM was unable to find the primitive array class, and
makeArrayClass refused to create one since it should already have
existed.
The problem was that the bootimage=true build uses an empty
Machine::BootstrapClassMap, and resolveArrayClass expected to find the
primitive array classes there. The fix is to check the
Machine::BootLoader map if we can't find it in
Machine::BootstrapClassMap.
This is an attempt to reproduce an issue reported on the discussion
group. However, the current form of the test is passing, so further
work will be necessary to trigger the bug.
As reported on the discussion group, there is a problem with the
ClassLoader.defineClass implementation sunch that this test is not
currently passing, at least for the mode=debug and bootimage=true
builds. I plan to address these failures soon, but I wanted to add a
test first to make sure I could reproduce them.
This rather large commit modifies the VM to use non-local returns to
throw exceptions instead of simply setting Thread::exception and
returning frame-by-frame as it used to. This has several benefits:
* Functions no longer need to check Thread::exception after each call
which might throw an exception (which would be especially tedious
and error-prone now that any function which allocates objects
directly or indirectly might throw an OutOfMemoryError)
* There's no need to audit the code for calls to functions which
previously did not throw exceptions but later do
* Performance should be improved slightly due to both the reduced
need for conditionals and because undwinding now occurs in a single
jump instead of a series of returns
The main disadvantages are:
* Slightly higher overhead for entering and leaving the VM via the
JNI and JDK methods
* Non-local returns can make the code harder to read
* We must be careful to register destructors for stack-allocated
resources with the Thread so they can be called prior to a
non-local return
The non-local return implementation is similar to setjmp/longjmp,
except it uses continuation-passing style to avoid the need for
cooperation from the C/C++ compiler. Native C++ exceptions would have
also been an option, but that would introduce a dependence on
libstdc++, which we're trying to avoid for portability reasons.
Finally, this commit ensures that the VM throws an OutOfMemoryError
instead of aborting when it reaches its memory ceiling. Currently, we
treat the ceiling as a soft limit and temporarily exceed it as
necessary to allow garbage collection and certain internal allocations
to succeed, but refuse to allocate any Java objects until the heap
size drops back below the ceiling.
We now check for stack overflow in the JIT build as well as the
interpreted build, throwing a StackOverflowError if the limit
(currently hard-coded to 64KB, but should be easy to make
configurable) is exceeded.
We weren't properly handling the case where a 64-bit value is
multipled with itself in multiplyRR, leading to wrong code. Also,
addCarryCR didn't know how to handle constants more than 8-bits wide.
Compiling the entire OpenJDK class library into a bootimage revealed
some corner cases which broke the compiler, including synchronization
in a finally block and gotos targeting the first instruction of an
unsynchronized method.
If the VM runs out of heap space and the "avian.heap.dump" system
property was specified at startup, the VM will write a heap dump to
the filename indicated by that property. This dump may be analyzed
using e.g. DumpStats.java.
This makes heap dumps more useful since these classes are now refered
to by name instead of number.
This commit also adds a couple of utilities for parsing heap dumps:
PrintDump and DumpStats.
This allows OpenJDK to access time zone data which is normally found
under java.home, but which we must embed in the executable itself to
create a self-contained build. The VM intercepts various file
operations, looking for paths which start with a prefix specified by
the avian.embed.prefix property and redirecting those operations to an
embedded JAR.
For example, if avian.embed.prefix is "/avian-embedded", and code
calls File.exists() with a path of
"/avian-embedded/javahomeJar/foo.txt", the VM looks for a function
named javahomeJar via dlsym, calls the function to find the memory
region containing the embeded JAR, and finally consults the JAR to see
if the file "foo.txt" exists.
The main changes in this commit ensure that we don't hold the global
class lock when doing class resolution using application-defined
classloaders. Such classloaders may do their own locking (in fact,
it's almost certain), making deadlock likely when mixed with VM-level
locking in various orders.
Other changes include a fix to avoid overflow when waiting for
extremely long intervals and a GC root stack mapping bug.
Whereas the GNU Classpath port used the strategy of patching Classpath
with core classes from Avian so as to minimize changes to the VM, this
port uses the opposite strategy: abstract and isolate
classpath-specific features in the VM similar to how we abstract away
platform-specific features in system.h. This allows us to use an
unmodified copy of OpenJDK's class library, including its core classes
and augmented by a few VM-specific classes in the "avian" package.
We were incorrectly returning an empty array when the input was empty,
whereas we ought to return an array containing a single empty string.
When the pattern to match was empty, we went into a loop to create an
infinite list of empty strings, only to crash once we've run out of
memory. This commit addresses both problems.
The shiftLeftC function in powerpc.cpp was miscompiling such shifts,
leading to crashes due to illegal instructions and other weirdness due
to instructions that meant something completely different. This
commit fixes that and adds a test to Longs.java to make sure it stays
fixed.
Previously, the stack frame mapping code (responsible for statically
calculating the map of GC roots for a method's stack frame during JIT
compilation) would assume that the map of GC roots on entry to an
exception handler is the same as on entry to the "try" block which the
handler is attached to. Technically, this is true, but the algorithm
we use does not consider whether a local variable is still "live"
(i.e. will be read later) when calculating the map - only whether we
can expect to find a reference there via normal (non-exceptional)
control flow. This can backfire if, within a "try" block, the stack
location which held an object reference on entry to the block gets
overwritten with a non-reference (i.e. a primitive). If an exception
is later thrown from such a block, we might end up trying to treat
that non-reference as a reference during GC, which will crash the VM.
The ideal way to fix this is to calculate the true interval for which
each value is live and use that to produce the stack frame maps. This
would provide the added benefit of ensuring that the garbage collector
does not visit references which, although still present on the stack,
will not be used again.
However, this commit uses the less invasive strategy of ANDing
together the root maps at each GC point within a "try" block and using
the result as the map on entry to the corresponding exception
handler(s). This should give us safe, if not optimal, results. Later
on, we can refine it as described above.
We were miscompiling methods which contained getfield, getstatic,
putfield, or putstatic instructions for volatile 64-bit primitives on
32-bit PowerPC due to not noticing that values in registers are clobbered
across function calls.
The solution is to create a separate Compiler::Operand instance for each
object monitor reference before and after the function call to avoid
confusing the compiler. To avoid duplicate entries in the constant pool,
we add code look for and, if found, reuse any existing entry for the same
constant.
