javaone 2016: life after modularity
TRANSCRIPT
An experience report
Dan Heidinga, J9 VM Interpreter Lead
@DanHeidinga
18Sept 2016
Life after modularity
Important disclaimers
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.
WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION
CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED.
ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED
ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR
INFRASTRUCTURE DIFFERENCES.
ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.
IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT
PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.
IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT
OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
– CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS
OR THEIR SUPPLIERS AND/OR LICENSORS
2
Who am I?
I've been involved with virtual machine development at IBM
since 2007 and am now the J9 Virtual Machine Team Lead.
J9 is IBM's independent implementation of the JVM.
I've represented IBM on both the JSR 292 ('invokedynamic')
and JSR 335 ('lambda') expert groups and lead J9's
implementation of both JSRs.
Currently leading my team in the adoption and implementation
of Java 9’s headline feature: Modularity.
I’ve also maintain the bytecode verifier and deal with various
other parts of the runtime.
3
Current JDK levels in production?
Java 8?
Java 7?
Java 6?
Java 5?
Java 1.4?
Have you tried Java 9 yet?
4
5
Pre-Java 9: wild west
https://www.flickr.com/photos/30456664@N03/5763126280
6
Enter Java 9
Why am I here?
The purpose of this talk is to share our experience implementing Modularity.
– We track the OpenJDK builds to implement the VM-side of features in J9.
– Running 10,000s of tests, we’ve tripped many of the immediate issues
Number one take away from this talk: Download JDK9 ea builds and test your app.
7
Migration path
This talk is organized in code snippets.
– Small sections of code that demonstrate the problem
– Not identical to the code we hit the problem with.
– Goal is small code snippets that are easy to digest.
Where possible, I’ve validated the issues have been experienced by others
– And have credited them when possible.
Were possible, I’ve shown how to work around the issues
– Some don’t have workarounds
– Some the EG are still investigating design changes
8
Module system overview
9
java.base
java.something
com.user.b
com.user.a
unnamed
Internal APIs are now hidden
There are “Java Platform SE APIs”, and then there are those useful, internal public types...
– Sometimes usage is for a defensible reason, e.g. performance, Unsafe operations,
extending the Platform
Java 9 will not discriminate!
– New module boundaries will only export public APIs to strangers
Your code may be clean, but how about your dependencies?
– Some types have migrated into API in later releases, e.g. Base64Encoder
Java 8 contains a tool called jdeps that can show you the damage$ jdeps -jdkinternals com.ibm.mq.explorer.jmsadmin_8.0.0.201502232022.jar
com.ibm.mq.explorer.jmsadmin_8.0.0.201502232022.jar -> java.naming
com.ibm.mq.explorer.jmsadmin.ui.JmsAdminCommon (com.ibm.mq.explorer.jmsadmin_8.0.0.201502232022.jar)
-> com.sun.jndi.ldap.LdapCtxFactory JDK internal API (java.naming)
com.ibm.mq.explorer.jmsadmin.ui.internal.contexts.AddJmsContextWizPage4 (com.ibm.mq.explorer.jmsadmin_8.0.0.201502232022.jar)
-> com.sun.jndi.ldap.LdapCtxFactory JDK internal API (java.naming)
Warning: JDK internal APIs are unsupported and private to JDK implementation that are
subject to be removed or changed incompatibly and could break your application.
Please modify your code to eliminate dependency on any JDK internal APIs.
For the most recent update on JDK internal API replacements, please check:
https://wiki.openjdk.java.net/display/JDK8/Java+Dependency+Analysis+Tool
IBM XML implementation is replaced
IBM XML implementation (XML4J, XLXPJ, XLTXE) is replaced by the OpenJDK version
Scan your code using the WAS Migration Toolkit to find issues
– https://developer.ibm.com/wasdev/downloads/#asset/tools-
Migration_Toolkit_for_Application_Binaries
java -jar binaryAppScanner.jar <binaryInputPath> --java9xml
java -jar binaryAppScanner.jar --help --java9xml
Scan your code using the jdeps tool included with Java 9
A migration guide is included in the IBM Java User Guides
Just run your app…
jre/bin/java: command not found
12
Just run your app…
13
The JDK directory layout has changed
Impact: If you rely on the file layout, your application may break
Recommendation: JDK9 early access builds are available
– Try them, feedback is critical to mitigate issues
.
├── bin
├── include
│ └── linux
├── jre
│ ├── bin
│ │ ├── classic
│ │ └── j9vm
│ ├── lib
│ │ ├── applet
│ │ ├── boot
│ │ ├── cmm
│ │ ├── deploy
│ │ ├── endorsed
│ │ ├── ext
│ │ ├── fonts
│ │ ├── management
│ │ ├── oblique-fonts
│ │ └── security
│ └── plugin
├── lib
└── properties
└── version
.
├── bin
├── conf
│ ├── management
│ └── security
├── include
│ └── linux
└── lib
├── modules
└── security
IBM Java 8 GA OpenJDK Java 9 dev
Old World (pre-JDK9) New World (JDK9)
Installs are JRE or JDK.JDKs contain a JRE, e.g. there are two bin directories.
There is now a single Java install that may contain development tools, or not.
Placing code in certain directories conveys rights, i.e. endorsed, and ext dirs.
Endorsed updates become “upgradeable modules”Extensions concept is abandoned.
Implementation is provided in various (well-known) JAR files such as rt.jar and tools.jar
No more JARs. IDEs etc will need to learn new file formats and locations.
Supporting paraphernalia such as timezone info and fonts can be seen
Implementation are hidden in modules
rt.jar / tools.jar replaced by lib/modules
In Java 8,
– Core class libraries for the platform were found in rt.jar
– Tools like javac where found in tools.jar
In Java 9, these classes are part of the new ‘lib/modules’ jimage file
Symptom: NoClassDefFoundError or ClassNotFoundException
Impact: fairly low
– More likely to break build scripts than application behavior
– Hand crafted jar URLs to these files will break
– Will need to use libraries to read the jimage format instead
Goodbye ext/ & endorsed/ directories
$ java -Djava.ext.dirs=ext Main-Djava.ext.dirs=ext is not supported. Use -classpath instead.Error: Could not create the Java Virtual Machine.Error: A fatal exception has occurred. Program will exit.
$ java -Djava.endorsed.dirs=ext Main-Djava.endorsed.dirs=ext is not supported. Endorsed standards and standalone APIs in modular form will be supported via the concept of upgradeable modules.Error: Could not create the Java Virtual Machine.Error: A fatal exception has occurred. Program will exit.
17
Goodbye ext/ & endorsed/ directories
Options:
Use “-cp <dirs>” instead
Use Java 9’s new “-upgrademodulepath <dirs>” option
Use “-Xbootclasspath/a:” to move them to the bootpath
–Only if you require elevated privileges
18
Java 8 Classloader hierarchy
19
Bootstrap Loader
Extension Loader
Application Loader
Loads classes from:• rt.jar• jre/lib/endorsed• -Djava.endorsed.dirs• -Xbootclasspath/a:• -Xbootclasspath/p:
Loads classes from:• jre/lib/ext• -Djava.ext.dirs
Loads classes from:• -cp
Java 9 Classloader hierarchy
20
Bootstrap Loader
Platform Loader
Application Loader
Loads classes from:• lib/modules• -Xbootclasspath/a:
Loads classes from:• lib/modules
Loads classes from:• -cp• -mp• -upgrademodulepath
Classloader invariants maintained
Bootloader classes still return null
Class.class.getClassLoader() == null
Loop to get extension / platform loader still works
ClassLoader iterator = Test.class.getClassLoader(); ClassLoader prev = null;while (iterator != null) {
prev = iterator;iterator = iterator.getParent();
}prev == extension / platform loader
21
Classloader parent example
22
Confirmation: https://issues.apache.org/jira/browse/SUREFIRE-1265
Classloader parent output
Works on Java 8, fails on Java 9
Classloaders should delegate to the Platform (extension) loader rather than the Bootloader
Can work around this so it runs on both 8 & 9 by using
ClassLoader.getSystemClassLoader().getParent()
or a getParent() loop to provide a different loader.
23
Classloader invariants maintained
Bootloader classes still return null
Class.class.getClassLoader() == null
Loop to get extension / platform loader still works
ClassLoader iterator = Test.class.getClassLoader(); ClassLoader prev = null;while (iterator != null) {
prev = iterator;iterator = iterator.getParent();
}prev == extension / platform loader
24
Who’s affected?
25
Bootstrap Loader Platform Loader Application Loader
java.*
jdk.*
See http://openjdk.java.net/jeps/261
URLClassloader casting
26
URLClassloader casting
ClassCastException is the only possible outcome.
Work around: none
– Well, nothing simple. The answer really depends on why you were casting
27
If all you wanted was URLs…
28
Confirmation: https://github.com/Mojang/LegacyLauncher/pull/11/files
getResource() APIs may return null
Java 8: jar:file:/home/vagrant/jdk8-111/jdk1.8.0_111/jre/lib/rt.jar!/java/lang/Class.class
Java 9: null
getResource doesn’t find named module resources
…. And all JDK classes are in named modules
See http://download.java.net/java/jdk9/docs/api/java/lang/ClassLoader.html
getResource hope
See http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000392.html
AccessibleObject.setAccessible’s kryptonite
32
Confirmation: https://issues.apache.org/jira/browse/CAMEL-10188
InaccessibleObjectException
Exception in thread "main" java.lang.reflect.InaccessibleObjectException: Cannot make a non-public member of class java.lang.reflect.AccessibleObject accessible
at jdk.internal.reflect.Reflection.throwInaccessibleObjectException(java.base@9-ea/Reflection.java:414)
at java.lang.reflect.AccessibleObject.checkCanSetAccessible(java.base@9-ea/AccessibleObject.java:190)
at java.lang.reflect.Method.checkCanSetAccessible(java.base@9-ea/Method.java:192)
at java.lang.reflect.Method.setAccessible(java.base@9-ea/Method.java:186)
at Test.main(Test.java:8)
33
Confirmation: https://issues.apache.org/jira/browse/CAMEL-10188
Most likely to cause problems for frameworks / libraries
Dealing with InaccessibleObjectException in Java 8
AccessibleObject#setAccessible’s new InaccessibleObjectException doesn’t exist in Java 8
– And can’t be caught in a try / catch block
Work around:
– Catch RuntimeException and check if the class name is InaccessibleObjectException
Better work around:
– Try not to use setAccessible
34
Confirmation: https://issues.apache.org/jira/browse/CAMEL-10188
Java version: From 1.8.x to 9.x
35
Java version: From 1.8.x to 9.x (fixed)
36
Java version: From 1.8.x to 9.x (fixed)
37
Java version: From 1.8.x to 9.x (fixed 9 only)
38
Increasing readability
--add-reads <source-module>=<target-module>
39
--add-reads java.naming=jdk.naming.rmi
java.lang.IllegalAccessException: Class com/sun/naming/internal/ResourceManager(module java.naming) can not access class com/sun/jndi/url/rmi/rmiURLContextFactory(module jdk.naming.rmi) because module java.naming does not read module jdk.naming.rmi
at java.lang.J9VMInternals.newInstanceImpl(java.base/Native Method)at java.lang.Class.newInstance(java.base/Class.java:1833)at ...
Breaking encapsulation
--add-exports <source-module>/<package>=<target-module>(,<target-module>)*
40
java.lang.IllegalAccessException: class sun.reflect.misc.Trampoline cannot access interface com.ibm.lang.management.JvmCpuMonitorMXBean (in module java.management) because module java.management does not export com.ibm.lang.management to unnamed module @59d80b37
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(java.management/MBeanIntrospector.java)at com.sun.jmx.mbeanserver.PerInterface.invoke(java.management/PerInterface.java:138)at com.sun.jmx.mbeanserver.MBeanSupport.invoke(java.management/MBeanSupport.java:252)at ….
Multiversion jar
Support multiple java versions with a single jar using different implementations
– A single jar
– Multiple implementations
jar --create --file mr.jar -C java8_dir Test.class --release 9 -C java9_dir Test.class
Take away
Try the JDK 9 early access builds.
Run JDEPs and fix the internal usages
Report your experience on the OpenJDK mailing lists.
42
43
Legal Notice
IBM and the IBM logo are trademarks or registered trademarks of IBM Corporation, in the United States, other
countries or both.
Java and all Java-based marks, among others, are trademarks or registered trademarks of Oracle in the United
States, other countries or both.
Other company, product and service names may be trademarks or service marks of others.
THE INFORMATION DISCUSSED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL
PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND
ACCURACY OF THE INFORMATION, IT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, AND IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT
OF THE USE OF, OR OTHERWISE RELATED TO, SUCH INFORMATION. ANY INFORMATION
CONCERNING IBM'S PRODUCT PLANS OR STRATEGY IS SUBJECT TO CHANGE BY IBM WITHOUT
NOTICE.