Eclipse APT Design Notes

# Overview

Annotation processors may implement either the Java 5 com.sun.mirror.apt interfaces or the Java 6 javax.annotation.processing interfaces. The two interfaces are functionally similar but the details are different. Note that although the Java 5 apt tool has been deprecated by Sun, it is still included in the Java 6 JDK and developers continue to write processors against it. However, the com.sun.mirror.apt interface is considered proprietary and there are no Java compliance tests which validate it.

# Build Process

The JDT system performs three kinds of build: full build, incremental build, and reconcile. When a single file is edited and saved (if autobuild is enabled) or manually built, an incremental build is performed: only the affected files will be rebuilt. This will include the edited file as well as any other files with relevant dependencies on the edited file. After a clean, or when incremental build detects an overwhelming number of files needing to be rebuilt, a full build will be performed; in this case every file in the project will be rebuilt. During an editing session, prior to saving, a file will frequently be reconciled, in order to identify errors while typing. Reconcile operates on only a single file at a time.

Java 5 and Java 6 processors are handled by different code and interact with the Eclipse Java compiler in different ways and at different points in the build cycle. This is partly for historical reasons and partly in order to support Java 6 processors in the context of the command-line Eclipse compiler (ecj.jar). The Java 5 annotation processing code is built against the JDT's public API; the Java 6 code uses internal interfaces within the compiler, which was necessary because the public APIs use platform classes not available in the command-line package. A pleasant consequence is that the Java 6 annotation processing code is somewhat easier to understand.

The main challenge of introducing APT into Eclipse is that annotation processors can contribute additional Java types, which must in turn be compiled; this compilation can affect previously compiled types, because a formerly unresolved reference can be resolved. Because of this, it is necessary for annotation processing to be implemented within the compile process, rather than as a later build step.

TODO: overview of build process, i.e., when Java 5 processors are called relative to Java 6 processors

## Java 5 Build-time Processing

The invocation path for Java 5 processors is quite complex, in part because when Java 5 processing was implemented it was not possible to integrate it closely with the compiler, and in part for silly historical reasons that serve no current purpose. It is ripe for refactoring.

A typical call stack when a Java 5 processor is invoked looks like this:

    	SomeProcessor.process() line: xxx
    	APTDispatchRunnable.dispatchToFileBasedProcessor(AbstractCompilationEnv, boolean, boolean) line: 628
    	APTDispatchRunnable.runAPTInFileBasedMode(BuildEnv) line: 317
    	APTDispatchRunnable.build(BuildEnv) line: 655
    	APTDispatchRunnable.access$1(APTDispatchRunnable, BuildEnv) line: 647
    	APTDispatchRunnable$1.run(AbstractCompilationEnv) line: 265
    	BuildEnv$CallbackRequestor.acceptBinding(String, IBinding) line: 611
    	CompilationUnitResolver.resolve(ICompilationUnit[], String[], ASTRequestor, int, Map, WorkingCopyOwner, int) line: 766
    	CompilationUnitResolver.resolve(ICompilationUnit[], String[], ASTRequestor, int, Map, IJavaProject, WorkingCopyOwner, int, IProgressMonitor) line: 478
    	ASTParser.createASTs(ICompilationUnit[], String[], ASTRequestor, IProgressMonitor) line: 737
    	BaseProcessorEnv.createASTs(IJavaProject, ICompilationUnit[], ASTRequestor) line: 856
    	BuildEnv.createASTs(BuildContext[]) line: 356
    	AbstractCompilationEnv.newBuildEnv(BuildContext[], BuildContext[], IJavaProject, AbstractCompilationEnv$EnvCallback) line: 111
    	APTDispatchRunnable.build() line: 271
    	APTDispatchRunnable.run(IProgressMonitor) line: 217
    	Workspace.run(IWorkspaceRunnable, ISchedulingRule, int, IProgressMonitor) line: 1800
    	APTDispatchRunnable.runAPTDuringBuild(BuildContext[], BuildContext[], Map<IFile,CategorizedProblem[]>, AptProject, Map<AnnotationProcessorFactory,Attributes>, Set<AnnotationProcessorFactory>, boolean) line: 142
    	AptCompilationParticipant.processAnnotations(BuildContext[]) line: 193
    	IncrementalImageBuilder(AbstractImageBuilder).processAnnotations(CompilationParticipantResult[]) line: 627
    	IncrementalImageBuilder(AbstractImageBuilder).compile(SourceFile[]) line: 338
    	IncrementalImageBuilder.build(SimpleLookupTable) line: 134
    	JavaBuilder.buildDeltas(SimpleLookupTable) line: 265
    	JavaBuilder.build(int, Map, IProgressMonitor) line: 193
    	[higher calls omitted for brevity]

Java 5 processors are invoked via the CompilationParticipant interface. The Java compiler calls AptCompilationParticipant.processAnnotations(), passing in a list of all the files being built. Note that the compiler may break large projects into multiple groups of files, resulting in multiple calls to processAnnotations(). Files are marked with whether or not they contain annotations.

Within processAnnotations(), annotation processor factories are discovered from the processor factory path (and cached for subsequent invocations); factories and files are then passed in to APTDispatchRunnable.runAPTDuringBuild(). This in turn stores the information in a new instance of APTDispatchRunnable, and invokes its run() method within a workspace.run() operation.

The run() method invokes BuildEnv.newBuildEnv(), passing in the AptDispatchRunnable as a callback. The newBuildEnv() method creates a BuildEnv, gets compilation units for each of the files being compiled, and then requests ASTs for each of the files. The parser is passed an ASTRequestor callback that is implemented by an inner class of the BuildEnv; in this way, when the ASTs are ready for consumption, APTDispatchRunnable.build(BuildEnv) is ultimately called, within the ASTRequestor callback.

Finally, within build(), processors are selected on the basis of the annotations they support, and each processor's process() method is called on each appropriate file, configuring the BuildEnv before each process() invocation so that it will provide the correct data. The details of this vary depending on whether the processor is normal (file-based) or is marked as requiring batch mode.

### Batch-mode Processors

Processors can be marked in the Advanced Factory Path Options dialog as requiring batch mode. If any of the processors on the project's factory path require batch mode, AptDispatchRunnable.build(BuildEnv) calls runAPTInMixedMode(); if none of the processors requires batch mode, runAPTInFileBasedMode() is called instead. The chief difference is that batch-mode processors are only run during a full build, and that the batch-mode implementation of AnnotationProcessorEnvironment.getTypeDeclarations() returns all the compilation units in the build, whereas the normal implementation returns only a single compilation unit at a time.

Additionally, a separate classloader is used to load batch-mode processors, and it is discarded after each build; thus the classes are reloaded for each build, and static variables are reinitialized. This is important because some processors written with command-line (non-incremental) compilation in mind store dynamic state in static variables and thus cannot be invoked multiple times.

### File-mode Processors

Most processors are designed (or at least should be designed) to support incremental compilation, meaning that within the AnnotationProcessor.process() invocation, if the processor calls AnnotationProcessorEnvironment.getTypeDeclarations(), it will only retrieve a single type at a time; the process() method will be called repeatedly during the course of a build so that all files are eventually processed.

## Java 6 Build-time Processing

The invocation process for Java 6 processors is more straightforward. A typical call stack looks like this:

    	ModelTesterProc.process(Set<TypeElement>, RoundEnvironment) line: 107
    	RoundDispatcher.handleProcessor(ProcessorInfo) line: 139
    	RoundDispatcher.round() line: 121
    	IdeAnnotationProcessorManager(BaseAnnotationProcessorManager).processAnnotations(CompilationUnitDeclaration[], ReferenceBinding[], boolean) line: 159
    	IdeAnnotationProcessorManager.processAnnotations(CompilationUnitDeclaration[], ReferenceBinding[], boolean) line: 134
    	Compiler.processAnnotations() line: 810
    	Compiler.compile(ICompilationUnit[]) line: 428
    	IncrementalImageBuilder(AbstractImageBuilder).compile(SourceFile[], SourceFile[], boolean) line: 364
    	IncrementalImageBuilder.compile(SourceFile[], SourceFile[], boolean) line: 321
    	IncrementalImageBuilder(AbstractImageBuilder).compile(SourceFile[]) line: 301
    	IncrementalImageBuilder.build(SimpleLookupTable) line: 134
    	JavaBuilder.buildDeltas(SimpleLookupTable) line: 265
    	JavaBuilder.build(int, Map, IProgressMonitor) line: 193
    	[higher calls omitted for brevity]

Note that the processing is performed by an IdeAnnotationProcessorManager; this is distinguished from the BatchAnnotationProcessorManager, which would be used if processing was invoked from a command line (ecj.jar) compilation. Command-line compilation with ecj.jar uses the same typesystem implementation as IDE compilation, but somewhat different Messager and Filer implementations.

## Reconcile-time Processing

Reconcile-time processing is only supported for Java 5 processors. This is chiefly because, after developing the Java 5 processing implementation, it became clear that writing highly-performant annotation processors requires a level of compiler experience that most programmers do not have, and if less-performant processors are inserted into the reconcile step, reconcile can become painfully and confusingly slow. Thus when the Java 6 processing implementation was added it was decided to not support reconcile-time processing. Eclipse's effectiveness as an IDE depends on reconcile being an extremely fast operation. In hindsight, edit-time processing should have been limited to reporting errors, and should have been implemented as a background validation task rather than within the reconcile step.

TODO: describe practical differences between reconcile and build.

The stack trace of a typical reconcile-time invocation looks like this:

    	SomeProcessor.process() line: xxx
    	APTDispatchRunnable.dispatchToFileBasedProcessor(AbstractCompilationEnv, boolean, boolean) line: 628
    	APTDispatchRunnable.access$0(APTDispatchRunnable, AbstractCompilationEnv, boolean, boolean) line: 598
    	APTDispatchRunnable$ReconcileEnvCallback.run(AbstractCompilationEnv) line: 77
    	ReconcileEnv$CallbackRequestor.acceptBinding(String, IBinding) line: 135
    	CompilationUnitResolver.resolve(ICompilationUnit[], String[], ASTRequestor, int, Map, WorkingCopyOwner, int) line: 766
    	CompilationUnitResolver.resolve(ICompilationUnit[], String[], ASTRequestor, int, Map, IJavaProject, WorkingCopyOwner, int, IProgressMonitor) line: 478
    	ASTParser.createASTs(ICompilationUnit[], String[], ASTRequestor, IProgressMonitor) line: 737
    	BaseProcessorEnv.createASTs(IJavaProject, ICompilationUnit[], ASTRequestor) line: 856
    	ReconcileEnv.openPipeline() line: 108
    	AbstractCompilationEnv.newReconcileEnv(ReconcileContext, AbstractCompilationEnv$EnvCallback) line: 97
    	APTDispatchRunnable.reconcile(ReconcileContext, IJavaProject) line: 211
    	APTDispatchRunnable.runAPTDuringReconcile(ReconcileContext, AptProject, Map<AnnotationProcessorFactory,Attributes>) line: 159
    	AptCompilationParticipant.reconcile(ReconcileContext) line: 223
    	ReconcileWorkingCopyOperation$1.run() line: 257
    	SafeRunner.run(ISafeRunnable) line: 42
    	ReconcileWorkingCopyOperation.notifyParticipants(CompilationUnit) line: 244
    	ReconcileWorkingCopyOperation.executeOperation() line: 94
    	ReconcileWorkingCopyOperation(JavaModelOperation).run(IProgressMonitor) line: 728
    	ReconcileWorkingCopyOperation(JavaModelOperation).runOperation(IProgressMonitor) line: 788
    	CompilationUnit.reconcile(int, int, WorkingCopyOwner, IProgressMonitor) line: 1242
    	JavaReconcilingStrategy.reconcile(ICompilationUnit, boolean) line: 126
    	[higher calls omitted for brevity]

# Processor Callbacks

When annotation processors are invoked, they can in turn call back into the compiler through various interfaces in order to generate new files and report errors. The interfaces are naturally not the same as those used within the JDT compiler, so wrappers and proxies are employed to glue the different systems together. The most significant difference in systems is that, in both the Java 5 and Java 6 processor interfaces, the typesystem is separated into Type and Element hierarchies, in which a Type object represents a particular reification of a type while an Element represents a declaration in code. The distinction is easiest to understand in the context of generics: for example, the following code sample contains declarations of classes Lister and Test, which contain declarations of method get() and fields 'strings' and 'numbers' respectively. The type of the return of get() is List<T>, the type of strings is Lister<String>, and the type of numbers is Lister<Number>. Note that the single declaration of Lister<T> was able to generate multiple Types.

      import java.util.List;
      class Lister<T> {
        List<T> get() { ... }
      }
      class Test {
        Lister<String> strings;
        Lister<Number> numbers;
      }

## Java 5 Type System

TODO

## Java 5 Filer and Messager

TODO

## Java 6 Type System

TODO

## Java 6 Filer and Messager

TODO