Fractal packaging & deployment, bis

[Eric Bruneton <Eric.Bruneton@xxxxxxxxxxxxxxxxxxxx>]

to continue the "discussion" on this topic (in fact we got very little 
feedback), here is a document that summarizes the result of a meeting 
on this subject that took place last week, between INRIA and FTR&D. 
Comments are more than welcome, they are needed!

the DREAM and Fractal teams

Title: packaging

1 First attempts

We first thought that components, packages and modules (see below for a
definition and discussion about modules) where three independent
concepts, i.e. that the content of a package (resp. a module) was not
necessarily equal to the content of a component. Indeed we thought that
a package (resp. a module) could contain one or more components, only
some parts of a component (in the case of packages, this is necessary
because of some legal constraints: for example, some standard Java APIs
from Sun cannot be packaged with other software), or even no component
at all (in the case of "legacy" packages, such as Linux packages, or in
the case of "data" only packages).

We then focused on the relations between components and packages, and
the management of dependencies and versions. At first, since we thought
components and packages were completely independent, it seemed that it
was necessary to manage dependencies and versions both for components
and packages. Which means that a package would have its own
dependencies, its own version, and that each component it contains
would also have its own dependencies and its own version:

package P v1.0   requires     package Q v1.1   contains     component a v2.1       sub component b v1.0     component b v1.0       sub component c v2.0

package Q v1.1   requires     nothing   contains     component c v2.0

Although packages and components are independent, their dependencies
and versions are not: some consistency constraints must be ensured
between the two systems (for example, the packages required by P must
contain the components that are required by the components contained in
P, with the correct versions). Another problem is that two formalisms
are required to express the packages versions and dependencies, and the
component versions and dependencies (the Fractal ADL, plus a new
formalism for packages). All this seemed overly complicated, and so we
then tried to eliminate the need for two inter related dependencies and
versioning systems.

A first idea was to limit the role of packages to an exchange format
between component
repositories: a package would be created to get a copy of some
components (explicitely designated by the user) inside a repository,
sent over a network to another repository where the components it
contains would be extracted and put inside the repository. After this
the package would be destroyed. A package would then be a short lived
entity, and would therefore not require a version. Package dependencies
are also not required in this scenario. In fact the package concept
would become almost transparent (the only abstraction visible to users,
and managed by component repositories, would be components). But it was
not sure that this solution could handle legacy packages containing no
Fractal component at all. Moreover, the package concept in this
solution is almost
transparent, but it is still there and different from the component
concept.

So we then tried to see if the package and component concepts could be
unified, and we found that the answer is yes. The solution we found is
described below.

2 Fractal packages

Definition: a Fractal package A.far
is a Fractal component A in a serialized form, which is described by a
Fractal ADL definition A.fractal contained in the package itself.

This definition unifies the package and component concepts, in the
sense that a package is just a special form of a component. As a
consequence, the dependencies of a package are the dependencies of the
corresponding component (which, in the Fractal model can only be of two
sorts: dependencies through component encapsulation, and dependencies
through component interfaces and bindings), and the version of a
package is the version of the corresponding component.

The precise format used to serialize a component is not defined yet,
but a package will at least contain one or more Fractal ADL file(s).
The only mandatory ADL file is the file describing the top level
component to which the package corresponds. The package can also
contain some of the ADL files describing the sub components of this top
level component. The sub components for which there is no ADL file are
supposed to be packaged separately. 

For example, the two packages of the previous section must now be named
and organized as follows:

package A   A.fractal:     <definition name="A" version="2.1">       <component name="b" definition="B" version="1.0"/>       ...     </definition>   B.fractal:     <definition name="B" version="1.0">       <component name="c" definition="C" version="2.0"/>       ...     </definition>   ... other, non ADL files ...

package C   C.fractal:     <definition name="C" version ="2.0">       ...     </definition>   ... other, non ADL files ...

In this example, package A is supposed to contain all the code
necessary to create A and, recursively, all its sub components for
which the package contains an ADL definition. The other sub components,
such as "c", are supposed to be packaged in their own packages (and not as hidden sub
components - such as B - in arbitrary packages, otherwise they would be
impossible to find). The ability of packages to include sub components
that cannot be accessed independently outside the package corresponds
to the ability of Fractal components to expose or hide (some parts of)
their content, by providing or not the ContentController interface.

2.1 Package versions and dependencies

The version of a package A is the version specified in the A.fractal
file it contains. 

The two types of dependencies between Fractal components give two types
of dependencies between Fractal packages:

• a containment dependency gives a strong dependency between two packages (in the above example, the containement of c inside b, gives a strong dependency between package A and package C).
• a dependency through interfaces and bindings gives a loose dependency between packages (in the example below, the mandatory client interface "c" gives a loose dependency between package D and any package that provides the "I" interface; an optional client interface gives optional package dependencies).

package D
  D.fractal:
    <definition name="D" version="1.3">
      <interface name="s" role="server" signature="I"/>
      <interface name="c" role="client" signature="I"/>
      ...
    </definition>
  ... other, non ADL files ...

2.2 Package installation

The ADL files inside a Fractal package can be processed in two ways:

• by a Fractal component factory to create corresponding Fractal component or Fractal template instances,
• by a package installation system, such as apt or dselect for debian packages. In this case the ADL files are processed to find the dependencies of the package to other packages. When trying to "install" (this term is not precisely defined here) a package A, the tool can then find, recursively, all the packages that are needed to install A, and can install them automatically (more precisely the strong package dependencies can be resolved automatically; user input or user policies are needed to select one package between all the packages that could resolve a loose dependency). The sources where the packages can be downloaded must be configured outside the package themselves (as with dselect; this allows users to choose their own package sources, based on bandwith or security constraints for example).
  

2.3 Legacy packages

It may seem that the above solution is very specific to the Fractal
model and to its ADL, and that it is therefore impossible to handle
legacy packages (such as a Java library, a native program or a data
archive) with this solution. In fact, since everything is optional and
extensible in the Fractal model, and in the Fractal ADL, we think it is
possible to describe legacy packages as Fractal packages, in the same
way that it is possible to consider plain old Java objects as Fractal
components (of level 0, i.e. with no introspection and no control
capabilities). In fact, the goal of the "everything is optional"
feature is precisely to be able to easily integrate legacy components
in the model.

A legacy package which requires precisely defined implementations of
other packages can be viewed as a Fractal component which contains
other components, which does not provide or require any interface, and
whose implementation is just a list of files (for an application, the
executable file used to start the application can be viewed as the
constructor of the application). For example, a "Zip" package requiring
a C library provided by a "Libc6" package could be described as follows:

package Zip   Zip.fractal:     <definition name="Zip" version="3.1.4">       <component name="libc" definition="Libc6" version="1.0"/>     </definition>   ... program and documentation files ...

package Libc6   Libc6.fractal:     <definition name="Libc6" version ="3.1.4">       ...     </definition>   ... libc.so.6, ...

A legacy package which requires any other package that provide a given
protocol or API can be viewed as a Fractal component with a client
interface representing this protocol or this API, which can be bound to
any component providing this interface. For example, an"Eclipse"
package which requires a package implementing a Java Virtual Machine,
can be described as follows (here we see that interface versions are
needed, in order to specify "JVM version 1.4". Note also that the "JVM"
interface is not a Java interface, but this is not a problem, since
Fractal components are not limited to a specific language):

package Eclipe   Eclipse.fractal:     <definition name="Eclipse" version="3.0M8">       <interface name="jvm" role="client" signature="JVM"/>     </definition>   ... jar and documentation files ...

package Blackdown   Blackdown.fractal:     <definition name="Blackdown" version ="1.0">       <interface name="jvm" role="server" signature="JVM"/>     </definition>   ... program and documentation files ...

Information usually found in legacy packages, such as author name,
comments, list of files in the package, checksum, digital signature,
.... can of course easily be described in a Fractal ADL module. But it is not
sure yet if the two types of
dependencies bewteen Fractal components, and the extensibility of the
Fractal ADL, are sufficient or not to represent all the features of the
existing packaging systems.

3 Modules

3.1 The need for modules

At runtime, a Fractal application is made of one or several components. Each component is made of one or several code snippets. These snippets are classes in a Java-based application, .so or .dll files in a C-based application, etc. One of the problems faced by Fractal application developers is the runtime management of these different snippets. In particular, it is difficult to simultaneously handle different versions of some code in the same address space. The following figure illustrates such a case: components A, B and C, D are bound using a Push interface that defines a push method with the following signature: void push (Message m). Nevertheless, components A and B require version 1.0 of Message, whereas components C and D require version 1.1 of Message.

The solution we propose is to associate code snippets with entities, called modules, that act as naming contexts. The role of a module is to map names to code snippets. Modules are created and managed by the Fractal bootstrap. Using modules, there exist two solutions to the above-mentioned problem:

1. The two versions of Message are registered under two different names. As a consequence, components A, B and C, D refer to messages using two different names (e.g. "message_v1_0" and "message_v1_1).
2. Components A, B and C, D use two different modules. As a consequence components A, B, C, and D can use the same name to refer to Message.

3.2 Definition of a module

As described in the previous section, the role of a module is to map names to code snippets. For that purpose, a module must implement the Binder interface defined in the ObjectWeb's naming context API.
package org.objectweb.naming;

package org.objectweb.naming;
interface Name {
   NamingContext getNamingContext ();
   byte[] encode () throws NamingException;
}
interface NamingContext {
   Name export (any o, any hints) throws NamingException;
   Name decode (byte[] b) throws NamingException;
}
interface Binder extends NamingContext {
   any bind (Name n, any hints) throws NamingException;
}

A description of this API in the context of component interfaces can be found in the Fractal specifications. In the particular case we are describing, the export operation is implemented by a module to create a name for a given code snippet. This operation takes as argument a code snippet and optional hints, and returns a name for this snippet. A module also implements a decode operation to deserialize names in serialized form. Finally, the bind method gives access to exported code snippets given their name.

Typical examples of modules are Java class loaders. Java class loaders are naming contexts that manage classes. Let us take the example of an URLClassLoader. Exported classes are the ones that are accessible from the URLs passed as parameter to the class loader's constructor. Names mapped to exported classes are Java strings, e.g. "org.objectweb.fractal.Push". The bind method corresponds to the loadClass method that returns a class given its name.

Note that nothing is said on the way modules should work. For instance, modules can cooperate: a module can use another module to access some code snippets. This is the case of Java class loaders which are organized in a tree-like hierarchy: a class loader always has a father and may have one or several children. Java class loaders delegate calls to the loadClass method to their parent class loader.

3.3 Constraints imposed by modules

The capabilities of modules are often constrained by the language they are associated to. For instance, a typical constraint imposed by Java class loaders is that names created by the export method are just a String representation of the class name (including the package name), e.g. "org.objectweb.fractal.Message". As a consequence, a class loader does not allow two versions of the same class to be exported.

Another typical limitation is that it is impossible to change a mapping, i.e. a name is mapped to the same class for the whole life of the class loader. As a consequence, a component that requires changing the version of a class it is using must change the class loader it is using.

A last constraint imposed by Java class loaders is that two indentical instances of the same class loaded by two different class loaders are incompatible (e.g. a cast would raise a ClassCastException exception). As a consequence, if the class "org.objectweb.fractal.Message" is loaded by two different modules (one for component A, and one for component B), it will not be possible for A and B to exchange objects of this class.

3.4 Integrating modules with the ADL

The ADL must allow the application developer to specify versions of the code used by a component. For that purpose, it can add a "version" attribute to "interface" and "content" elements. It can also add "shared-code" sub-elements to specify code snippets that should be used in particular versions. A "shared-code" element has two attributes: "name" and "version". Note that "name" is not necessarily the name used to retrieve this code snippet from a module. Each of these elements (versioned interface, versioned content, and shared-code) must uniquely identify the code snippets they refer to.

Provided this information about code snippets used in a component, the deployment process must answer the following questions:

1. In which modules the specified snippets are exported?
2. Should two different snippets be loaded in the same module or not?

Answers to these questions depend on the module system that is used. Let us, for instance, consider a module system that allows several versions of the same code snippet to be exported under different names in a same module. A solution to the above-mentioned problem is to export all the code snippets into the same module. This solution is comparable to the one implemented in the SOFA Java-based component model, which uses bytecode manipulation to dynamically change the name of the classes from which components are built. Note however that this solution has some drawbacks: it forbids the use of methods such as Class.forName().

On the contrary, this solution cannot be implemented with modules behaving like class loaders do. In such module system, the first question can be answered by asking each module for the different code snippets. The second question requires handling the version attributes and "shared-code" elements described in the ADL, as exemplified in the next section. 3.5 Example This section gives an example of a module system for the Java language. A module corresponds to a class loader.

Scenario 1:

The application is not supposed to be updated. Moreover two different versions of a same code snippet are not allowed. The chosen solution is to have one module that is used to load every code snippet. This is the solution that is used in the current version of Julia.

Scenario 2:

The application developer requires the ability to update the application. For instance, he wants to be able to dynamically change the implementation of a component. Such change can result in simultaneously having two different versions of a same interface (one implemented by an "old" component, and one implemented by the "updated" component).

Due to class loader limitations mentioned in section 3.3,

• Each component must be associated to separate modules,
• Interfaces and implementation of a component must be exported into different modules,
• Classes of objects exchanged by two components (called shared classes) must be exported into a third module,
• Classes used by component implementation and never exchanged with other components (called inner classes) can be exported in the module associated to the component implementation.

Classes associated to interfaces and implementation can be deduced from the ADL description. Shared classes can be specified using the "shared-code" element previously described. Note that additional information could be provided in the ADL to impose another module organization. For instance, two components could be associated to the same modules if the same updates have to be done on both components.

Attachment: fractal_code_version.JPG
Description: JPEG image