OSGi and WSO2

The Carbon platform is a collection of totally independent components which can be added or removed from a solution dynamically. This behaviour is achieved through the use of Open Services Gateway Initiative(OSGi) framework. In OSGi, independent components are called bundles. Therefore, Carbon can also be referred to as a collection of highly dynamic bundles.

Legacy Java applications is collection of Jar files while OSGi applications are a collection of modules or bundles. Bundle is just a jar file with additional OSGi specific meta data in a MANIFEST.MF file. This file located in the META-INF folder of the jar file. Just as normal jar files, it can contains .class files, images, html files etc. Additionally, it can embed other jar files at the root level.

Since WSO2 Carbon is based on OSGi the platform can only be extend by adding OSGi bundles. A mechanism is provided to convert legacy jar files to OSGi bundles too.

OSGi uses Java package as the unit of information hiding. Hence OSGi allows bundles to share packages among other bundles or to hide package from other bundles. OSGi achieves this level of information hiding by assigning a separate class loader for a bundle. And this bundle class loader can load classes from system class loader, other bundles class loader and from the bundle’s jar files.

A system bundle is a bundle which represent the OSGi framework. Implementation classes of the OSGi framework reside inside the system bundle. It exports packages loaded from the system classpath. i.e. from the application class loader.
Fragments are bundle which can be attached to host bundles by the framework. Once attached fragments are considered as part of the host bundle and the fragment shares the host bundles class loader.
Framework extension bundles is a special type of fragment bundles. They can be attached to the system bundle. They can provide functionality which should be reside on the boot class path. This is necessary because certain Java package implementation assume that they are on the boot class patch or required to be available to all clients. An example of a boot class path extension is an implementation of java.sql such as JSR 169. One other use case of extension bundles is to provide optional parts of the framework implementation.

Source: https://wso2.com/library/knowledgebase/add-external-jar-libraries-wso2-carbon-based-products/

Creating a bundle for WSO2

When creating a bundle to be deployed within the WSO2 OSGi container the following should be kept in mind.

Create a bundle for the new component, trying to re-use the dependencies (and specific version) known to exist already within the product. This minimises the inclusion of additional dependencies.
Where this is not possible, either wrap the dependencies in their own OSGi bundle or embed them within the component bundle. Refer to the OSGi dependencies documentation.
If classes in the bundle is loaded by name, i.e. via a config file, then ensure the Import-Package instruction includes *;resolution:=optional directive as the last value.
When using libraries including Java 9 or later code, add the -noee directive to ensure the bundle loads in WSO2. See the section below about problems since Java 9.

Manifest Headers

Thie MANIFEST.MF file contains a set of manifest headers that indicate OSGi meta data. All these headers are defined in the section 3.2.1 of the OSGi specification. Most important headers which are frequently used are described below:

Almost all the Manifest Headers defined in the specification are optional. If some header is mandatory, it is mentioned under the particular header description below.

Bundle-Classpath

This is a comma separated list of directories and jar file path names which exist within the bundle itself. Bundle class loader can see the resources located in these specified locations. The period ('.') is the default value which indicates the entire root directory of the bundle and is also the default value.

Bundle classpath example

Bundle-Classpath: org.wso2.cabon.example, example.jar

Bundle-SymbolicName

This is the unique name give for the bundle. This is a mandatory header that must be set by the bundle author. Name should be based on the reverse domain name convention.

Bundle symbolic name example

Bundle-SymbolicName: org.wso2.carbon.example

Bundle-Version

This header specifies the version of this bundle. The default value is 0.0.0 . Using versions is important when different versions of the same bundle exist in the system. OSGi specification[9] defines a set of version rules under sections 3.2.4 and 3.2.5.

Bundle version example

Bundle-Version: 1.0.0

Export-Package

This header lists all the packages exported to the OSGi environment for this bundle. These packages either exist within bundle packages or inside some other jar file which is embedded within this bundle. Once this bundle is activated within the OSGi environment (In our case the Carbon framework), other bundles can use these exported packages by importing them. Each package can be exported with a version (optional) which is specific to that particular package. This version should be specified in front of the package and a semi colon (';') is used to separate the package and the version. Using versions is important when different versions of the same package exist in the system at the same time. All exported packages should be given as a comma separated list.

Export package example

Export-Package: org.wso2.carbon.foo;version="1.4.0",  org.wso2.carbon.bar;version="1.2.1"

Import-Package

This is also a comma separated list of packages with versions (optional). This is the list of packages which are needed by this bundle from other bundles of the OSGi environment. Those packages should be exported from some other bundle before it can be imported. If all the imported packages are not found, this particular bundle will not be activated properly once it is dropped into the OSGi environment.

Import package example

Import-Package: org.wso2.carbon.foo;version="1.1.0",  org.wso2.carbon.bar;version="1.1.6"

Although this is the default behaviour, you can make the resolution of a particular package optional. In that case, even if that package is not found in the OSGi environment, bundle will be running.

Optional resolution example

Import-Package: org.wso2.carbon.foo; resolution:=optional; version="1.1.0"

In cases where WSO2 loads classes by their name specified in config files an additional wildcard optional directive has to be added.

For instance, when setting the KeyValidationHandler class;

api-manager.xml

<APIKeyValidator>
    ...
    <KeyValidationHandlerClassName>org.wso2.carbon.apimgt.keymgt.handlers.DefaultKeyValidationHandler</KeyValidationHandlerClassName>
</APIKeyValidator>

In such cases add as the last directive of Import-Package;

Wildcard optional directive

Import-Package: <other imports>, *;resolution:=optional

DynamicImport-Package

This header contains a comma-separated list of package names that should be dynamically imported when needed while the bundle is active within the OSGi environment. These packages are not needed to start the bundle. But at run time these are loaded when needed.

Dynamic import example

DynamicImport-Package: org.wso2.carbon.example

It is a best practice to use this header for all dynamically needed packages (indicated by '*') when you are writing complex bundles as you may forget to import some packages using Import-Package header.

Dynamic import all packages

DynamicImport-Package: *

Private-Package

This header also contains a comma separated list of packages which are not exported to the outside. Those packages are private to the bundle.

Private package example

Private-Package: org.wso2.carbon.example.internal

Fragment-Host

A bundle can be included into the OSGi environment as an extension to an already existing bundle. In this case, the existing bundle is called the host bundle and the attached bundle is called the fragment bundle. So if your bundle should be a fragment to some other bundle, you have to specify the host bundle under this Fragment-Host header. Resources in the fragment bundle are added to the host bundle class path.

Fragment host example

Fragment-Host: org.wso2.carbon.ui

Require-Bundle

If your bundle needs all exports made by some other bundle, you can specify that bundle under this header. If this header is set, your bundle can’t be started if the other bundle doesn’t exist in the system. It’s not a good practice to use this header and best method is to import the needed packages using Import-Package header or DynamicImport-Package.

Required bundle example

Require-Bundle: org.wso2.carbon.example

Embed-Dependency

If and external JAR should be embeded within the bundle jar, this header should be set. This is a pipe ('|') separated list of artifact ids of the jars to embedded.

Embed dependency example

Embed-Dependency: org.apache.foo | org.wso2.bar

Bundle-Activator

This header specifies the class used to start and stop the bundle within the OSGi framework. This activator class must implement the BundleActivator interface which has a start method and a stop method. The start method is called by the OSGi framework when the bundle starts after installing it in an environment. If you have any particular logic to be executed at start time, you can include it in this start method. Similarly, stop method is executed when the bundle is uninstalled from the OSGi environment.

It is important to remember that a bundle Activator class is optional for an OSGi bundle. If you don’t have any specific logic to be executed on bundle startup or shutdown, you don’t have to write an activator class and omit the Bundle-Activator header.

Example

Bundle-Activator: org.wso2.carbon.example.Activator

Bundle-ManifestVersion

This header indicates the OSGi specification version implemented in this bundle. The default value is 1 and indicates version 3. Value 2 indicates version 4 and higher.

Bundle manifest example

Bundle-ManifestVersion: 2

Version 4 is used by all WSO2 Carbon related bundles.

Creating the manifest headers

Manifest headers described above are the most commonly used headers in Carbon components. But there are some other rarely used headers which are defined in the OSGi specification.

Writing the MANIFEST.MF file is a complex task when the bundle is complex. For example, if you have many import and export packages, you have to list all those one by one in the MANIFEST.MF file. WSO2 Carbon uses Maven2 as the build system. Therefore, Maven bundle plugin is used to generate MANIFEST.MF file for all the Carbon components.

Users can write the MANIFEST.MF file manually or using a tool. It will work within the Carbon environment as far as it is written correct. But the easiest way to generate a correct MANIFEST.MF file is to use the Maven bundle plugin.

Maven bundle plugin

When constructing an OSGi bundle using Maven2, the first step is to write the normal pom.xml with all the needed dependencies and Maven attributes. Refer to the Maven2 documentation for additional information.

Once the project builds correctly use the maven-bundle-plugin under plugins section of the pom.xml to generate the OSGi meta data. The meta data explained above can be defined under the <instructions> section. When the project is built the bundle jar file will be created, including an OSGi compliant MANIFEST.MF file.

Sample configuration for a fictional package "my.app"

<plugin>
    <groupId>org.apache.felix</groupId>
    <artifactId>maven-bundle-plugin</artifactId>
    <version>${maven-bundle-plugin.version}</version>
    <extensions>true</extensions>
    <configuration>
        <instructions>
            <Bundle-SymbolicName>${project.artifactId}</Bundle-SymbolicName>
            <Bundle-Name>${project.artifactId}</Bundle-Name>
            <Bundle-Version>${project.version}</Bundle-Version>
            <Bundle-ClassPath>{maven-dependencies},.</Bundle-ClassPath>
            <Bundle-Vendor>>DeARX Services</Bundle-Vendor>
            <Bundle-Activator>my.app.Activator</Bundle-Activator>
            <Export-Package>
                my.app;version="${project.version}"
            </Export-Package>
            <Import-Package>
                org.wso2.carbon.apimgt.*;version="${carbon.apimgt.version}",
                org.apache.log4j.*; version="1.2.0",
                *;resolution:=optional
            </Import-Package>
            <Embed-Dependency>*;scope=compile|runtime;inline=false</Embed-Dependency>
            <Embed-Transitive>true</Embed-Transitive>
            <_noee>true</_noee>
        </instructions>
    </configuration>
</plugin>

The star (*) character can be used to indicate all packages under a particular parent package. The bundle plugin will write all matching packages into the MANIFEST.MF file.

Exported packages are removed from the imported list using '!' character to avoid cyclic dependencies as a best practice.

If no Import-Pachages directive is supplied the plugin will automatically write all external packages used within the bundle under the Import-Package section of the MANIFEST.MF file.

Handling local or transient dependencies

Dependencies can be provided as additional OSGi bundles or by embedding the dependencies. Refer the the article on OSGi dependencies.

However, in the above example any dependent libraries were embedded. These Maven dependencies were defined as per normal and then the OSGi bundle dependencies was declared as dependencies with the <scope>provided</scope> directive. This allows the Embed-Dependency to use a scope selector, as used in the example above. This technique will then only select those Maven dependencies not marked as provided for embedding.

For example;

<dependencies>
    <dependency>
        <groupId>org.wso2.carbon.apimgt</groupId>
        <artifactId>org.wso2.carbon.apimgt.usage.publisher</artifactId>
        <version>${carbon.apimgt.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-api</artifactId>
        <version>2.8.2</version>
    </dependency>
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-core</artifactId>
        <version>2.8.2</version>
    </dependency>
<dependencies>

An alternative approach is to use the package names in the Embed-Dependency directive, for example;

<Embed-Dependency>artifactId=log4j-api|log4j-core</Embed-Dependency>

The pipe (|) character has to be used to separate the artifactId s.

In the example above the dependencies are included as separate JAR files in the root of the bundle. The Bundle-Classpath uses a variable to provide an expansion of these library files.

The libraries can also be included using the inline directive, where each dependency JAR is extracted and placed amongst the class files of the bundle.

Using the inline directive for embedded dependencies

<Embed-Dependency>*;scope=compile|runtime;inline=true</Embed-Dependency>

Configuration files

The OSGi framework makes it difficult to load configuration files from the file system, as libraries does not share a common classpath.

Each bundle in the environment has a unique bundle classLoader which resolves into a unique classpath. In such scenarios there are three possible solutions which are:

Bundle the config files along with the library class files. This will ensure that config files always resides in the bundle classpath. The biggest disadvantage is that, you cannot edit the config files during server restarts.
If the third party library accepts the config file path via a system property, we can set that system property prior to invocation of the library. This is a very clean approach.
Bundle the config file as a fragment of the main library and attach them during the startup of the server.

More information can be found at the link below.

Source: https://docs.wso2.com/display/Carbon4411/Config+Files+for+Third+Party+JARs

Deployment

The OSGi repository of WSO2 Carbon is located inside the $CARBON_HOME/repository/components folder, where $CARBON_HOME is the folder where the product is installed.

The following directories are provided; (Source: https://docs.wso2.com/display/Carbon4411/Adding+External+Libraries)

Directory	Purpose
$CARBON_HOME/repository/components/dropins	Third-party libraries, which are already OSGI bundles.
$CARBON_HOME/repository/components/lib	JAR files that will be converted to OSGi bundles and copied to the dropins directory during server startup.
$CARBON_HOME/repository/components/extensions	JAR files that will be converted to OSGi Framework Extension bundles and copied to the dropins directory during server startup.

Problem with bundles created since Java 9

When Java 9 was released a new feature called multi-release jars was included in the Java 9 JDK. This allow several versions of the same class to be package at once, which can be consumption by different version of the runtime. For example, if you run on JDK 8, the Java runtime would use the Java 8 version of the class, but if you run on Java 9, it would use the Java 9 specific implementation.

This new feature result in the maven-bundle-plugin detecting the later module-info.class, which forces an incorrect setting of Require-Capability in the MANIFEST.MF file despite the bundle being built for JDK8 compatibility.

Incorrect manifest header setting

Require-Capability: osgi.ee;filter:="(&(osgi.ee=JavaSE)(version=9.0))"

WSO2, and other OSGi environments, might offer a lower version of the osgi.ee version, and then the bundle will not activate.

In WSO2 this can be seen by deploying the bundle, starting the application with the -DosgiConsole setting and using the console to inspect the state of the bundles.

For instance, run:

> ./bin/wso2server.sh -DosgiConsole
osgi> ss

The ss command will list the bundle as INSTALLED, but not ACTIVE.

For instance;

Example where bundle 260 is not ACTIVE

259     ACTIVE      org.wso2.carbon.apimgt.usage.client_6.4.50
260     INSTALLED   org.wso2.carbon.apimgt.usage.elkpublisher_7.0.0
261     ACTIVE      org.wso2.carbon.apimgt.usage.publisher_6.4.50

By diagnosing the non-active bundle the following information is revealed;

osgi> diag 260
reference:file:../dropins/org.wso2.carbon.apimgt.usage.elkpublisher-7.0.0.jar [260]
  Direct constraints which are unresolved:
    Missing required capability Require-Capability: osgi.ee; filter="(&(osgi.ee=JavaSE)(version=9.0))"

This is solved by building the bundle with the -noee instruction. For instance;

pom.xml - maven-bundle-plugin configuration section

<configuration>
    <instructions>
        <Bundle-SymbolicName>${project.artifactId}</Bundle-SymbolicName>
        <Bundle-Name>${project.artifactId}</Bundle-Name>
        <Bundle-Version>${project.version}</Bundle-Version>
        ...
        <_noee>true</_noee> (1)
    </instructions>
</configuration>

1	Set to true to disable the `osgi.ee` capability constraint.

Read more about what multi-release JARs are and the benefits and concerns about it at the Gradle blog.

Interact with runtime bundles

The WSO2 products ship with an included OSGi Console which can be activated by starting the product with the -DosgiConsole command line parameter.

Starting a WSO2 product with the OSGi Console enabled

sh wso2server.sh -DosgiConsole

Once the server has started press Enter to display the osgi prompt.

There are some OSGi commands useful in troubleshooting.

List

Use the ss command to list all bundles loaded by the OSGi container, which will include with their bundle ID and statuses.

ss

Start

This command can be used to start a particular bundle. If there is any trouble starting, reasons will be printed on the console.

start <bundle-id> (Ex: osgi> start 18)

Print

The b command can be used to print all meta-data related to this bundle. That includes imported packages, exported packages, host bundle, required bundles etc.

b <bundle-id> (Ex: osgi> b 18)

Diagnose

The diag command can be used to diagnose a particular bundle. It will show a list of missing imported packages.

diag <bundle-id> (Ex: osgi> diag 18)

Package dependency

The package command can be used to list all bundles which use a given package.

packages <package-name> (Ex: osgi> packages org.wso2.foo)

Install

This command can be used to install a bundle into a running OSGi environment. Use this command to install a bundle instead of copying it into plugins directory prior to starting the server. After installing, use start command to activate the bundle.

install file:<file-path> (Ex: osgi> install file:/home/temp/bundle1.jar)

Uninstall

This command can be used to remove a bundle from the OSGi environment.

uninstall <bundle-id> (Ex: osgi> uninstall 18)

Refresh

This command can be used to refresh the system. All package resolutions are refreshed when this is executed. Always refresh the system after uninstalling a bundle as a best practice.

refresh (Ex: osgi> refresh)