Mirror of the Rel4tion website/wiki source, view at <http://rel4tion.org>

Clone

HTTPS: git clone https://vervis.peers.community/repos/yEzqv

SSH: git clone USERNAME@vervis.peers.community:yEzqv

Branches

• master

Tags

manual.mdwn

This is the manual of Tosaf, a C++ plugin framework. The idea is to move this text into the source package because it needs to have separate version control and stable versions. In order to make different versions visible as web pages, a new Partager subdomain will be created. It will probably be implemented as a new system user which pulls the Tosaf git repo once a day/hour, builds the docs and installs locally under a “latest” folder. It will also keep compiled versions for the tags, i.e. stable versions, under folders whose names are the version numbers. This is similar to how the GNOME online API docs and manuals work. [[TODO|TODO/OPEN]] make this work and write a tutorial. See if there’s already some program/debian package which does things like this

Introduction

There are many existing plugin systems for C++. Many of them are specific to the problem domain for which they were written, but there are several ones which are general purpose. However, these general purpose systems are often parts of much larger interfaces (Glib, Qt), and some of them are somewhat old. Another concern of the author is software freedom, and most (if not all) C++ plugin systems use a license which allows use by proprietary software and doesn’t contribute (or maybe partially contributes) to spreading the message of free software.

Tosaf aims to provide a modern general-purpose flexible C++ plugin framework, that is easy and straight-forward to use, without locking the user to a specific plugin model. The protocol between the program and the plugins is minimal, giving the user full flexibility to build any plugin interfaces they wish, on top of the generic framework.

The documentation doesn’t leave you to handle the domain-specific part of your plugin system alone. Since a significant part is specific to your usage, this manual also explains how to extend the generic framework provided by Tosaf into the specific solution you need. [[TODO|TODO/OPEN]] make sure this is true

License

If you look at Tosaf’s source code, you’ll see its license is GPLv3+. This is a message to both users and developers: Software freedom is an important fundamental right, that every user of computers must have. If you’re a developer, protect that freedom. If you’re a user, insist on having your freedom respected.

Tosaf is provided for the benefit of mankind, and for the advancement and progress of science and technology for all people’s happiness.

If you prefer permissive licenses, please reconsider your priorities. Permissive licenses don’t sent a message of freedom and don’t help spread or protect it. They just allow popularity and the “user base” to be maximized, which has more to do with the selfish feeling of popularity and with potential “profit”, than with values or ethics.

If you’re against copyright and use CC0 or public domain for your code, don’t worry. I’m against copyright as well! The reason that I use GPL and not CC0 is that while CC0 helps making copyright obsolete, it doesn’t help protect software freedom. The severe lack of software freedom is a serious immediate problem which opens the door for companies and governments to spy on people, divide them, make them helpless and unable to help their neighbors. Until this issue becomes less severe, I am using the GPL. Not because of the use of license restrictions and the law, but because it sends a message.

Technically you can’t copy GPL code into CC0 code, but since we’re both freedom fighters, I’m sure we can figure out a solution together! A license specifically for you to use Tosaf under CC0 is an option. Don’t hesitate to ask me.

However, if you write things under permissive licenses like BSD and MIT just because you want to maximize the availability of your code, and you don’t mind or care whether proprietary software is derived from your code - like I already said, please reconsider this, and consider moving to GPL instead. In any case, unless you have a really good reason (like “my software saves life and for reason ____ (fill here) I have to use the BSD license”), please don’t ask me for a license to use Tosaf’s code in permissive-licensed software or to link to it from such software, because I strongly believe in software freedom and will therefore not give such a license.

What are Plugins

A plugin is somewhat like a callback, or a set of callbacks. An application or a library can declare that it can load extension modules and allow them to extend the program functionality in an application specific way. Then third parties can write such plugins, and they will be loaded at runtime. The application doesn’t need to know what they do - the idea is to allow any extensions, even ones not foreseen by the application developer.foresee

For example, a GUI application can allow plugins to add new menu items and dialogs, get user input and perform changes to the open document. Or it can allow plugins to define new export or import formats. The possibilities are endless.

Plugins therefore allow a program or a library to open itself to changes, updates, improvements and extensions. Even by independent teams. New capabilities can be added without changing the core program, and loaded and unloaded dynamically at runtime.

Some applications and system are “plugin based”, which means they contain just a minimal core, a “skeleton”, and all the functionality is written as plugins. For example, an IDE can define a generic framework for IDEs, and let plugins add support for specific programming languages and specific tool support like debugging and memory usage tracking, syntax highlighting, code completion and so on. A text editor can define a minimal UI and a text area, and allow plugins to implement all the editing capabilities, modes, automatic indentation, line wrapping, highlighting, snippet insertion, spell checking and so on.

Plugins in C++

The Compatibility Problem

Plugins in C++ are not trivial but also not difficult. Actually, if you are careful, it’s just like C plugins with a small layer of added support for C++.

Binary (i.e. compiled) software components often need to have a clearly defined interface for communication. This kind of interface describes data structures and protocols on the binary level, and can deal with things like function calling conventions and virtual table structures. It’s called ABI - Application Binary Interface. It’s basically the binary equivalent of an API.

The C language has a formally defined ABI, which means that components compiled by different compilers (or by different versions of the same compiler) can work together. They all speak the same “machine level language”. Therefore, when writing a C plugin, you just need to implement the functions required by the application, compile the plugin into a shared library, install to the plugin folder and you’re done.

C++ is much more platform specific and compiler specific, because it has no ABI. Each compiler defines its own, and has its own way of doing things. It’s even possible for different versions of the same compiler to have incompatible ABIs. In particular, what can be a problem for C++ plugins is name mangling and the virtual table structure.

Name Mangling

Since each C function has a unique name, the symbol name stored in the binary compiled file is the function name itself. You can compile a shared library with one compiler, load it from a program compiled with another, and all the symbols you need will be correctly detected and loaded. In C++, there are many additional features, due to which the name of a function is not unique. For example, functions can be overloaded by parameter list, overloaded by constness, be class members, be static class members, be template instantiations… of course it doesn’t mean it’s impossible to define rules for generating symbol names consistently, but C++ has no such definition and each compiler defines its own scheme. The scheme used to translate source code entities into shared library symbols is called “name mangling”.

Virtual Tables

In C++, objects can contain a virtual function table (vtable) in addition to the data members. Due to the existence of inheritance (and multiple inheritance and virtual inheritance), there needs to be a clearly defined way to pack all the necessary information in the object. If a class has at least one virtual function, all instances will need to have vtable.

The vtable structure is not part of the C++ standard, and each compiler defines its own structure. I read somewhere that the vtable structure is more stable and relying on its stability is less a problem than relying on name mangling. However I’m not an expert. [[TODO|TODO/OPEN]] read about it and say facts

Module Loading

C and C++ don’t define any interfaces for loading of external modules. As a result, each platform has its own interface and its own implementation. POSIX defines a set of functions for this purpose, including dlopen and dlsym, which is supported on GNU/Linux and even provides some extensions. Other platforms, such as MicrooftWindow, provide their own API.

[[TODO|TODO/OPEN]] explain about the need to has a multi platform abstraction and mention Qt and GModule, but first think about the idea to write something like gmodule.

Tosaf’s Solution

Tosaf provides a low-level straight forward wrapper over the bare symbol loading. This wrapper fulfills two roles:

1. Use an OOP approach to plugins
2. Avoid relying on name mangling

In order for the plugin system to be compatible across compilers and platforms, it would need to be based on C calls between the program and the plugin. But if we did that, we’d be limited to what C offers. Of course the C calls could then be wrapped by C++ classes, but you would have to define interfaces in plain C. There are two ways to avoid that:

1. Let the user write C++ but then convert to C using a code generator
2. Allow ourselves to rely on vtable stability, which means we can use polymorphism, i.e. an OOP approach

Tosaf aims to use standard C++. A code generator would cause a lot of trouble: things like templates and exceptions would be impossible to use, the code would be unexpected and difficult to debug and so on. Relying on vtable stability is not a big issue, while allowing us to use an OOP approach. This is what Tosaf does. As long as the plugin and the program are compiled with compilers which use the same vtable structure, we’re fine.

If you call member functions of plugin objects, then you do rely on name mangling stability. But there is no need to do that: Tosaf uses C calls and polymorphism when accessing functions stored in the plugin. C has a standard ABI and polymoprhism relies on vtable stability, so there’s no need to rely on name mangling stability. You’d rely on it if you called plugin C++ functions directly by name, but you can always do that through polymorphism so there’s no need to.

How It Works

In C, the program normally provides a set of functions and each plugin needs to implement them (some of them may be optional, depending on how the program defines it). The OOP equivalent can be something like this: There is a C++ (possibly abstract) base class which declares the functions which plugins can implement. Each plugin is a C++ object of a class derived from that base class, and implements the functions according to its specific purpose. You can see these C++ plugin objects as “singletons”, although that is not strictly required.

This is what Tosaf does. Each plugin stores a C++ object implementing the plugin interface defined by the program.

In order to access this object without relying on name mangling stability, Tosaf uses a C symbol. Each plugin defines a C global function which takes no arguments and returns a pointer to the C++ object. For a plugin whose object implements the interface defined by base class BaseClass, the signature of the C getter function would be:

BaseClass* get ();

Tosaf handles the C part behind the scenes. The program developer just needs to define BaseClass, and then plugin access is through C++ objects. A plugin developer just needs to implement a derived class and tell Tosaf that it’s a plugin (using standard clean C++ without macros or hacks), and Tosaf handles the “singleton” object and the C function that returns a pointer to it.

Terminology

In order to make the API clean and make it easy to explain how things work, Tosaf defines several simple terms. It gives each entity a unique name and avoids confusion. For example, what is a plugin exactly? It is the object instance of the derived class, or is it the compiled shared library? Let’s make these things clear.

• Library: A binary file (e.g. in ELF format on GNU/Linux) whose role is a “shared object” (SO), i.e. it can be loaded as a module at runtime and be used by several independent programs and processes. A library contains function definitions and data objects which can be accessed by a program after loading the library (from the hard drive into main memory). The class tosaf::library represents a library.
• Symbol: A library can contain many functions and objects. But not all of them are necessarily public, e.g. some internal functions are compiled into the library for use by the API functions, but are themselves not exposed in the API. In order to define its public interface to programs using it, a library contains symbols, which are name strings associated with the functions and objects stored in the library. These symbols allow the program to ask for the address of a specific library function, referring to it by its symbol name. By the way, executables (i.e. programs) can contain symbols too, which is useful for debugging. Tosaf uses just a single symbol per library, whose name is get. It refers to the C getter function.
• Interface: A C++ class with virtual member functions, which serves as a base class for plugins. Each plugin derives this class, overriding the virtual functions in its specific way, according to its purpose.
• Plugin: A dynamically loadable module which provides an implementation of a specific interface. The program developer can ask Tosaf for a pointer to the object implementing the interface, get a polymorphic interface pointer is return and call virtual member functions through it. A plugin doesn’t have to be a C++ plugin: For example, it can invoke a Python interpreter and forward the C++ virtual calls to a Python object. Then the plugin developer can write a plugin in Python, if a proper Python wrapper of the interface class is provided. Plugins therefore allow plugin developers to implement the interface in any provided language (if Tosaf and the program support it), while the program’s plugin usage mechanism just sees the polymorphic interface pointer in pure C++. The cross-language support is transparent to it.
• C++ Plugin: A plugin written in C++ (but can internally use other languages, as this is entirely up to the plugin developer) and implements an instanciable class derived from the interface defined by the program. Currently this is the only kind of plugin Tosaf supports. The class tosaf::plugin represents a C++ plugin.
• Hook: Some programs provide a single interface for plugins. In this case, each plugin is just “a plugin”. In this case, the program is said to have a single hook. There’s a single interface and each plugin of any kind just needs to implement it. The interface may give plugins access to the program’s UI elements and its data structures, so that plugins can add new functionality and UI components. However, some programs have more than one hook. For example, a diagram editor application may allow plugins to add new toolbar buttons adding new functionality, or define new export file formats, or define new import file formats, or add new diagram object types. Each such “kind” of plugin has its own interface. A hook is a set of plugins fulfilling the same role. In Tosaf, a hook is responsible for detecting, loading and unloading plugins which fulfill the role associated with it. For example, the diagram editor will have a separate hook for diagram object plugins. The class tosaf::hook represents a hook.
• Role: A kind of plugin offered by the program. Each role should have a hook associated with it. A role is just a name. For example, in the diagram editor the roles could be “export format”, “import format”, “functionality” and “diagram object”. Note that there is no strict one-to-one mapping between roles and interfaces. It’s possible for a program to use exactly the same interface for different roles. For example, an application may have two different screens for different uses, and each allows plugins to add new UI elements. The interface class could be the same, but each plugin would be used for just one of the roles, according to its purpose and where it belongs logically. Just to make it clear: Role is just the name or the concept, while a hook is a programming tool providing access to plugins fulfulling a role.
• Provider: This may refer either to a class derived from the interface class by a plugin, or to an object of such a derived class. Since in Tosaf each such class has a single object stored in the library as some sort of singleton, the class and the object are essentially one. When it does matter, the terms “provider class” and “provider object” will be used. A plugin developer tells Tosaf that a derived class is a provider by making it inherit the class tosaf::provider in addition to inheriting the interface class.

Code Components

Tosaf is quite simple. It should be easy to use. The code should be short, readable and hackable. Here’s an overview of the components, each of which is a C++ class:

• tosaf::library: Allows loading and unloading shared libraries, and resolving their symbols, i.e. getting the address for a given symbol name. This class is used behind the scenes by the C++ plugin implementation. Program and plugin developers normally won’t need to use it directly.
• tosaf::plugin: Allows loading and unloading a plugin, and getting a pointer to the provider object stored in the plugin library. Single plugins can be manipulated using this class, but most of the time they can be accessed using hooks, which should be easier and more convenient.
• tosaf::hook: Provides access to plugins fulfilling a specific role. It can detect plugins (e.g. by reading from the file system), load and unload them. It’s possible to rescan at any time, see what has changed and un/load plugins accordingly. There is also support for automatic detection of changes in the plugin directory using the file monitoring services provided by the operating system kernel, allowing plugins to be loaded to programs simply by installing them. This makes is very easy to make a constantly running server to load a newly installed plugin in realtime without a server restart. [[TODO|TODO/OPEN]] implement this feature!
• tosaf::provider: Unlike the other components, this class is meant for use by plugin developers. A class derived from the interface class should also derive from this class. Then Tosaf causes the library to be built into a plugin, by defining the provider object and the get function behind the scenes.

Coding Time

If you are writing a program and want to give it plugin support:

You need to define the interface class (which is a regular C++ class). Then you can give your main program class a hook member. When the program starts, you ask the hook to load the available plugins. During the program flow, you ask the hook to call a virtual function implemented by the plugins, when appropriate events occur. If the function returns a result, you collect the results returned by the plugins and use them. When the program closes, you can tell the hook to unload the plugins. Actually, you can just make sure your plugins release any resources they allocated and that you hold no references to the provider objects, and then the hook’s destructor will automatically unload the plugins for you.

If you’re writing a plugin:

You need to write the provider class, which is a regular C++ class derived from the interface class. In order to build it as a plugin that Tosaf can load, make the provider class also derive from tosaf::provider. A common pattern in OOP is make the provider class be a factory class which can create and destroy objects of a specific type. For example, a diagram editor plugin could provide a factory that creates diagram objects of a specific kind, e.g. UML components. But Tosaf isn’t limited to any pattern - you can write any provider class you want.

Runtime

Each hook knows where to look for plugins to load. During program runtime, a hook finds and loads plugins, and then any provider functions can be called by the program. For example, when the UI is initialized, the program can ask the plugins to pass it any UI elements they want to add, e.g. a new menu item or a toolbar buttom or a keyboard shortcut.

When a hook isn’t needed anymore (or when the program wants to exit, whichever comes first), it can be told to unload the plugins. Any resources allocated by plugins must be released before that, and references to the provider object or to anything else inside the plugin must not be used after plugin unloading.

Tutorial

This section describes how to use Tosaf in practice. It does so by example. We are going to write a simple exporter plugin for a diagram editor, which can export a diagram into an SVG file. We will skip all the parts related to diagrams and image formats, and focus on the parts relevant to us.

Once you get the basics, Tosaf’s API reference pages will give you all the details of the classes and the functions.

Problem Description

We have a diagram editor program. After editing and preparing a diagram, you usually want to generate an image from it, and then distribute the image or include it in a web page or in a document. There are many image formats and many ways to represent an image digitally. We’d like our program to be able to support any image format, i.e. the only talk is implementing the image generation process.

Since there are many image formats, we’d like other developers to be able to provide new export capabilities to new formats. We’d like to allow those developers to work independently, and be able to add new formats without depending on us, without needing to insert code in the program itself and without asking for permission.

In other words, a developer should be able to implement an exporter into a new image format, and somehow inform the program that this format is not available for export. The program would then offer it to the user, without requiring any coding-time modifications to the program. Format discovery would happen at runtime.

Solution Approach

We’ll use plugins, of course. Our program will define an “exporter” hook which detects exporter plugins. It will then use the available exporters according to the user’s request.

The user needs to be able to do two things:

1. Get a list of export formats to choose from
2. Export a diagram into a given format

Our plugins will therefore need to provide the following functionality:

• return the abbeviated name of the format
• return the full name of the format
• take a diagram and an output stream, generate an image, write to stream and report whether it was successful

In addition, since some formats may require significant processing, we’d like to allow plugins to make some preparations before they are used. For example, a plugin may initialize some kind of cache for use when generating the image. Plugins should also get a chance to return any such resources they own, before being unloaded.

Application Side

First, create an interface class for the exporter hook. Personally, I prefer to make all my virtual functions protected and define public wrappers of them, but for simplicity let’s use public virtual functions.

DiagramExporter.hpp:

class Diagram;

#include <iostream>
#include <string>

class DiagramExporter
{
public:
	virtual std::string get_abbr_format_name () const = 0;
	virtual std::string get_full_format_name () const = 0;
	virtual bool export (const Diagram& diagram, std::ostream& out) = 0;
	virtual void init ();
	virtual void finish ();
};

DiagramExporter.cpp:

#include "DiagramExporter.hpp"

void DiagramExporter::init ()
{
}

void DiagramExporter::finish ()
{
}

In your main program class or in some kind of action manager - depending on your specific code, usage and preference - add a data member that is a Tosaf hook. When initializing the hook, give it the name of the role, “exporter”.

Application.hpp: [[TODO|TODO/OPEN]] see if 2nd include can instead be a forward decl

#include <tosaf.hpp>
#include "DiagramExporter.hpp"

class Application
{
public:
	/* ... */
private:
	/* ... */
	tosaf::hook<DiagramExporter> exporter_hook;
};

Application.cpp:

#include "Application.hpp"

Application::Application (/* ... */)
: /* ... */, exporter_hook ("exporter")
{
	/* ... */
}

Somewhere at the beginning of the application’s run, when you decide it’s time, ask the hook to look for plugins and load the plugins it finds. Also, since we have the init function which is a plugin’s chance to allocate and initialize resources, call the plugins’ init function.

[[TODO|TODO/OPEN]] check if references can be polymorphic just like pointers. If yes, use a redirecting iterator adapter in order to expose references in the iter interface. Otherwise, expose the pointers as is. But then you need a way to prevent the user from setting the pointer, i.e. you can’t have the expression *iter return a reference to the container element, which is a pointer. You’d have to return a pointer by value. Non standard behavior, but somewhat safer.

exporter_hook.load ();
for (DiagramExporter& provider : exporter_hook.providers ())
{
	provider.init ();
}

When the user opens the “Export diagram…” dialog, you want to give the user a list of the available export formats. In order to generate such a list, use the interface functions:

ExportFormatChooserWidget chooser;
for (const DiagramExporter& provider : exporter_hook.providers ())
{
	std::string format_display_name =
	    provider.get_full_format_name () +
	    " (" +
	    provider.get_abbr_format_name () +
	    ")";
	chooser.add_format (format_display_name);
}

When the user chooses a format, we need to apply the corresponding exporter plugin in order to perform the actual export. How the user chooses the format and how the corresponding plugin is matched is application specific, but for simplicity assume we get a string containing the format’s abbreviated name, and then we use a simple linear search in order to find the right plugin.

std::string choice = chooser.get_chosen_format ();
auto iter = std::find_if (exporter_hook.providers ().begin (),
                          exportet_hook.providers ().end (),
                          [&choice](const DiagramExporter& provider)
                          {
                              return provider.get_abbr_format_name () ==
                                     choice;
                          });
if (iter == exporter_hook.providers ().end ()
{
	/* No plugin found! Strange, if we offered the format to the user it
	   should be available here. There is probably a bug in our code. */
}
else
{
	std::ofstream ofs ("diagram-export");
	bool success = iter->export (diagram, ofs);
	if (success)
	{
		/* ... */
	}
	else
	{
		/* ... */
	}
}

When the application is closed, we need to give the plugins a chance to release their resources. Then we can safely unload them. However, manually unloading them in the source code isn’t strictly required: If we don’t do it, the hook’s destructor will simply take care of it for us. But just for completeness of this example, let’s see how it’s done manually. Note that we must manually make sure the plugins have a chance to release resources, because the hook doesn’t understand the purpose of the finish function: for the hook it’s just a regular interface function.

for (DiagramExporter& provider : exporter_hook.providers ())
{
	provider.finish ();
}
exporter_hook.unload ();

[[TODO|TODO/OPEN]] check if shared libraries can automatically have some code invoked at loading and just before unloading. If yes, explain about this option, either here or where the init and finish functions are introduced.

Now, let’s tell the build system to link the diagram editor program to Tosaf. Note that even though all we used from Tosaf is hook, which is a class template, we still need the Tosaf library. The reason is that hook may use (and usually does use) Tosaf classes and functions which aren’t templates.

[[TODO|TODO/OPEN]] explain how to use pkg-config to get ldflags and cflags for Tosaf. Just look how the sigc++ or gtkmm front API ref page says it, and use the same convention.

[[TODO|TODO/OPEN]] explain how to do it with autotools - use the macro which finds dependency libraries and use the resulting defined variables in Makefile.am.

Plugin Side

Useful Links