Mirror of the Rel4tion website/wiki source, view at <http://rel4tion.org>

[[ 🗃 ^yEzqv rel4tion-wiki ]] :: [📥 Inbox] [📤 Outbox] [🐤 Followers] [🤝 Collaborators] [🛠 Commits]

Clone

HTTPS: git clone https://vervis.peers.community/repos/yEzqv

SSH: git clone USERNAME@vervis.peers.community:yEzqv

Branches

Tags

master :: projects / tosaf / old /

work-items.mdwn

Library Backend

This item now has its own new project, [[projects/Sif]].

Provider Setup

I was sure that compiling a plugin could work just by making the provider class inherit tosaf::provider. But it’s not really so simple. I need to find another way. If the only way is to use a macro, I need to update the manual, because it already proudly declares that no macros are used.

Let’s go over what I need to define. Given classes Interface and Derived, the addition Tosaf needs to make is:

namespace
{
	Derived instance;
}

//TODO use extern C
Interface* get ()
{
	return &instance;
}

The problem is, none of this can be done with templates, because get has to be a pure C function from the outside view. Here’s an idea: Keep the C++ provider class. Make it have a private static instance, a private getter and a - wait a second… the idea includes passing types to the C code using a typedef. But if you do that, the C++ class isn’t needed at all. Let’s see how.

As a plugin developer, you define the provider class in a pair of files: an hpp file and a cpp file. Now, we need another file, whose name may be for example provider.cpp. It may look like this:

#include "FileStorage.hpp"

typedef Storage Interface;
typedef FileStorage Provider;

#include <tosaf_setup.hpp>

The last header comes with Tosaf, but isn’t included by tosaf.hpp. It may look like this:

//TODO static assert the inheritance relashionship

namespace
{
	Provider instance;
}

//TODO make this extern C
Interface* get ()
{
	return &instance;
}

Hmmm… I’m not sure I like how this code works. It’s a bit ugly to put the instance in the C-like code. What if we do use Tosaf’s CRTP base class? tosaf::provider will have a static Derived member object and a function which returns a pointer to it as Interface*. It also declares a friend function, void* tosaf_get_plugin_instance (). Now the new content of the tosaf_setup.hpp header is:

//TODO assert inheritance

namespace
{
	Interface* tosaf_get_plugin_instance ()
	{
		return & tosaf::provider<Interface, Derived>::instance;
	}
}

//TODO use extern C
Interface* get ()
{
	return tosaf_get_plugin_instance ();
}

No, wait… does this not cause an instantiation of the tosaf::provider template and make a new fresh copy of the instance? Hmmm… maybe the idea of inheriting this class is really a useless one.

I’d like to introduce a new concept: C templates. These are headers which use typedefs with assumed names as template arguments. In order to instantiate such a template, you need to define the typedefs and #include the template header. It’s exactly what I already did above, but with a new way to explain it to people: It’s a bit like templates, but for C.

Updating code and manual… done.

Multi Path

I’d like to add support for loading plugins from more than one location. But how? Should this support be on the library level, on the plugin leve or on the hook level? Let’s discuss this.

There are basically two ways to use Tosaf (that are expected to be common):

  1. Manage separate plugins with tosaf::plugin
  2. Manage whole hooks with tosaf::hook

Since the first way is not really the intended way, and is more like a workaround to get more fine grained control in special cases, it doesn’t have to support 100% of the features. For example, automatic detection of installation and removal is just a bonus. However, it’s better to move things from tosaf::hook to tosaf::plugin when it makes sense.

The first question I asked myself was whether a plugin should still represent a specific single file, or manage all the possible copies/versions in the various library locations. When a plugin is reloaded, the older library becomes irrelevant anyway (at least for a while), so maybe it doesn’t even matter whether it’s the same plugin object or a new one. It may make a difference if at some point plugin objects will have more settings. Or do they already have some? I’ll check. Also, a plugin as a single specific file is quite a thin boring wrapper around tosaf::library. If support for multi-path can be elegantly added to it, it will be great.

Hmmm indeed, plugins may be thin wrappers but they do have some extra state. Copying it “manually” on the hook level doesn’t sound like the best design. I’d like to try managing location changes on the plugin level.

Now let’s examine the ways hooks are used, to get better understanding of how it will be best to implement multi-path for plugins.

First, when starting the program, the hook examines all the library locations, collects all the plugins it finds and loads them according to location priorities. Then, changes can happen in three ways:

  1. Manually loading and unloading specific plugins or the entire hook
  2. Rescanning the directories, collecting changes and applying them
  3. Listening to updates and responding in real-time

Regardless of which way is used, the intended way to access the plugin functionality through the hook is by invoking a certain function on all the available provider instances. It doesn’t matter for this purpose whether file changes caused new plugin objects to be created, or existing objects simply loaded new library objects. However, it may be a good idea to “distribute the load” and give the plugin class any reasonable responsibilities it can handle, rather than putting the entire logic implementation in the hook class.

I’d like to start with an assumption. It’s a reasonable assumption for now which simplifies things, and reverting it later shouldn’t be too difficult. The assumption is that plugin objects don’t necessarily automatically handle updates. I’m not sure it helps, but we’ll see.

So let’s start by letting a plugin object handle all its copies and versions. A new plugin object is given a name and a role, and its actual path is stored symbolically (in addition to what the library object stores) as an index into the list maintained by tosaf::path, or maybe as an iterator, since the list doesn’t change after locking it (but then add an extra warning to tosaf::path’s documentation, to explain that late writes will quickly cause a segfault). The initial index comes from the hook, which detects actual files and delegates the loading task to plugin objects. Later, when there’s a new file found - assume the hook found it - the plugin gets the location’s index, and after comparison with its current index decides whether it should reload.

Note that automatic reload is generally a very bad idea! The only safe way to do it is to take a callback from the user, that safely released any resources the plugin allocated and removes any references to it! Since automatic reload is not the only option anyway, there should be a way to track changes until they are applied. First, which changes are possible?

  1. A library file is removed (e.g. package removal)
  2. A library file is modified (e.g. package upgrade)
  3. A library file is created (e.g. package installation)
  4. A library file is moved (probably manually)

Since the last option is very rare, it’s okay to treat is as a series of two events: removal and creation. A move event can be added later if it’s critical to anyone. So now we have just removal, modification and creation. Now another reduction: modification to a binary file is basically the same as removal followed by creation, at least as far as Tosaf is concerned. Therefore, we’re actually left with two events: removal and creation.

Should the list of changes stay in the hook, or be moved to the plugins themselves? The weakness of moving is that each plugin object will have an additional pointer and size fields, and memory will be allocated separately even though the use of change lists is going to be very minimal. Plugins change so rarely compared to how much they are used, implementing this change management may hardly be worth the work on it! But the alternative is to let the program developer use a file monitor to find changes automatically or to run rescans manually. Since none of that is specific to any program, Tosaf is probably the right place for it.

Another option in the middle is to allow plugins to rescan and react instantly, without change storage, and have delay support by change storage only in the hook. But how much sense does it make?

Another thing to consider is perhaps the number of plugins. How many plugins does a single program have? If each gets an additional change list, and from time to time few of those lists get 1-2 values, maybe it’s not such a big deal. Maybe the more significant issue with that is how Tosaf registers for file monitoring and how it does rescans. Doing these things once from a central place may be faster than having each plugin do them separately. But wait, let’s examine this one level deeper. A folder scan is simply a nested loop. It doesn’t really matter how many classes are involved. I’d say letting each plugin do its own rescans is fine. The question is whether it also stores any changes it finds, or just runs some callback/signal handlers. On the file monitor side, it depends on how inofity works. Is it more efficient when a single report-all-changes-under-this-folder-tree inofity entry is used?

Another idea which can solve that: Make Tosaf run such a central recursive hook, and have plugins register to that hook inside Tosaf instead of talking directly and separately to inofity! Then, plugins can even support standalone use by plugging them into their own inofity hook instead of Tosaf’s one.

Before a first summary and actual work on this, there’s one decision left to make. On manual rescan of a specific plugin, it may detect - wait a moment. I have a cool idea! What if instead of a vector of changes, which is a dynamically allocated array, we use a constant-size bit array? The size of the array will now be the number of search locations. It’s true that the number of unhandled changes in a row is expected to be just 1-2 even with 10 locations, but 10 bits is still less than a pointer and a size (which on 32-bit systems would be 8 bytes together, i.e. 80 bits! And not to mention is saves a dynamic memory allocation. Who has 80 library search locations??? Maybe merely being constant sized would make it worth the savings, even if a whole bool byte is used per location!

The other option is that each rescan takes a callback, or emits a signal. Then, hooks can cause rescan results to be written directly into their change lists. In general, with large change intervals keeping a list is better than a state array, but the state array is tiny in this case. Hmmm… what if I need more than 1 bit state in the future? What then? Will it still be the best solution?

Wait a moment… who said a change storage is needed? Why not just notify that some change happened, and do a rescan again just before handling it? Hmmm can make a lot of sense. Another thought I have is about the signals: Using signals with objects that can be changed by direct access requires synchronization. It can be an ugly world to start messing with. And the gain is minimal: How many creative things are there to do when there’s a new plugin? It’s just about either ignoring it, or reloading it at some point. I’d say “let’s ignore change storage and just do rescans to start with”, but what do you do after a rescan? Ignore the results?

Let’s see. Since it’s a single plugin, things are simple: The minimal piece of useful info is a single boolean! It just says whether the plugin needs to be unloaded to react to the change. The full info is:

  1. Whether need to unload
  2. Whether need to load again after unload
  3. The new location/version

Maybe an elegant way to return this information is a pair of indices or iterators: The first is the index of the current location, and the second is the index of the new location. Comparing them determines whether an unload is needed, and the second one decides whether a reload is needed. A special value such as max+1 or the vector’s max size could mean “no new location”. Actually, if a plugin can say its current index, it only needs to return a single value! Perhaps a pair - index, and a boolean to say whether the index is real.

I’d like to suggest two common patterns when dealing with plugins. The first pattern can perhaps be called “hot swap”. When a plugin is uninstalled, it’s unloaded in real time. When a plugin is installed, it may be loaded in real time depeding on the program. This approach works well for plugins that don’t provide state, but rather functionality. For example, plugins that add menu items and buttons to GUI applications, and plugins that hook into some program’s triggers to provide integration with various external tools.

The second pattern is the static pattern. Whatever happens after the program starts is irrelevant. Plugins which were loaded stay loaded. New plugins are not detected. Uninstalled plugins are not unloaded. Maybe a manual rescan happens when the user opens the plugin preferenes dialog.

What do we learn from that? Two main scenarios must be supported well:

Note that triggers don’t necessarily respond instantly. Sometimes there is some work to do before unloading a plugin. Should changes be stored at all? It should be possible to do a rescan, and then go over all the changes and decide which ones to react to, and which ones to ignore. Of course it can be done by mapping The plugin rescans into a list of results, but we could also have a “dirty” flag for plugins.

Hmmm on the other hand, what if a change is made and then reverted? Problem! Should plugins be able to clean the “dirty” state in such a case? If yes, they must be smarter than a single flag. Perhaps there’s a smarter way to do this than storing a bit per location, but I’m not sure optimization is necessary. If storing changes, it will basically be at most one change per location. So it’s practically the same as the bit vector, but starting with 0 bits and adding up to 5 bits if needed. I have an idea: When there are predefined locations, their number is usually small. When many locations are possible, it’s usually simply because any location is possible, and plugins can be loaded by arbitrarily choosing files from the file system. Let’s start with a vector per plugin, and with a simple rescan function. An iterator will point to the current location, and the end() iterator will function as the max-value index.

Working on this…

Specific Path

I’m implementing support for loading from a specific path. I’d like to discuss the options here. Assume that at some point, some plugin is loaded from a location specified as a string by the user/program. What does the user mean?

First, meaning of the path:

  1. An additional library path with roles etc. under it (unlikely)
  2. A direct path to load from (likely)

Then the scan/reload policy:

  1. Entirely ignore the library path and use just what was specified
  2. Use the library path only if more recent version is found there
  3. Use the library path only if anything is found there
  4. Use the library path only if the specific path’s file is remove

Actually 2 and 3 are different only if a custom comparison is used, and that is still an optional planned feature. For now, we can summarize the options, as follows. Assume you load a plugin from a special location. Now you can say:

  1. Ignore any other location, use just this
  2. If the library path has anything, use it next time you scan
  3. Use the library path only as fallback

So it basically determines the priority between the special location and the library path. When there is no specific location, 2 or 3 can be used. But for uniformity we could choose one of them. Probably 2 is better.

I’d like to add 2 ideas now to the code:

  1. path_priority: Enumeration for use as discussed just now
  2. provider: A class that holds a library and does nothing except calling the get function inside and returning the instance. I already had thoughts about plugin being a mix of this minimal logic and all the extra convenience of roles, names and the library path. Now these things can be put into separate classes.
[See repo JSON]