Mirror of the Rel4tion website/wiki source, view at <http://rel4tion.org>

Clone

HTTPS: git clone https://vervis.peers.community/repos/yEzqv

SSH: git clone USERNAME@vervis.peers.community:yEzqv

Branches

• master

Tags

afifon.mdwn

Afifon is planned to be a system for sharing software project information between development platform instances. Today, distributed search of projects is completely missing! The best way I know is to run a distributed web search using Yacy. But Yacy is heavy and does much more than needed, and I haven’t examined how it works with semantic tagging. It’s another thing to examine.

Working on short term goals will hopefully help solve the problem of project hosting centralization and proprietary tools (especially g1thu8 which sadly many people use, even software freedom advocates), and in the process give me the knowledge, understanding and experience needed for the long term goal of bringing the same tools to all kinds of software.

“Afifon” ([[!hewiktionary “עפיפון”]]) means kite in Hebrew (the flying toy on a string).

Table of Contents

[[!toc levels=3]]

Data Model

Before the federation part, there must be a regular data model for repo hosting. I’d like to start minimal, because having the full package with wikis and issue tracking and CI etc. makes the federation core work unnecessarily harder.

Therefore, the working assumption is that the system just stores repositories.

There are no groups and no organizations and no projects - just a flat namespace for repos, and a flat namespace for users.

Permissions, for now, will be simple too. Given a user and a repo, the permission system can determine whether the user can push changes to the repo, and whether the user can manage the permissions of the repo. That’s all, just these two booleans. The permissions for a repo are therefore a pair of lists. One list contains users who may make changes to the repo, and the other list contains users who may add and remove users from these lists for this repo.

From the permissions of the instance’s repos, for each user it’s possible to compute and maintain a list of repos she can push to. However, as merge requests are possible too, people can participate in other projects too.

A merge request is an ordered pair of branches from different repos. It means “please merge the first branch into the second branch”. For each repo, a list of these should be maintained.

[[model.dia]]

Federation

Minimal

What is the minimal feature set required for removing all the inconvenience of decentralized repo hosting?

• Shared user accounts
• Repos, projects, tickets, wikis - all support grouping and collaboration between users of different instances, transparently
• Forks and merge requests across instances
• Global search and access to all data: Can reach any person, project, repo, group, wiki, etc. by starting to search or browse from any instance - no need to know the home base instance in advance. HTTP redirection between instances is OK, just make it happen automatically during the user browsing workflow
• Maybe share user, project, repo, etc. name space, so that they’re all unique and can be moved between instances without name collisions. Another option is to assign unique IDs, e.g. using UUIDs, without requiring unique names

User Operations

• vcs operations: as usual from command line vcs program
• (TODO)

Federation Features

• Search for users and repositories worldwide
• Give permissions to users from other instances
• Take merge requests from users from other instances
• Display user and repo links, icons, etc. for remote ones in the same way as for local ones, making the federation transparent. The UI doesn’t differentiate between local and remote objects.

Cases to Study

• Gitolite
• Darcsden
• Gogs
• Kallithea

Implementation Plan

• Storage backend which stores all the meta info (users, permissions, etc.)
• Repo/file for configuration by server admin
• Shell for SSH access, allowing commands according to permissions
• Library which abstracts VCS access - [[/projects/repository]]
• Library which implements all the data manipulations
• Web API which wraps it
• Some kind of UI - either web-based or desktop GUI, not critical right now

User Stories

Storage Sharing

Using:

• Alice has a repository R on instance S (denoted S.R)
• Alice makes a local commit on her computer, using her local copy
• Alice pushes the commit to the remote branch on instance S
• Repository R has two read-only backup mirrors, on instances T and U
• Either immediately after the push, or peiodically, S pushes the changes in S.R to the backup repositories T.R and U.R, which aren’t necessarily visible publicly (maybe through some admin UI or backup mirror UI, not the main UI anyway)

Choosing:

• Alice creates a new repository R on instance S
• S chooses, based on stats and the list of participating instances, two backup instances for R - instances T and U
• Periodically, and/or when T or U have uptime or responsiveness problems, or they announce expected downtime, S may choose new backup instances, so that there are always two of them. When one or both are down, this period of lack of backup is minimized by detecting this and choosing new instances.

Distributed Search

Today ways to find projects and be found are:

• Have your project hosted in a proprietary centralized system like g1thu8
• Have your project hosted in a free software system like gitorious, savannah
• Run your own hosting platform with gitorious, kallithea, gogs, gitolite, etc.
• Use a proprietary user-tracking centralized search engine, like g00gle
• Use a distributed free software search engine, like Yacy

Getting all the proprietary-ness, centralization and greed out of the picture, possible ideas are:

• Make hosting platforms federate, i.e. each one has integrated project search and integration for merge requests, usernames, bugs, wiki pages etc. across instances over the network
• Make distributed web search support semantic search and have instances communicate like social network nodes (e.g. Diaspora* pods), which probably doesn’t need a DHT
• Have a shared vocabulary and API for dev platforms to provide info, use info, declare features and subsystems, etc.

Storage Distribution

Should there be a clear concept of “hosting provider” in the traditional sense? In other words, should storage be completely distributed (i.e. a project doesn’t belong to any specific hosting server), or should each project have a “home base”?

From the user’s point of view, it doesn’t matter much in the technical sense. If a user can see project details using any instance, the actual storage location doesn’t matter. But e.g. unlike with Tahoe LAFS, the information is public, so the LAFS concept and its overhead aren’t needed here. It may not be critical to have a single specific upstream instance visible to the user, but in any case there should be several backup instances for each project.

What about the regular version control system usage? Having no upstream instance means that things like git clone will be slower. The server will have to figure out the physical upstream, clone from it and stream the data to the requesting user client. Instead, if people get the URL of the physical upstream, they can clone it like always.

Problem: What happens if physical upstream is down? How can people still work with the repo? Should they be able to push too, or just pull and clone?

If a local GUI app handles detecting the functioning mirror, then no physical visible upstream is needed in the first place. Pulling only is trivial: Just detect the mirror and pull. Pushing means that once the upstream is down, one of the backups detects this and becomes the new upstream (there’s a shared known protocol for choosing which backup it is). From that point on, users can work with it transparently.

Question: Should I work on all these ideas, or should I focus on a minimal addition to the existing centralized model?

Answer: Start minimal. In particular, it means storage sharing isn’t needed.

Tasks

• Is Yacy good for this? Can it do semantic search? How good is it today at finding independently hosted software projects?
• Understand DHTs, examine existing ones
• Write some basic ontology for project info
• How do GNU Social, Pump.io and Diaspora* instances federate?
• Given semantic search, is DHT still a good option for collaboration features? If yes, it may be better to use it for search too than rely on general-purpose search
• Is a general-purpose quadstore DHT possible? Does it make sense? Maybe it can a meta-store which just says “who knows what” or “who’s online”, and querying can be done using direct connections to the found nodes?
• Maybe it’s much easier and reusable to use an existing DHT based system! For example, GNUnet and I2P are general-purpose (e.g. see how file sharing and instant messaging work with them) and maybe Freenet is relevant too. Also cjdns. Make a list of candidates to examine.

Features and Ideas

• Allow instances to community with other instances, for queries and commands
• DHT for distributed access where needed
• Fork projects from other instances
• Send merge requests to projects in other instances
• Report bugs to projects in other instances
• Use GPG WoT for trust between users and/or between instances
• In the future try to use distributed storage for repos, or at least for backup. For now, just let projects be hosted on specific hosts
• Global semantic project search
• Transparent federation: UI doesn’t make you handle local/remote difference. It gets abstracted, like Diaspora and GNU Social transparently connect you to people from other instances
• Global uniform username space. Maybe avoid using hosts in the name, so moving between hosts is easy.
• Easy 1-click move of projects between instances, without breaking anything

Random

A development platform may consist of many components, each providing a solution for a particular need. Afifon should rely on any specific combination, because the common choice of components is arbitrary. What Afifon can and should rely on, is theory behind the combination of components.

Under application [[/projects]] and under the [[/projects/Kiwi]] ontologies, I have been working on general-purpose models for wikis, issue tracking, discussions and much more. There is still a lot of work to do on these models and on the deployment aspects of solutions based on them. At least for now, Afifon won’t rely on these models, because it may take a lot of time for them to be ready.

The initial Afifon will just use existing common practices. Examples for common components are: version controlled repository, wiki, issue tracker, mailing list, forum, generated manual and API reference.

For the very beginning, only core features will be supported. No integrated issue tracking, no wiki, no discussions. Just the version control and code repository aspects.

[[This|http://tsyesika.co.uk/u/federated-code-issue-hosting.html]] is a plan by Jessica Tallon, which at the time of writing is concerned with the federation messages. Afifon should also support things like DHT, distributed DNS, storage sharing, p2p, routing and more. GNUnet may be a good idea here.

Collaborating on that project may be an awesome idea, especially the information model. Also, the JSON snippets. While [[/projects/Idan]] may be more readable and more writable for humans, it’s much too complicated for the basic simple needs of machine communication. Machines don’t need convenience features like references and generators. So I suppose that with some inspiration from [[!wikipedia JSON-LD]], I could define a mapping between JSON and [[/projects/Smaoin]]. Anyway, we’ll see soon.