Mirror of the Rel4tion website/wiki source, view at <http://rel4tion.org>

Clone

HTTPS: git clone https://vervis.peers.community/repos/yEzqv

SSH: git clone USERNAME@vervis.peers.community:yEzqv

Branches

• master

Tags

diary.mdwn

Skapa: The Source Code Golf-Stick

Meta

Since I’m using the jfm laptop right now, I don’t have access to all my personal material. The documentation of Skapa is there, so I can’t reuse it here at the moment. I’ll start with what I remember.

Concept

Skapa wraps common operations done on free software code repositories and management tools. It abstracts away the specifics and conventions of each project, which makes it much easier and faster to make quick changes or external operations like filing/closing a bug or uploading a PO translation.

It means the maintainers of the software need to tell Skapa once how to handle the user commands according to the rules and tools of their specific project. After that, this specific configuration is transparent to the Skapa user. However, Skapa does not in any way attempt or hope to hide these details: It is important that people know how to use git, how to write source files from scratch and how to use Autotools in C/C++/etc. projects. Skapa just helps make the processes easier, but it is a very good idea to provide instructions for the tools used in the backend so people can learn how that level works, under Skapa’s abstraction layer.

If you already know these layers and participate in a free software project, Skapa is essentially a command-line flexible IDE. But it doesn’t provide a text editor or impose any rules or conventions or depend on any heavy graphical tools. It simply makes repeated tasks easier by doing the repeated parts for you. It is fully configurable and extensible, and you are encouraged to modify, configure and extend it as much as you wish or need.

Design

Skapa is going to be built using a simple extensible design. There is an exensible set of commands, e.g. implemented as Perl modules, and the entry point in a Perl program with a command-line interface which calls the commands. It is responsible for finding them, loading them, verifying authenticity if needed, report errors returned and so on.

Each command is written in a similar manner: There is a core functionality, and any parameters or configuration sets or subcommands or templates can be added dynamically like plugins.

Since Skapa mostly does regex substitutions and wraps other programs, making it a shell script makes sense, maybe even more than using Perl. I will think about it.

Plan

My idea was to port ‘makeclass’ to Perl. But since I can’t access it from here, and since I don’t remember exactly what ‘makeclass’ does, I’ll just define a simple subset of Skapa, a single command, and implement it.

The command I’ll implement is source code file creation. It will not update any Autotools files or other lists, or run any git commands. The command simply creates a new file based on a template. I’m going to try implementing it with several tools:

• Perl
• sed
• awk
• gpp or some other preprocessor
• Maybe some template software

And I will write it and modify gradually, adding features step by step:

1. Run hard-coded substitutions on file from given filename and write result to the same file
2. Write to given filename or to stdout by default (so it can be piped and redirected), same for input file and stdin
3. Have a configuration file (even some plain format is okay for now) which specifies input file(s)
4. Have the substitutions in the config file
5. Make it a YAML file
6. Make a set of prioritized config files, like git uses system config, the global (user) config, then repo-specific
7. Add command-line parameter support
8. Add a main program which finds commands as plugins
9. Add support for loading commands and implementations from both the system and from project’s git repo local clone

Let’s start.

Let’s call the creation command ‘create’ for now. In the end, it may take parameters such as:

• What kind of source to create
• What name to give the new source
• Where to place the new file(s)

Actually that’s all. The name is used both for naming the file and for generating the substitutions. Here’s an example of a typical call:

skapa create class MyClass .

This command goes to the config file to see where the ‘class’ template is, and what it needs to substiture. Then it does the substitution and stores the result in a path relative to the one given. In this case, it’s ‘.’, i.e. the current folder. The ‘class’ template may be configured to create a ‘.c’ file in ./src and an ‘.h’ file in ./include. Instead it makes sense to use the template name to implicitly specify where to put the files, or to use a “source type” instead of a path, where each such type is mapped in the config file to a relative path.

Since many things can be “created”, the name ‘create’ is not exactly descriptive. But for now I think it can work. I’m thus starting a file named ‘create.pl’.

First task: Open a hardcoded-named file, run a simple substitution, print the result and exit.

Done.

Task: Write result to output file. If the filename is "“, write to stdout instead. See what happens if input and output files are the same file. Also read from stdin if input file is”".

Done.

Task: Have a configuration file which specifies input and output filenames. There now may be more than one file - each input file simply gets its own output file. The value “-” means stdin or stdout, depending on the list.

Let’s do this. The config file is going to be taken from 3 places, and each will override the options in the lower priority ones. This is taken from man git-config:

If not set explicitly with --file, there are three files where git
config will search for configuration options:

$GIT_DIR/config
	Repository specific configuration file.

~/.gitconfig
	User-specific configuration file. Also called "global"
	configuration file.

$(prefix)/etc/gitconfig
	System-wide configuration file.

Let’s do something similar. But first, what exactly is $prefix? I have no idea, and I can’t find any material. Maybe where git was installed? Possibly. Anyway let’s ignore it for now and use the following three paths:

1. For now don’t bother with tree-walking, and just use the file “.skapa-local/config” from current directory
2. ~/.skapa-config
3. /etc/skapa.conf
4. The config inside the project source itelf, set by its developers and put in git: .skapa/config

I’m adding functions for reading the config - for now will simply be a list of inputs one per line, then blank line, then list of outputs one per line, which are matched to the inputs by order of appearance.

2014-05-24

I wrote the config reader function. Now write the config getter and make the code use the config. Problems I see now:

1. If one of the config lines was blank my mistake, it may totally ruin the relation between inputs and outputs, because it is based entirely on the order. Solution: Since it’s just temporary practice coding, no big deal. I could switch from list to hash, where each line has an input-output pair separates by space or colon.

2014-06-01

Task done! Config file is in place.

Next task: Have the substitutions in the config file too. Also maybe read a bit about OO/structs, because my data is getting complex and I want some OO of some form.

Also, I’d like to think about the actual config structure I need. Right now, there’s just a mapping from input to output and a single substitution. Instead, I’m going to have “template”. Each template has a name which maps it to its configuration. Therefore, the configuration should contain at least two things:

• auxiliary flat variables (e.g. user’s name and email)
• associative array which maps template names to their configs

What does each template need to specify? It’s a set of configs actually, one for each file. Each such config has a template file, whose base path is determined by the template name. It also has a file path for the generated file, which may contain substituted parts as well. Then, it has the substitution pattern. However, it is not simply a pair of strings. It’s basically a mapping with several strings, which maps strings from the file to parameters received from the user. But there’s more to it: It can take just a single subst_to value, and generate all the others from it. It can work by plugging in a script, e.g. instead of specifying the mapping from subst_to to subst_from, specify a filename of a script which takes parameters (coming from the user) and generates a mapping.

Good. Let’s try again, now bottom-up. There are several key entities:

• parameter sets
• mappings
• template files
• templates

Parameter Sets

A parameter is an entity which appears in meta-template files, and is expected to be replaced with a user-chosen input when generating a template. A parameter is basically just a symbolic name, denoting the meaning of the entity, and not the exact string which appears in the files. It’s very important, because it allows to decouple the meta-template strings and the replacement strings. The meta-template strings can easily be changed without affecting the configuration, because the configuration uses the symbolic names, and these don’t have to be changed because they aren’t pasted in any output.

Example:

In a source file, we may wish to have the class name auto-filled when generating the file. But we have several forms:

• The filename (e.g. myclass)
• The header guard (e.g. MY_CLASS)
• The class declaration (e.g. MyClass)
• A human readable label (e.g. My Class)

But we don’t want to state and promise we will use specific forms for each. For example, right now we may be using class names in the form MyClass, but later we may prefer to use my_class and reflect the change in the meta-template file so that it serves as a better example. So we give each form a symbolic name:

• filename
• header-guard
• declaration
• label

Mappings

There are two uses for mappings:

1. Mapping a parameter set to actual strings used in meta-template files
2. Mapping a set of substitution strings given by the user to a parameter set

Each mapping can be performed in one of two ways:

1. Supplying an associative array which defines the mapping
2. Supplying a script which takes parameters from the user and generates the associative array

Template Files

A template file is usually a static file sitting in a repository, and new files are generated from it by substituting parts of it with user-supplied input. The input doesn’t come directly, however. Each template file is associated with a mapping from a set of strings found in the file to a parameter set. The user input used when generating the files is a mapping from the parameter set to a set of user-supplied strings. Applying these mappings generates the output file.

Templates

A template is a set of transformations, and it has a label string. Each transformation specifies a template file and a set of output files to generate from it. For each output file it is specified whether the mapping from the parameter set to the user strings is supplied directly, or there’s a script which generates it - and a reference to the script is given.

2014-06-02

Task done, the patterns are now inside the config file.

Task: Add models of the components described above to the code, even as an example of a single case. Also try to add some function to manipulate the components and generate templates.