RSS Atom

In case you need to add a parser to a C/C++ project, this should be a useful starting point with an empty lexer/parser combination that uses no global variables and accurately tracks location.

You can replace the string "project" in these files with the name of your parser and then rename the files accordingly. Obviously these defaults are not correct for every project.

Lexer Template

%option 8bit
%option never-interactive
%option noyywrap
%option nodefault
%option bison-bridge
%option bison-locations
%option reentrant
%option warn
%option yylineno
%option outfile="project_lex.cc"
%option header-file="project_lex.hh"

%{
#include "project_parse.hh"
#include <stdio.h>

#define YY_USER_ACTION \
    { \
        yylloc->first_line = yylloc->last_line; \
        yylloc->first_column = yylloc->last_column; \
        yylloc->last_line = yylineno; \
        yylloc->last_column = yycolumn; \
        yycolumn += yyleng; \
    }
%}

%%

\n*     yycolumn = 1;
.       fprintf(stderr, "%d:%d:Unhandled character %02x\n", yylineno, yycolumn, (unsigned int)(unsigned char)(yytext[0]));

Newlines need explicit handling in a rule, as the column counter needs to be reset to 1. If there is no return statement in this line, this also means that newlines do not appear as tokens in the token stream, so if these are significant in your language, you will need to adapt this line accordingly. The yylineno variable is silently incremented when newlines are matched, so no special handling is required here.

The catch-all rule at the bottom generates an error message with location data for otherwise unhandled input, then ignores these characters for further parsing.

The YY_USER_ACTION macro updates the standard YYLTYPE as defined by Bison and the internal yycolumn variable. If you define your own YYLTYPE, e.g. because you need to add a file name, this needs to be adjusted as well.

The parser definition needs the lexer declarations (for the token types), so this is included here as well.

Parser Template

%define api.pure full
%define parse.error verbose
%param {yyscan_t scanner}
%parse-param {toplevel &top}
%locations

%output "project_parse.cc"
%defines "project_parse.hh"

%union {
}

%code requires {
#include "project_tree.h"
    typedef void *yyscan_t;
}

%code {
#include "project_lex.hh"

#include <stdio.h>

void yyerror(YYLTYPE *yylval, yyscan_t, toplevel &, char const *msg)
{
    fprintf(stderr, "%d:%d: %s\n", yylval->first_line, yylval->first_column, msg);
}

}

%token end_of_file 0 "end of file"

%%

start:

The order of the include files here is tricky, as the parser definition needs the lexer declarations, which in turn needs the parser declarations.

Also, the user rules in the parser require the syntax tree declarations, which I normally keep in a separate file.

As the pure parser doesn't pass the value of the top production outside of yyparse, the top production needs to copy it somewhere. For this, a separate parameter is added to the parser, a reference to a toplevel object. This is available in all levels of yyparse as well as in yyerror, so an alternative error handling method could store errors in a list inside the toplevel.

The typedef void *yyscan_t; is knowledge we're not supposed to have here, but this definition needs to be available before the declaration of yyparse in the parser header file, which normally doesn't have the lexer definition visible. Including the AST definitions that early makes them available for use in the generated YYSTYPE declaration, so AST types can be used as value types for productions.

Invocation Template

To use the generated parser, you need to open a FILE * stream, attach it to the lexer and invoke the parser (which will call the lexer as required):

toplevel top;

FILE *in = fopen(input, "r");
if(!in)
    /* handle error */ ;
yyscan_t scanner;
yylex_init(&scanner);
yylex_set_in(in, scanner);
int ret = yyparse(scanner, top);
yylex_destroy(scanner);
fclose(in);

The return code from yyparse indicates if the top production was matched successfully, the action in this production should then update the toplevel object.

Makefile Template

To compile the files, just invoke flex and bison with the respective file as a single argument. The output names are listed in the input files, and both outputs are generated at the same time, so make sure dependencies are declared correctly:

project_lex.cc: project_lex.hh
    @

project_lex.hh: project.ll
    flex $<

project_parse.cc: project_parse.hh
    @

project_parse.hh: project.yy
    bison $<

If you generate dummy dependency files somewhere, include the generated headers in them:

%.d:
    @echo >$@ "$*.o: project_lex.hh project_parse.hh"

This ensures that the headers are generated before any source is compiled on the first build — subsequent builds will have accurate dependency information anyway.

So, once again I had a box that had been installed with the kind-of-wrong Debian architecture, in this case, powerpc (32 bit, bigendian), while I wanted ppc64 (64 bit, bigendian). So, crossgrade time.

If you want to follow this, be aware that I use sysvinit. I doubt this can be done this way with systemd installed, because systemd has a lot more dependencies for PID 1, and there is also a dbus daemon involved that cannot be upgraded without a reboot.

To make this a bit more complicated, ppc64 is an unofficial port, so it is even less synchronized across architectures than sid normally is (I would have used jessie, but there is no jessie for ppc64).

Step 1: Be Prepared

To work around the archive synchronisation issues, I installed pbuilder and created 32 and 64 bit base.tgz archives:

pbuilder --create --basetgz /var/cache/pbuilder/powerpc.tgz
pbuilder --create --basetgz /var/cache/pbuilder/ppc64.tgz \
    --architecture ppc64 \
    --mirror http://ftp.ports.debian.org/debian-ports \
    --debootstrapopts --keyring=/usr/share/keyrings/debian-ports-archive-keyring.gpg \
    --debootstrapopts --include=debian-ports-archive-keyring

Step 2: Gradually Heat the Water so the Frog Doesn't Notice

Then, I added the sources to sources.list, and added the architecture to dpkg:

deb [arch=powerpc] http://ftp.debian.org/debian sid main
deb [arch=ppc64] http://ftp.ports.debian.org/debian-ports sid main
deb-src http://ftp.debian.org/debian sid main

dpkg --add-architecture ppc64
apt update

Step 3: Time to Go Wild

apt install dpkg:ppc64

Obviously, that didn't work, in my case because libattr1 and libacl1 weren't in sync, so there was no valid way to install powerpc and ppc64 versions in parallel, so I used pbuilder to compile the current version from sid for the architecture that wasn't up to date (IIRC, one for powerpc, and one for ppc64).

Manually installed the libraries, then tried again:

apt install dpkg:ppc64

Woo, it actually wants to do that. Now, that only half works, because apt calls dpkg twice, once to remove the old version, and once to install the new one. Your options at this point are

apt-get download dpkg:ppc64
dpkg -i dpkg_*_ppc64.deb

or if you didn't think far enough ahead, cursing followed by

cd /tmp
ar x /var/cache/apt/archives/dpkg_*_ppc64.deb
cd /
tar -xJf /tmp/data.tar.xz
dpkg -i /var/cache/apt/archives/dpkg_*_ppc64.deb

Step 4: Automate That

Now, I'd like to get this a bit more convenient, so I had to repeat the same dance with apt and aptitude and their dependencies. Thanks to pbuilder, this wasn't too bad.

With the aptitude resolver, it was then simple to upgrade a test package

aptitude install coreutils:ppc64 coreutils:powerpc-

The resolver did its thing, and asked whether I really wanted to remove an Essential package. I did, and it replaced the package just fine.

So I asked dpkg for a list of all powerpc packages installed (since it's a ppc64 dpkg, it will report powerpc as foreign), massage that into shape with grep and sed, and give the result to aptitude as a command line.

Some time later, aptitude finished, and I had a shiny 64 bit system. Crossgrade through an ssh session that remained open all the time, and without a reboot. After closing the ssh session, the last 32 bit binary was deleted as it was no longer in use.

There were a few minor hiccups during the process where dpkg refused to overwrite "shared" files with different versions, but these could be solved easily by manually installing the offending package with

dpkg --force-overwrite -i ...

and then resuming what aptitude was doing, using

aptitude install

So, in summary, this still works fairly well.

I'm fairly busy at the moment, so I don't really have time to work on free software, and when I do I really want to do something else than sit in front of a computer.

I have declared email bankruptcy at 45,000 unread mails. I still have them, and plan to deal with them in small batches of a few hundred at a time, but in case you sent me something important, it is probably stuck in there. I now practice Inbox Zero, so resending it is a good way to reach me.

For my Debian packages, not much changes. Any package with more than ten users is team maintained anyway. Sponsoring for the packages where I agreed to do so goes on.

For KiCad, I won't get around to much of what I'd planned this year. Fortunately, at this point no one expects me to do anything soon. I still look into the CI system and unclog anything that doesn't clear on its own within a week.

Plans for December:

actually having my own place. While I like the room I'm staying at, it is still fairly expensive because it's paid by the day, and living out of a suitcase without access to my library is kind of annoying after some time.
finishing the paperwork for 2016. Except for some small bits, most of it is in place.
33C3. This time, instead of the "two monitors, three computers" setup, my plan is to have a single laptop only, and have it closed most of the time so the battery lasts the whole day.
See how far I'll get with the controller board for the CNC mill in the Munich Maker Lab. Absolutely no pressure there, it's only the most complex and expensive PCB I ever made.

Plans for January:

Getting settled in.
Back to the Carbon Monoxide detector board that we started in early November. The board is simple enough.
Visiting a demoparty in Finland

Plans for February:

FOSDEM. I plan to hang out in the EDA devroom most of the time, and go to dinner with friends.
Party. Specifically, a housewarming party for whatever flat I'll have then.

Other than that, reading lots of books and meeting other people.

The Arduino IDE does not work properly with tiling window managers, because they do some interesting reparenting.

To solve this, add

_JAVA_AWT_WM_NONREPARENTING=1
export _JAVA_AWT_WM_NONREPARENTING

to the start script or your environment.

Credit: "Joost"

On the other hand,

export EDITOR='sed -ie "/^\+/s/ override//"'
yes e | git add -p

is a ~~good~~ way to commit all your changes except the addition of C++11 override specifiers.

Just starting a small side project...

So, a customer uses a FortiNet VPN gateway. Because I have perfectly fine IPsec software already installed, the only thing missing are appropriate settings. As they use IKEv1 in aggressive mode, there is not much of an error reply if you get any of them wrong.

So, here's a StrongSwan setup that works for me:

conn fortinet
        left=%any
        leftauth=psk
        leftid=""
        leftauth2=xauth
        xauth_identity="your username"
        leftsourceip=%config
        right=gateway IP address
        rightsubnet=VPN subnet
        rightauth=psk
        keyexchange=ikev1
        aggressive=yes
        ike=aes128-sha1-modp1536!
        esp=aes128-sha1-modp1536!
        auto=add

Not sure if that can be optimized further by getting the VPN subnet through mode_config as well, but I'm basically happy with the current state.

In addition to that, you need the PSK and XAUTH secrets, obviously.

So, finally I've also joined the club.

As most distributions have switched to systemd as default "init", some people have asked why we actually keep sysvinit around, as it's old and crusty, and systemd can do so much more.

My answer is: systemd and sysvinit solve entirely different problems, and neither can ever replace the other fully.

The big argument in favour of systemd is integration: when you keep everything in one process, it is easy to make sure that components talk to each other. That is nice, but kind of misses the point: integration could be done via IPC as well, and systemd uses a lot of it to separate privileges anyway.

The tighter integration comes from something else: the configuration format. Where sysvinit uses a lot of shell scripts ("imperative" format), systemd uses explicit keywords ("descriptive" format), and this is the important design decision here: descriptive formats cannot do anything unanticipated, neither good nor bad.

This is important for UI programmers: if there is a button that says "if I close the lid switch, suspend the laptop", then this needs to be connected to functionality that behaves this way, and this connection needs to work in both directions, so the user will find the active setting there when opening the dialog. So, we need to limit ourselves to a closed set of choices, so the UI people can prepare translations.

This is the design tradeoff systemd makes: better integration in exchange for less flexibility.

Of course, it is simple to allow "custom" actions everywhere, but that starts to break the "well-integrated" behaviour. There is an old blog post from Joey Hess about how to implement a simple alarm clock, using systemd, and this shows both how powerful and how limited the system is.

On one hand, the single line WakeSystem=true takes care of the entire dependency chain attached to making sure the system is actually on -- when the system is going to suspend mode, some program needs to determine that there are scheduled jobs that should wake up the system, determined which of these is the next one, program the wakeup time into the hardware and only then allow the suspend to continue.

On the other hand, the descriptive framework already breaks down in this simple example, because the default behaviour of the system is to go to sleep if the lid switch is closed, and the easiest way to fix this is to tell systemd to ignore the lid switch while the program is running. The IPC permissions disallow scheduled user jobs from setting this, so a "custom" action is taken that sets up the "ignore the lid switch" policy, changes to an unprivileged user, and runs the actual job.

So, it is possible to get back flexibility, but this is traded for integration: while the job is running, any functionality that the "suspend laptop when the lid is closed" button had, is broken.

For something simple as a switch on a personal laptop that breaks if the user configures something that is non-standard, that is acceptable. However, the keywords in the service unit files, the timer files, etc. are also part of a descriptive framework, and the services using them expect that the things they promise are kept by the framework, so there is a limit to how many exceptions you can grant.

SystemV init really comes from the other direction here. Nothing is promised to init scripts but a vague notion that they will be run with sufficient privileges. There is no dependency tracking that ensures that certain services will always start before others, just priorities. You can add a dependency system like insserv if you like and make sure that all the requirements for it to work (i.e. services declaring their dependencies) are given, and there is a safe fallback of not parallelizing so we can always play it safe.

Because there is so little promised to scripts, writing them becomes a bit tedious -- all of them have a large case statement where they parse the command line, all of them need to reimplement dropping privileges and finding the PID of a running process. There are lots of helpers for the more common things, like the LSB standard functions for colorful and matching console output, and Debian's start-stop-daemon for privilege and PID file handling, but in the end it remains each script's responsibility.

Systemd will never be able to provide the same flexibility to admins while at the same time keeping the promises made in the descriptive language, and sysvinit will never reach the same amount of easy integration. I think it is fairly safe to predict both of these things, and I'd even go a step further: if any one of them tried, I'd ask the project managers what they were thinking.

The only way to keep systems as complex as these running is to limit the scope of the project.

With systemd, the complexity is kept in the internals, and it is important for manageability that all possible interactions are well-defined. That is a huge issue when writing adventure games, where you need to define interaction rules for each player action in combination with each object, and each pair of objects that can be combined with the "Use" action. Where adventure games would default to "This doesn't seem to work." or "I cannot use this.", this would not be a helpful error message when we're trying to boot, so we really need to cut down on actions and object types here.

Sysvinit, on the other hand, is both more extreme in limiting the scope to very few actions (restart, wait, once) and only one object type (process that can be started and may occasionally terminate) in the main program, and a lot more open in what scripts outside the main program are allowed to do. The prime example for this is that the majority of configuration takes place outside the inittab even -- a script is responsible for the interface that services are registered by dropping init scripts into specific directories, and scripts can also create additional interfaces, but none of this is intrinsic to the init system itself.

Which system is the right choice for your system is just that: your choice. You would choose a starting point, and customize what is necessary for your computer to do what you want it to do.

Here's the thing: most users will be entirely happy with fully uncustomized systemd. It will suspend your laptop if you close the lid, and even give your download manager veto power. I fully support Debian's decision to use systemd as the default init for new installs, because it makes sense to use the default that is good for the vast majority of users.

However, my opening statement still stands: systemd can never support all use cases, because it is constrained by its design. Trying to change that would turn it into an unmaintainable mess, and the authors are right in rejecting this. There are use cases that define the project, and use cases that are out of scope.

This is what we need sysvinit for: the bits that the systemd project managers rightfully reject, but that are someone's use case. Telling these users to get lost would be extremely bad style for the "universal operating system."

For example, if someone needs a service that asks a database server for a list of virtual machines to start, runs each in its private network namespace that is named after replacing part of the virtual machine name with the last two octets of the host machine's IP address, then binds a virtual Ethernet device into the network namespace and sets up a NAT rule that allows devices to talk to the public Internet and a firewall rule to stop VMs from talking to each other.

Such a beast would live outside of systemd's world view. You can easily start it, but systemd would not know how to monitor it (as long as there is some process still running, is that a good sign), not know how to shut down one instance, not know how to map processes to instances and so on. SystemV init has the advantage here that none of these things are defined by the system, and as long as my init script has some code to handle "status", I can easily check up on my processes with the same interface I'd use for everything else.

This requires me to have more knowledge of how the init system works, that is true, however I'd also venture that the learning curve for sysvinit is shallower. If you know shell scripting, you can understand what init scripts do, without memorizing special rules that govern how certain keywords interact, and keeping track of changes to these rules as the feature set is expanded.

That said, I'm looking at becoming the new maintainer of the sysvinit package so we can continue to ship it and give our users a choice, but I'd definitely need help, as I only have a few systems left that won't run properly with systemd because of its nasty habit to keep lots of code in memory where sysvinit would unload it after use (by letting the shell process end), and these don't make good test systems.

The overall plan is to set up a CI system that

tests sysvinit -> systemd transition
tests systemd -> sysvinit transition
tests debootstrap with systemd
tests debootstrap with sysvinit
tests whether service start works for a list of services, so we can catch regressions
scans packages that have systemd unit files for init scripts, and vice versa

Ideally, I'd like to reach a state where we can actually ensure that the default configuration is somewhat similar regardless of which init system we use, and that it is possible to install with either (this is a nasty regression in jessie: bootstrapping a foreign-architecture system broke).

This will take time, and step one is probably looking at bugs first -- I'm grateful for any help I can get here, primarily I'd like to hear about regressions via bug reports.

tl;dr: as a distribution, we need both systemd and sysvinit. If you suggest that systemd can replace sysvinit, I will probably understand that you think that the systemd maintainers don't know project management. I'm reluctantly looking at becoming the official maintainer because I think it is important to keep around.

As I have a few hours of downtime on the train, I'm working on one of my favourite KiCad features again: the pin table.

This silently appeared in the last KiCad release -- when you have a schematic component open in the editor, there is a new icon, currently on the right of the toolbar, that brings up a table of all pins.

Currently, I have the pin number, name and type in the table (as well as the coordinates on the grid, but that's probably not that useful).

For the release, this table is read-only, but can at least be useful for proofreading: left mouse click on a column header sorts (as would be expected), and right mouse click on a column header creates a grouping (i.e. all pins that have the same value in this column are collapsed into a single row).

The latter is entirely unintuitive, but I have no idea how to show the user that this option exists at all.

The good old DirectoryOpus on Amiga had a convention that buttons that had right-click functionality had an earmark to indicate that a "backside" exists, but that is a fairly obscure UI pattern, and would be hard to reproduce across all platforms.

I could make the right click drop down a menu instead of immediately selecting this column for grouping -- at least this way it would be explained what is happening, but advanced users may get annoyed at the extra step.

I could add a status bar with a hint, but that will always cost me screen space, and this only papers over the issue.

I could add a menu with sorting/grouping options, which would allow novice and intermediate users to find the function, but that still does not hint at the shortcut, and the functions to set the sorting would have to be disabled on Linux because wxGTK cannot set the sort column from a program.

For editing, another problem turns up in the collapsed rows: what should editing do when we are basically editing multiple pins at once. The most useful mapping would be:

Editing the pin numbers adds and removes pins. The column is displayed and interpreted as a comma-separated list of pins, so going to edit mode and adding ",5" at the end is an obvious way to quickly create a pin with the number 5.
Editing the pin name would ask for confirmation if not all the pins had the same name before, and then set the same name for all.
Editing the pin type would do the same -- set the same type for all, possibly after confirmation.

However, I'd like to later on introduce multiple-function pins, which would add multiple (partially) exclusive pin names per pin number, which would change the semantics:

As before, editing the pin number adds and removes pins.
Editing the pin name adds and removes functions
Having commas in both the pin name and pin number is legal if e.g. we've collapsed on pin type (so all the outputs are in one line), but editing either of them now will create a broken state.

At the moment, I'm mostly inclined to have separate modes for "grouped by pin number" and "grouped by pin name" -- these modes could be switched by the current grouping column, but that is neither intuitive, nor does it provide a full solution (because grouping by pin type is also supported).

Time to look at the use cases:

I want to quickly enter pins from tables. There are two major cases here:
1. I have pins with the same function, e.g. Vcc or GND. I'd like to enter these as "1,5,17GNDp".
2. I have pins with multiple configurations. I'd like to enter these via "3IO3,DIFFIO4+,CONF_DONEb,b,o".
When I'm done entering data, I want a quick overview whether the data entered is complete.
For existing components, I want to quickly check and correct data.

At 32C3, I've implemented a quick completeness check, consisting of a summary line that collapses the pin number column into a comma-separated list of ranges (where a range is defined as consecutive numbers in the last group of digits).

I wonder if there is a group of developers with an UI focus who are happy about solving problems like these...