The Enterprise

53 days of GitHub contributions

Michael Pankov •

I wanted to write this post earlier than I, in the end, do. But as it is with many things, it’s better late than never.

This year was fruitful for me—I got a lot of ideas while working on programs, both at my daily job and as a hobby. But what makes the year fruitful are, of course, not the ideas themselves, but their execution. Not only did I manage to release two open-source projects, one of them also gained sufficient traction so that I could call it “notable”.

I decided I want to share the experience of building open-source software. It’s interesting not only as a solitary activity, although programming is often defined as being one.


So, to start a project, one first has to have an idea of what to automate, simplify, or simply implement.


My first area of interest was build systems— I had to re-implement from scratch and then grow and maintain a build system for several interdependent libraries. As one of the libraries was a C run-time library for ARM native tool-chain to use, building was not straight-forward. A lot of thoughts came to my mind as I gardened the build 1. Not all of them were baked enough to introduce implementation of them to build of a project, nearing its production use. Several loosely connected teams used this build, so taking chances near the deadline wasn’t our choice.

So we ended up with a big and messy build system written in Make mostly by one person. That made it nearly impossible to try to simplify anything of it “in-tree”—as even being the only person understanding most of it, I already started to forget the details. Besides, many decisions were made without me, and the project was extended by fellow programmers—in a way that was possible given short deadlines and possibility to understand convoluted contraptions of Make build.

At the same time, I saw a lot of possible improvements, would the build be implemented from scratch, again. Even without changing the technology itself. GNU Make, as old and complex and an-hoc it is, is still a powerful general-purpose dependency tracker, that can also be used to automate deployment (we needed to install stuff to separate devices), testing and even debugging.

So I wanted to try to squeeze every bit of goodness the old program could give. I started to poke around several prototypes.

The name—Qake—came a bit later as something resembling “Make”, but all other “first-letter-replacement” names (such as Bake, Cake, Rake, Fake) were already taken. So I took this, and invented the meaning—I attempted to cause a “quake”, as in “earthquake”—and destroy the Make’s backwards compatibility that was getting in the way all the time.

I’ll tell you soon what came of it, but first I wanted to describe the second project, too.


At some point of learning Rust 2, I decided I want to implement something that would be slightly less contrived than calculating the factorial. I doubt I came up with significantly more useful goal, but I decided to implement a simple arithmetic game, running in the terminal.

Why in terminal? This has to do with enjoying Emacs and text-based interfaces—because they are ubiquitous, easy to implement and remind me of the gone era of early Unix hackers doing away with mere terminal on PDP-11 3.

So here am I—after some two weeks I got a dirty first implementation. For the sake of being a subtle kind of eye candy—in particular, a terminal-based colorized eye candy—the game manipulated colors and other attributes of terminal. Doing this in imperative fashion every time one wants to do formatted output is not very convenient:

    write(term, "I am bold");

So I wanted a tiny library that would allow embedded formatting tokens to be inserted, like this:

    render([Bold, Literal("I am bold"), Reset]);

Asking on Rust’s IRC channel determined there’s possible some similar code in Rust’s compiler. But it would be needed to be extracted to a nice library without external dependencies first. I decided this little act of bike-shedding was useful as a learning exercise, and set out to implement the library myself.

How open source works

Running a bit ahead of train, I have to say that another two weeks later I had working implementation of what I initially wanted. I came to IRC channel and announced it. Nobody was particularly interested—that’s understandable, given vague purpose of the library, but that’s not the point.

For me, this was the fulfillment of a dream. It was a tiny fulfillment—of the size of the project, but nevertheless.

You know, how it’s often being said:

– You need something in an open source project? Why not build it yourself?

And I did exactly that—came and asked, then went away and came back, having what I deemed useful, implemented.

This was for the first time for me. I never participated in others' open source projects. But now I understand how some projects or features of projects get the reputation of not user friendly, barely documented, fragile programs with little practical use.

Someone at some point in time thought it would be useful, implemented it, and contributed. The contribution might have been rejected if the project has strict policy on documentation or tests. Whether it should have been rejected is out of discussion for now. I don’t claim this is precisely the case, but the thought seems plausible.

Enthusiasts work on things they find interesting, and some specialists find obscure complex features infinitely more interesting to implement, than something a user would actually use and find practical. This is an interesting paradox, I believe.

What I learned from Qake

Qake, a GNU Make based build system, attempted to be a lot more practical, than terminal coloring :) I thought that it might be even useful to GNU Make maintainers—to know how their program is being used and what problems do users encounter. It may seem as strange thought, given that Make is nearly 40 years old.

In the end, I posted a link to Hacker News on first release. Interestingly enough, my own post didn’t get up-votes nearly at all. But someone else posted the same link some 12 hours later and it gained around 50 points with several comments, both substantial and not. I got two pull request during the first day, one of them adding OS X support, but these were the only PRs I got.

I also wrote a long email to GNU Make Help mailing list about what would be cool to have in Make so that implementing stuff like Qake would be easier.

No maintainer ever replied to that. I think there’s some degree of cynicism, especially given that I wasn’t a frequent poster to the mailing list. Decades of usage probably already discovered all the points I made—and maintainers just couldn’t do anything in particular about them, because of, mainly, backwards compatibility. There were some backwards incompatible changes between versions 3.81 and 3.82 already, and that cost the 3.82 a lot of adoption. Many distributions continued to use 3.81 for many years and many still do. 4.0 reverted these changes. It seems it’s impossible to challenge de-facto usage of three dozen years old program.

I think that many people already did what I did, because the one reply I got in that mailing list, is from another Make user. He said he did a similar thing at his previous job—and that’s pretty much the same story as with me. I also initially got the urge to implement these things at work, but at work, I had no time to do that. So there are probably hundreds of tiny project-specific advanced build systems, based on Make. The point being, we can never know, because not a single one of them is open source. So we have dozens of ad-hoc solutions precisely because there’s no single central repository to collect code and no community to discuss the implementation.

There probably were many attempts to improve the usage of Make-based build systems, but they mostly happened on proprietary projects.

And we now come to the purpose of GNU itself, as a project and software license. It’s viral exactly for the purpose of sharing every attempt to solve problems, implement some feature, or fix a bug. When forced to share the modified code, implementer will at least make their—maybe unfinished—work the property of community. Community then has a chance to improve it. Or in case nobody goes on to better the unfinished contribution, it’s at least there for people who consider to maybe do something similar. If a failed attempt is open sourced, everyone can study it and discern the reasons of failure. Then they may be able to mitigate these conditions and solve the problem better, next time.


This post is already too long, so I have to wrap up.

Contributing to open source development by starting a project of my own taught me a lot. It was a great experience even though there’s no activity on the project currently.

I now value the viral nature of GNU slightly more—or at least have several more points to consider why it might be a good discipline to develop software.

It also now seems a bit more practical to consider someone’s open source contributions during job application. It shows the attitude of programmer more than particular technical decisions done in the projects.

I think people who never started an open source project or participated in others' ones should absolutely give it a try.

It teaches a programmer a lot.

  1. “Gardening” here refers to programming, in its current state, being similar to gardening more than to engineering, as described in “Pragmatic Programmer: from Journeyman to Master”. 

  2. In case you’re interested in zero-overhead language with focus on system programming, you should absolutely give it a try, too. 

  3. I refer to Thompson, Kernighan and Ritchie—people who invented Unix and C. A very good book on the subject is “The Art of Unix Programming”. It’s not very technical, but explains a lot of thinking, decisions and history instead. 

The Virtual Memory: What and Why

Michael Pankov •

Today we're going to dive a bit deeper than usual. We'll talk about virtual memory — a facility, now omnipresent in most devices considered to be "computers" (1).

With virtual memory, programs operate virtual addresses — rather than physical ones. The virtual addresses are mapped arbitrarily to physical ones on a per-process basis. So, same virtual address in different processes will refer to different areas in memory.
Physical memory operation

The mappings are stored in the form of virtual memory area descriptions in the kernel and page tables in the memory management unit (MMU). Both of these can be thought of as just dictionaries with virtual addresses as keys and physical ones as values. Real implementation is quite involved, though, and differs significantly in software and hardware worlds.

Virtual memory scheme
Let's now define why do we need it. Here are some advantages of virtual addressing comparing to physical addressing:

  1. Simplification of programming.
    User programs become much simpler, because they can assume that they own entire address space. There's no need to manage potential overlaps of memory areas of different programs. This work is done by kernel.
  2. Increase of security and stability.
    With virtual memory, processes can't arbitrarily access each other's memory. That is, they can't corrupt somebody else's data, either intentionally or not. In case some program crashes, it's the only one that goes down. Processes also can't read passwords from memory.
    Oh, and I almost forgot: OS is also in that same address space, so you can't corrupt, crash, or modify it for your inner hacker's delight.
  3. Ability to implement features like swapping, copy-on-write, etc.
    We'll talk about this in later post.
I personally think the main reason back in the days (around 1960s (2)) was the number 1. The second one came just as a consequence, because nobody ever cares about security out of good will (and they probably shouldn't). And the third one is a nice added bonus.

Virtual memory operation
Okay, so far so good, but what are the downsides, so to say, of the virtual memory addressing? Here they are:

  1. Hardware support is required.
    Before CPUs could maintain tables of pages, it was impossible to implement virtual memory.
    Without MMU, we have no way to interrupt intruders in other's address spaces — they all have equivalent physical addresses, which are the same for all processes.
    With MMU, however, we get hardware interrupt in case someone tries to do something not permitted by its virtual mapping. We setup translation tables and switch them for every process — and same virtual address in two different programs may refer to two distinct physical locations. (3)
  2. Determinism is lost.
    As the virtual address space looks all the same across its' entirety, there's no more a guarantee that memory access will always succeed, within a given time frame.
    There are several reasons for that. The main one is that virtual memory address can actually refer not to memory, that is, RAM of the computer, but to hard disk, flash drive, or even a network location. One of the most known examples of virtual memory addresses referring to out-of-memory places is swapping.
    As there can be more virtual memory than physical one, it's possible that a program that consumes no memory in its' address space will run out of resources because some other program consumed physical memory in their address space.
These are the main ones. They both keep virtual memory out of really-tightly-embedded systems, and second is a particularly important disadvantage for real-time systems.

Next post in the series will talk more about implementation details and, probably, about advanced features one can easily implement with virtual memory. Stay tuned!

(1) There are whole classes of exceptions, of course: microcontrollers, digital signal processors, maybe something else. In particular, some configurations of ARM processors come without memory management unit.
(2) Yes, virtual memory was developed in 1960s. The concept itself was introduced by German physicist Fritz-Rudolf Güntsch in 1956. First commercial implementation came out in 1961, but it used segmentation rather than paging. All this stuff wasn't popular on personal computers until Intel 80286 (1982).
(3) In case you're OS guru, please accept my apologies before going to bash me in comments. This is intended oversimplification of the matter for educational purposes.

Building a Personal Database with SQLite and Haskell (part 2)

Michael Pankov •

Last time we've built a nice podcast scheduling tool with Haskell, HDBC, and SQLite. Unfortunately, currently it only supports saving podcasts to the database, but not showing them. Today we're going to fix that.

In case you're interested in the final code — it's available on GitHub.

Firstly, we need to add a command to allow passing a number of podcasts to show:

data Commands = ...
| ShowPodcasts { numberToShow :: Int }
deriving (Typeable, Data, Eq, Show)

What we do is just add another variant with an Int argument to our sum type Commands.

As you remember, Commands is used by dispatchR function to parse the command line.

We also add the description of the new command to the Attributes instance:

instance Attributes Commands where
attributes _ = group "Noname" [
group "Showing the podcasts" [
numberToShow %> [ Help "How much podcasts to display"
, Default (1 :: Int) ]

We use the default value of one podcast to show.

And we then add a description of the mode to RecordCommand instance:

instance RecordCommand Commands where
run' cmd@(ShowPodcasts {}) _ = showPodcasts cmd
mode_summary (ShowPodcasts {}) = "show podcasts"

Our showPodcasts function looks like this:

showPodcasts :: Commands -> IO ()
showPodcasts p = do
let number = numberToShow p
conn <- connectSqlite3 databaseFilePath
rows <- quickQuery conn ("SELECT number, topics, guests, start_ " ++
" FROM podcasts ORDER BY start_ DESC LIMIT ?;")
[toSql number]
mapM_ showPodcast rows
disconnect conn
return ()

Nothing complex. First we pull the number of entities to show from the Commands record p, passed to the function by dispatchR. Then we establish a connection, and do a query with the only parameter — LIMIT, to only show as many podcasts as requested. We also sort the results by start date descending, so that newer podcasts are on top. The showPodcast function (shown below) is then mapped on the results, and the database is disconnected.

Additional challenge: you can try disconnecting before mapping.

showPodcast :: [SqlValue] -> IO ()
showPodcast values = do
let number : topics : guests : start_ : [] = values
putStrLn $ "Episode " ++ show (fromSql number :: Int)
putStrLn $ "Topics:\n " ++ (
concat $ intersperse "\n " (lines $ (fromSql topics :: String)))
putStrLn $ "Guests:\n " ++ (
concat $ intersperse ", " (lines $ (fromSql guests :: String)))
putStrLn $ "Start: " ++ show (fromSql start_ :: UTCTime)
putStrLn ""

Here we decompose a list into four values, then convert them using fromSql and output them to the terminal. The only interesting part is the concat $ intersperse "\n  " — it allows as to have nice indentation of contents relative to the headers.

See example below to know what I mean:

➜  haskpod git:(master) ✗ cabal run -- show-podcasts --number-to-show=2
Preprocessing executable 'haskpod' for haskpod-
[1 of 1] Compiling Main ( Main.hs, dist/build/haskpod/haskpod-tmp/Main.o )
Linking dist/build/haskpod/haskpod ...
Episode 2
A, B, C.Zugge
Start: 2014-04-06 12:14:00 UTC

That's it! We developed a simple command-line utility with a database in under two hours. It's pretty concise (111 lines) and robust, i.e. it reacts to errors adequately:

➜  haskpod git:(master) ✗ cabal run --                                   
Preprocessing executable 'haskpod' for haskpod-
help show help for a command or commands overview
add-podcast add a podcast
show-podcasts show podcasts
FATAL: Command required.

It's also easy to extend, keeping everything working at the same time — thanks to the strong static typing.

As a homework, I suggest adding support of custom ordering when showing podcast. User should be able to choose by which column to sort. This should take no more than 10 lines of code ;)

I think I'll continue posting some code in Haskell. For the next post, however, we're going to dive deep into the operating system internals and start a virtual memory & dynamic libraries detour.

Problems of Reproducibility in Computer Science

Michael Pankov •

There is a word "science" in "Computer Science". But is the real science there in our field of study?

In this post, I will go over the foundations of scientific method and describe what's wrong with it in studies of computers. I'll also try to identify the means of fixing the current, quite sad, state of affairs.

The scientific method, at its' core, is a universal practice, consisting of following steps:
  1. Form a theory.
  2. Predict the events should the theory be true.
  3. Gain experimental data.
  4. Reason about data and correct theory (if necessary).
The steps are very simple — and yet, they lie at the basis of the critical exploration of the world since the beginning of mankind.

Nevertheless, there are many cases when the approach doesn't immediately work. It's frequently because of lack of supporting evidence, or misinterpretation of gathered data. Geocentric model of the Solar System only staggered when the telescope was invented. Luminiferous aether plagued minds of scientists for a long time, thanks to the dual nature of light being hard to explain.

In Computer Science, we collect data on programs being executed on some computer system: time of completion, power consumption, latency of request handling. And the many factors involved in the interesting number being just that number we've got include CPU cache size, memory clock speed, and loop unrolling optimization being enabled in your compiler (among the other hundred). There are, of course, hundreds of other affecting features.

This is real. Two loops being run one after another may yield better performance result than doing the both actions in the same loop body. Or may not, depending on size of the array.

Printing "B" letter to the terminal may be 30 times slower than printing a "#" character. Sorting an array may miraculously make its' processing 6 times faster.

There are countless examples of such counter-intuitive behavior of computers. They are also barely predictable, although explainable post factum. But such an explanation is just anti-scientific — it's known as postdiction, an effect of "knew-it-all-along" thinking.

To carefully perform an experiment in Computer Science, one has to control for many variables, most of which are controlled for almost never. Our caches are state machines (and their state is not observable). Our CPUs can arbitrarily change their frequencies and voltages out of power saving concerns. When a scientific paper is published, there are only vague specifications of systems academics used: i.e., "Used CPU is Intel Xeon E5420" — as if it's everything what matters.

Speaking of scientific papers — there are many and many of them which are barely reproducible at all (1). They often refer to never released source code, and use ad-hoc methods for optimization of programs and measuring effects of the optimizations. Papers being behind the paywalls (2) of big publishers don't help the matter either.

We explain faster fall of a weight compared to a feather by the mass of the former being bigger. (Well, of course we don't, anymore!) That's the equivalent of what we're doing now in computational experimentation. It may even work in the most primitive cases and give seemingly accurate predictions. But once you try to build a catapult or launch a rocket, you need to understand that there's air friction and the more general law of gravitational force. And when you want to understand events on the Universe scale, you have to introduce relativity.

But we're walking in the dark, as the ancient people explaining a thunderbolt by wrath of Zeus.

There are many people who are concerned with current state of affairs. We need a systematic approach to collection and analysis of the data. We need APIs to make experiments reproducible. We should store everything in a universal format, so that later scientists have access to the many experiments we performed.

There are Collective Tuning and Collective Mind (3) initiatives with Grigori Fursin behind them — and I'm glad I participated in one them (although briefly). There's also my humble attempt in the area — Adaptor. It's in no way general enough or complete, but that was an attempt to reuse a lot of existing tools, while Collective Mind built everything from the ground up.

Frameworks such as presented above try to take out as much variables as possible. Nearly all software dependencies are managed by the tool, and the only thing left is OS and hardware. There are also unified statistics and machine learning methods of analysis of the collected experimental data.

Computational science being science again means a lot (4).

And we can help it (5).

(1) There are also critics of those skeptics.
(2) A rather controversial article, but there's a message.
(3) This is an actual working framework for computational data collection & analysis. In case you plan to do something in the area, you absolutely should give it a try!
(4) A presentation in PDF, slides 39-50 are particularly interesting.
(5) A Coursera course on reproducible research.

Building a Personal Database with SQLite and Haskell (part 1)

Michael Pankov •

As some of you may know, I and Alexander Turok record a podcast. It's called "Code Speak Loop" and we discuss programming and related stuff.

To somehow manage the topics, guests and schedule, I decided to develop a small-ish tool to store all the planned podcasts in a sensible way.

To be able to easily sort and query the podcasts directory, we're going to use SQLite embedded database. And since I like Haskell, the program is going to be written in this language.

Second part of the post is available here.

Beware: this post is not very educational! It shows some quickly-written dirty Haskell code, which you may find both ugly and unconventional and hard to understand for a novice (and it is).
In case you don't want to follow-through, there's a GitHub repository with this source code.

I'm going to use GHC 7.6.3 on Ubuntu 13.10 x64, which I built from source myself using this instruction. To install SQLite, I had to run
apt-get install libsqlite3-dev sqliteman sqlite3
That also installs a GUI tool for managing SQLite databases: sqliteman. It's pretty straight-forward.

For GHC interfacing with SQLite, we're going to need HDBC and HDBC-sqlite3. But first, let's create the project and a cabal sandbox.
mkdir haskpod
cd haskpod
cabal init

>Package name? [default: haskpod]
>Package version? [default:]
>Please choose a license:
>Your choice? [default: (none)] 10
>Author name? [default: Michael Pankov]
>Maintainer email? [default:]
>Project homepage URL?
>Project synopsis? A podcast management tool
>Project category:
>Your choice? [default: (none)] 18
>What does the package build:
>Your choice? 2
>What base language is the package written in:
>Your choice? [default: Haskell2010]
>Include documentation on what each field means (y/n)? [default: n] y
>Guessing dependencies...
>Generating LICENSE...
>Warning: unknown license type, you must put a copy in LICENSE yourself.
>Generating Setup.hs...
>Generating haskpod.cabal...
>You may want to edit the .cabal file and add a Description field.
cabal sandbox init
cabal install --only-dependencies
Then let's create the simplest Main.hs
module Main where

main = putStrLn "Hello world"

And then we can build and run the project!
cabal configure
> ...
cabal build
> ...
cabal run
> ...
> Hello world
Now, to the database.

Let's start with main function, adapted from the cmdlib example.
main = getArgs >>= executeR Main {} >>= \opts -> do
let ts = start opts
Just t = (parseTime defaultTimeLocale "%D %R" ts) :: Maybe UTCTime
gs = csv $ guests opts
ps = csv $ topics opts
p = Podcast (episodeNumber opts) gs ps t
d <- doesFileExist databaseFilePath
when (not d) initDatabase
savePodcast p
print p
executeR is cmdlib's way of processing the arguments when using record style of arguments specification. "%D %R" format of time gives us "06/30/14 12:30 PM UTC".  start, guests, topics and episodeNumber are all defined in the cmdlib record type:
data Main = Main { episodeNumber :: Int
, guests :: String
, topics :: String
, start :: String }
deriving (Typeable, Data, Eq, Show)
I tried to use lists of strings for guests and topics, but it proved not worth it: simpler to just split the string we get from parser. The start time suffered the same problem, so I decided to just parse it myself.

The record fields we described are going to be bound to the actual options passed to program by Attributes instance and a RecordCommand instance from cmdlib:
instance Attributes Main where
attributes _ = group "Options" [
episodeNumber %> [ Help "Episode number", Default (1 :: Integer) ],
guests %> Help "Guests of the episode",
topics %> Help "Topics of the episode",
start %> [ Help "Start date and time"
, Default (Data.DateTime.fromGregorian 2014 1 1 0 0 0) ]

instance RecordCommand Main where
mode_summary _ = "Podcast management tool."

The defaults, where not specified, are empty.

Then, we need some extensions for cmdlib to work:
 {-# LANGUAGE FlexibleInstances, MultiParamTypeClasses, DeriveDataTypeable #-}
I didn't delve into them. It seems there's nothing as controversial as Template Haskell, so I just let it be.

I implemented a Podcast record type as well. It is of not much use right now, but it shows the structure of entity we want to store, more accurately.
data Podcast = Podcast { episodeNumber_ :: Int
, guests_ :: [String]
, topics_ :: [String]
, start_ :: UTCTime
deriving (Show)

After all this preparation, we can get to actual database initialization:
initDatabase = do
conn <- connectSqlite3 databaseFilePath
run conn ("CREATE TABLE podcasts" ++
" , number INTEGER NOT NULL " ++
" , topics TEXT NOT NULL " ++
" , guests TEXT NOT NULL " ++
" , start_ INTEGER NOT NULL ) ") []
commit conn
disconnect conn
return ()

databaseFilePath = "podcasts.db"

Notice some interesting things:
  • We have to specify the id as INTEGER PRIMARY KEY ASC to be able to not specify it when doing INSERT later.
  • We have to commit the transaction. There's a nicer to do it: withTransaction, which provides context to wrapped IO action.
  • I used underscore in the end of start, since start was highlighted as a keyword when I tried the query manually. Didn't check it though!
Now we can implement podcast saving:
savePodcast p = do
conn <- connectSqlite3 databaseFilePath
run conn "INSERT INTO podcasts VALUES (NULL, ?, ?, ?, ?)"
[toSql $ episodeNumber_ p, toSql $ unlines $ topics_ p,
toSql $ unlines $ guests_ p, toSql $ start_ p]
commit conn
disconnect conn
return ()
Nothing particularly interesting. toSql uses Convertible instance to convert, pretty much, everything to everything. It even starts to feel like dynamic typing! :)

There's also small helper I used to separate the string of comma-separated guests' names and topics into lists of strings:
csv :: String -> [String]
csv s = case dropWhile isComma s of
"" -> []
s' -> w : csv s''
where (w, s'') =
break isComma s'
where isComma = (== ',')

It's a verbatim copy of words standard function, except for delimiter. Might be quite stupid to copy it like that.

That's all we need to implement saving of our naively defined Podcast records to SQLite. It took me a bit more than an hour, what probably supports the argument that Haskell is a great language for prototyping.

See the next post implementing showing the podcasts we saved. And stay tuned!

Just for reference: haskpod.cabal build-depends section:
  build-depends:       base == 4.6.*
-- we'll need to handle date and time of start of podcast recording
, time == 1.4.*
-- command line arguments handling
, cmdlib == 0.3.*
-- wrappers around time package
, datetime == 0.2.*
-- locale, for time formatting
, old-locale == 1.0.*
-- filesystem querying, i.e. does file exist?
, directory == 1.2.*
-- database interface stuff
, HDBC-sqlite3 == 2.3.*
, HDBC == 2.4.*
And the module header:
module Main where

import Control.Monad
import Data.Time
import Data.DateTime
import Database.HDBC
import Database.HDBC.Sqlite3
import System.Console.CmdLib (Attributes(..), RecordCommand(..), Typeable, Data,
Attribute(..), (%>), executeR, group, getArgs)
import System.Directory
import System.Locale

How we developed our Free-to-Play game

Michael Pankov •

Disclaimer: This post is still personal and I explain a lot about my motives and goals, rather than focusing on the business strategy, if you wish.

Quite a long while ago I developed a game with Oleg Leonov. Now it's almost not active, and I decided to reflect on our enterprise a bit. There are some lessons to be learned from mistakes we made.

The Dream

A product we set out to develop was an Android game in arcade/smasher genre. The idea was proposed by Oleg initially, I think — and for some reason I picked it up and decided it was my chance to shine as a game designer. I'm hardcore player and have some education in the field — some courses and books.

It was not just one of these plain bug smashers — we intended to have a career mode with global map, character development, plenty of specials, and In-App Purchases. That last part worried me a bit, but the hype around Free-to-Play games was so high, the temptation to try was hard to overcome. 

We decided to have a nice cartoon graphic style and comic-like intro to campaign. And the setting was a tiny bit crazy — but we stuck to the idea of heroic mission of one slipper (a "male" one) to save another (a "female" one) from the terrorist cockroaches.

This is never published graphic by Jane, my girlfriend — lo and behold, fans of the game!

We were naive and dreaming of lots of dollars the delighted gamers would put into our pockets. We started to work.

The Plan

Most of the planning was done by me in the form of design document in the beginning of the project. I wrote it sitting on a bench in a park on Bolotnaya square, using some nearby cafe's Wi-Fi — and I remember that moment vividly. The enthusiasm was raging inside.

The plan was to try to make the game in about 3 months, without making it the primary job of any of us. Besides me and Oleg, there were some friends who helped us from time to time — with opinions, playtesting, and ideas. We used BitBucket as the project platform and tracked the issues there. The reason we needed that platform and bureaucracy is simple: I lived in Moscow, and Oleg did in Tula — some two hundred kilometers to the south of the capital.

Oleg was already experienced with Android development, so he took all the technical work. Later he also became an actual stakeholder by investing some money in graphics, which we outsourced to the freelancers. My job was making sure the gameplay was fun and the monetization worked — that's basically game design, playtesting, and balance tuning.

The one thing we both missed already at that time is the actual monetization. We knew what to sell — we introduced a lot of bonuses and slippers skins. We didn't know how to sell. Or rather we didn't want to sell the stuff the way the most successful F2P games do. You know how it works: the game offers you a tiny bit for free, waits until you have a habit of playing, and starts nagging you for bucks. We hated that and decided that people will just pay for optional stuff. You might have guessed what happened.

The Execution

As we both kept working on our day jobs, there was not quite a lot of time for the project. The progress was slow. We underestimated the complexity of the project significantly. Some deadlines were missed. That would probably be a show-stopper for "real" businessmen. But we, amateurs, just continued making this game for fun.

At some point we had to outsource the graphic design. I tried to recruit some of my friends — but, well, none got a lot of interest and motivation to work. That was most probably my bad HR skills — nevertheless, as I mentioned above, we ended up paying for the job.

That part went quite good — the graphics looked nice. I discovered some glitches later, but well, I believe they were not obvious to the casual players. And while we're at it: we never aimed for casual ones. I believed the game should be pretty hardcore. But the fallacy of not wanting to beg users for money already led me to balancing the thing to be hard-but-passable without paying.

That also led to another debatable decision: the character progression was session-wide and randomized. Once you had "game over", you had to start with level one. That was intended to keep players entertained but it's hard to say if it worked.

This was, however, playtested (although mostly by our friends), and seemed to be enjoyable. People were trying to get further in the waves of insects.

All-in-all, we had the game in production for far much time, than we initially planned. It was over a year. The morale started to fall and we had to cut most of the features — namely, global character development and career mode.

At that moment we decided that we have to release what we have, otherwise we risk failing the project completely.

I'll tell you about the results in a moment, but let's side-step to discuss another important aspect: promotion.


We never had a proper site of the game. We've set up a few groups in social networks, and at Indie DB. The traffic was not particularly high, and we were not featured by anyone. I suppose there was not enough traction — most of our friends seemed to be not very interested in the game itself. Maybe the setting was not quite right for the demographic, or genre, or it wasn't ambitious enough — it's hard to tell.

I decided to do a press release as we were nearing the finish of the development — but somehow missed it. I wasn't quite sure in the quality, and thought that we'll advertise more after we fix some problems of the game.

It's worthy to note we also did some alpha and beta releases on forums. There was no particular reaction. People downloaded the completely free non-finished game some thousands times, however. And then it leaked to the China and…

We had around 10000 new users from China in one or two days
That all was before actual release. We weren't able or wishing to control the spread of the game and probably missed a good demographic.

The Release

The day we went online in the Google Play we were downloaded a mere 30 times. The "release tweet" was retweeted just ten or so times. But numbers started to climb. We were in the "New releases" tab of the market for a month — as any new app — and people installed the game and rated it pretty well. For some time we had an average rating of 4.8 (today it's only 4.1).

Anyway, the game entered the top of Arcade & Action, All Games, and even All Apps in at least Russia. We were in eighties of the arcades roster when the free featuring in "New releases" ended.

That would be quite good numbers — we had 2 thousands installs per day for some time — if not for conversion. The monetization was such an utter disaster that we didn't reach the payment ratio of even 0.1% on our nearly fifty-thousand-user base.

The Conclusion

Today the game has just 500 active users. The retention was quite bad, too.

I think we had a success as a free game. As a business, we failed miserably.

For me, that underlined the dangers of F2P model. The developers of such games must always tune them, leaving gamers wanting even if they payed, and quite blatantly use the whales. As a frequent gamer myself, I am now disgusted by blunt techniques of game developing corporations, trying to suck all the money from the player. I often think that our financial results would be much higher should we publish a paid version with all the benefits unlocked — rather than relying on non-mandatory payments.

I never perceived the business' purpose as to maximize profit — and maybe I'm wrong on that. But what I would like to do is making an excellent product — a game, a program, a library — which is loved by users. And I will try again.

Silence is golden

Michael Pankov •

You surely have heard this quote already:
Silence is golden.
It's often used in context of human relations. A slightly less known formulation of the thought is this:
Rule of Silence: Developers should design programs so that they do not print unnecessary output. This rule aims to allow other programs and developers to pick out the information they need from a program's output without having to parse verbosity.
This is the one of the design rules of Unix philosophy.

Let's consider an example of copying a file on a typical *nix system:
# cp source target
The program doesn't output anything. It just silently does its' job and quits. And it does so even if the file is several gigabytes in size and it takes some minutes to copy it.

What's good about it?

Most importantly, this principle emphasizes the value of user's attention: a program shouldn't distract the person launching it with unnecessary messages. If everything goes "as usual", it should not print anything — even progress messages for long-running operations. Then it's easy to spot irregularities in the output. Well, spotting any output is certainly easier, when there's none for successful completion.

For example, when an error occurs, the output appears, making it clear something is out-of-order:
# cp nonexistent target
cp: cannot stat ‘nonexistent’: No such file or directory
This way you don't have to examine the return code of the program to know if it was successful.

The principle also means it's easy to sequence programs together to do a complex task. Let's consider an example of getting sizes of all the files in the directory. (I know there are other ways, which might be considered more optimal. This is just to explain what I'm talking about).
# ls -l | awk '{print $5}'

What would happen if ls would explain what it's doing first, like this:
# ls -l
listing all files in the directory

ls -l | awk '{print $5}'
Notice the "the". That's the thing. Now it's obviously silly to try to pass the sizes to some other program which would like numbers as its' input — "the" spoils the plan.

Arguably the annotated output is easier to understand for non-familiar user — but well, to run the command the user had to know what does -l switch do in the first place.

Now to whose who do violate the rule. For starters, there's GNU Make. Yes, this example output is long. I'm not hiding it on purpose. Go ahead, scroll it.
➜  redis git:(464fef9) make -j8
cd src && make all
make[1]: Entering directory `/home/constantius/redis/src'
rm -rf redis-server redis-sentinel redis-cli redis-benchmark redis-check-dump redis-check-aof *.o *.gcda *.gcno *.gcov lcov-html
(cd ../deps && make distclean)
make[2]: Entering directory `/home/constantius/redis/deps'
(cd hiredis && make clean) > /dev/null || true
(cd linenoise && make clean) > /dev/null || true
(cd lua && make clean) > /dev/null || true
(cd jemalloc && [ -f Makefile ] && make distclean) > /dev/null || true
(rm -f .make-*)
make[2]: Leaving directory `/home/constantius/redis/deps'
(rm -f .make-*)
echo STD=-std=c99 -pedantic >> .make-settings
echo WARN=-Wall >> .make-settings
echo OPT=-O2 >> .make-settings
echo MALLOC=jemalloc >> .make-settings
echo CFLAGS= >> .make-settings
echo LDFLAGS= >> .make-settings
echo REDIS_CFLAGS= >> .make-settings
echo REDIS_LDFLAGS= >> .make-settings
echo PREV_FINAL_CFLAGS=-std=c99 -pedantic -Wall -O2 -g -ggdb   -I../deps/hiredis -I../deps/linenoise -I../deps/lua/src -DUSE_JEMALLOC -I../deps/jemalloc/include >> .make-settings
echo PREV_FINAL_LDFLAGS=  -g -ggdb -rdynamic >> .make-settings
(cd ../deps && make hiredis linenoise lua jemalloc)
make[2]: Entering directory `/home/constantius/redis/deps'
(cd hiredis && make clean) > /dev/null || true
(cd linenoise && make clean) > /dev/null || true
(cd lua && make clean) > /dev/null || true
(cd jemalloc && [ -f Makefile ] && make distclean) > /dev/null || true
(rm -f .make-*)
(echo "" > .make-ldflags)
(echo "" > .make-cflags)
MAKE hiredis
MAKE linenoise
cd hiredis && make static
MAKE lua
cd linenoise && make
cd lua/src && make all CFLAGS="-O2 -Wall -DLUA_ANSI " MYLDFLAGS=""
MAKE jemalloc
cd jemalloc && ./configure --with-jemalloc-prefix=je_ --enable-cc-silence CFLAGS="-std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops " LDFLAGS=""
make[3]: Entering directory `/home/constantius/redis/deps/linenoise'
cc  -Wall -Os -g  -c linenoise.c
make[3]: Entering directory `/home/constantius/redis/deps/lua/src'
cc -O2 -Wall -DLUA_ANSI    -c -o lapi.o lapi.c
cc -O2 -Wall -DLUA_ANSI    -c -o lcode.o lcode.c
make[3]: Entering directory `/home/constantius/redis/deps/hiredis'
cc -std=c99 -pedantic -c -O3 -fPIC  -Wall -W -Wstrict-prototypes -Wwrite-strings -g -ggdb  net.c
cc -std=c99 -pedantic -c -O3 -fPIC  -Wall -W -Wstrict-prototypes -Wwrite-strings -g -ggdb  hiredis.c
cc -O2 -Wall -DLUA_ANSI    -c -o ldebug.o ldebug.c
cc -O2 -Wall -DLUA_ANSI    -c -o ldo.o ldo.c
cc -O2 -Wall -DLUA_ANSI    -c -o ldump.o ldump.c
cc -std=c99 -pedantic -c -O3 -fPIC  -Wall -W -Wstrict-prototypes -Wwrite-strings -g -ggdb  sds.c
cc -O2 -Wall -DLUA_ANSI    -c -o lfunc.o lfunc.c
cc -O2 -Wall -DLUA_ANSI    -c -o lgc.o lgc.c
make[3]: Leaving directory `/home/constantius/redis/deps/linenoise'
cc -O2 -Wall -DLUA_ANSI    -c -o llex.o llex.c
cc -O2 -Wall -DLUA_ANSI    -c -o lmem.o lmem.c
cc -O2 -Wall -DLUA_ANSI    -c -o lobject.o lobject.c
cc -O2 -Wall -DLUA_ANSI    -c -o lopcodes.o lopcodes.c
cc -O2 -Wall -DLUA_ANSI    -c -o lparser.o lparser.c
cc -O2 -Wall -DLUA_ANSI    -c -o lstate.o lstate.c
cc -O2 -Wall -DLUA_ANSI    -c -o lstring.o lstring.c
cc -O2 -Wall -DLUA_ANSI    -c -o ltable.o ltable.c
cc -std=c99 -pedantic -c -O3 -fPIC  -Wall -W -Wstrict-prototypes -Wwrite-strings -g -ggdb  async.c
cc -O2 -Wall -DLUA_ANSI    -c -o ltm.o ltm.c
cc -O2 -Wall -DLUA_ANSI    -c -o lundump.o lundump.c
cc -O2 -Wall -DLUA_ANSI    -c -o lvm.o lvm.c
cc -O2 -Wall -DLUA_ANSI    -c -o lzio.o lzio.c
cc -O2 -Wall -DLUA_ANSI    -c -o strbuf.o strbuf.c
checking for xsltproc... no
checking for gcc... gcc
cc -O2 -Wall -DLUA_ANSI    -c -o lauxlib.o lauxlib.c
cc -O2 -Wall -DLUA_ANSI    -c -o lbaselib.o lbaselib.c
cc -O2 -Wall -DLUA_ANSI    -c -o ldblib.o ldblib.c
checking whether the C compiler works... cc -O2 -Wall -DLUA_ANSI    -c -o liolib.o liolib.c
checking for C compiler default output file name... a.out
checking for suffix of executables... cc -O2 -Wall -DLUA_ANSI    -c -o lmathlib.o lmathlib.c
cc -O2 -Wall -DLUA_ANSI    -c -o loslib.o loslib.c

checking whether we are cross compiling... ar rcs libhiredis.a net.o hiredis.o sds.o async.o
cc -O2 -Wall -DLUA_ANSI    -c -o ltablib.o ltablib.c
cc -O2 -Wall -DLUA_ANSI    -c -o lstrlib.o lstrlib.c
make[3]: Leaving directory `/home/constantius/redis/deps/hiredis'
cc -O2 -Wall -DLUA_ANSI    -c -o loadlib.o loadlib.c
cc -O2 -Wall -DLUA_ANSI    -c -o linit.o linit.c
cc -O2 -Wall -DLUA_ANSI    -c -o lua_cjson.o lua_cjson.c
checking for suffix of object files... cc -O2 -Wall -DLUA_ANSI    -c -o lua_struct.o lua_struct.c
cc -O2 -Wall -DLUA_ANSI    -c -o lua_cmsgpack.o lua_cmsgpack.c
checking whether we are using the GNU C compiler... lua_cmsgpack.c: In function ‘table_is_an_array’:
lua_cmsgpack.c:370:21: warning: variable ‘max’ set but not used [-Wunused-but-set-variable]
     long count = 0, max = 0, idx = 0;
cc -O2 -Wall -DLUA_ANSI    -c -o lua.o lua.c
checking whether gcc accepts -g... cc -O2 -Wall -DLUA_ANSI    -c -o luac.o luac.c
cc -O2 -Wall -DLUA_ANSI    -c -o print.o print.c
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E
ar rcu liblua.a lapi.o lcode.o ldebug.o ldo.o ldump.o lfunc.o lgc.o llex.o lmem.o lobject.o lopcodes.o lparser.o lstate.o lstring.o ltable.o ltm.o lundump.o lvm.o lzio.o strbuf.o lauxlib.o lbaselib.o ldblib.o liolib.o lmathlib.o loslib.o ltablib.o lstrlib.o loadlib.o linit.o lua_cjson.o lua_struct.o lua_cmsgpack.o # DLL needs all object files
ranlib liblua.a
cc -o lua  lua.o liblua.a -lm 
cc -o luac  luac.o print.o liblua.a -lm 
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... liblua.a(loslib.o): In function `os_tmpname':
loslib.c:(.text+0x29c): warning: the use of `tmpnam' is dangerous, better use `mkstemp'
make[3]: Leaving directory `/home/constantius/redis/deps/lua/src'
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking size of void *... 8
checking size of int... 4
checking size of long... 8
checking size of intmax_t... 8
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking whether __asm__ syntax is compilable... yes
checking whether __attribute__ syntax is compilable... yes
checking whether compiler supports -fvisibility=hidden... yes
checking whether compiler supports -Werror... yes
checking whether tls_model attribute is compilable... no
checking for a BSD-compatible install... /usr/bin/install -c
checking for ranlib... ranlib
checking for ar... /usr/bin/ar
checking for ld... /usr/bin/ld
checking for autoconf... /usr/bin/autoconf
checking for memalign... yes
checking for valloc... yes
checking configured backtracing method... N/A
checking for sbrk... yes
checking whether utrace(2) is compilable... no
checking whether valgrind is compilable... no
checking STATIC_PAGE_SHIFT... 12
checking pthread.h usability... yes
checking pthread.h presence... yes
checking for pthread.h... yes
checking for pthread_create in -lpthread... yes
checking for _malloc_thread_cleanup... no
checking for _pthread_mutex_init_calloc_cb... no
checking for TLS... yes
checking whether a program using ffsl is compilable... yes
checking whether atomic(9) is compilable... no
checking whether Darwin OSAtomic*() is compilable... no
checking whether to force 32-bit __sync_{add,sub}_and_fetch()... no
checking whether to force 64-bit __sync_{add,sub}_and_fetch()... no
checking whether Darwin OSSpin*() is compilable... no
checking for stdbool.h that conforms to C99... yes
checking for _Bool... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating doc/html.xsl
config.status: creating doc/manpages.xsl
config.status: creating doc/jemalloc.xml
config.status: creating include/jemalloc/jemalloc.h
config.status: creating include/jemalloc/internal/jemalloc_internal.h
config.status: creating test/jemalloc_test.h
config.status: creating config.stamp
config.status: creating bin/
config.status: creating include/jemalloc/jemalloc_defs.h
config.status: executing include/jemalloc/internal/size_classes.h commands
jemalloc version   : 3.2.0-0-g87499f6748ebe4817571e817e9f680ccb5bf54a9
library revision   : 1

CC                 : gcc
CFLAGS             : -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -fvisibility=hidden
LDFLAGS            : 
LIBS               :  -lm -lpthread
RPATH_EXTRA        : 

XSLTPROC           : 
XSLROOT            : 

PREFIX             : /usr/local
BINDIR             : /usr/local/bin
INCLUDEDIR         : /usr/local/include
LIBDIR             : /usr/local/lib
DATADIR            : /usr/local/share
MANDIR             : /usr/local/share/man

srcroot            : 
abs_srcroot        : /home/constantius/redis/deps/jemalloc/
objroot            : 
abs_objroot        : /home/constantius/redis/deps/jemalloc/

install_suffix     : 
autogen            : 0
experimental       : 1
cc-silence         : 1
debug              : 0
stats              : 1
prof               : 0
prof-libunwind     : 0
prof-libgcc        : 0
prof-gcc           : 0
tcache             : 1
fill               : 1
utrace             : 0
valgrind           : 0
xmalloc            : 0
mremap             : 0
munmap             : 0
dss                : 0
lazy_lock          : 0
tls                : 1
cd jemalloc && make CFLAGS="-std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops " LDFLAGS="" lib/libjemalloc.a
make[3]: Entering directory `/home/constantius/redis/deps/jemalloc'
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/jemalloc.o src/jemalloc.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/arena.o src/arena.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/atomic.o src/atomic.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/base.o src/base.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/bitmap.o src/bitmap.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/chunk.o src/chunk.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/chunk_dss.o src/chunk_dss.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/chunk_mmap.o src/chunk_mmap.c
src/jemalloc.c: In function ‘je_realloc’:
src/jemalloc.c:1082:9: warning: variable ‘old_rzsize’ set but not used [-Wunused-but-set-variable]
  size_t old_rzsize JEMALLOC_CC_SILENCE_INIT(0);
src/jemalloc.c: In function ‘je_free’:
src/jemalloc.c:1230:10: warning: variable ‘rzsize’ set but not used [-Wunused-but-set-variable]
   size_t rzsize JEMALLOC_CC_SILENCE_INIT(0);
src/jemalloc.c: In function ‘je_rallocm’:
src/jemalloc.c:1477:9: warning: variable ‘old_rzsize’ set but not used [-Wunused-but-set-variable]
  size_t old_rzsize JEMALLOC_CC_SILENCE_INIT(0);
src/jemalloc.c: In function ‘je_dallocm’:
src/jemalloc.c:1622:9: warning: variable ‘rzsize’ set but not used [-Wunused-but-set-variable]
  size_t rzsize JEMALLOC_CC_SILENCE_INIT(0);
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/ckh.o src/ckh.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/ctl.o src/ctl.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/extent.o src/extent.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/hash.o src/hash.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/huge.o src/huge.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/mb.o src/mb.c
src/ctl.c: In function ‘epoch_ctl’:
src/ctl.c:1112:11: warning: variable ‘newval’ set but not used [-Wunused-but-set-variable]
  uint64_t newval;
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/mutex.o src/mutex.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/prof.o src/prof.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/quarantine.o src/quarantine.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/rtree.o src/rtree.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/stats.o src/stats.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/tcache.o src/tcache.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/util.o src/util.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/tsd.o src/tsd.c
ar crus lib/libjemalloc.a src/jemalloc.o src/arena.o src/atomic.o src/base.o src/bitmap.o src/chunk.o src/chunk_dss.o src/chunk_mmap.o src/ckh.o src/ctl.o src/extent.o src/hash.o src/huge.o src/mb.o src/mutex.o src/prof.o src/quarantine.o src/rtree.o src/stats.o src/tcache.o src/util.o src/tsd.o
make[3]: Leaving directory `/home/constantius/redis/deps/jemalloc'
make[2]: Leaving directory `/home/constantius/redis/deps'
    CC adlist.o
    CC ae.o
    CC anet.o
    CC dict.o
    CC redis.o
    CC sds.o
    CC zmalloc.o
    CC lzf_c.o
    CC lzf_d.o
    CC pqsort.o
    CC zipmap.o
    CC sha1.o
    CC ziplist.o
    CC release.o
    CC networking.o
    CC util.o
    CC object.o
    CC db.o
    CC replication.o
    CC rdb.o
    CC t_string.o
    CC t_list.o
    CC t_set.o
    CC t_zset.o
    CC t_hash.o
    CC config.o
    CC aof.o
    CC pubsub.o
    CC multi.o
    CC debug.o
    CC sort.o
    CC intset.o
    CC syncio.o
    CC migrate.o
    CC endianconv.o
    CC slowlog.o
    CC scripting.o
    CC bio.o
    CC rio.o
    CC rand.o
    CC memtest.o
    CC crc64.o
    CC bitops.o
    CC sentinel.o
    CC notify.o
    CC setproctitle.o
    CC redis-cli.o
    CC redis-benchmark.o
    CC redis-check-dump.o
    CC redis-check-aof.o
    LINK redis-check-aof
    LINK redis-check-dump
    LINK redis-benchmark
    LINK redis-cli
    LINK redis-server
    INSTALL redis-sentinel

Hint: To run 'make test' is a good idea ;)

make[1]: Leaving directory `/home/constantius/redis/src'

What useful information can you make out of this sheet of text? None, I bet. I wouldn't even read it. By the way, did you notice several warnings from the compiler? No? Well, that's the point.

Having lengthy verbose output makes you ignore most of it. If you're extremely wary most of the time, you can try to force yourself to parse the ever-increasing log for useful information — such as aforementioned compiler warnings. But most developers won't do that. That's irrational thing to do.

That last part of make log is slightly better:
CC adlist.o
It's understandable and gives just the most general information — more like just progress report. Which is still way below the high standard of conciseness set by Unix. Consider this:
➜  redis git:(464fef9) make -j8

src/jemalloc.c: In function ‘je_realloc’:
src/jemalloc.c:1082:9: warning: variable ‘old_rzsize’ set but not used [-Wunused-but-set-variable]
  size_t old_rzsize JEMALLOC_CC_SILENCE_INIT(0);
src/jemalloc.c: In function ‘je_free’:
src/jemalloc.c:1230:10: warning: variable ‘rzsize’ set but not used [-Wunused-but-set-variable]
   size_t rzsize JEMALLOC_CC_SILENCE_INIT(0);
src/jemalloc.c: In function ‘je_rallocm’:
src/jemalloc.c:1477:9: warning: variable ‘old_rzsize’ set but not used [-Wunused-but-set-variable]
  size_t old_rzsize JEMALLOC_CC_SILENCE_INIT(0);
src/jemalloc.c: In function ‘je_dallocm’:
src/jemalloc.c:1622:9: warning: variable ‘rzsize’ set but not used [-Wunused-but-set-variable]
  size_t rzsize JEMALLOC_CC_SILENCE_INIT(0);
src/ctl.c: In function ‘epoch_ctl’:
src/ctl.c:1112:11: warning: variable ‘newval’ set but not used [-Wunused-but-set-variable]
  uint64_t newval;

The warnings are right there now. Up front and clear.

What does it cost? Just suppress all the output to stdout. Or, if you're writing your own tool: don't output anything except your immediate result to standard output until asked explicitly.

To be honest, I myself am guilty of verbose output of build system. I've implemented CC file type of output and that's still too much even for our not-so-big project. I guess it's easy to get attached to feeling of importance given by lots of output of your system.

Okay, let's not stop on my favorite kind of software, which is build systems :)

Recently I've come across a test, which output something along these lines:
case 12: passed
case 13: skipped
case 14: passed
You know what's "skipped" here? It doesn't mean the test can't run on current configuration or was disabled manually. Here it means the code for this particular case was removed to distinct testing executable and therefore the case is "reserved for future use". Could you guess it?

What I'm getting at is this: designing proper information, error, and debugging messages is hard. To be both helpful and short, the output should be carefully crafted. And until this is the case, the program should be silent — to not mislead anyone.

It's better to have a proper error code of the tool and no error message besides vague "Error", than elaborate screenful of explanation what went wrong. Having the error code one can look it up in manual or in the source. But having some complex (or even worse — outdated) explanation of error may induce a misunderstanding and false sense of trust to help message. Seeing some message developer won't even consider investigating the error further — the program already tells what's wrong! Only it does it the wrong way.

We're coming to generalization of the rule: one shouldn't supply the client with information until client wants to have it. When the user wants to know, he's already primed to be thinking in right context. When the program just spits all the information it has, user is tempted to blindly trust it.

The reasoning applies not only to command-line tools. Unnecessary modal message boxes, verbose dialogues, and blinking buttons are all species of the same kind.

So, just to summarize and repeat it once again: the principle of silence is really just another expression of law of encapsulation.

Black boxes are silent. And silence is golden.

Fix the cause

Michael Pankov •

How do you usually react when you discover a small bug in your code? Do you fix it right away? Or do you stop and try to find why it was introduced in the first place?

Consider a simple scenario: developer tries to build the project and at some point he discovers that not all files are rebuilt. He has two options. First is filing a bug against the build system maintainer (or at least putting it onto a post-it note). And the second is using the fact that the bug doesn't happen when you clean the project build directory first.

If one goes with the latter option, it's worth putting the `make clean; make` sequence to the script. The script that may accidentally get committed to repository under the name ''. And someone else may start using this inherently broken solution.

It will take more time to build the program — but that's "lazy time". One doesn't really do anything while waiting for compilation to finish. At least not when the build time is at a scale of several seconds or minutes. So — it's very easy to work around the problem. Moreover, this solution provides false motivation to implement it — it offers a free chance to procrastinate a bit more on the internet.

Of course, that's pretty obvious to any engineer, that the "right" way to handle this is to fix the bug in the build system. The problem is we are often forced to act in "not right" way. Not because of our laziness, but because the business requires quick solution, not the right one. Amount of "right" is hard to qualify, and amount of time spent is easy.

That actually got me thinking: what if we could unite the "right" and "quick" ways to fix the problem? It probably means going with one of two options, again. Either make the "quick" way the sole option, or only provide the "right" way.

What I'm talking about is kind of "Python vs. Haskell" decision. With first, you have "mostly sane" default behavior, and engineering is pretty straightforward. Straightforward in the sense it's designed to be easy to make progress. With second, you are forced to solve small tactical problems at the moment they're introduced — i.e. make the types correct. It may be hard to make progress at times, but the built program is more robust because you're not allowed to take shortcuts. You stop to think what you're doing.

While I understand that's probably not the best possible analogy, I think the reader can grasp what I mean: one approach emphasizes the importance of good design, and the other strives to achieve the goal of program implementation no matter what. One is trying to make a good program, and the other is trying to solve the problem the program is intended to solve.

Perception of implementation in a programming language differs significantly in these two approaches. For "quick" the implementation itself is only a tool to reach a non-technical goal. A business goal, so to say. And for "right" one, the right design of program is recognized as a separate and not less important goal, than the business one.

The focus on the business problem sometimes goes as far as discarding the importance of tools altogether. People taking this point of view claim that technologies — programming languages, editors, databases, etc. — are mere appliances. Like a mixer in your kitchen. Except studying to properly operate your high-tech bytes mixer — programming language — is orders of magnitude harder and longer process.

We wandered off quite far from the main topic of discussion. But I believe the examples help to understand what I mean. When something breaks, it's an opportunity to re-consider whether you're on the right path at all. Maybe the stuff doesn't work because you're trying to implement some awful workarounds and there's actually a better way.

Occurrence of some bug is also a chance to think about prevention of such bug in the future. You discovered a goto fail? Introduce an automatic check of your code with static analyzer. Your colleague found a memory leak? Urge him to start not with fixing the immediate problem, but to add a valgrind run to the commit test suite. It's in spirit of TDD: first you introduce a non-passing test, and only then make it pass.

Behaving in such a way solves not only current issues, but prevents many of them in future. And the same bugs occur over and over again more times than any individual developer notices. That's why it pays off to do the right thing at the time it becomes needed, not postponing it "until next time". It's easy to become overwhelmed with the recurring problems because you weren't able to implement a proper prevention measure when you first encountered a particular bug.

Don't fix the consequences of an issue. Fix the cause.

The use of prototyping

Michael Pankov •

I recently watched a remarkably fun and motivating talk on software engineering, engineering in general, and history of humankind space exploration. It's called "To the moon" and you should absolutely watch it, if you're interested in at least one of the three topics.

The thing that I found particularly interesting is that when doing a project, especially a big one, you should start with simplest thing possible (the link is to the specific point in the talk).

The crux is this: one should start with something simple to the point of being silly, and the harder the project, the sillier your first task should be.

Why is that? Because we often overestimate our abilities. We underestimate unknown obstacles. The larger the project, the more costly it will be to overcome the obstacles later — when we've already built something based on the assumption, which may not hold. We don't know, because we didn't test it.

So test it. Test your every assumption in quickest way available, and throw the prototype out of the window.

The "throwing out of the window part" is also (if not more) important, than the first one. Because once we aim for speed, we don't aim for code quality. And for prototype, we shouldn't aim for quality — it's only needed to check if it's a good idea in principle. The final design will be a compromise of various factors (including the time limits and required quality of code), and it should be done differently than the initial attempt at understanding the problem domain.

This is the idea behind "hello world" programs. A newly coming developer may assume all the infrastructure will work, and start writing some actual program. Just to discover that they miss required libraries on development machine, that their editor messes up the source file encoding, or that the tutorial one reads is based on newer version of the language, than they have.

These are all petty problems. But they add up, and in case you're testing some actual code that does the job, it will have it's own, not-so-petty mistakes. They will overwhelm and demotivate you. Don't let that happen.

When starting to learn a new programming language, write a "hello world".

When starting a Moon program, launch a rocket and crash it on the surface of the Earth's satellite.

When starting a new task, test every thing that you think you're sure in, first.

The Traps of Obviousness

Michael Pankov •

Note: This text was initially intended to be a question on, but I decided there's no "hard answer" and the question may easily be considered off-topic. So I'm putting a more developed version of it here instead.

There's an often attitude in software engineering: calling big, complex things obvious.

The problem is present not only in software engineering, it probably just happens more often there because of "high density of knowledge" in programming. The gap between knowing a single detail and not knowing it is huge due to many abstraction levels, which one has to deal with. Because of that, "full-stack" engineers are highly valued because they have understanding of all the layers of the computing system designed to do the job.

In any case, for the purpose of this discussion, let's not concern ourselves with other fields, and talk just about programming.

So, there's a common phenomenon: a person who is familiar and experienced with a particular technology (1) has a discussion with less familiar developer (2) and when (2) finds some disconnection between their understanding and actual workings of this technology, (1) exclaims: "It's obvious!". Of course (2) doesn't think it's obvious because it took some effort trying to understand the particular thing, which (2) may initially consider a quirk.
Situation I'm talking about is along these lines, although I'm intentionally exaggerating (but only a bit):
1: So, what's the problem? 
2: I'm trying to create a 2 billion-sized array of integers on stack in C, can't wrap my head why it doesn't work. (Doesn't know about limited stack size and probably doesn't understand concept of process in OS).
1: But it's obvious! (Does know the implications (2) doesn't know about).
In some extreme cases (1) may go as far as criticizing (2) for their lack of knowledge or even attack (2) personally.

Now, I think (1) behaving in such a way is wrong for several reasons, among which:

In any case, 

But there's also another sort of obviousness, illustrated by the following example:
1: Let's implement our custom test case description format! 
2: How should the long lines should be handled in this format? (Knows about C string literals, which can be concatenated, but also can be continued via `\`, and also remembers about Markdown, which continues every "string" by default (actually, a paragraph) and uses empty lines for line breaks.)
1: It's obvious that it should be done via `\`!
Here (1) behaves if not worse, then at least no better, than in previous case. They demonstrate a significant amount of ignorance by assuming there are no other options. They also imply that their solution is the best and is not on topic for discussion.

This is, essentially, the authoritative supervisor's way, leading to poor motivation of executor and misunderstanding. Apart from that, it results in bad design due to lack of attention to other ways to solve the problem.

I think every self-respecting programmer will agree, that stating the obviousness of the architecture, software design, or particular way to solve a tactical problem, as implemented or intended, doesn't help anyone.

This word — obvious — can only really be used as a pun, a joke, hinting to a really poorly designed system.

Footnote: It was probably obvious to someone, that -j option of tar means bzip2 archive format, -J means xz and -z means gzip.

Break things when something serious happens

Michael Pankov •

As I wrote in my last post speaking about breaking backward compatibility, errors are better than warnings. I'm going to elaborate on what I mean and why is this so.


Let's start with an example. At some point it is decided that the project build should include a separate stage managed by a different configuration file and performing custom commands, like packaging the built executables and external resources into distribution archive. This archive is composed from files given their paths and corresponding names in the archive. Paths could be relative to the root of project — for the files which are built in this project. Or they could be absolute — for the files which are not the part of this project.

A developer implements a tool which creates this archive given the configuration file. He has a choice: when the tool can't find the file specified in the configuration (i.e. because it doesn't exist), it could just issue a warning to terminal and continue, or stop with error. Issuing a warning seems like a good option, and the programmer chooses it. Let's think what consequences will this decision have.


The tool is implemented and is integrated to the build. Well, "integrated" may be a strong word. Let's remember that configuration files are not stored in version control system directly. Rather, an original, sample, version of such file (think .config.sample) is put into git, and then the developer should create the working version of that file (i.e. .config) — in many cases by copying the original. So the integration actually stops on adding the logic to the build system and putting the sample configuration file under VCS.

Then, depending on the level of consciousness and experience of the implementer, the build system may or may not detect the absence of the working copy of configuration file. That's not the main point. Either way, the developer which just pulled the VCS and got an updated build finally figured out they need a local .config to supplement the .config.sample. What's the simplest way to make a local copy of VCS-tracked file? Just copying! Bam, we're in trouble.

The .config, as we intended, contains paths — both relative and absolute. Do you see what may be wrong? The .config.sample may, by mistake (or just as an example!), contain some absolute paths from the original implementer's machine, and they will be copied to the .config. Even if not, the .config will contain the paths to the files which don't exist yet — because the executables we want to be packed are not built.

Then, for some reason, the files needed by tool can be not built — the new build stage wasn't not thoroughly tested yet, after all. Or it's okay in "vanilla" situation, but then you started hacking on your task and broke the build. Some file to be packaged was removed and is not built now.

But the packager then proceeds to process the .config, tries to pack the non-existing files, but doesn't fail — rather just emits several lines of "warning:" lines to the build log, which spans over several screens, even when build commands are not output.

So, it's impossible to notice that something went wrong. If some of the needed files do exist, and some don't — the resulting package will get created, but will be incomplete. And you'll get to know that only after the package is deployed — either by fellow developer, or the DevOps team, or by customer themselves.

Then, depending on quality of error handling in the rest of the system, the team will spend anywhere in a range of from an hour to the couple of days debugging what's going wrong. In the end, the package was meant to include all the required files! And no errors were encountered during the build, so what could be broken?

It all could have been avoided, should the initial implementer of the packaging tool opt-in for errors instead of warnings. Just emit a "file doesn't exist" as soon as a file mentioned in configuration is not found.

Okay, you may think that the scenario I described is inconceivable. That it doesn't happen the way it's shown here. That people are not that stupid and that there not so many mistakes in build systems.

Well, it does happen. Depending on overall responsibility of programmers, size of the team, and pace of development it may happen more, or less, often, but it still happens. In my experience — the actual errors are mistakenly masked by someone (including me) unforgivably frequently. Which brings us to the conclusion…

Warnings are ignored most of the time

Because warnings are just that — non-blocking hints on what could cause some trouble. I'll elaborate on this matter in another post later, but for now — admit it, nobody is ever going to perceive a warning as something which can cause a complete failure of operation of your software.

No manager is going to understand why the package is broken if the build is successful. No DevOps will stop deployment of a new version because it compiles with warnings. Especially when deadlines are pressing and the supervisor is not satisfied.

It's not that hard to steer clear of all that trouble: don't emit warnings which should have been errors. Do not contribute to the confusion.

Backward compatibility

Michael Pankov •

I often come across a software engineering practice of retaining as much backward compatibility as possible. This strikes me as a poor decision, leading to a lot of problems, which I'll try to describe in this post.

First, let's introduce some context. I work as a system programmer, the team is mostly (but not totally) located in one office and is around 2 dozens of people. There's local issue tracker, but not a lot of formalized planning — to the point that the tasks are often only set informally in a talk between a supervisor and an implementor, so features in development are often invisible for others. We also do code-review, but it's essentially "commit-wise". The project is moving out of research stage into production.

So, what sort of backward compatibility am I talking about? It's the one when a developer implements a replacement feature because of change of requirements. The existing feature is clearly going to be deprecated. And the developer hesitates to deprecate it and wants to leave the old feature in place as well. They don't make an atomic change of replacement of old feature with the new one — they rather add one more way to do things.

What does it mean in the long term?

That the old feature may be not deprecated at all ever — because of priority shift, lack of time, reorganization of team. The initial implementor may get sick, or go to a vacation, and forget about removing the old feature on return. Even if it's in the issue tracker — because issue tracker doesn't bother you. You may have tens and hundreds of tasks there and still not care. The old feature may get finally removed, say, in a year — a great time-span, in which it can cause a lot of confusion and trouble.

And by confusion I mean the following. If the feature isn't deprecated — that means essentially removed, at the moment a replacement is ready — you have to support both options. Someone may build their work off the old feature — because they may not know about replacement at all. It gets even worse with growth of team and geographical distribution of developers — considering little formalized planning, there's no way to know that the old feature is meant to be deprecated, when both options are present.

Moreover, it's much more likely that the developers know how to work with old feature and will be lazy to move their code to use the new one. To them, it will look like the old feature is going to be here for a while and they may rely on it. And this will bite them at the very last moment — because the new feature was implemented exactly to mitigate some problems, about which this particular developer, who wants to use the old feature, may be not aware of. Not knowing about these problems, a developer sticks with the compatibility — and right before shipment it turns out this old, compatible feature stopped working. It was known to the implementor of replacement feature that it will stop working at some time, that's why the replacement feature was implemented. But the user of the feature didn't know it. And they might not use the old feature, if it weren't there — because they use what's there. And out of general laziness and common sense — why care, if a feature exists and apparently does it's job? No comments or documentation are going to stop a developer — because reading them is optional. The way the feature should be used may be apparent just from the other code that also uses the old feature. And the developer will just copy and paste. Because the old feature was there, and the "example code" using it also was there. Only the compiler spitting out errors may stop them.

Don't get me wrong — I don't consider every developer a stupid ignorant schoolboy with sky-high ambitions. But occasionally all of us are stupid, or ignorant, or thinking "oh it's all clear from this code, I won't go ask the guy who implemented this", or tired of 12-hour drag because of burning deadline, or stressed because of pressure from the customers. And we, programmers, should protect programmers, including (maybe most importantly) ourselves, from making mistakes. This thought isn't new at all (see e.g. Pragmatic Programmer; it's great read, by the way). But it's worth repeating.

Apart from all the issues described above, the subject practice also violates one good principle, which may sound like Python's "there's one right way to do stuff", or like "don't create unneeded entities", with the latter probably known to reader as principle of Occam's razor. Being lazy to replace the feature, removing the old one, leads to code bloat, increasing the compilation time and executable size — because there are two ways of doing stuff.

It's also interesting to consider this love of backward compatibility, taken to the extreme — what will happen, if no features will ever get removed? I'll leave this thought experiment to the reader.

Don't retain the backward compatibility. Make your changes atomically, replace the old features, and make other people learn about them. It's better to let them know sooner rather than later.

It's also better to let them know through a non-buildable project, than a compilation warning or a email notification. But, this is a topic of another post, which I hope I'll publish soon.

Why sending patches via email is bad

Michael Pankov •

This week, I've received emails with source code changes thrice from different people. And at first I tried to persuade the colleague to commit changes to git instead, and failed. When I got changes by email third time this week, I decided to write a blog post about it.


We work on same project and all have access to same git repository. There's no policy prohibiting arbitrary branch creation on server, but I generally remove old branches as they get merged. There's also no formalized changes review.
People who sent me changes did it in three forms:
  1. Just plain text source code in body of the email with comments like "I changed this place in that file to the following"
  2. Source code files, attached to the email message
  3. Patches, attached to the email message
I won't discuss the first two forms — they obviously indicate a decent amount of ignorance of good software engineering practices, but let's stop on the third one.

Why sending patches via email is bad

So, to send me changes, these people who sent me patches, did the following:
  1. Ran git format-patch in the working copy, with the revision interval specified. This is already unnecessary complicated — I'd rather not specify any revision interval. It also works only with established history, and I have no ways to, say, selectively share changes. In case of multiple commits it gets even more complicated on the receiving side — but let's talk about it later.
  2. Navigated to the folder where the patches are created.
  3. Created an email message, dragged-n-dropped patch files there.
  4. Sent the message.
And what I have to do to apply these changes:
  1. Open the email message, choose "Save as..." for each patch.
  2. Navigate to the location of the project to save patches there.
  3. Go to the terminal, use git apply for each patch, and hope the changes will apply successfully. At this point, I just have no information on from which branch were the patches made, so in case I'm working on a pretty divergent branch I'll have conflicts all over the place.
Let's compare this to a normal branching workflow. On the sending side:
  1. git checkout -b <issue name>
  2. git commit -a
  3. git push origin <issue_name>
On the receiving one:
  1. git fetch
  2. git checkout <issue_name>
I think it's pretty clear what is simpler and more precise.

Why using branches is good

Using distinct branch allows to review changes relative to upstream before merging, and still makes it clear that this is an independent unit of work.

Making the changes the independent unit of work matters, because you never know if it's only one commit. Maybe you broke something you didn't notice, maybe there's a cleaner solution, anyway — the receiving end will possibly build something on top of your changes, not just use it right from the shelf.

Please use branches, don't send patches.