Planet GStreamer

Break the kettles and sink the boats

2012-05-19T18:22:32+00:00

This is it. The “ges” branch of Pitivi has been merged to “master”.

The changes that have been merged are quite significant: over four hundred commits, 175 files changed, 13943 insertions and 30341 deletions (net result: more than 16 thousand lines of code removed while at the same time adding features and solving longstanding bugs).

While “master” is certainly far from bug-free, it is now a decent base to build upon. Perfect is the enemy of the good. Let us now move ahead resolutely and without fear to achieve victory: 0.16 will be a superb release and I am thrilled by what’s going to happen this summer with the GSoC projects.

See this page for build instructions, including a script that automates the process.

By the way, if you happen to be around, I will be presenting in Hong Kong at GNOME.Asia next month.

Notes on Apple IR remotes reverse-engineering

2012-05-18T17:47:05+00:00

In 2009, I wrote a driver to make the infra-red remote on my original MacBookAir work out-of-the-box on Linux. The driver was rejected upstream on the basis that the device would soon be supported through more generic means. In the meanwhile, it lived in Fedora's kernel tree, and I took some notes about implementing pairing, so that only your remote would work with your computer.

I'm posting this now because I wanted to poke at a MacOS X application today, and couldn't for the life of me remember the name of the program to monitor disk-activity. Hope this finds its way to a search engine near you.

Launched the System Preferences, Security, and unlock the panel.
In a terminal: sudo fs_usage -f filesys -w and check the output when enabling/disabling the remote.
We can see the modified file is /Library/Preferences/com.apple.driver.AppleIRController.plist
Installed PlistEditPro and opened the file up.
Now try to pair a remote (menu and next together)
You can see the UID value changing in the file. I named the remotes I had available to me:

New remote: UID = 145
Old clean remote: UID = 24
Old dirty remote: UID = 227

After adding some debug to the aforementioned appleir driver, in Linux, I got:

New remote: appleir: received (5 bytes) 25 87 e0 91 02
Old clean remote: appleir: received (5 bytes) 25 87 e0 18 03
Old dirty remote: appleir: received (5 bytes) 25 87 e0 e3 02

So the 4th byte is the remote's UID.

Now one could implement remote pairing using a sysfs attribute, a udev helper to apply the pairing across reboots, and PolicyKit helper to set and save the paired UID.

This will be left as an exercise to the reader :)

Those Funny Funguloids

2012-05-16T12:23:37+00:00

Overview

Recently I’ve spent a little time getting the Ogre3D engine into a state where it’ll work on the Nokia N9 and N950 MeeGo phones.

To test the port out a bit more extensively I decided it’d be a good idea to try porting an existing game, at which point rzr suggested Those Funny Funguloids. So after a fair amount of hacking to make it compatible with my version of Ogre and to make the controls work reasonably on a touch screen device I have a playable version.

The aim of the game is to collect mushrooms (…IN SPACE), and bring them back to your base without running into any asteroids. Only mushrooms of the same colour as your base are added to your score and your base changes colour each level (but you can hang on to mushrooms between levels and score them later). The controls are fairly simple, touch the left side of the screen to turn left, the right side to turn right and the middle to start moving.

When I have a bit more time I’ll see about writing a little tutorial on making use of my modified version of Ogre3D to create 3D applications and games in a way that’s easy to distribute.

Screenshots

stranger in these parts

2012-05-16T11:35:25+00:00

My JSConf 2012 video is out! Check it out:

Visit blip.tv to watch in your browser

The talk is called "Stranger in these parts: A hired gun in the JS corral", and in it I talk about my experiences as a Schemer in the implementation world, with a focus on JavaScriptCore, the JS implementation of the WebKit project.

If you want, you can fetch the slides or the notes. If you were unable to play the video in the browser, you can download it directly (25 minutes, ~80 MB, CC-BY-SA).

Special thanks to the A/V team for the fine recording. My talk was the first one that used the wireless mics, and it turned out there was some intermittent interference. They corrected this in later talks by double-miking the speakers. In my case, it was fortunate that they recorded the room as well, and with (I would imagine) a fair amount of post-processing the sound is perfectly intelligible. Cheers!

Finally, there were a number of other interesting talks whose recordings are starting to come out. I especially liked David Nolen's funhouse of ClojureScript and logic programming. It was pleasantly strange to see him mention Peter Norvig's 1992 book, Paradigms of Artificial Intelligence Programming, because I did too, and I think someone else did as well. Three people mentioning a somewhat obscure 20-year-old book, what are the odds?

I also liked Vyacheslav's amusing talk on V8's optimizing compiler. He actually showed some assembler! Folks that read this webrag might find it interesting. Dan Ingalls' talk was fun too. The ending scene was pretty surreal; be sure to watch all the way through.

Thanks again to the JSConf organizers for the invitation to speak. It was a pleasure to get to hang around the lively and energetic JS community. Happy hacking, all!

doing it wrong: cse in guile

2012-05-14T17:07:14+00:00

Greetings, readers! It's been a little while, but not because I haven't been doing anything, or nothing worth writing about. No, the reason I haven't written recently is because the perceived range of my ignorance has been growing faster than the domain of my expertise.

Knowledge may well be something one can dominate, but ignorance must forever be a range, stretching off to a hazy horizon. Climbing the hill of JavaScript implementations has let me see farther out on that plain. I think it's only the existence of the unknown unknowns that can let one muster up the arrogance to write at all.

But back to domains and dominators later. To begin, I note that there is very little in the way of correct, current, and universal folk wisdom as to how to implement a programming language. Compiler hackers are priests of their languages, yes, but their temples are in reality more or less isolated cults, in which the values of one's Gods may be unknown or abhorrent to those of others. Witness the attention paid to loop optimizations in some languages, compared to garbage collection in others, or closures in still others.

In my ecumenical capacity as abbot of Guile and adjunct deacon of JavaScriptCore, sometimes I'm tempted to mix rites: to sprinkle the holy water of lexical scope optimizations on JS, and, apropos of this article, to exorcise common subexpressions in Scheme.

As one might well imagine, the rites of one cult must be adapted to circumstances. I implemented CSE for Guile, but I don't know if it was actually a win. In this article I'll go into what CSE is, how it works in Guile, why it probably won't survive in its present form.

cse: common subexpression elimination

I implemented a source-to-source optimization pass in Guile that eliminates common subexpressions. It actually does both more and less than that: it propagates predicates and eliminates effect-free statements as well, and these latter optimizations are why I implemented the pass in the first place.

Let me give an example. Let's imagine we implement a binary tree in Guile, using the records facility.

(use-modules (srfi srfi-9))

(define-record-type btree
  (make-btree elt left right)
  btree?
  (elt btree-elt)
  (left btree-left)
  (right btree-right))

(define *btree-null* #f)

(define (btree-cons head tail)
  (if (btree? tail)
      (let ((elt (btree-elt tail)))
        (if (< elt head)
            (make-btree elt
                        (btree-left tail)
                        (btree-cons head (btree-right tail)))
            (make-btree elt
                        (btree-cons head (btree-left tail))
                        (btree-right tail))))
      (make-btree head
                  *btree-null*
                  *btree-null*)))

That's enough to illustrate my point, I think. We have the data type, the base case, and a constructor. Of course in Haskell or something with better data types it would be much cleaner, but hey, let's roll with this.

If you look at btree-cons, it doesn't seem to be amenable in its current form to classic common subexpression elimination. People don't tend to write duplicated code. You see that I bound the temporary elt instead of typing (btree-elt btree) each time, and that was partly because of typing, and partly out of some inner efficiency puritan, telling me I shouldn't write duplicate expressions. (Cult behavior, again!)

But, note that all of these record abstractions will probably inline, instead of calling out to procedures. (They are implemented as inlinable procedures. Yes, it would be better to have cross-module inlining, but we don't, so this is what we do.) In general, syntactic abstraction in Scheme can lead to duplicate code. Apologies in advance for this eyeball-rending torrent, but here's a listing of what btree-cons reduces to, after expansion and partial evaluation:

(define (btree-cons head tail)
  (if (and (struct? tail)
           (eq? (struct-vtable tail) btree))
      (let ((elt (if (eq? (struct-vtable tail) btree)
                     (struct-ref tail 0)
                     (throw 'wrong-type-arg
                            'btree-elt
                            "Wrong type argument: ~S"
                            (list tail)
                            (list tail)))))
        (if (< elt head)
            (let ((left (if (eq? (struct-vtable tail) btree)
                            (struct-ref tail 1)
                            (throw 'wrong-type-arg
                                   'btree-left
                                   "Wrong type argument: ~S"
                                   (list tail)
                                   (list tail))))
                  (right (btree-cons
                           head
                           (if (eq? (struct-vtable tail) btree)
                               (struct-ref tail 2)
                               (throw 'wrong-type-arg
                                      'btree-right
                                      "Wrong type argument: ~S"
                                      (list tail)
                                      (list tail))))))
              (make-struct/no-tail btree elt left right))
            (let ((left (btree-cons
                          head
                          (if (eq? (struct-vtable tail) btree)
                              (struct-ref tail 1)
                              (throw 'wrong-type-arg
                                     'btree-left
                                     "Wrong type argument: ~S"
                                     (list tail)
                                     (list tail)))))
                  (right (if (eq? (struct-vtable tail) btree)
                             (struct-ref tail 2)
                             (throw 'wrong-type-arg
                                    'btree-right
                                    "Wrong type argument: ~S"
                                    (list tail)
                                    (list tail)))))
              (make-struct/no-tail btree elt left right))))
      (let ((left *btree-null*) (right *btree-null*))
        (make-struct/no-tail btree head left right))))

Again, I'm really sorry about that, and it's not just for your eyes: it's also because that's a crapload of code for what should be a simple operation. It's also redundant! There are 6 checks that btree is in fact a btree, when only one is needed semantically. (Note that the null case is not a btree, of course.)

Furthermore, all of the checks in the first arm of the if are redundant. The code above is what the optimizer produces -- which is, you know, turrible.

So, I thought, we could run a pass over the source that tries to propagate predicates, and then tries to fold predicates whose boolean value we already know.

And that's what I did. Here's what Guile's optimizer does with the function, including the CSE pass:

(define (btree-cons head tail)
  (if (and (struct? tail)
           (eq? (struct-vtable tail) btree))
      (let ((elt (struct-ref tail 0)))
        (if (< elt head)
            (let ((left (struct-ref tail 1))
                  (right (btree-cons head (struct-ref tail 2))))
              (make-struct/no-tail btree elt left right))
            (let ((left (btree-cons head (struct-ref tail 1)))
                  (right (struct-ref tail 2)))
              (make-struct/no-tail btree elt left right))))
      (let ((left *btree-null*) (right *btree-null*))
        (make-struct/no-tail btree head left right))))

This is much better. It's quite close to the source program, except the symbolic references like btree-elt have been replaced with indexed references. The type check in the predicate of the if expression propagated to all the other type checks, causing those nested if expressions to fold.

Of course, CSE can also propagate bound lexicals:

(let ((y (car x)))
  (car x))
=> (let ((y (car x)))
     y)

This is the classic definition of CSE.

but is it a win?

I should be quite pleased with the results, except that CSE makes Guile's compiler approximately twice as slow. Granted, in the long run, this should be acceptable: code is usually run many more times than it is compiled. But this is a fairly expensive pass, and yet at the same time it's not as good as it could be.

In order to get to the heart of the matter, I need to explain a little about the implementation. CSE is a post-pass, that runs after partial evaluation (peval). I tried to make it a part of peval, as the two optimizations are synergistic -- oh yes, let's revel in that word -- are you feeling it? -- but it was too complicated in the end. The reason is that in functions like this:

(define (explode btree)
  (unless (btree? btree)
    (error "not a btree" btree))
  (values (btree-head btree)
          (btree-left btree)
          (btree-right btree)))

Here we have a sequence of two expressions. Since the first one bails out if the predicate is false, we should propagate a true predicate past the first expression. This means that running CSE on an expression returns two values: the rewritten expression, given the predicates already seen; and a new set of predicates that the expression asserts. We should be able to use these new assertions to elide the type checks in the second expression. And indeed, Guile can do this.

Perhaps you can see the size of the problem now. CSE is a pass that runs over all code, building up an ordered set of expressions that were evaluated, and in what context. When it sees a new expression in a test context -- as the predicate in an if -- it checks to see if the set contains that expression (or its negation) already, in test context, and if so tries to fold the expression to true or false. Already doing this set lookup and traversal is expensive -- at least N log N in the total size of the program, with some quadratic components in the case an expression is found, and also with high constant factors due to the need for custom hash and equality predicates.

The quadratic factor comes in when walking the set to see if the elimination is valid. Consider:

(if (car x)
    (if (car x) 10 20)
    30)

Here, we should be able to eliminate the second (car x), folding to (if (car x) 10 30). However, in this one:

(if (car x)
    (begin
      (y)
      (if (car x) 10 20))
    30)

If we don't know what (y) does, then we can't fold the second test, because perhaps (y) will change the contents of the pair, x. The information that allows us to make these decisions is effects analysis. For the purposes of Guile's optimizer, (car x) has two dependencies and can cause two effects: it depends on the contents of a mutable value, and on the value of a toplevel (x), and can cause the effect of an unbound variable error when resolving the toplevel, or a type error when accessing its car. Two expressions commute if neither depends on effects that the other causes.

I stole the idea of doing a coarse effects analysis, and representing it as bits in a small integer, from V8. Guile's version is here: effects.scm. The ordered set is a form of global value numbering. See the CSE pass here: cse.scm.

The commute test is fairly cheap, but the set traversal is currently a bit expensive.

and for what?

As I have indicated, the pass does do something useful on real programs, as in the binary tree example. But it does not do all it could, and it's difficult to fix that, for a few reasons.

Unlike traditional CSE, Guile's version of it is interprocedural. Instead of operating on just one basic block or one function, it operates across nested functions as well. However, only some dependencies can commute across a function boundary. For example:

(lambda (x)
  (if (pair? x)
      (let ((y (car x)))
        (lambda ()
          (and (pair? x) (car x))))))

Can the first pair? test propagate to the second expression? It can, because pair? does not depend on the values of mutable data, or indeed on any effect. If it's true once, it will always be true.

But can we replace the second (car x) with y? No, because (car x) has a dependency on mutable data, and because we don't do escape analysis on the closure, we don't let those dependencies commute across a procedure boundary. (In this case, even if we did escape analysis, we'd have the same conclusion.)

However, not all lambda abstractions are closures. Some of them might end up being compiled to labels in the function. Scheme uses syntactically recursive procedures to implement loops, after all. But Guile's CSE does poorly for lambda expressions that are actually labels. The reason is that lexical scope is not a dominator tree.

MLton hacker Stephen Weeks says it better than I do:

Thinking of it another way, both CPS and SSA require that variable definitions dominate uses. The difference is that using CPS as an IL requires that all transformations provide a proof of dominance in the form of the nesting, while SSA doesn't. Now, if a CPS transformation doesn't do too much rewriting, then the partial dominance information that it had from the input tree is sufficient for the output tree. Hence tree splicing works fine. However, sometimes it is not sufficient.
As a concrete example, consider common-subexpression elimination. Suppose we have a common subexpression x = e that dominates an expression y = e in a function. In CPS, if y = e happens to be within the scope of x = e, then we are fine and can rewrite it to y = x. If however, y = e is not within the scope of x, then either we have to do massive tree rewriting (essentially making the syntax tree closer to the dominator tree) or skip the optimization. Another way out is to simply use the syntax tree as an approximation to the dominator tree for common-subexpression elimination, but then you miss some optimization opportunities. On the other hand, with SSA, you simply compute the dominator tree, and can always replace y = e with y = x, without having to worry about providing a proof in the output that x dominates y. (i.e. without putting y in the scope of x)
[MLton-devel] CPS vs SSA

See my article on SSA and CPS for more context.

So that's one large source of lost CSE opportunities, especially in loops.

Another large source of lost opportunities is that the Tree-IL language, which is basically a macro-expanded Scheme, has the same property that Scheme does, that the order of evaluation of operands is unspecified.

Consider the base-case clause of my btree-cons above:

(let ((left *btree-null*) (right *btree-null*))
  (make-struct/no-tail btree head left right))

Now, *btree-null* is a toplevel lookup, that might have an unbound-variable effect. We should be able to eliminate one of them, though. Why doesn't the CSE pass do it? Because in Tree-IL, the order of evaluation of the right-hand-sides of left and right is unspecified. This gives Guile some useful freedoms, but little information for CSE.

This is an instance of a more general problem, that Tree-IL might be too high-level to be useful for CSE. For example, at runtime, not all lexical references are the same -- some are local, and some reference free variables. For mutated free variables, the variable is itself in a box, so to reference it you would load the box into a local and then dereference the box. CSE should allow you to eliminate duplicate loads of the box, even in the case that it can't eliminate duplicate references into the box.

conclusion

It is nice to be able to eliminate the duplicate checks, but not at any price. Currently the bootstrapping time cost is a bit irritating. I have other ideas on how to fix that, but ultimately we probably need to re-implement CSE at some lower level. More on that in a future post. Until then, happy hacking.

GStreamer Core 0.11.91, Base Plugins 0.11.91, Good Plugins 0.11.91, Bad Plugins 0.11.91, Ugly Plugins 0.11.91, libav Plugins 0.11.91 unstable release

2012-05-13T15:17:31+00:00

The GStreamer team announces a new release of the GStreamer core, Base/Good/Bad/Ugly/libav modules for the 0.11 GStreamer unstable release series.

This is the second release candidate of the upcoming 1.0 release. It is intended for developers and people wanting to port their plugins and applications to the new series. Only minor or absolutely necessary changes to the core/base API/ABI will happen between this release and the final 1.0.0 release.

Check out release notes for gstreamer core or gst-plugins-base, gst-plugins-good, gst-plugins-bad, gst-plugins-ugly, gst-libav, or download tarballs for gstreamer or gst-plugins-base, gst-plugins-good, gst-plugins-bad, gst-plugins-ugly, gst-libav,

PulseAudio 2.0: Twice The Goodness!

2012-05-12T10:50:34+00:00

That’s right, it’s finally out! Thanks go out to all our contributors for the great work (there’s too many — see the shortlog!). The highlights of the release follow. Head over to the announcement or release notes for more details.

Dynamic sample rate switching by Pierre-Louis Bossart: This makes PulseAudio even more power efficient.
Jack detection by David Henningsson: Separate volumes for your laptop speakers and headphones, and more stuff coming soon.
Major echo canceller improvements by me: Based on the WebRTC.org audio processing library, we now do better echo cancellation, remove the need to fiddle with the mic volume knob and have fixed AEC between laptop speakers and a USB webcam mic.
A virtual surround module by Niels Ole Salscheider: Try it out for some virtual surround sound shininess!
Support for Xen guests by Giorgos Boutsiouki: Should make audio virtualisation in guests more efficient.

Special thanks from me to Collabora for giving me some time for upstream work.

Packages are available on Gentoo, Arch, and probably soon on other distributions if they’re not already there.

PyGObject 3.2.1 released

2012-05-10T19:10:52+00:00

I am pleased to announce version 3.2.1 of the Python bindings for
GObject. This is a maintenance release for the stable branch bringing
you a couple of bug fixes.

Download

http://download.gnome.org/sources/pygobject/3.2/pygobject-3.2.1.tar.xz

sha256sum:
0f07401289fa33e8caee7a9342f5c838bb44cd0a386cb99093505edb5e42817a

Highlights

Add missing override for TreeModel.iter_previous() (Martin Pitt)
GSettings: allow extra keyword arguments (Giovanni Campagna)
pygtkcompat: Correct Userlist module use (Jose Rostagno)
GTK overrides: Add missing keyword arguments (Martin Pitt)
pygi-convert.py: Drop obsolete drag method conversions (Martin Pitt)
Fix len_arg_index for array arguments (Bastian Winkler)
Add missing GObject.TYPE_GTYPE (Martin Pitt)
Add GtkComboBoxEntry compatibility (Paolo Borelli)

Changes

http://download.gnome.org/sources/pygobject/3.2/pygobject-3.2.1.changes

About PyGObject

GObject is a object system used by GTK+, GStreamer and other libraries.

PyGObject provides a convenient wrapper for use in Python programs when
accessing GObject libraries.

Like the GObject library itself PyGObject is licensed under the GNU
LGPL, so is suitable for use in both free software and proprietary
applications. It is already in use in many applications ranging from
small single purpose scripts up to large full featured applications.

PyGObject now dynamically accesses any GObject libraries that uses
GObject Introspection. It replaces the need for separate modules such as
PyGTK, GIO and python-gnome to build a full GNOME 3.0 application. Once
new functionality is added to gobject library it is instantly available
as a Python API without the need for intermediate Python glue.

Quart de siècle

2012-05-10T04:17:47+00:00

Today, I am turning 25 years old. Holy carp. A quarter of a century old. At this rate, how the heck can I solve the massive amount of bugs remaining in GNOME and GStreamer in only a hundred years?

Video decoding in a sandbox

2012-05-08T09:01:45+00:00

I would like to explain a bit the stuff I've been working on recently at Igalia. It is about playing with GStreamer and a sandboxing system to try and make the playback of untrusted media more secure. Hopefully writing this will be an occasion for me to get more distance and understand things better, and for others to give me feedback and ideas. Particularly, even though that is for me a field of interest, I do not claim to have any real expertise in security, therefore comments by people who know better would be gladly welcome.

This story started when I decided to have a look at chromium and its internals. It turns out that one very specific aspect of this application is its sandboxing system. In a nutshell, a sandbox is a virtual container in which untrusted programs can be safely run. In the real world, sandboxes are rarely perfect, but they are a significant security improvement over not using one. Chromium uses a sandbox to run its rendering engine (WebKit), which is basically the part that transforms the code of a web page into the graphical representation of it that you see on your screen. The rationale for running WebKit in a sandbox is not that it is untrusted code in itself, but rather that it is a big and complex project that is bound to have bugs, like all big and complex projects. On top of that, the input given to it is quite often data from untrusted sources, which could potentially be forged so that it exploits security bugs to do bad things to your computer and your beloved files. Now, with WebKit running in a sandbox, if a web page has been forged by an attacker to exploit a vulnerability in WebKit, the attacker will only have access to the sandbox environment, which means that it won't be able to do things like access the data on your computer, install software or connect to remote hosts.

As you might know, I like to play with multimedia things, and have hacked quite a bit on or around GStreamer. Therefore, I quite automatically thought of something else that might be worth running in a sandbox: demuxers and decoders. They are relatively big and complex pieces of software to which we regularly pass a whole bunch of untrusted data, would it be in a web context or a more traditional desktop or mobile context.

Fortunately, Julien Tinnes, a developer of the chromium sandbox for GNU/Linux made a stand alone version of it called setuid-sandbox, which can be used by other projects to easily sandbox any process.

Architecture

The way setuid-sandbox works is rather straightforward: there is a sandboxme command that needs to be installed setuid root. You run sandboxme my_command and then from inside my_command, you first set up the file descriptors that you will need (being careful not to put there anything that could allow to escape the sandbox, more on that later), and then you call the provided chrootme() function, which will tell the sandboxme process to restrict the privileges that my_command has (e.g. it can still read and write on the fds that it has open, but it cannot open new ones).

Here is how I organised my integration of setuid-sandbox into GStreamer. What I want to do for now is to put what I think are the "most dangerous" parts (demuxing and decoding) in the sandbox, while letting the other components (mainly source and sinks) outside of the sandbox (for now at least). I decided to create a small program (called gst-decoder) that receives the original muxed and encoded video stream and outputs the decoded video and audio buffers. gst-decoder needs 3 channels of communication with the "controlling" process outside the sandbox (which is called the broker):

one to pass the original stream from the source element in the broker to gst-decoder

one to pass the video buffers from gst-decoder to the video sink element in the broker

one to pass the audio buffers from gst-decoder to the audio sink element in the broker

In the future, more channels for subtitle support or other features could be desirable.

Since I am lazy, I wanted to use off the shelf GStreamer elements to handle these communication channels. For the cases explained above, that would be:

the fdsink element on the broker side, and the fdsrc element in the sandbox

shmsink (in gst-decoder) and shmsrc (in the broker)

same elements as above

Since I expect other people to be equally lazy^W^W^Wwant their life to be made easier, my goal is to try and have this reasonably integrated in GStreamer, and easy to integrate in applications. For that, my best idea so far was to make a sandboxedecodebin element that, from the outside, works like decodebin or decodebin2, at least for simple cases: it has a sink pad that can take any format you would throw at decodebin, and it has an audio and a video source pads that output the decoded result. In the future, it might or might not be a good idea to try to integrate the "sandboxed" functionality in decodebin directly.

I implemented sandboxeddecodebin as a subclass of GstBin, and it has the following flow inside it:

fdsink -> [gst-decoder] | -> shmsrc (video) -> gdpdepay
                        | -> shmsrc (audio) -> gdpdepay

Note that gst-decoder is an external (sandboxed) process, and not a GStreamer element like the other entities of this data flow graph. The sink pad of fdsink and the source pads of the two gdpdepay elements are exported by sandboxeddecodebin through ghost pads, which provides a decodebin-like interface.

The gst-decoder program basically runs a pipeline that looks like that:

fdsrc ! decodebin2 name=decoder
decoder. ! video/x-raw-yuv;video/x-raw-rgb ! gdppay ! shmsink (video)
decoder. ! audio/x-raw-int;audio/x-raw-float ! gdppay ! shmsink (audio)

and it also makes sure to get the privilege dropped at the right time, which is discussed below.

When to drop privileges?

The ordering of operations needs to be thought carefully to combine GStreamer, and these elements in particular, with setuid-sandbox. Each of them brings its own set of conditions.

For setuid-sandbox, inside the sandbox (in gst-decoder):

before we call chrootme(): we can open new fds and do a lot of nice initialisation, and we don't want to parse any untrusted data

after we call chrootme(): we can't open new fds any more, or do similar initialisation tasks, but we can work on the data we received.

GStreamer has several states in which an element can be, with some rules on what should be done in which state. From the design documentation, the states are defined as follow:

NULL: This is the initial state of an element.

READY: The element should be prepared to go to PAUSED.

PAUSED: The element should be ready to accept and process data. Sink elements however only accept one buffer and then block.

PLAYING: The same as PAUSED except for live sources and sinks. Sinks accept and rendering data. Live sources produce data.

In particular, the elements that interest us here behave in the following way:

shmsink is responsible for the creation and destruction of the shared memory object and the associated control socket and creates them when going from NULL to READY and destroys them when going from READY to NULL. Since shmsink is used from inside the sandbox, this means that the state change NULL to READY needs to happen before chrootme(). This also means that it won't be able to clean up properly the shared memory object and the control socket.

fdsrc doesn't create nor destroy the fd it uses, so that can be done separately. Moreover, in the case of stdin, we leave that responsibility to the system.

And quite obviously, we want gst-decoder to handle buffers only after it has called chrootme(), so that it is ready to run potentially unsafe operations.

This is relatively easy: all we have to do is, in gst-decoder, to call chrootme() once we are in the READY state and before going to PAUSED.

Another issue with the privilege drop is that we use decodebin2 (things would be the same with decodebin), and it only loads the plugins it needs once it knows what kind of data it will have to decode. That is, it needs to load plugins after it has started to analyse potentially unsafe data. My solution to that is to preload all the installed plugins when gst-decoder starts, so that decodebin2 doesn't need any privilege to have access to the plugins it wants (they are already in memory).

This is obviously suboptimal in memory consumption. I can think of two ways to improve that:

use a white/black list of plugins to avoid loading plugins we are not likely to need (there are many things we're pretty sure not to need in gst-decoder, such as all sources and sinks or gnonlin)

use a separate typefinding sandboxed process that will determine what plugins are needed, then have gst-decoder take as argument the plugins that it needs to load before dropping privileges

Synchronising broker and sandbox

Another synchronisation issue is that the broker has to wait for the sandboxed process to be ready before interacting with it. As seen before, we have 3 channels through which they interact, and they are of two different types:

the pipe to which the broker writes, which points to stdin in the sandboxed process

the shared memory areas, and their associated control sockets created by the two shmsink

The first one is easy to synchronise: as long as the sandboxed process is not ready, it won't read on the pipe, and fdsink on the broker will just wait until it can write.

The second one is more complex: the shared memory areas are announced over the control socket when they are ready, so this part gets done correctly for free by shmsrc. But the control sockets need to exist when shmsrc tries to connect to them (this happens when going from READY to PAUSED). For now, my workaround is to sleep() for 2 seconds when sandboxeddecodebin goes from NULL to READY, after launching the subprocess. With this, the control sockets are very likely to be created when shmsrc goes from READY to PAUSED.

This is obviously very hackish , and I think I would prefer to use GFileMonitors to check when the sockets are created. Also, I don't know if it's better to do that in sandboxeddecodebin (blocking the switch to READY, but using that file monitoring instead of a sleep(), or going to READY asynchronously if that's possible?) or in shmsrc (in which case I think it should be optional and probably make shmsrc go to PAUSED asynchronously).

Making Preroll work

On the broker side, we have another tricky situation. We typically run a pipeline that contains all of this (the parts between angle brackets are outside of sandboxeddecodebin and given as examples):

<filesrc> ! fdsink (passes data to gst-decoder)
shmsrc (gets data from gst-decoder) ! gdpdepay ! <autoaudiosink>
shmsrc (gets data from gst-decoder) ! gdpdepay ! <autovideosink>

This pipeline is atypical in that it has a sink that is not really at the downstream end of it (fdsink, which sandboxeddecodebin uses to pass data to gst-decoder). Data would go through it, then through gst-decoder and its own pipeline, and then emerge back in the broker's pipeline in the shmsrc elements.

This is a problem at the preroll phase. Preroll is what usually happens when going to PAUSED: the sinks wait until they have a buffer to render before committing the state to PAUSED. The issue with our pipeline, is that the "real" sinks will only get the data they need to commit to the PAUSED state if fdsink lets the data through, but fdsink only passes data once it is in PLAYING state (apart maybe from one initial buffer). On top of that sandboxeddecodebin is a subclass of GstBin. By default, GstBin only changes to the next state (e.g. PLAYING) once all its elements have reached the previous one (e.g. PAUSED). This gives us a nice deadlock: the final (downstream) sinks are waiting for data to come to them to commit their change to PAUSED, GstBin is waiting for all its elements (including final sinks) to finish their transition to PAUSED before asking them to go to PLAYING, and fdsink is waiting to be asked to switch to PLAYING before it lets the data through (that the final sinks are waiting on). My workaround to solve this deadlock is to manually request fdsink to go to PLAYING when sandboxeddecodebin is switching to PAUSED. That way, fdsink is "one state ahead" of the rest, and lets the data go through. I haven't decided yet if it's a very ugly way of solving that issue or if it's an awesome clever hack. If you have an idea of a cleaner solution, feel free to suggest it in the comments!

Analysis of open file descriptors

Once the privileges have been dropped, the sandboxed process is very limited in what it can do, but it still can use all the fds that it has open, which might be a way for it to escape the limitations we want to put on it. For instance, imagine that the sandboxed process has an open fd on the device that contains your home directory (say, /dev/sda). By reading it, it can access all your data, even though the sandbox is designed not to let it open more files.

This precise example is very unlikely to happen in our case, but some less obvious fds could lead to ways to escape the sandbox. That is why I think it is necessary to analyse the file descriptors that are open in the sandboxed process and to try to understand the risks they bring.

I took a "snapshot" of the open fds of gst-decoder while it was decoding a video, and here is what it looks like:

guijemont@thirtytwo:~$ ls -lv /proc/5860/fd
total 0
lr-x------ 1 guijemont guijemont 64 2012-04-18 18:17 0 -> pipe:[8348338]
lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 1 -> /dev/pts/5
lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 2 -> /dev/pts/5
lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 3 -> anon_inode:[eventfd]
lr-x------ 1 guijemont guijemont 64 2012-04-18 18:17 4 -> pipe:[8348342]
l-wx------ 1 guijemont guijemont 64 2012-04-18 18:17 5 -> pipe:[8348342]
lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 6 -> socket:[8358884]
lr-x------ 1 guijemont guijemont 64 2012-04-18 18:17 7 -> pipe:[8359036]
l-wx------ 1 guijemont guijemont 64 2012-04-18 18:17 8 -> pipe:[8359036]
lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 9 -> anon_inode:[timerfd]
lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 10 -> /run/shm/shmpipe. 5860.    0
lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 11 -> socket:[8358886]
lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 12 -> socket:[8358887]
lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 13 -> socket:[8358888]
lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 14 -> /run/shm/shmpipe. 5860.    1
lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 15 -> socket:[8358890]
lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 16 -> socket:[8358891]
lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 17 -> socket:[8358892]
lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 18 -> socket:[8358893]
lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 19 -> socket:[8358894]
lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 20 -> socket:[8358895]
lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 21 -> socket:[8348346]
lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 22 -> socket:[8348347]

I used the "usual suspects" (strace and gdb) to look further into this and understand where each fd comes from, and try to get an idea of how necessary it is and of how much of a risk it brings.

O: that is stdin, and it is the pipe I create when starting sandboxme gst-decoder. Also, it is read only. I don't think an attacker could do much with this, and we need it anyway.
1 and 2: stdout and stderr, plugged to the pseudo tty where my test gst-launch command was running. This is clearly not necessary, and could be exploited for privilege escalation if there's a bad bug somewhere in devpts. I modified the code to make stdout and stderr point to /dev/null instead when gst-decoder is launched. There is an environment variable that can prevent that from happening when one wants to see the debug messages that are output by gst-decoder.
3: This is an event fd used by the GMainContext. I suspect that at least a few of the components we run to decode our stuff needs a GMainLoop, and therefore a GMainContext. And I don't think this is very dangerous, though I don't know much about the complexity and safety of the event system.
4 and 5: this is a pipe used by the GLib unix signal code. Both ends of the pipe are inside the sandbox, so I don't think this would be much of a problem.
6 and 21: shm area control socket for audio. There is one fd created by socket() that is bound to the right temporary file, then another fd is created by accept() when the broker connects. We definitely need that if we want to use shm, which I think we do for performance reasons (I did not run benchmarks though).
7 and 8: pipe open by some code in /usr/lib/frei0r-1/facedetect.so when it is g_module_open()'ed. I don't think we need that at all, and it might be a good motivation to try and not load all plugins. A limited risk though, since both ends of the pipe are inside the sandbox.
9: a timer fd opened in the same conditions as the pipe of fds 7 and 8 (by frei0r's facedetect). This one definitely looks like an unnecessary risk, though I don't know how much of a risk it actually is.
10 and 14: these are the shared memory areas (one for audio, one for video), so I think we definitely want them. The alternative would be to use regular sockets instead to pass the buffers, but I fear it might cost us much in performances for little added security, though this issue could deserve more investigation.
11, 12, 15, 16, 17, 18, 19 and 20: these 8 fds are actually 4 socket pairs, with each time both ends inside the sandbox. They are all created by gst_poll_new(), by the following pieces of code:
- shmsink in gst_shm_sink_start(). It does that twice: once for audio, once for video.
- fdsrc in gst_fd_src_start().
- the system clock (in gst_system_clock_init(), via gst_poll_new_timer()).
13 and 22: shm area control socket for video. There is one fd created by socket() that is bound to the right temporary file, then another fd is created by accept() when the broker connects. We definitely need that if we want to use shm, which I think we do for performance reasons (I did not run benchmarks though).

Play with it!

You can check out the code from its github repository, instructions are available here.

GStreamer Conference 2012 Announced

2012-05-02T15:31:00+00:00

The popular GStreamer user and developers Conference is back for the third year, this year going to San Diego in the USA. The conference will take place on August 27th and August 28th. Check out our conference page for details and the Call for Papers and mark the dates in your calendar. See you in San Diego!

Androidifying your autotools build the easy way

2012-05-01T19:48:15+00:00

Derek Foreman has finally written up a nice blog post about his Androgenizer tool, which we’ve used for porting PulseAudio, GStreamer, Wayland, Telepathy and most of their dependencies to Android.

If you’ve got an autotools-based project that you’d like to build on Android, whether on the NDK or system-wide this is really useful.

Flumotion Transcoder 0.11.1 "Habushu" released

2012-05-01T01:00:00+00:00

The Flumotion team announces a new release of the development 0.11 branch of the Flumotion Transcoding Platform.

0.11.1 release notes for more details.

PulseAudio on Android: Part 2

2012-04-30T10:35:27+00:00

Some of you might’ve noticed that there has been a bunch of work happening here at Collabora on making cool open source technologies such as GStreamer, Telepathy, Wayland and of course, PulseAudio available on Android.

Since my last blog post on this subject, I got some time to start looking at replacing AudioFlinger (recap: that’s Android’s native audio subsystem) with PulseAudio (recap: that’s the awesome Linux audio subsystem). This work can be broken up into 3 parts: playback, capture, and policy. The roles of playback and capture are obvious. For those who aren’t aware of system internals, the policy bits take care of audio routing, volumes, and other such things. For example, audio should play out of your headphones if they’re plugged in, off Bluetooth if you’ve got a headset paired, or the speakers if nothing’s plugged in. Also, depending on the device, the output volume might change based on the current output path.

I started by looking at solving the playback problem first. I’ve got the first 80% of this done (as we all know, the second 80% takes at least as long ;) ). This is done by replacing the native AudioTrack playback API with a simple wrapper that translates into the libpulse PulseAudio client API. There’s bits of the API that seem to be rarely used(loops and markers, primarily), and I’ve not gotten around to those yet. Basic playback works quite well, and here’s a video showing this. (Note: this and the next video will be served with yummy HTML5 goodness if you enabled the YouTube HTML5 beta).

(if the video doesn’t appear, you can watch it on YouTube)

Users of PulseAudio might have spotted that this now frees us up to do some fairly nifty things. One such thing is getting remote playback for free. For a long time now, there has been support for streaming audio between devices running PulseAudio. I wrote up a quick app to show this working on the Galaxy Nexus as well. Again, seeing this working is a lot more impressive than me describing it here, so here’s another video:

(if the video doesn’t appear, you can watch it on YouTube)

This is all clearly work in progress, but you can find the code for the AudioTrack wrapper as a patch for now. This will be a properly integrated tree that you can just pull and easily integrate into your Android build when it’s done. The PA Output Switcher app code is also available in a git repository.

I’m hoping to be able to continue hacking on the capture and policy bits. The latter, especially, promises to be involved, since there isn’t always a 1:1 mapping between AudioFlinger and PulseAudio concepts. Nothing insurmountable, though. :) Watch this space for more updates as I wade through the next bit.

GStreamer backend for video in Firefox

2012-04-29T22:19:34+00:00

Good news to hear that the GStreamer backend for video playback in Firefox has landed, due to a flurry of work by Alessandro Decina in the last few months. Of course, this isn’t part of the standard Firefox build (but maybe some day?), but it’s very useful for putting Firefox on mobile and embedded platforms, since GStreamer has a well-established ecosystem of vendor-provided plugins for hardware decoding.

Seagull Strike

2012-04-27T17:58:01+00:00

Overview

A while back a wrote a little game for the Nokia N9 and N950, mostly as a way to explore the QML Box2D bindings. However, whilst packaging up my StatusNet client I thought I might as well release this as well since it can be a fairly enjoyable distraction. Below is a short video demonstrating how it works, and at the end of the post you can find details on where to get hold of it.

Video

Download

You can either download it via Apps for Meego, or get hold of the package directly here: http://mikeasoft.com/~mike/seagullstrike_0.1-2_armel.deb.

Source

License: GPL version 3.0 or later
Gitorious repository: https://gitorious.org/seagull-strike
Ohloh project page: https://www.ohloh.net/p/seagull-strike

StatusNet for MeeGo 0.2.1

2012-04-26T00:57:58+00:00

Overview

StatusNet for MeeGo makes it possible for Nokia N9, N950 and similar phones to connect to StatusNet compatible microblogging services such as Identi.ca. It supports viewing statuses in the phone’s event feed alongside Twitter and Facebook updates, viewing conversations, posting new status updates and replying to other people.

New Features

Support for followings links in posts.
Links to groups, tags, and users.
Support for posting messages using UTF-8 characters.
Slight improvements to the clarity of the login screen.

Download

Ovi Store: StatusNet for MeeGo
Direct download: statusnet-meego_0.2.1-1_armel.deb

Source

License: GPL version 3.0 or later
Gitorious repository: https://gitorious.org/statusnet-meego-plugin
Ohloh project page: https://www.ohloh.net/p/statusnet-meego

Starting a new chapter

2012-04-25T17:48:29+00:00

Having signed and sent of my contract I think the time has come to let the every one know that I am starting to work for what I think is a very cool company in just two Months, Red Hat. I have been a Red Hat and later Fedora user ever since I first tried Linux back in the late 90ties (by installing either 4.1 or 4.2 of Red Hat Linux), so it does feel like a bit of a homecoming for me. And in my opinion Red Hat is still what they where back then, the biggest driving force behind pushing linux and open source forward. And this is what I hope to be a part of by joining Red Hat, to play my part in continuing to push new innovations into the operating system and through that make it an even better choice for more and more users and organisations out there.

It is still with a bit of melancholy that I leave Collabora behind as I am very proud of the team we built there over the last five years. The amount of features and maturity we managed to put into GStreamer over those years and all the new usecases we managed to cover is quite amazing. But with Rob and Philippe at the helm I am sure the company will continue to prosper even without me.
I will of course continue to be a part of the GStreamer community and will be working with both Collabora and the GStreamer community for instance on organizing this years GStreamer Conference (hope to have the GStreamer Conference 2012 page up in a few days). So while I am now moving onto a new challenge I do not plan on leaving all the great people and friends I made behind, in fact since my new role at Red Hat will be as part of the Desktop team I will continue to haunt a lot of the same conferences and gatherings in the years to come .

Also as part of this new job I will be moving to Brno in the Czech Republic, joining the 400 strong Red Hat team there. For me, the best thing about living in Brno, in addition to being a very nice city, is that it will allow me to put even more pressure on Jimmac to create a new icon for Transmageddon, as he will be living only a few hours away by car

Thumbnails are back

2012-04-23T17:00:21+00:00

…it’s back. And it will not stop until it has accomplished its mission.

This week, Daniel Thul fixed and reactivated the clip thumbnailing code for Pitivi’s timeline. But he didn’t just stop there. No sir. He implemented caching too, and that made my week.

We don’t just process thumbnails on-demand anymore, we now save them to disk for reuse. Notice the length of the clip below? It is two hours long. And its thumbnails load instantly at startup.

That’s not all. The thumbnails you see above were using “nearest neighbor” scaling. If you are using GStreamer 0.10.36 or newer, you get better thumbnail quality, because we now use multitap Lanczos image scaling:

Update: Daniel also blogged about his findings.

La pression sociale

2012-04-22T21:05:02+00:00

Voici quelques pensées générales sur la responsabilité, le déterminisme et sur la société dans laquelle nous vivons.

Je vais à contre-courant sur une poignée de choses dans la vie:

Je refuse d’utiliser Facebook pour gérer mes relations avec mes véritables amis (en d’autres termes, les « relations privées »), pour les raisons énumérées précédemment.
Je n’utiliserai plus jamais Windows comme mon système d’opération, ou, en fait, n’importe quel système d’opération propriétaire. Avec les années, j’ai vu des choses, dans l’industrie, que certains auraient peine à croire. Pour quelque chose d’aussi crucial (pour mon travail et ma vie personnelle) que mon ordinateur, je ne peux aucunement faire confiance à autre chose qu’un système open-source. Ce n’est pas juste une question philosophique, c’est également une question de pragmatisme, pérénité et sécurité. Mais ça, c’est une autre discussion.
Je n’ai pas et n’aurai pas d’abonnement de téléphone cellulaire, tant que les forfaits ne correspondront pas à ce que je recherche en termes de fonctionnalités et de prix, ou à moins qu’on me paie pour l’utiliser. D’ici là, j’ai un système combinant ligne fixe et VoIP avec boîte vocale qui fonctionne parfaitement bien et je ne manque les appels que dans trois situations: quand je suis sur la route, lorsque je ne veux pas déranger ou interrompre mon entourage ou quand je dors.
Je refuse également d’avoir une carte de crédit.

Pourtant, quand on dévie de la norme sur les points que j’ai énoncés plus tôt, on passe pour un hippie venu de Pluton. Les hippies de Pluton devraient-ils être ostracisés parce que notre société valorise plus que jamais l’hédonisme, la pensée à court terme et le matérialisme?

On fait tous des choix dans la vie. Certains de mes choix font d’ailleurs en sorte que je suis libre de dettes. Par exemple, ne pas fumer (et résister pendant des années aux pressions sociales en ce sens), rester plus longtemps chez mes parents plutôt que de me précipiter à vivre en appartement, ne pas posséder de voiture et tous les coûts que ça implique, ne pas être à la dernière mode (je porte encore des t-shirts qui datent de 1997), aller à la bibliothèque au lieu d’acheter mes livres (incluant les manuels obligatoires de cours), utiliser des ordinateurs usagés, ne pas avoir d’iPhone/iPad dernier cri, éviter de manger au restaurant (en faveur de l’expérimentation culinaire à la maison!), choisir de terminer mes jeux de PS1 et PS2 avant de considérer l’idée d’une nouvelle console, etc.

Mon mode de vie vous semble peut-être frugal, et pourtant je suis probablement parmi les gens les plus heureux qui soient.

La stimulation socio-technologique constante à la Facebook n’est qu’une « amélioration de la solitude », et ce n’est qu’illusion: on est toujours connectés les uns aux autres par ces moyens technologiques, mais ces contacts sont fragmentés et perdent leur sens; on est connectés mais on ne ressent rien, parce que c’est un simulacre.

Surviving Git

2012-04-13T16:14:21+00:00

Git can feel extremely unnatural to those who came from Bazaar. Only after a lot of advice from friends, the right tools and a few months of intensive usage did I feel confident enough to use it without fear. Many are probably in the same boat as me though, so I took a couple of hours to clean up my personal notes and make a proper “crash course” tutorial for Git in the Pitivi context. I have also made a nice video on how to use interactive rebase, if you’re interested.

GStreamer Core 0.11.90, Base Plugins 0.11.90, Good Plugins 0.11.90, Bad Plugins 0.11.90, Ugly Plugins 0.11.90, libav Plugins 0.11.90 unstable release

2012-04-13T09:33:00+00:00

The GStreamer team announces a new release of the GStreamer core, Base/Good/Bad/Ugly/libav modules for the 0.11 GStreamer unstable release series.

This is the first release candidate of the upcoming 1.0 release. It is intended for developers and people wanting to port their plugins and applications to the new series. Only minor or absolutely necessary changes to the core/base API/ABI will happen between this release and the final 1.0.0 release.

Plotting the future of Transmageddon

2012-04-07T10:42:52+00:00

After a long period of very slow development of Transmageddon where most work has been in debugging issues with GStreamer 0.11 and porting existing features to GTK3 and GStreamer 0.11, I decided it was time to start thinking about where to go forward again. That said, there are still some issues with the new GTK3 and GStreamer 0.11 version, but I needed a break from just tedious porting/re-implementation work and instead start looking at some new ideas and features.

There are two major items I am thinking of. The first is what to do with the usecase of people just wanting to create a manual transcode. The goal of Transmageddon is to make transcoding simple, but the problem is that I am trying to simplify something that is inherently quite complex. The current user interface doesn’t really let you set a lot of things and I know that sometimes that will create files that doesn’t conform to the needs of the user, for instance a lot of settings are just kept from the original output file, like number of audio channels or bitrate used and so on. And many others just rely on the default values of the GStreamer elements used. I don’t want to try to support tweaking all of them through the user interface as there is no way of doing so without making the user interface either cluttered or filled with what will be for most people just gibberish. It is the problem I feel for instance Handbrake is suffering from, that yes it does let you set everything you could ever need, so it can create files that will be useable in some cases where the default user interface of Transmageddon falls short, but it also becomes a hard to navigate jungle for people. My answer to the need for that level of settings is and will continue to be the device profiles option, which I also have some plans for. That said there are a couple of features I do think would be useful to enable from the non-preset interface, like being able to resize the video, choose if you want deinterlacing or not and to allow you to choose stereo output for the audio and finally I want to allow the creation of transcoding queues, so you don’t have to wait until one transcode is finished before configuring another one.

The one special feature I already got in there, the ability to rotate the video, so that if your video for instance was shot with a camera held sideways, also felt rather arbitrary (it was put in there due to a very early bug request) to have exposed in the userinterface.

So what I have been experimenting with is as user interface which has an Advanced menu option in which you can set certain things and also enable certain extra options in the user interface.

Idea around adding an Advanced option in Transmageddon

So as you see in the screenshot above there are certain options which you just turn on/off and certain options which when enabled will add extra elements to the userinterface as shown below.
If I go down this route the question of course comes what options to put into this Advanced menu and also if I should make the settings persistent accross runs of Transmageddon or if I should let Transmageddon return to its default settings every time you start it.

Transmageddon with all advanced options enabled

The Transmageddon editor

Of course the Advanced menu will never become the solution to all transcoding challenges, the device profiles will still be that. But I realized that editing textfiles is not for everyone and it also makes creating advanced profiles a task for even fewer people than there needs to be. I don’t know how many people out there have made their own profiles, but the only one I ever got submitted was from Stefan Kost who made one for the N900. That is probably mostly due to lack of documentation of how to create profiles and where to put them, a problem I have been planning to remedy. That said I realized that maybe creating some kind of editor would be an even better solution, as it could provide a lot of helpful tools for profile creation and thus making it accessible to a lot more people. Which is why I been trying to prototype what such an user interface could look like.

Screenshot of prototype for Transmageddon profile editor

The interface above is what I got so far, and it is just a glorified profile viewer atm, but my hope is to make it fully functional and hopefully useful to people. One feature I really want to do is to allow you to take an existing video, have the profile editor analyze it and create a new profile that will allow you to replicate the settings of that file. So when you get a new phone or device the manufacturer most likely put a sample file on it, you can then load that sample into the editor and the editor will create a profile that matches it. This will enable you to transcode other files to that profile and thus make them work on your device.

That said this will be a major task creating this editor, because I want to to contain a lot of clever logic, so that it doesn’t just end up being a glorified text editor, but I will need to test and experiement to figure out what that logic will be and how to expose it in the user interface.

Anyway, I am hoping to hear back from the community on these two new things and playing with to hear what you think, both from a usability standpoint and of course ideas for how the features could work or should not work, and what you would need to make Transmageddon suit your needs.

StatusNet for MeeGo 0.2

2012-04-07T00:45:47+00:00

Overview

Assuming no issues are encountered by testers I plan on submitting this version to the Apps For MeeGo repository, and later also the Nokia Ovi Store.

You’ll need to restart your phone before statuses start appearing in the events feed, although you’ll be able to use the main app straight away. Because of some changes in the dbus services its also necessary to restart your phone even if you’ve already installed version 0.1 (this won’t be needed after future updates though).

New features

Posting updates.
Posting replies.
Viewing messages in the application.
Display full conversations.
Load additional statuses from earlier in time.
Splash screen while connecting.
Improved dbus handling.
Asynchronous message loading and posting.

Planned features

Support for displaying rich content (pictures, videos, etc.)
Ability to mark statuses as being “favourites”.
Option to subscribe to new users in the app.

Download

Ovi Store: StatusNet for MeeGo
Direct download: statusnet-meego_0.2-4_armel.deb

Source

License: GPL version 3.0 or later
Gitorious repository: https://gitorious.org/statusnet-meego-plugin
Ohloh project page: https://www.ohloh.net/p/statusnet-meego

OggStreamer: audio capture and streaming device

2012-04-06T00:30:39+00:00

Recently learned about a cool new open hardware project called OggStreamer. They’re designing and making a small device that records an analog audio signal and streams it using Ogg/Vorbis. It’s an open hardware project, so all the schematics and PCB layout is provided.

PulseAudio in Google Summer of Code 2012

2012-04-04T12:27:43+00:00

If you’re a student participating in this year’s edition of Google Summer of Code and want to get your hands dirty with some fun low-level hacking, here’s a quick reminder that PulseAudio is a participating organisation for the first time, and we have some nice ideas for you to hack on.

The deadline for applications is 2 days away, so get those applications in soon! If you’ve got questions, feel free to drop by #pulseaudio on the Freenode IRC network and ping us. (I’m Ford_Prefect there for those who don’t know)

3 Apr 2012

2012-04-03T19:40:09+00:00

Buzztard & Distro Versions

I started hacking on the transport settings. The idea here is to have a generic scheme for how to control playback (start, stop, seek) externally. This can be a midi-controller, but also qjackcontrol or a MPRIS applet. The setting now show the available modules and allow to enable master and slave mode as well as mode specific settings. I could remove the combobox from the toolbar again.

As a next big thing I started to refactor a big chunk of duplicated code in core. Machines have global parameters, per voice parameters and per incoming wire parameters. Now we have a parameter-group class that deals with those.

I spend the rest of the month cleaning up the test suite and the autofoo setup. On the latter I bumped the required versions. The policy now is to support the linux distributions from about the last two years. This allowed me to remove large chunks of conditional code. Unfortunately it is not easy to do that. The easy part is to check what versions we require, by looking at the pkg-config macros in configure.ac. Then we can also grep for conditional section in the code:
find . $ -name "*.c" -o -name "*.h" $ -exec egrep -o "[A-Z]*_CHECK_VERSION$.*$" {} \; | sort | uniq
Now the tricky part is to figure what distributions ship. It is somewhat easier for debian/ubuntu as they have pages like:

http://packages.debian.org/search?suite=all&arch=any&searchon=names&keywords=libgstreamer0.10-0http://packages.ubuntu.com/search?suite=all&section=all&arch=any&searchon=names&keywords=libgstreamer0.10

For fedora Company suggested to look at the spec files in git and iterate over the branch names
http://pkgs.fedoraproject.org/gitweb/?p=gstreamer.git;a=blob;f=gstreamer.spec;hb=f17
For opensuse Vuntz suggested to use an osc query at which I failed misserably and in the end checked the packages in the repo and iterated over the versions:
http://download.opensuse.org/distribution/12.1/repo/oss/suse/i586/
Anyone know better tricks? Please share them!

One more motivation for doing this is upcomming gsoc. If I am lucky to get a student who will port buzztard to gstreamer 1.0. We'll release a 0.7 of buzztard before we switch to the new gstreamer api.

212 files changed, 7954 insertions(+), 7778 deletions(-)

StatusNet client for the Nokia N9/N950

2012-04-03T15:58:04+00:00

There’s one thing that my Nokia N950 has been lacking and that’s a StatusNet client, so I set about hacking one together. I love the way that the N9/N950 events feed displays messages from Twitter and Facebook, so my main focus so far has been to integrate with this. I’ve now got my client to a stage at which it covers most of my own needs so I thought I’d make an early release for others to play with.

Once I’ve extended it a little further I plan on submitting it to both Apps For Meego and the general Nokia Ovi Store, but for now you can download the Debian package directly here.

One thing to note is that you will need to restart your phone before events will start to appear (this is due to a limitation in the way the SyncFW framework loads plugins).

The client is written mostly in Python, with a little C++ for integrating with the events synchronisation framework. It makes use of the StatusNet module written for the command line StatusNet client IdentiCurse and the python-eventfeed module written by Thomas Perl.

Features

* OAuth login for identi.ca.
* Standard login for any other StatusNet services.
* Shows messages in the events feed.
* Fetches (and caches) user avatars.
* Displays messages from the events feed in the browser when clicked.
* Automatically fetches new messages in the events feed based on your events feed preferences.
* Can manually fetch new messages via the events feed ‘Refresh’ button.

Not yet implemented

* Posting messages.
* Displaying messages locally within the application.
* Displaying rich content (messages with multimedia attachments)

Download

Ovi Store: StatusNet for MeeGo
Direct download: statusnet-meego_0.1-3_armel.deb

Source

License: GPL version 3.0 or later
Gitorious repository: https://gitorious.org/statusnet-meego-plugin
Ohloh project page: https://www.ohloh.net/p/statusnet-meego

Update on the GStreamer DeckLink Elements

2012-04-03T04:56:50+00:00

A little more than a year ago, I posted about GStreamer support for SDI and HD-SDI using DeckLink hardware from BlackMagic Design. In the meantime, the decklinksrc and decklinksink elements have grown up a bit, and work with most devices in the DeckLink and Intensity line of hardware. A laundry list of features:

Multiple device support
Multiple input and output support on a single device
HDMI, component analog, and composite input and output with Intensity Pro
Analog, AES/EBU, and embedded (HDMI/SDI) audio input
SDI, HD-SDI, and Optical SDI input and output with DeckLink
Works on Linux, OS/X (new), and Windows
8-bit and 10-bit support for SDI/HD-SDI
Supports most video modes in the DeckLink SDK
Implements GstPropertyProbe interface for proper detection as a source element
Lots of bug fixes from previous releases

Kudos to Blake Tregre and Joshua Doe for submitting several of the patches implementing the above list. There still a bunch of outstanding bug reports (some with patches) that need to be fixed. Several of these relate to output, which is currently rather clumsy and broken.

People have asked me about automatically detecting the video mode for input. Some DeckLink hardware has this capability, but not any of the hardware I have to test with. However, I’ve had some success with cycling through the video modes at the application level, with a 200 ms timeout between modes, stopping when it finds a mode than generates output. This works ok, except that it tends to confuse 60i and 30p modes (and 50i with 25p), which can be differentiated with a bit of processing on the images. At some point I’d like to integrate this functionality into decklinksrc, but wouldn’t be upset if someone else did it first.

PiTiVi GES spring sitrep

2012-04-02T22:54:11+00:00

Now that the Internet is back to normal, here’s a status update on what I’ve been busy doing lately.

The new media library clip previewer, to see all the important details.

You may remember my initial donation experiment last month where I presented the new realtime trimming preview feature—I was not kidding. It’s merged in the pitivi “ges” development branch now, with some additional improvements tacked on.

Since my last blog post, I’ve been spending a lot of spare time:

Preparing Pitivi for GSoC (and in general, every detail counts to make the lives of new contributors easier)
Mentoring GSoC candidates
Filing bugs and cleaning up my shared internal “Pitivi GES TODO”
Making GNOME git the official development repository for PiTiVi
Reviewing and merging patches. One of the new visible features I’m enthusiastic about is the ability to export standalone archives of your project (thanks to Paul Lange), which can come in handy for debugging!
Documenting various things:

What GES is and how important the port to GES is to us.
Creating some new brainstorming pages for the title editor and for grouping/nesting.
The situation with GTK 3/GSt 1.0 (hopefully, with some help we will resolve that quite soon, actually)

Searching for creative ways to get people to contribute to GES and solve the bugs that prevent us from making a release right now.

If you take a step back and compare to the latest stable release (0.15.1), on the pitivi side you have:

Clip thumbnails in the media library are not only much prettier: they load instantaneously too.
An automated project backup/restore system. The new code for it is beautiful. I spent a crazy amount of time testing it (those of you who know me know that I’m probably the most meticulous tester that can be), and as far as I can tell, it’s bulletproof.
Cleaned up menus and keyboard shortcuts
The ability to export the current frame
Sleep inhibition (thanks to Stephen Griffiths)
An improved clip previewer that gives you more space to view the video, yet stays out of your way (pictured above)
Better filename/paths handling
Less CPU usage when idle
Match GTK’s appearance regarding button icons
Various little UI fixes all over the place
A huge code cleanup and refactoring all around, with a vastly simplified codebase and much simpler architecture

That’s just the surface. With GES, in addition to the simplified Pitivi codebase, we have some other niceties:

Playback performance back to the 0.13.1 levels (or better?). Want to play a 1080p clip? No problem. A 4K clip? Then you’ll probably benefit from hardware decoding acceleration when we switch to GStreamer 1.0, if your video hardware and driver supports it—goodbye, CPU bottlenecks.
Importing clips is blazing fast and asynchroneous (it won’t block the UI).
Loading a project timeline is instantaneous.
Various backend bugs that plagued earlier releases are possibly a thing of the past.
Faster application startup.
Boundless possibilities…

It’s not ready just yet though. We need your help to be able to get 0.16 out of the door. There are some feature regressions (like drawing the clip thumbnails or waveforms on the timeline), but more importantly there are a handful of serious bugs in GES that are blocking us (they are not many, but they’re hard to solve). If we can get help to fix these we might finally be able to get a release (or at least an alpha) out of the door.

Evolution backup recovery

2012-04-01T14:36:12+00:00

I pretty much never drink and hack, and last Friday’s evening is a good reason why. I was having a rare beer and managed to spill part of it on my keyboard and desk. So I turned the keyboard around, started cleaning it as fast as I could, forgetting to actually unplug it. I called it a night because nothing good was going to come from that night anymore.

And on Saturday morning I noticed that my INBOX was gone. Hm, is it really gone? Yep, gone from my laptop too. Crap, must have deleted it on the server by accident while cleaning my keyboard…

And because my NAS is a little full lately, I haven’t been as diligent with backups as I normally have been. Hm, and the modest cache on my N900 isn’t very useful either…

Luckily, evolution on my work machine was shut down for some reason, so yay, it has a reasonably fresh cache of my INBOX!

Except that it’s not all that straightforward to actually get this cache back into Evolution. Just copying its contents to an existing or new folder doesn’t do anything. The files themselves are split up versions of the actual email, assumingly because the evo guys thought it would be faster to search header and body by splitting them off from the attachments and saving them separately, inventing their own caching format. Which is fine, but makes it impossible to actually restore a backup with…

After lots of Googling, I stumbled upon this tool that did the trick for me. A lot of hours wasted over a bunch of emails… But what would happen if I really lost my IMAP server mail ? Run this script by hand on all the folders ? Shudder…

git bash prompt

2012-03-29T19:39:30+00:00

I’ve been having fun recently on a new project where I put myself through all sorts of pain by nesting git submodules into team submodules into platform submodules and so on. The goal here is to be able to tag a root repository and thus identify exact commit hashes of all the submodules to any level. This was an idea Andoni had when he was working on livetranscoding in response to a request of mine where I want to be able to use a single ‘tag’ to identify a complete deployment.

That’s been working better than I expected, and I even hacked git-submodule-tools so that I can do git rlog and get a recursive git log between two root version tags, and get a list of every commit between the master and all submodules. That’s pretty neat for writing out release notes.

However, the way I embedded submodules causes a bit of pain when going back and forth. One of my hackers once gave me a PS1 bash prompt that includes info of which git branch you’re on in your shell prompt. So today I decided to extend that a little, and I now have this:

(b:release-0.2.x d:deploy-pro-2012-03-29) [thomas@otto platform]$ ls Makefile platform puppet RELEASE-0.2.1 (b:release-0.2.x d:deploy-pro-2012-03-29) [thomas@otto platform]$ cd puppet/pro/ (s:puppet/pro b:release-0.2.x d:v0.2.1) [thomas@otto pro]$

This is showing me submodule name, branch, and description of the current commit.

If you want this for your prompting fun too, here’s the github repo

In the near future, simple portknocking for fun and profit with bash!

GStreamer Google Summer of Code

2012-03-28T11:47:37+00:00

You can now submit an application as a student to be part of this years Google Summer of Code. So if you want to do a GStreamer project be sure to ping us on IRC (#gstreamer, irc.freenode.net) or send an email to the gstreamer-devel mailing list and start discussing any project ideas you might have, or to ask questions about items from our SoC ideas page..

Details on how to submit an application you can find our our Student information page.

Try to submit your application as soon as possible because if you wait until the day before the deadline of April the 6th, then the amount of time available for feedback and improvements will be small and thus decrease your chances of getting approved.

Puppet pains

2012-03-27T13:53:24+00:00

The jury is still out on puppet as far as I’m concerned.

On the one hand, of course I relish that feeling of ultimate power you are promised over all those machines… I appreciate the incremental improvements it lets you make, and have it give you the feeling that anything will be possible.

But sometimes, it is just so painful to deal with. Agent runs are incredibly slow. It really shouldn’t take over a minute for a simple configuration with four machines. Also, does it really need to be eating 400 MB of RAM while it does so ? And when running with the default included web server (is that webrick ?), I have to restart my puppetmaster for every single run because there is this one multiple definition that I can’t figure out that simply goes away when you restart, but comes back after an agent run:
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Class[Firewall::Drop] is already defined; cannot redefine at /etc/puppet/environments/testing/modules/manifests/firewall/drop.pp:19 on node esp

And sometimes it’s just painfully silly. I just spent two hours trying to figure out why my production machine couldn’t complete its puppet run.

All it was telling me was
Could not evaluate: 'test' is not executable

After a lot of googling, I stumbled on this ticket. And indeed, I had a file called ‘test’ in my /root directory.

I couldn’t agree with the reporter more:

I find it incredibly un-pragmatic to have policies fail to run whenever someone creates a file in root which matches the name of an executable I am running.

ac

2012-03-26T22:12:48+00:00

HDTV Color Matrix

2012-03-24T03:07:09+00:00

Digital video is a time series of pictures, and each picture is comprised of an array of pixels, and each pixel is comprised of three numbers representing how brightly the red, green, and blue LCD dots (or CRT phosphors, if you’re old school) glow. The representation in memory, however, is not of RGB values, but of YCbCr values, which one calculates by multiplying a 3×3 matrix with the RGB values, and then adding/subtracting some offsets. This converts the components into a gray value (Y, or luma) and Cb and Cr (chroma blue and chroma red). The reason for doing this is because the human visual system is more sensitive to variations in luma compared to variations in chroma (er, actually luminance and chrominance, see below). Furthermore, for this reason, typically half or 3/4 of the chroma values are dropped and not stored — the missing ones are interpolated when converting back to RGB for display.

There are various theoretical reasons for choosing a particular matrix, and I’ve recently become interested if these reasons are actually valid. For historical reasons, early digital video copied analog precedent and used a matrix that is theoretically suboptimal. This matrix is used in standard definition (SD) video, but was changed to the theoretically correct matrix for high-definition (HD) video. There are other technical differences between SD and HD video, but this is the most significant for color accuracy.

For some time, I’ve been curious how much of a visual difference there is between the two matrices. Here are two stills from Big Buck Bunny, the first is the original, correct image, and the second is the same picture converted to YCbCr with the HDTV matrix and then back to RGB with the SDTV matrix. (To best see the differences, open the images in separate browser tabs and flip between them.)

If you are like me, you probably have trouble seeing the difference side by side, but flipping between them makes it fairly obvious. I chose this image because it has relatively saturated green and greenish-yellow, which shows off some of the largest differences.

The RGB values for the pixels that are used in computation are not proportional to the actual amount of power output by a monitor. This is known as gamma correction, and is a clever byproduct of the fact that the response curve of television phosphors (the amount of light output for a given voltage) is approximately similar to the response curve of the eye (the perceived brightness based on the amount of light). Thus voltage became synonymous with perceived brightness, televisions had fewer vacuum tubes, and we’re left with that legacy. But it’s not a bad legacy, because just like dropping chroma values, it makes it easier to compress images.

However, color comes along and messes with that simplicity a bit. Luminance in color theory is used to describe how the brain interprets the brightness of a particular pixel, which is proportional to the RGB values in linear light space, i.e., the amount of light emanating from a display. Luma is proportional to the RGB values in gamma-corrected (actually, gamma-compressed) space. This means that luma doesn’t simply depend on luminance, and contains some variation due to color. This messes with our idea that matrixing RGB values will separate variations in brightness from variations in color. How visible is it? I took the above picture and squashed the luma to one value, leaving chroma values the same (HD matrix):

What you see here is that saturated areas appear brighter than the grey areas. This is chroma (i.e., the color values we use in calculations) feeding into luminance (i.e., the perception of brightness).

How much does this matter for image and video compression efficiency? It’s a minor inefficiency of a subtle visual difference. In other words, not very much.

Earlier I mentioned that the HD matrix was theoretically more correct than the SD matrix. What about in practice? Here’s the same luma-squashed image with the SD matrix. Notice that there’s a lot more leakage from chroma into luminance, especially in the green leaves:

GStreamer Core 0.11.3, Base Plugins 0.11.3, Good Plugins 0.11.2, Bad Plugins 0.11.2, Ugly Plugins 0.11.2, FFmpeg Plugins 0.11.2 unstable release

2012-03-22T15:58:00+00:00

The GStreamer team announces a new release of the GStreamer core, Base/Good/Bad/Ugly/FFmpeg modules for the 0.11 GStreamer unstable release series.

This is hopefully the last release of the unstable 0.11 core and base series that will lead up to the next 1.0 stable series next.

Check out release notes for gstreamer core or gst-plugins-base, gst-plugins-good, gst-plugins-bad, gst-plugins-ugly, gst-ffmpeg, or download tarballs for gstreamer or gst-plugins-base, gst-plugins-good, gst-plugins-bad, gst-plugins-ugly, gst-ffmpeg,

GStreamer Participating in Google Summer of Code 2012

2012-03-21T13:00:00+00:00

The GStreamer project is happy to announce that we are a mentoring organization in the 2012 Google Summer of Code. This means students from all over the world who are interested in open source and multimedia can apply to do a project related to GStreamer as their paid summer job this year. You can find more information about this and how to apply on the GStreamer Google Summer of Code 2012 wiki page..

GStreamer and the Google Summer of Code 2012

2012-03-21T12:37:01+00:00

A big thanks to Google for making GStreamer part of also this years Google Summer of Code. For those of you who might not know the Google Summer of Code it is basically a program that lets you do a summer internship with various leading open source projects and get paid for it (if you are a student).

So if you always wanted to get involved with the GStreamer project and become a multimedia wizard, this is a great way to do so. The process is quite simply, you write a proposal for what you want to do (some possible suggestions and info found here) and submit it to Google when the application period starts on the 26th of March. Before then you would do well to speak with us on IRC or email lists to discuss your project to maximise your chances of getting selected.

We have a special Google Summer of Code student information page with information on what the GStreamer project specifically expects of you and your proposal. So for those interested that should be a good starting point.

Hope to see many great applications this year on either GStreamer itself or GStreamer using projects. And remember, you don’t have to feel bound by our suggestion list, if you have a great idea for a project that involves GStreamer, then we are happy to mentor it, just be sure to talk to us as soon as possible so we can let you know if we think it is a suitable Summer of Code project.

Picking your battles

2012-03-20T02:52:16+00:00

Most of you have no doubt already seen that Mozilla will be changing their position on H.264 support for HTML5 video in future releases. This is an extremely important decision that I’ve been hoping to see for a while now, and I am really glad this is being done.

There is no doubt that we need patent-unencumbered standards for web codecs (or as much as is possible given the dismal patent ecology today), and while much giddy anticipation followed Google/On2′s release of VP8 into the open, I don’t believe it ever made sense to expect the codec landscape to change drastically in the short timespan everyone expected. There’s a lot of the hardware and software out there that needs to change (see any SoCs with VP8 support yet?), not to mention the interests of the MPEG-LA ~~maf~~consortium.

I love Firefox, both as a product and what it means for an open web (for those of you that know me, this might be hard to believe given all my ranting, but it’s true!). I’m glad Mozilla chose to live to fight another day rather than go out in a blaze of glory and (or a flicker of irrelevance).

p.s.: these are my views and do not necessarily represent those of my employer

p.p.s.: Alessandro’s been doing some great work to get the GStreamer multimedia backend going again (this makes so much more sense than going the NIH route!)

GSoC 2012 and PiTiVi

2012-03-18T20:50:53+00:00

Are you a student who wants to get involved in shaping the future of open source video editing? Are you looking for a friendly project providing stimulating challenges and a well-established codebase and expertise? Then consider applying for the Summer of Code programme to work on Pitivi or GES! See this page for the list of ideas for Pitivi. Don’t wait though: the deadline for applying is April 6th.

Pitivi SoC students at the Desktop Summit last year. Important note: a moustache is NOT required to apply as a GSoC student!

Not a programmer? You can help raise awareness about this (and about Pitivi in general). Maybe you know a brilliant hacker friend/family member or a top-notch computer science student, who is just waiting for a chance to make a big difference in the world. Tell that person about how cool and welcoming PiTiVi is and how getting involved is the best way to advance free, powerful and intuitive video editing for everyone!

DAD hacking

2012-03-17T23:06:59+00:00

On the bad side of life, I was planning to go to an awesome Calcotada in Lleida today, but I spent last night awake until 4:30 with an upset stomach, so I had to cancel and stay home feeling like shit.

On the good side of life, I really had no excuse left to not do a little long overdue hacking on Digital Audio Database.

I still use it regularly to listen to music, but the GNonLin-based player is just really not very stable. I should really just rewrite it using simply adder just like roughly ten years ago, but my brain won’t be able to do that. So instead I decided to clean up the web-based WebSockets using player I prototyped at OVC last year.

I started with some refactoring, clearly defining model/view/controller base classes and adapting the player and playerview classes to them.

WebSocket code seems to need an update every few months – I pulled the latest revision of txWebsocket on my fork, so my recent browsers actually play music again.

Since the last time I hacked on this, I actually added my 1500+ freshly ripped cd’s, in FLAC format – which browsers don’t actually support.

So, first off, I added an option for the scheduler – responsible for picking tracks, and picking audio files to represent them – to filter by extension. It’s not ideal, but it will do for now, and I punched that filter through the levels of abstraction in DAD. I now start it filtering on .mp3 and .oga, and so Chrome can play back all the tracks the scheduler throws at it.

The web-based player just loads tracks and timing info from the scheduler relative to page load time. I’ve been wanting to make that absolute for a while, so I did just that – the player server schedules tracks for epoch seconds now through websockets.

I had an entertaining half hour listening to the awesome echo effects obtained by having three chrome pages simultaneously playing the jukebox schedule – each page being slightly out of sync with the others.

As I’ll be wanting to use a smallish computer for music playback using a browser, I adapted the code to not use localhost any longer, but do everything with relative URL’s. Voila – the laptop now plays music too, a little bit more out of sync, and of course through its own speakers, adding to the eerie effect.

As an encore, I wanted to stumble my way through some jquery code, to which I’m a certified newbie. I want a nice background slideshow related to the current artist, and I pulled together echonest and bgstretcher-2 as an experiment.

That seems to work relatively well, except that the slideshow plugin doesn’t let you reload a new set of images to cycle through. And some of the other ones I tried instead after that seemed to have the same problem.

Oh well, it’s a start. If anyone knows of a good jquery background slideshow plugin that lets me update the list of url’s for images at any time, let me know!

Pourquoi j’ai quitté Facebook: des précisions

2012-03-17T16:45:44+00:00

Il y a deux ans je quittais définitivement Facebook. Depuis ce temps ma vie s’en est trouvée significativement améliorée, mais on me questionne encore sur les raisons de cette décision. Chaque fois, on me regarde comme si j’étais autiste. J’ai pensé aujourd’hui mettre à jour le billet que j’avais posté jadis, mais comme l’explication risque d’être longue, autant faire un nouveau billet:

J’avais tendance à y revenir à tout bout de champ dans la journée, à y passer des heures et perdre mon temps à lire un flux ininterrompu de choses qui ne me concernent la plupart du temps pas, ou au contraire à «chercher des nouveautés» qui n’existaient pas, créant alors un vague sentiment de vide et de déception.
Facebook n’est pas le produit/service. Je suis le produit que Facebook vend. Google fait évidemment la même chose, mais au moins Google m’est utile (pour GMail c’est une autre histoire: je ne me fais tout simplement pas confiance à héberger ma propre infrastructure de courriels de manière compétente pour avoir un taux de disponibilité de 100% et une bonne résistance au spam).
Si j’utilise Twitter et G+, c’est pour gérer les relations publiques de Pitivi; si j’utilise Facebook, c’est supposément pour gérer mes relations privées… et il n’y a aucune raison pourquoi je ne pourrais pas gérer mes relations privées «dans le vrai monde» au lieu de confier ça à Zuckerberg. Et si je voulais gérer mes relations privées à travers un site web, il faudrait que ce soit une plateforme ouverte et décentralisée comme «DIASPORA*».
Facebook favorise l’invasion passive de la vie privée et le mélodrame (non seulement dans les messages qui s’y échangent, mais aussi dans la simple notion d’accepter ou refuser des «amis»). Les gens se sentaient insultés quand je ne les rajoute pas à mon réseau social, et on ne rentre même pas encore dans l’épineuse question des relations de travail ou des relations amoureuses. Toutes ces sphères de la vie d’une personne entrent alors de force en collision, qu’on le veuille ou non.
Non seulement le modèle d’affaires de Facebook est-il de creuser et revendre mes informations personnelles, mon «profil» socio-psychologique et mes habitudes de consommation (incluant qui sont mes amis et où j’ai grandi et étudié)… mais même si je ne fournis pas moi-même ces informations, par phénomène d’aggrégation, mes amis vont le faire pour moi, sans que je puisse faire quoi que ce soit pour empêcher le phénomène. Excepté boycotter Facebook entièrement (et encore).
Facebook détruit à petit feu plusieurs fondements de l’Internet et le retransforme en un AOL nouveau genre. Pourtant, je persiste à utiliser l’Internet comme une collection décentralisée de sites ouverts, à utiliser le téléphone, la messagerie instantanée et le courriel (d’ailleurs, je préfère nettement l’ergonomie d’un véritable logiciel de courriel comme Evolution ou Thunderbird à l’inefficacité d’une interface web, mais je suis certainement une espèce en voie de disparition sur ce point).
Parce que Facebook ne se limite pas à Facebook, et tente de me suivre absolument partout sur Internet (comme je le craignais en 2010). Vous savez, il y a une bonne raison pourquoi, même après deux ans, Facebook est encore entièrement bloqué au niveau de mon routeur (sauf quand un invité me fait la demande de le débloquer temporairement durant son séjour. Je ne suis pas un monstre). Vous n’avez pas idée de la quantité de scripts/frames cachés sont présents partout sur les sites que je visite (et on ne remarque pas leur présence avant de les avoir bloqués).
Les «applications» et autres gadgets du genre (s’ils existent encore sur Facebook) me font chier: ils sont «opt-out», insistants et «constamment dans ma face». Pire encore, ils constituent de sérieux risques de sécurité informatique et de vie privée (et des brèches de ce côté là, ça s’est vu).
Parce que Facebook est une tendance technologique qui peut aussi bien être remplacée (bien que, tristement, ce soit improbable). Il y a environ cinq ans, ça n’existait pas et on vivait très bien. Qui sait ce que sera Facebook en 2015, si Facebook existera encore, ou s’il sera remplacé par un autre dispositif orwellien à la mode?

Histoire de vous foutre un peu la chienne (comme on dirait en bon québécois): deux ans plus tard, je viens de faire le test et de recréer un compte, sous un faux nom, sans y entrer aucune information personnelle autre que mon adresse courriel, sans donner l’autorisation à Facebook de fouiller dans mon compte courriel et sans ajouter aucun «ami»:

… et Facebook se souvient encore de tous mes amis, me suggérant qui ajouter. Même si j’avais manuellement purgé mes données puis complètement supprimé mon compte (pas juste désactivé) et même si Facebook m’assurait du contraire, ils ont effectivement conservé toutes leurs données sur moi depuis deux ans.

Ce n’est pas tout: Facebook ne se souvient pas seulement de mes amis, collaborateurs et collègues d’antan… Facebook m’a même proposé d’ajouter quelqu’un que j’ai rencontré dans un café il y a deux semaines. Alors que je ne n’avais pas de compte Facebook. Et dire qu’il y en a qui trouvent que j’exagère…

J’ai ici uniquement énuméré quelques raisons pourquoi je refuse encore et toujours d’utiliser Facebook, même si ça me vaut la marginalisation… phénomène que je pourrais couvrir dans un prochain billet.

Switching PiTiVi’s main git repository

2012-03-15T18:03:45+00:00

Due to public demand, the official main repository for PiTiVi is now git.gnome.org/pitivi instead of git://git.pitivi.org/git/pitivi.git

This has many advantages:

It reduces confusion.
It prevents potential contributors from thinking we’re a dead project or something. We’re a killer project, that’s completely different.
No need to manage two “official” repositories (one for development and one for translations) anymore. No more strange merges to sync the repositories.
Those who want email notifications can subscribe through GNOME’s “commits-list”
We don’t have five times the word “git” when we do a clone of pitivi. You know, we were worried that people were not aware of the fact that they’re using git, turns out that’s not the case ;)
People might stop thinking that this is some kind of French conspiracy. Well, many contributors are francophones and we do have plans for world domination, but that’s entirely unrelated.

I updated all the relevant website and wiki pages accordingly. No more excuses now: ladies and gentlemen, start hacking (but make sure to do so on the “ges” branch).

What the heck is this GES thing anyway?

2012-03-14T02:56:32+00:00

“GES. You keep using that word. I do not think it means what you think it means.”

Unless you’ve been following the PiTiVi’s development very closely lately, it is often hard to imagine what GES does and how important it is to PiTiVi. I therefore wrote an explanatory page about it and updated the architecture overview on the wiki, in the hope of clearing things up. Let me know if something’s missing or if you have suggestions to enhance those pages.

Also, a quick status update on my call for donations experiment: we now have a total of $121.50 CAD. This is not enough for a hackfest, but certainly enough to pay a couple of beers to PiTiVi contributors at GUADEC.

I would like to extend my sincere thanks to the following people who kindly donated some money through Paypal: Felipe Almeida Lessa, Frederik Elwert, Garrett Gandy, Joseph Tennies, Leon Handreke, Pierre Bornancin, Stéphane Maniaci. I’d also like to thank the dozen of people who “flattr’d” us (but Flattr doesn’t tell me the full list of names).

the merry month of ma

2012-03-12T20:27:53+00:00

or, from the department of self-inflicted injuries

Recently I saw a bunch of errors in my server logs. People were asking for pages on my web site, but only if they were newer than Thu, 08 Ma 2012 22:44:59 GMT. "Ma"? What kind of a month is that? The internets have so many crazy things.

On further investigation, it seemed this was just a case of garbage in, garbage out; my intertube was busted. I was the one returning a Last-Modified with that date. It was invalid, but client software sent it back with the conditional request.

Thinking more on this, though, and on the well-known last-modified hack in which that field can be used as an unblockable cookie, I think I have to share some blame with the clients again.

So, clients using at least Apple-PubSub/65.28, BottomFeeder/4.1, NetNewsWire, SimplePie, Vienna, and Windows-RSS-Platform/2.0 should ask the people that implement their RSS software to only pass a Last-Modified date if it's really a valid date. Implementors of the NetVibes and worio.com bots should also take a look at their HTTP stacks. I don't guess that there's much that you can do with an etag though, for better or for worse.

Previously, in a related department.

Prévisualisation en temps réel du redimensionnement de séquences dans Pitivi

2012-03-10T03:10:00+00:00

Ceci est une traduction d’un billet que j’ai publié avant-hier sur Planet GNOME. Je ne sais pas trop comment traduire « trimming », alors je me base sur le terme « redimensionnement » de la version française du manuel de PiTiVi (même si l’usage du mot sonne étrange). La question du « financement de logiciels libres complexes n’ayant pas de modèle d’affaire clairement établi » est quelque chose qui m’a toujours intrigué, vous trouverez donc quelque chose à ce sujet en deuxième partie de ce billet.

Dans mon temps libre cette semaine, j’ai implanté une fonctionnalité manquante de PiTiVi qui me tient énormément à coeur: la prévisualisation en temps réel lorsque l’on change le début ou la fin d’une séquence. Ceci permet enfin de faire de l’édition avec une précision inégalée, sans essais-erreurs. De la productivité à l’état pur, comme la vidéo ci-dessous le démontre:

Cette implémentation est assez performante pour prévisualiser de manière fluide la version 2K de Sintel et pour rester raisonnablement rapide même avec des vidéos en résolution 4K. Ceci est sans même prendre en compte l’accélération matérielle du décodage vidéo, qui sera disponible dans GStreamer 1.0 (j’ai bien hâte de tester lorsque ce sera possible).

La fonctionnalité dont je vous parle ici, je l’avais en fait demandée à travers un rapport de bug il y a trois ans (dans la période où j’avais ouvert plus d’une centaine de rapports de bugs sur PiTiVi en l’espace de trois mois). Pour ceux parmi vous qui doutaient de mon génie avant-gardiste, vous en avez maintenant la preuve 😃

Je suis certain que les autres logiciels de montage vidéo libre vont se précipiter pour ripper mon élégantissime code et cloner cette fonctionnalité en l’espace de quelques heures. Retenez toutefois ceci: PiTiVi l’a fait le premier. PiTiVi est le seul logiciel de montage vidéo libre qui possède cette fonctionnalité. Prenez ça dans vos dents, détracteurs de PiTiVi !

Essayons maintenant une petite expérience sociologique. J’ai toujours été curieux de savoir s’il pourrait y avoir une demande significative pour « un PiTiVi solide comme le roc », un logiciel de montage vidéo libre à la fois intuitif et répondant aux besoins des usagers plus exigeants. Eh bien, vous avez maintenant l’opportunité d’effectuer une donation au projet.

« QUOI? Sale cochon capitaliste! »

Ne vous méprenez pas: je ne suis pas en train de mendier ou de chercher à faire un tour de passe-passe financier. J’ai un emploi bien rémunéré et j’ai toujours contribué à PiTiVi par plaisir, par passion et sans motif financier. Ce que je désire faire ici, c’est de jauger la possibilité de soutenir le projet PiTiVi à un point où il pourrait être développé plus rapidement, plus régulièrement et aboutir vers un logiciel à toute épreuve:

Si nous obtenons 100€, on pourrait payer quelques bières aux contributeurs PiTiVi à GUADEC cet été.
Si on obtient 10 000€ par année, ça devient un peu plus sérieux: on pourrait organiser un ou deux hackfests par année, comme celui-ci et celui-là. Avec ce montant on pourrait payer tout juste les frais de transport des développeurs.
Si on obtient 100 000 à 500 000€ par année (ouais, c’est beau de rêver, sait-on jamais!)… alors les choses changeraient de manière drastique. Ça voudrait dire qu’on pourrait payer des experts de GStreamer pour travailler à plein temps sur PiTiVi pour en faire le logiciel de montage vidéo professionnel que nous attendions depuis si longtemps. Ou bien on pourrait embaucher des ninjas pour infiltrer le quartier général de Avid, ou un truc du genre. Je ne dis pas ici que, pour le cas de cette expérimentation, il nous faut des centaines de milliers d’euros/dollars: avec un montant significativement élevé, je pourrais finalement faire la preuve de mon plan de bataille (ou plan de développement « durable », ou « plan d’affaires », appelez ça comme vous voulez), et lancer les choses en grand. Cela pourrait surprendre certains d’entre vous, mais je ne suis pas juste un de ces passionnés irréductibles du logiciel libre, je possède également des connaissances en création et gestion d’entreprise.

On ne pourra jamais savoir avant d’avoir essayé, n’est-ce pas? Alors, maintenant, tentons l’expérience. Vous pouvez faire votre déclaration d’amour à travers Paypal ou Flattr. Si vous pensez que je rigole, allez voir sur le site de PiTiVi et vous y trouverez lesdits boutons Flattr et Paypal (au bas de la page d’accueil, ou bien sur la page « Contributing »). Voici le même bouton Flattr, s’il y en a parmi vous qui êtes tellement excités qu’il vous faut absolument un truc sur lequel cliquer immédiatement:

Je vais observer et étudier ce qui se passe à présent. Soyez assuré que je n’utiliserai pas ces donations à titre personnel. Tout ira vers le projet PiTiVi, il me faudra seulement résoudre les questions techniques/administrative de la chose. J’ai discuté depuis plusieurs mois avec la fondation GNOME, et je voudrais pouvoir évaluer si la quantité de donations à laquelle nous faisons face pourrait justifier les complications qu’une telle structure impliquerait.

Soit dit en passant, j’ai d’autres améliorations/fonctionnalités dans mon sac à surprises, mais elles ne sont pas encore prètes. Restez à l’écoute (sur Planet PiTiVi ou Planet GNOME)! En attendant, n’hésitez pas à partager avec le reste du monde la grande étape que nous avons franchie pour l’édition efficace de séquences. Et si vous avez envie de faire l’écho de mon appel aux donations, vous pouvez faire le lien vers mon billet de blog ou utiliser ce lien ou encore ce lien (page « Contributing ») pour me faciliter le suivi.

an in-depth look at the performance of guile's web server

2012-03-08T22:44:59+00:00

What ho, ladies! And what ho, gentlemen! The hack is on and apace. Today, the topic is performance: of Guile and of its web server, in microseconds and kiloinstructions. Brew up a pot of tea; this is a long article.

the problem

I have been poking at Guile's web server recently. To recap, Guile is an implementation of Scheme. It is byte-compiled, and has a set of runtime libraries written in C and, increasingly, in Scheme itself.

Guile includes some modules for dealing with the web: HTTP, clients, servers, and such. It's all written in Scheme. It runs this blog you're reading right now.

To be precise, this web log is served by tekuti, an application written on top of Guile's web server; and actually, in this case there is Apache in front of everything right now, so you're not talking directly to Guile. Perhaps that will change at some point.

But anyway, a few months ago, this blog was really slow to access. That turned out to be mostly due to a bug in tekuti, the blog application, in which generating the "related links" for a post would always end up invoking git. (The database for the blog is implemented in git.) Spawning another process was slow. Fellow hacker and tekuti user Aleix Conchillo Flaqué fixed the problem a year ago, but it took a while for it me to finally review and merge it.

So then, at that point, things were tolerable, but I had already contracted the performance bug, so I went on to spend a couple months drilling down, optimizing Guile and its web server -- the layers below tekuti.

10K requests per second: achievement unlocked!

After a lot of tweaking, to the compiler, runtime, HTTP libraries, and to the VM, Guile can now serve over 10000 requests per second on a simple "Hello world" benchmark. This is out of the box, so to speak, if the master branch in git were a box. You just check out Guile, build it using the normal autoreconf -vif & & ./configure && make dance, and then run the example uninstalled:

$ meta/guile examples/web/hello.scm

In another terminal, you can connect directly to the port to see what it does. Paste the first paragraph of this and press return, and you should see the second part:

$ nc localhost 8080
GET / HTTP/1.0
Host: localhost:8080
User-Agent: ApacheBench/2.3
Accept: */*

HTTP/1.0 200 OK
Content-Length: 13
Content-Type: text/plain;charset=utf-8

Hello, World!

I should say a little more about what the server does, and what the test application is. It reads the request, and parses all the headers to native Scheme data types. This is strictly more work than is needed for a simple "pong" benchmark, but it's really useful for applications to have all of the headers parsed for you already. It also filters out bad requests.

Guile then passes the request and request body to a handler, which returns the response and body. This example's handler is very simple:

(define (handler request body)
  (values '((content-type . (text/plain)))
          "Hello, World!"))

It uses a shorthand, that instead of building a proper response object, it just returns an association-list of headers along with the body as a string, and relies on the web server's sanitize-response to produce a response object and encode the body as a bytevector. Again, this is more work for the server, but it's a nice convenience.

Finally, the server writes the response and body to the client, and either closes the port, as in this case for HTTP/1.0 with no keep-alive, or moves the client back to the poll set if we have a persistent connection. The reads and writes are synchronous (blocking), and the web server runs in one thread. I'll discuss this a bit more later.

You can then use ApacheBench to test it out:

$ ab -n 100000 -c100 http://localhost:8080/
Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        13 bytes

Concurrency Level:      100
Time taken for tests:   9.631 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      9200000 bytes
HTML transferred:       1300000 bytes
Requests per second:    10383.03 [#/sec] (mean)
Time per request:       9.631 [ms] (mean)
Time per request:       0.096 [ms] (mean, across all concurrent requests)
Transfer rate:          932.85 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       3
Processing:     1   10   1.7      9      20
Waiting:        1   10   1.7      9      20
Total:          3   10   1.7      9      20

Percentage of the requests served within a certain time (ms)
  50%      9
  66%      9
  75%     12
  80%     12
  90%     12
  95%     12
  98%     13
  99%     13
 100%     20 (longest request)

I give you the whole printout because I find it interesting. There's no huge GC pause anywhere. This laptop has an Intel i7-2620M 2.70GHz CPU. It was bought last year, so it's recent but not fancy. There are two cores, and so part of one core is being used by ApacheBench, and the whole other one used by Guile.

Of course, this is a flawed benchmark. If you really care about this sort of thing, Mark Nottingham wrote a nice article on HTTP benchmarking last year that shows all the ways in which this test is wrong.

However, flawed though it is, this test serves my purposes: to understand the overhead that Guile and its web server imposes on a web application.

So in that light, I'd like to take apart this test and try to understand its performance. I'll look at it from three directions: bottom-up (using low-level profiling), top-down (using a Scheme profiler), and transverse (profiling the garbage collector).

bottom-up: clock cycles, instructions retired

The best way to measure the performance of an application is with a statistical profiler. Statistical profiling samples what's really happening, without perturbing the performance characteristics of your application.

On GNU/Linux systems, we have a few options. None are really easy to use, however. Perf comes the closest. You just run perf record -g guile examples/web/hello.scm, and it records its information. Then you run perf report and dive into the details through the mostly pleasant curses application.

However, for some reason perf could not capture the call graphs, in my tests. My machine is x86-64, which does not include frame pointers in its call frames, so perhaps it was a naive stack-walking algorithm. The associated DWARF information does include the necessary stack-walking data.

Anyway, at least you can get a good handle on what individual functions are hot, and indeed what source lines and instructions take the most time. These lowest-level statistical profilers typically sample based on number of clock cycles, so they correspond to real time very well.

So much for measurement. For actually understanding and improving performance, I find valgrind's callgrind tool much more useful. You run it like this:

valgrind --tool=callgrind --num-callers=20 /path/to/guile examples/web/http.scm

Valgrind can record the execution of your program as it runs, tracking every call and every instruction that is executed. You can then explore the resulting log with the kcachegrind graphical tool. It's the best thing there is for exploring low-level execution of your program.

Now, valgrind is not a statistical profiler. It slows down and distorts the execution of your program, as you would imagine. Beyond those caveats, though, you have to keep two things in mind: firstly, that a count of instructions executed does not correspond directly to clock cycles spent. There are memory latencies, cache misses, branch mispredictions, and a whole host of randomnesses that can affect your runtime. Secondly, valgrind only records the user-space behavior of your program. I'll have more to say on that later.

If I run the "hello.scm" benchmark under valgrind, and hit it with a hundred thousand connections, I can get a pretty good idea what the aggregate behavior of my program is. But I can do better than big-picture, for this kind of test. Given that this test does the same thing a hundred thousand times, I can use valgrind's accurate call and instruction counts to give me precise information on how much a single request takes.

So, looking at the total instruction and call counts, and dividing by the number of requests (100K), I see that when Guile handles one request, the algorithmic breakdown is as follows:

instructions per request	percentage of total	procedure	calls per request
250K	100	(total)	-
134K	52.5	bytecode interpretation	-
47K	18.5	port buffer allocation	1
18K	7.2	`display`	1
7.6K	3	allocation within VM (closures, pairs, etc)	83
7.2K	2.84	`read-line`	5
3.6K	1.40	`substring-downcase`	3
3.2K	1.24	`string->symbol`	4
3.1K	1.23	`accept`	1
2.7K	1.06	`substring`	11
2.6K	1.03	`string-index`	8
2.5K	1.00	`hashq-ref`	14
23K	9	other primitives < 1% each	-

Guile's ports can be buffered, like C's stdio FILE* streams, and the web server does turn on buffering. The 18.5% of the time spent allocating port buffers seems like a ripe place for optimization, but digging into it, almost all of the time within those routines is spent in the garbage collector, and most of that marking the heap. Switching to a generational collector could help here, but I'm not sure how much, given that port buffers are probably 4 KB each for input and output, and thus might not fit into a young generation. Marking from more threads at once could help -- more on that in some future essay.

There are some primitives that can be optimized as well, but with the VM taking up 52% of the runtime, and 23% for allocation and the garbage collector, Amdahl's law is against us: making the primitives twice as fast would result only a 15% improvement in throughput.

Turning Amdahl's argument around, we can predict the effect of native compilation on throughput. If Guile 2.2 comes out with native compilation, as it might, and that makes Scheme code run 5 times faster (say), then the 50% of the instructions that are currently in the VM might drop to 10% -- leading to an expected throughput improvement of 67%.

top-down: scheme-level profiling with statprof

But what is going on in the VM? For that, I need a Scheme profiler. Fortunately, Guile comes with one, accessible at the REPL:

$ guile --no-debug
> ,profile (load "examples/web/hello.scm")
%     cumulative   self             
time   seconds     seconds      name
 15.42      1.55      1.55  close-port
 11.61      1.17      1.17  %after-gc-thunk
  6.20      1.65      0.62  setvbuf
  4.29      0.43      0.43  display
  3.34      0.35      0.34  accept
  2.86      0.35      0.29  call-with-error-handling
  2.38      0.24      0.24  hashq-ref
  2.23      9.16      0.22  with-default-trap-handler
  2.07      0.67      0.21  build-response
  1.75      1.17      0.18  sanitize-response
...
Sample count: 629
Total time: 10.055736686 seconds (1.123272351 seconds in GC)

(I use the --no-debug argument to avoid some per-VM-instruction overhead imposed by running Guile interactively; see VM Hooks for more.)

Here I get a really strange result. How is close-port taking all the time? It's implemented as a primitive, not in Scheme, and Valgrind only thought it took 0.40% of the instructions. How is that?

To answer this, we need to remember a couple things. First of all, we recall that Guile's statistical profiler uses setitimer to get signals delivered periodically, after some amount of time spent on the program's behalf, including time spent in the kernel. Valgrind doesn't account for system time. So here we are seeing that close-port is indeed taking time, specifically to flush out the buffered writes. The time is really being spent in the write system call.

So this is good! We know now that we should perhaps look at tuning the kernel to buffer our writes better.

We can also use the profiling data counts to break down the time spent in serving one request from a high level. For example, http-read handles traversing the poll set and accepting connections, and tail-calls read-request to actually read the request. Looking at the cumulative times in the chart tells me that out of each request, the time spent breaks down like this:

microseconds per request	procedure
100	(total)
29	poll and accept
16	reading request
0	request handler
12	sanitize-response
8	writing response headers
0	writing response body
15	close-port
20	other

Of course, since this is statistical, there is some uncertainty about the whole thing. Still, it seems sensible enough.

transverse: who is doing all the allocating?

Often when you go to optimize a Scheme program, you find that it's spending a fair amount of time in garbage collection. How do you optimize that? The rookie answer to this is to try to patch the collector to allocate faster, or less frequently, or something. Veterans know that the solution to GC woes is usually just to allocate less. But how do you know what is allocating? GC is a fairly transverse cost, in that it can charge the good parts of your program for the expenses of the bad.

Guile uses the Boehm-Demers-Weiser conservative collector. For what it is, it's pretty good. However its stock configuration does not provide very much insight into the allocation patterns of your program. One approximation that can be made, though, is that the parts of the program that cause garbage collection to run are the parts that are allocating the most.

Based on this insight, Guile includes a statistical profiler that samples when the garbage collector runs. One thing to consider is that GC probably doesn't run very often, so for gcprof tests, one might need to run the test for longer. In this example, I increased the load to a million requests.

$ guile --no-debug
> (use-modules (statprof))
> (gcprof (lambda () (load "examples/web/hello.scm")))
%     cumulative   self             
time   seconds     seconds      name
 86.55     82.75     82.75  setvbuf
  9.37      8.96      8.96  accept
  1.70      1.63      1.63  call-with-error-handling
  1.31      1.25      1.25  %read-line
  0.29      0.28      0.28  substring
  0.24      0.23      0.23  string-downcase
  0.10     91.80      0.09  http-read
  0.10      0.09      0.09  parse-param-list
  0.10      0.09      0.09  write-response
---
Sample count: 2059
Total time: 95.611532937 seconds (9.371730448 seconds in GC)

Here we confirm the result that we saw in the low-level profile: that the setvbuf Scheme procedure, which can cause Guile to allocate buffers for ports, is the primary allocator in this test.

Another interesting question to answer is, "how much are we allocating, anyway?" Using the statistics REPL command, I can see that the 100K requests entailed a total allocation of about 1.35 GB, which divides out to 13.5 KB per request. That sounds reasonable: about 4 KB each for the read and write buffers, and some 4 KB of various other allocation: strings, pairs, the final bytevector for output, etc.

The test incurred 220 stop-the-world garbage collections. So, about 1 out of 450 (0.2%) of requests trigger a GC. The average GC time seems to be about 5 ms (1100 ms / 220 times). That squares fairly well with our last-percentile ApacheBench results.

The total heap size is modest: 14 megabytes. It does not leak memory, thankfully. If I run mem_usage.py on it, I get:

Mapped memory:
               Shared            Private
           Clean    Dirty    Clean    Dirty
    r-xp    1236        0     1028        0  -- Code
    rw-p      24        0       28      160  -- Data
    r--p      60        0     1580      112  -- Read-only data
    ---p       0        0        0        0
    r--s      24        0        0        0
   total    1344        0     2636      272
Anonymous memory:
               Shared            Private
           Clean    Dirty    Clean    Dirty
    r-xp       4        0        0        0
    rw-p       0        0        0    11652  -- Data (malloc, mmap)
    ---p       0        0        0        0
   total       4        0        0    11652
   ----------------------------------------
   total    1348        0     2636    11924

Native ahead-of-time compilation would allow for more shareable, read-only memory. Still, as it is, this seems acceptable.

a quick look at more dynamic tests

That's all well and good, you say, but it's a fairly static test, right?

For that I have a couple of data points. One is the simple SXML debugging test in examples/test/debug-sxml.scm, which simply spits back the headers that it receives, wrapped in an HTML table. The values are printed as the Scheme object that they parse to. Currently, I have a version of this script running on my site. (You can see the headers that Apache adds on, there.)

Testing it as before tells me that Guile can serve 6000 of these pages per second, on my laptop. That's pretty respectable for an entirely dynamic page that hasn't been optimized at all. You can search around the net for comparable tests in other languages; I think you'll find Guile's performance to be very good.

The other point to mention is Tekuti itself. Tekuti includes a caching layer in the application, so after the first request, it's not really a dynamic test. It does check to make sure its caches are fresh on every round, though, by checking the value of the refs/heads/master ref. But still, it is a test of pushing a lot of data; for my test, the page is 50 KB, and Guile still reaches 5700 requests per second on this one core, serving 280 megabytes per second of... well, of my blather, really. But it's a powerful blather-pipe!

related & future work

Here's a similarly flawed test from a year ago of static servers, serving small static files over localhost. Flawed, but it's a lot like this "hello world" test in semantics. We see that nginx gets up to about 20K requests per second, per core. It also does so with a flat memory profile, which is nice.

Guile's bundled web server is currently single-threaded and blocking, which does not make it a good frontend server. There's room for a project to build a proper web server on top of Guile, I think, but I probably won't do it myself. In the meantime though, I do want to offer the possibility for the built-in web server to be multithreaded, with some number of I/O threads and some more limited number of compute threads. I've been testing out some code in that direction -- in fact, this server is running that code -- but as yet Guile's synchronization primitives have too much overhead for it to be a real win. There's more runtime and compiler work to do here.

As far as web servers written in safe languages, it would be remiss to not mention Warp, a Haskell web server. Again, their tests effectively utilize multiple cores, but it seems that 20K per core is the standard.

Unlike Haskell, Guile lacks a proper event manager. I'm not sure whether to work on such a thing; Havoc seems to think it's necessary, and who am I to argue.

Finally, I mention a benchmark of python WSGI servers from a couple years ago. (Is March the month of benchmarks?) The python performance is notably worse; hopefully PyPy has improved things in the meantime. On the other hand, GEvent's use of greenlets is really nice, and makes me envious.

conclusions

Hey, you read the thing! Congratulations to you! Good thing you didn't just skip down to the end :)

If I have a message to send, it's this: that you should consider using Guile to be perfectly acceptable for implementing dynamic web applications with high performance requirements.

It's a modest point, I know. There are all kinds of trade-offs here, but hey, Guile is plucky and still a little bit shy, but would love it if you to ask it to the hack.

If it works for you, boast about it to your colleagues! And if it doesn't, let us know, over at the usual places (guile-user@gnu.org, and #guile on freenode). Happy hacking!

Trim like a professional hair stylist

2012-03-08T04:59:21+00:00

In my spare time this week I implemented one of my favorite missing features in PiTiVi: a live preview of what you’re trimming. No more fuzzying around and moving the playhead all the time to figure out if you cut your scene right. No more trial and error. Just pure, unadulterated productivity. HTML5 video below:

It is fast enough to seek through the 2K version of Sintel like a breeze and to stay reasonably responsive with 4K videos. But that’s without hardware acceleration, mind you (I’m looking forward to trying this again in GStreamer 1.0…).

I actually filed a bug about this feature three years ago (you know, when I filed a little over a hundred bug reports within the space of three months). You hadn’t noticed how I am a visionary way ahead of my time? Well, now you’ve got proof.

I’m sure that all the other open source video editors will rip my beautiful code and clone this feature within 12 hours. But mark my words: PiTiVi had it first. Yes: PiTiVi is the only open source video editor that has this feature. Take that, haters.

When nekohayo starts touching your gst pipelines, you should be very afraid.

Let’s try a small social experiment here. I’ve always been curious to see if there was a significant demand for a “serious-business-I-kid-you-not pitivi”. So now you have an opportunity to donate some money.

“WHAT? You capitalist pig!”

Don’t get me wrong: I’m not begging for money here. I have a job and I’ve always contributed to Pitivi without any financial motive. What I want to do here is to gauge the possibility of sustaining Pitivi to the point where it would be developed faster and be much more robust:

If I get 10$, I might be able to afford a sandwich.
If we get 100$, we could pay a couple of beers for contributors at GUADEC.
If we get 10 000$ per year, now we’re starting to get serious enough to be able to organize one or two PiTiVi hackfests per year (still, that’s just nearly enough to pay a port of travel fees for developers for a hackfest).
If we get 100 000 to 500 000$ a year (yeah, right!)… then things would change very drastically around here. Like actually contracting GStreamer hackers to make Pitivi the professional-grade open-source video editing solution on Linux. Or hiring ninjas to infiltrate Avid’s headquarters, that sort of thing. I’m not saying we need hundreds of thousands of dollars either: with a significant-enough amount, I could finally prove my case to some select folks I know, and get the ball rolling. It might come as a surprise, but I don’t just have a crazy passion for Free Software, I also have some nice business skills.

You never know until you try, right? So, for now, let’s do this initial experiment. You can send some love through Paypal or with Flattr. I placed the Flattr and Paypal buttons on the pitivi.org website if you think I’m joking (they are located at the bottom of the home page, or in the “Contributing” page). And here’s the same Flattr button if you’re so excited that you need a target to click right now:

I’ll just watch and study what happens. By the way, I won’t be using the money for myself as an individual. It will go towards the project, I just need to figure out the technicalities of doing so. I’ve been talking with the GNOME foundation in the past few months, and I’d like to know if the amount of donations we’re facing would warrant the additional overhead.

Oh, and I have some more features in the cooker, but they’re not ready yet. Stay tuned. In the meantime, feel free to let the world know about the important milestone we’ve reached today. And if you want to spread the word about donations, you may use this URL or this URL (“contributing” page) when linking to the pitivi website, to help me keep track of what’s going on.

Le SSD du pauvre

2012-03-08T03:33:34+00:00

Un de mes amis possède encore son netbook Dell Mini 9. Or, il y a un an ou deux, le SSD bon marché à l’intérieur a rendu l’âme (même le BIOS voit le disque dur comme étant de capacité 0 MB).

J’avais entretenu l’idée de ressusciter cet ordinateur pour un usage de serveur ou «media center» ou autre chose en live USB, mais ce n’était qu’une idée. Récemment, l’ami en question a décidé de tenter la chose et s’est procuré une clé USB, amenant le coût total de la réincarnation à environ 10 dollars.

J’ai d’abord tenté de faire fonctionner le tout simplement en créant un live USB Fedora 16 «persistant», malgré les limitations inhérentes de cette approche, comme le fait que l’espace ne se libère jamais même si on supprime les fichiers:

One very important note about using the “primary” persistent overlay for system changes is that due to the way it’s currently implemented (as a LVM copy-on-write snapshot), every single change to it (writes AND deletes) subtracts from its free space, so it will eventually be “used up” and your USB stick will no longer boot

… puis j’ai rencontré une embûche qui avait échappé à ma planification: la carte sans fil était une Broadcom 4312 (donc nécessitant un module kernel non-libre).

Utilisant mon téléphone android (acheté d’occasion l’été dernier) pour partager la connexion WiFi par USB, je tentai néanmoins d’installer «akmod-wl» de rpmfusion (c’est le paquet pour faire marcher les chips Broadcom sous Fedora. Oubliez les kmod-wl). Cependant, une fausse manipulation mit l’ordinateur en veille, et il ne voulait plus se réveiller correctement par la suite. Comme le barbare que je suis, j’ai croisé les doigts, éteint de force et redémarré le système… pour constater que le système de fichiers de la clé USB était corrompu.

Après quelques minutes de réflexion pondérées de «FFFUUUU», j’ai retourné le problème dans un sens moins conventionnel. Le «live USB» n’est pas la solution. «Installer vers la clé USB» est la solution.

Il nous faut donc utiliser deux clés USB:

L’une pour booter et lancer l’installation
L’autre (une clé 16 GB vierge, sans aucune partition dessus) qui sert comme cible d’installation. Il a fallu la partitionner comme suit: une partition de type «BIOS BOOT» de 2 MB, une partition swap de 50 MB (juste parce que Fedora insiste pour me faire croire qu’il faut plus d’un gigaoctet de mémoire pour installer… donc à désactiver post-installation) et le reste en ext4.

… et ça marche. Après l’installation, j’ai effectué quelques hacks pour améliorer la performance et augmenter un peu (en théorie) la durée de vie de la clé USB:

Modifier /etc/fstab pour désactiver le swap et ajouter l’option «noatime» à la partition /
Désactiver presto dans /etc/yum/pluginconf.d/presto.conf
Désactiver le firewall et selinux
Supprimer abrt

Après avoir fait fonctionner la carte sans-fil Broadcom, il ne restait qu’à appliquer les ~400 mises à jour, rajouter LibreOffice et les plugins GStreamer. Heureusement que j’avais désactivé presto et fait effectuer cette opération durant mon sommeil: le tout a nécessité un peu plus de six heures (processeur Atom N270 et lenteur de clé USB obligent)!

L’aspect intéressant de tout ça est que nous sommes maintenant en présence d’un véritable système d’opération USB portable, sans les limitations d’un live USB persistant conventionnel. Peu importe dans quel ordinateur on le branche. C’est quand même pas mal moins cher qu’un «Cotton Candy» quand on y pense.

Those darn strings

2012-03-06T21:32:33+00:00

It's long overdue, but I finally added the promised section about "How to deal with strings" to the Python GTK+3 Tutorial.

It all started with bug 663610 which got me confused because with PyGObject for Python 2.x you can supply a unicode instance to a method that expects a string, although GTK+ can only deal with UTF-8 encoded 8-bit strings. However, every GTK+ method that returns a string will always be in the form of a str instance. This is not necessarily PyGObject's fault, it is exceptionally easy to do bad things in Python 2.x when mixing unicode and str (hello UnicodeDecodeError). The lesson hear is that if you use PyGObject for Python 2.x, don't use unicode and stick to str all the time. If you instead use Python 3.x, you don't have to worry about this. A more detailed description and a little bit of background can be found in the "How to deal with strings" section of the tutorial.

I also want to thank everybody who helped to improve the tutorial as part of Google Code-In 2011. Students wrote sections about Drag and Drop, sub-classing GObject, Gtk.Builder, Gtk.IconView, Gtk.Clipboard, and Gtk.TreeSortable. They all did an outstanding job in improving the tutorial to enable others to learn GTK+ 3 in Python.

More adventures in puppet

2012-03-04T22:32:31+00:00

After last week’s Linode incident I was getting a bit more worried about security than usual. That coincided with the fact that I found I couldn’t run puppet on one of my linodes, and some digging turned up that it was because /tmp was owned by uid:gid 1000:1000. Since I didn’t know the details of the breakin (and I hadn’t slept more than 4 hours for two nights, one of which involving a Flumotion DVB problem), I had no choice but to be paranoid about it. And it took me a good half hour to realize that I had inflicted this problem on myself – a botched rsync command (rsync arv . root@somehost:/tmp).

So I wasn’t hacked, but I still felt I needed to tighten security a bit. So I thought I’d go with something simple to deploy using puppet – port knocking.

Now, that would be pretty easy to do if I just deployed firewall rules in a single set. But I started deploying firewall rules using the puppetlabs firewall module, which allows me to group rules per service. So that’s the direction I wanted to head off into.

On saturday, I worked on remembering enough iptables to actually understand how port knocking works in a firewall. Among other things, I realized that our current port knocking is not ideal – it uses only two ports. They’re in descending order, so usually they would not be triggered by a normal port scan, but they would be triggered by one in reverse order. That is probably why most sources recommend using three ports, where the third port is between the first two, so they’re out of order.

So I wanted to start by getting the rules right, and understanding them. I started with this post, and found a few problems in it that I managed to work out. The fixed version is this:
UPLINK="p21p1" # # Comma seperated list of ports to protect with no spaces. SERVICES="22,3306" # # Location of iptables command IPTABLES='/sbin/iptables'

# in stage1, connects on 3456 get added to knock2 list
${IPTABLES} -N stage1
${IPTABLES} -A stage1 -m recent --remove --name knock
${IPTABLES} -A stage1 -p tcp --dport 3456 -m recent --set --name knock2

# in stage2, connects on 2345 get added to heaven list
${IPTABLES} -N stage2
${IPTABLES} -A stage2 -m recent --remove --name knock2
${IPTABLES} -A stage2 -p tcp --dport 2345 -m recent --set --name heaven

# at the door:
# - jump to stage2 with a shot at heaven if you're on list knock2
# - jump to stage1 with a shot at knock2 if you're on list knock
# - get on knock list if connecting t0 1234
${IPTABLES} -N door
${IPTABLES} -A door -m recent --rcheck --seconds 5 --name knock2 -j stage2
${IPTABLES} -A door -m recent --rcheck --seconds 5 --name knock -j stage1
${IPTABLES} -A door -p tcp --dport 1234 -m recent --set --name knock

${IPTABLES} -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
${IPTABLES} -A INPUT -p tcp --match multiport --dport ${SERVICES} -i ${UPLINK} -m recent --rcheck --seconds 5 --name heaven -j ACCEPT
${IPTABLES} -A INPUT -p tcp --syn -j door

# close everything else
${IPTABLES} -A INPUT -j REJECT --reject-with icmp-port-unreachable

And it gives me this iptables state:

So the next step was to reproduce these rules using puppet firewall rules.

Immediately I ran into the first problem – we need to add new chains, and there doesn’t seem to be a way to do that in the firewall resource. At the same time, it uses the recent iptables module, and none of that is implemented either. I spent a bunch of hours trying to add this, but since I don’t really know Ruby and I’ve only started using Puppet for real in the last two weeks, that wasn’t working out well. So then I thought, why not look in the bug tracker and see if anyone else tried to do this ? I ask my chains question on IRC, while I find a ticket about recent support. A minute later danblack replies on IRC with a link to a branch that supports creating chains – the same person that made the recent branch.

This must be a sign – the same person helping me with my problem in two different ways, with two branches? Today will be a git-merging to-the-death hacking session, fueled by the leftovers of yesterday’s mexicaganza leftovers.

I start with the branch that lets you create chains, which works well enough, bar some documentation issues. I create a new branch and merge this one on, ending up in a clean rebase.

Next is the recent branch. I merge that one on. I choose to merge in this case, because I hope it will be easier to make the fixes needed in both branches, but still pull everything together on my portknock branch, and merge in updates every time.

This branch has more issues – rake test doesn’t even pass. So I start digging through the failing testcases, adding print debugs and learning just enough ruby to be dangerous.

I slowly get better at fixing bugs. I create minimal .pp files in my /etc/puppet/manifests so I can test just one rule with e.g. puppet apply manifests/recent.pp

The firewall module hinges around being able to convert a rule to a hash as expressed in puppet, and back again, so that puppet can know that a rule is already present and does not need to be executed. I add a conversion unit test for each of the features that tests these basic operations, but I end up actually fixing the bugs by sprinkling print’s and testing with a single apply.

I learn to do service iptables restart; service iptables stop to reset my firewall and start cleanly. It takes me a while to realize when I botched the firewall so that I can’t even google (in my case, forgetting to have -A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT
) – not helped by the fact that for the last two weeks the network on my home desktop is really flaky, and simply stops working after some activity, forcing me to restart NetworkManager and reload network modules.

I start getting an intuition for how puppet’s basic resource model works. For example, if a second puppet run produces output, something’s wrong. I end up fixing lots of parsing bugs because of that – once I notice that a run tells me something like
notice: /Firewall[999 drop all other requests]/chain: chain changed '-p' to 'INPUT' notice: Firewall[999 drop all other requests](provider=iptables): Properties changed - updating rule
I know that, even though the result seems to work, I have some parsing bug, and I can attack that bug by adding another unit test and adding more prints for a simple rule.

I learn that, even though the run may seem clean, if the module didn’t figure out that it already had a rule (again, because of bogus parsing), it just adds the same rule again – another thing we don’t want. That gets fixed on a few branches too.

And then I get to the point where my puppet apply brings all the rules together – except it still does not work. And I notice one little missing rule: ${IPTABLES} -A INPUT -p tcp –syn -j door

And I learn about –syn, and –tcp-flags, and to my dismay, there is no support for tcp-flags anywhere. There is a ticket for TCP flags matching support, but nobody worked on it.

So I think, how hard can it be, with everything I’ve learned today? And I get onto it. It turns out it’s harder than expected. Before today, all firewall resource properties swallowed exactly one argument – for example, -p (proto). In the recent module, some properties are flags, and don’t have an argument, so I had to support that with some hacks.

The rule_to_hash function works by taking an iptables rule line, and stripping off the parameters from the back in reverse order one by one, but leaving the arguments there. At the end, it has a list of keys it saw, and hopefully, a string of arguments that match the keys, but in reverse order. (I would have done this by stripping the line of both parameter and argument(s) and putting those on a list, but that’s just me)

But the –tcp-flags parameter takes two arguments – a mask of flags, and a list of flags that needs to be set. So I hack it in by adding double quotes around it, so it looks the same way a –comment does (except –comment is always quoted in iptables –list-rules output), and handle it specially. But after some fidgeting, that works too!

And my final screenshot for the day:

So, today’s result:

Now, I have a working node that implements port knocking:
node 'ana' {

$port1 = '1234'
$port2 = '3456'
$port3 = '2345'

$dports = [22, 3306]

$seconds = 5

firewall { "000 accept all icmp requests":
proto => "icmp",
action => "accept",
}

firewall { "001 accept all established connections":
proto => "all",
state => ["RELATED", "ESTABLISHED"],
action => "accept",
}

firewall { "999 drop all other requests":
chain => "INPUT",
proto => "tcp",
action => "reject",
}

firewallchain { [':stage1:', ':stage2:', ':door:']:
}

# door
firewall { "098 knock2 goes to stage2":
chain => "door",
recent_command => "rcheck",
recent_name => "knock2",
recent_seconds => $seconds,
jump => "stage2",
require => [
Firewallchain[':door:'],
Firewallchain[':stage2:'],
]
}

firewall { "099 knock goes to stage1":
chain => "door",
recent_command => "rcheck",
recent_name => "knock",
recent_seconds => $seconds,
jump => "stage1",
require => [
Firewallchain[':door:'],
Firewallchain[':stage1:'],
]
}

firewall { "100 knock on port $port1 sets knock":
chain => "door",
proto => 'tcp',
recent_name => 'knock',
recent_command => 'set',
dport => $port1,
require => [
Firewallchain[':door:'],
]
}

# stage 1
firewall { "101 stage1 remove knock":
chain => "stage1",
recent_name => "knock",
recent_command => "remove",
require => Firewallchain[':stage1:'],
}

firewall { "102 stage1 set knock2 on $port2":
chain => "stage1",
recent_name => "knock2",
recent_command => "set",
proto => "tcp",
dport => $port2,
require => Firewallchain[':stage1:'],
}

# stage 2
firewall { "103 stage2 remove knock":
chain => "stage2",
recent_name => "knock",
recent_command => "remove",
require => Firewallchain[':stage2:'],
}

firewall { "104 stage2 set heaven on $port3":
chain => "stage2",
recent_name => "heaven",
recent_command => "set",
proto => "tcp",
dport => $port3,
require => Firewallchain[':stage2:'],
}

# let people in heaven
firewall { "105 heaven let connections through":
chain => "INPUT",
proto => "tcp",
recent_command => "rcheck",
recent_name => "heaven",
recent_seconds => $seconds,
dport => $dports,
action => accept,
require => Firewallchain[':stage2:'],
}

firewall { "106 connection initiation to door":
# FIXME: specifying chain explicitly breaks insert_order !
chain => "INPUT",
proto => "tcp",
tcp_flags => "FIN,SYN,RST,ACK SYN",
jump => "door",
require => [
Firewallchain[':door:'],
]
}
}

and I can log in with
nc -w 1 ana 1234; nc -w 1 ana 3456; nc -w 1 ana 2345; ssh -A ana

Lessons learned today:

watch iptables -nvL is an absolutely excellent way of learning more about your firewall – you see your rules and the traffic on them in real time. It made it really easy to see for example the first nc command triggering the knock.
Puppet is reasonably hackable – I was learning quickly as I progressed through test and bug after test and bug.
I still don’t like ruby, and we may never be friends, but at least it’s something I’m capable of learning. Puppet might just end up being the trigger.

Tomorrow, I need to clean up the firewall rules into something reusable, and deploy it on the platform.

1 Mar 2012

2012-03-01T20:52:00+00:00

GStreamer & Buzztard

As a big leftover promise from my talk at the GStreamer conference in 2011 I spend more time to understand latencies inside GStreamer. For plain playback the latencies are not an issue, but for anything interactive, be it entering notes or changing parameters while the song is playing, we want much lower latencies. For the talk I did some measurements by having a pipeline with two branches - src1 ! sink and src2 ! fx1 ! ... ! fxn ! sink. The fx elements where plain volume elements. The branches where panned hard left and right. For the experiment, I was pulling down the volume on the two sources down to 0 at the same time and recoding the audio on the sink. Looking at the wave, one could see the delay in the signal that went through the effect-chain. The longer the effect-chain the larger becomes the delay. Don't misunderstand this, GStreamer is properly handling A/V sync. The buffers that get mixed in front of the sink have the same time stamps, the problem is that we have a lot more buffers traveling the the effect-branch. This is because of the queue elements. In buzztard one can freely link generators to effects, effects to effects and effects to sinks. This includes diamond shaped connections. Here we need queues to not stall processing on one branch. It is sufficient for the queues to keep only 1 buffer. Still even knowing the buffer duration, I could not tell a formula to explain the measured delays.
This month I looked at the issue with a new idea. I wrote a small example that build pipelines close to what I have in buzztard, but stripped of many details. In this code I add buffer probed to all pads. The probe is comparing the time-stamp on the buffer to the pipeline clock. This tells how early the buffer is. Ideally we want buffers to be generated and processed as much in time as possible. When generating audio one needs to configure audio-buffer sizes on the source elements and two properties on the audio-sink - buffer-time and latency-time. A good scheme is to use buffer-time = 2 * latency-time. That configures the sink to have two audio-segments. Initially I also set the buffer sizes on the source elements to have a duration of latency-time. Now one problem with that is that we will have one buffer waiting on each queue. Thus if there are two queues the actual latency is (n-queues + 1) * latency-time. One way to improve that a bit is to half the buffer sizes on the sources, then the 2nd buffer is calculated when the first one is needed. As the first one won't be sufficient to fill the gap, the calculation of the 2nd buffer is scheduled right away. The disadvantage of this scheme is that one gets quite jittery latencies. In the end I settled on finishing the subtick timing in buzztard. Each tick will have n subticks, where n is setup so that we get down to the desired latency. So far I get nice low latency on all my machines (inlcuding a atom based netbook).

We also ported more buzzmachines and have 49 machines right now. When porting we often also fix bugs as gcc is quite good at warnings these days. Finally, the machines now also install docs (where available).

In the UI I got rid of the GtkRuler copy again. The analyzer window now has own code to draw the rulers. I think they look nice, a lot less noisy then the older rulers.

62 files changed, 1957 insertions(+), 1844 deletions(-)

Wacom tablets in GNOME 3.4 (part #2)

2012-02-29T16:36:01+00:00

Since the last report, "mapping" has been the order of the day. We now have support for:

Display mapping

Not as pretty as mocked up, but it's functional. We hope to have the nicer version done for the next development cycle.

and

Button mapping

The button mapping will show you the actual keys, with translated names, for your device, instead of every possible combination of buttons which end up doing nothing.

We've also made a big number of layout changes and fixes, which means that the main window isn't as awkward as it used to be.

There are a number of features that unfortunately didn't make the cut, mostly due to the lack of support from the X.org drivers. Mode switching currently doesn't advertise the current mode by changing the LED, and the OLEDs that show the bindings for each key stand empty.

Y U NO GTK3 YET?

2012-02-28T23:27:23+00:00

Some folks were probably wondering why PiTiVi is still using GTK2. Now you know:

Collabora and Fluendo collaborate fluently!

2012-02-27T12:01:14+00:00

Well, this sure has been a long time in the making.

Fluendo and Collabora have a checkered past which I won’t get into, but on paper it has always made sense for these two companies to collaborate and making GStreamer work commercially. One company specializes in products, the other in consulting (I’m sure you can figure out which is which), and complement each other perfectly to make GSstreamer more successful commercially.

I personally have always believed that we need to get GStreamer to other platforms and make them as easy to use as possible. Windows was an obvious target in the past, and now Android is another. There is a big difference between a successful open source project, and a commercially successful one. Flumotion’s Andoni Morales who came with me to the GStreamer 0.11 hackfest in Malaga is going to be working on this one SDK to rule them all.

Christian beat me to it in the blogosphere, but the word is now officially out! Feel free to read Fluendo’s press release.

Collabora and Fluendo partners to invest in GStreamer

2012-02-27T10:54:24+00:00

So working up to MWC in Barcelona this week we just announced a great new effort by Collabora and Fluendo around the GStreamer project. The goal is to ease adoption of GStreamer by making sure that no matter what platform you are in, there are well tested and easy to install and use binaries of GStreamer available. This means that not matter if you are running Windows, MacOS X or Linux, you will have a GStreamer SDK available to use. Eventually we will also cover various embedded platforms with this SDK, making it easier than ever to use GStreamer to create great cross platform multimedia applications. You can find out more about this effort by checking out the press release which have all the relevant details.

Marché du travail

2012-02-25T00:41:58+00:00

J’ai terminé mes études en décembre dernier. J’ai décroché un emploi comme testeur chez Collabora il y a environ une semaine. C’est une nouvelle étape qui commence, on verra où ça me mènera :)

Note: ça n’a rien à voir avec PiTiVi, auquel je continue de contribuer par loisir.

Export-macosx alt=

2012-02-24T02:10:38+00:00

Syphon Processing

2012-02-21T23:54:24+00:00