November 30, 2016

GStreamerGStreamer 1.10.2 stable release (binaries)

(GStreamer)

Pre-built binary images of the 1.10.2 stable release of GStreamer are now available for Windows 32/64-bit, iOS and Mac OS X and Android.

See /releases/1.10/ for the full list of changes.

The builds are available for download from: Android, iOS, Mac OS X and Windows.

November 30, 2016 05:45 PM

November 29, 2016

GStreamerGStreamer 1.10.2 stable release

(GStreamer)

The GStreamer team is pleased to announce the second bugfix release in the stable 1.10 release series of your favourite cross-platform multimedia framework!

This release only contains bugfixes and it should be safe to update from 1.10.0. For a full list of bugfixes see Bugzilla.

See /releases/1.10/ for the full release notes.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

Check out the release notes for GStreamer core, gst-plugins-base, gst-plugins-good, gst-plugins-ugly, gst-plugins-bad, gst-libav, gst-rtsp-server, gst-python, gst-editing-services, gst-validate, gstreamer-vaapi, or gst-omx, or download tarballs for gstreamer, gst-plugins-base, gst-plugins-good, gst-plugins-ugly, gst-plugins-bad, gst-libav, gst-rtsp-server, gst-python, gst-editing-services, gst-validate, gstreamer-vaapi, or gst-omx.

November 29, 2016 02:00 PM

GStreamerRevamped documentation and gstreamer.com switch-off

(GStreamer)

The GStreamer project is pleased to announce its new revamped documentation featuring a new design, a new navigation bar, search functionality, source code syntax highlighting as well as new tutorials and documentation about how to use GStreamer on Android, iOS, macOS and Windows.

It now contains the former gstreamer.com SDK tutorials which have kindly been made available by Fluendo & Collabora under a Creative Commons license. The tutorials have been reviewed and updated for GStreamer 1.x. The old gstreamer.com site will be shut down with redirects pointing to the updated tutorials and the official GStreamer website.

Thanks to everyone who helped make this happen.

This is just the beginning. Our goal is to provide a more cohesive documentation experience for our users going forward. To that end, we have converted most of our documentation into markdown format. This should hopefully make it easier for developers and contributors to create new documentation, and to maintain the existing one. There is a lot more work to do, do get in touch if you want to help out. The documentation is maintained in the new gst-docs module.

If you encounter any problems or spot any omissions or outdated content in the new documentation, please file a bug in bugzilla to let us know.

November 29, 2016 12:00 PM

November 24, 2016

Sebastian DrögeWriting GStreamer Elements in Rust (Part 3): Parsing data from untrusted sources like it’s 2016

(Sebastian Dröge)

And again it took quite a while to write a new update about my experiments with writing GStreamer elements in Rust. The previous articles can be found here and here. Since last time, there was also the GStreamer Conference 2016 in Berlin, where I had a short presentation about this.

Progress was rather slow unfortunately, due to work and other things getting into the way. Let’s hope this improves. Anyway!

There will be three parts again, and especially the last one would be something where I could use some suggestions from more experienced Rust developers about how to solve state handling / state machines in a nicer way. The first part will be about parsing data in general, especially from untrusted sources. The second part will be about my experimental and current proof of concept FLV demuxer.

Parsing Data

Safety?

First of all, you probably all saw a couple of CVEs about security relevant bugs in (rather uncommon) GStreamer elements going around. While all of them would’ve been prevented by having the code written in Rust (due to by-default array bounds checking), that’s not going to be our topic here. They also would’ve been prevented by using various GStreamer helper API, like GstByteReader, GstByteWriter and GstBitReader. So just use those, really. Especially in new code (which is exactly the problem with the code affected by the CVEs, it was old and forgotten). Don’t do an accountant’s job, counting how much money/many bytes you have left to read.

But yes, this is something where Rust will also provide an advantage by having by-default safety features. It’s not going to solve all our problems, but at least some classes of problems. And sure, you can write safe C code if you’re careful but I’m sure you also drive with a seatbelt although you can drive safely. To quote Federico about his motivation for rewriting (parts of) librsvg in Rust:

Every once in a while someone discovers a bug in librsvg that makes it all the way to a CVE security advisory, and it’s all due to using C. We’ve gotten double free()s, wrong casts, and out-of-bounds memory accesses. Recently someone did fuzz-testing with some really pathological SVGs, and found interesting explosions in the library. That’s the kind of 1970s bullshit that Rust prevents.

You can directly replace the word librsvg with GStreamer here.

Ergonomics

The other aspect with parsing data is that it’s usually a very boring aspect of programming. It should be as painless as possible, as easy as possible to do it in a safe way, and after having written your 100th parser by hand you probably don’t want to do that again. Parser combinator libraries like Parsec in Haskell provide a nice alternative. You essentially write down something very close to a formal grammar of the format you want to parse, and out of this comes a parser for the format. Other than parser generators like good, old yacc, everything is written in target language though, and there is no separate code generation step.

Rust, being quite a bit more expressive than C, also made people write parser generator libraries. They are all not as ergonomic (yet?) as in Haskell, but still a big improvement over anything else. There’s nom, combine and chomp. All having a slightly different approach. Choose your favorite. I decided on nom for the time being.

A FLV Demuxer in Rust

For implementing a demuxer, I decided on using the FLV container format. Mostly because it is super-simple compared to e.g. MP4 and WebM, but also because Geoffroy, the author of nom, wrote a simple header parsing library for it already and a prototype demuxer using it for VLC. I’ll have to extend that library for various features in the near future though, if the demuxer should ever become feature-equivalent with the existing one in GStreamer.

As usual, the code can be found here, in the “demuxer” branch. The most relevant files are rsdemuxer.rs and flvdemux.rs.

Following the style of the sources and sinks, the first is some kind of base class / trait for writing arbitrary demuxers in Rust. It’s rather unfinished at this point though, just enough to get something running. All the FLV specific code is in the second file, and it’s also very minimal for now. All it can do is to play one specific file (or hopefully all other files with the same audio/video codec combination).

As part of all this, I also wrote bindings for GStreamer’s buffer abstraction and a Rust-rewrite of the GstAdapter helper type. Both showed Rust’s strengths quite well, the buffer bindings by being able to express various concepts of the buffers in a compiler-checked, safe way in Rust (e.g. ownership, reability/writability), the adapter implementation by being so much shorter (it’s missing features… but still).

So here we are, this can already play one specific file (at least) in any GStreamer based playback application. But some further work is necessary, for which I hopefully have some time in the near future. Various important features are still missing (e.g. other codecs, metadata extraction and seeking), the code is rather proof-of-concept style (stringly-typed media formats, lots of unimplemented!() and .unwrap() calls). But it shows that writing media handling elements in Rust is definitely feasible, and generally seems like a good idea.

If only we had Rust already when all this media handling code in GStreamer was written!

State Handling

Another reason why all this took a bit longer than expected, is that I experimented a bit with expressing the state of the demuxer in a more clever way than what we usually do in C. If you take a look at the GstFlvDemux struct definition in C, it contains about 100 lines of field declarations. Most of them are only valid / useful in specific states that the demuxer is in. Doing the same in Rust would of course also be possible (and rather straightforward), but I wanted to try to do something better, especially by making invalid states unrepresentable.

Rust has this great concept of enums, also known as tagged unions or sum types in other languages. These are not to be confused with C enums or unions, but instead allow multiple variants (like C enums) with fields of various types (like C unions). But all of that in a type-safe way. This seems like the perfect tool for representing complicated state and building a state machine around it.

So much for the theory. Unfortunately, I’m not too happy with the current state of things. It is like this mostly because of Rust’s ownership system getting into my way (rightfully, how would it have known additional constraints I didn’t know how to express?).

Common Parts

The first problem I ran into, was that many of the states have common fields, e.g.

enum State {
    ...
    NeedHeader,
    HaveHeader {header: Header, to_skip: usize },
    Streaming {header: Header, audio: ... },
    ...
}

When writing code that matches on this, and that tries to move from one state to another, these common fields would have to be moved. But unfortunately they are (usually) borrowed by the code already and thus can’t be moved to the new variant. E.g. the following fails to compile

match self.state {
        ...
        State::HaveHeader {header, to_skip: 0 } => {
            
            self.state = State::Streaming {header: header, ...};
        },
    }

A Tree of States

Repeating the common parts is not nice anyway, so I went with a different solution by creating a tree of states:

enum State {
    ...
    NeedHeader,
    HaveHeader {header: Header, have_header_state: HaveHeaderState },
    ...
}

enum HaveHeaderState {
    Skipping {to_skip: usize },
    Streaming {audio: ... },
}

Apart from making it difficult to find names for all of these, and having relatively deeply nested code, this works

match self.state {
        ...
        State::HaveHeader {ref header, ref mut have_header_state } => {
            match *have_header_state {
                HaveHeaderState::Skipping { to_skip: 0 } => {
                    *have_header = HaveHeaderState::Streaming { audio: ...};
                }
        },
    }

If you look at the code however, this causes the code to be much bigger than needed and I’m also not sure yet how it will be possible nicely to move “backwards” one state if that situation ever appears. Also there is still the previous problem, although less often: if I would match on to_skip here by reference (or it was no Copy type), the compiler would prevent me from overwriting have_header for the same reasons as before.

So my question for the end: How are others solving this problem? How do you express your states and write the functions around them to modify the states?

Update

I actually implemented the state handling as a State -> State function before (and forgot about that), which seems conceptually the right thing to do. It however has a couple of other problems. Thanks for the suggestions so far, it seems like I’m not alone with this problem at least.

Update 2

I’ve went a bit closer to the C-style struct definition now, as it makes the code less convoluted and allows me to just get forwards with the code. The current status can be seen here now, which also involves further refactoring (and e.g. some support for metadata).

by slomo at November 24, 2016 11:10 PM

November 22, 2016

GStreamerGStreamer 1.10.1 stable release (binaries)

(GStreamer)

Pre-built binary images of the 1.10.1 stable release of GStreamer are now available for Windows 32/64-bit, iOS and Mac OS X and Android.

See /releases/1.10/ for the full list of changes.

The builds are available for download from: Android, iOS, Mac OS X and Windows.

November 22, 2016 12:10 PM

November 19, 2016

Víctor JáquezGStreamer-VAAPI 1.10 (now 1.10.1)

A lot of things have happened last month and I have neglected this report. Let me try to fix it.

First, the autumn GStreamer Hackfest and the GStreamer Conference 2016 in Berlin.

The hackfest’s venue was the famous C-Base, where we talked with friends and colleagues. Most of the discussions were leaned towards the release 1.10. Still, Julien Isorce and I talked about our approaches for DMABuf sharing with downstream; we agreed on to explore how to implement an allocator based on GstDmaBufAllocator. We talked with Josep Torra about the performance he was getting with gstreamer-vaapi encoders (and we already did good progress on this). Also we talked with Nicolas Dufresne about v4l2 and kmssink. And many other informal chats along with beer and pizza.

GStreamer Hackfest a the C-Base (from @calvaris tweet)GStreamer Hackfest a the C-Base (from @calvaris tweet)

Afterwards, in the GStreamer Conference, I talked a bit about the current status of GStreamer-VAAPI and what we can expect from release 1.10. Here’s the video that the folks from Ubicast recorded. You can see the slides here too.

I spent most of the conference with Scott, Sree and Josep tracing the bottlenecks in the frame uploading to VA Intel driver, and trying to answer the questions of the folks who gently approached me. I hope my answers were any helpful.

And finally, the first of November, the release 1.10 hit the streets!

As it is already mentioned in the release notes, these are the main new features you can find in GStreamer-VAAPI 1.10

  • All decoders have been split, one plugin feature per codec. So far, the available ones, depending on the driver, are: vaapimpeg2dec, vaapih264dec, vaapih265dec, vaapivc1dec, vaapivp8dec, vaapivp9dec and vaapijpegdec (which already was split).
  • Improvements when mapping VA surfaces into memory, differenciating from negotiation caps and allocations caps, since the allocation memory for surfaces may be bigger than one that is going to be mapped.
  • vaapih265enc got support to constant bitrate mode (CBR).
  • Since several VA drivers are unmaintained, we decide to keep a white list with the va drivers we actually test, which is mostly the i915 and, in some degree, gallium from mesa project. Exporting the environment variable GST_VAAPI_ALL_DRIVERS disable the white list.
  • The plugin features are registered, in run-time, according to its support by the loaded VA driver. So only the decoders and encoder supported by the system are registered. Since the driver can change, some dependencies are tracked to invalidate the GStreamer registry and reload the plugin.
  • DMA-Buf importation from upstream has been improved, gaining performance.
  • vaapipostproc now can negotiate through caps the buffer transformations.
  • Decoders now can do reverse playback! But they only shows I frames, because the surface pool is smaller than the required by the GOP to show all the frames.
  • The upload of frames onto native GL textures has been optimized too, keeping a cache of the internal structures for the offered textures by the sink, improving a lot the performance with glimagesink.

This is the short log summary since branch 1.8:

  1  Allen Zhang
 19  Hyunjun Ko
  1  Jagyum Koo
  2  Jan Schmidt
  7  Julien Isorce
  1  Matthew Waters
  1  Michael Olbrich
  1  Nicolas Dufresne
 13  Scott D Phillips
  9  Sebastian Dröge
 30  Sreerenj Balachandran
  1  Stefan Sauer
  1  Stephen
  1  Thiago Santos
  1  Tim-Philipp Müller
  1  Vineeth TM
154  Víctor Manuel Jáquez Leal

Thanks to Igalia, Intel Open Source Technology Center and many others for making GStreamer-VAAPI possible.

by vjaquez at November 19, 2016 11:31 AM

November 17, 2016

GStreamerGStreamer 1.10.1 stable release

(GStreamer)

The GStreamer team is pleased to announce the first bugfix release in the stable 1.10 release series of your favourite cross-platform multimedia framework!

This release only contains bugfixes and it should be safe to update from 1.10.0. For a full list of bugfixes see Bugzilla.

See /releases/1.10/ for the full release notes.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

Check out the release notes for GStreamer core, gst-plugins-base, gst-plugins-good, gst-plugins-ugly, gst-plugins-bad, gst-libav, gst-rtsp-server, gst-python, gst-editing-services, gst-validate, gstreamer-vaapi, or gst-omx, or download tarballs for gstreamer, gst-plugins-base, gst-plugins-good, gst-plugins-ugly, gst-plugins-bad, gst-libav, gst-rtsp-server, gst-python, gst-editing-services, gst-validate, gstreamer-vaapi, or gst-omx.

November 17, 2016 03:00 PM

Jean-François Fortin TamHelping Purism structure its messaging

I meant to finish writing and posting this a month or two ago, but urgent tasks and life kept getting in the way. I don’t often talk about client work here, but since this is public-facing ongoing work for a company that is insanely pro-Free-Software (not just “open source”), a company that ships GNOME3 by default on their laptops (something I have awaited for years), I guess it makes sense to talk about what I’ve been up to recently.

So, for a few weeks three months now, I have been helping Purism structure its messaging and get its business in a better shape. Purism is, in itself, a hugely interesting endeavour. Heck, I could go out on a limb and say this venture, alongside the work Endless is doing, is quite possibly one of the most exciting things that has happened to the Free desktop for the past decade—and yet almost nobody heard of it.

2016-06-01

Before I can even consider visual branding work (maybe someday—when I get to that point, that would mean things are going really well), there was a fundamental need to fill various gaps in the strategy and daily operations, and to address messaging in a way that simultaneously resonates with:

  • hardcore Free Software enthusiasts;
  • “Linux” (GNU/Linux) users and developers just looking for great ultraportable workhorses;
  • the privacy/security-conscious crowd;
  • the public at large (hopefully).

Purism’s website and previous campaigns were full of relevant content, but it was disjointed, sometimes outdated, and sometimes a bit imprecise, confusing, or just plain boring. You really had to be abnormally motivated to dig deep and digest a ton of fairly arid content. I set out to fix all that, so I looked at every single page on the website and external campaigns to rearchitect everything back into a solid foundation, something insightful and pleasant to read.

There was so much buried and scattered information to analyze, I coudn’t have done this with computer screens, no matter how large. I decided to print out a summary for easy annotation. Even with super compacted text (small fonts, tight paragraphs, fewer images, etc.) and redundant text passages removed, my “content review summary” document of the core content was still a whopping 28 pages in “US legal” size (8.5×14 inches). It easily occupies all my office’s desk space:

2016-11-16--18.08.06 2016-11-16--18.14.022016-11-16--18.11.35 2016-11-16--18.08.40

Serious business.

Having no time to waste, and having made a very clear list of issues to fix, I applied my proposed changes directly after completing the semantic and structural analysis.

Nike Just do it basket

The taxonomy and ontology in the blog section was all wrong, too. So I rethought and reorganized all the tags and categories, to create a reliable system for visitors to browse through. For example, I made the tags go from what a noisy mess (pictured on the left) to something much more meaningful (on the right):

beforeafter

The website overhaul led to the new “Why Purism?” section, the new Global Shipping Status page, a much nicer stylesheet (my eyes were bleeding trying to read anything on the website and its blog), URLs that actually make sense, and a rethought product discovery and purchase navigation flow. Some more details on the overall changes can be found here. Hopefully, the current state of the website is much more enticing now. It may not be perfect (feedback welcome) but it’s a good start.

I also paid attention to a bunch of papercuts when it comes to interacting with the community. For example, blog posts lacked a relatable face, so I added avatars for everybody. Forums had an absolutely horrible mail notification system, so I fixed that. People were unsure about warranties, delivery dates, etc., so I made sure to help clarify those as best as possible throughout the website. That sort of thing.

The next big steps, in my view:

  • Helping ensure the business’ viability, beyond just “survival”. Running a “normal” hardware manufacturing business is already fairly complicated from the get go—so making an independent “organically grown” one profitable, that is much harder. Purism is at a turning point right now, and the coming months will be a real test of the feasibility of reconciling Free Software with modern hardware. Most of us have been waiting so long for this, failure is not an option.
  • Improving the customer experience.
  • Improving the product design.

By the way, I have been testing out a “Librem 15” laptop extensively for about two months, and I am hoping to publish a very detailed review and feedback video soon (I don’t do superficial/half-assed reviews, hence the delay). Stay tuned!

The test unit I have transplanted my testing drive into

The review unit I have transplanted my (ThinkPad’s) testing drive into for further testing (this is why you can see the dash to dock extension active)

by nekohayo at November 17, 2016 05:52 AM

November 15, 2016

Bastien NoceraLyon GNOME Bug day #1

(Bastien Nocera) Last Friday, both a GNOME bug day and a bank holiday, a few of us got together to squash some bugs, and discuss GNOME and GNOME technologies.

Guillaume, a new comer in our group, tested the captive portal support for NetworkManager and GNOME in Gentoo, and added instructions on how to enable it to their Wiki. He also tested a gateway related configuration problem, the patch for which I merged after a code review. Near the end of the session, he also rebuilt WebKitGTK+ to test why Google Docs was not working for him anymore in Web. And nobody believed that he could build it that quickly. Looks like opinions based on past experiences are quite hard to change.

Mathieu worked on removing jhbuild's .desktop file as nobody seems to use it, and it was creating the Sundry category for him, in gnome-shell. He also spent time looking into the tracker blocker that is Mozilla's Focus, based on disconnectme's block lists. It's not as effective as uBlock when it comes to blocking adverts, but the memory and performance improvements, and the slow churn rate, could make it a good default blocker to have in Web.

Haïkel looked into using Emeus, potentially the new GTK+ 4.0 layout manager, to implement the series properties page for Videos.

Finally, I added Bolso to jhbuild, and struggled to get gnome-online-accounts/gnome-keyring to behave correctly in my installation, as the application just did not want to log in properly to the service. I also discussed Fedora's privacy policy (inappropriate for Fedora Workstation, as it doesn't cover the services used in the default installation), a potential design for Flatpak support of joypads and removable devices in general, as well as the future design of the Network panel.

by Bastien Nocera (noreply@blogger.com) at November 15, 2016 09:48 AM

November 10, 2016

Christian SchallerMp3 support now coming to Fedora Workstation 25

(Christian Schaller)

So, in Fedora Workstation 24 we added H264 support through OpenH264. In Fedora Workstation 25 I am happy to tell you all that we are taking another step in improving our codec support by adding support for mp3 playback. I know this has been a big wishlist item for a long time for a lot of people so I am really happy that we are finally in a position to fulfil that wish. You should be able to download the mp3 plugin on day 1 through GNOME Software or through the missing codec installer in various GStreamer applications. For Fedora Workstation 26 I would not be surprised if we decide to ship it on the install media.

Fo the technically inclined out there, our initial enablement is through the mpeg123 library and corresponding GStreamer plugin. The main reason we choose this library over all the others available out there was a combination of using the same license as GStreamer (lgpl v2) and being a well established library used by a lot of different applications already. There might be other mp3 decoders added in the future depending on interest in and effort by the community. So get ready to install Fedora Workstation 25 when its released soon and play some tunes :)

P.S. To be 110% clear we will not be adding encoding support at this time.

by uraeus at November 10, 2016 07:34 PM

November 04, 2016

Arun RaghavanGStreamer and Synchronisation Made Easy

A lesser known, but particularly powerful feature of GStreamer is our ability to play media synchronised across devices with fairly good accuracy.

The way things stand right now, though, achieving this requires some amount of fiddling and a reasonably thorough knowledge of how GStreamer’s synchronisation mechanisms work. While we have had some excellent talks about these at previous GStreamer conferences, getting things to work is still a fair amount of effort for someone not well-versed with GStreamer.

As part of my work with the Samsung OSG, I’ve been working on addressing this problem, by wrapping all the complexity in a library. The intention is that anybody who wants to implement the ability for different devices on a network to play the same stream and have them all synchronised should be able to do so with a few lines of code, and the basic know-how for writing GStreamer-based applications.

I’ve started work on this already, and you can find the code in the creatively named gst-sync-server repo.

Design and API

Let’s make this easier by starting with a picture …

Big picture of the architecture

Let’s say you’re writing a simple application where you have two ore more devices that need to play the same video stream, in sync. Your system would consist of two entities:

  • A server: this is where you configure what needs to be played. It instantiates a GstSyncServer object on which it can set a URI that needs to be played. There are other controls available here that I’ll get to in a moment.

  • A client: each device would be running a copy of the client, and would get information from the server telling it what to play, and what clock to use to make sure playback is synchronised. In practical terms, you do this by creating a GstSyncClient object, and giving it a playbin element which you’ve configured appropriately (this usually involves at least setting the appropriate video sink that integrates with your UI).

That’s pretty much it. Your application instantiates these two objects, starts them up, and as long as the clients can access the media URI, you magically have two synchronised streams on your devices.

Control

The keen observers among you would have noticed that there is a control entity in the above diagram that deals with communicating information from the server to clients over the network. While I have currently implemented a simple TCP protocol for this, my goal is to abstract out the control transport interface so that it is easy to drop in a custom transport (Websockets, a REST API, whatever).

The actual sync information is merely a structure marshalled into a JSON string and sent to clients every time something happens. Once your application has some media playing, the next thing you’ll want to do from your server is control playback. This can include

  • Changing what media is playing (like after the current media ends)
  • Pausing/resuming the media
  • Seeking
  • “Trick modes” such as fast forward or reverse playback

The first two of these work already, and seeking is on my short-term to-do list. Trick modes, as the name suggets, can be a bit more tricky, so I’ll likely get to them after other things are done.

Getting fancy

My hope is to see this library being used in a few other interesting use cases:

  • Video walls: having a number of displays stacked together so you have one giant display — these are all effectively playing different rectangles from the same video

  • Multiroom audio: you can play the same music across different speakers in a single room, or multiple rooms, or even group sets of speakers and play different media on different groups

  • Media sharing: being able to play music or videos on your phone and have your friends be able to listen/watch at the same time (a silent disco app?)

What next

At this point, the outline of what I think the API should look like is done. I still need to create the transport abstraction, but that’s pretty much a matter of extracting out the properties and signals that are part of the existing TCP transport.

What I would like is to hear from you, my dear readers who are interested in using this library — does the API look like it would work for you? Does the transport mechanism I describe above cover what you might need? There is example code that should make it easier to understand how this library is meant to be used.

Depending on the feedback I get, my next steps will be to implement the transport interface, refine the API a bit, fix a bunch of FIXMEs, and then see if this is something we can include in gst-plugins-bad.

Feel free to comment either on the Github repository, on this blog, or via email.

And don’t forget to watch this space for some videos and measurements of how GStreamer synchronised fares in real life!

by Arun at November 04, 2016 10:16 AM

November 02, 2016

GStreamerGStreamer 1.10.0 stable release (binaries)

(GStreamer)

Pre-built binary images of the 1.10.0 stable release of GStreamer are now available for Windows 32/64-bit, iOS and Mac OS X and Android.

See /releases/1.10/ for the full list of changes.

The builds are available for download from: Android, iOS, Mac OS X and Windows.

November 02, 2016 04:10 PM

November 01, 2016

GStreamerGStreamer 1.10.0 stable release

(GStreamer)

The GStreamer team is proud to announce a new major feature release in the stable 1.x API series of your favourite cross-platform multimedia framework!

As always, this release is again packed with new features, bug fixes and other improvements.

See /releases/1.10/ for the full list of changes.

Binaries for Android, iOS, Mac OS X and Windows will be provided shortly after the source release by the GStreamer project during the stable 1.10 release series.

Check out the release notes for GStreamer core, gst-plugins-base, gst-plugins-good, gst-plugins-ugly, gst-plugins-bad, gst-libav, gst-rtsp-server, gst-python, gst-editing-services, gst-validate, gstreamer-vaapi, or gst-omx, or download tarballs for gstreamer, gst-plugins-base, gst-plugins-good, gst-plugins-ugly, gst-plugins-bad, gst-libav, gst-rtsp-server, gst-python, gst-editing-services, gst-validate, gstreamer-vaapi, or gst-omx.

November 01, 2016 04:30 PM

Christian SchallerHybrid Graphics and Fedora Workstation 25

(Christian Schaller)

When we started the Fedora Workstation effort one thing we wanted to do was to drain the proverbial swamp of make sure that running Linux on a laptop is a first rate experience. As you see from my last blog entry we have been working on building a team dedicated to that task. There are many facets to this effort, but one that we kept getting asked about was sorting out hybrid graphics. So be aware that some of this has been covered in previous blog entries, but I want to be thorough here. So this blog will cover the big investments of time and effort we are putting into Wayland and X Windows, GNOME Shell and Nouveau (the open source driver for NVidia GPU hardware).

A big part of this story is also that we are trying to make it easier to run the NVidia binary driver on Fedora in general which is actually a fairly large undertaking as there are a lot challenges with trying to do that without requiring any manual configuration from the user and making sure it works fully in the hybrid graphics usecase. This is one of the most requested areas of improvement for Fedora. Many users are trying to run Fedora on their existing laptops or other hardware. Rather than allow this gap to be a reason for people to not run Fedora, we want to provide a better experience. Of course users will have the freedom to make their own choice about installing these drivers, using information provided by Software.

Hybrid graphics is the term used for when you have a laptop with two GPUs, usually one Intel and one NVidia GPU, but there are also some laptops available which comes with Intel/AMD CPU + AMD GPU. The purpose of hybrid graphics is to have your (Intel) Integrated GPU be your ‘day to day GPU’ for running your desktop yet not drawing to much power, but then if you want to for instance play a game on your system you can activate your secondary GPU which got a lot more power, but which also draws more electricity.

What complicated this even more was the fact that most users who wanted to use this setup wanted to use it in combination with the binary NVidia driver, in order to get top performance from their second GPU, which is not surprising since that is the whole point of switching to it. The main challenge with this was that the Mesa and binary NVidia driver both provided an OpenGL implementation that due to the way things works in the X Window System ended up overwriting each other, basically breaking the overwritten driver. The solution for this was a system called Bumblebee which employed some clever hacks to work around this issue, but Bumblebee is a solution to a problem we shouldn’t have to begin with.

Of course dealing with the OpenGL stack wasn’t the only challenge here. For instance different display outputs ended up being connected to different GPUs, with one of the most common setups being that the internal screen is connected to your Intel GPU and your external HDMI and DisplayPort connections are connected to your NVidia or AMD GPU. One some systems there where hardware connectors allowing you to display using either GPU to any screen, but a setup we see becoming more common is that the drivers for both card needs to be initialized in all cases, allowing the display connected GPU to slave itself to the other GPU to allow rendering on one GPU and displaying on the other. Within Mesa this is fairly straightforward to handle, so if both Intel and NVidia as rendering using Mesa things are fairly straightforward. What makes this a lot more challenging here is the NVidia binary driver, so once you install the binary driver we need to find a way to bridge and interoperate between these two stacks.

And of course that is just the highlights, like anything complex like this there are a long laundry list of items from the point where you can checkbox having a feature to it working really well and seamless.

What to expect in Fedora Workstation 25
Lets start with a word of caution here, we are working on a tight deadline here to get all the pieces lined up, so I can not be 100% sure what will make it for the day of release. What we are confident about though is that we will have all the low level plumbing sorted so that we can improve this over the course of the Fedora 25 lifecycle. As always I hope the community will join us in testing and developing this, to ensure we have even the corner cases worked out for Fedora Workstation 26.

The initial step we took and this is quite some time ago was that Dave Airlie worked on making sure we could handle two GPUs within the Mesa stack. As a bit of wordplay on the NVidia solution being called Optimus the open source Mesa solution was named Prime. This work basically allowed you to use Mesa for your Intel card and use Mesa (Nouveau) for your NVidia card and launch applications on the NVidia card while the screen was connected to the Intel card. You can choose which one is used by setting the DRI_PRIME=1 environment variable.

The second step was Adam Jackson collaborating with NVidia on something called libglvnd. Libglvnd stands for GL Vendor Neutral Dispatch and it is basically a dispatch layer that allows your OpenGL calls to be dispatched to more than one OpenGL implementation. Nvidia created this specification and have been steadily working on supporting it in their binary driver, but Adam Jackson stepped up to help review their patches and work on supporting glvnd in the Mesa drivers. This means that for Fedora Workstation 25 you should for the first time ever be able to have both the binary NVidia driver and Mesa installed without any file conflicts. So the X server now has the infrastructure to route your OpenGL call to the correct stack when that DRI_PRIME variable is set. That said there is a major catch here, due to the way things currently work once the NVidia binary driver is installed it expects to be rendering the screen all the time. So we need for the short term to figure out a way to allow it to do that, and in the long run we need to work with NVidia to figure out how the Intel open source driver can collaborate with the Nvidia driver to allow us to only use the Intel driver at times (to save on power for instance). We are still pushing hard on trying to have the short term fix in place, but as I write this we haven’t fully nailed this down yet.

The third step is work that Ben Skeggs has been doing on dealing with the monitor handling here, which includes adding MST support to Nouveau, because a lot of external ports and docking stations have not been working with these hybrid graphics setups due to the external screens all being routed through the NVidia chip. These patches just got accepted upstream and we will be including them in Fedora Workstation 25.

The fourth step has been work that Hans de Goede has been doing around Prime, fixing the modesetting driver and fixing cursor handling with hybrid graphics. In some sense the work Hans has been doing, and check his blog entry linked, is part of that washlist I talked about, the many smaller items that needs to work smoothly for this to go from ‘checkbox works’ to ‘actually works’. He is also working with the DNF team to allow us to control the kernel updates if you use the binary NVidia driver, meaning that we will only bump up your kernel driver when we have the binary NVidia driver module ready too. If for any reason the binary Nvidia driver doesn’t work we want to have graceful fallback to Nouveau.

Fifth step has been Jonas Ådahls work on enabling the binary NVidia driver for Wayland. He has put together a set of patches to be able to support NVidias EGLStreams interface, which means that starting from Fedora Workstation 25 you will be able to use Wayland also with NVidias binary driver.
You can see his work in progress patches here. Jonas will also be looking at implementing hybrid graphics support in Wayland, so we ensure that also for this usecase Wayland is on par with X for Fedora Workstation 26.

The sixth step has been work done by Bastien Nocera to ensure we expose this functionality in the user interface. The idea is that you should be able to configure on a per application basis which GPU they are being launched on. It also means that you can now get all your GPU information in the GNOME about screen also when you have dual-GPU systems. More details in his blog post here.
gnome-shell-discrete-gpu-menuChoosing discete GPU in GNOME Shell

The seventh step is the work that Simone Caronni from Negativo17 has been doing on working with us on packaging the binary NVidia driver in a way that makes all this work. You can find his repo here on Negativo17, but Simone has also been working with Kalev Lember and Richard Hughes to ensure the driver show up correctly in GNOME Software once you have the repository enabled.
NVidia driver in GNOME SoftwareNVidia driver in GNOME Software
The plan is to offer the driver in Fedora Workstation 25 as third party software, but we haven’t yet made the formal proposal due to wanting to be sure we ironed out all important bugs first, both on our side and on NVidia side.

So as you can see from all of this there are a lot of components involved and since we are trying to support both X and Wayland with all of this the job hasn’t exactly been easy. There are for sure some things that will not be ready in time for Fedora Workstation 25, for instance if you want to use the binary NVidia driver we will not be able to make that work with XWayland. So native Wayland applications will be fine, including games using SDL on top of Wayland, but due to how the stack is architected we haven’t gotten around to implementing bridging OpenGL from Xwayland down to the binary NVidia driver. With Nouveau it should work however. This corner case we will have to figure out for Fedora Workstation 26, but for now we have decided that if we detect a Optimus system we will default you to X instead of Wayland.

Get involved
As always we love for more people to join us in this effort and one good way to get started with that is to join our Hybrid graphics test day on Thursday the 3rd of November!

by uraeus at November 01, 2016 02:05 PM

October 31, 2016

Bastien NoceraFlatpak cross-compilation support: Epilogue

(Bastien Nocera) You might remember my attempts at getting an easy to use cross-compilation for ARM applications on my x86-64 desktop machine.

With Fedora 25 approaching, I'm happy to say that the necessary changes to integrate the feature have now rolled into Fedora 25.

For example, to compile the GNU Hello Flatpak for ARM, you would run:

$ flatpak install gnome org.freedesktop.Platform/arm org.freedesktop.Sdk/arm
Installing: org.freedesktop.Platform/arm/1.4 from gnome
[...]
$ sudo dnf install -y qemu-user-static
[...]
$ TARGET=arm ./build.sh

For other applications, add the --arch=arm argument to the flatpak-builder command-line.

This example also works for 64-bit ARM with the architecture name aarch64.

by Bastien Nocera (noreply@blogger.com) at October 31, 2016 12:00 PM

October 28, 2016

Reynaldo VerdejoWayland uninstalled. The easy way


This article is an adaptation of another one first posted on the Samsung Open Source Group blog.

I recently started looking at some GStreamer & Wayland integration issues and, as everyone would, commenced by trying to setup a Wayland development environment.

Before getting my feet wet though, I decided to have a chat about this with Derek Forman, OSG's resident Wayland expert. This isn't surprising because on our team, pretty much every task starts by having a conversation with one of our field specialists. The idea is to save time, as you might have guessed.

This time around I was looking for a fairly trivial piece of info:

Me - "Hey Derek, I have Wayland installed on my distro for some reason - I don't really want to take a look at now - and I would like to setup an upstream Wayland environment without messing it up. Do you have some script like GStreamer's gst-uninstalled so I can perform this feat without messing with my local install?

Derek - "Hey Reynaldo, No."

He said something else I forgot already. Guess he was ranting as usual. Anyhow, the conversation continued with me taking care of the usual I-forgot-to-ask-how-you-were-doings, and later, I'm not sure who's idea it was but we decided to try upstreaming a
wl_uninstalled script --The name had to match existing infrastructure, but that's a boring story you don't want to know about-- that, like gst-uninstalled, would enable developers to quickly setup and use a build/run-time environment consisting of an uninstalled set of interdependent Wayland repos. The idea is not new, but it's still extremely useful. Sure, you can perform similar feats by littering your local disk with custom-prefix installs of pretty much anything, but hey, it is bad taste OK? and you will waste a lot of time, at the very least once.

As if the foreword wasn't boring enough and to continue trying your resilience, here are the instructions to use the thing we come up with:

First the What

Essentially, wl_uninstalled is a helper script that automates the setup needed to build and work with an uninstalled Wayland/Weston environment comprised at least of the wayland, wayland-protocols, libinput, and weston repositories. The provides a shell environment where all build and run-time dependencies are resolved so the uninstalled versions of these modules take precedence.

Then the How

I'll use Weston as an example although other Wayland-based projects should work as well.
Edit a local copy of the script to make $WLD point to the base directory where the repositories are located, make sure to use the absolute path. Then, after executing the script, issue the following commands to get everything built and Weston running from the uninstalled environment:
cd $WLD
for i in wayland wayland-protocols libinput weston; do cd $i && ./autogen.sh && make && cd ..; done
weston &

This work is now upstream, you can see the discussions by searching for 'uninstalled' in the Wayland mailing lists archive. As an added bonus, it turned out that fiddling with this uninstalled environment mambo-jambo lead to the uncovering of a few additional broken pieces of buildsystem/pkg-config cruft that we fixed too. 'Cause we are good as that see ;). Not everything has been merged though. A word on that at the end.

Back to my own use case --GStreamer / Wayland integration work-- now I can simply call the wl_uninstalled script before gst-uninstalled and I'm all set. My uninstalled GStreamer environment will use my (in-turn) uninstalled Wayland/Weston environment making debugging, fixing and integrating against the upstream version of both projects a breeze --Not really but still--, and my filesystem more FHS-compliant. But then again, who cares about that; you probably just want all this to be quick and painless. Hopeless.

Note:

While all patches for this are on Wayland's public mailing list, some have not been merged yet. If you want to try the procedure described here you will need to manually apply the missing ones till they find their way into the official repository:
The actual script is in the wayland-build-tools repository now.

by reynaldo (noreply@blogger.com) at October 28, 2016 05:14 PM

October 26, 2016

Bastien NoceraDual-GPU integration in GNOME

(Bastien Nocera)
Thanks to the work of Hans de Goede and many others, dual-GPU (aka NVidia Optimus or AMD Hybrid Graphics) support works better than ever in Fedora 25.

On my side, I picked up some work I originally did for Fedora 24, but ended up being blocked by hardware support. This brings better integration into GNOME.

The Details Settings panel now shows which video cards you have in your (most likely) laptop.

dual-GPU Graphics

The second feature is what Blender and 3D video games users have been waiting for: a contextual menu item to launch the application on the more powerful GPU in your machine.

Mooo Powaa!

This demonstration uses a slightly modified GtkGLArea example, which shows which of the GPUs is used to render the application in the title bar.

on the integrated GPU

on the discrete GPU

Behind the curtain

Behind those 2 features, we have a simple D-Bus service, which runs automatically on boot, and stays running to offer a single property (HasDualGpu) that system components can use to detect what UI to present. This requires the "switcheroo" driver to work on the machine in question.

Because of the way applications are launched on the discrete GPU, we cannot currently support D-Bus activated applications, but GPU-heavy D-Bus-integrated applications are few and far between right now.

Future plans

There's plenty more to do in this area, to polish the integration. We might want applications to tell us whether they'd prefer being run on the integrated or discrete GPU, as live switching between renderers is still something that's out of the question on Linux.

Wayland dual-GPU support, as well as support for the proprietary NVidia drivers are also things that will be worked on, probably by my colleagues though, as the graphics stack really isn't my field.

And if the hardware becomes more widely available, we'll most certainly want to support hardware with hotpluggable graphics support (whether gaming laptop "power-ups" or workstation docks).

Availability

All the patches necessary to make this work are now available in GNOME git (targeted at GNOME 3.24), and backports are integrated in Fedora 25, due to be released shortly.

by Bastien Nocera (noreply@blogger.com) at October 26, 2016 02:37 PM

Jean-François Fortin TamPour en finir avec la congestion nasale: comment effectuer le rinçage nasal correctement

Ce billet pourrait aussi s’intituler “Comment en finir avec les allergies et court-circuiter le rhume, la grippe ou la sinusite”, car selon mon expérience la congestion nasale est un symptôme qui empire tout le reste au point de prolonger la maladie. Quand on élimine la congestion nasale et la sinusite, on récupère beaucoup plus rapidement (ou, du moins, beaucoup moins péniblement). Regardez comme cet homme semble heureux:

neti pot sinus rinsing epic dude

La dernière fois que j’ai dû faire appel à cette technique, c’était il y a trois ans lors d’une grippe qui m’avait jeté par terre pendant un mois. Mon ami médecin a alors insisté—à plusieurs reprises—pour que j’utilise du “Sinus Rinse”. Ce truc là avec la boîte chelou:

neilmed sinus rinse

Ça vaut la peine d’acheter le kit d’origine pour avoir la bouteille en même temps que les sachets de solution saline “de grade pharmaceutique”, mais le “jetez la bouteille tous les trois mois” et “n’achetez que les sachets de NeilMed” c’est de la bullshit commerciale. On trouve facilement des sachets de solution saline “grade pharmaceutique” génériques pour moins cher. Pour ce qui est de la bouteille, pas de raison de remplacer ça, surtout que je fais bouillir l’eau à la casserole avant de la refroidir et la mettre dans la bouteille de rinçage—je ne mets pas la bouteille de rinçage au four à micro-ondes.

C’est donc à ce moment-là que j’ai découvert le principe du rinçage nasal avec des produits tels que Sinus Rinse, ce qui m’a permis de me rétablir en quelques jours. Leçon apprise: les médecins ont parfois raison.

Trois ans plus tard, cet automne, j’ai été en proie à des allergies persistantes qui ont résisté à mon rinçage nasal. Après plus d’un mois de “comment ça se fait que ça persiste?”, je découvre que j’avais oublié les détails de la technique secrète. Ce billet de blog vous partage ma redécouverte.

Tout d’abord pour comprendre le principe général du rinçage nasal, regardez cette vidéo avec cette charmante demoiselle qui vous le présente avec le sourire:

Ah, quelle femme! Et comme le disait Panoramix en Égypte, “quel nez!”

Maintenant, voici ce qu’il faut savoir de plus, concrètement :

  • Il ne faut pas lésiner sur l’eau et le sel utilisé : le quart de la bouteille y passe à chaque fois dans chaque narine. Le flot d’eau doit être continu, il ne s’agit pas ici de faire des “petites shot à pression” comme j’avais tendance à faire.
  • Il est normal que vous sentiez qu’une partie se rend dans votre gorge. Si ça vous sort par la bouche, toutefois, c’est que vous n’êtes pas assez penché.
  • Dans le cas d’un rinçage “d’urgence” (ex: après des jours/semaines de congestion), j’ai quelques astuces plus radicales. Identifions “A” la narine où on introduit la bouteille, et “B” la narine où la merde sort.
    • Après avoir rincé en injectant dans la narine “A”, attendez quelques secondes, puis plantez votre doigt mouillé (dégueulasse? On s’en fout, vous êtes tout seul en pantoufles dans la salle de bain) dans la narine “A” afin de créer une obstruction partielle, et expirez profondément pour expulser le mucus liquéfié par la narine “B”.
    • Utilisez également votre doigt mouillé pour enlever l’excédent de mucus pendouillant dans la narine B. Hey, j’ai pas dit que c’était pour les âmes sensibles.
    • Faire la même chose en inversant A et B.
    • Après avoir ainsi rincé est purgé les deux narines, attendre un moment (quelques minutes, une demie, heure, etc.) pour laisser le mucus se liquéfier davantage (vous le sentirez) puis se moucher de nouveau “avec la technique du doigt magique” et répéter le tout depuis le début.

Si vous faites un grand débloquage d’un coup, faites-le en matinée et durant la journée, pas avant d’aller dormir. À savoir, lorsqu’on dort couché sur le dos le nez a tendance à se boucher, et quand on dort à plat ventre il a tendance à couler.

Bref, aux grands mucus les grands moyens.

puking rainbows

by nekohayo at October 26, 2016 01:29 PM

October 25, 2016

Jean-François Fortin Tam2015 Annual Reports mailed out

While the latest GNOME annual reports sold like hotcakes at GUADEC, there is still a need to send some of them by snailmail, like I did last year. I was under a pretty big rush from July to mid-October, and since nobody was available to help me determine the list of recipients, I had to wait for the end of the rush to allow myself to spend time devising that list. And so I did, this month.

2016-10-24--15.22.19

Packaged reports just before I shipped them out.

The shipping cost will be roughly half of last year’s, but it still turned out to be a surprising amount of copies to be sent out. Thanks to Adelia, Cassandra and Nuritzi (from the Engagement team) for writing postcards to accompany the reports, that saved me quite a bit of time!

You thought the annual reports weighted 3 MB? You were wrong. They weigh 3.8 kg.

You thought the annual reports weighted 3 MB?
You were wrong. They weigh 3.8 kg.

by nekohayo at October 25, 2016 03:00 PM

October 12, 2016

Andy WingoAn incomplete history of language facilities for concurrency

(Andy Wingo)

I have lately been in the market for better concurrency facilities in Guile. I want to be able to write network servers and peers that can gracefully, elegantly, and efficiently handle many tens of thousands of clients and other connections, but without blowing the complexity budget. It's a hard nut to crack.

Part of the problem is implementation, but a large part is just figuring out what to do. I have often thought that modern musicians must be crushed under the weight of recorded music history, but it turns out in our humble field that's also the case; there are as many concurrency designs as languages, just about. In this regard, what follows is an incomplete, nuanced, somewhat opinionated history of concurrency facilities in programming languages, with an eye towards what I should "buy" for the Fibers library I have been tinkering on for Guile.

* * *

Modern machines have the raw capability to serve hundreds of thousands of simultaneous long-lived connections, but it’s often hard to manage this at the software level. Fibers tries to solve this problem in a nice way. Before discussing the approach taken in Fibers, it’s worth spending some time on history to see how we got here.

One of the most dominant patterns for concurrency these days is “callbacks”, notably in the Twisted library for Python and the Node.js run-time for JavaScript. The basic observation in the callback approach to concurrency is that the efficient way to handle tens of thousands of connections at once is with low-level operating system facilities like poll or epoll. You add all of the file descriptors that you are interested in to a “poll set” and then ask the operating system which ones are readable or writable, as appropriate. Once the operating system says “yes, file descriptor 7145 is readable”, you can do something with that socket; but what? With callbacks, the answer is “call a user-supplied closure”: a callback, representing the continuation of the computation on that socket.

Building a network service with a callback-oriented concurrency system means breaking the program into little chunks that can run without blocking. Whereever a program could block, instead of just continuing the program, you register a callback. Unfortunately this requirement permeates the program, from top to bottom: you always pay the mental cost of inverting your program’s control flow by turning it into callbacks, and you always incur run-time cost of closure creation, even when the particular I/O could proceed without blocking. It’s a somewhat galling requirement, given that this contortion is required of the programmer, but could be done by the compiler. We Schemers demand better abstractions than manual, obligatory continuation-passing-style conversion.

Callback-based systems also encourage unstructured concurrency, as in practice callbacks are not the only path for data and control flow in a system: usually there is mutable global state as well. Without strong patterns and conventions, callback-based systems often exhibit bugs caused by concurrent reads and writes to global state.

Some of the problems of callbacks can be mitigated by using “promises” or other library-level abstractions; if you’re a Haskell person, you can think of this as lifting all possibly-blocking operations into a monad. If you’re not a Haskeller, that’s cool, neither am I! But if your typey spidey senses are tingling, it’s for good reason: with promises, your whole program has to be transformed to return promises-for-values instead of values anywhere it would block.

An obvious solution to the control-flow problem of callbacks is to use threads. In the most generic sense, a thread is a language feature which denotes an independent computation. Threads are created by other threads, but fork off and run independently instead of returning to their caller. In a system with threads, there is implicitly a scheduler somewhere that multiplexes the threads so that when one suspends, another can run.

In practice, the concept of threads is often conflated with a particular implementation, kernel threads. Kernel threads are very low-level abstractions that are provided by the operating system. The nice thing about kernel threads is that they can use any CPU that is the kernel knows about. That’s an important factor in today’s computing landscape, where Moore’s law seems to be giving us more cores instead of more gigahertz.

However, as a building block for a highly concurrent system, kernel threads have a few important problems.

One is that kernel threads simply aren’t designed to be allocated in huge numbers, and instead are more optimized to run in a one-per-CPU-core fashion. Their memory usage is relatively high for what should be a lightweight abstraction: some 10 kilobytes at least and often some megabytes, in the form of the thread’s stack. There are ongoing efforts to reduce this for some systems but we cannot expect wide deployment in the next 5 years, if ever. Even in the best case, a hundred thousand kernel threads will take at least a gigabyte of memory, which seems a bit excessive for book-keeping overhead.

Kernel threads can be a bit irritating to schedule, too: when one thread suspends, it’s for a reason, and it can be that user-space knows a good next thread that should run. However because kernel threads are scheduled in the kernel, it’s rarely possible for the kernel to make informed decisions. There are some “user-mode scheduling” facilities that are in development for some systems, but again only for some systems.

The other significant problem is that building non-crashy systems on top of kernel threads is hard to do, not to mention “correct” systems. It’s an embarrassing situation. For one thing, the low-level synchronization primitives that are typically provided with kernel threads, mutexes and condition variables, are not composable. Also, as with callback-oriented concurrency, one thread can silently corrupt another via unstructured mutation of shared state. It’s worse with kernel threads, though: a kernel thread can be interrupted at any point, not just at I/O. And though callback-oriented systems can theoretically operate on multiple CPUs at once, in practice they don’t. This restriction is sometimes touted as a benefit by proponents of callback-oriented systems, because in such a system, the callback invocations have a single, sequential order. With multiple CPUs, this is not the case, as multiple threads can run at the same time, in parallel.

Kernel threads can work. The Java virtual machine does at least manage to prevent low-level memory corruption and to do so with high performance, but still, even Java-based systems that aim for maximum concurrency avoid using a thread per connection because threads use too much memory.

In this context it’s no wonder that there’s a third strain of concurrency: shared-nothing message-passing systems like Erlang. Erlang isolates each thread (called processes in the Erlang world), giving each it its own heap and “mailbox”. Processes can spawn other processes, and the concurrency primitive is message-passing. A process that tries receive a message from an empty mailbox will “block”, from its perspective. In the meantime the system will run other processes. Message sends never block, oddly; instead, sending to a process with many messages pending makes it more likely that Erlang will pre-empt the sending process. It’s a strange tradeoff, but it makes sense when you realize that Erlang was designed for network transparency: the same message send/receive interface can be used to send messages to processes on remote machines as well.

No network is truly transparent, however. At the most basic level, the performance of network sends should be much slower than local sends. Whereas a message sent to a remote process has to be written out byte-by-byte over the network, there is no need to copy immutable data within the same address space. The complexity of a remote message send is O(n) in the size of the message, whereas a local immutable send is O(1). This suggests that hiding the different complexities behind one operator is the wrong thing to do. And indeed, given byte read and write operators over sockets, it’s possible to implement remote message send and receive as a process that serializes and parses messages between a channel and a byte sink or source. In this way we get cheap local channels, and network shims are under the programmer’s control. This is the approach that the Go language takes, and is the one we use in Fibers.

Structuring a concurrent program as separate threads that communicate over channels is an old idea that goes back to Tony Hoare’s work on “Communicating Sequential Processes” (CSP). CSP is an elegant tower of mathematical abstraction whose layers form a pattern language for building concurrent systems that you can still reason about. Interestingly, it does so without any concept of time at all, instead representing a thread’s behavior as a trace of instantaneous events. Threads themselves are like functions that unfold over the possible events to produce the actual event trace seen at run-time.

This view of events as instantaneous happenings extends to communication as well. In CSP, one communication between two threads is modelled as an instantaneous event, partitioning the traces of the two threads into “before” and “after” segments.

Practically speaking, this has ramifications in the Go language, which was heavily inspired by CSP. You might think that a channel is just a an asynchronous queue that blocks when writing to a full queue, or when reading from an empty queue. That’s a bit closer to the Erlang conception of how things should work, though as we mentioned, Erlang simply slows down writes to full mailboxes rather than blocking them entirely. However, that’s not what Go and other systems in the CSP family do; sending a message on a channel will block until there is a receiver available, and vice versa. The threads are said to “rendezvous” at the event.

Unbuffered channels have the interesting property that you can select between sending a message on channel a or channel b, and in the end only one message will be sent; nothing happens until there is a receiver ready to take the message. In this way messages are really owned by threads and never by the channels themselves. You can of course add buffering if you like, simply by making a thread that waits on either sends or receives on a channel, and which buffers sends and makes them available to receives. It’s also possible to add explicit support for buffered channels, as Go, core.async, and many other systems do, which can reduce the number of context switches as there is no explicit buffer thread.

Whether to buffer or not to buffer is a tricky choice. It’s possible to implement singly-buffered channels in a system like Erlang via an explicit send/acknowlege protocol, though it seems difficult to implement completely unbuffered channels. As we mentioned, it’s possible to add buffering to an unbuffered system by the introduction of explicit buffer threads. In the end though in Fibers we follow CSP’s lead so that we can implement the nice select behavior that we mentioned above.

As a final point, select is OK but is not a great language abstraction. Say you call a function and it returns some kind of asynchronous result which you then have to select on. It could return this result as a channel, and that would be fine: you can add that channel to the other channels in your select set and you are good. However, what if what the function does is receive a message on a channel, then do something with the message? In that case the function should return a channel, plus a continuation (as a closure or something). If select results in a message being received over that channel, then we call the continuation on the message. Fine. But, what if the function itself wanted to select over some channels? It could return multiple channels and continuations, but that becomes unwieldy.

What we need is an abstraction over asynchronous operations, and that is the main idea of a CSP-derived system called “Concurrent ML” (CML). Originally implemented as a library on top of Standard ML of New Jersey by John Reppy, CML provides this abstraction, which in Fibers is called an operation1. Calling send-operation on a channel returns an operation, which is just a value. Operations are like closures in a way; a closure wraps up code in its environment, which can be later called many times or not at all. Operations likewise can be performed2 many times or not at all; performing an operation is like calling a function. The interesting part is that you can compose operations via the wrap-operation and choice-operation combinators. The former lets you bundle up an operation and a continuation. The latter lets you construct an operation that chooses over a number of operations. Calling perform-operation on a choice operation will perform one and only one of the choices. Performing an operation will call its wrap-operation continuation on the resulting values.

While it’s possible to implement Concurrent ML in terms of Go’s channels and baked-in select statement, it’s more expressive to do it the other way around, as that also lets us implement other operations types besides channel send and receive, for example timeouts and condition variables.


1 CML uses the term event, but I find this to be a confusing name. In this isolated article my terminology probably looks confusing, but in the context of the library I think it can be OK. The jury is out though.

2 In CML, synchronized.

* * *

Well, that's my limited understanding of the crushing weight of history. Note that part of this article is now in the Fibers manual.

Thanks very much to Matthew Flatt, Matthias Felleisen, and Michael Sperber for pushing me towards CML. In the beginning I thought its benefits were small and complication large, but now I see it as being the reverse. Happy hacking :)

by Andy Wingo at October 12, 2016 01:45 PM

October 09, 2016

Thomas Vander StichelePuppet/puppetdb/storeconfigs validation issues

(Thomas Vander Stichele)

Over the past year I’ve chipped away at setting up new servers for apestaart and managing the deployment in puppet as opposed to a by now years old manual single server configuration that would be hard to replicate if the drives fail (one of which did recently, making this more urgent).

It’s been a while since I felt like I was good enough at puppet to love and hate it in equal parts, but mostly manage to control a deployment of around ten servers at a previous job.

Things were progressing an hour or two here and there at a time, and accelerated when a friend in our collective was launching a new business for which I wanted to make sure he had a decent redundancy setup.

I was saving the hardest part for last – setting up Nagios monitoring with Matthias Saou’s puppet-nagios module, which needs External Resources and storeconfigs working.

Even on the previous server setup based on CentOS 6, that was a pain to set up – needing MySQL and ruby’s ActiveRecord. But it sorta worked.

It seems that for newer puppet setups, you’re now supposed to use something called PuppetDB, which is not in fact a database on its own as the name suggests, but requires another database. Of course, it chose to need a different one – Postgres. Oh, and PuppetDB itself is in Java – now you get the cost of two runtimes when you use puppet!

So, to add useful Nagios monitoring to my puppet deploys, which without it are quite happy to be simple puppet apply runs from a local git checkout on each server, I now need storedconfigs which needs puppetdb which pulls in Java and Postgres. And that’s just so a system that handles distributed configuration can actually be told about the results of that distributed configuration and create a useful feedback cycle allowing it to do useful things to the observed result.

Since I test these deployments on local vagrant/VirtualBox machines, I had to double their RAM because of this – even just the puppetdb java server by default starts with 192MB reserved out of the box.

But enough complaining about these expensive changes – at least there was a working puppetdb module that managed to set things up well enough.

It was easy enough to get the first host monitored, and apart from some minor changes (like updating the default Nagios config template from 3.x to 4.x), I had a familiar Nagios view working showing results from the server running Nagios itself. Success!

But all runs from the other vm’s did not trigger adding any exported resources, and I couldn’t find anything wrong in the logs. In fact, I could not find /var/log/puppetdb/puppetdb.log at all…

fun with utf-8

After a long night of experimenting and head scratching, I chased down a first clue in /var/log/messages saying puppet-master[17702]: Ignoring invalid UTF-8 byte sequences in data to be sent to PuppetDB

I traced that down to puppetdb/char_encoding.rb, and with my limited ruby skills, I got a dump of the offending byte sequence by adding this code:

Puppet.warning "Ignoring invalid UTF-8 byte sequences in data to be sent to PuppetDB"
File.open('/tmp/ruby', 'w') { |file| file.write(str) }
Puppet.warning "THOMAS: is here"

(I tend to use my name in debugging to have something easy to grep for, and I wanted some verification that the File dump wasn’t triggering any errors)
It took a little time at 3AM to remember where these /tmp files end up thanks to systemd, but once found, I saw it was a json blob with a command to “replace catalog”. That could explain why my puppetdb didn’t have any catalogs for other hosts. But file told me this was a plain ASCII file, so that didn’t help me narrow it down.

I brute forced it by just checking my whole puppet tree:


find . -type f -exec file {} \; > /tmp/puppetfile
grep -v ASCII /tmp/puppetfile | grep -v git

This turned up a few UTF-8 candidates. Googling around, I was reminded about how terrible utf-8 handling was in ruby 1.8, and saw information that puppet recommended using ASCII only in most of the manifests and files to avoid issues.

It turned out to be a config from a webalizer module:

webalizer/templates/webalizer.conf.erb: UTF-8 Unicode text

While it was written by a Jesús with a unicode name, the file itself didn’t have his name in it, and I couldn’t obviously find where the UTF-8 chars were hiding. One StackOverflow post later, I had nailed it down – UTF-8 spaces!

00004ba0 2e 0a 23 c2 a0 4e 6f 74 65 20 66 6f 72 20 74 68 |..#..Note for th|
00004bb0 69 73 20 74 6f 20 77 6f 72 6b 20 79 6f 75 20 6e |is to work you n|

The offending character is c2 a0 – the non-breaking space

I have no idea how that slipped into a comment in a config file, but I changed the spaces and got rid of the error.

Puppet’s error was vague, did not provide any context whatsoever (Where do the bytes come from? Dump the part that is parseable? Dump the hex representation? Tell me the position in it where the problem is?), did not give any indication of the potential impact, and in a sea of spurious puppet warnings that you simply have to live with, is easy to miss. One down.

However, still no catalogs on the server, so still only one host being monitored. What next?

users, groups, and permissions

Chasing my next lead turned out to be my own fault. After turning off SELinux temporarily, checking all permissions on all puppetdb files to make sure that they were group-owned by puppetdb and writable for puppet, I took the last step of switching to that user role and trying to write the log file myself. And it failed. Huh? And then id told me why – while /var/log/puppetdb/ was group-writeable and owned by puppetdb group, my puppetdb user was actually in the www-data group.

It turns out that I had tried to move some uids and gids around after the automatic assignment puppet does gave different results on two hosts (a problem I still don’t have a satisfying answer for, as I don’t want to hard-code uids/gids for system accounts in other people’s modules), and clearly I did one of them wrong.

I think a server that for whatever reason cannot log should simply not start, as this is a critical error if you want a defensive system.

After fixing that properly, I now had a puppetdb log file.

resource titles

Now I was staring at an actual exception:

2016-10-09 14:39:33,957 ERROR [c.p.p.command] [85bae55f-671c-43cf-9a54-c149cede
c659] [replace catalog] Fatal error on attempt 0
java.lang.IllegalArgumentException: Resource '{:type "File", :title "/var/lib/p
uppet/concat/thomas_vimrc/fragments/75_thomas_vimrc-\" allow adding additional
config through .vimrc.local_if filereadable(glob(\"~_.vimrc.local\"))_\tsource
~_.vimrc.local_endif_"}' has an invalid tag 'thomas:vimrc-" allow adding additi
onal config through .vimrc.local
if filereadable(glob("~/.vimrc.local"))
source ~/.vimrc.local
endif
'. Tags must match the pattern /\A[a-z0-9_][a-z0-9_:\-.]*\Z/.
at com.puppetlabs.puppetdb.catalogs$validate_resources.invoke(catalogs.
clj:331) ~[na:na]

Given the name of the command (replace catalog), I felt certain this was going to be the problem standing between me and multiple hosts being monitored.

The problem was a few levels deep, but essentially I had code creating fragments of vimrc files using the concat module, and was naming the resources with file content as part of the title. That’s not a great idea, admittedly, but no other part of puppet had ever complained about it before. Even the files on my file system that store the fragments, which get their filename from these titles, happily stored with a double quote in its name.

So yet again, puppet’s lax approach to specifying types of variables at any of its layers (hiera, puppet code, ruby code, ruby templates, puppetdb) in any of its data formats (yaml, json, bytes for strings without encoding information) triggers errors somewhere in the stack without informing whatever triggered that error (ie, the agent run on the client didn’t complain or fail).

Once again, puppet has given me plenty of reasons to hate it with a passion, tipping the balance.

I couldn’t imagine doing server management without a tool like puppet. But you love it when you don’t have to tweak it much, and you hate it when you’re actually making extensive changes. Hopefully after today I can get back to the loving it part.

flattr this!

by Thomas at October 09, 2016 08:31 PM

Jean-François Fortin TamBack from the GNOME Summit

In my previous blog post, where I was providing an update on the 2016 GNOME Summit I was organizing in Montréal, I wrote,

With a change of attendees comes a change of the nature of the event: instead of being an extremely technical, “deep end of the pool” event as it has been in the past, this edition will be focused on newcomers and potential contributors, with presentations and workshops targetted for this purpose.

In practice, it turned out to be a great event. It was halfway between the traditional highly technical gathering and the “event aimed at new contributors”, with 13 attendees including myself.

ets-maison-du-logiciel-libre-2015

After welcoming participants and opening the event with a rough action plan, I gave a short presentation to set the stage and provide an overview of what GNOME is (a collection of software, a community, and the GNOME Foundation). The point was to have an icebreaker talk that would ensure all participants are on the same level of understanding as to who they were dealing with. This eventually morphed into a joint talk with Hubert that looked more like a Saturday Night Live show with two hosts sitting in fancy chairs in front of the audience (wish I had pictures of the two of us taken from the outside to illustrate the “TV show” feel).

2016-10-08--13.58.13

My co-animator

We gradually got people to open up and get involved by probing participants with questions. After all, the chief goal is to motivate people into contributing to the GNOME, showing them the various ways they can get involved, how the GNOME Project’s release cycle works (and where the timing of their contributions comes in) and what technologies are involved. We sang the praise of Flatpak and GNOME Builder (even though he uses Emacs and I use Gedit ;)

Our introductory discussion was then followed by a presentation by Guillaume Poirier-Morency on Vala, Meson, pollcore and Valum.

2016-10-08--14.50.45

Guillaume presenting

David Mirza Ahmad also presented Subgraph OS, the problem it aims to solve and how GNOME fits into the picture. Their distro runs a hardened GNOME3 and it’s quite relevant to our interests. One could have a fun week-end in Manchester with these folks!

2016-10-08--15.19.03

David talking about Subgraph, security, and GNOME

It seems the Subgraph talk captured attendees’ interest and ended up morphing into a security discussion that took the rest of the afternoon.

While I initially thought we wouldn’t have enough content to fill more than a day, we ended up not having time to do newcomers workshops (short & sweet, as I would say). There was a shared interest among participants to try creating a GNOME community in Montréal to have such meetings more often. So, for starters, I have now created the #gnome-montreal IRC channel on irc.gnome.org. Get your butt over there if you’re in the area.

I would like to thank ÉTS and ML2 for providing us with a nice meeting space for the event, and the Subgraph folks for sponsoring dinner on Saturday night (a very nice touch), where lively debates were facilitated by beer and food.

by nekohayo at October 09, 2016 12:53 PM

October 03, 2016

GStreamerGStreamer Core, Plugins, RTSP Server, Editing Services, Python, Validate, VAAPI, OMX 1.10.0 release candidate 1 (1.9.90) (binaries)

(GStreamer)

Pre-built binary images of the 1.9.90 release candidate of GStreamer are now available for Windows 32/64-bit, iOS and Mac OS X and Android.

The builds are available for download from: Android, iOS, Mac OS X and Windows.

October 03, 2016 12:00 AM

September 30, 2016

Jean-François Fortin TamDownscaling the Summit

Due to unexpectedly low international attendance numbers this year, we have decided to scale down the 2016 GNOME Summit to one day, most likely to be on the afternoon of Saturday the 8th instead of three whole days. I have updated the wiki page accordingly; some small changes might still occur on that page as plans settle down, but that should be the jist of it. With a change of attendees comes a change of the nature of the event: instead of being an extremely technical, “deep end of the pool” event as it has been in the past, this edition will be focused on newcomers and potential contributors, with presentations and workshops targetted for this purpose. We might even speak French or Python!

by nekohayo at September 30, 2016 01:49 PM

GStreamerGStreamer Core, Plugins, RTSP Server, Editing Services, Python, Validate, VAAPI, OMX 1.10.0 release candidate 1 (1.9.90)

(GStreamer)

The GStreamer team is pleased to announce the first release candidate of the stable 1.10 release series. The 1.10 release series is adding new features on top of the 1.0, 1.2, 1.4, 1.6 and 1.8 series and is part of the API and ABI-stable 1.x release series of the GStreamer multimedia framework.

Binaries for Android, iOS, Mac OS X and Windows will be provided in the next days.

Check out the release notes for GStreamer core, gst-plugins-base, gst-plugins-good, gst-plugins-ugly, gst-plugins-bad, gst-libav, gst-rtsp-server, gst-python, gst-editing-services, gst-validate, gstreamer-vaapi, or gst-omx, or download tarballs for gstreamer, gst-plugins-base, gst-plugins-good, gst-plugins-ugly, gst-plugins-bad, gst-libav, gst-rtsp-server, gst-python, gst-editing-services, gst-validate, gstreamer-vaapi, or gst-omx.

September 30, 2016 10:00 AM

September 21, 2016

Andy Wingois go an acceptable cml?

(Andy Wingo)

Yesterday I tried to summarize the things I know about Concurrent ML, and I came to the tentative conclusion that Go (and any Go-like system) was an acceptable CML. Turns out I was both wrong and right.

you were wrong when you said everything's gonna be all right

I was wrong, in the sense that programming against the CML abstractions lets you do more things than programming against channels-and-goroutines. Thanks to Sam Tobin-Hochstadt to pointing this out. As an example, consider a little process that tries to receive a message off a channel, and times out otherwise:

func withTimeout(ch chan int, timeout int) (result int) {
  var timeoutChannel chan int;
  var msg int;
  go func() {
    sleep(timeout)
    timeoutChannel <- 0
  }()
  select {
    case msg = <-ch: return msg;
    case msg = <-timeoutChannel: return 0;
  }
}

I think that's the first Go I've ever written. I don't even know if it's syntactically valid. Anyway, I think we see how it should work. We return the message from the channel, unless the timeout happens before.

But, what if the message is itself a composite message somehow? For example, say we have a transformer that reads a value from a channel and adds 1 to it:

func onePlus(in chan int) (result chan int) {
  var out chan int
  go func () { out <- 1 + <-in }()
  return out
}

What if we do a withTimeout(onePlus(numbers), 0)? Assume the timeout fires first and that's the result that select chooses. There's still that onePlus goroutine out there trying to read from in and at some point probably it will succeed, but nobody will read its value. At that point the number just vanishes into the ether. Maybe that's OK in certain domains, but certainly not in general!

What CML gives you is the ability to express an event (which is kinda like a possibility of sending or receiving a message on a channel) in such a way that we don't run into this situation. Specifically with the wrap combinator, we would make an event such that receiving on numbers would run a function on the received message and return that as the message value -- which is of course the same as what we have, except that in CML the select wouldn't actually read the message off unless it select'd that channel for input.

Of course in Go you could just rewrite your program, so that the select statement looks like this:

select {
  case msg = <-ch: return msg + 1;
  case msg = <-timeoutChannel: return 0;
}

But here we're operating at a lower level of abstraction; we were forced to intertwingle our concerns of adding 1 and our concerns of timeout. CML is more expressive than Go.

you were right when you said we're all just bricks in the wall

However! I was right in the sense that you can build a CML system on top of Go-like systems (though possibly not Go in particular). Thanks to Vesa Karvonen for this comment and the link to their proof-of-concept CML implementation in Clojure's core.async. I understand Vesa also has an implementation in F# as well.

Folks should read Vesa's code, after reading the Reppy papers of course; it's delightfully short and expressive. The basic idea is that event composition operators like choose and wrap build up data structures instead of doing things. The sync operation then grovels through those data structures to collect a list of channels to pass on to core.async's equivalent of select. When select returns, sync determines which event that chosen channel and message corresponds to, and proceeds to "activate" the event (and, as a side effect, possibly issue NACK messages to other channels).

Provided you can map from the chosen select channel/message back to the event, (something that core.async can mostly do, with a caveat; see the code), then you can build CML on top of channels and goroutines.

o/~ yeah you were wrong o/~

On the other hand! One advantage of CML is that its events are not limited to channel sends and receives. I understand that timeouts, thread joins, and maybe some other event types are first-class event kinds in many CML systems. Michael Sperber, current Scheme48 maintainer and functional programmer, tells me that simply wrapping events in channels+goroutines works but can incur a big performance overhead relative to supporting those event types natively, due to the need to make the new goroutine and channel and the scheduling costs. He quotes 10X as the overhead!

So although CML and Go appear to be inter-expressible, maybe a proper solution will base the simple channel send/receive interface on CML rather than the other way around.

Also, since these events are now second-class, it must be OK to lose these events, for the same reason that the naïve withTimeout could lose a message from numbers. This is the case for timeouts usually but maybe you have to think about this more, and possibly provide an infinite stream of the message. (Of course the wrapper goroutine would be collected if the channel becomes unreachable.)

you were right when you said this is the end

I've long wondered how contemporary musicians deal with the enormous, crushing weight of recorded music. I don't really pick any more but hoo am I feeling this now. I think for Guile, I will continue hacking on fibers in a separate library, and I think that things will remain that way for the next couple years and possibly more. We need more experience and more mistakes before blessing and supporting any particular formulation of highly concurrent programming. I will say though that I am delighted that we are able to actually do this experimentation on a library level and I look forward to seeing what works out :)

Thanks again to Vesa, Michael, and Sam for sharing their time and knowledge; all errors are of course mine. Happy hacking!

by Andy Wingo at September 21, 2016 09:29 PM

September 20, 2016

Andy Wingoconcurrent ml versus go

(Andy Wingo)

Peoples! Lately I've been navigating the guile-ship through waters unknown. This post is something of an echolocation to figure out where the hell this ship is and where it should go.

Concretely, I have been working on getting a nice lightweight concurrency system rolling for Guile. I'll write more about that later, but you can think of it as being modelled on Go, though built as a library. (I had previously described it as "Erlang-like", but that's just not accurate.)

Earlier this year at Curry On this topic was burning in my mind and of course when I saw the language-hacker fam there I had to bend their ears. My targets: Matthew Flatt, the amazing boundary-crossing engineer, hacker, teacher, researcher, and implementor of Racket, and Matthias Felleisen, the godfather of the PLT research family. I saw them sitting together and I thought, you know what, what can they have to say to each other? These people have been talking together for 30 years right? Surely they are actually waiting for some ignorant dude to saunter up to the PL genius bar, right?

So saunter I do, saying, "if someone says to you that they want to build a server that will handle 100K or so simultaneous connections on Racket, what abstraction do you tell them to use? Racket threads?" Apparently: yes. A definitive yes, in the case of Matthias, with a pointer to Robby Findler's paper on kill-safe abstractions; and still a yes from Matthew with the caveat that for the concrete level of concurrency that I described, you'd have to run tests. More fundamentally, I was advised to look at Concurrent ML (on which Racket's concurrency facilities were based), that CML was much better put together than many modern variants like Go.

This was very interesting and new to me. As y'all probably know, I don't have a formal background in programming languages, and although I've read a lot of literature, reading things only makes you aware of the growing dimension of the not-yet-read. Concurrent ML was even beyond my not-yet-read horizon.

So I went back and read a bunch of papers. Turns out Concurrent ML is like Lisp in that it has a tribe and a tightly-clutched history and a diaspora that reimplements it in whatever language they happen to be working in at the moment. Kinda cool, and, um... a bit hard to appreciate in the current-day context when the only good references are papers from 10 or 20 years ago.

However, after reading a bunch of John Reppy papers, here is my understanding of what Concurrent ML is. I welcome corrections; surely I am getting this wrong.

1. CML is like Go, composed of channels and goroutines. (Forgive the modern referent; I assume most folks know Go at this point.)

2. Unlike Go, in CML a channel is never buffered. To make a buffered channel in CML, you spawn a thread that manages a buffer between two channels.

3. Message send and receive operations in CML are built on a lower-level primitive called "events". (send ch x) is instead euivalent to (sync (send-event ch x)). It's like an event is the derivative of a message send with respect to time, or something.

4. Events can be combined and transformed using the choose and wrap combinators.

5. Doing a sync on an event created by choose allows a user to build select in "user-space", as a library. Cool stuff. So this is what events are for.

6. There are separate event type implementations for timeouts, channel send/recv blocking operations, file descriptor blocking operations, syscalls, thread joins, and the like. These are supported by the CML implementation.

7. The early implementations of Concurrent ML were concurrent but not parallel; they did not run multiple "goroutines" on separate CPU cores at the same time. It was only in like 2009 that people started to do CML in parallel. I do not know if this late parallelism has a practical impact on the viability of CML.

ok go

What is the relationship of CML to Go? Specifically, is CML more expressive than Go? (I assume the reverse is not the case, but that would also be an interesting result!)

There are a few languages that only allow you to select over message receives (not sends), but Go's select doesn't have this limitation, so that's not a differentiator.

Some people say that it's nice to have events as the common denominator, but I don't get this argument. If the only event under consideration is message send or receive over a channel, events + choose + sync is the same in expressive power as a built-in select, as far as I can see. If there are other events, then your runtime already has to support them either way, and something like (let ((ch (make-channel))) (spawn-fiber (lambda () (put-message ch exp))) (get-message ch)) should be sufficient for any runtime-supported event in exp, like sleeps or timeouts or thread joins or whatever.

To me it seems like Go has made the right choices here. I do not see the difference, and that's why I wrote all this, is to be shown the error of my ways. Choosing channels, send, receive, and select as the primitives seems to have the same power as SML events.

Let this post be a pentagram on the floor, then, to summon the CML cognoscenti. Well-actuallies are very welcome; hit me up in the comments!

[edit: Sam Tobin-Hochstadt tells me I got it wrong and I believe him :) In the meantime while I work out how I was wrong, examples are welcome!]

by Andy Wingo at September 20, 2016 09:33 PM

September 15, 2016

GStreamerGStreamer Conference 2016: Last chance for early-bird discount on tickets

(GStreamer)

This is a quick reminder that registration for the GStreamer conference 2016 is open, and if you register today you can still benefit from the discounted early-bird registration fee, which is only available until Thursday 15 September 2016 (inclusive). After that the registration fee for professional tickets will rise to 340 EUR.

Register now for the GStreamer Conference!

GStreamer Conference 2016 Berlin

About the GStreamer Conference

The GStreamer Conference 2016 will take place on 10-11 October 2016 in Berlin (Germany), and will take place in the same week as the Embedded Linux Conference Europe. More information and details how to register can be found on the conference website.

September 15, 2016 10:00 AM

September 14, 2016

GStreamerGStreamer Conference 2016: Collabora Platinum Sponsor

(GStreamer)

The GStreamer project is pleased to welcome back Collabora as Platinum level sponsor at this year's GStreamer Conference in Berlin.

Collabora (https://www.collabora.com) is a consultancy with more than 10 years of experience in open source technologies. As well as employing several core contributors, they have been sponsoring the GStreamer conference for every year since the very first conference.

Thanks Collabora!

Collabora

About the GStreamer Conference

The GStreamer Conference 2016 will take place on 10-11 October 2016 in Berlin (Germany), and will take place in the same week as the Embedded Linux Conference Europe. More information and details how to register can be found on the conference website.

September 14, 2016 01:00 PM

September 11, 2016

Jean-François Fortin TamTaking down my online portfolio

Cleaning up my apartment today, hoping to get rid of a pile of draft papers that has been cluttering my space for six months, I’m taking the opportunity to write about what this particular pile of paper means (yes, my blogging backlog goes that far—I am draining the swamp one post at a time!)

2016-09-11--10.40.19

Pictured: my floor, littered with intermediate drafts of my new portfolio, along with a few printer calibration photos (upper-left) and a copy of the new Annual Report sitting around

For the longest of times, as a designer flying under the radar (my management and marketing work is more “visible”, ironically), I did not have a proper portfolio. In fact, I had made a portfolio only once in my life, assembled in a few days in 2011. While my work as idéemarque in the past few years has led me to do a fair amount of design work, I was so busy with business work in general that I never took the time to do the massive amount of research and planning required to make a new one.

In early 2016, I finally sat down and made a new portfolio. Those who have worked with me know I’m a perfectionist, and since I had finally set my mind on doing this, I would be doing it right. It indeed took me 2-3 months of research, writing, designing, revising, redesigning, print testing, revising again and fixing issues, until I ran out of issues to fix.

From the start, I decided that it would be a top-quality print-only edition. I used beautiful, legible and readable fonts to make reading on paper a delightful experience. I invested in premium quality paper (including some semi-transparent sheets) and used a high-end printer, but not a commercial printing press as that would have been ridiculously expensive and inflexible for printing only a handful of custom-made copies. A print shop couldn’t have provided all the fancy features I required anyway (fancy binding, specialty paper, embossing, etc.). Therefore each of my portfolios is a hand-made craft—even if it might not seem that way as I made it full bleed (edge to edge). This whole project made me learn way too much about troubleshooting printer color problems, printing on unconventional paper types, faking bleeds, and the traditional bookbinding techniques of my ancestors.

Pictured: some really thick paper used for the portfolio

Pictured: some really thick and heavy paper used for the portfolio

In 2016, a printed portfolio might seem outdated compared to an online portfolio. Yet, mine is designed for print, precisely because the web and cheap emails have become the norm in this era of constant noise, and because I prefer to control everything in the reader’s experience—including the delivery, the typography, the texture, size, depth, resolution, and the physical (and mental) weight of the document. And not having to “/%$?%*)&!?” around with CSS and “modern” web development is a huge plus.

CSS is awesome mug

Pictured: not my mug

Some said, “How can this be such an involved process, taking you months?” to which I replied that, among other things, the document was roughly thirty pages. This got me some expressions of bewilderment, until I explained, “It’s not a graphic design portfolio, it’s a human-computer interaction design research portfolio.”

While it took a long time, the project was worth it, both as a personal challenge and from the reactions I got afterwards.

I have now decided to remove any traces of my portfolio from my website (except the photo and illustrations gallery). A website is not only a pain in the ass to maintain regularly, it’s typically ignored or skimmed anyway (basically, Schrödinger’s Visitor). For those who requested an electronic version of my portfolio over the past few months, I have insisted on shipping them the print version instead—tailored for them, of course.

by nekohayo at September 11, 2016 06:38 PM