Planet GStreamer
http://gstreamer.freedesktop.org/planet/
Planet GStreamer - http://gstreamer.freedesktop.org/planet/GStreamer: GStreamer 1.22.8 stable bug fix release
https://gstreamer.freedesktop.org/news/#2023-12-18T14:00:00Z
<p>
The GStreamer team is pleased to announce another bug fix release
in the stable 1.22 release series of your favourite cross-platform
multimedia framework!
</p><p>
This release only contains bugfixes and security fixes and it should be safe
to update from 1.22.x.
</p><p>
<b>Highlighted bugfixes:</b>
<ul>
<li><a href="https://gstreamer.freedesktop.org/security/">Security fixes</a> for the AV1 codec parser</li>
<li>[Security fixes](https://gstreamer.freedesktop.org/security/) for the AV1 video codec parser</li> <!-- FIXME: update advisory -->
<li>avdec video decoder: fix another possible deadlock with FFmpeg 6.1</li>
<li>qtdemux: reverse playback and seeking fixes for files with raw audio streams</li>
<li>v4l2: fix "newly allocated buffer ... is not free" warning log flood</li>
<li>GstPlay + GstPlayer library fixes</li>
<li>dtls: Fix build failure on Windows when compiling against OpenSSL 3.2.0</li>
<li>d3d11screencapturesrc: Fix wrong color with HDR enabled</li>
<li>Cerbero build tool: More python 3.12 string escape warning fixes; make sure to bundle build tools as well</li>
<li>various bug fixes, build fixes, memory leak fixes, and other stability and reliability improvements</li>
</ul>
</p><p>
See the <a href="https://gstreamer.freedesktop.org/releases/1.22/#1.22.8">GStreamer 1.22.8 release notes</a>
for more details.
</p><p>
Binaries for Android, iOS, Mac OS X and Windows will be available shortly.
</p><p>
Release tarballs can be downloaded directly here:
<ul>
<a href="https://gstreamer.freedesktop.org/src/gstreamer/gstreamer-1.22.8.tar.xz">gstreamer</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-base/gst-plugins-base-1.22.8.tar.xz">gst-plugins-base</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-good/gst-plugins-good-1.22.8.tar.xz">gst-plugins-good</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-ugly/gst-plugins-ugly-1.22.8.tar.xz">gst-plugins-ugly</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-bad/gst-plugins-bad-1.22.8.tar.xz">gst-plugins-bad</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-libav/gst-libav-1.22.8.tar.xz">gst-libav</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-rtsp-server/gst-rtsp-server-1.22.8.tar.xz">gst-rtsp-server</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-python/gst-python-1.22.8.tar.xz">gst-python</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-editing-services/gst-editing-services-1.22.8.tar.xz">gst-editing-services</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-devtools/gst-devtools-1.22.8.tar.xz">gst-devtools</a>,
<a href="https://gstreamer.freedesktop.org/src/gstreamer-vaapi/gstreamer-vaapi-1.22.8.tar.xz">gstreamer-vaapi</a>,
<a href="https://gstreamer.freedesktop.org/src/gstreamer-sharp/gstreamer-sharp-1.22.8.tar.xz">gstreamer-sharp</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-omx/gst-omx-1.22.8.tar.xz">gst-omx</a>, or
<a href="https://gstreamer.freedesktop.org/src/gstreamer-docs/gstreamer-docs-1.22.8.tar.xz">gstreamer-docs</a>.
</ul>
</p>2023-12-18T14:00:00+00:00Andy Wingo: service update
https://wingolog.org/archives/2023/12/14/service-update
<div><p>Late last year I switched blog entries and comments to be written in <a href="https://github.com/wingo/tekuti/blob/master/tekuti/marxdown.scm">a dialect of markdown</a>, but there was a bug that I never noticed: if a text consisted only of a single paragraph or block, it would trigger an error that got reported back to the user in a very strange way, and which would prevent the comment from being posted.</p><p>I had never seen the error myself because blog posts are generally more than a paragraph, but it must have been quite irritating when commenting. Sorry about that; it should be fixed now. Should you experience more strange errors, please do send me an email with the comment to <tt>wingo@igalia.com</tt>. Cheers.</p></div>2023-12-14T14:28:45+00:00Andy WingoAndy Wingo: sir talks-a-lot
https://wingolog.org/archives/2023/12/12/sir-talks-a-lot
<div><p>I know, dear reader: of course you have already seen all my talks this year. Your attentions are really too kind and I thank you. But those other people, maybe you share one talk with them, and then they ask you for more, and you have to go stalking back through the archives to slake their nerd-thirst. This happens all the time, right?</p><p>I was thinking of you this morning and I said to myself, why don’t I put together a post linking to all of my talks in 2023, so that you can just send them a link; here we are. You are very welcome, it is really my pleasure.</p><h3>2023 talks</h3><p><i>Scheme + Wasm + GC = MVP: Hoot Scheme-to-Wasm compiler update.</i> Wasm
standards group, Munich, 11 Oct 2023.
<a href="https://wingolog.org/pub/wasm-cg-2023-10-scheme-slides.pdf">slides</a></p><p><i>Scheme to Wasm: Use and misuse of the GC proposal.</i>
Wasm GC subgroup, 18 Apr 2023.
<a href="https://wingolog.org/pub/wasm-gc-2023-04-scheme-slides.pdf">slides</a></p><p><i>A world to win: WebAssembly for the rest of us.</i>
BOB, Berlin, 17 Mar 2023.
<a href="https://wingolog.org/archives/2023/03/20/a-world-to-win-webassembly-for-the-rest-of-us">blog</a>
<a href="https://wingolog.org/pub/2023-bobkonf-wasm-for-the-rest-of-us-slides.pdf">slides</a>
<a href="https://www.youtube.com/watch?v=lUCegCa7A08&pp=ygUKYW5keSB3aW5nbw%3D%3D">youtube</a></p><p><i>Cross-platform mobile UI: “Compilers, compilers everywhere”.</i>
EOSS, Prague, 27 June 2023.
<a href="https://wingolog.org/pub/2023-eoss-cross-platform-ui-slides.pdf">slides</a>
<a href="https://www.youtube.com/watch?v=r6ctxZphtkI&pp=ygUKYW5keSB3aW5nbw%3D%3D">youtube</a>
<a href="https://wingolog.org/archives/2023/06/15/parallel-futures-in-mobile-application-development">blog</a>
<a href="https://wingolog.org/archives/2023/05/02/structure-and-interpretation-of-ark">blog</a>
<a href="https://wingolog.org/archives/2023/04/26/structure-and-interpretation-of-flutter">blog</a>
<a href="https://wingolog.org/archives/2023/04/24/structure-and-interpretation-of-nativescript">blog</a>
<a href="https://wingolog.org/archives/2023/04/21/structure-and-interpretation-of-react-native">blog</a>
<a href="https://wingolog.org/archives/2023/04/20/structure-and-interpretation-of-capacitor-programs">blog</a></p><p><i>CPS Soup: A functional intermediate language.</i>
Spritely, remote, 10 May 2023.
<a href="https://wingolog.org/archives/2023/05/20/approaching-cps-soup">blog</a>
<a href="https://wingolog.org/pub/2023-spritely-cps-soup-slides.pdf">slides</a></p><p><i>Whippet: A new GC for Guile.</i>
FOSDEM, Brussels, 4 Feb 2023.
<a href="https://wingolog.org/archives/2023/02/07/whippet-towards-a-new-local-maximum">blog</a>
<a href="https://fosdem.org/2023/schedule/event/whippet/">event</a>
<a href="https://wingolog.org/pub/fosdem-whippet-2023-slides.pdf">slides</a></p><h3>but wait, there’s more</h3><p>Still here? The full <a href="https://wingolog.org/talks/">talks archive</a> will surely fill your cup.</p></div>2023-12-12T15:18:14+00:00Andy WingoGStreamer: New GStreamer Matrix chat space
https://gstreamer.freedesktop.org/news/#2023-12-12T09:30 :00Z
<p>Hello everyone,
</p><p>As part of the ongoing effort to provide better tooling and services for the GStreamer community at wide, there is now a
<a href="https://matrix.org/" target="_blank" rel="noopener">Matrix
</a> instance for GStreamer available at
<a href="https://matrix.to/#/#community:gstreamer.org" target="_blank" rel="noopener">https://matrix.to/#/#community:gstreamer.org
</a> . You can join using a
<a href="https://matrix.org/ecosystem/clients/" target="_blank" rel="noopener">matrix client
</a> and a matrix account (the GStreamer instance does not provide account creation).
</p><p>The decision comes after discussions that took place during the gstreamer conference/hackfest and discussions here :
<a href="https://discourse.gstreamer.org/t/a-replacement-for-irc/151" target="_blank" rel="noopener">https://discourse.gstreamer.org/t/a-replacement-for-irc/151
</a> . But the summary is that Matrix provides a modern chat system with features people expect in this day and age, while not being a closed silo.
</p><p>There are already some rooms up:
</p><ul>
<li>
<a href="https://matrix.to/#/#gstreamer:gstreamer.org" target="_blank" rel="noopener">General Discussion
</a> The main discussion channel
</li>
<li>
<a href="https://matrix.to/#/#newcomers:gstreamer.org" target="_blank" rel="noopener">GStreamer newcomers
</a> for those new to GStreamer
</li>
<li>
<a href="https://matrix.to/#/#releases:gstreamer.org" target="_blank" rel="noopener">GStreamer Releases
</a> for preparing releases
</li>
<li>
<a href="https://matrix.to/#/#rust:gstreamer.org" target="_blank" rel="noopener">Rust 🦀 GStreamer
</a> for the crustaceous multimedians
</li>
<li>
<a href="https://matrix.to/#/#python:gstreamer.org" target="_blank" rel="noopener">Python 🐍 GStreamer
</a> for the pythonistas
</li>
<li>
<a href="https://matrix.to/#/#sysadmin:gstreamer.org" target="_blank" rel="noopener">Sysadmin and service support
</a> for discussions and questions on the services behind
<a href="http://gstreamer.org" target="_blank" rel="noopener">gstreamer.org
</a> (matrix, discourse, mail, …)
</li>
<li>And more rooms will be made available on a as-needed basis
</li>
</ul><p>Note that a client with support for spaces and threads is recommended (but not mandatory).
</p><p>In terms of the OFTC IRC channel, it will remain bridged to matrix (2/3 users in
<code>#gstreamer
</code> on IRC are already accessing it via Matrix), but we recommend switching over to using the above matrix rooms since:
</p><ul>
<li>They provide the full matrix experience (history, threads, search, emojis, calls,…)
</li>
<li>They offer more rooms than the (single) IRC room
</li>
<li>Using the IRC bridge requires you to handle IRC commands if you wish to join/speak there (NickServ)
</li>
<li>The IRC bridge isn’t maintained by the GStreamer community and we cannot guarantee it will always be around/up.
</li>
</ul>2023-12-12T09:30:00+00:00Andy Wingo: a simple hdr histogram
https://wingolog.org/archives/2023/12/10/a-simple-hdr-histogram
<div><p>Good evening! This evening, a note on high-dynamic-range (HDR)
histograms.</p><h3>problem</h3><p>How should one record garbage collector pause times?</p><p>A few options present themselves: you could just record the total pause
time. Or, also record the total number of collections, which allows you
to compute the average. Maximum is easy enough, too. But then you
might also want the median or the p90 or the p99, and these percentile
values are more gnarly: you either need to record all the pause times,
which can itself become a memory leak, or you need to approximate via a
histogram.</p><p>Let’s assume that you decide on the histogram approach. How should you
compute the bins? It would be nice to have microsecond accuracy on the
small end, but if you bin by microsecond you could end up having
millions of bins, which is not what you want. On the other end you
might have multi-second GC pauses, and you definitely want to be able to
record those values.</p><p>Consider, though, that it’s not so important to have microsecond
precision for a 2-second pause. This points in a direction of wanting
bins that are <i>relatively</i> close to each other, but whose <i>absolute</i>
separation can vary depending on whether we are measuring microseconds
or milliseconds. You want approximately uniform precision over a high
dynamic range.</p><h3>logarithmic binning</h3><p>The basic observation is that you should be able to make a histogram
that gives you, say, 3 significant figures on measured values. Such a
histogram would count anything between 1230 and 1240 in the same bin,
and similarly for 12300 and 12400. The gap between bins increases as
the number of digits grows.</p><p>Of course computers prefer base-2 numbers over base-10, so let’s do
that. Say we are measuring nanoseconds, and the maximum number of
seconds we expect is 100 or so. There are about 2<sup>30</sup>
nanoseconds in a second, and 100 is a little less than 2<sup>7</sup>, so
that gives us a range of 37 bits. Let’s say we want a precision of 4
significant base-2 digits, or 4 bits; then we will have one set of
2<sup>4</sup> bins for 10-bit values, another for 11-bit values, and
so-on, for a total of 37 × 2<sup>4</sup> bins, or 592 bins. If we use a
32-bit integer count per bin, such a histogram would be 2.5kB or so,
which I think is acceptable.</p><p>Say you go to compute the bin for a value. Firstly, note that there are
some values that do not have 4 significant bits: if you record a
measurement of 1 nanosecond, presumably that is just 1 significant
figure. These are like the
<a href="https://en.wikipedia.org/wiki/Subnormal_number">denormals</a> in
floating-point numbers. Let’s just say that recording a value <i>val</i> in
[0, 2<sup>4</sup>-1] goes to bin <i>val</i>.</p><p>If <i>val</i> is 2<sup>4</sup> or more, then we compute the <i>major</i> and
<i>minor</i> components. The major component is the number of bits needed to
represent <i>val</i>, minus the 4 precision bits. We can define it like this
in C, assuming that <i>val</i> is a 64-bit value:</p><pre>#define max_value_bits 37
#define precision 4
uint64_t major = 64ULL - __builtin_clzl(val) - precision;
</pre><p>The <tt>64 - __builtin_clzl(val)</tt> gives us the ceiling of the base-2
logarithm of the value. And actually, to take into account the denormal
case, we do this instead:</p><pre>uint64_t major = val < (1ULL << precision)
? 0ULL
: 64ULL - __builtin_clzl(val) - precision;
</pre><p>Then to get the minor component, we right-shift <i>val</i> by <i>major</i> bits,
unless it is a denormal, in which case the minor component is the value
itself:</p><pre>uint64_t minor = val < (1 << precision)
? val
: (val >> (major - 1ULL)) & ((1ULL << precision) - 1ULL);
</pre><p>Then the histogram bucket for a given value can be computed directly:</p><pre>uint64_t idx = (major << precision) | minor;
</pre><p>Of course, we would prefer to bound our storage, hence the consideration
about 37 total bits in 100 seconds of nanoseconds. So let’s do that,
and record any out-of-bounds value in the special last bucket,
indicating that we need to expand our histogram next time:</p><pre>if (idx >= (max_value_bits << precision))
idx = max_value_bits << precision;
</pre><p>The histogram itself can then be defined simply as having enough buckets
for all major and minor components in range, plus one for overflow:</p><pre>struct histogram {
uint32_t buckets[(max_value_bits << precision) + 1];
};
</pre><p>Getting back the lower bound for a bucket is similarly simple, again
with a case for denormals:</p><pre>uint64_t major = idx >> precision;
uint64_t minor = idx & ((1ULL << precision) - 1ULL);
uint64_t min_val = major
? ((1ULL << precision) | minor) << (major - 1ULL)
: minor;
</pre><h3>y u no library</h3><p>How many lines of code does something need to be before you will include
it as a library instead of re-implementing? If I am honest with myself,
there is no such limit, as far as code goes at least: only a limit of
time. I am happy to re-implement anything; time is my only enemy. But
strategically speaking, time too is the fulcrum: if I can save time by
re-implementing over integrating a library, I will certainly hack.</p><p>The canonical library in this domain is
<a href="https://hdrhistogram.github.io/HdrHistogram/">HdrHistogram</a>. But even
the C port is thousands of lines of code! A histogram should not take
that much code! Hence this blog post today. I think what we have above is sufficient. HdrHistogram’s documentation speaks
in terms of base-10 digits of precision, but that is easily translated
to base-2 digits: if you want 0.1% precision, then in base-2 you’ll need
10 bits; no problem.</p><p>I should of course mention that HdrHistogram includes an API that compensates for <a href="https://groups.google.com/g/mechanical-sympathy/c/icNZJejUHfE/m/BfDekfBEs_sJ">coordinated
omission</a>, but I think such an API is <a href="https://www.javadoc.io/static/org.hdrhistogram/HdrHistogram/2.1.12/org/HdrHistogram/AbstractHistogram.html#recordValueWithExpectedInterval-long-long-">straigtforward to build on
top of the basics</a>.</p><p>My code, for what it is worth, and which may indeed be buggy, is <a href="https://github.com/wingo/whippet/blob/main/api/gc-histogram.h">over
here</a>.
But don’t re-use it: write your own. It could be much nicer in C++ or
Rust, or any other language.</p><p>Finally, I would note that somehow this feels very basic; surely there
is prior art? I feel like in 2003, Google would have had a better
answer than today; alack. Pointers
appreciated to other references, and if you find them, do tell me more about your search strategy, because mine is inadequate. Until then, gram you later!</p></div>2023-12-10T21:27:59+00:00Andy WingoAndy Wingo: v8's mark-sweep nursery
https://wingolog.org/archives/2023/12/08/v8s-mark-sweep-nursery
<div><p>Today, a followup to <a href="https://wingolog.org/archives/2023/12/07/the-last-5-years-of-v8s-garbage-collector">yesterday’s note</a> with some more details on V8’s new
young-generation implementation, <i>minor mark-sweep</i> or <i>MinorMS</i>.</p><p>A
caveat again: these observations are just from reading the code; I
haven’t run these past the MinorMS authors yet, so any of these details
might be misunderstandings.</p><p>The MinorMS nursery consists of <i>pages</i>, each of which is 256 kB, unless
huge-page mode is on, in which case they are 2 MB. The total default
size of the nursery is 72 MB by default, or 144 MB if <a href="https://v8.dev/blog/pointer-compression">pointer
compression</a> is off.</p><p>There can be multiple threads allocating into the nursery, but let’s
focus on the <a href="https://chromium.googlesource.com/v8/v8.git/+/refs/heads/main/src/heap/main-allocator.h">main
allocator</a>,
which is used on the main thread. Nursery allocation is bump-pointer,
whether in a MinorMS page or scavenger semi-space. Bump-pointer regions
are called <i>linear allocation buffers</i>, and often abbreviated as <tt>Lab</tt>
in the source, though the class is
<a href="https://chromium.googlesource.com/v8/v8.git/+/refs/heads/main/src/heap/linear-allocation-area.h"><tt>LinearAllocationArea</tt></a>.</p><p>If the current bump-pointer region is too small for the current
allocation, the nursery implementation finds another one, or triggers a
collection. For the MinorMS nursery, each page collects the set of
allocatable spans in a free list; if the free-list is non-empty, it pops
off one entry as the current and tries again.</p><p>Otherwise, MinorMS needs another page, and specifically a <i>swept page</i>:
a page which has been visited since the last GC, and whose spans of
unused memory have been collected into a free-list. There is a
concurrent sweeping task which should usually run ahead of the mutator,
but if there is no swept page available, the allocator might need to
sweep some. This logic is in
<a href="https://chromium.googlesource.com/v8/v8.git/+/refs/heads/main/src/heap/main-allocator.cc#625"><tt>MainAllocator::RefillLabMain</tt></a>.</p><p>Finally, if all pages are swept and there’s no Lab big enough for the
current allocation, we trigger collection from the roots. The initial
roots are the <i>remembered set</i>: pointers from old objects to new
objects. Most of the trace happens concurrently with the mutator; when
the nursery utilisation rises over 90%, V8 will kick off concurrent
marking tasks.</p><p>Then once the mutator actually runs out of space, it pauses, drains any pending marking work, marks
conservative roots, then drains again. I am not sure whether MinorMS
with conservative stack scanning visits the whole C/C++ stacks or
whether it manages to install some barriers (i.e. “don’t scan deeper
than 5 frames because we collected then, and so all older frames are
older”); dunno. All of this logic is in
<a href="https://chromium.googlesource.com/v8/v8.git/+/refs/heads/main/src/heap/minor-mark-sweep.cc#618"><tt>MinorMarkSweepCollector::MarkLiveObjects</tt></a>.</p><p>Marking traces the object graph, setting object mark bits. It does not
trace pages. However, the MinorMS space promotes in units of pages. So
how to decide what pages to promote? The answer is that <i>sweeping</i> partitions the MinorMS pages into empty,
recycled, aging, and promoted pages.</p><p>Empty pages have no surviving
objects, and are very useful because they can be given back to the
operating system if needed or shuffled around elsewhere in the system. If they are re-used for allocation, they do not need to be swept.</p><p>Recycled pages have some survivors, but not many; MinorMS keeps the page
around for allocation in the next cycle, because it has enough empty
space. By default, a page is recyclable if it has 50% or more free
space after a minor collection, or 30% after a major collection.
MinorMS also promotes a page eagerly if in the last cycle, we only
managed to allocate into 30% or less of its empty space, probably due to
fragmentation. These pages need to be swept before re-use.</p><p>Finally, MinorMS doesn’t let pages be recycled indefinitely:
after 4 minor cycles, a page goes into the <i>aging</i> pool, in which it is
kept unavailable for allocation for one cycle, but is not yet promoted.
This allows any new allocations on that page in the previous cycle age
out and probably die, preventing premature tenuring.</p><p>And that’s it. Next time, a note on a way in which generational
collectors can run out of memory. Have a nice weekend, hackfolk!</p></div>2023-12-08T14:34:03+00:00Andy WingoAndy Wingo: the last 5 years of V8's garbage collector
https://wingolog.org/archives/2023/12/07/the-last-5-years-of-v8s-garbage-collector
<div><p>Captain, status report: I’m down here in a Jeffries tube, poking at V8’s
garbage collector. However, despite working on other areas of the project
recently, V8 is now so large that it’s necessary to ignore whole subsystems when working on any given task. But now I’m looking at the GC in anger: what is its deal? What does V8’s GC even look like these days?</p><p>The <a href="https://v8.dev/blog/trash-talk">last public article on the structure of V8’s garbage
collector</a> was in 2019; fine enough, but
dated. Now in the evening of 2023 I think it could be useful to revisit
it and try to summarize the changes since then. At least, it would have
been useful to me had someone else written this article.</p><p>To my mind, work on V8’s GC has had three main goals over the last 5
years: improving interactions between the managed heap and C++,
improving security, and increasing concurrency. Let’s visit these in
turn.</p><h3>C++ and GC</h3><p>Building on the 2018 <a href="https://docs.google.com/document/d/1Hs60Zx1WPJ_LUjGvgzt1OQ5Cthu-fG-zif-vquUH_8c/edit#heading=h.nh3gzht95k4n">integration of the Oilpan tracing garbage
collector into the Blink web
engine</a>,
there was some refactoring to <a href="https://chromium.googlesource.com/v8/v8/+/main/include/cppgc/README.md">move the implementation of Oilpan into V8
itself</a>.
Oilpan is known internally as <i>cppgc</i>.</p><p>I find the cppgc name a bit annoying because I can never remember what
it refers to, because of the other thing that has been happpening in C++
integration: a migration away from <a href="https://github.com/v8/v8/blob/main/src/handles/handles.h">precise
roots</a> and
instead towards <a href="https://bugs.chromium.org/p/v8/issues/detail?id=13257">conservative
root-finding</a>.</p><p>Some notes here: with conservative stack scanning, we can hope for
better mutator throughput and fewer bugs. The throughput comes from not
having to put all live pointers in memory; the compiler can keep them in
registers, and avoid managing the HandleScope. You may be able to avoid
the compile-time and space costs of stack maps (<a href="https://wingolog.org/archives/2023/10/16/on-safepoints">side tables telling the
collector where the pointers
are</a>). There
are also two classes of bug that we can avoid: holding on to a handle
past the lifetime of a handlescope, and holding on to a raw pointer
(instead of a handle) during a potential GC point.</p><p>Somewhat confusingly, it would seem that conservative stack scanning has
garnered the acronym “CSS” inside V8. <i>What does CSS have to do with
GC?</i>, I ask. I know the answer but my brain keeps asking the question.</p><p>In exchange for this goodness, conservative stack scanning means that
because you can’t be sure that a word on the stack refers to an object
and isn’t just a spicy integer, you can’t move objects that might be the
target of a conservative root. And indeed the conservative edge might
actually not point to the start of the object; it could be an interior
pointer, which places additional constraints on the heap, that it be
able to resolve internal pointers.</p><h3>Security</h3><p>Which brings us to security and the admirable nihilism of the <a href="https://docs.google.com/document/d/1FM4fQmIhEqPG8uGp5o9A-mnPB5BOeScZYpkHjo0KKA8/edit#heading=h.xzptrog8pyxf">sandbox
effort</a>.
The idea is that everything is terrible, so why not just assume that no
word is safe and that an attacker can modify any word they can address. The only way to limit the scope of an attacker’s modifications is then to
limit the address space. This happens firstly by <a href="https://v8.dev/blog/pointer-compression">pointer
compression</a>, which happily
also has some delightful speed and throughput benefits. Then the
pointer cage is placed within a larger cage, and off-heap data such as
Wasm memories and array buffers go in that larger cage. Any needed
<a href="https://docs.google.com/document/d/1CPs5PutbnmI-c5g7e_Td9CNGh5BvpLleKCqUnqmD82k/edit#heading=h.xzptrog8pyxf">executable
code</a>
or <a href="https://docs.google.com/document/d/1V3sxltuFjjhp_6grGHgfqZNK57qfzGzme0QTk0IXDHk/edit">external
object</a>
is accessed indirectly, through dedicated tables.</p><p>However, this indirection comes with a cost of a proliferation in the
number of spaces. <a href="https://chromium.googlesource.com/v8/v8.git/+/43d26ecc3563a46f62a0224030667c8f8f3f6ceb/src/spaces.h#36">In the
beginning</a>,
there was just an evacuating newspace, a mark-compact oldspace, and a
non-moving large object space. Now there are closer to 20 spaces: a
separate code space, a space for <i>read-only</i> objects, a space for
<i>trusted</i> objects, a space for each kind of indirect descriptor used by
the sandbox, in addition to spaces for objects that might be shared
between threads, newspaces for many of the preceding kinds, and so on.
From what I can see, managing this complexity has taken a significant
effort. The result is pretty good to work with, but you pay for what
you get. (Do you get security guarantees? I don’t know enough to say.
Better pay some more to be sure.)</p><p>Finally, the C++ integration has also had an impact on the spaces
structure, and with a security implication to boot. The thing is,
conservative roots can’t be moved, but the original evacuating newspace
required moveability. One can get around this restriction by
pretenuring new allocations from C++ into the mark-compact space, but
this would be a performance killer. The solution that V8 is going for
is to use the block-structured mark-compact space that is already used for the old-space, but for new
allocations. If an object is ever traced during a young-generation
collection, its page will be promoted to the old generation, without
copying. Originally called <i>minor mark-compact</i> or <i>MinorMC</i> in the
commit logs, it was renamed to <i>minor mark-sweep</i> or <i>MinorMS</i> to
indicate that it doesn’t actually compact. (V8’s mark-compact old-space
doesn’t <i>have</i> to compact: V8 usually chooses to just mark in place.
But we call it a mark-compact space because it has that capability.)</p><p>This last change is a performance hazard: yes, you keep the desirable
bump-pointer allocation scheme for new allocations, but you lose on
locality in the old generation, and the rate of promoted bytes will be
higher than with the semi-space new-space. The only relief is that for
a given new-space size, you can allocate twice as many objects, because
you don’t need the copy reserve.</p><p>Why do I include this discussion in the security section? Well, because
<a href="https://bugs.chromium.org/p/v8/issues/detail?id=12612">most MinorMS commits mention this locked
bug</a>. One day
we’ll know, but not today. I speculate that evacuating is just too rich
a bug farm, especially with concurrency and parallel mutators, and that
never-moving collectors will have better security properties. But
again, I don’t know for sure, and I prefer to preserve my ability to
speculate rather than to ask for too many details.</p><h3>Concurrency</h3><p>Speaking of concurrency, ye gods, the last few years have been quite the
ride I think. Every phase that can be done in parallel (multiple
threads working together to perform GC work) is now fully parallel:
semi-space evacuation, mark-space marking and compaction, and sweeping.
Every phase that can be done <i>concurrently</i> (where the GC runs threads
while the mutator is running) is concurrent: marking and sweeping. A
major sweep task can run concurrently with an evacuating minor GC. And,
V8 is preparing for multiple mutators running in parallel. It’s all a
bit terrifying but again, with engineering investment and a huge farm of
fuzzers, it seems to be a doable transition.</p><p>Concurrency and threads means that V8 has sprouted new schedulers:
should a background task have incremental or concurrent marking? How
many sweepers should a given isolate have? How should you pause
concurrency when the engine needs to do something gnarly?</p><p>The latest in-progress work would appear to be <a href="https://bugs.chromium.org/p/v8/issues/detail?id=13012">concurrent marking of
the new-space</a>.
I think we should expect this work to result in a lower overall
pause-time, though I am curious also to learn more about the model: how
precise is it? Does it allow a lot of slop to get promoted? It seems
to have a black allocator, so there will be some slop, but perhaps it
can avoid promotion for those pages. I don’t know yet.</p><h3>Summary</h3><p>Yeah, GCs, man. I find the move to a non-moving young generation is
quite interesting and I wish the team luck as they whittle down the last
sharp edges from the conservative-stack-scanning performance profile.
The sandbox is pretty fun too. All good stuff and I look forward to
spending a bit more time with it; engineering out.</p></div>2023-12-07T12:15:45+00:00Andy WingoAndy Wingo: colophonwards
https://wingolog.org/archives/2023/12/05/colophonwards
<div><p>A brief meta-note this morning: for the first time in 20 years, I
finally got around to updating the web design of
<a href="https://wingolog.org">wingolog.org</a> recently and wanted to share a bit
about that.</p><p>Back when I made <a href="http://web.archive.org/web/20041230062546/http://wingolog.org/">the initial wingolog
design</a>,
I was using the then-brand-new Wordpress, <a href="https://en.wikipedia.org/wiki/Internet_Explorer_6">Internet Explorer
6</a> was the most common web browser, CSS wasn’t very good, the Safari browser had
just made its first release, smartphones were yet to be invented, and
everyone used low-resolution CRT screens. The original design did use CSS instead
of tables, thankfully, but it was very narrow and left a lot up to the
user agent (notably font choice and size).</p><p>These days you can do much better. Even HTML has moved on, with
<a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/article"><tt><article></tt></a>
and
<a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/aside"><tt><aside></tt></a>
and
<a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/section"><tt><section></tt></a>
elements. CSS is powerful and interoperable, with grid layout and
media queries and <tt>:has()</tt> and <tt>:is()</tt> and all kinds of fun selectors.
And, we have web fonts.</p><p>I probably would have stuck with the old design if it were readable, but
with pixel counts growing, the saturated red bands on the sides flooded
the screen, leaving the reader feeling like they were driving into
headlights in the rain.</p><p>Anyway, the new design is a bit more peaceful, I hope. Feedback
welcome.</p><p>I’m using grid layout, but not in the way that I thought I would. From
reading the documentation, I had the impression that the element with
<tt>display: grid</tt> would be a kind of flexible corkboard which could be
filled up by any child element. However, that’s not quite true: it only
works for direct children, which means your HTML does have to match the
needs of the grid. Grandchildren can take their rows and columns from
grandparents via <tt>subgrid</tt>, but only really display inside themselves:
you can’t pop a grandkid out to a grandparent grid area. (Or maybe you
can! It’s a powerful facility and I don’t claim to fully understand
it.)</p><p>Also, as far as I can tell there is no provision to fill up one grid
area with multiple children. Whereas I thought that on the root page,
each blog entry would end up in its own grid area, that’s just not the
case: you put the <tt><main></tt> (another new element!) in a grid area and let
it lay out itself. Fine enough.</p><p>I would love to have <a href="https://meyerweb.com/eric/thoughts/2023/09/12/nuclear-anchored-sidenotes/">proper
side-notes</a>,
and I thought the grid would do something for me there, but it seems
that I have to wait for CSS anchor positioning. Until then you
can use <tt>position: absolute</tt> tricks, but side-notes may overlap unless
the source article is careful.</p><p>For fonts, I dearly wanted proper fonts, but I was always scared of the
<a href="https://fonts.google.com/knowledge/glossary/foit">flash of invisible
text</a>. It turns out
that with <tt>font-display: swap</tt> you can guarantee that the user can read
text if for some reason your fonts fail to load, at the cost of a later
layout shift when the fonts come in. At first I tried <a href="https://practicaltypography.com/charter.html">Bitstream
Charter</a> for the body
typeface, but I was unable to nicely mix it with <a href="https://bboxtype.com/typefaces/FiraSans/#!layout=specimen">Fira
Mono</a> without
line-heights getting all wonky: a <tt><code></tt> tag on a line would make that
line too high. I tried all kinds of adjustments in the <tt>@font-face</tt> but
finally decided to follow my heart and buy a font. Or two. And then
sheepishly admit it to my spouse the next morning. You are reading this
in <a href="https://mbtype.com/fonts/valkyrie/">Valkyrie</a>, and the headings are
<a href="https://mbtype.com/fonts/hermes-maia/">Hermes Maia</a>. I’m pretty happy
with the result and I hope you are too. They are loaded from my server,
to which the browser already has a TCP and TLS connection, so it would
seem that the performance impact is minimal.</p><p>Part of getting performance was to inline my CSS file into the web pages
produced by the <a href="https://wingolog.org/projects/tekuti/">blog software</a>,
allowing the browser to go ahead and lay things out as they should be
without waiting on a chained secondary request to get the layout.</p><p>Finally, I did finally break down and teach my blog software’s <a href="https://github.com/wingo/tekuti/blob/master/tekuti/marxdown.scm">marxdown
parser</a>
about “smart quotes” and em dashes and en dashes. I couldn’t make this
post in good faith without it; “the guy yammers on about web design and
not only is he not a designer, he uses ugly quotes”, &c, &c...</p><p>Finally finally, some recommendations: I really enjoyed reading Erik
Spiekermann’s <a href="https://static.googleusercontent.com/media/fonts.google.com/en//knowledge/stop_stealing_sheep.pdf">Stop Stealing Sheep, 4th
ed.</a>
on typography and type, which led to a raft of book purchases. Eric
Meyer and Estelle Weyl’s <a href="https://meyerweb.com/eric/books/css-tdg/">CSS: The Definitive Guide</a> was very useful
for figuring out what is possible with CSS and how to make it happen.
It’s a guide, though, and is not very opinionated; you might find
Matthew Butterick’s <a href="https://practicaltypography.com/">Practical
Typography</a> to be useful if you are
looking for pretty-good opinions about what to make.</p><p>Onwards and upwards!</p></div>2023-12-05T11:36:57+00:00Andy WingoChristian Schaller: Fedora Workstation 39 and beyond
https://blogs.gnome.org/uraeus/2023/11/29/fedora-workstation-39-and-beyond/
<p>I have not been so active for a while with writing these Fedora Workstation updates and part of the reason was that I felt I was beginning to repeat myself a lot, which I partly felt was a side effect of writing them so often, but with some time now since my last update I felt that time was ripe again. So what are some of the things we have been working on and what are our main targets going forward? This is not a exhaustive list, but hopefully items you find interesting. Apologize for weird sentences and potential spelling mistakes, but it ended up a a long post and when you read your own words over for the Nth time you start going blind to issues :)</p>
<h1><font color="blue">PipeWire</font></h1>
<p><img src="https://blogs.gnome.org/uraeus/files/2023/11/Firefly-lines-of-musical-notes-spiraling-46102-300x300.jpg" alt="PipeWire 1.0 is available!" width="300" height="300" class="aligncenter size-medium wp-image-10672" align="right" /> PipeWire keeps the Linux Multimedia revolution rolling[/caption]So lets start with one of your favorite topics, PipeWire. As you probably know <strong>PipeWire 1.0</strong> is now out and I feel it is a project we definitely succeeded with, so big kudos to Wim Taymans for leading this effort. I think the fact that we got both the creator of JACK, Paul Davis and the creator of PulseAudio Lennart Poettering to endorse it means our goal of unifying the Linux audio landscape is being met. I include their endorsement comments from the PipeWire 1.0 release announcement here :</p>
<blockquote><p> “PipeWire represents the next evolution of audio handling for Linux, taking<br />
the best of both pro-audio (JACK) and desktop audio servers (PulseAudio) and<br />
linking them into a single, seamless, powerful new system.”<br />
– Paul Davis, JACK and Ardour author</p></blockquote>
<blockquote><p> “PipeWire is a worthy successor to PulseAudio, providing a feature set<br />
closer to how modern audio hardware works, and with a security model<br />
with today’s application concepts in mind. Version 1.0 marks a<br />
major milestone in completing the adoption of PipeWire in the standard<br />
set of Linux subsystems. Congratulations to the team!”<br />
– Lennart Poettering, Pulseaudio and systemd author</p></blockquote>
<p>So for new readers, PipeWire is a audio and video server we created for Fedora Workstation to replace PulseAudio for consumer audio, JACK for pro-audio and add similar functionality for video to your linux operating system. So instead of having to deal with two different sound server architectures users now just have to deal with one and at the same time they get the same advantages for video handling. Since PipeWire implemented both the PulseAudio API and the JACK API it is a drop in replacement for both of them without needing any changes to the audio applications built for those two sound servers. Wim Taymans alongside the amazing community that has grown around the project has been hard at work maturing PipeWire and adding any missing feature they could find that blocked anyone from moving to it from either PulseAudio and JACK. Wims personal focus recently has been on an IRQ based ALSA driver for PipeWire to be able to provide 100% performance parity with the old JACK server. So while a lot of Pro-audio users felt that PipeWire’s latency was already good enough, this work by Wim shaves of the last few milliseconds to reach the same level of latency as JACK itself had.</p>
<p>In parallel with the work on PipeWire the community and especially Collabora has been hard at work on the new 0.5 release of WirePlumber, the session manager which handles all policy issues for PipeWire. I know people often get a little confused about PipeWire vs WirePlumber, but think of it like this: PipeWire provides you the ability to output audio through a connected speaker, through a bluetooth headset, through an HDMI connection and so on, but it doesn’t provide any ‘smarts’ for how that happens. The smarts are instead provided by WirePlumber which then contains policies to decide where to route your audio or video, either based on user choice or through preset policies making the right choices automatically, like if you disconnect your USB speaker it will move the audio to your internal speaker instead. Anyway, WirePlumber 0.5 will be a major step forward for WirePlumber moving from using lua scripts for configuration to instead using JSON for configuration while retaining lua for scripting. This has many advantages, but I point you to this <a href="https://www.collabora.com/news-and-blog/blog/2022/10/27/from-lua-to-json-refactoring-wireplumber-configuration-system/">excellent blog post by Collabora’s Ashok Sidipotu for the details</a>. Ashok got <a href="https://www.collabora.com/news-and-blog/blog/2023/10/30/wireplumber-exploring-lua-scripts-with-event-dispatcher/">further details about WirePlumber 0.5 that you can find here</a>.</p>
<p>With PipeWire 1.0 out the door I feel we are very close to reaching one of our initial goals with PipeWire, to remove the need for custom pro-audio distributions like Fedora JAM or Ubuntu Studio, and instead just let audio folks be able to use the same great Fedora Workstation as the rest of the world. With 1.0 done Wim plans next to look a bit at things like configuration tools and similar used by pro-audio folks and also dive into the Flatpak portal needs of pro-audio applications more, to ensure that Flatpaks + PipeWire is the future of pro-audio.</p>
<p>On the video handling side its been a little slow going since there applications need to be ported from relying directly on v4l. Jan Grulich has been working with our friends at Mozilla and Google to get PipeWire camera handling support into Firefox and Google Chrome. At the moment it looks like the Firefox support will land first, in fact Jan has set up a <a href="https://copr.fedorainfracloud.org/coprs/jgrulich/Firefox_PipeWire_Camera/">COPR that lets you try it out here</a>. For tracking the upstream work in WebRTC to add PipeWire support <a href="https://bugs.chromium.org/p/webrtc/issues/detail?id=15654">Jan set up this tracker bug</a>. Getting the web browsers to use PipeWire is important both to enable the advanced video routing capabilities of PipeWire, but it will also provide applications the ability to use <a href="https://libcamera.org/">libcamera</a> which is a needed for new modern MIPI cameras to work properly under Linux.</p>
<p>Another important application to get PipeWire camera support into is OBS Studio and the great thing is that community member <a href="https://feaneron.com/">Georges Stavracas</a> is working on getting the PipeWire patches merged into <a href="https://obsproject.com/">OBS Studio</a>, hopefully in time for their planned release early next year. <a href="https://github.com/obsproject/obs-studio/pull/9771">You can track Georges work in this pull request</a>. </p>
<p>For more information about PipeWire 1.0 I recommend our <a href="https://fedoramagazine.org/pipewire-1-0-an-interview-with-pipewire-creator-wim-taymans/">interview with Wim Taymans in Fedora Magazin</a>e and also the <a href="https://linuxunplugged.com/538">interview with Wim on Linux Unplugged podcast</a>.</p>
<p><strong><font color="blue">HDR</font></strong><br />
<img src="https://blogs.gnome.org/uraeus/files/2023/11/Firefly-swirls-of-bright-colors-reaching-towards-the-sky-10821-300x300.jpg" alt="HDR" width="300" height="300" class="alignright size-medium wp-image-10690" />HDR, or High Dynamic Range, is another major effort for us. HDR is a technology I think many of you have become familiar with due to it becoming quite common in TVs these days. It basically provides for greatly increased color depth and luminescence on your screen. This is a change that entails a lot of changes through the stack, because when you introduce into an existing ecosystem like the Linux desktop you have to figure out how to combine both new HDR capable applications and content and old non-HDR applications and content. <a href="https://fosstodon.org/@swick">Sebastian Wick</a>, <a href="https://twitter.com/j_adahl">Jonas Ådahl</a>, <a href="https://en.wikipedia.org/wiki/Olivier_Fourdan">Oliver Fourdan</a>, <a href="https://www.linkedin.com/in/daenzer/">Michel Daenzer</a> and more on the team has been working with other members of the ecosystem from Intel, AMD, NVIDIA, <a href="https://www.collabora.com/">Collabora</a> and more to pick and define the standards and protocols needed in this space. A lot of design work was done early in the year so we been quite focused on implementation work across the drivers, Wayland, Mesa, GStreamer, Mutter, GTK+ and more. Some of the more basic scenarios, like running a fullscreen HDR application is close to be ready, while we are still working hard on getting all the needed pieces together for the more complex scenarios like running SDR and HDR windows composited together on your desktop. So getting for instance full screen games to run in HDR mode with Steam should happen shortly, but the windowed support will probably land closer to summer next year.</p>
<p><strong><font color="blue">Wayland remoting</font></strong><br />
One feature we been also spending a lot of time on is enabling remote logins to a Wayland desktop. You have been able to share your screen under Wayland more or less from day one, but it required your desktop session to be already active. But lets say you wanted to access your Wayland desktop running on a headless system you been out of luck so far and had to rely on the old X session instead. So putting in place all the pieces for this has been quite an undertaking with work having been done on PipeWire, on Wayland portals, gnome remote desktop daemon, <a href="https://gitlab.freedesktop.org/whot/libei">libei</a>; the new input emulation library, gdm and more. The pieces needed are finally falling into place and we expect to have everything needed landed in time for GNOME 46. This support is currently done using a private GNOME API, but a vendor less API is being worked on to replace it. </p>
<p>As a sidenote here not directly related to desktop remoting, but libei has also enabled us to bring xtest support to XWayland which was important for various applications including Valves gamescope.</p>
<p><strong><font color="blue">NVIDIA drivers</font></strong><br />
One area we keep investing in is improving the state of NVIDIA support on Linux. This comes both in the form of being the main company backing the continued development of the <a href="https://nouveau.freedesktop.org/">Nouveau</a> graphics driver. So the challenge with Nouveau is that for the longest while it offered next to no hardware acceleration for 3D graphics. The reason for this was that the firmware that NVIDIA provided for Nouveau to use didn’t expose that functionality and since recent generations of NVIDIA cards only works with firmware signed by NVIDIA this left us stuck. So Nouveau was a good tool for doing an initial install of a system, but if you where doing any kind of serious 3D acceleration, including playing games, then you would need to install the NVIDIA binary driver. So in the last year that landscape around that has changed drastically, with the release of the new out-of-tree open source driver from NVIDIA. Alongside that driver a new firmware has also been made available , one that do provide full support for hardware acceleration.<br />
Let me quickly inject a quick explanation of out-of-tree versus in-tree drivers here. An in-tree driver is basically a kernel driver for a piece of hardware that has been merged into the official Linux kernel from Linus Torvalds and is thus being maintained as part of the official Linux kernel releases. This ensures that the driver integrates well with the rest of the Linux kernel and that it gets updated in sync with the rest of the Linux kernel. So Nouveau is an in-tree kernel driver which also integrates with the rest of the open source graphics stack, like Mesa. The new NVIDIA open source driver is an out-of-tree driver which ships as a separate source code release on its own schedule, but of course NVIDIA works to keeps it working with the upstream kernel releases (which is a lot of work of course and thus considered a major downside to being an out of tree driver).</p>
<p>As of the time of writing this blog post NVIDIAs out-of-tree kernel driver and firmware is still a work in progress for display usercases, but that is changing with NVIDIA exposing more and more display features in the driver (and the firmware) with each new release they do. But if you saw the original announcement of the new open source driver from NVIDIA and have been wondering why no distribution relies on it yet, this is why. So what does this mean for Nouveau? Well our plan is to keep supporting Nouveau for the foreseeable future because it is an in-tree driver, which is a lot easier to ensure keeps working with each new upstream kernel release. </p>
<p>At the same time the new firmware updates allows Nouveau to eventually offer performance levels competitive with the official out-of-tree driver, kind of how the open source AMD driver with MESA offers comparable performance to AMD binary GPU driver userspace. So Nouvea maintainer Ben Skeggs spent the last year working hard on refactoring Nouveau to work with the new firmware and we now have a new release of Nouveau out showing the fruits of that labor, enabling support for NVIDIAs latest chipset. Over time we will have it cover more chipset and expand Vulkan and OpenGL (using Zink) support to be a full fledged accelerated graphics driver.<br />
So some news here, Ben after having worked tirelessly on keeping Nouveau afloat for so many years decided he needed a change of pace and thus decided to leave software development behind for the time being. A big thank you to Ben from all us at Red Hat and Fedora ! The good news is that Danilo Krummrich will take over as the development lead, with Lyude Paul taking on working on the Display side specifically of the driver. We also expect to have other members of the team chipping in too. They will pick up Bens work and continue working with NVIDIA and the community on a bright future for Nouveau.</p>
<p>So as I mentioned though the new open source driver from NVIDIA is still being matured for the display usercase and until it works fully as a display driver neither will Nouveau be able to be a full alternative since they share the same firmware. So people will need to rely on the binary NVIDIA Driver for some time still. One thing we are looking at there and discussing is if there are ways for us to improve the experience of using that binary driver with Secure Boot enabled. Atm that requires quite a bit of manual fiddling with tools like mokutils, but we have some ideas on how to streamline that a bit, but it is a hard nut to solve due to a combination of policy issues, legal issues, security issues and hardware/UEFI bugs so I am making no promises at this point, just a promise that it is something we are looking at.</p>
<p><strong><font color="blue">Accessibility</font></strong><br />
<img src="https://blogs.gnome.org/uraeus/files/2023/11/laptopshine-300x300.jpg" alt="" width="300" height="300" class="alignright size-medium wp-image-10696" />Accessibility is an important feature for us in Fedora Workstation and thus we hired <a href="https://fedoramagazine.org/accessibility-in-fedora-workstation/">Lukáš Tyrychtr</a> to focus on the issue. Lukáš has been working through across the stack fixing issues blocking proper accessibility support in Fedora Workstation and also participated in various accessibility related events. There is still a lot to do there so I was very happy to hear recently that the GNOME Foundation got a million Euro sponsorship from the <a href="https://www.sovereigntechfund.de/">Sovereign Tech Fund</a> to improve various things across the stack, especially improving accessibility. So the combination of Lukáš continued efforts and that new investment should make for a much improved accessibility experience in GNOME and in Fedora Workstation going forward. </p>
<p><strong><font color="blue">GNOME Software</font></strong><br />
Another area that we keep investing in is improving GNOME Software, with Milan Crha working continuously on bugfixing and performance improvements. GNOME Software is actually a fairly complex piece of software as it has to be able to handle the installation and updating of RPMS, OSTree system images, Flatpaks, fonts and firmware for us in addition to the formats it handles for other distributions. For some time it felt was GNOME Software was struggling with the load of all those different formats and usercases and was becoming both slow and with a lot of error messages. Milan has been spending a lot of time dealing with those issues one by one and also recently landed some major performance improvements making the GNOME Software experience a lot better. One major change that Milan is working on that I think we will be able to land in Fedora Workstation 40/41 is porting GNOME Software to use DNF5. The main improvement end users will probably notice is that it unifies the caches used for GNOME Software and using dnf on the command line, saving you storage space and also ensuring the two are fully in sync on what RPMS is installed/updated at any given time.</p>
<p><strong><font color="blue">Fedora and Flatpaks</font></strong><br />
<img src="https://blogs.gnome.org/uraeus/files/2023/11/flatpak-300x300.jpg" alt="" width="300" height="300" class="alignright size-medium wp-image-10702" /><br />
Flatpaks is another key element of our strategy for moving the Linux desktop forward and as part of that we have now enabled all of Flathub to be available if you choose to enable 3rd party repositories when you install Fedora Workstation. This means that the huge universe of applications available on <a href="https://flathub.org/">Flathub</a> will be easy to install through GNOME Software alongside the content available in Fedora’s own repositories. That said we have also spent time improving the ease of making Fedora Flatpaks. Owen Taylor jumped in and removed the dependency on a technology called ‘<a href="https://asamalik.fedorapeople.org/modularity-docs-faq-pr/modularity/">modularity</a>‘ which was initially introduced to Fedora to bring new features around having different types of content and ease keeping containers up to date. Unfortunately it did not work out as intended and instead it became something that everyone just felt made things a lot more complicated, including building Flatpaks from Fedora content. With <a href="https://fedoraproject.org/wiki/Changes/FlatpaksWithoutModules">Owens updates</a> building Flatpaks in Fedora has become a lot simpler and should help energize the effort building Flatpaks in Fedora.</p>
<p><strong><font color="blue">Toolbx</font></strong><br />
<img src="https://blogs.gnome.org/uraeus/files/2023/11/toolbxtoolbox-300x300.jpg" alt="" width="300" height="300" class="alignright size-medium wp-image-10759" />As we continue marching towards a vision for Fedora Workstation to be a highly robust operating we keep evolving <a href="https://containertoolbx.org/">Toolbx</a>. Our tool for making running your development environment(s) inside a container and thus allows you to both keep your host OS pristine and up to date, while at the same time using specific toolchains and tools inside the development container. This is a hard requirement for immutable operating systems such as <a href="https://fedoraproject.org/silverblue/">Fedora Silverblue</a> or <a href="https://universal-blue.org/">Universal blue</a>, but it is also useful on operating systems like Fedora Workstation as a way to do development for other platforms, like for instance Red Hat Enterprise Linux. </p>
<p>A major focus for Toolbx since the inception is to get it a stage where it is robust and reliable. So for instance while we prototyped it as a shell script, today it is written in Go to be more maintainable and also to confirm with the rest of the container ecosystem. A recent major step forward for getting that stability there is that starting with <a href="https://fedoraproject.org/wiki/Changes/ToolbxReleaseBlocker">Fedora 39, the toolbox image is now a <a href="https://docs.fedoraproject.org/en-US/releases/f39/blocking">release blocking deliverable</a>. This means it is now built as part of the nightly compose and the whole Toolbx stack (ie. the fedora-toolbox image and the toolbox RPM) is part of the release-blocking test criteria. This shows the level of importance we put on Toolbx as the future of Linux software development and its criticality to Fedora Workstation. Earlier, we built the fedora-toobox image as a somewhat separate and standalone thing, and people interested in Toolbx would try to test and keep the whole thing working, as much as possible, on their own. This was becoming unmanageable because Toolbx integrates with many parts of the distribution from Mutter (ie, the Wayland and X sockets) to Kerberos to RPM (ie., %_netsharedpath in /usr/lib/rpm/macros.d/macros.toolbox) to glibc locale definitions and translations. The list of things that could change elsewhere in Fedora, and end up breaking Toolbx, was growing too large for a small group of Toolbx contributors to keep track of.</a></p>
<p>We the next release we now also have built-in support for Arch Linux and Ubuntu through the –distro flag in toolbox.git main, thanks again to the community contributors who worked with us on this allowing us to widen the amount of distros supported while keeping with our policy of reliability and dependability. And along the same theme of ensuring Toolbx is a tool developers can rely on we have added lots and lots of new tests. We now have more than 280 tests that run on CentOS Stream 9, all supported Fedoras and Rawhide, and Ubuntu 22.04.</p>
<p>Another feature that Toolbx maintainer Debarshi Ray put a lot of effort into is setting up full RHEL containers in Toolbx on top of Fedora. Today, thanks to Debarshi work you do <code>subscription-manager register --username user@domain.name</code> on the Fedora or RHEL host, and the container is automatically entitled to RHEL content. We are still looking at how we can provide a graphical interface for that process or at least how to polish up the CLI for doing <code>subscription-manager register</code>. If you are interested in this feature, <a href="https://debarshiray.wordpress.com/2023/08/25/fedora-meets-rhel-upgrading-ubi-to-rhel/">Debarshi provides a full breakdown here.</a></p>
<p>Other nice to haves added is support for enterprise FreeIPA set-ups, where the user logs into their machine through Kerberos and support for automatically generated shell completions for Bash, fish and Z shell.</p>
<p><strong><font color="blue">Flatpak and Foreman & Katello</font></strong><br />
For those out there using <a href="https://theforeman.org/">Foreman</a> to manage your fleet of Linux installs we have some good news. We are in the process of implementing support for Flatpaks in these tools so that you can manage and deploy applications in the Flatpak format using them. This is still a work in progress, but relevant Pulp and Katello commits are Pulp commit <a href="https://github.com/pulp/pulp_container/commit/eec513314ad9d55843f225802ea9fb45adad16a3">Support for Flatpak index endpoints</a> and Katello commits <a href="https://github.com/Katello/katello/commit/f6a1688f8078bc50bc1140ca8ea66115cdd9aecf">Reporting results of docker v2 repo discovery</a>” and <a href="https://github.com/Katello/katello/commit/e75f1f149504b2dfe3be54a797fe16dee0e86072">Support Link header in docker v2 repo discovery</a>“.</p>
<p><strong><font color="blue">LVFS</font></strong><br />
<img src="https://blogs.gnome.org/uraeus/files/2023/11/firmware-300x300.jpg" alt="" width="300" height="300" class="alignright size-medium wp-image-10768" />Another effort that Fedora Workstation has brought to the world of Linux and that is very popular arethe LVFS and fwdup formware update repository and tools. Thanks to that effort we are soon going to be <strong>passing one hundred million firmware updates on Linux devices </strong>soon! These firmware updates has helped resolve countless bugs and much improved security for Linux users. </p>
<p>But we are not slowing down. Richard Hughes worked with industry partners this year to define a <a href="https://uefi.org/blog/firmware-sbom-proposal">Bill of Materials defintion to firmware updates </a>allowing usings to be better informed on what is included in their firmware updates.</p>
<p>We now support over 1400 different devices on the LVFS (covering 78 different protocols!), with over 8000 public firmware versions (image below) from over 150 OEMs and ODMs. We’ve now done over 100,000 static analysis tests on over 2,000,000 EFI binaries in the firmware capsules!</p>
<p>Some examples of recently added hardware:<br />
* AMD dGPUs, Navi3x and above, AVer FONE540, Belkin Thunderbolt 4 Core Hub dock, CE-LINK TB4 Docks,CH347 SPI programmer, EPOS ADAPT 1×5, Fibocom FM101, Foxconn T99W373, SDX12, SDX55 and SDX6X devices, Genesys GL32XX SD readers, GL352350, GL3590, GL3525S and GL3525 USB hubs, Goodix Touch controllers, HP Rata/Remi BLE Mice, Intel USB-4 retimers, Jabra Evolve 65e/t and SE, Evolve2, Speak2 and Link devices, Logitech Huddle, Rally System and Tap devices, Luxshare Quad USB4 Dock, MediaTek DP AUX Scalers, Microsoft USB-C Travel Hub, More Logitech Unifying receivers, More PixartRF HPAC devices, More Synaptics Prometheus fingerprint readers, Nordic HID devices, nRF52 Desktop Keyboard, PixArt BLE HPAC OTA, Quectel EM160 and RM520, Some Western Digital eMMC devices, Star Labs StarBook Mk VIr2, Synaptics Triton devices, System76 Launch 3, Launch Heavy 3 and Thelio IO 2, TUXEDO InfinityBook Pro 13 v3, VIA VL122, VL817S, VL822T, VL830 and VL832, Wacom Cintiq Pro 27, DTH134 and DTC121, One 13 and One 12 Tablets</p>
<p><strong><font color="blue">InputLeap on Wayland</font></strong><br />
One really interesting feature that landed for Fedora Workstation 39 was the support for InputLeap. It’s probably not on most peoples radar, but it’s an important feature for system administrators, developers and generally anyone with more than a single computer on their desk.</p>
<p>Historically, <a href="https://github.com/input-leap/input-leap">InputLeap</a> is a fork of <a href="https://github.com/debauchee/barrier">Barrier</a> which itself was a fork of <a href="https://synergy-project.org/">Synergy</a>, it allows to share the same input devices (mouse, keyboard) across different computers (Linux, Windows, MacOS) and to move the pointer between the screens of these computers seamlessly as if they were one.</p>
<p>InputLeap has a client/server architecture with the server running on the main host (the one with the keyboard and mouse connected) and multiple clients, the other machines sitting next to the server machine. That implies two things, the InputLeap daemon on the server must be able to “capture” all the input events to forward them to the remote clients when the pointer reaches the edge of the screen, and the InputLeap client must be able to “replay” those input events on the client host to make it as if the keyboard and mouse were connected directly to the (other) computer. Historically, that relied on X11 mechanisms and neither InputLeap (nor Barrier or even Synergy as a matter of fact) would work on Wayland.</p>
<p>This is one of the use cases that Peter Hutterer had in mind when he started <a href="https://gitlab.freedesktop.org/libinput/libei">libEI</a>, a low-level library aimed at providing a separate communication channel for input emulation in Wayland compositors and clients (even though libEI is not strictly tied to Wayland). But libEI alone is far from being sufficient to implement InputLeap features, with Wayland we had the opportunity to make things more secure than X11 and take benefit from the XDG portal mechanisms.</p>
<p>On the client side, for replaying input events, it’s similar to remote desktop but we needed to <a href="https://github.com/flatpak/xdg-desktop-portal/pull/762">update the existing RemoteDesktop portal to pass the libEI socket</a>. On the server side, it required a <a href="https://github.com/flatpak/xdg-desktop-portal/pull/714">brand new portal for input capture </a>. These also required their counterparts in the GNOME portal, for both <a href="https://gitlab.gnome.org/GNOME/xdg-desktop-portal-gnome/-/merge_requests/97">RemoteDesktop</a> and <a href="https://gitlab.gnome.org/GNOME/xdg-desktop-portal-gnome/-/merge_requests/61">InputCapture</a> [8], and of course, all that needs to be supported by the Wayland compositor, in the case of GNOME that’s <a href="https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/2628">mutter</a>. That alone was a lot of work.</p>
<p>Yet, even with all that in place, that’s just the basic requirements to support a Synergy/Barrier/InputLeap-like feature, the tools in question need to have support for the portal and libEI implemented to benefit from the mechanisms we’ve put in place and for the all feature to work and be usable. So libportal was also updated to support the new portal features and a new “Wayland” backend alongside the X11, Windows and Mac OS backends was <a href="https://github.com/input-leap/input-leap/pull/1594">contributed to InputLeap</a>.</p>
<p>The merge request in InputLeap was accepted very early, even before the libEI API was completely stabilized and before the rest of the stack was merged, which I believe was a courageous choice from Povilas (who maintains InputLeap) which helped reduce the time to have the feature actually working, considering the number of components and inter-dependencies involved. Of course, there are still features missing in the Wayland backend, like copy/pasting between hosts, but a <a href="https://github.com/flatpak/xdg-desktop-portal/pull/852">clipboard interface was fairly recently added</a> to the remote desktop portal and therefore could be used by InputLeap to implement that feature.</p>
<p>Fun fact, Xwayland also grew support for libEI also using the <a href="https://gitlab.freedesktop.org/xorg/xserver/-/merge_requests/975">remote desktop portal</a> and wires that to the XTEST extension on X11 that InputLeap’s X11 backend uses, so it might even be possible to use the X11 backend of InputLeap in the client side through Xwayland, but of course it’s better to use the Wayland backend on both the client and server sides.</p>
<p>InputLeap is a great example of collaboration between multiple parties upstream including key contributions from us at Red Hat to implement and contribute a feature that has been requested for years upstream..</p>
<p><em><strong>Thank you to Olivier Fourdan, Debarshi Ray, Richard Hughes, Sebastian Wick and Jonas Ådahl for their contributions to this blog post.</strong></em></p>2023-11-29T17:55:08+00:00Andy Wingo: tree-shaking, the horticulturally misguided algorithm
https://wingolog.org/archives/2023/11/24/tree-shaking-the-horticulturally-misguided-algorithm
<div><p>Let’s talk about tree-shaking!</p><h3>looking up from the trough</h3><p>But first, I need to talk about WebAssembly’s dirty secret: despite the
hype, WebAssembly has had limited success on the web.</p><p>There is
<a href="https://medium.com/@addyosmani/photoshop-is-now-on-the-web-38d70954365a">Photoshop</a>,
which does appear to be a real success. 5 years ago there was
<a href="https://www.figma.com/blog/figma-faster/">Figma</a>, though they don’t
talk much about Wasm these days. There are quite a number of little NPM
libraries that use Wasm under the hood, usually compiled from C++ or
Rust. I think
<a href="https://dotnet.microsoft.com/en-us/apps/aspnet/web-apps/blazor">Blazor</a>
probably gets used for a few in-house corporate apps, though I could be
fooled by their marketing.</p><p>You might recall the hyped demos of 3D first-person-shooter
games with Unreal engine again from 5 years ago, but that was the previous major release of
Unreal and was always experimental; the current Unreal 5 does not support targetting
WebAssembly.</p><p>Don’t get me wrong, I think WebAssembly is great. It is having fine
success in off-the-web environments, and I think it is going to be a key
and growing part of the Web platform. I suspect, though, that we are
only just now getting past the <a href="https://en.wikipedia.org/wiki/Gartner_hype_cycle">trough of
disillusionment</a>.</p><p>It’s worth reflecting a bit on the nature of web Wasm’s successes and
failures. Taking Photoshop as an example, I think we can say that Wasm
does very well at bringing large C++ programs to the web. I know that
it took quite some work, but I understand the end result to be
essentially the same source code, just compiled for a different target.</p><p>Similarly for the JavaScript module case, Wasm finds success in getting
legacy C++ code to the web, and as a way to write new web-targetting
Rust code. These are often tasks that JavaScript doesn’t do very well
at, or which need a shared implementation between client and server
deployments.</p><p>On the other hand, WebAssembly has not been a Web success for DOM-heavy
apps. Nobody is talking about rewriting the front-end of
<a href="https://wordpress.com">wordpress.com</a> in Wasm, for example. Why is
that? It may sound like a silly question to you: Wasm just isn’t good
at that stuff. But <i>why</i>? If you dig down a bit, I think it’s that the
programming models are just too different: the Web’s primary programming
model is JavaScript, a language with dynamic typing and managed memory,
whereas WebAssembly 1.0 was about static typing and linear memory.
Getting to the DOM from Wasm was a hassle that was overcome only by the
most ardent of the true Wasm faithful.</p><p>Relatedly, Wasm has also not really been a success for languages that
aren’t, like, C or Rust. I am guessing that wordpress.com isn’t written
mostly in C++. One of the sticking points for this class of language.
is that C#, for example, will want to ship with a garbage collector, and
that it is annoying to have to do this. Check my <a href="https://wingolog.org/archives/2023/03/20/a-world-to-win-webassembly-for-the-rest-of-us">article from March
this year for more
details</a>.</p><p>Happily, this restriction is going away, as all browsers are going to
ship support for reference types and garbage collection within the next
months; Chrome and Firefox already ship Wasm GC, and Safari shouldn’t be
far behind thanks to the efforts from my colleague Asumu Takikawa. This
is an extraordinarily exciting development that I think will kick off a
whole ‘nother Gartner hype cycle, as more languages start to update
their toolchains to support WebAssembly.</p><h3>if you don’t like my peaches</h3><p>Which brings us to the meat of today’s note: web Wasm will win where
compilers create compact code. If your language’s compiler toolchain
can manage to produce useful Wasm in a file that is less than a handful
of over-the-wire kilobytes, you can win. If your compiler can’t do that
yet, you will have to instead rely on hype and captured audiences for
adoption, which at best results in an unstable equilibrium until you
figure out what’s next.</p><p>In the JavaScript world, managing bloat and deliverable size is a huge
industry. Bundlers like <a href="https://esbuild.github.io/">esbuild</a> are a
ubiquitous part of the toolchain, compiling down a set of JS modules to
a single file that should include only those functions and data types
that are used in a program, and additionally applying domain-specific
size-squishing strategies such as minification (making monikers more
minuscule).</p><p>Let’s focus on tree-shaking. The visual metaphor is that you write a
bunch of code, and you only need some of it for any given page. So you
imagine a tree whose, um, branches are the modules that you use, and
whose leaves are the individual definitions in the modules, and you then
violently shake the tree, probably killing it and also annoying any
nesting birds. The only thing that’s left still attached is what is
actually needed.</p><p>This isn’t how trees work: holding the trunk doesn’t give you
information as to which branches are somehow necessary for the tree’s
mission. It also <a href="https://pvk.ca/Blog/2012/02/19/fixed-points-and-strike-mandates/">primes your mind to look for the wrong fixed
point</a>,
removing unneeded code instead of keeping only the necessary code.</p><p>But, tree-shaking is an evocative name, and so despite its horticultural
and algorithmic inaccuracies, we will stick to it.</p><p>The thing is that maximal tree-shaking for languages with a thicker
run-time has not been a huge priority. Consider Go: <a href="https://zchee.github.io/golang-wiki/WebAssembly/">according to the
golang wiki</a>, the most
trivial program compiled to WebAssembly from Go is 2 megabytes, and
adding imports can make this go to 10 megabytes or more. Or look at
Pyodide, the Python WebAssembly port: the <a href="https://pyodide.org/en/stable/console.html">REPL
example</a> downloads about 20
megabytes of data. These are fine sizes for technology demos or, in the
limit, very rich applications, but they aren’t winners for web
development.</p><h3>shake a different tree</h3><p>To be fair, both the built-in Wasm support for Go and the Pyodide port
of Python both derive from the upstream toolchains, where producing
small binaries is nice but not necessary: on a server, who cares how big
the app is? And indeed when targetting smaller devices, we tend to see
alternate implementations of the toolchain, for example
<a href="https://micropython.org/">MicroPython</a> or
<a href="https://github.com/tinygo-org/tinygo">TinyGo</a>. TinyGo has a Wasm
back-end that can apparently go down to less than a kilobyte, even!</p><p>These alternate toolchains often come with <a href="https://tinygo.org/docs/reference/lang-support/">some restrictions or
peculiarities</a>, and
although we can consider this to be an evil of sorts, it is to be
expected that the target platform exhibits some co-design feedback on
the language. In particular, running in the sea of the DOM is
sufficiently weird that a Wasm-targetting Python program will
necessarily be different than a “native” Python program. Still, I think
as toolchain authors we aim to provide the same language, albeit
possibly with a different implementation of the standard library. I am
sure that the ClojureScript developers would prefer to remove their
<a href="https://clojurescript.org/about/differences">page documenting the differences with
Clojure</a> if they could, and
perhaps if Wasm becomes a viable target for Clojurescript, they will.</p><h3>on the algorithm</h3><p>To recap: now that it supports GC, Wasm could be a winner for web
development in Python and other languages. You would need a different
toolchain and an effective tree-shaking algorithm, so that user
experience does not degrade. So let’s talk about tree shaking!</p><p>I work on the <a href="https://gitlab.com/spritely/guile-hoot/">Hoot Scheme
compiler</a>, which targets Wasm
with GC. We manage to get down to 70 kB or so right now, in the minimal
“main” compilation unit, and are aiming for lower; auxiliary compilation
units that import run-time facilities (the current exception handler and
so on) from the main module can be sub-kilobyte. Getting here has been
tricky though, and I think it would be even trickier for Python.</p><p>Some background: <a href="https://wingolog.org/archives/2023/11/16/a-whiff-of-whiffle">like
Whiffle</a>,
the Hoot compiler prepends a
<a href="https://gitlab.com/spritely/guile-hoot/-/blob/main/module/hoot/prelude.scm">prelude</a>
onto user code. Tree-shakind happens in a number of places:</p><ul><li><p><a href="https://git.savannah.gnu.org/cgit/guile.git/tree/module/language/tree-il/peval.scm">partial
evaluation</a>
will evaluate unused bindings for effect, possibly eliding them</p></li><li><p><a href="https://git.savannah.gnu.org/cgit/guile.git/tree/module/language/tree-il/fix-letrec.scm">fixing
letrec</a>
will do the same</p></li><li><p>CPS frequently traverses the program, following only referenced
function, value, and control edges, e.g. via
<a href="https://git.savannah.gnu.org/cgit/guile.git/tree/module/language/cps/renumber.scm">renumbering</a></p></li><li><p>There is an explicit <a href="https://git.savannah.gnu.org/cgit/guile.git/tree/module/language/cps/dce.scm">dead-code
elimination</a>
pass which tries to elide unused effect-free allocations, a situation
that can arise due to other optimizations</p></li><li><p>Finally there is a <a href="https://gitlab.com/spritely/guile-hoot/-/blob/main/module/hoot/stdlib.scm?ref_type=heads">standard library written in raw-ish
WebAssembly</a>,
whose definitions (globals, tables, imports, functions, etc) are
included in the residual binary <a href="https://gitlab.com/spritely/guile-hoot/-/blob/main/module/wasm/link.scm?ref_type=heads">only as
neeeded</a>.</p></li></ul><p>Generally speaking, procedure definitions (functions / closures) are the
easy part: you just include only those functions that are referenced by
the code. In a language like Scheme, this gets you a long way.</p><p>However there are three immediate challenges. One is that the
evaluation model for the definitions in the prelude is <tt>letrec*</tt>: the
scope is recursive but ordered. Binding values can call or refer to
previously defined values, or capture values defined later. If
evaluating the value of a binding requires referring to a value only
defined later, then that’s an error. Again, for procedures this is
trivially OK, but as soon as you have non-procedure definitions,
sometimes the compiler won’t be able to prove this nice “only refers to
earlier bindings” property. In that case the <a href="https://legacy.cs.indiana.edu/~dyb/pubs/letrec-reloaded-abstract.html">fixing <tt>letrec</tt>
(reloaded)</a>
algorithm will end up residualizing bindings that are <tt>set!</tt>, which of
all the tree-shaking passes above require the delicate DCE pass to
remove them.</p><p>Worse, some of those non-procedure definitions are <i>record types</i>, which
have vtables that define how to print a record, how to check if a value
is an instance of this record, and so on. These vtable callbacks can
end up keeping a lot more code alive even if they are never used. We’ll
get back to this later.</p><p>Similarly, say you print a string via
<a href="https://www.r6rs.org/final/html/r6rs-lib/r6rs-lib-Z-H-9.html#node_idx_832"><tt>display</tt></a>.
Well now not only are you bringing in the whole buffered I/O facility,
but you are also calling a highly polymorphic function: <tt>display</tt> can
print anything. There’s a case for bitvectors, so you pull in code for
bitvectors. There’s a case for pairs, so you pull in that code too.
And so on.</p><p>One solution is to instead call <tt>write-string</tt>, which only writes
strings and not general data. You’ll still get the generic buffered I/O
facility (ports), though, even if your program only uses one kind of
port.</p><p>This brings me to my next point, which is that optimal tree-shaking is a
flow analysis problem. Consider <tt>display</tt>: if we know that a program
will never have bitvectors, then any code in <tt>display</tt> that works on
bitvectors is dead and we can fold the branches that guard it. But to
know this, we have to know what kind of arguments <tt>display</tt> is called
with, and for that we need <a href="https://wingolog.org/archives/2014/07/01/flow-analysis-in-guile">higher-level flow
analysis</a>.</p><p>The problem is exacerbated for Python in a few ways. One, because
<a href="https://matt.might.net/articles/implementation-of-kcfa-and-0cfa/">object-oriented dispatch is higher-order
programming</a>.
How do you know what <tt>foo.bar</tt> actually means? Depends on <tt>foo</tt>, which
means you have to thread around representations of what <tt>foo</tt> might be
everywhere and to everywhere’s caller and everywhere’s caller’s caller
and so on.</p><p>Secondly, lookup in Python is generally more dynamic than in Scheme: you
have <tt>__getattr__</tt> methods (is that it?; been a while since I’ve done
Python) everywhere and users might indeed use them. Maybe this is not
so bad in practice and flow analysis can exclude this kind of dynamic
lookup.</p><p>Finally, and perhaps relatedly, the object of tree-shaking in Python is
a mess of modules, rather than a big term with lexical bindings. This
is like JavaScript, but without the established ecosystem of
tree-shaking bundlers; Python has its work cut out for some years to go.</p><h3>in short</h3><p>With GC, Wasm makes it thinkable to do DOM programming in languages
other than JavaScript. It will only be feasible for mass use, though,
if the resulting Wasm modules are small, and that means significant
investment on each language’s toolchain. Often this will take the form
of alternate toolchains that incorporate experimental tree-shaking
algorithms, and whose alternate standard libraries facilitate the
tree-shaker.</p><p>Welp, I’m off to lunch. Happy wassembling, comrades!</p></div>2023-11-24T11:41:37+00:00Andy WingoMichael Sheldon: Pied Beta
https://blog.mikeasoft.com/2023/11/20/pied-beta/
<p>For the past couple of months I’ve been working on <a href="https://pied.mikeasoft.com">Pied</a>, an application that makes it easy to use modern, natural sounding, text-to-speech voices on Linux. It does this by integrating the <a href="https://github.com/rhasspy/piper">Piper</a> neural text-to-speech engine with <a href="https://freebsoft.org/speechd">speech-dispatcher</a>, so most existing software will work with it out of the box.</p>
<p><center></center></p>
<p>The first beta version is now available in the snap store:<br />
<a href="https://snapcraft.io/pied"><br />
<img alt="Get it from the Snap Store" src="https://snapcraft.io/static/images/badges/en/snap-store-black.svg" /><br />
</a><br />
(Other package formats will follow)</p>
<p>I’d appreciate any feedback if you’re able to test it, thanks!</p>2023-11-20T21:11:37+00:00Andy Wingo: a whiff of whiffle
https://wingolog.org/archives/2023/11/16/a-whiff-of-whiffle
<div><p>A couple nights ago I <a href="https://wingolog.org/archives/2023/11/14/whiffle-a-purpose-built-scheme">wrote about a superfluous Scheme
implementation</a>
and promised to move on from sheepishly justifying my egregious behavior in
my next note, and finally mention some results from this experiment.
Well, no: I am back on my bullshit. Tonight I write about a couple of
implementation details that discerning readers may find of interest:
value representation, the tail call issue, and the standard library.</p><h3>what is a value?</h3><p>As a Lisp, Scheme is one of the early “dynamically typed” languages.
These days when you say “type”, people immediately think <a href="https://www.pure.ed.ac.uk/ws/files/20001186/propositions_as_types.pdf">propositions
as
types</a>,
mechanized proof of program properties, and so on. But “type”
has another denotation which is all about values and almost not at all
about terms: one might say that
<a href="https://man.scheme.org/vector-ref.3scm"><tt>vector-ref</tt></a> has a type, but
it’s not part of a proof; it’s just that if you try to <tt>vector-ref</tt> a
pair instead of a vector, you get a run-time error. You can imagine
values as being associated with <i>type tags</i>: annotations that can be
inspected at run-time for, for example, the sort of error that
<tt>vector-ref</tt> will throw if you call it on a pair.</p><p>Scheme systems usually have a finite set of type tags: there are
fixnums, booleans, strings, pairs, symbols, and such, and they all have
their own tag. Even a Scheme system that provides facilities for
defining new disjoint types (<tt>define-record-type</tt> et al) will implement these via a secondary type tag layer: for example that all record
instances are have the same primary tag, and that you have to retrieve
their record type descriptor to discriminate instances of different
record types.</p><p>Anyway. In Whiffle there are immediate types and heap types. All values
have a low-bit tag which is zero for heap objects and nonzero for
immediates. For heap objects, the first word of the heap object has
tagging in the low byte as well. The <a href="https://github.com/wingo/whiffle/blob/main/include/whiffle/types.h#L43-L44">3-bit heap tag for pairs is chosen so that
pairs can just be two words, with no header
word</a>.
There is another 3-bit heap tag for forwarded objects, which is used but the
GC when evacuating a value. Other objects <a href="https://github.com/wingo/whiffle/blob/main/include/whiffle/types.h#L49-L57">put their heap tags in the low 8
bits of the first word</a>.
Additionally there is a “busy” tag word value, used to prevent races
when evacuating from multiple threads.</p><p>Finally, for generational collection of objects that can be “large” –
the definition of large depends on the collector implementation, and is
not nicely documented, but is more than, like, 256 bytes – anyway these
objects might need to have space for a “remembered” bit in the object
themselves. This is not the case for pairs but is the case for, say,
vectors: even though they are prolly smol, they might not be, and they
need space for a remembered bit in the header.</p><h3>tail calls</h3><p>When I started Whiffle, I thought, let’s just compile each Scheme
function to a C function. Since all functions have the same type, clang and gcc will have
no problem turning any tail call into a proper tail call.</p><p>This intuition was right and wrong: at optimization level <tt>-O2</tt>, this
works great. We don’t even do any kind of loop recognition /
contification: loop iterations are tail calls and all is fine. (Not the
most optimal implementation technique, but the assumption is that for
our test cases, GC costs will dominate.)</p><p>However, when something goes wrong, I will need to debug the program to
see what’s up, and so you might think to compile at <tt>-O0</tt> or <tt>-Og</tt>. In
that case, somehow gcc does not compile to tail calls. One time while
debugging a program I was flummoxed at a segfault during the <tt>call</tt>
instruction; turns out it was just stack overflow, and the <tt>call</tt> was
trying to write the return address into an unmapped page. For clang, I
could use the <a href="https://clang.llvm.org/docs/AttributeReference.html#musttail"><tt>musttail</tt>
attribute</a>;
perhaps I should, to allow myself to debug properly.</p><p>Not being able to debug at <tt>-O0</tt> with gcc is annoying. I feel like if GNU were an actual thing, we would have had the equivalent of a <tt>musttail</tt> attribute 20 years ago already. But it’s not, and we still don’t.</p><h3>stdlib</h3><p>So Whiffle makes C, and that C uses some <a href="https://github.com/wingo/whiffle/blob/main/include/whiffle/vm.h">primitives defined as inline
functions</a>.
Whiffle actually <a href="https://github.com/wingo/whiffle/blob/main/module/whiffle/input.scm#L42-L45">lexically embeds user Scheme
code</a>
with a
<a href="https://github.com/wingo/whiffle/blob/main/runtime/prelude.scm">prelude</a>,
having exposed a <a href="https://github.com/wingo/whiffle/blob/main/module/whiffle/primitives.scm">set of
primitives</a>
to that prelude and to user code. The assumption is that the compiler
will open-code all primitives, so that the conceit of providing a
primitive from the Guile compilation host to the Whiffle guest magically
works out, and that any reference to a free variable is an error. This
works well enough, and it’s similar to <a href="https://gitlab.com/spritely/guile-hoot/-/blob/main/module/hoot/compile.scm?ref_type=heads#L2592">what we currently do in
Hoot</a>
as well.</p><p>This is a quick and dirty strategy but it does let us <a href="https://www.youtube.com/watch?v=lw6TaiXzHAE">grow the
language</a> to something
worth using. I think I’ll come back to this local maximum later if I
manage to write about what Hoot does with modules.</p><h3>coda</h3><p>So, that’s Whiffle: the Guile compiler front-end for Scheme, applied to
an expression that prepends a user’s program with a prelude, in a
lexical context of a limited set of primitives, compiling to very simple
C, in which tail calls are just <tt>return f(...)</tt>, relying on the C
compiler to inline and optimize and all that.</p><p>Perhaps next up: some results on using Whiffle to test Whippet. Until
then, good night!</p></div>2023-11-16T21:11:43+00:00Andy WingoAndy Wingo: whiffle, a purpose-built scheme
https://wingolog.org/archives/2023/11/14/whiffle-a-purpose-built-scheme
<div><p>Yesterday I <a href="https://wingolog.org/archives/2023/11/13/i-accidentally-a-scheme">promised an
apology</a>
but didn’t actually get past the admission of guilt. Today the
defendant takes the stand, in the hope that an awkward
cross-examination will persuade the jury to take pity on a poor
misguided soul.</p><p>Which is to say, let’s talk about
<a href="https://github.com/wingo/whiffle">Whiffle</a>: what it actually is, what it is
doing for me, and why on earth it is that [I tell myself that] writing a
new programming language implementation is somehow preferable than
re-using an existing one.</p><h3><s>graphic design</s>garbage collection is my passion</h3><p>Whiffle is purpose-built to test the
<a href="https://github.com/wingo/whippet">Whippet</a> garbage collection library.</p><p>Whiffle lets me create Whippet test cases in C, without actually writing
C. C is fine and all, but the problem with it and garbage collection is
that you have to track all stack roots manually, and this is an
error-prone process. Generating C means that I can more easily ensure
that each stack root is visitable by the GC, which lets me make test
cases with more confidence; if there is a bug, it is probably not
because of an untraced root.</p><p>Also, Whippet is mostly meant for programming language runtimes, not for
direct use by application authors. In this use-case, probably you can
use less “active” mechanisms for ensuring root traceability: instead of
eagerly recording live values in some kind of
<a href="https://github.com/v8/v8/blob/main/src/handles/handles.h">handlescope</a>,
you can keep a <a href="https://github.com/v8/v8/blob/7fdc2728f2114b083ebbd24710a32fe4574c0c57/src/codegen/safepoint-table.h#L201">side
table</a>
that is only consulted as needed during garbage collection pauses. In
particular since Scheme uses the <a href="https://wingolog.org/archives/2014/03/17/stack-overflow">stack as a data
structure</a>, I
was worried that using handle scopes would somehow distort the
performance characteristics of the benchmarks.</p><p>Whiffle is not, however, a high-performance Scheme compiler. It is not
for number-crunching, for example: garbage collectors don’t care about
that, so let’s not. Also, Whiffle should not go to any effort to remove
allocations (<a href="https://wingolog.org/archives/2014/08/25/revisiting-common-subexpression-elimination-in-guile">sroa / gvn /
cse</a>);
creating nodes in the heap is the purpose of the test case, and eliding
them via compiler heroics doesn’t help us test the GC.</p><p>I settled on a
<a href="https://wingolog.org/archives/2020/06/03/a-baseline-compiler-for-guile">baseline-style</a>
compiler, in which I re-use the Scheme front-end from
<a href="https://gnu.org/s/guile">Guile</a> to expand macros and create an
<a href="https://www.gnu.org/software/guile/manual/html_node/Tree_002dIL.html">abstract syntax
tree</a>.
I do run some optimizations on that AST; in the spirit of <a href="https://www.youtube.com/watch?v=LIEX3tUliHw">the macro
writer’s bill of rights</a>,
it does make sense to provide some basic reductions. (These reductions
can be surprising, but I am used to the <a href="https://wingolog.org/archives/2011/10/11/partial-evaluation-in-guile">Guile’s flavor of cp0
(peval)</a>,
and this project is mostly for me, so I thought it was risk-free; I was
<a href="https://mastodon.social/@wingo/111093597389556069">almost right!</a>).</p><p>Anyway the result is that Whiffle uses an explicit stack. A
<a href="https://wingolog.org/archives/2023/10/16/on-safepoints">safepoint</a> for
a thread simply records its stack pointer: everything between the stack
base and the stack pointer is live. I do have a lingering doubt about
the representativity of this compilation strategy; would a conclusion
drawn from Whippet apply to Guile, which uses a different stack
allocation strategy? I think probably so but it’s an unknown.</p><h3>what’s not to love</h3><p>Whiffle also has a number of design goals that are better formulated in
the negative. I mentioned compiler heroics as one thing to avoid, and
in general the desire for a well-understood correspondence between
source code and run-time behavior has a number of other corrolaries:
Whiffle is a pure ahead-of-time (AOT) compiler, as just-in-time (JIT)
compilation adds noise. Worse, speculative JIT would add
unpredictability, which while good on the whole would be anathema to
understanding an isolated piece of a system like the GC.</p><p>Whiffle also should produce stand-alone C files, without a thick
run-time. I need to be able to understand and reason about the residual
C programs, and depending on third-party libraries would hinder this
goal.</p><p>Oddly enough, users are also an anti-goal: as a compiler that only
exists to test a specific GC library, there is no sense in spending too
much time making Whiffle nicer for other humans, humans whose goal is
surely not just to test Whippet. Whiffle is an interesting object, but
is not meant for actual use or users.</p><h3>corners: cut</h3><p>Another anti-goal is completeness with regards to any specific language
standard: the point is to test a GC, not to make a useful Scheme.
Therefore Whippet gets by just fine without flonums, fractions,
continuations (delimited or otherwise), multiple return values, ports,
or indeed any library support at all. All of that just doesn’t matter
for testing a GC.</p><p>That said, it has been useful to be able to import standard Scheme
garbage collection benchmarks, such as
<a href="https://github.com/wingo/whiffle/blob/main/examples/earley.scm">earley</a>
or
<a href="https://github.com/wingo/whiffle/blob/main/examples/nboyer.scm">nboyer</a>.
These have required very few modifications to run in Whippet, mostly
related to Whippet’s test harness that wants to spawn multiple threads.</p><h3>and so?</h3><p>I think this evening we have elaborated a bit more about the “how”,
complementing yesterday’s note about the “what”. Tomorrow (?) I’ll see
if I can dig in more to the “why”: what questions does Whiffle let me
ask of Whippet, and how good of a job does it do at getting those
answers? Until then, may all your roots be traced, and happy hacking.</p></div>2023-11-14T22:10:58+00:00Andy WingoAndy Wingo: i accidentally a scheme
https://wingolog.org/archives/2023/11/13/i-accidentally-a-scheme
<div><p>Good evening, dear hackfriends. Tonight’s missive is an apology: not
quite in the sense of expiation, though not quite
<i>not</i> that, either; rather, apology in the sense of explanation, of
exegesis: apologia. See, I accidentally made a Scheme. I know I have
enough Scheme implementations already, but I went and made another one.
It’s for a maybe good reason, though!</p><h3>one does not simply a scheme</h3><p>I feel like we should make this the decade of leaning into your problems,
and I have a Scheme problem, so here we are. See, I co-maintain
<a href="https://gnu.org/s/guile">Guile</a>, and have been noodling on a new
garbage collector (GC) for Guile,
<a href="https://github.com/wingo/whippet">Whippet</a>. Whippet is designed to be
embedded in the project that uses it, so one day I hope it will be just
copied into Guile’s source tree, replacing the venerable
<a href="https://github.com/ivmai/bdwgc">BDW-GC</a> that we currently use.</p><p>The thing is, though, that GC implementations are complicated. A bug in
a GC usually manifests itself far away in time and space from the code
that caused the bug. Language implementations are also complicated, for
similar reasons. Swapping one GC for another is something to be done
very carefully. This is even more the case when the switching cost is
high, which is the case with BDW-GC: as a collector written as a library
to link into <a href="https://hboehm.info/spe_gc_paper/">“uncooperative”</a>
programs, there is more cost to moving to a conventional collector than
in the case where the embedding program is already aware that (for
example) garbage collection may relocate objects.</p><p>So, you need to start small. First, we need to <a href="https://wingolog.org/archives/2023/02/07/whippet-towards-a-new-local-maximum">prove that the new GC
implementation is promising in some way</a>, that it might improve on
BDW-GC. Then... embed it directly into Guile? That sounds like a bug
farm. Is there not any intermediate step that one might take?</p><p>But also, how do you actually test that a GC algorithm or
implementation is interesting? You need a workload, and you need the
ability to compare the new collector to the old, for that workload. In
Whippet I had been writing some benchmarks in C
(<a href="https://github.com/wingo/whippet/blob/main/benchmarks/mt-gcbench.c">example</a>),
but this approach wasn’t scaling: besides not sparking joy, I was
starting to wonder if what I was testing would actually reflect usage in
Guile.</p><p>I had an early approach to rewrite a simple language implementation like
the <a href="https://wingolog.org/archives/2022/08/18/just-in-time-code-generation-within-webassembly">other Scheme implementation I made to demonstrate JIT code
generation in
WebAssembly</a>,
but that soon foundered against what seemed to me an unlikely rock: the
compiler itself. In my <a href="https://github.com/wingo/wasm-jit/">wasm-jit</a>
work, the “compiler” itself was in C++, using the C++ allocator for
compile-time allocations, and the result was a tree of AST nodes that were
interpreted at run-time. But to embed the benchmarks in Whippet itself
I needed something C, which is less amenable to abstraction of any
kind... Here I think I could have made a different choice: to somehow
allow C++ or something as a dependency to write tests, or to do more
mallocation in the “compiler”...</p><p>But that wasn’t fun. A lesson I learned long ago is that if something
isn’t fun, I need to turn it into a compiler. So I started writing a
compiler to a little bytecode VM, initially in C, then in Scheme because
C is a drag and why not? Why not just generate the bytecode C file from
Scheme? Same dependency set, once the C file is generated. And then,
as long as you’re generating C, why go through bytecode at all? Why not
just, you know, generate C?</p><h3>after all, why not? why shouldn’t i keep it?</h3><p>And that’s how I accidentally made a
Scheme, <a href="https://github.com/wingo/whiffle">Whiffle</a>. Tomorrow I’ll write a little
more on what Whiffle is and isn’t, and what it’s doing for me. Until
then, happy hacking!</p></div>2023-11-13T21:36:08+00:00Andy WingoGStreamer: GStreamer 1.22.7 stable bug fix release
https://gstreamer.freedesktop.org/news/#2023-11-13T12:00:00Z
<p>
The GStreamer team is pleased to announce another bug fix release
in the stable 1.22 release series of your favourite cross-platform
multimedia framework!
</p><p>
This release only contains bugfixes and security fixes and it should be safe
to update from 1.22.x.
</p><p>
<b>Highlighted bugfixes:</b>
<ul>
<li><a href="https://gstreamer.freedesktop.org/security/">Security fixes</a> for the MXF demuxer and AV1 codec parser</li>
<li>glfilter: Memory leak fix for OpenGL filter elements</li>
<li>d3d11videosink: Fix toggling between fullscreen and maximized, and window switching in fullscreen mode</li>
<li>DASH / HLS adaptive streaming fixes</li>
<li>Decklink card device provider device name string handling fixes</li>
<li>interaudiosrc: handle non-interleaved audio properly</li>
<li>openh264: Fail gracefully if openh264 encoder/decoder creation fails</li>
<li>rtspsrc: improved whitespace handling in response headers by certain cameras</li>
<li>v4l2codecs: avoid wrap-around after 1000000 frames; tiled formats handling fixes</li>
<li>video-scaler, audio-resampler: downgraded "Can't find exact taps" debug log messages</li>
<li>wasapi2: Don't use global volume control object</li>
<li>Rust plugins: various improvements in aws, fmp4mux, hlssink3, livesync, ndisrc, rtpav1depay, rsfilesink, s3sink, sccparse</li>
<li>WebRTC: various webrtchttp, webrtcsrc, and webrtcsink improvements and fixes</li>
<li>Cerbero build tools: recognise Windows 11; restrict parallelism of gst-plugins-rs build on small systems</li>
<li>Packages: ca-certificates update; fix gio module loading and TLS support on macOS</li>
</ul>
</p><p>
See the <a href="https://gstreamer.freedesktop.org/releases/1.22/#1.22.7">GStreamer 1.22.7 release notes</a>
for more details.
</p><p>
Binaries for Android, iOS, Mac OS X and Windows will be available shortly.
</p><p>
Release tarballs can be downloaded directly here:
<ul>
<a href="https://gstreamer.freedesktop.org/src/gstreamer/gstreamer-1.22.7.tar.xz">gstreamer</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-base/gst-plugins-base-1.22.7.tar.xz">gst-plugins-base</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-good/gst-plugins-good-1.22.7.tar.xz">gst-plugins-good</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-ugly/gst-plugins-ugly-1.22.7.tar.xz">gst-plugins-ugly</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-bad/gst-plugins-bad-1.22.7.tar.xz">gst-plugins-bad</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-libav/gst-libav-1.22.7.tar.xz">gst-libav</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-rtsp-server/gst-rtsp-server-1.22.7.tar.xz">gst-rtsp-server</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-python/gst-python-1.22.7.tar.xz">gst-python</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-editing-services/gst-editing-services-1.22.7.tar.xz">gst-editing-services</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-devtools/gst-devtools-1.22.7.tar.xz">gst-devtools</a>,
<a href="https://gstreamer.freedesktop.org/src/gstreamer-vaapi/gstreamer-vaapi-1.22.7.tar.xz">gstreamer-vaapi</a>,
<a href="https://gstreamer.freedesktop.org/src/gstreamer-sharp/gstreamer-sharp-1.22.7.tar.xz">gstreamer-sharp</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-omx/gst-omx-1.22.7.tar.xz">gst-omx</a>, or
<a href="https://gstreamer.freedesktop.org/src/gstreamer-docs/gstreamer-docs-1.22.7.tar.xz">gstreamer-docs</a>.
</ul>
</p>2023-11-13T12:00:00+00:00Víctor Jáquez: GStreamer Conference 2023
https://blogs.igalia.com/vjaquez/2023/10/31/gstreamer-conference-2023/
<p>This year the <a href="https://gstreamer.freedesktop.org/conference/2023/">GStreamer Conference</a> happened in A Coruña, basically at home, along with the hackfest.</p>
<p>The conference was the first after a long hiatus of four years of pandemics. The community craved it and long expected it. Some igalians helped to the GStreamer Foundation and our warm community with the organization and logistics. I’m very thankful with my peers and the sponsors of the event. Personally, I’m happy with the outcome. Though, I ought to say, organizing a conference like this is quite a challenge and very demanding.</p>
<p><img src="http://blogs.igalia.com/vjaquez/files/2023/10/palexco-marina.jpg" alt="Palexco Marina" title="Marina from Palexco" /></p>
<p>The conference were recorded and streamed by <a href="https://www.ubicast.eu/">Ubicast</a>. And you can watch any presentation of the conference in their <a href="https://gstconf.ubicast.tv/channels/#gstreamer-conference-2023">GStreamer Archive</a>.</p>
<p><img src="http://blogs.igalia.com/vjaquez/files/2023/10/state-of-the-union.jpg" alt="Tim sharing the State of the Union" title="Tim sharing the State of the Union" /></p>
<p><img src="http://blogs.igalia.com/vjaquez/files/2023/10/palexco-lunch.jpg" alt="Lunch time in Palexco" title="Lunch time in Palexco" /></p>
<p>This is the list of talks where fellow Igalians participated:</p>
<p><a target="_blank" title="Vulkan Video in GStreamer" href="https://gstconf.ubicast.tv/permalink/v1266612a0e8e8cbie1q/iframe/" rel="noopener noreferrer"><img src="https://gstconf.ubicast.tv/thumb/v1266612a0e8e8cbie1q/play/" alt="Vulkan Video in GStreamer" /></a></p>
<p><a target="_blank" title="Video Editing with GStreamer: an update" href="https://gstconf.ubicast.tv/permalink/v12666129e8f3rl98fq2/iframe/" rel="noopener noreferrer"><img src="https://gstconf.ubicast.tv/thumb/v12666129e8f3rl98fq2/play/" alt="Video Editing with GStreamer: an update" /></a></p>
<p><a target="_blank" title="MSE and EME on GStreamer based WebKit ports" href="https://gstconf.ubicast.tv/permalink/v1266611b4503y89eyrf/iframe/" rel="noopener noreferrer"><img src="https://gstconf.ubicast.tv/thumb/v1266611b4503y89eyrf/play/" alt="MSE and EME on GStreamer based WebKit ports" /></a></p>
<p><a target="_blank" title="High-level WebRTC APIs in GStreamer" href="https://gstconf.ubicast.tv/permalink/v1266611b4d2ayspl9ed/iframe/" rel="noopener noreferrer"><img src="https://gstconf.ubicast.tv/thumb/v1266611b4d2ayspl9ed/play/" alt="High-level WebRTC APIs in GStreamer" /></a></p>
<p><a target="_blank" title="GstPipelineStudio version 0.3.0 is out !" href="https://gstconf.ubicast.tv/permalink/v1266612985bboidk020/iframe/" rel="noopener noreferrer"><img src="https://gstconf.ubicast.tv/thumb/v1266612985bboidk020/play/" alt="GstPipelineStudio version 0.3.0 is out !" /></a></p>
<p><a target="_blank" title="WebCodecs in WebKit, with GStreamer!" href="https://gstconf.ubicast.tv/permalink/v1266612983d6jfb6hdq/iframe/" rel="noopener noreferrer"><img src="https://gstconf.ubicast.tv/thumb/v1266612983d6jfb6hdq/play/" alt="WebCodecs in WebKit, with GStreamer!" /></a></p>
<p><a target="_blank" title="Updates in GStreamer VA-API (and DMABuf negotiation)" href="https://gstconf.ubicast.tv/permalink/v1266611b48c76zr2gxx/iframe/" rel="noopener noreferrer"><img src="https://gstconf.ubicast.tv/thumb/v1266611b48c76zr2gxx/play/" alt="Updates in GStreamer VA-API (and DMABuf negotiation)" /></a></p>
<p><a target="_blank" title="GstWebRTC in WebKit, current status and challenges" href="https://gstconf.ubicast.tv/permalink/v1266611ab0caxu0i73z/iframe/" rel="noopener noreferrer"><img src="https://gstconf.ubicast.tv/thumb/v1266611ab0caxu0i73z/play/" alt="GstWebRTC in WebKit, current status and challenges" /></a></p>
<p>There were two days of conference. The following two were for the hackfest, at Igalia’s Head Quarters.</p>
<p><img src="http://blogs.igalia.com/vjaquez/files/2023/10/igaliahq-hackfest.jpg" alt="Day one of the Hackfest" title="Day one of the
Hackfest" /></p>2023-10-31T13:10:30+00:00Michael Sheldon: How to make Flutter’s AlertDialog screen reader friendly
https://blog.mikeasoft.com/2023/10/23/how-to-make-flutters-alertdialog-screen-reader-friendly/
<p>While developing <a href="https://pied.mikeasoft.com">Pied</a>, I discovered that Flutter’s <a href="https://api.flutter.dev/flutter/material/AlertDialog-class.html">AlertDialog</a> widget isn’t accessible to screen reader users (there’s been a <a href="https://github.com/flutter/flutter/issues/101289">bug report</a> filed against this for over a year). Text within the AlertDialog doesn’t get sent to the screen reader when the dialog is focused. The user has no indication that the dialog is present, let alone what its contents are.</p>
<h3>Example</h3>
<p>The following video demonstrates a typical Flutter AlertDialog with the <a href="https://help.gnome.org/users/orca/stable/">Orca screen reader</a> enabled. When the <i>‘Launch inaccessible alert’</i> button is pressed an alert dialog appears, but the screen reader is unable to read its contents. The <i>‘Launch accessible alert’</i> button is then pressed and an accessible alert dialog is demonstrated, with the screen reader successfully reading the contents of the dialog.</p>
<p><center></center></p>
<p>The example application used in the video can be found in the <a href="https://github.com/Elleo/accessible_alert_example">accessible_alert_example GitHub repository.</a></p>
<h3>Creating a standard alert dialog</h3>
<p>The following code will create a normal Flutter <i>AlertDialog</i> as recommended in the official Flutter documentation. Unfortunately this doesn’t work correctly for screen reader users:</p>
<pre>
showDialog(
context: context,
builder: (context) => AlertDialog(
actions: [
TextButton(
onPressed: () {
Navigator.pop(context);
},
child: const Text('Close Alert'),
)
],
content: Text(
'This text is invisible to screen readers. There is no indication that this dialog has even appeared.'
),
)
);
</pre>
<h3>Creating an accessible alert dialog</h3>
<p>By making a few small changes to the above code we can make it work correctly with screen readers. First we wrap the <i>AlertDialog</i> in a <i><a href="https://api.flutter.dev/flutter/widgets/Semantics-class.html">Semantics</a></i> widget. This allows us to attach a screen reader friendly label to the <i>AlertDialog</i>. The <i>Semantics</i> <i>label</i> text should be the same as the text displayed in the <i>AlertDialog</i>. Finally, to have this text read as soon as the alert is triggered, we enable <i>autofocus</i> on the <i>TextButton</i>:</p>
<pre>
showDialog(
context: context,
builder: (context) => <b>Semantics(
label: 'This text will be read by a screen reader. It is clear to the user that something has happened.',
enabled: true,
container: true,
child: </b>AlertDialog(
actions: [
TextButton(
<b>autofocus: true</b>,
onPressed: () {
Navigator.pop(context);
},
child: const Text('Close Alert'),
)
],
content: Text(
'This text will be read by a screen reader. It is clear to the user that something has happened.'
),
)
)
);
</pre>2023-10-23T17:33:22+00:00Andy Wingo: requiem for a stringref
https://wingolog.org/archives/2023/10/19/requiem-for-a-stringref
<div><p>Good day, comrades. Today’s missive is about strings!</p><h3>a problem for java</h3><p>Imagine you want to compile a program to WebAssembly, with the new <a href="https://github.com/WebAssembly/gc/blob/master/proposals/gc/MVP.md">GC
support for
WebAssembly</a>.
Your WebAssembly program will run on web browsers and render its
contents using the DOM API:
<a href="https://developer.mozilla.org/en-US/docs/Web/API/Document/createElement"><tt>Document.createElement</tt></a>,
<a href="https://developer.mozilla.org/en-US/docs/Web/API/Document/createTextNode"><tt>Document.createTextNode</tt></a>,
and so on. It will also use DOM interfaces to read parts of the page
and read input from the user.</p><p>How do you go about representing your program in WebAssembly? The GC
support gives you the ability to define a number of different kinds of
<a href="https://webassembly.github.io/gc/core/syntax/types.html#aggregate-types">aggregate data
types</a>:
structs (records), arrays, and functions-as-values. Earlier versions of
WebAssembly gave you 32- and 64-bit integers, floating-point numbers,
and opaque references to host values (<tt>externref</tt>). This is what you
have in your toolbox. But what about strings?</p><p>WebAssembly’s historical answer has been to throw its hands in the air
and punt the problem to its user. This isn’t so bad: the direct user of
WebAssembly is a compiler developer and can fend for themself. Using
the primitives above, it’s clear we should represent strings as some
kind of array.</p><p>The source language may impose specific requirements regarding string
representations: for example, in Java, you will want to use an <tt>(array i16)</tt>, because Java’s strings are specified as sequences of UTF-16¹ code
units, and Java programs are written assuming that random access to a
code unit is constant-time.</p><p>Let’s roll with the Java example for a while. It so happens that
JavaScript, the main language of the web, also specifies strings in
terms of 16-bit code units. The DOM interfaces are optimized for
JavaScript strings, so at some point, our WebAssembly program is going
to need to convert its <tt>(array i16)</tt> buffer to a JavaScript string. You
can imagine that a high-throughput interface between WebAssembly and the
DOM is going to involve a significant amount of copying; could there be
a way to avoid this?</p><p>Similarly, Java is going to need to perform a number of gnarly
operations on its strings, for example, locale-specific collation. This
is a hard problem whose solution basically amounts to shipping a copy of
<a href="https://unicode-org.github.io/icu/userguide/icu/"><tt>libICU</tt></a> in their
WebAssembly module; that’s a lot of binary size, and it’s not even clear
how to compile <tt>libICU</tt> in such a way that works on GC-managed arrays
rather than linear memory.</p><p>Thinking about it more, there’s also the problem of regular expressions.
A high-performance regular expression engine is a lot of investment, and
not really portable from the native world to WebAssembly, as the main
techniques require just-in-time code generation, which is unavailable on
Wasm.</p><p>This is starting to sound like a terrible system: big binaries, lots of
copying, suboptimal algorithms, and a likely ongoing functionality gap.
What to do?</p><h3>a solution for java</h3><p>One observation is that in the specific case of Java, we could just use
JavaScript strings in a web browser, instead of implementing our own
string library. We may need to make some shims here and there, but the
basic functionality from JavaScript gets us what we need: constant-time
UTF-16¹ code unit access from within WebAssembly, and efficient access
to browser regular expression, internationalization, and DOM
capabilities that doesn’t require copying.</p><p>A sort of minimum viable product for improving the performance of Java
compiled to Wasm/GC would be to represent strings as
<a href="https://webassembly.github.io/gc/core/syntax/types.html#syntax-heaptype"><tt>externref</tt></a>,
which is WebAssembly’s way of making an opaque reference to a host
value. You would operate on those values by importing the equivalent of
<tt>String.prototype.charCodeAt</tt> and friends; to get the receivers right
you’d need to run them through
<a href="https://github.com/WebAssembly/stringref/issues/5#issuecomment-1243487288"><tt>Function.call.bind</tt></a>.
It’s a somewhat convoluted system, but a WebAssembly engine could be
taught to recognize such a function and compile it specially, using the
same code that JavaScript compiles to.</p><p>(Does this sound too complicated or too distasteful to implement?
Disabuse yourself of the notion: it’s happening already. V8 does this
and other JS/Wasm engines will be forced to follow, as users file bug
reports that such-and-such an app is slow on e.g. Firefox but fast on
Chrome, and so on and so on. It’s the same dynamic that led <tt>asm.js</tt>
adoption.)</p><p>Getting properly good performance will require a bit more, though.
String literals, for example, would have to be loaded from e.g. UTF-8 in
a WebAssembly data section, then transcoded to a JavaScript string. You
need a function that can convert UTF-8 to JS string in the first place;
let’s call it <tt>fromUtf8Array</tt>. An engine can now optimize the
<tt>array.new_data</tt> + <tt>fromUtf8Array</tt> sequence to avoid the intermediate
array creation. It would also be nice to tighten up the typing on the
WebAssembly side: having everything be <tt>externref</tt> imposes a dynamic
type-check on each operation, which is something that can’t always be
elided.</p><h3>beyond the web?</h3><p>“JavaScript strings for Java” has two main limitations: JavaScript and
Java. On the first side, this MVP doesn’t give you anything if your
WebAssembly host doesn’t do JavaScript. Although it’s a bit of a
failure for a universal virtual machine, to an extent, the WebAssembly
ecosystem is OK with this distinction: there are different compiler and
toolchain options when targetting the web versus, say, Fastly’s edge
compute platform.</p><p>But does that mean you can’t run Java on Fastly’s cloud? Does the Java
compiler have to actually implement all of those things that we were
trying to avoid? Will Java actually implement those things? I think
the answers to all of those questions is “no”, but also that I expect a
pretty crappy outcome.</p><p>First of all, it’s not technically required that Java implement its own
strings in terms of <tt>(array i16)</tt>. A Java-to-Wasm/GC compiler can keep
the strings-as-opaque-host-values paradigm, and instead have these
string routines provided by an auxiliary WebAssembly module that itself
probably uses <tt>(array i16)</tt>, effectively polyfilling what the browser
would give you. The effort of creating this module can be shared
between e.g. Java and C#, and the run-time costs for instantiating the
module can be amortized over a number of Java users within a process.</p><p>However, I don’t expect such a module to be of good quality. It doesn’t
seem possible to implement a good regular expression engine that way,
for example. And, absent a very good run-time system with an adaptive
compiler, I don’t expect the low-level per-codepoint operations to be as
efficient with a polyfill as they are on the browser.</p><p>Instead, I could see non-web WebAssembly hosts being pressured into
implementing their own built-in UTF-16¹ module which has accelerated
compilation, a native regular expression engine, and so on. It’s nice
to have a portable fallback but in the long run, first-class UTF-16¹
will be everywhere.</p><h3>beyond java?</h3><p>The other drawback is Java, by which I mean, Java (and JavaScript) is
outdated: if you were designing them today, their strings would not be
UTF-16¹.</p><p>I keep this little “¹” sigil when I mention UTF-16 because Java (and
JavaScript) don’t actually use UTF-16 to represent their strings.
UTF-16 is standard <i>Unicode encoding form</i>. A Unicode encoding form
encodes a sequence of <i>Unicode scalar values</i> (USVs), using one or two
16-bit <i>code units</i> to encode each USV. A USV is a <i>codepoint</i>: an
integer in the range [0,0x10FFFF], but excluding <i>surrogate
codepoints</i>: codepoints in the range [0xD800,0xDFFF].</p><p>Surrogate codepoints are an accident of history, and occur either when
accidentally slicing a two-code-unit UTF-16-encoded-USV in the middle,
or when treating an arbitrary <tt>i16</tt> array as if it were valid UTF-16.
They are annoying to detect, but in practice are here to stay: no amount
of wishing will make them go away from Java, JavaScript, C#, or other
similar languages from those heady days of the mid-90s. Believe me, I have
engaged in some serious wishing, but if you, the virtual machine
implementor, want to support Java as a source language, your strings
have to be accessible as 16-bit code units, which opens the door
(eventually) to surrogate codepoints.</p><p>So when I say UTF-16¹, I really mean
<a href="https://simonsapin.github.io/wtf-8/">WTF-16</a>: sequences of any 16-bit
code units, without the UTF-16 requirement that surrogate code units be
properly paired. In this way, WTF-16 encodes a larger language than
UTF-16: not just USV codepoints, but also surrogate codepoints.</p><p>The existence of WTF-16 is a consequence of a kind of original sin,
originating in the choice to expose 16-bit code unit access to the Java
programmer, and which everyone agrees should be somehow firewalled off
from the rest of the world. The usual way to do this is to prohibit
WTF-16 from being transferred over the network or stored to disk: a
message sent via an HTTP <tt>POST</tt>, for example, will never include a
surrogate codepoint, and will either replace it with the U+FFFD
replacement codepoint or throw an error.</p><p>But within a Java program, and indeed within a JavaScript program, there
is no attempt to maintain the UTF-16 requirements regarding surrogates,
because any change from the current behavior would break programs. (How
many? Probably very, very few. But productively deprecating web
behavior is hard to do.)</p><p>If it were just Java and JavaScript, that would be one thing, but WTF-16
poses challenges for using JS strings from non-Java languages. Consider
that any JavaScript string can be invalid UTF-16: if your language
defines strings as sequences of USVs, which excludes surrogates, what do
you do when you get a fresh string from JS? Passing your string <i>to</i> JS
is fine, because WTF-16 encodes a superset of USVs, but when you receive
a string, you need to have a plan.</p><p>You only have a few options. You can eagerly check that a string is
valid UTF-16; this might be a potentially expensive O(n) check, but
perhaps this is acceptable. (This check may be <a href="https://github.com/tc39/proposal-is-usv-string">faster in the
future</a>.) Or, you can
replace surrogate codepoints with U+FFFD, when accessing string
contents; lossy, but preserves your language’s semantic domain. Or, you
can extend your language’s semantics to somehow deal with surrogate
codepoints.</p><p>My point is that if you want to use JS strings in a non-Java-like
language, your language will need to define what to do with invalid
UTF-16. Ideally the browser will give you a way to put your policy into
practice: replace with U+FFFD, error, or pass through.</p><h3>beyond java? (reprise) (feat: snakes)</h3><p>With that detail out of the way, say you are compiling Python to
Wasm/GC. <a href="https://docs.python.org/3/reference/datamodel.html#immutable-sequences">Python’s language
reference</a>
says: “A string is a sequence of values that represent Unicode code
points. All the code points in the range U+0000 - U+10FFFF can be
represented in a string.” This corresponds to the domain of
JavaScript’s strings; great!</p><p>On second thought, how do you actually access the contents of the
string? Surely not via the equivalent of JavaScript’s
<tt>String.prototype.charCodeAt</tt>; Python strings are sequences of codepoints, not
16-bit code units.</p><p>Here we arrive to the second, thornier problem, which is less about
domain and more about idiom: in Python, we expect to be able to access
strings by codepoint index. This is the case not only to access string
contents, but also to refer to positions in strings, for example when
extracting a substring. These operations need to be fast (or fast
enough anyway; CPython doesn’t have a very high performance baseline to
meet).</p><p>However, the web platform doesn’t give us O(1) access to string
codepoints. Usually a codepoint just takes up one 16-bit code unit, so
the (zero-indexed) 5th codepoint of JS string <tt>s</tt> may indeed be at
<tt>s.codePointAt(5)</tt>, but it may also be at offset 6, 7, 8, 9, or 10. You
get the point: finding the <i>n</i>th codepoint in a JS string requires a
linear scan from the beginning.</p><p>More generally, <i>all</i> languages will want to expose O(1) access to
<i>some</i> primitive subdivision of strings. For Rust, this is bytes; 8-bit
bytes are the <i>code units</i> of UTF-8. For others like Java or C#, it’s
16-bit code units. For Python, it’s codepoints. When targetting
JavaScript strings, there may be a performance impedance mismatch
between what the platform offers and what the language requires.</p><p>Languages also generally offer some kind of string iteration facility,
which doesn’t need to correspond to how a JavaScript host sees strings.
In the case of Python, one can implement <tt>for char in s: print(char)</tt>
just fine on top of JavaScript strings, by decoding WTF-16 on the fly.
Iterators can also map between, say, UTF-8 offsets and WTF-16 offsets,
allowing e.g. Rust to preserve its preferred “strings are composed of
bytes that are UTF-8 code units” abstraction.</p><p>Our O(1) random access problem remains, though. Are we stuck?</p><h3>what does the good world look like</h3><p>How should a language represent its strings, anyway? Here we depart
from a precise gathering of requirements for WebAssembly strings, but in
a useful way, I think: we should build abstractions not only for what
is, but also for what should be. We should favor a better future;
imagining the ideal helps us design the real.</p><p>I keep returning to Henri Sivonen’s authoritative article, <a href="https://hsivonen.fi/string-length/">It’s Not
Wrong that “🤦🏼♂️”.length == 7, But It’s Better that “🤦🏼♂️”.len() == 17
and Rather Useless that len(“🤦🏼♂️”) ==
5</a>. It is <i>so</i> good and if you have
reached this point, pop it open in a tab and go through it when you can.
In it, Sivonen argues (among other things) that random access to
codepoints in a string is not actually important; he thinks that if you
were designing Python today, you wouldn’t include this interface in its
standard library. Users would prefer extended grapheme clusters, which
is variable-length anyway and a bit gnarly to compute; storage wants
bytes; array-of-codepoints is just a bad place in the middle. Given
that UTF-8 is more space-efficient than either UTF-16 or
array-of-codepoints, and that it embraces the variable-length nature of
encoding, programming languages should just use that.</p><p>As a model for how strings are represented, array-of-codepoints is
outdated, as indeed is UTF-16. Outdated doesn’t mean irrelevant, of
course; there is lots of Python code out there and we have to support it
somehow. But, if we are designing for the future, we should nudge our
users towards other interfaces.</p><p>There is even a case that a JavaScript engine should represent its
strings as UTF-8 internally, despite the fact that JS exposes a UTF-16
view on strings in its API. The pitch is that UTF-8 takes less memory,
is probably what we get over the network anyway, and is probably what
many of the low-level APIs that a browser uses will want; it would be
faster and lighter-weight to pass UTF-8 to text shaping libraries, for
example, compared to passing UTF-16 or having to copy when going to JS
and when going back. JavaScript engines already have a dozen internal
string representations or so (narrow or wide, cons or slice or flat,
inline or external, interned or not, and the product of many of those);
adding another is just a Small Matter Of Programming that could show
benefits, even if some strings have to be later transcoded to UTF-16
because JS accesses them in that way. I have talked with JS engine
people in all the browsers and everyone thinks that UTF-8 has a chance
at being a win; the drawback is that actually implementing it would
take a lot of effort for uncertain payoff.</p><p>I have two final data-points to indicate that UTF-8 is the way. One is
that <a href="https://swift.org/">Swift</a> used to use UTF-16 to represent its
strings, but was able to <a href="https://www.swift.org/blog/utf8-string/">switch to
UTF-8</a>. To adapt to the newer
performance model of UTF-8, Swift maintainers designed new APIs to allow
users to request a <i>view</i> on a string: treat this string as UTF-8, or
UTF-16, or a sequence of codepoints, or even a sequence of extended
grapheme clusters. Their users appear to be happy, and I expect that
many languages will follow Swift’s lead.</p><p>Secondly, as a maintainer of the <a href="https://gnu.org/s/guile/">Guile
Scheme</a> implementation, I also want to switch
to UTF-8. Guile has long used Python’s representation strategy: array
of codepoints, with an optimization if all codepoints are “narrow” (less
than 256). The Scheme language exposes codepoint-at-offset
(<tt>string-ref</tt>) as one of its fundamental string access primitives, and
array-of-codepoints maps well to this idiom. However, we do plan to move to
UTF-8, with a Swift-like
<a href="https://www.swift.org/blog/utf8-string/#breadcrumbs">breadcrumbs</a>
strategy for accelerating per-codepoint access. We hope to lower memory
consumption, simplify the implementation, and have general (but not
uniform) speedups; some things will be slower but most should be faster.
Over time, users will learn the performance model and adapt to prefer
string builders / iterators (“string ports”) instead of <tt>string-ref</tt>.</p><h3>a solution for webassembly in the browser?</h3><p>Let’s try to summarize: it definitely makes sense for Java to use
JavaScript strings when compiled to WebAssembly/GC, when running on the
browser. There is an OK-ish compilation strategy for this use case
involving <tt>externref</tt>, <tt>String.prototype.charCodeAt</tt> imports, and so on,
along with some engine heroics to specially recognize these operations.
There is an <a href="https://github.com/WebAssembly/js-string-builtins/blob/main/proposals/js-string-builtins/Overview.md">early
proposal</a>
to sand off some of the rough edges, to make this use-case a bit more
predictable. However, there are two limitations:</p><ol><li><p>Focussing on providing JS strings to Wasm/GC is only really good for
Java and friends; the cost of mapping <tt>charCodeAt</tt> semantics to,
say, Python’s strings is likely too high.</p></li><li><p>JS strings are only present on browsers (and Node and such).</p></li></ol><p>I see the outcome being that Java will have to keep its implementation
that uses <tt>(array i16)</tt> when targetting the edge, and use JS strings on
the browser. I think that polyfills will not have acceptable
performance. On the edge there will be a binary size penalty and a
performance and functionality gap, relative to the browser. Some edge
Wasm implementations will be pushed to implement fast JS strings by
their users, even though they don’t have JS on the host.</p><p>If the JS string builtins proposal were a local maximum, I could see
putting some energy into it; it does make the Java case a bit better.
However I think it’s likely to be an unstable saddle point; if you are
going to infect the edge with WTF-16 anyway, you might as well step back and
try to solve a problem that is a bit more general than Java on JS.</p><h3>stringref: a solution for webassembly?</h3><p>I think WebAssembly should just bite the bullet and try to define a
string data type, for languages that use GC. It should support UTF-8
and UTF-16 views, like Swift’s strings, and support some kind of
iterator API that decodes codepoints.</p><p>It should be abstract as regards the concrete representation of strings,
to allow JavaScript strings to stand in for WebAssembly strings, in the
context of the browser. JS hosts will use UTF-16 as their
internal representation. Non-JS hosts will likely prefer UTF-8, and
indeed an abstract API favors migration of JS engines away from UTF-16
over the longer term. And, such an abstraction should give the user control
over what to do for surrogates: allow them, throw an error, or replace
with U+FFFD.</p><p>What I describe is what the
<a href="https://github.com/WebAssembly/stringref/blob/main/proposals/stringref/Overview.md">stringref</a>
proposal gives you. We don’t yet have consensus on this proposal in the
Wasm standardization group, and we may never reach there, although I think
it’s still possible. As I understand them, the objections are two-fold:</p><ol><li><p>WebAssembly is an instruction set, like AArch64 or x86. Strings are
too high-level, and should be built on top, for example with <tt>(array i8)</tt>.</p></li><li><p>The requirement to support fast WTF-16 code unit access will mean
that we are effectively standardizing JavaScript strings.</p></li></ol><p>I think the first objection is a bit easier to overcome. Firstly,
WebAssembly now defines quite a number of components that don’t map to
machine ISAs: typed and extensible locals, <tt>memory.copy</tt>, and so on.
You could have defined <tt>memory.copy</tt> in terms of primitive operations,
or required that all local variables be represented on an explicit stack
or in a fixed set of registers, but WebAssembly defines higher-level
interfaces that instead allow for more efficient lowering to machine
primitives, in this case SIMD-accelerated copies or machine-specific
sets of registers.</p><p>Similarly with garbage collection, there was a very interesting
<a href="https://github.com/WebAssembly/exception-handling/issues/105">“continuation marks”
proposal</a>
by Ross Tate that would give a low-level primitive on top of which users
could implement root-finding of stack values. However when choosing
what to include in the standard, the group preferred a more high-level
facility in which a Wasm module declares managed data types and allows
the WebAssembly implementation to do as it sees fit. This will likely
result in more efficient systems, as a Wasm implementation can more
easily use concurrency and parallelism in the GC implementation than a
guest WebAssembly module could do.</p><p>So, the criteria of what to include in the Wasm standard is not “what is
the most minimal primitive that can express this abstraction”, or even
“what looks like an ARMv8 instruction”, but rather “what makes Wasm a
good compilation target”. Wasm is designed for its compiler-users, not
for the machines that it runs on, and if we manage to find an abstract
definition of strings that works for Wasm-targetting toolchains, we
should think about adding it.</p><p>The second objection is trickier. When you compile to Wasm, you need a
good model of what the performance of the Wasm code that you emit will
be. Different Wasm implementations may use different stringref
representations; requesting a UTF-16 view on a string that is already
UTF-16 will be cheaper than doing so on a string that is UTF-8. In the
worst case, requesting a UTF-16 view on a UTF-8 string is a linear
operation on one system but constant-time on another, which in a loop
over string contents makes the former system quadratic: a real
performance failure that we need to design around.</p><p>The stringref proposal tries to reify as much of the cost model as
possible with its “view” abstraction; the compiler can reason that any
cost may incur then rather than when accessing a view. But, this
abstraction can leak, from a performance perspective. What to do?</p><p>I think that if we look back on what the expected outcome of the
JS-strings-for-Java proposal is, I believe that if Wasm succeeds as a
target for Java, we will probably already end up with WTF-16 everywhere.
We might as well admit this, I think, and if we do, then this objection
goes away. Likewise on the Web I see UTF-8 as being potentially
advantageous in the medium-long term for JavaScript, and certainly
better for other languages, and so I expect JS implementations to also
grow support for fast UTF-8.</p><h3>i’m on a horse</h3><p>I may be off in some of my predictions about where things will go, so
who knows. In the meantime, in the time that it takes other people to
reach the same conclusions, stringref is in a kind of hiatus.</p><p>The <a href="https://gitlab.com/spritely/guile-hoot/">Scheme-to-Wasm compiler</a>
that I work on does still emit stringref, but it is purely a toolchain
concept now: we have a <a href="https://gitlab.com/spritely/guile-hoot/-/blob/main/module/wasm/lower-stringrefs.scm">post-pass that lowers stringref to
WTF-8</a>
via <tt>(array i8)</tt>, and which emits calls to host-supplied conversion
routines when passing these strings to and from the host. When compiling
to Hoot’s built-in Wasm virtual machine, we can leave stringref in, instead of lowering it down,
resulting in more efficient interoperation with the host Guile than if we had to bounce through byte arrays.</p><p>So, we wait for now. Not such a bad situation, at least we have GC coming soon to all the browsers. appy hacking to all my stringfolk, and until next time!</p></div>2023-10-19T10:33:58+00:00Andy WingoStéphane Cerveau: Introducing GstPipelineStudio 0.3.4
https://dabrain34.github.io/jekyll/update/2023/10/19/GPS_0_3_4.html
<h2 id="gstpipelinestudio">GstPipelineStudio</h2>
<p>As it’s not always convenient to use the powerful command line based, <code class="language-plaintext highlighter-rouge">gst-launch</code> tool and also manage all the debug possibilities on all the platforms supported by GStreamer,
I started this personal project in 2021 to facilitate the adoption to the GStreamer framework and help newbies as confirmed engineers enjoy the power of it.</p>
<p>Indeed a few other projects, such as <a href="https://github.com/virinext/pipeviz">Pipeviz</a> (greatly inspired from…) or <a href="https://gitlab.gnome.org/GNOME/gst-debugger">gst-debugger</a>, already tried to offer this GUI capability,
my idea with GPS was to provide a cross-platform tool written in Rust with the powerful framework, gtk-rs.</p>
<p>The aim of this project is to provide the GUI for GStreamer but also being able to remote debug existing pipeline while offering
a very simple and accessible interface as back in the days I discovered DirectShow with the help of <a href="https://learn.microsoft.com/en-us/windows/win32/directshow/using-graphedit">graphedit</a> or <a href="https://www.videohelp.com/software/GraphStudio">GraphStudioNext</a></p>
<h2 id="project-details">Project details</h2>
<p><img src="https://dabrain34.github.io/jekyll/update/2023/10/19/images/gps_annotated.png" alt="GPS" /></p>
<p>The interface includes 5 important zones:</p>
<ul>
<li>(1) The registry area gives you an access to the GStreamer registry including all the plugins/elements available on your system. It provides
you details on each elements. You can also access to a favorite list.</li>
<li>(2) the main drawing area is where you can add elements from the registry area and connect them together. This area
allows you to have multiple independent pipelines with its own player for each drawing area.</li>
<li>(3) The control playback area, each pipeline can be controlled from this area including basic play/stop/pause but also a seekbar.</li>
<li>(4) the debug zone where you’ll receive the messages from the application.</li>
<li>(5) The render zone where you can have a video preview if a video sink has been added to the pipeline. Future work includes to have tracers or audio analysis in this zone.</li>
</ul>
<p>The project has been written in Rust to offer more stability and thanks to the wonderful work to use the GTK framework, it
was perfectly fitting to this project as it gives an easy way to use it over the 3 platforms targeted such as GNU/Linux, MacOS and
Windows.
On this last platform which is quite well “implanted” in the desktop eco-system, the use of GStreamer
can lead to difficulties, that’s why <a href="https://gitlab.freedesktop.org/dabrain34/GstPipelineStudio/uploads/ce4b8443e2d3161eeaed089071cdc402/GstPipelineStudio-0.3.4.msi">GstPipelineStudio Windows MSI</a> will be a perfect match to test the power of the GStreamer framework.</p>
<p>This project has been written under the GPL v3 License.</p>
<h2 id="how-it-works-under-the-hood">How it works under the hood</h2>
<p>The trick is quite simple as it uses the power of <a href="https://gstreamer.freedesktop.org/documentation/gstreamer/gstparse.html?gi-language=c#gst_parse_launch">gst-parse-launch API</a> to build a pipeline as a transformation of the visual pipeline to a command line.</p>
<p>So its a clearly a sibling of <code class="language-plaintext highlighter-rouge">gst-launch</code>.</p>
<p>Right now its directly linked to the GStreamer installed on your system but future work could be to connect it over daemons
such as <a href="https://github.com/dabrain34/gpop">GstPrinceOfParser</a>
or <a href="https://developer.ridgerun.com/wiki/index.php/GStreamer_Daemon">gstd</a></p>
<h2 id="whats-new-in-034">What’s new in <a href="https://gitlab.freedesktop.org/dabrain34/GstPipelineStudio/-/releases">0.3.4</a></h2>
<p>The main feature of this release is the cross platform ready state. These are beta versions but the CI is now ready
to build and deploy Flathub (Linux), Mac OS and Windows version of GstPipelineStudio.</p>
<p>You can download the installers from the project <a href="https://dabrain34.pages.freedesktop.org/GstPipelineStudio/">page</a> or with:</p>
<ul>
<li>Linux: <code class="language-plaintext highlighter-rouge">flatpak install org.freedesktop.dabrain34.GstPipelineStudio</code></li>
<li>MacOS: <a href="https://gitlab.freedesktop.org/dabrain34/GstPipelineStudio/uploads/5ac641779cfb7e8fffdf9be6a61fba17/GstPipelineStudio-0.3.4.dmg">DMG file</a></li>
<li>Windows: <a href="https://gitlab.freedesktop.org/dabrain34/GstPipelineStudio/uploads/ce4b8443e2d3161eeaed089071cdc402/GstPipelineStudio-0.3.4.msi">MSI file</a></li>
</ul>
<p>Here is a list of main features added to the app:</p>
<ul>
<li>
<p>Open a pipeline from the command line and it will be drawn automatically on the screen. This feature
allows you to take any command line pipeline and draw it on the screen to allow any new play tricks with the pipeline, such as change of elements, properties etc.</p>
</li>
<li>
<p>Multiple graphview allows you to draw multiple independent pipeline in the same instance of GstPipelineStudio.
The playback state is totally independent for each of the views.</p>
</li>
<li>
<p>Capsfilter has been added to the links allowing to add this crucial feature of GStreamer pipelines.</p>
</li>
<li>
<p>gstreamer-1.0 wrap support to the build system. So you can
build your own version of GPS using a dedicated GStreamer version.</p>
</li>
</ul>
<h2 id="whats-next-in-the-pipeline">What’s next in the pipeline</h2>
<p>Among multiple use case, key and debug features, the most upcoming features are:</p>
<ul>
<li>Support the zoom on the graphview. As a pipeline can be quite big, the zoom is a key/must features for GPS. See <a href="https://gitlab.freedesktop.org/dabrain34/GstPipelineStudio/-/merge_requests/53">MR</a></li>
<li>Debug sections such as receiving events/tags/messages or tracers and terminal support, see <a href="https://gitlab.freedesktop.org/dabrain34/GstPipelineStudio/-/merge_requests/55">MR</a></li>
<li>Elements compatibility to check if an element can connect to the previous/next one.</li>
<li>Remote debugging: A tracer wsserver is currently under development allowing to send over websocket the pipeline events such as connections, properties or element addition.
A <a href="https://gitlab.freedesktop.org/dabrain34/GstPipelineStudio/-/merge_requests/52">MR</a> is under development to connect this tracer and render the corresponding pipeline.</li>
<li>Auto plugging according to the rank of each compatible elements for a given pad caps.</li>
<li>Display the audio signal in a dedicated render tab.</li>
<li>Translations</li>
<li>Documentation</li>
<li>Unit tests</li>
</ul>
<p>Here is a <a href="https://gstconf.ubicast.tv/videos/gstpipelinestudio-version-030-is-out/">lighning talk</a>, I gave about this release (0.3.3), during the 2023 GStreamer conference.</p>
<p>Hope you’ll enjoy this tool and please feel free to provide
new features with an RFC <a href="https://gitlab.freedesktop.org/dabrain34/GstPipelineStudio/-/issues/new">here</a> or merge requests <a href="https://gitlab.freedesktop.org/dabrain34/GstPipelineStudio/-/merge_requests">here</a>.</p>
<p>As usual, if you would like to learn more about GstPipelineStudio, GStreamer or any other open multimedia framework, please contact <a href="https://www.igalia.com/">us</a>!</p>2023-10-19T10:10:45+00:00Andy Wingo: on safepoints
https://wingolog.org/archives/2023/10/16/on-safepoints
<div><p>Hello all, a brief note today. For context, my big project over the
last year and a half or so is
<a href="https://github.com/wingo/whippet">Whippet</a>, a new garbage collector
implementation. If everything goes right, Whippet will finding a better
point on the space/time tradeoff curve than
<a href="https://gnu.org/s/guile/">Guile</a>‘s current garbage collector,
<a href="https://github.com/ivmai/bdwgc/">BDW-GC</a>, and will make it into Guile,
well, sometime.</p><p>But, testing a garbage collector is... hard. Methodologically it’s very
tricky, though there has been some <a href="https://users.cecs.anu.edu.au/~steveb/pubs/papers/lbo-ispass-2022.pdf">recent progress in assessing
collectors in a more systematic
way</a>.
Ideally of course you test against a real application, and barring that,
against a macrobenchmark extracted from a real application. But garbage
collectors are deeply intertwined with language run-times; to maximize
the insight into GC performance, you need to minimize everything else,
and to minimize non-collector cost, that means relying on a
high-performance run-time (e.g. the JVM), which... is hard! It’s hard
enough to get toy programs to work, but testing against a beast like the
JVM is not easy.</p><p>In the case of Guile, it’s even more complicated, because the BDW-GC is
a conservative collector. BDW-GC doesn’t require precise annotation of
stack roots, and scans intraheap edges conservatively (unless you go out
of your way to do otherwise). How can you test this against a semi-space
collector if Guile can’t precisely enumerate all object references? The
Immix-derived collector in Whippet can be configured to run with conservative stack roots
(and even heap links), but how then to test the performance tradeoff of
conservative vs precise tracing?</p><p>In my first iterations on Whippet, I hand-coded some benchmarks in C,
starting with the classic
<a href="https://hboehm.info/gc/gc_bench.html"><tt>gcbench</tt></a>. I used
<a href="https://github.com/wingo/whippet/blob/main/benchmarks/simple-roots-api.h">stack-allocated linked lists for precise
roots</a>,
when run in precise-rooting mode. But, this is excruciating and
error-prone; trying to write some more tests, I ran up against a wall of
gnarly nasty C code. Not fun, and I wasn’t sure that it was
representative in terms of workload; did the handle discipline have some
kind of particular overhead? The usual way to do things in a
high-performance system is to instead have <i>stack maps</i>, where the
collector is able to precisely find roots on the stack and registers
using a side table.</p><p>Of course the usual solution if something is not fun is to make it into
a tools problem, so I wrote a little Scheme-to-C compiler,
<a href="https://github.com/wingo/whiffle/">Whiffle</a>, purpose-built for testing
Whippet. It’s a <a href="https://wingolog.org/archives/2020/06/03/a-baseline-compiler-for-guile">baseline-style
compiler</a>
that uses the C stack for control (function call and return), and a
packed side stack of temporary values. The goal was to be able to
generate some C that I could include in the collector’s benchmarks; I’ve
gotten close but haven’t done that final step yet.</p><p>Anyway, because its temporaries are all in a packed stack, Whippet can
always traverse the roots precisely. This means I can write bigger benchmark
programs without worry, which will allow me to finish testing the collector. As a nominally separate project, Whiffle also
tests that Whippet’s API is good for embedding. (Yes, the names are
similar; if I thought that in the future I would need to say much more
about Whiffle, I would rename it!)</p><p>I was able to <a href="https://github.com/wingo/whiffle/tree/main/examples">translate over the sparse benchmarks that Whippet already
had into Scheme</a>, and set about checking that they worked as expected.
Which they did... mostly.</p><h3>the bug</h3><p>The Whippet interface abstracts over different garbage collector
implementations; the choice of which collector to use is made at compile-time. There’s the
basic semi-space copying collector, with a large object space and
ephemerons; there’s the Immix derived “whippet” collector (yes, it
shares the same name); and there’s a shim that provides the Whippet API
via the BDW-GC.</p><p>The benchmarks are multithreaded, except when using the
semi-space collector which only supports one mutator thread. (I should
fix that at some point.) I tested the benchmarks with one mutator
thread, and they worked with all collectors. Yay!</p><p>Then I tested with multiple mutator threads. The Immix-derived
collectors worked fine, in both precise and conservative modes. The
BDW-GC collector... did not. With just one mutator thread it was fine, but with multiple threads I would get segfaults deep inside libgc.</p><h3>the side quest</h3><p>Over on the fediverse, Daphe Preston-Kendal asks, <a href="https://c.im/@dpk/111094069310103918">how does Whiffle deal
with tail calls?</a> The answer is,
“sloppily”. All of the generated function code has the same prototype,
so return-calls from one function to another <i>should</i> just optimize to
jumps, and indeed they do: at <tt>-O2</tt>. For <tt>-O0</tt>, this doesn’t work, and
sometimes you do want to compile in that mode (no optimizations) when
investigating. I wish that C had a <tt>musttail</tt> attribute, and I use GCC
too much to rely on <a href="https://clang.llvm.org/docs/AttributeReference.html#musttail">the one that only Clang
supports</a>.</p><p>I found the <tt>-foptimize-sibling-calls</tt> flag in GCC and somehow convinced
myself that it was a sufficient replacement, even when otherwise at <tt>-O0</tt>. However most of my calls
are indirect, and I think this must have fooled GCC. Really not sure.
But I <i>was</i> sure, at one point, until a week later I realized that the
reason that the <tt>call</tt> instruction was segfaulting was because it
couldn’t store the return address in <tt>*$rsp</tt>, because I had blown the
stack. That was embarrassing, but realizing that and trying again at
-O2 didn’t fix my bug.</p><h3>the second side quest</h3><p>Finally I recompiled <tt>bdw-gc</tt>, which I should have done from the
beginning. (I use <a href="https://guix.gnu.org/">Guix</a> on my test machine, and
I wish I could have entered into an environment that had a specific set
of dependencies, but with debugging symbols and source code; there are many things about
Guix that I would think should help me develop software but which don’t.
Probably there is a way to do this that I am unaware of.)</p><p>After recompiling, it became clear that BDW-GC was trying to scan a
really weird range of addresses for roots, and accessing unmapped
memory. You would think that this would be a common failure mode for
BDW-GC, but really, this <i>never</i> happens: it’s an astonishingly reliable
piece of software for what it does. I have used it for almost 20 years and its problems are elsewhere. Anyway, I determined that the thread
that it was scanning was... well it was stuck somewhere in some <tt>rr</tt> support library,
which... why was that anyway?</p><p>You see, the problem was that since my strange spooky segfaults on stack
overflow, I had been living in <a href="https://rr-project.org/"><tt>rr</tt></a> for a
week or so, because I needed to do hardware watchpoints and
reverse-continue. But, somehow, strangely, oddly, <i><tt>rr</tt> was corrupting
my program</i>: it worked fine when not run in <tt>rr</tt>. Somehow <tt>rr</tt> caused
GCC to grab the wrong <tt>$rsp</tt> on a remote thread.</p><h3>the actual bug</h3><p>I had found the actual bug in the shower, some days before, and fixed it
later that evening. Consider that both precise Immix and conservative
Immix are working fine. Consider that conservative Immix is, well,
stack-conservative just like BDW-GC. Clearly Immix finds the roots
correctly, why didn’t BDW-GC? What is the real difference?</p><p>Well, friend, you have read the article title, so perhaps you won’t be
surprised when I say “safepoints”. A safepoint is a potential stopping
place in a program. At a safepoint, the overall state of the program
can be inspected by some independent part of the program. In the case
of garbage collection, safepoints are places where heap object
references in a thread’s stack can be enumerated.</p><p>For my Immix-derived collectors, safepoints are cooperative: the only
place a thread will stop for collection is in an allocation call, or if
the thread has explicitly signalled to the collector that is doing a
blocking operation such as a thread join. Whiffle usually keeps the
stack pointer for the side value stack in a register, but for allocating
operations that need to go through the slow path, it makes sure to
<a href="https://github.com/wingo/whiffle/blob/main/include/whiffle/vm.h#L178">write that stack pointer into a shared data
structure</a>,
so that other threads can read it if they need to walk its stack.</p><p>The BDW collector doesn’t work this way. It can’t rely on
cooperation from the host. Instead, it installs a signal handler for
the process that can suspend and resume any thread at any time. When
BDW-GC needs to trace the heap, it stops all threads, finds the roots
for those threads, and then traces the graph.</p><p>I had installed a custom <a href="https://github.com/wingo/whiffle/blob/main/runtime/whiffle-gc.h#L172-L184">BDW-GC mark function for Whiffle
stacks</a>
so that even though they were in anonymous mmap’d memory that BDW-GC
doesn’t usually trace for roots, Whiffle would be sure to let BDW-GC
know about them. That mark function used the safepoints that I was
already tracking to determine which part of the side stack to mark.</p><p>But here’s the thing: <i>cooperative safepoints can be lazy, but
preemptive safepoints must be eager</i>. For BDW-GC, it’s not enough to
ensure that the thread’s stack is traversable at allocations: it must
be traversable <i>at all times</i>, because a signal can stop a thread at any
time.</p><p>Safepoints also differ on the axis of precision: precise safepoints must
include no garbage roots, whereas conservative safepoints can be sloppy.
For precise safepoints, if you bump the stack pointer to allocate a new
frame, you can’t include the slots in that frame in the root set until
they have values. Precision has a cost to the compiler and to the
run-time in side table size or shared-data-structure update overhead.
As far as I am aware, no production collector uses fully preemptive,
precise roots. (Happy to be corrected, if people have counterexamples;
I know that this used to be the case, but my understanding is that since
multi-threaded mutators have been common, people have mostly backed away
from this design.)</p><p>Concretely, as I was using a stack discipline to allocate temporaries, I
had to <a href="https://github.com/wingo/whiffle/blob/main/include/whiffle/vm.h#L121-L124"><i>shrink</i> the precise
safepoint</a>
at allocation sites, but I was neglecting to
<a href="https://github.com/wingo/whiffle/blob/main/include/whiffle/vm.h#L126C24-L133"><i>expand</i> the
conservative safepoint</a>
whenever the stack pointer would be restored.</p><h3>coda</h3><p>When I was a kid, baseball was not my thing. I failed out at the
earlier phase of tee-ball, where the ball isn’t thrown to you by the
pitcher but is instead on a kind of rubber stand. As often as not I
would fail to hit the ball and instead buckle the tee under it, making
an unsatisfactory “flubt” sound, causing the ball to just fall to the
ground. I probably would have persevered if I hadn’t also caught a
ground ball to the face one day while playing right field, knocking me
out and giving me a nosebleed so bad that apparently they called off the
game because so many other players were vomiting. I woke up in the
entryway of the YMCA and was given a lukewarm melted push-pop.</p><p>Anyway, “flubt” is the sound I heard in my mind when I realized that
<tt>rr</tt> had been perturbing my use of BDW-GC, and instead of joy and relief
when I finally realized that it worked already, it tasted like melted
push-pop. It be that way sometimes. Better luck next try!</p></div>2023-10-16T13:46:55+00:00Andy WingoSebastian Pölsterl: scikit-survival 0.22.0 released
https://k-d-w.org/blog/2023/10/scikit-survival-0.22.0-released/
<p>I am pleased to announce the release of <a href="https://scikit-survival.readthedocs.io/" target="_blank">scikit-survival</a> 0.22.0.
The highlights for this release include</p>
<ul>
<li>Compatibility with scikit-learn 1.3.</li>
<li>Missing value support for <a href="https://scikit-survival.readthedocs.io/en/v0.22.0/api/generated/sksurv.tree.SurvivalTree.html#sksurv.tree.SurvivalTree" target="_blank">SurvivalTree</a>.</li>
<li>A reduced memory mode for <a href="https://scikit-survival.readthedocs.io/en/v0.22.0/api/generated/sksurv.ensemble.RandomSurvivalForest.html#sksurv.ensemble.RandomSurvivalForest" target="_blank">RandomSurvivalForest</a>, <a href="https://scikit-survival.readthedocs.io/en/v0.22.0/api/generated/sksurv.ensemble.ExtraSurvivalTrees.html#sksurv.ensemble.ExtraSurvivalTrees" target="_blank">ExtraSurvivalTrees</a>, and <a href="https://scikit-survival.readthedocs.io/en/v0.22.0/api/generated/sksurv.tree.SurvivalTree.html#sksurv.tree.SurvivalTree" target="_blank">SurvivalTree</a>.</li>
<li>Support for <a href="https://scikit-survival.readthedocs.io/en/v0.22.0/api/generated/sksurv.meta.Stacking.html#sksurv.meta.Stacking.predict_cumulative_hazard_function" target="_blank">predict_cumulative_hazard_function()</a> and <a href="https://scikit-survival.readthedocs.io/en/v0.22.0/api/generated/sksurv.meta.Stacking.html#sksurv.meta.Stacking.predict_survival_function" target="_blank">predict_survival_function()</a> in <a href="https://scikit-survival.readthedocs.io/en/v0.22.0/api/generated/sksurv.meta.Stacking.html" target="_blank">Stacking</a>.</li>
</ul>
<h2 id="missing-values-support-in-survivaltree">Missing Values Support in SurvivalTree</h2>
<p>Based on the <a href="https://scikit-learn.org/1.3/modules/tree.html#tree-missing-value-support" target="_blank">missing value support</a>
in scikit-learn 1.3, a <a href="https://scikit-survival.readthedocs.io/en/v0.22.0/api/generated/sksurv.tree.SurvivalTree.html#sksurv.tree.SurvivalTree" target="_blank">SurvivalTree</a>
can now deal with missing values if it is fit with <code>splitter='best'</code>.</p>
<p>If the training data contained no missing values, then during prediction missing values are mapped
to the child node with the most samples:</p>
<pre><code class="language-python">X, y = load_veterans_lung_cancer()
X_train = np.asarray(X.loc[:, ["Karnofsky_score"]], dtype=np.float32)
est = SurvivalTree(max_depth=1)
est.fit(X_train, y)
X_test = np.array([[np.nan]])
surv_fn = est.predict_survival_function(X_test, return_array=True)
mask = X_train[:, 0] > est.tree_.threshold[0]
km_x, km_y = kaplan_meier_estimator(
y[mask]["Status"], y[mask]["Survival_in_days"]
)
plt.step(km_x, km_y, where="post", linewidth=5)
plt.step(
est.unique_times_, surv_fn[0], where="post", linewidth=3, linestyle="dotted"
)
plt.ylim(0, 1)
</code></pre>
<img src="https://k-d-w.org/blog/2023/10/scikit-survival-0.22.0-released/img/survival-tree-missing-values-1.svg" width="400" />
<p>If a tree is fit to training data with missing values, the splitter will evaluate
each split with all samples with missing values going to the left or the right child node.</p>
<pre><code class="language-python">X, y = load_veterans_lung_cancer()
X_train = np.asarray(X.loc[:, ["Age_in_years"]], dtype=np.float32)
X_train[-50:, :] = np.nan
est = SurvivalTree(max_depth=1)
est.fit(X_train, y)
X_test = np.array([[np.nan]])
surv_fn = est.predict_survival_function(X_test, return_array=True)
mask = X_train[:, 0] > est.tree_.threshold[0]
mask |= np.isnan(X_train[:, 0])
km_x, km_y = kaplan_meier_estimator(
y[mask]["Status"], y[mask]["Survival_in_days"]
)
plt.step(km_x, km_y, where="post", linewidth=5)
plt.step(
est.unique_times_, surv_fn[0], where="post", linewidth=3, linestyle="dotted"
)
plt.ylim(0, 1)
</code></pre>
<img src="https://k-d-w.org/blog/2023/10/scikit-survival-0.22.0-released/img/survival-tree-missing-values-2.svg" width="400" />
<p>These rules are identical to those of <a href="https://scikit-learn.org/1.3/modules/tree.html#tree-missing-value-support" target="_blank">scikit-learn’s missing value support</a>.</p>
<h2 id="low-memory-mode-for-survivaltree-and-randomsurvivalforest">Low-memory Mode for SurvivalTree and RandomSurvivalForest</h2>
<p>The last release already saw
<a href="https://k-d-w.org/blog/2023/06/scikit-survival-0.21.0-released/" target="_blank">performance improvments to SurvivalTree and RandomSurvivalForest</a>.
This release adds the <code>low_memory</code> option to
<a href="https://scikit-survival.readthedocs.io/en/v0.22.0/api/generated/sksurv.ensemble.RandomSurvivalForest.html#sksurv.ensemble.RandomSurvivalForest" target="_blank">RandomSurvivalForest</a>, <a href="https://scikit-survival.readthedocs.io/en/v0.22.0/api/generated/sksurv.ensemble.ExtraSurvivalTrees.html#sksurv.ensemble.ExtraSurvivalTrees" target="_blank">ExtraSurvivalTrees</a>, and <a href="https://scikit-survival.readthedocs.io/en/v0.22.0/api/generated/sksurv.tree.SurvivalTree.html#sksurv.tree.SurvivalTree" target="_blank">SurvivalTree</a>.</p>
<p>If low-memory mode is <em>disabled</em>, which is the default, calling <code>predict</code> on a sample will require memory
in the order of unique event times in the training data, because the cumulative hazard function
is computed as an intermediate value.
If low-memory mode is <em>enabled</em>, then the risk score is computed directly, without computing
the cumulative hazard function. However, low-memory mode disables using
<code>predict_cumulative_hazard_function</code> and <code>predict_survival_function</code>.</p>
<h2 id="install">Install</h2>
<p>Pre-built conda packages are available for Linux, macOS, and Windows, either</p>
<p>via pip:</p>
<pre><code class="language-bash">pip install scikit-survival
</code></pre>
<p>or via conda</p>
<pre><code class="language-bash"> conda install -c sebp scikit-survival
</code></pre>2023-10-02T20:39:15+00:00GStreamer: New GStreamer Discourse forum
https://gstreamer.freedesktop.org/news/#2023-09-23T18:30:00Z
<p>
The GStreamer project is thrilled to announce that there is now a new
GStreamer Discourse forum up and running at
<a href="https://discourse.gstreamer.org">https://discourse.gstreamer.org</a>,
offering a modern and welcoming platform for support requests and
discussions.
</p><p>
Anybody is welcome to join and discuss, ask questions, reply, etc...
</p><p>
We look forward to seeing you all there.
</p><p>
<a href="https://discourse.gstreamer.org"><img src="https://gstreamer.freedesktop.org/news/discourse.png" alt="Discourse forum screenshot" width="80%" /></a>
</p>2023-09-23T18:30:00+00:00GStreamer: GStreamer 1.22.6 stable bug fix release
https://gstreamer.freedesktop.org/news/#2023-09-20T20:00:00Z
<p>
The GStreamer team is pleased to announce another bug fix release
in the stable 1.22 release series of your favourite cross-platform
multimedia framework!
</p><p>
This release only contains bugfixes and security fixes and it should be safe
to update from 1.22.x.
</p><p>
<b>Highlighted bugfixes:</b>
<ul>
<li><a href="https://gstreamer.freedesktop.org/security/">Security fixes</a> for the MXF demuxer and H.265 video parser</li>
<li>Fix latency regression in H.264 hardware decoder base class</li>
<li>androidmedia: fix HEVC codec profile registration and fix coded_data handling</li>
<li>decodebin3: fix switching from a raw stream to an encoded stream</li>
<li>gst-inspect: prettier and more correct signal and action signals printing</li>
<li>rtmp2: Allow NULL flash version, omitting the field, for better RTMP server compatibility</li>
<li>rtspsrc: better compatibility with buggy RTSP servers that don't set a clock-rate</li>
<li>rtpjitterbuffer: fix integer overflow that led to more packets being declared lost than have been lost</li>
<li>v4l2: fix video encoding regression on RPi and fix support for left and top padding</li>
<li>waylandsink: Crop surfaces to their display width height</li>
<li>cerbero: Recognise Manjaro; add Rust support for MSVC ARM64; cmake detection fixes</li>
<li>various bug fixes, memory leak fixes, and other stability and reliability improvements</li>
</ul>
</p><p>
See the <a href="https://gstreamer.freedesktop.org/releases/1.22/#1.22.6">GStreamer 1.22.6 release notes</a>
for more details.
</p><p>
Binaries for Android, iOS, Mac OS X and Windows will be available shortly.
</p><p>
Release tarballs can be downloaded directly here:
<ul>
<a href="https://gstreamer.freedesktop.org/src/gstreamer/gstreamer-1.22.6.tar.xz">gstreamer</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-base/gst-plugins-base-1.22.6.tar.xz">gst-plugins-base</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-good/gst-plugins-good-1.22.6.tar.xz">gst-plugins-good</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-ugly/gst-plugins-ugly-1.22.6.tar.xz">gst-plugins-ugly</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-bad/gst-plugins-bad-1.22.6.tar.xz">gst-plugins-bad</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-libav/gst-libav-1.22.6.tar.xz">gst-libav</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-rtsp-server/gst-rtsp-server-1.22.6.tar.xz">gst-rtsp-server</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-python/gst-python-1.22.6.tar.xz">gst-python</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-editing-services/gst-editing-services-1.22.6.tar.xz">gst-editing-services</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-devtools/gst-devtools-1.22.6.tar.xz">gst-devtools</a>,
<a href="https://gstreamer.freedesktop.org/src/gstreamer-vaapi/gstreamer-vaapi-1.22.6.tar.xz">gstreamer-vaapi</a>,
<a href="https://gstreamer.freedesktop.org/src/gstreamer-sharp/gstreamer-sharp-1.22.6.tar.xz">gstreamer-sharp</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-omx/gst-omx-1.22.6.tar.xz">gst-omx</a>, or
<a href="https://gstreamer.freedesktop.org/src/gstreamer-docs/gstreamer-docs-1.22.6.tar.xz">gstreamer-docs</a>.
</ul>
</p>2023-09-20T20:00:00+00:00GStreamer: GStreamer Conference 2023: Full Schedule, Talk Abstracts and Speakers Biographies now available
https://gstreamer.freedesktop.org/news/#2023-09-16T16:30:00Z
<p>
The GStreamer Conference team is pleased to announce that the full conference
schedule including talk abstracts and speaker biographies is now available for
this year's lineup of talks and speakers, covering again an exciting range of
topics!
</p><p>
The <b>GStreamer Conference 2023</b> will take place on <b>25-26 September 2023
in A Coruña, Spain</b>, followed by a hackfest.
</p><p>
Details about the conference, hackfest and how to register can be found on the
<a href="https://gstreamer.freedesktop.org/conference/2023/">conference website</a>.
</p><p>
<b>This year's topics and speakers:</b>
<ul>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/174/">
GStreamer State of the Union
</a>
<br />
<small><i>Tim-Philipp Müller, Centricular</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/238/">
Video Editing with GStreamer: an update
</a>
<br />
<small><i>Thibault Saunier, Igalia</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/239/">
GstWebRTC in WebKit, current status and challenges
</a>
<br />
<small><i>Philippe Normand, Igalia</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/221/">
Updates in GStreamer VA-API (and DMABuf negotiation)
</a>
<br />
<small><i>Victor Manuel Jáquez Leal, Igalia</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/222/">
Vulkan Video in GStreamer
</a>
<br />
<small><i>Victor Manuel Jáquez Leal, Igalia; Stéphane Cerveau, Igalia</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/240/">
NVIDIA Deepstream and GStreamer for AI workloads
</a>
<br />
<small><i>Brad Greenway, Equature</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/256/">
API translation layers: the benefits and challenges of using GStreamer in Wine
</a>
<br />
<small><i>Zeb Figura, CodeWeavers</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/258/">
Deep Upstream : Hardware Agnostic GStreamer Analytics
</a>
<br />
<small><i>Aaron Boxer, Collabora; Daniel Morin, Collabora</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/248/">
New MPEG-5 part 2 (LCEVC) plugin for GStreamer
</a>
<br />
<small><i>Julian Bouzas, Collabora</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/241/">
GstWASM: GStreamer for the web
</a>
<br />
<small><i>Jorge Zapata, Fluendo</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/223/">
DgiStreamer: a Gstreamer Pipeline Editor
</a>
<br />
<small><i>Mattia Angelini, Cyens</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/224/">
RidgeRun / Texas Instruments: Edge AI GStreamer Plugins
</a>
<br />
<small><i>Marco Herrera-Valverde, RidgeRun</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/225/">
GstPluginPylon: A Study of Dynamic Element Properties in Basler Cameras
</a>
<br />
<small><i>Miguel Taylor-Lopez, RidgeRun</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/264/">
Lessons learnt in new playback and adaptive components
</a>
<br />
<small><i>Edward Hervey, Centricular</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/226/">
ICE: How to find your way through the internet
</a>
<br />
<small><i>Matthew Waters, Centricular</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/227/">
Closed Captions: What GStreamer Can Do
</a>
<br />
<small><i>Matthew Waters, Centricular</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/242/">
WirePlumber 0.5: Making PipeWire policies easier
</a>
<br />
<small><i>George Kiagiadakis, Collabora</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/247/">
Patching a 3rd-party plugin in runtime
</a>
<br />
<small><i>Alexander Slobodeniuk, Fluendo</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/243/">
More Efficient Streaming using Linux DRM Modifiers
</a>
<br />
<small><i>Nicolas Dufresne, Collabora</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/257/">
Building your own USB camera with GStreamer
</a>
<br />
<small><i>Michael Grzeschik, Pengutronix</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/244/">
How we are building a distributed multi-camera real-time sports tracking system using GStreamer and Rust
</a>
<br />
<small><i>Robin Gustavsson, Spiideo; Daniel Pendse, Spiideo</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/249/">
The evolution of HTTP based signalling for WebRTC in GStreamer
</a>
<br />
<small><i>Taruntej Kanakamalla, asymptotic.io</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/250/">
Bridging WebRTC and SIP using GStreamer & SIPjs
</a>
<br />
<small><i>Sanchayan Maity, asymptotic.io</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/251/">
Server-side Media Processing with GStreamer
</a>
<br />
<small><i>Arun Raghavan, asymptotic.io</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/228/">
Applications of GStreamer in Surgical Devices
</a>
<br />
<small><i>Matthias Fuchs, ZEISS Group</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/229/">
MSE and EME on GStreamer based WebKit ports
</a>
<br />
<small><i>Xabier Rodríguez Calvar, Igalia</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/252/">
High-level WebRTC APIs in GStreamer
</a>
<br />
<small><i>Thibault Saunier, Igalia; Mathieu Duponchelle, Centricular; Loïc Le Page, Igalia</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/254/">
WebRTC in Axis cameras and in the surveillance industry
</a>
<br />
<small><i>Jonas Cremon, Axis Communications</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/230/">
Adding Rust to a C++ GStreamer WebRTC application
</a>
<br />
<small><i>Johan Sternerup, Axis Communications</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/234/">
ONVIF metadata streams in Network security cameras
</a>
<br />
<small><i>Linus Svensson, Axis Communications; Johan Bjäreholt, Axis Communications</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/246/">
Fluster: A framework for multimedia decoder conformance testing
</a>
<br />
<small><i>Rubén Gonzalez, Fluendo; Michalis Dimopoulos, Fluendo</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/235/">
HYPE: HYbrid Parallel Encoder
</a>
<br />
<small><i>Rubén Gonzalez, Fluendo</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/236/">
Flumes: Scan and index your multimedia files
</a>
<br />
<small><i>Michalis Dimopoulos, Fluendo</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/237/">
Improved RTSP connection latency for live streams
</a>
<br />
<small><i>Jacob Johnsson, Axis Communications</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/231/">
GStreamer and VSCode: a love story
</a>
<br />
<small><i>Andoni Morales Alastruey, Fluendo</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/232/">
Transcription, Translation, Closed Captions
</a>
<br />
<small><i>Mathieu Duponchelle, Centricular</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/245/">
Writing GStreamer applications with C#
</a>
<br />
<small><i>Andoni Morales Alastruey, Fluendo</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/259/">
HLS/LL-HLS and DASH playback
</a>
<br />
<small><i>Jan Schmidt, Centricular</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/253/">
Developing Low latency Video Telephony solutions using GStreamer
</a>
<br />
<small><i>Devarsh Thakkar, Texas Instruments</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/233/">
Variations on a WebRTC relay architecture (featuring Janus and WebRTCSrc + WebRTCSink)
</a>
<br />
<small><i>François Laignel, Centricular</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/260/">
Adding W3C Media Source Extensions and Encrypted Media Extensions to GStreamer
</a>
<br />
<small><i>Jordan Yelloz, Collabora</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/255/">
libcamerasrc: Introduction and usage of libcamera's gstreamer element
</a>
<br />
<small><i>Umang Jain, IdeasOnBoard</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/261/">
Pexip + GStreamer
</a>
<br />
<small><i>Håvard Graff, Pexip</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/262/">
Four Years of Cross-platform Improvements
</a>
<br />
<small><i>Nirbheek Chauhan, Centricular</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/263/">
Splitting GStreamer pipelines
</a>
<br />
<small><i>Jan Schmidt, Centricular</i></small>
</li>
</ul>
</p><p>
<b>Lightning Talks:</b>
<ul>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/218/">Update of Four Years of V4L2 Support</a>
<small><br /><i>Nicolas Dufresne, Collabora</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/170/">WebCodecs in WebKit, with GStreamer!</a>
<small><br /><i>Philippe Normand, Igalia</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/172/">GStreamer Daemon Project Update</a>
<small><br /><i>Miguel Taylor-Lopez, RidgeRun</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/173/">Playing around with Artistic Style Transfer with GStreamer + NNStreamer</a>
<small><br /><i>Hosang Lee, LG Electronics</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/219/">libcamerasrc: Sensor Configuration and Enumeration</a>
<small><br /><i>Umang Jain, IdeasOnBoard</i></small></li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/220/">3 milliseconds from Diaphragm to Diaphragm</a>
<small><br /><i>Nirbheek Chauhan, Centricular</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/265/">GstPipelineStudio version 0.3.0 is out!</a>
<small><br /><i>Stéphane Cerveau, Igalia</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/266/">Streamway: From multistreaming to Browser based live stream studio & beyond</a>
<small><br /><i>Ramjivan Jangid, Streamway</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/267/">Exploring the Future of Fun: 5G, Interactive Entertainment, and GStreamer Magic</a>
<small><br /><i>Ramjivan Jangid, Streamway</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/171/">GStreamer plugin in Rust on webOS OSE</a>
<small><br /><i>Seungwook Cha, LG Electronics</i></small>
</li>
<li>
<a href="https://indico.freedesktop.org/event/5/contributions/269/">Behind the GStreamer Conferences video archive</a>
<small><br /><i>Florent Thiéry, Ubicast</i></small>
</li>
<li>...and more to come</li>
<li><i>Submit <u>your</u> lightning talk now!</i></li>
</ul>
</p><p>
Many thanks to our amazing sponsors ‒ Platinum sponsors
<a href="https://igalia.com">Igalia</a>,
<a href="https://www.collabora.com">Collabora</a>,
<a href="https://fluendo.com">Fluendo</a>, and
<a href="https://www.pexip.com">Pexip</a>,
Gold sponsors
<a href="https://www.axis.com">Axis Communications</a>, and
<a href="https://centricular.com/">Centricular</a>,
and Silver sponsors
<a href="https://zeiss.com">Zeiss</a>,
<a href="https://ridgerun.com">RidgeRun</a>,
<a href="https://laerdal.com">Laerdal Labs</a> and
<a href="https://asymptotic.io">asymptotic</a>,
without whom the conference would not be possible in this form. And to
<a href="http://www.ubicast.eu">Ubicast</a> who will be recording and live streaming the talks again.
</p><p>
We hope to see you all A Coruña! Don't forget to <a href="https://gstreamer2023.serglo.es/">register</a>!
</p>2023-09-16T16:30:00+00:00Jean-François Fortin Tam: Help us make GNOME Calendar rock-solid by expanding the test suite!
https://fortintam.com/blog/call-for-help-writing-gnome-calendar-compliance-unit-tests/
<p><a href="https://wiki.gnome.org/Apps/Calendar" target="_blank" rel="noopener">GNOME Calendar</a> 45 will be a groundbreaking release in terms of UX (more on that later?), performance, and to some extent, reliability (we’ve at least solved two complex crashers recently, including a <a rel="noreferrer noopener" href="https://mastodon.social/@nekohayo/110939643457489280" target="_blank">submarine Cthulhu</a> crasher <a rel="noreferrer noopener" href="http://en.wikipedia.org/wiki/Unusual_software_bug" target="_blank">heisen</a>bug and its offspring)… and yet, I think this might be “just the beginning” of a new era. And <em>a beginning… is a very delicate time.</em></p>
<span id="more-6495"></span>
<p>If you’ve tried to use GNOME Calendar in the past decade or so, you’ve certainly encountered one of the many bugs related to timezones, daylight saving time, and all of that crazy <a rel="noreferrer noopener" href="https://www.youtube.com/watch?v=-5wpm-gesOY" target="_blank">stuff that would make Tom Scott curl up in a corner and cry</a>. But it doesn’t have to be that way, and in fact, there <em>is</em> a way for anyone who knows a bit of C programming to help us build the tools to solve this mission-critical problem.</p>
<p>Today, I’d like to <em>urge</em> you to help in writing some automated tests.<br />The sooner our test suite can grow, the faster we can make GNOME Calendar <em>rock-solid.</em></p>
<img width="500" height="544" src="https://fortintam.com/blog/wp-content/uploads/GNOME-Calendar-anime-girl-hiding-from-automated-tests.jpg" alt="The " />
<p>As I explain in <a rel="noreferrer noopener" href="https://gitlab.gnome.org/GNOME/gnome-calendar/-/issues/1093" target="_blank">this epic blocker ticket</a>:</p>
<blockquote class="wp-block-quote">
<p>[…] There currently are <a href="https://gitlab.gnome.org/GNOME/gnome-calendar/-/tree/main/tests" target="_blank" rel="noopener">some unit tests</a> in GNOME Calendar, but not nearly enough, and we <em>need</em> to cover a lot more in order to fix the pile of <a href="https://gitlab.gnome.org/GNOME/gnome-calendar/-/issues?label_name%5B%5D=Timezones" target="_blank" rel="noopener">timezone-related issues</a> […]</p>
<p>We really need to use <a rel="noreferrer noopener" href="https://en.wikipedia.org/wiki/Test-driven_development" target="_blank">test-driven development</a> here, otherwise these bugs will be a nightmare to fix and to <em>keep fixed forever</em>. Before we start fixing those bugs, we need as many unit tests as possible to cover spec compliance.</p>
<p>By helping write unit tests to ensure GNOME Calendar complies with standard calendaring specifications, your contribution will make a <strong>huge</strong> impact in GNOME Calendar’s future reliability, as an improved test suite will make it 100x easier for us to fix existing bugs while preventing introducing new bugs.</p>
<p>Doing this will also help us more confidently expand GNOME Calendar’s featureset to handle <a href="https://gitlab.gnome.org/GNOME/gnome-calendar/-/issues?label_name%5B%5D=Meetings" target="_blank" rel="noopener">meetings</a> planning functionality. […]</p>
</blockquote>
<h2 class="wp-block-heading">Why is this suddenly mission-critical <em>now?</em></h2>
<p>I believe a boosted test suite is now becoming mission-critical, because in my <a rel="noreferrer noopener" href="https://gitlab.gnome.org/GNOME/gnome-calendar/-/boards" target="_blank">logical ordering</a> of the dependency chain for <a rel="noreferrer noopener" href="https://gitlab.gnome.org/GNOME/gnome-calendar/-/issues/965" target="_blank">this new roadmap</a> I devised in 2023, I have determined that timezones/DST/midnight backend+views reliability blocks almost everything left: you can’t expand the UI and UX featureset to encompass meetings & timezones management unless you have a rock-solid, reliable backend and views/representation that you can trust to begin with (why create nice UIs if the users won’t trust the resulting events/data/views?)</p>
<p>So we need to fix those ugly issues <em>once and for all</em>. And there’s pretty much only one way to do this kind of daunting bugfixing work efficiently and sustainably: with test-driven development.</p>
<p>Once we have those tests, I trust that we will be able to make the app pretty much <em>bulletproof.</em></p>
<img width="600" height="367" src="https://fortintam.com/blog/wp-content/uploads/Testing-a-bulletproof-vest-1923-cropped.jpg" alt="" class="wp-image-2477" />
<p>Having extensive automated test coverage will allow us to confidently/safely fix some of the first “fundamental” compliance bugs there, and I can bet that by doing so, we would incidently solve <em>tons</em> of other issues at once—I wouldn’t be surprised if this would allow us to solve 50 issues or more. This will not only make a ton of existing users happy (and make GNOME Calendar viable for a <em>ton</em> more new users), but also make <em>contributors</em> happy because we will be able to resolve and close a <em>ton</em> of tickets cluttering our view. It’s all part of a flywheel of sustainability here.</p>
<h2 class="wp-block-heading">“How hard can it be?”</h2>
<p>While writing such tests might not be easy to do for a “beginner programmer who does not know the C language”, it <em>is</em> a good “newcomer contributor” task for someone who already knows C programming and are looking to make a lasting quality impact on a popular FLOSS desktop productivity application that many people love.</p>
<p>Maybe you, or a friend or colleague, would be interested by this challenge (while existing GNOME Calendar contributors keep fixing bugs at the same time)? This blog post is <a rel="noreferrer noopener" href="https://mastodon.social/@nekohayo/111018739466816748" target="_blank">retootable</a> if you want to spread the word there, too.</p>
<h2 class="wp-block-heading">Ready to help us improve reliability?</h2>
<p>If the above sounds like an interesting way to help the project, read <a rel="noreferrer noopener" href="https://gitlab.gnome.org/GNOME/gnome-calendar/-/issues/1093" target="_blank">the detailed ticket</a> for more details, join us on the cozy “<a rel="noreferrer noopener" href="https://matrix.to/#/#gnome-calendar:gnome.org" target="_blank">#gnome-calendar:gnome.org</a>” <a rel="noreferrer noopener" href="https://wiki.gnome.org/GettingInTouch/Matrix" target="_blank">Matrix</a> channel (mapped to the <a href="irc://irc.libera.chat/gnome-calendar">#gnome-calendar IRC channel on irc.libera.chat</a>) and get started!</p>2023-09-06T14:12:43+00:00Jean-François Fortin Tam: Why I picked the biggest furry elephant as my microblogging platform (and refuse to self-host a Mastodon server)
https://fortintam.com/blog/migration-from-twitter-to-mastodon-biggest-server/
<p class="has-small-font-size">This article will require <strong>between 1 and 2 minutes of your attention</strong> if you read only the first half; obviously double that if you also feel like reading the second (more philosophical & strategic) half.</p>
<span id="more-6464"></span>
<hr class="wp-block-separator has-alpha-channel-opacity" />
<p class="has-drop-cap">As you may know, in addition to this blog here, I have also been microblogging very actively for years (whether <a rel="noreferrer noopener" href="https://twitter.com/nekohayo" target="_blank">on Twitter</a> or <a rel="noreferrer noopener" href="https://linkedin.com/in/fortintam/" target="_blank">on LinkedIn</a>), particularly the day-to-day / work-in-progress of my Free & Open Source software contributions across <a rel="noreferrer noopener" href="https://gnome.org" target="_blank">GNOME</a> and the FreeDesktop, and that habit shall outlive Twitter’s 2022-2023 <a rel="noreferrer noopener" href="https://www.theverge.com/2022/11/11/23451931/elon-musk-twitter-bankrupt-verification-ftc" target="_blank">chaotic</a> hostile takeover and sabotage by its new majority shareholder/owner. I have (reluctantly) found refuge in the shire that is the fediverse, a quirky platform filled with countless technical & usability challenges, but eh, what else have we got left? Tis the last bastion we have (we’ll see what happens when Meta/Facebook “enters the chat”, will it be like <a rel="noreferrer noopener" href="https://fortintam.com/blog/the-death-of-instant-messaging-in-2013/" target="_blank">what happened with XMPP</a>? 🤷)…</p>
<a href="https://www.peppercarrot.com/en/viewer/misc__2023-06-21_MAY-I-JOIN-YOU_by-David-Revoy.html" target="_blank" rel="noopener"><img width="1500" height="952" src="https://fortintam.com/blog/wp-content/uploads/David-Revoys-speedpainting-on-Meta-and-the-fediverse-May-I-join-you.jpg" alt="" /></a>“<a href="https://www.peppercarrot.com/en/viewer/misc__2023-06-21_MAY-I-JOIN-YOU_by-David-Revoy.html" target="_blank" rel="noreferrer noopener">MAY I JOIN YOU</a>” by David Revoy − CC-BY-SA 4.0, with fair-use elements
<p>And so, some of you might be pleased to hear that I have been dragged—kicking and screaming—to <em>the Mastodon,</em> as my replacement for <em>the Twitter. </em>I just, uh, “forgot” to tell y’all. </p>
<p>Seriously, it was a <em>really</em> long and busy R&D winter! I didn’t have time to announce this properly here; when registrations reopened and I <em>finally</em> could sign up for the largest* general-purpose instance (maintained by the nonprofit <em>Mastodon gGmbH</em>), it was already “income tax season” <em>and</em> I was busy helping the GNOME <a rel="noreferrer noopener" href="https://wiki.gnome.org/Apps/Calendar" target="_blank">Calendar</a>, <a rel="noreferrer noopener" href="https://en.wikipedia.org/wiki/GNOME_Files" target="_blank">Nautilus</a>, and <a rel="noreferrer noopener" href="https://en.wikipedia.org/wiki/GNOME_Web" target="_blank">Epiphany</a> projects… so I pretty much only had time to change my Twitter profile banner to this picture (because any blog post like what you’re reading is at <em>least</em> 5 to 7 hours of work):</p>
<a href="https://mastodon.social/@nekohayo" target="_blank" rel="noreferrer noopener"><img width="1024" height="341" src="https://fortintam.com/blog/wp-content/uploads/2023-04-social-migration-banner-1024x341.jpg" alt="A profile banner on my old social media microblogging account, pointing to my personal Mastodon account" class="wp-image-6465" title="Yes, this is the " /></a>
<p>So yeah, follow <a rel="noreferrer noopener" href="https://mastodon.social/@nekohayo" target="_blank">my personal Mastodon account</a> if you hadn’t found me already (and if you ain’t on Mastodon, I guess there’s still <a rel="noreferrer noopener" href="https://fortintam.com/blog/2019-starting-an-email-list/" target="_blank">this blog’s email notification system</a> in addition to RSS).</p>
<p>Due to the inherent cultural difference of Mastodon (more on that below), my account there is somewhat more specialized than it was on Twitter. As stated in my <a rel="noreferrer noopener" href="https://mastodon.social/@nekohayo/110035313535650002" target="_blank">introduction toot</a>, I focus mainly on creative, positive & uplifting stuff. It’s more fun, and it pretty much allows me to not worry about the whole “contents warning” minefield. My account is primarily a way for me to share:</p>
<ul>
<li>My findings and contributions to Free & Open Source software “as they happen in the day-to-day”, particularly around GNOME & FreeDesktop-related projects on Linux; I often post my bug reports there, particularly if it is about usability & interaction design, or performance optimization. I sometimes publish awareness and advocacy posts related to such topics.</li>
<li>Occasionally, posts related to cycling, ecology, sustainability, gardening, urbanism and societal improvements.</li>
</ul>
<p>If you’d be interested in things beyond FLOSS and casual life updates, you can also follow my three FLOSS-friendly businesses:</p>
<ul>
<li>the <a rel="noreferrer noopener" href="https://idmark.ca" target="_blank">idéemarque creative agency</a>‘s Mastodon account <a href="https://mastodon.social/@ideemarque" target="_blank" rel="noreferrer noopener">here</a>;</li>
<li>the <a rel="noreferrer noopener" href="https://atypi.ca" target="_blank">Atypica studio</a>‘s Mastodon account <a href="https://mastodon.social/@atypica" target="_blank" rel="noreferrer noopener">here</a>;</li>
<li>the <a rel="noreferrer noopener" href="https://regen.to" target="_blank">Regento strategic marketing</a> & executive services firm’s Mastodon account <a href="https://mastodon.social/@regento" target="_blank" rel="noreferrer noopener">here</a>.</li>
</ul>
<p>As you can see, those three extra accounts have been very quiet. They are not going to be corporatey spammage / annoyances, they are simply intended to be a way for interested parties to be notified when I have some neat accomplishments to show (whether in FLOSS-related marketing work, or in other industries) or when we publish a new article in those various fields of work. I don’t expect that the Mastodon/fediverse crowd will show much interest in those, but I’ll be flattered if some of you do!</p>
<hr class="wp-block-separator has-alpha-channel-opacity" />
<div class="wp-block-image">
<a href="https://www.peppercarrot.com/en/viewer/misc__2023-04-13_Goodbye-Blue-Bird_by-David-Revoy.html" target="_blank" rel="noreferrer noopener"><img src="https://fortintam.com/blog/wp-content/uploads/Goodbye-Blue-Bird-by-David-Revoy.jpg" alt="" /></a><a href="https://www.peppercarrot.com/en/viewer/misc__2023-04-13_Goodbye-Blue-Bird_by-David-Revoy.html" target="_blank" rel="noreferrer noopener">“Goodbye Blue Bird”</a> by David Revoy − <a href="https://creativecommons.org/licenses/by/4.0/deed.en" target="_blank" rel="noreferrer noopener">CC-BY 4.0</a></div>
<p>While I kept servicing the Twitter accounts at the same time as the Mastodon accounts for a couple of months, lately it has been abundantly clear that Twitter has truly become a deserted place, and the continued sabotage of its features & reliability has made it pretty much impossible to use professionally and personally.</p>
<p>That’s the end of the article if you simply want to follow my adventures… (◕‿◕✿)</p>
<p>Below are some short philosophical & sustainability observations on the whole situation.</p>
<hr class="wp-block-separator has-alpha-channel-opacity" />
<h2 class="wp-block-heading">Brief thoughts on the societal bubble of the fediverse</h2>
<p>As you can see by my high amount of activity on my personal Mastodon account, I have taken a certain liking to it, even though I was <em>sure</em> I would hate it. I guess this platform sort of works if your audience is primarily <a href="https://en.wikipedia.org/wiki/Free_and_open-source_software" target="_blank" rel="noreferrer noopener">FLOSS</a> enthusiasts and you spent months studying the cultural & technical quirks. And yet, I am not blind to the fact that I happen to live in one of the very few bubbles that had high affinity with Mastodon to begin with.</p>
<img width="602" height="395" src="https://fortintam.com/blog/wp-content/uploads/Matrix-white-rabbit-tattoo-lady.jpg" alt="" />
<p>I still miss the ability to do topical research and feel the “<a rel="noreferrer noopener" href="https://techcrunch.com/2013/01/15/twitters-social-impact-cant-be-measured-but-its-the-pulse-of-the-planet/" target="_blank">pulse of the planet</a>“, and to follow more “mainstream” people like my local policymakers and organizations.</p>
<ul>
<li>Pretty much none of the local architects, urbanists, public institutions, mayors, journalists, businesses, neighbors, and many other “normal” people, are there to engage with. Hoping any of them will join—when even <em>I</em> didn’t want to join unless I had no other options left—is wishful thinking; talking about the fediverse to normies gets you the same blank stares as when you say you run Linux and use FLOSS software—completely abstract to them even if you explain, and they will never care beyond the two-minutes casual conversation. At best they will say “Are you <a href="https://www.youtube.com/watch?v=K7Hn1rPQouU" target="_blank" rel="noreferrer noopener">a hacker</a>?!” and then you inevitably say, “Depends on your definition of the word <a href="https://en.wikipedia.org/wiki/Hacker" target="_blank" rel="noreferrer noopener">hacker</a>…”</li>
<li>There could be a major event occuring in my neighborhood and I would never hear about it in realtime through the fediverse. My neighborhood has over 20 thousand people living there—with higher density than <a rel="noreferrer noopener" href="https://en.wikipedia.org/wiki/Brooklyn" target="_blank">Brooklyn</a>—and yet I know of exactly <em>two</em> other individuals from my neighborhood who are on Mastodon… and <em>of course</em> they work in tech.</li>
</ul>
<p>Twitter’s global and serendipitous system was the biggest value “for users” (especially journalists) that I feel we have lost, and there is no alternative.</p>
<h2 class="wp-block-heading">Some thoughts on instances’ sustainability</h2>
<p class="has-text-align-left">*: And now, a big footnote for those who wonder why I specifically picked the biggest instance I could, and who might say:</p>
<div class="wp-block-image is-style-default">
<img src="https://fortintam.com/blog/wp-content/uploads/mastodon.social-is-too-mainstream.jpg" alt="Hipster Ariel says, " /></div>
<p>I do not intend to switch to some niche Mastodon server/instance, unless my life depends on it.</p>
<ul>
<li>There are <em>many</em> technical (and social) reasons why I do not want to be confined to a tiny server where it feels like <a rel="noreferrer noopener" href="https://www.youtube.com/watch?v=nUsDk8wjRPs" target="_blank">a Linuxcon happening in a deserted strip mall</a>, and you experience the full extent of decentralization bugs & technical limitations (I have at least two dozen bookmarks to bug reports related to that);</li>
<li>I don’t want to lawyer up to figure out which server to pick, and to be subjected to complicated and/or arbitrary rules (FOSStodon users have been learning <a rel="noreferrer noopener" href="https://discussion.fedoraproject.org/t/move-mastodon-account-off-of-fosstodon/87843" target="_blank">this lesson</a> in the last few days; I already learned the hard way in 2020 that there was such a thing as <a rel="noreferrer noopener" href="https://github.com/getting-things-gnome/gtg/issues/294#issuecomment-621319344" target="_blank">picking the “wrong” server or username</a> and realizing that you can’t cleanly change some things)</li>
<li>I don’t want to be dependent on a tiny overworked team of volunteers burning out, suddenly disappearing along with the instance, or getting hit by a <a rel="noreferrer noopener" href="https://en.wikipedia.org/wiki/Bus_factor" target="_blank">bus</a>.</li>
</ul>
<p>Decentralization is <em>very</em> neat conceptually, but I’ll take my chances with the biggest mainstream instance that has the best odds of still being around and able to pay their servers & moderators in two years. Remember, kids, Twitter was not just a failwhale propped up by <a rel="noreferrer noopener" href="https://www.youtube.com/watch?v=y8OnoxKotPQ" target="_blank">invisible ropes and ducktape</a>, it was also a white elephant whose value never was in its FUBAR software & infrastructure, but in the brand safety (for advertisers) provided by <a href="https://www.theverge.com/2022/10/28/23428132/elon-musk-twitter-acquisition-problems-speech-moderation">its paid moderation team,</a> and the handling of <a rel="noreferrer noopener" href="https://denise.dreamwidth.org/91757.html" target="_blank">legal/moral/safety/security/liability nightmares that user-generated content represent</a>. As I summarized my readings in that tweet, back when tweeting was still a thing:</p>
<div class="wp-block-embed__wrapper">
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">It's not the 90's anymore.<br />Public service with user-generated content = soul-crushing moderation & infosec nightmare + legal inferno.<br /><br />Twitter isn't software. It's people paid to handle uptime, moderation, compliance.<br /><br />This thread is why you do not want to run a Mastodon server. <a href="https://t.co/LhABlsh3uH">https://t.co/LhABlsh3uH</a></p>— Jeff 🎆 (@nekohayo) <a href="https://twitter.com/nekohayo/status/1594864822889418752?ref_src=twsrc%5Etfw">November 22, 2022</a></blockquote>
</div>
<p>A couple of years prior, I had also educated various clients about the risks and liabilities of running social media platforms (my clients were, as a result, in a better position to make their own decisions according to their own objectives, threat models, resources and risk tolerance). </p>
<p>I’ve seen enough safety & security nightmares out there throughout the years, so personally speaking, I’ll save myself <em>that</em> hassle, at least; I’m not going to be running my own fediverse instances.</p>2023-08-25T17:23:49+00:00Nirbheek Chauhan: What is WebRTC? Why does it need ‘Signalling’?
http://blog.nirbheek.in/2023/07/webrtc-signalling.html
<p>If you’re new to WebRTC, you must’ve heard that it’s a way to do video calls in your browser without needing to install an app. It’s pretty great!</p><p>However, it uses a bunch of really arcane terminology because it builds upon older technologies such as RTP, RTCP, SDP, ICE, STUN, etc. To understand what WebRTC Signalling is, you must first understand these foundational technologies.</p><p>Readers who are well-versed in this subject might find some of the explanations annoyingly simplistic to read. They will also notice that I am omitting a lot of of detail, leading to potentially misleading statements.</p><p>I apologize in advance to these people. I am merely trying to avoid turning this post into a book. If you find a sub-heading too simplistic, please feel free to skip it. :-)</p>
<h2 id="RTP">RTP</h2><p><strong>Real-time Transport Protocol</strong> is a <a href="https://en.wikipedia.org/wiki/Internet_Engineering_Task_Force" target="_blank" rel="noopener">standardized</a> way of taking video or audio data (media) and chopping it up into “packets” (you can literally think of them as packets / parcels) that are sent over the internet using <strong>UDP</strong>. The purpose is to try and deliver them to the destination as quickly as possible.</p><p><strong>UDP</strong> (user datagram protocol) is a packet-based alternative to <strong>TCP</strong> (transmission control protocol), which is connection-based. So when you send something to a destination (<a href="https://en.wikipedia.org/wiki/IP_address" target="_blank" rel="noopener">IP address</a> + port number), it will be delivered if possible but you have no protocol-level mechanism for finding out if it was received, unlike, say, <a href="https://en.wikipedia.org/wiki/Transmission_Control_Protocol#Connection_establishment" target="_blank" rel="noopener">TCP ACKs</a>.</p><p>You can think of this as chucking parcels over a wall towards someone whom you can’t see or hear. A bunch of them will probably be lost, and you have no straightforward way to know how many were actually received.</p><p>UDP is used instead of TCP for a number of reasons, but the most important ones are:</p><ol>
<li>
<p>TCP is designed for perfect delivery of all data, so networks will often try too hard to do that and use ridiculous amounts of buffering (sometimes 30 seconds or more!), which leads to latencies that are too large for two people to be able to talk over a call.</p>
</li>
<li>
<p>UDP doesn’t have that problem, but the trade-off is that it gives no guarantees of delivery at all!</p>
<p>You’d be right to wonder why nothing new has been created to be a mid-way point between these two extremes. The reason is that new transport protocols don’t get any uptake because existing systems on the Internet (operating systems, routers, switches, etc) don’t (want to) support them. This is called <a href="https://en.wikipedia.org/wiki/Protocol_ossification" target="_blank" rel="noopener">Protocol ossification</a>, and it's a big problem for the Internet.</p>
<p>Due to this, new protocols are just built on top of UDP and try to add mechanisms to detect packet loss and such. One such mechanism is…</p>
</li>
</ol>
<h2 id="RTCP">RTCP</h2><p><strong>RTP Control Protocol</strong> refers to <a href="https://www.rfc-editor.org/rfc/rfc3550#section-6" target="_blank" rel="noopener">standardized messages</a> (closely related to RTP) that are sent by a media sender to all receivers, and also messages that are sent back by the receiver to the sender (feedback). As you might imagine, this message-passing system has been extended to do a lot of things, but the most important are:</p><ol>
<li>Receivers use this to send feedback to the sender about how many packets were actually received, what the latency was, etc.</li>
<li>Senders send information about the stream to receivers using this, for instance to synchronize audio and video streams (also known as <a href="https://en.wikipedia.org/wiki/Lip_sync" target="_blank" rel="noopener">lipsync</a>), to tell receivers that the stream has ended (a BYE message), etc.</li>
</ol><p>Similar to RTP, these messages are also sent over UDP. You might ask “what if these are lost too”? Good question!</p><p>RTCP packets are sent at regular intervals, so you’d know if you missed one, and network routers and switches will prioritize RTCP packets over other data, so you’re unlikely to lose too many in a row unless there was a complete loss of connectivity.</p><h2 id="Peer">Peer</h2><p>WebRTC is often called a “peer-to-peer” (P2P) protocol. You might’ve heard that phrase in a different context: P2P file transfer, such as Bittorrent.</p><p>The word “peer” contrasts with “server-client” architectures, in which “client” computers can only talk to (or via) “server” computers, not directly to each other.</p><p>We can contrast server-client architecture with peer-to-peer using a real-world example:</p><ul>
<li>If you send a letter to your friend using a postal service, that’s a server-client architecture.</li>
<li>If you leave the letter in your friend’s mailbox yourself, that’s peer-to-peer.</li>
</ul><p>But what if you don’t know what kind of messages the recipient can receive or understand? For that we have…</p>
<h2 id="SDP">SDP</h2><p>Stands for <strong>Session Description Protocol</strong> which is a <a href="https://tools.ietf.org/html/rfc4566" target="_blank" rel="noopener">standardized</a> message format to tell the other side the following:</p><ul>
<li>Whether you want to send and/or receive, audio and/or video</li>
<li>How many streams of audio and/or video you want to send / receive</li>
<li>What formats you can send or receive, for audio and/or video</li>
</ul><p>This is called an “offer”. Then the other peer uses the same message format to reply with the same information, which is called an “answer”.</p><p>This constitutes media “negotiation”, also called “SDP exchange”. One side sends an “offer” SDP, the other side replies with an “answer” SDP, and now both sides know what to do.</p><p>As you might expect, there’s a bunch of other technical details here, and you can know all about them at <a href="https://webrtchacks.com/sdp-anatomy/" target="_blank" rel="noopener">this excellent page that explains every little detail</a>. It even explains the format for ICE messages! Which is…</p>
<h2 id="ICE">ICE</h2><p><strong>Interactive Connectivity Establishment</strong> is a <a href="https://www.rfc-editor.org/rfc/rfc5245.html" target="_blank" rel="noopener">standardized</a> mechanism for peers to tell each other how to transmit and receive UDP packets. The simplest way to think of it is that it’s just a list of IP address and port pairs.</p><p>Once both sides have successfully sent each other (“exchanged”) ICE messages, both sides know how to send RTP and RTCP packets to each other.</p><p>Why do we need IP address + port pairs to know how to send and receive packets? For that you need to understand…</p>
<h2 id="How-The-Internet-Works">How The Internet Works</h2><p>If you’re connected to the internet, you always have an <a href="https://en.wikipedia.org/wiki/IP_address" target="_blank" rel="noopener">IP address</a>. That’s usually something like <code>192.168.1.150</code> – a private address that is specific to your local (home) network and has no meaning outside of that. Having someone’s private IP address is basically like having just their house number but no other parts of their address, like the street or the city. Useful if you're living in the same building, but not otherwise.</p>
<p>Most personal devices (computer or phone or whatever) with access to the Internet don’t actually have a <em>public</em> IP address. Picking up the analogy from earlier, a public IP address is the internet equivalent of a full address with a house number, street address, pin code, country.</p>
<p>When you want to connect to (visit) a website, your device actually talks to an ISP (internet service provider) router, which will then talk to the web server on your behalf and ask it for the data (website in this case) that you requested. This process of packet-hopping is called “<a href="https://en.wikipedia.org/wiki/Routing" target="_blank" rel="noopener">routing</a>” of network packets.</p>
<p>This ISP router with a public address is called a NAT (<a href="https://en.wikipedia.org/wiki/Network_address_translation" target="_blank" rel="noopener">Network Address Translator</a>). Like the name suggests, its job is to translate the addresses embedded in packets sent to it from public to private and vice-versa.</p>
<p>Let’s say you want to send a UDP packet to <code>www.google.com</code>. Your browser will resolve that domain to an IP address, say <code>142.250.205.228</code>. Next, it needs a port to send that packet to, and both sides have to pre-agree on that port. Let’s pick <code>16789</code> for now.</p><p>Your device will then allocate a port on your device from which to send this packet, let’s say <code>11111</code>. So the packet header looks a bit like this:</p>
<table>
<thead>
<tr>
<th>From</th>
<th>To</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>192.168.1.150:11111</code></td>
<td><code>142.250.205.228:16789</code></td>
</tr>
</tbody>
</table>
<p>Your ISP’s NAT will intercept this packet, and it will replace your private address and port in the <code>From</code> field in the packet header to its own public address, say <code>169.13.42.111</code>, and it will allocate a new sender port, say <code>22222</code>:</p>
<table>
<thead>
<tr>
<th>From</th>
<th>To</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>169.13.42.111:22222</code></td>
<td><code>142.250.205.228:16789</code></td>
</tr>
</tbody>
</table>
<p>Due to this, the web server never sees your private address, and all it can see is the public address of the NAT.</p><p>When the server wants to reply, it can send data back to the <code>From</code> address, and it can use the same port that it received the packet on:</p>
<table>
<thead>
<tr>
<th>From</th>
<th>To</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>142.250.205.228:16789</code></td>
<td><code>169.13.42.111:22222</code></td>
</tr>
</tbody>
</table>
<p>The NAT remembers that this port <code>22222</code> was recently used for <em>your</em> <code>From</code> address, and it will do the reverse of what it did before:</p>
<table>
<thead>
<tr>
<th>From</th>
<th>To</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>142.250.205.228:16789</code></td>
<td><code>192.168.1.150:11111</code></td>
</tr>
</tbody>
</table>
<p>And that’s how packets are send and received by your phone, computer, tablet, whatever when talking to a server.</p><p>Since at least one side needs to have a public IP address for this to work, how can your phone send messages to your friend’s phone? Both only have private addresses.</p>
<h3 id="Solution-1-Just-Use-A-Server-As-A-Relay">Solution 1: Just Use A Server As A Relay</h3><p>The simplest solution is to have a server in the middle that relays your messages. This is how all text messaging apps such as iMessage, WhatsApp, Instagram, Telegram, etc work.</p><p>You will need to buy a server with a public address, but that’s relatively cheap if you want to send small messages.</p><p>For sending RTP (video and audio) this is accomplished with a TURN (<a href="https://en.wikipedia.org/wiki/Traversal_Using_Relays_around_NAT" target="_blank" rel="noopener">Traversal Using Relays around NAT</a>) server.</p><p>Bandwidth can get expensive very quickly, so you don’t want to always use a TURN server. But this is a fool-proof method to transmit data, so it’s used a backup.</p>
<h3 id="Solution-2-STUN-The-NAT-Into-Doing-What-You-Want">Solution 2: STUN The NAT Into Doing What You Want</h3><p><strong>STUN</strong> stands for “<a href="https://en.wikipedia.org/wiki/STUN" target="_blank" rel="noopener">Simple Traversal of UDP through NATs</a>”, and it works due to a fun trick we can do with most NATs.</p><p>Previously we saw how the NAT will remember the mapping between a “port on its public address” and “your device’s private address and port”. With <a href="https://en.wikipedia.org/wiki/Network_address_translation#Full-cone_NAT" target="_blank" rel="noopener">many NATs</a>, this actually works for <em>any</em> packet sent on that public port by anyone.</p><p>This means if a public server can be used to create such mappings on the NATs of both peers, then the two can send messages to each other from NAT-to-NAT without a relay server!</p><p>Let’s dig into this, and let’s substitute hard-to-follow IP addresses with simple names: <code>AlicePhone</code>, <code>AliceNAT</code>, <code>BobPhone</code>, <code>BobNAT</code>, and finally <code>STUNServer:19302</code>.</p><p>First, <code>AlicePhone</code> follows this sequence:</p><ol>
<li>
<p><code>AlicePhone</code> sends a STUN packet intended for <code>STUNServer:19302</code> using UDP</p>
<table>
<thead>
<tr>
<th>From</th>
<th>To</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>AlicePhone:11111</code></td>
<td><code>STUNServer:19302</code></td>
</tr>
</tbody>
</table>
</li>
<li>
<p><code>AliceNAT</code> will intercept this and convert it to:</p>
<table>
<thead>
<tr>
<th>From</th>
<th>To</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>AliceNAT:22222 </code></td>
<td><code>STUNServer:19302</code></td>
</tr>
</tbody>
</table>
</li>
<li>
<p>When <code>STUNServer</code> receives this packet, it will know that if someone wants to send a packet to <code>AlicePhone:11111</code>, they could use <code>AliceNAT:22222</code> as the <code>To</code> address. This is an example of an ICE candidate.</p>
</li>
<li>
<p><code>STUNServer</code> will then send a packet back to <code>AlicePhone</code> with this information.</p>
</li>
</ol>
<p>Next, <code>BobPhone</code> does the same sequence and discovers that if someone wants to send a packet to <code>BobPhone:33333</code> they can use <code>BobNAT:44444</code> as the <code>To</code> address. This is <code>BobPhone</code>’s ICE candidate.</p><p>Now, <code>AlicePhone</code> and <code>BobPhone</code> must exchange these ICE candidates.</p><p>How do they do this? They have no idea how to talk to each other yet.</p><p>The answer is… they <a href="https://notes.centricular.com/5FL6lO_kSpa18N4NPyHBWQ?both#Solution-1-Just-Use-A-Server-As-A-Relay">Just Use A Server As A Relay</a>! The server used for this purpose is called a <strong>Signalling Server</strong>.</p><p>Note that these called “candidates” because this mechanism won’t work if one of the two NATs changes the public port also based on the public <code>To</code> address, not just the private <code>From</code> address. This is called a <a href="https://en.wikipedia.org/wiki/Network_address_translation#Symmetric_NAT" target="_blank" rel="noopener">Symmetric NAT</a>, and in these (and other) cases, you have to fallback to <a href="https://notes.centricular.com/5FL6lO_kSpa18N4NPyHBWQ?both#Solution-1-Just-Use-A-Server-As-A-Relay">TURN</a>.</p>
<h2 id="Signalling-Server">Signalling Server</h2><p><strong>Signalling</strong> is a technical term that simply means: “a way to pass small messages between peers”. In this case, it’s a way for peers to exchange SDP and ICE candidates.</p><p>Once these small messages have been exchanged, the peers know how to send data to each other over the internet without needing a relay.</p><p>Now open your mind: you could use <em>literally any</em> out of band-mechanism for this. You can use <a href="https://docs.aws.amazon.com/kinesisvideostreams/latest/dg/API_Operations_Amazon_Kinesis_Video_Signaling_Channels.html" target="_blank" rel="noopener">Amazon Kinesis Video Signalling Channels</a>. You can use a custom <a href="https://en.wikipedia.org/wiki/WebSocket" target="_blank" rel="noopener">websocket</a> server or a <a href="https://en.wikipedia.org/wiki/Protocol_Buffers" target="_blank" rel="noopener">ProtoBuf</a> server.</p><p>Heck, Alice and Bob can copy/paste these messages into iMessage on both ends. In theory, you can even use carrier pigeons — it’ll just take a very long time to exchange messages 😉</p>
<p>That’s it, this is what <strong>Signalling</strong> means in a WebRTC context, and why it’s necessary for a successful connection!</p><p>What a <strong>Signalling Server</strong> gives you on top of this is <strong>state management</strong>: checking whether a peer is allowed to send messages to another peer, whether a peer is allowed to join a call, can be invited to a call, which peers are in a call right now, etc.</p><p>Based on your use-case, this part can be really easy to implement or really difficult and heavy in corner-cases. Most people can get away with a really simple protocol, just by adding authorization to <a href="https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-examples/webrtc/signalling/Protocol.md#multi-party-calls-with-a-room" target="_blank" rel="noopener">this multi-party protocol</a> I wrote for the <a href="https://gitlab.freedesktop.org/gstreamer/gstreamer/-/tree/main/subprojects/gst-examples/webrtc/multiparty-sendrecv" target="_blank" rel="noopener">GStreamer WebRTC multiparty send-receive examples</a>. More complex setups require a more bespoke solution, where all peers aren’t equal.</p>2023-08-21T18:25:52+00:00NirbheekArun Raghavan: To Conference Organisers Everywhere…
https://arunraghavan.net/2023/08/to-conference-organisers-everywhere/
<p>(well, not exactly <em>everywhere</em> …)</p>
<p>This is not an easy post for me to write, being a bit of a criticism / “you can do better” note for organisers of conferences that cater to a global community.</p>
<p>It’s not easy because most of the conferences I attend are community driven, and I have helped organise <a href="https://en.wikipedia.org/wiki/FOSS.IN">community conferences</a> in the past. It is a thankless job, a labour of love, and you mostly do not get to enjoy the fruits of that labour as others do.</p>
<p>The problem is that these conferences end up inadvertently excluding members who live in, for lack of a better phrase, the Global South.</p>
<h1>Visas</h1>
<p>It always surprises me when I meet someone who doesn’t realise that I can’t just book tickets to go anywhere in the world. Not because this is information that everyone <em>should</em> be aware of, but because this is such a basic aspect of travel for someone like me. As a holder of an Indian passport, I need to apply for a visa to travel to … well <a href="https://en.wikipedia.org/wiki/Visa_requirements_for_Indian_citizens">most countries</a>.</p>
<p>The list of countries that require a visa are clearly defined by post-colonial geopolitics, this is just a fact of life and not something I can do very much about.</p>
<h2>Getting a Visa</h2>
<p>Applying for a visa is a cumbersome and intrusive process that I am now used to. The process varies from country to country, but it’s usually something like:</p>
<ul>
<li>Get an invitation letter from conference organisers</li>
<li>Book all the tickets and accommodation for the trip</li>
<li>Provide bank statements for 3-6 months, income tax returns for ~3 years (in India, those statements need attestation by the bank)</li>
<li>Maybe provide travel and employment history for the past few years (how many years depends on the country)</li>
<li>Get an appointment from the embassy of the country you’re traveling to (or their service provider)</li>
<li>Submit your passport and application</li>
<li>Maybe provide documentation that was not listed on the provider’s website</li>
<li>Wait for your passport with visa (if granted) to be mailed back to you</li>
</ul>
<p>The duration of visa (that is how long you can stay in the country) depends on the country.</p>
<p>In the EU, I am usually granted a visa for the exact dates of travel (so there is no flexibility to change plans). The UK allows you to pay more for a longer visa.</p>
<p>The US and Canada grant multi-year visas that allow one to visit for up to 6 months by default (in the US, whether you are permitted to enter and how long you may stay are determined by the person at the border).</p>
<h1>Timelines</h1>
<p>Now we get to the crux of the problem: this process can take anywhere from a few days (if you are <em>very</em> lucky) to a few months (if you are not).</p>
<p>Appointments are granted by the embassy or the third party that countries delegate application collection to, and these may or may not be easily available. Post-pandemic, I’ve seen that several embassies just aren’t accepting visitor visa appointments or have a multi-month wait.</p>
<p>If you do get an appointment, the processing time can vary again. Sometimes, it’s a matter of a few days, sometimes a few weeks. A lot of countries I have applied to recommend submitting your application at least 6 weeks in advance (this is from the date of your visa appointment which might be several weeks in the future).</p>
<h1>Conference Schedules</h1>
<p>If you’re organising a conference, there are a few important dates:</p>
<ul>
<li>When the conference dates are announced</li>
<li>When the call for participation goes out</li>
<li>When it ends</li>
<li>When speakers are notified</li>
<li>The conference itself</li>
</ul>
<p>These dates are based on a set of complex factors — venue availability and confirmation, literally writing and publishing all the content of the website, paper committee availability, etc.</p>
<p>But if you’re in my position, you need <em>at least</em> 2-3 months between the first and the last step. If your attendance is conditional on speaking at the conference (for example, if your company will only sponsor you if you’re speaking), then you need a minimum of 2-3 months between when speakers are notified and the conference starts.</p>
<p>From what I see, this is not something that is top-of-mind for conference organisers. That may happen for a host of perfectly understandable reasons, but it also has a cost to the community and individuals who might want to participate.</p>
<h1>Other Costs</h1>
<p>Applying for a visa costs money. This can be anything from a few hundred to over a 1000 US dollars.</p>
<p>It also costs you time — filling in the application, getting all the documentation in place, getting a physical visa photo (must be no older than 6 months), traveling to an appointment, waiting in line, etc. This can easily be a matter of a day if not more.</p>
<p>Finally, there is an <em>emotional</em> cost to all this — there is constant uncertainty during the process, and a visa rejection means <em>every</em> visa you apply for thereafter needs you to document that rejection and reason. And you may find out just days before your planned travel whether you get to travel or not.</p>
<h1>What Can One Do?</h1>
<p>All of this clearly sucks, but the problem of visas is too big and messy for any of us to have any real impact on, at least in the short term. But if you’re organising a conference, and you want a diverse audience, here are a few things you <em>can</em> do:</p>
<ul>
<li>Announce the dates of the conference as early as possible (allows participants to book travel, visa appointments, maybe club multiple conferences)</li>
<li>Provide invitation letters in a timely manner</li>
<li>Call for participation as early as possible</li>
<li>Notify speakers as soon as you can</li>
</ul>
<p>I know of conferences that <em>do</em> some if not all of these things — you know who you are and you have my gratitude for it.</p>
<p>If you made it this far, thank you for reading.</p>2023-08-17T21:42:00+00:00Bastien Nocera: New responsibilities
https://www.hadess.net/2023/08/new-responsibilities.html
<p><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">As part of the same process outlined in Matthias Clasen's <a href="https://lwn.net/Articles/933525/">"LibreOffice packages" email</a></span><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">, my management chain has made the decision to stop all
upstream and downstream work on desktop Bluetooth, multimedia
applications (namely totem, rhythmbox and sound-juicer) and
libfprint/fprintd. The rest of my upstream and downstream work will be
reassigned depending on Red Hat's own priorities (see below), as I am
transferred to another team that deals with one of a list of Red Hat’s
priority projects.</span></p><div class="ace-line" id="magicdomid3"></div><div class="ace-line" id="magicdomid4"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">I'm
very disappointed, because those particular projects were already
starved for resources: I spent less than 10% of my work time on them in
the past year, with other projects and responsibilities taking most of
my time.</span></div><div class="ace-line" id="magicdomid5"><br /></div><div class="ace-line" id="magicdomid6"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">This means that, in the medium-term at least, all those GNOME projects will go without a maintainer, reviewer, or triager:</span></div><div class="ace-line" id="magicdomid7"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">- gnome-bluetooth (including Settings panel and gnome-shell integration)</span></div><div class="ace-line" id="magicdomid8"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">- totem, totem-pl-parser, gom</span></div><div class="ace-line" id="magicdomid9"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">- libgnome-volume-control</span></div><div class="ace-line" id="magicdomid10"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">- libgudev</span></div><div class="ace-line" id="magicdomid11"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">- geocode-glib</span></div><div class="ace-line" id="magicdomid12"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">- gvfs AFC backend</span></div><div class="ace-line" id="magicdomid13"><br /></div><div class="ace-line" id="magicdomid14"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">Those freedesktop projects will be archived until further notice:</span></div><div class="ace-line" id="magicdomid15"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">- power-profiles-daemon</span></div><div class="ace-line" id="magicdomid16"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">- switcheroo-control</span></div><div class="ace-line" id="magicdomid17"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">- iio-sensor-proxy</span></div><div class="ace-line" id="magicdomid187"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">- low-memory-monitor</span></div><div class="ace-line" id="magicdomid189"><br /></div><div class="ace-line" id="magicdomid388"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">I will not be available for reviewing libfprint/fprintd, upower, grilo/grilo-plugins, gnome-desktop thumbnailer sandboxing patches, or any work related to XDG specifications.</span></div><div class="ace-line" id="magicdomid19"><br /></div><div class="ace-line" id="magicdomid371"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">Kernel
work, reviews and maintenance, including recent work on SteelSeries
headset and Logitech devices kernel drivers, USB revoke for Flatpak
Portal support, or core USB is suspended until further notice.</span></div><div class="ace-line" id="magicdomid21"><br /></div><div class="ace-line" id="magicdomid181"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">All my <a href="https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/WRHVGQBKKFU74CBO3CHIJC3Q5VEKH2AV/">Fedora </a></span><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z"><a href="https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/WRHVGQBKKFU74CBO3CHIJC3Q5VEKH2AV/">packages
were orphaned</a> about a month and a half ago, it's likely that there are
still some that are orphaned, if there are takers. RHEL packages were
unassigned about 3 weeks ago, they've been reassigned
since then, so I cannot point to the new maintainer(s).</span></div><div class="ace-line" id="magicdomid23"><br /></div><div class="ace-line" id="magicdomid24"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">If
you are a partner, or a customer, I would recommend that you get in
touch with your Red Hat contacts to figure out what the plan is going
forward for the projects you might be involved with.</span></div><div class="ace-line" id="magicdomid25"><br /></div><div class="ace-line" id="magicdomid26"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">If
you are a colleague that will take on all or part of the 90% of the
work that's not being stopped, or a community member that was relying on
my work to further advance your own projects, get in touch, I'll do my
best to accommodate your queries, time permitting.</span></div><div class="ace-line" id="magicdomid26"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z"><br /></span></div><div class="ace-line" id="magicdomid26"><span class="author-a-bpz78z48jz69zz85zrfz84zz76zz84zbz89zz65z">I'll try to make sure to update this post, or create a new one if and when any of the above changes.<br /></span></div>2023-08-14T10:31:50+00:00Bastien NoceraJean-François Fortin Tam: Please help test (and fix) GTG’s GTK 4 port
https://fortintam.com/blog/call-for-testing-gtg-gtk4-branch/
<p>As you know, even with a “simple” language like Python, porting a desktop application to a new version of <a rel="noreferrer noopener" href="https://gtk.org" target="_blank">GTK</a> can be a pretty significant amount of work; doubly so when it is accompanied by major refactoring of the app itself at the same time.</p>
<span id="more-6456"></span>
<p>In <a rel="noreferrer noopener" href="https://wiki.gnome.org/Apps/GTG/" target="_blank">Getting Things GNOME</a>‘s case, this has been <a href="https://github.com/getting-things-gnome/gtg/issues/737">over a year in the making</a>, for various technical and personal/life reasons.</p>
<div class="wp-block-image">
<img src="https://fortintam.com/blog/wp-content/uploads/branch-needs-testing-it-must-be-destroyed.jpg" alt="" class="wp-image-6457" width="250" height="288" /></div>
<p>We need your help to <strong>determine when/if it would be “safe” to merge our long-running core rewrite & GTK 4 port branch</strong> to <a rel="noreferrer noopener" href="https://wiki.gnome.org/Apps/GTG/" target="_blank">GTG</a>‘s main branch and repository. We <em>think</em> it mostly works, but we can’t be sure until you try it out (preferrably while using a copy of your real world data file).</p>
<p>If you have an up-to-date Linux distro with recent versions of GTK (ex: 4.10.4+), and would like to help test & fix the upcoming GTK 4 version of GTG, check out <a rel="noreferrer noopener" href="https://github.com/getting-things-gnome/gtg/pull/894" target="_blank">this branch / PR</a> and give it a try in the coming days. You can then help by:</p>
<ul>
<li><strong>Finding and reporting remaining showstopper issues</strong>: problems specific to that branch should be reported <a rel="noreferrer noopener" href="https://github.com/diegogangl/gtg/issues" target="_blank">here</a> (in Diego’s repository) for now.
<ul>
<li>We’re looking for problems that would be considered unacceptable/non-functional <em>for the sake of merging to the main repository.</em> Especially, anything related to data or dangerously broken behavior, if any.</li>
<li>Broken plugins are acceptable at this point.</li>
<li>If you can’t find any unreported issues after extensive testing… let us know too, it’s nice to know our code is perfect 😉</li>
</ul>
</li>
<li><strong>Sending <a rel="noreferrer noopener" href="https://github.com/diegogangl/gtg/pulls" target="_blank">merge requests</a></strong><a rel="noreferrer noopener" href="https://github.com/diegogangl/gtg/pulls" target="_blank"> to Diego’s repository</a> (check the issues and existing PRs, if any, to avoid duplicate work) would help a ton, of course, because we’ve been stretched very thin lately.</li>
</ul>
<p>Being able to confidently merge this branch would make it much easier for others in the GTG community to have a stable base to work on, and provide the remaining fixes to bring version 0.7 to the finish line.</p>
<hr class="wp-block-separator has-alpha-channel-opacity" />
<p>P.s.: remember that we also have a Matrix discussion <a rel="noreferrer noopener" href="https://matrix.to/#/#gtg:gnome.org" target="_blank">channel</a> (see the GTG wiki page for details) if you’d like to coordinate some aspects in a more informal and interactive manner before posting to GitHub.</p>2023-08-09T18:56:31+00:00Víctor Jáquez: DMABuf modifier negotiation in GStreamer
https://blogs.igalia.com/vjaquez/2023/08/08/dmabuf-modifier-negotiation-in-gstreamer/
<p>It took almost a year of design and implementation but finally the DMABuf modifier negotiation in GStreamer is merged. Big kudos to all the people involved but mostly to <a href="https://gitlab.freedesktop.org/He_Junyan">He Junyan</a>, who did the vast majority of the code.</p>
<p><strong>What’s a DMAbuf modifier?</strong></p>
<p>DMABuf are the Linux kernel mechanism to share buffers among different drivers or subsystems. A particular case of DMABuf are the <em>DRM PRIME buffers</em> which are buffers shared by the <a href="https://en.wikipedia.org/wiki/Direct_Rendering_Manager">Display Rendering Manager (DRM) subsystem</a>. They allowed sharing video frames between devices with <em>zero copy</em>.</p>
<p>When we initially added support for DMABuf in GStreamer, we assumed that only color format and size mattered, just as old video frames stored in system memory. But we were wrong. Beside color format and size, also the memory layout has to be considered when sharing DMABufs. By not considering it, the produced output had horrible tiled artifacts in screen. This memory layout is known as <em>modifier</em>, and it’s <a href="https://github.com/torvalds/linux/blob/master/include/uapi/drm/drm_fourcc.h">uniquely described by an uint64 number</a>.</p>
<p><strong>How was designed and implemented?</strong></p>
<p>First, we wrote a <a href="https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-docs/markdown/additional/design/dmabuf.md">design document for caps negotiation with dmabuf<br />
modifiers</a>, where we added a new color format (<code>DMA_DRM</code>) and a new caps field (<code>drm-format</code>). This new caps field holds a string, or a list of strings, composed by the tuple <code>DRM_color_format : DRM_modifier</code>.</p>
<p>Second, we extended the <a href="https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-base/gst-libs/gst/video/video-info-dma.h">video info object</a> to support DMABuf with helper functions that parse and construct the <code>drm-format</code> field.</p>
<p>Third, we added the dmabuf caps negotiation in <code>glupload</code>. This part was the most difficult one, since the capability of importing DMABufs to OpenGL (which <em>is only available in EGL/GLES</em>) is run-time defined, by querying the hardware. Also, there are two code paths to import frames: direct or RGB-emulated. Direct would be the most efficient, but it depends on the presence of GLES2 API in the driver; while RGB-emulated is imported as a set of RGB images where each component is an image. At the end more than a thousand lines of code were added to the <code>glupload</code> element, beside the code added to EGL context object.</p>
<p>Fourth, and unexpectedly, <code>waylandsink</code> also got DMABuf caps negotiation<br />
support.</p>
<p>And lastly, decoders in `va** plugin merged theirs DMABuf caps negotiation support.</p>
<p><strong>How I can test it?</strong></p>
<p>You need, of course, to user the current main branch of GStreamer, since it’s just fresh and there’s no release yet. Then you need a box with VA support. And if you inspect, for example, <code>vah264dec</code>, you might see this output if your box is Intel (but also AMD through Mesa is supported though the negotiated memory is linear so far):</p>
<pre><code class="text">Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-h264
profile: { (string)main, (string)baseline, (string)high, (string)progressive-high, (string)constrained-high, (string)constrained-baseline }
width: [ 1, 4096 ]
height: [ 1, 4096 ]
alignment: au
stream-format: { (string)avc, (string)avc3, (string)byte-stream }
SRC template: 'src'
Availability: Always
Capabilities:
video/x-raw(memory:VAMemory)
width: [ 1, 4096 ]
height: [ 1, 4096 ]
format: NV12
video/x-raw(memory:DMABuf)
width: [ 1, 4096 ]
height: [ 1, 4096 ]
format: DMA_DRM
drm-format: NV12:0x0100000000000002
video/x-raw
width: [ 1, 4096 ]
height: [ 1, 4096 ]
format: NV12
</code></pre>
<p>What it’s saying, for <code>memory:DMABuf</code> caps feature, the <code>drm-format</code> to negotiate is <code>NV12:0x0100000000000002</code>.</p>
<p>Now some tests:</p>
<p>NOTE: These commands assume that <code>va</code> decoders are primary ranked (see <a href="https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/2312">merge request 2312</a>), and that you’re in a Wayland session.</p>
<pre><code>$ gst-play-1.0 --flags=0x47 video.file --videosink=waylandsink
</code></pre>
<pre><code>$ GST_GL_API=gles2 gst-play-1.0 --flags=0x47 video.file --videosink=glimagesink
</code></pre>
<pre><code>$ gst-play-1.0 --flags=0x47 video.file --videosink='glupload ! gtkglsink'
</code></pre>
<p>Right now it’s required to add <code>--flags=0x47</code> to <code>playbin</code> because it adds video filters that still don’t negotiate the new DMABuf caps.</p>
<p><code>GST_GL_API=gles2</code> instructs GStreamer OpenGL to use GLES2 API, which allows direct importation of YUV images.</p>
<p>Thanks to all the people involved in this effort!</p>
<p>As usual, if you would like to learn more about DMABuf, VA-API, GStreamer or any other open multimedia framework, <a href="https://www.igalia.com/">contact us</a>!</p>2023-08-08T13:00:20+00:00Andy Wingo: a negative result
https://wingolog.org/archives/2023/08/08/a-negative-result
<div><p><i>Update: I am delighted to have been wrong! See the end.</i></p><p>Briefly, an interesting negative result: consider benchmarks <tt>b1</tt>, <tt>b2</tt>,
<tt>b3</tt> and so on, with associated <tt>.c</tt> and <tt>.h</tt> files. Consider libraries
<tt>p</tt> and <tt>q</tt>, with their <tt>.c</tt> and <tt>.h</tt> files. You want to run each
benchmark against each library.</p><p>P and Q implement the same API, but they have different ABI: you need to
separately compile each benchmark for each library. You also need to
separate compile each library for each benchmark, because <tt>p.c</tt> also
uses an abstract API implemented by <tt>b1.h</tt>, <tt>b2.h</tt>, and so on.</p><p>The problem: can you implement a short GNU <tt>Makefile</tt> that produces
executables <tt>b1.p</tt>, <tt>b1.q</tt>, <tt>b2.p</tt>, <tt>b2.q</tt>, and so on?</p><p>The answer would appear to be “no”.</p><p>You might think that with <tt>call</tt> and all the other functions available
to you, that surely this could be done, and indeed it’s easy to take the
cross product of two lists. But what we need are new rules, not just
new text or variables, and you can’t programmatically create rules. So
we have to look at rules to see what facilities are available.</p><p>Consider the rules for one target:</p><pre>b1.p.lib.o: p.c
$(CC) -o $@ -include b1.h $<
b1.p.bench.o: b1.c
$(CC) -o $@ -include p.h $<
b1.p: b1.p.lib.o b1.p.bench.o
$(CC) -o $@ $<
</pre><p>With <a href="https://www.gnu.org/software/make/manual/html_node/Pattern-Rules.html">pattern
rules</a>,
you can easily modify these rules to parameterize <i>either</i> over
benchmark or over library, but not both. What you want is something
like:</p><pre>*.%.lib.o: %.c
$(CC) -o $@ -include $(call extract_bench,$@) $<
%.*.bench.o: %.c
$(CC) -o $@ -include $(call extract_lib,$@) $<
%: %.lib.o %.bench.o
$(CC) -o $@ $<
</pre><p>But that doesn’t work: you can’t have a wildcard (<tt>*</tt>) in the pattern
rule. (Really you would like to be able to match multiple patterns, but
the above is the closest thing I can think of to what <tt>make</tt> has.)</p><p><a href="https://www.gnu.org/software/make/manual/html_node/Static-Pattern.html">Static pattern
rules</a>
don’t help: they are like pattern rules, but more precise as they apply
only to a specific set of targets.</p><p>You might think that you could use <tt>$*</tt> or other special variables on
the right-hand side of a pattern rule, but that’s not the case.</p><p>You might think that <a href="https://www.gnu.org/software/make/manual/html_node/Secondary-Expansion.html">secondary
expansion</a>
might help you, but then you open the door to an annoying set of
problems: sure, you can mix variable uses that are intended to be
expanded once with those to be expanded twice, but the former set better
be idempotent upon second expansion, or things will go weird!</p><p>Perhaps the best chance for a <tt>make</tt>-only solution would be to recurse
on generated makefiles, but that seems to be quite beyond the pale.</p><p>To be concrete, I run into this case when benchmarking
<a href="https://github.com/wingo/whippet-gc">Whippet</a>: there are some number of
benchmarks, and some number of collector configurations. Benchmark code
will inline code from collectors, from their header files; and
collectors will inline code from benchmarks, to implement the
trace-all-the-edges functionality.</p><p>So, with Whippet I am left with the strange conclusion that the only
reasonable thing is to generate the
<a href="https://github.com/wingo/whippet-gc/blob/main/Makefile"><tt>Makefile</tt></a>
with a little custom generator, or at least generate the part of it to
do this benchmark-library cross product. It’s hard to be certain about
negative results with <tt>make</tt>; perhaps there is a trick. If so, do let
me know!</p><h3>epilogue</h3><p>Thanks to <a href="https://mastodon.gamedev.place/@amonakov/110853258601803460">a kind note from Alexander Monakov</a>, I am very happy to be proven wrong.</p><p>See, I thought that <a href="https://www.gnu.org/software/make/manual/html_node/Functions.html"><tt>make</tt> functions</a> were only really good in variables and rules and the like, and couldn’t be usefully invoked “at the top level”, to define new rules. But that’s not the case! <a href="https://www.gnu.org/software/make/manual/html_node/Eval-Function.html"><tt>eval</tt></a> in particular can define new rules.</p><p>So a solution with <tt>eval</tt> might look something like this:</p><pre>BENCHMARKS=b1 b2 b3
LIBS=p q
define template
$(1).$(2).lib.o: $(2).c
$$(CC) -o $$@ -include $(1).h $$<
$(1).$(2).bench.o: $(1).c
$$(CC) -o $$@ -include $(2).h $$<
$(1).$(2): $(1).$(2).lib.o $(1).$(2).bench.o
$$(CC) -o $$@ $$<
end
$(foreach BENCHMARK,$(BENCHMARKS),\
$(foreach LIB,$(LIBS),\
$(eval $(call template,$(BENCHMARK),$(LIB)))))
</pre><p>Thank you, Alexander!</p></div>2023-08-08T09:03:25+00:00Andy WingoGStreamer: GStreamer Conference 2023: Registration now open
https://gstreamer.freedesktop.org/news/#2023-07-31T17:00:00Z
<p><b><i>About the GStreamer Conference</i></b></p><p>
The GStreamer Conference 2023 will take place on 25-26 September 2023 in A Coruña, Spain.
</p><p>
It is a conference for developers, community members, decision-makers,
industry partners, researchers, students and anyone else interested
in the GStreamer multimedia framework or Open Source and cross-platform
multimedia.
</p><p>
There will also be a hackfest just after the conference.
</p><p><b><i>Registration now open</i></b></p><p>
You can now register for the <a href="https://gstreamer.freedesktop.org/conference/2023/">GStreamer Conference 2023</a>
on the <a href="https://gstreamer.freedesktop.org/conference/2023/">conference website</a>.
</p><p>
A reduced early-bird registration fee for professionals is available
until August 20th, so hurry!
</p><p>
Speakers will receive free registration, so if you intend to submit
a talk you may want to hold off with registration for now.
</p><p>
We hope to see you there!
</p><p><b><i>Call for Papers is open</i></b></p><p>
The call for papers is now open for talk proposals and lightning talks.
Please submit your talk now!
</p><p>
Talks can be on almost anything multimedia related, ranging from talks about
applications to challenges in the lower levels in the stack or hardware.
</p><p>
We want to hear from you, what you are working on, what you are using
GStreamer for, what difficulties you have encountered and how you solved them,
what your plans are, what you like, what you dislike, how to improve things!
</p><p>
Please submit all proposals through the <b><a href="https://indico.freedesktop.org/event/5/">GStreamer Conference 2023 Paper Submission Portal</a></b>.
</p><p><b><i>Sponsors</i></b></p><p>
Many thanks to our Platinum sponsors <a href="https://igalia.com">Igalia</a>,
<a href="https://collabora.com">Collabora</a> and
<a href="https://fluendo.com">Fluendo</a>, our Gold sponsors
<a href="https://axis.com">Axis Communications</a> and
<a href="https://centricular.com">Centricular</a>,
and our Silver sponsor <a href="https://zeiss.com">Zeiss</a>
for sponsoring the conference this year!
</p>2023-07-31T17:00:00+00:00GStreamer: GStreamer 1.20.7 old-stable bug fix release
https://gstreamer.freedesktop.org/news/#2023-07-26T18:00:00Z
<p>
The GStreamer team is pleased to announce another bug fix release in the
now old-stable 1.20 release series of your favourite cross-platform
multimedia framework!
</p><p>
This release only contains bug fixes. It should be safe to update from 1.20.x.
</p><p>
<b>Highlighted bugfixes:</b>
<ul>
<li><a href="https://gstreamer.freedesktop.org/security/">Security fixes</a> for flacparse, dvdspu, and subparse, and the RealMedia demuxer</li>
<li>h265parse: Fix framerate handling</li>
<li>filesink: Fix buffered mode writing of buffer lists and buffers with multiple memories</li>
<li>asfmux, rtpbin_buffer_list test: fix possible unaligned write/read on 32-bit ARM</li>
<li>ptp clock: Work around bug in ptpd in default configuration</li>
<li>qtdemux: fix reverse playback regression with edit lists</li>
<li>rtspsrc: various control path handling server compatibility improvements</li>
<li>avviddec: fix potential deadlock on seeking with FFmpeg 6.x</li>
<li>cerbero: Fix pango crash on 32bit Windows; move libass into non-GPL codecs</li>
<li>Miscellaneous bug fixes, memory leak fixes, and other stability and reliability improvements</li>
</ul>
</p><p>
See the <a href="https://gstreamer.freedesktop.org/releases/1.20/#1.20.7">GStreamer 1.20.7 release notes</a>
for more details.
</p><p>
Binaries for Android, iOS, macOS and Windows will be available shortly.
</p><p>
Release tarballs can be downloaded directly here:
<ul>
<a href="https://gstreamer.freedesktop.org/src/gstreamer/gstreamer-1.20.7.tar.xz">gstreamer</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-base/gst-plugins-base-1.20.7.tar.xz">gst-plugins-base</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-good/gst-plugins-good-1.20.7.tar.xz">gst-plugins-good</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-ugly/gst-plugins-ugly-1.20.7.tar.xz">gst-plugins-ugly</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-bad/gst-plugins-bad-1.20.7.tar.xz">gst-plugins-bad</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-libav/gst-libav-1.20.7.tar.xz">gst-libav</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-rtsp-server/gst-rtsp-server-1.20.7.tar.xz">gst-rtsp-server</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-python/gst-python-1.20.7.tar.xz">gst-python</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-editing-services/gst-editing-services-1.20.7.tar.xz">gst-editing-services</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-devtools/gst-devtools-1.20.7.tar.xz">gst-devtools</a>,
<a href="https://gstreamer.freedesktop.org/src/gstreamer-vaapi/gstreamer-vaapi-1.20.7.tar.xz">gstreamer-vaapi</a>,
<a href="https://gstreamer.freedesktop.org/src/gstreamer-sharp/gstreamer-sharp-1.20.7.tar.xz">gstreamer-sharp</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-omx/gst-omx-1.20.7.tar.xz">gst-omx</a>, or
<a href="https://gstreamer.freedesktop.org/src/gstreamer-docs/gstreamer-docs-1.20.7.tar.xz">gstreamer-docs</a>.
</ul>
</p><p>
The GStreamer 1.20 series has now been superseded by the new stable
<a href="https://gstreamer.freedesktop.org/releases/1.22/">GStreamer 1.22 series</a>
and we recommend you upgrade at your earliest convenience.
</p>2023-07-26T18:00:00+00:00Andy Wingo: on « the ministry for the future »
https://wingolog.org/archives/2023/07/25/on-the-ministry-for-the-future
<div><p>Good evening, comrades. This evening, words on reading.</p><h3>obtaining readables </h3><p>Since the pandemic, or maybe a bit before, I picked up a new habit: if I hear of a book which is liked by someone I find interesting, I buy a copy.</p><p>Getting to this point required some mental changes. When I was younger, buying lots of books wasn’t really thinkable; as a student I was well-served by libraries in my mother tongue, and never picked up the habit of allocating my meager disposable income to books. I did like bookstores, especially used bookstores, but since leaving university I emigrated to countries that spoke different languages; the crate-digging avenue was less accessible to me.</p><p>I also had to make peace with capitalism, or as the comic says it, “participating in society”. We all know that Amazon imposes a dialectic on the written word: it makes it astonishingly easy and cheap to obtain anything you want, but also makes it hard to earn a living from writing, producing, and distributing books; consuming words via Amazon undermines the production of the words you want to read.</p><p>So I didn’t. Participate, I mean; didn’t order off Amazon. Didn’t read much either. At this point in my life, though, I see it everywhere: purity is the opposite of praxis. I still try to avoid Amazon, quixotically ordering mostly off <a href="https://www.abebooks.com/">AbeBooks</a>, a now-fully-owned subsidiary of Amazon which fulfills via independent bookstores, but not avoiding Amazon entirely.</p><p>If you have access to and a habit of borrowing via a library in your language, you are almost certainly a better reader than I. But for the rest of us, and if you have the cash, maybe you too can grant yourself permission: when you hear of an interesting book, just buy it.</p><h3>the ministry for the future</h3><p>It was surely in this way that I heard of Kim Stanley Robinson’s « The Ministry for the Future ». My copy sat around for a year or so before I got around to it. We all hear the doom and gloom about the climate, and it’s all true; at least, everything about the present. But so much of it comes to us in a way that is fundamentally disempowering, that there is nothing to be done. I find this to be painful, viscerally; I prefer no news rather than bad news that I can’t act on, or at least incorporate into a longer narrative of action.</p><p>This may be a personal failing, or a defect. I have juvenile cultural tastes: I don’t like bad things to happen to good people. Jane Eyre was
excruciating: to reach an ultimately heavenly resolution, as a reader I had to suffer with every step. Probably this is recovering-Catholic scar damage.</p><p>In any case, climate fiction tends to be more hell than heaven, and I hate it all the more for another reason, that we are here in this moment and we <i>can</i> do things: why tell us that nothing matters? Why would you write a book like that? What part of praxis is that?</p><p>Anyway it must have been the premise that made Robinson’s book pass my pre-filters: that the COP series of meetings established a new UN agency that was responsible for the interests of future generations, notably as regards climate. The book charts the story of this administration from its beginnings over a period of some 40 years or so. It’s a wonderfully human, gripping story, <i>and we win</i>.</p><p>We don’t win inevitably; the book isn’t a prediction, but more of a prophecy in a sense: an exercise in narrative genesis, a precursor to history, a seed: one not guaranteed to germinate, but which might.</p><p>At this point I find the idea of actually participating in a narrative to be almost foreign, or rather alienated. When I was younger, the talk was “after the revolution” this, or “in the future” that, but with enough time and deracination, I lost my vision of how to get there from here. I don’t have it back, not yet anyway, but I think it’s coming. Story precedes history; storytellers are the ones that can breathe life into the airships of the mind. Robinson has both the airships and the gift to speak them into existence. Check it out, comrades, it is required reading!</p></div>2023-07-25T19:49:27+00:00Andy WingoStéphane Cerveau: Discover GStreamer full static mode
https://dabrain34.github.io/jekyll/update/2023/07/25/gstfullstatic.html
<h1 id="how-to-embed-statically-your-own-tailored-version-of-gstreamer-in-your-application">How to embed statically your own tailored version of GStreamer in your application</h1>
<p>Since the <a href="https://dabrain34.github.io/2021/10/04/shrinking_gstreamer.html">gstreamer-full effort</a>, it was possible to create a shared library which will embed the GStreamer framework in addition to its set of plugins.</p>
<p>Within this effort, it was also possible to register the selected plugins/features automatically by calling the <code class="language-plaintext highlighter-rouge">gst_init</code> method in your application linking with <code class="language-plaintext highlighter-rouge">gstreamer-full</code>.</p>
<p>This method was offering a <code class="language-plaintext highlighter-rouge">gstreamer-full</code> package with library, headers and pc files but it was not possible to embed GStreamer statically in your application and use it transparently.</p>
<h1 id="gstvkvideoparser-a-standalone-solution">GstVkVideoParser: a standalone solution</h1>
<p>In the journey to bring an open source solution for a video parser to the <a href="https://github.com/KhronosGroup/VK-GL-CTS">Vulkan Conformance Test Suite</a>, we chose first to use GStreamer as it was bringing all the parsing facilities
necessary to support the needed codecs such as H26x or VPx. This solution was supposed to be also cross platform and dragging as less as possible system dependencies.
Seen that GStreamer is usually dragging its own dependencies such as glib or orc and as we wanted to have a standalone <a href="https://github.com/Igalia/GstVkVideoParser">GstVkVideoParser</a> library supported on Windows, a little bit
of work and love was necessary to add this to GStreamer.</p>
<p>Unfortunately this solution has not be retained by the Vulkan Video TSG, not because it was not working but another parser has been made available and easy to integrate to the CTS at source level avoiding binary linkage, see Vulkan Video <a href="https://github.com/KhronosGroup/VK-GL-CTS/commit/e5db10e7ae436dbd9b46dad9518f3254dd6eeea2">change</a>.</p>
<h1 id="gstreamer-as-a-full-static-library">GStreamer as a full static library</h1>
<p>With the <code class="language-plaintext highlighter-rouge">gstreamer-full</code> work, everything was almost ready to be used except to have <code class="language-plaintext highlighter-rouge">gstreamer-full</code> as a real static library and be able to link with it in any application.</p>
<p>Here is the <a href="https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4128">MR</a> merged and the challenges taken up:</p>
<h2 id="adding-gst-full-target-typestatic">Adding gst-full-target-type=static</h2>
<p>To generate the <code class="language-plaintext highlighter-rouge">gstreamer-full</code> dependency which will be statically linked into the application, we decided to introduce a new gst meson options, <code class="language-plaintext highlighter-rouge">gst-full-target-type</code>.</p>
<p>By default the <code class="language-plaintext highlighter-rouge">gstreamer-full</code> will be built as a <em>shared library</em> as before.</p>
<p>By passing <code class="language-plaintext highlighter-rouge">gst-full-target-type=static</code>, only static object will be generated and a package config file will be generated for <code class="language-plaintext highlighter-rouge">gstreamer-full</code> allowing the application to avoid to know what static library it needs to add the link line.
The GStreamer build system will take care of enabling/disabling the features/libraries you (do/don’t) need.</p>
<h2 id="initialize-the-pluginsfeatures-automatically">Initialize the plugins/features automatically</h2>
<p>To avoid multiple call necessary to initialize GStreamer, it was also necessary to call the <code class="language-plaintext highlighter-rouge">gst_init_static_plugins</code> along with <code class="language-plaintext highlighter-rouge">gst_init</code> only in full-static mode but it was leading to a build issue.</p>
<p>Indeed most of tools/examples/tests are linking with <code class="language-plaintext highlighter-rouge">libgstreamer-1.0</code> which owns <code class="language-plaintext highlighter-rouge">gst_init ()</code> but to faciliate the plugins registration, it was necessary to move all the tools build after the <code class="language-plaintext highlighter-rouge">gstreamer-full</code> stage. A first <a href="https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1581">MR</a> has been performed to let gstreamer tools be built against gstreamer-full
but additional work was necessary for some core tools or helpers such as <code class="language-plaintext highlighter-rouge">gst-transcoder</code> or <code class="language-plaintext highlighter-rouge">gst-plugin-scanner</code> to avoid a linking issue.</p>
<h2 id="disable-tests-and-examples">Disable tests and examples</h2>
<p>In a future work all the tools/examples/tests should support the <code class="language-plaintext highlighter-rouge">full-static</code> mode but as GStreamer aims to be a shared object framework, we decided to leave this work for later and disable all the examples/tests in full static mode as most of the application using a tailor build won’t need the examples and tests.</p>
<h2 id="windows-support">Windows support</h2>
<p>One of the goal of this work was to provide a Windows library to the Vulkan CTS free of dependency, which has been achieved but some additional work might be necessary to support
all of the use case, the GStreamer framework offer, especially on supporting library-dependent plugins.</p>
<h2 id="give-me-an-example-">Give me an example …</h2>
<p>In the <a href="https://github.com/Igalia/GstVkVideoParser">GstVkVideoParser project</a>, various jobs are building Linux and Windows versions generating a library without any GStreamer/glib dependencie, everything is embedded inside the library, as you can see in this <a href="https://github.com/Igalia/GstVkVideoParser/actions">GitHub’ Actions</a>.</p>
<p>In this project, GStreamer is used as a meson subproject/wrap which allows to build GStreamer along of GstVkVideoParser. This can be possible easily by adding the following file to your meson project</p>
<p><em>subprojects/gstreamer-1.0.wrap</em></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[wrap-git]
directory=gstreamer-1.0
url=https://gitlab.freedesktop.org/gstreamer/gstreamer.git
revision=main
[provide]
dependency_names = gstreamer-1.0, gstreamer-base-1.0, gstreamer-video-1.0, gstreamer-audio-1.0
</code></pre></div></div>
<p>and then add the following lines to your <code class="language-plaintext highlighter-rouge">meson.build</code> to depend on <code class="language-plaintext highlighter-rouge">gstreamer-full</code></p>
<p><em>meson.build</em></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gstreamer_full_dep = dependency('gstreamer-full-1.0', fallback: ['gstreamer-1.0'], required :true)
</code></pre></div></div>
<p>In order to build a project, library or application which is using a tailored version of GStreamer you can follow this <a href="https://github.com/Igalia/GstVkVideoParser/blob/main/configure_gst_full.py">configure example</a>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ meson buildfull-static --default-library=static --force-fallback-for=gstreamer-1.0,glib,libffi,pcre2 -Dauto_features=disabled -Dglib:tests=false -Djson-glib:tests=false -Dpcre2:test=false -Dvkparser_standalone=enabled -Dgstreamer-1.0:libav=disabled -Dgstreamer-1.0:ugly=disabled -Dgstreamer-1.0:ges=disabled -Dgstreamer-1.0:devtools=disabled -Dgstreamer-1.0:default_library=static -Dgstreamer-1.0:rtsp_server=disabled -Dgstreamer-1.0:gst-full-target-type=static_library -Dgstreamer-1.0:gst-full-libraries=gstreamer-video-1.0, gstreamer-audio-1.0, gstreamer-app-1.0, gstreamer-codecparsers-1.0 -Dgst-plugins-base:playback=enabled -Dgst-plugins-base:app=enabled -Dgst-plugins-bad:videoparsers=enabled -Dgst-plugins-base:typefind=enabled
</code></pre></div></div>
<p>In this case we are disabling everything in GStreamer by using <code class="language-plaintext highlighter-rouge">-Dauto_features=disabled</code> and some enabled features such as <code class="language-plaintext highlighter-rouge">ges</code>, <code class="language-plaintext highlighter-rouge">libav</code>, etc. and enable only what we need as plugins, <code class="language-plaintext highlighter-rouge">playback</code>, <code class="language-plaintext highlighter-rouge">app</code>, <code class="language-plaintext highlighter-rouge">videoparsers</code> and <code class="language-plaintext highlighter-rouge">typefind</code>.</p>
<p>And finally we are enabling the static build with <code class="language-plaintext highlighter-rouge">--default-library=static</code> and <code class="language-plaintext highlighter-rouge">-Dgstreamer-1.0:gst-full-target-type=static_library</code>.</p>
<h2 id="next-">Next …</h2>
<p>As you can see, it’s quite easy now to build an application and depends on <code class="language-plaintext highlighter-rouge">gstreamer-full</code> static build, but there is still some issues to address such as the plugins dependencies which might be not static and some other platform specific issue such as the <code class="language-plaintext highlighter-rouge">gstreamer-full</code> symbols export on Windows.</p>
<p>You can follow some open issues such as:</p>
<ul>
<li>
<p><a href="https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1629">Static build is failing on Windows platform</a></p>
</li>
<li>
<p><a href="https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/2104">gstfull: Build only static library on Windows</a></p>
</li>
</ul>
<p>As usual, if you would like to learn more about Vulkan Video, GStreamer or any other open multimedia framework, please contact <a href="https://www.igalia.com/">us</a>!</p>2023-07-25T10:10:45+00:00GStreamer: GStreamer 1.22.5 stable bug fix release
https://gstreamer.freedesktop.org/news/#2023-07-20T16:00:00Z
<p>
The GStreamer team is pleased to announce another bug fix release
in the stable 1.22 release series of your favourite cross-platform
multimedia framework!
</p><p>
This release only contains bugfixes and security fixes and it should be safe
to update from 1.22.x.
</p><p>
<b>Highlighted bugfixes:</b>
<ul>
<li><a href="https://gstreamer.freedesktop.org/security/">Security fixes</a> for the RealMedia demuxer</li>
<li>vaapi decoders, postproc: Disable DMAbuf from caps negotiation to fix garbled video in some cases</li>
<li>decodebin3, playbin3, parsebin fixes, especially for stream reconfiguration</li>
<li>hlsdemux2: fix early seeking; don't pass referer when updating playlists; webvtt fixes</li>
<li>gtk: Fix critical caused by pointer movement when stream is getting ready</li>
<li>qt6: Set sampler filtering method, fixes bad quality with qml6glsink and gstqt6d3d11</li>
<li>v4l2src: handle resolution change when buffers are copied</li>
<li>videoflip: update orientation tag in auto mode</li>
<li>video timecode: Add support for framerates lower than 1fps and accept 119.88 (120/1.001) fps</li>
<li>webrtcsink: fixes for x264enc and NVIDIA encoders</li>
<li>cerbero: Pull ninja from system if possible, avoid spurious bootstrap of cmake</li>
<li>packages: Recipe updates for ffmpeg, libsoup, orc</li>
<li>various bug fixes, memory leak fixes, and other stability and reliability improvements</li>
</ul>
</p><p>
See the <a href="https://gstreamer.freedesktop.org/releases/1.22/#1.22.5">GStreamer 1.22.5 release notes</a>
for more details.
</p><p>
Binaries for Android, iOS, Mac OS X and Windows will be available shortly.
</p><p>
Release tarballs can be downloaded directly here:
<ul>
<a href="https://gstreamer.freedesktop.org/src/gstreamer/gstreamer-1.22.5.tar.xz">gstreamer</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-base/gst-plugins-base-1.22.5.tar.xz">gst-plugins-base</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-good/gst-plugins-good-1.22.5.tar.xz">gst-plugins-good</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-ugly/gst-plugins-ugly-1.22.5.tar.xz">gst-plugins-ugly</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-bad/gst-plugins-bad-1.22.5.tar.xz">gst-plugins-bad</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-libav/gst-libav-1.22.5.tar.xz">gst-libav</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-rtsp-server/gst-rtsp-server-1.22.5.tar.xz">gst-rtsp-server</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-python/gst-python-1.22.5.tar.xz">gst-python</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-editing-services/gst-editing-services-1.22.5.tar.xz">gst-editing-services</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-devtools/gst-devtools-1.22.5.tar.xz">gst-devtools</a>,
<a href="https://gstreamer.freedesktop.org/src/gstreamer-vaapi/gstreamer-vaapi-1.22.5.tar.xz">gstreamer-vaapi</a>,
<a href="https://gstreamer.freedesktop.org/src/gstreamer-sharp/gstreamer-sharp-1.22.5.tar.xz">gstreamer-sharp</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-omx/gst-omx-1.22.5.tar.xz">gst-omx</a>, or
<a href="https://gstreamer.freedesktop.org/src/gstreamer-docs/gstreamer-docs-1.22.5.tar.xz">gstreamer-docs</a>.
</ul>
</p>2023-07-20T16:00:00+00:00GStreamer: GStreamer Conference 2023 - Call for Papers is open!
https://gstreamer.freedesktop.org/news/#2023-07-10T22:00:00Z
<p>
The GStreamer project is thrilled to announce that this year's GStreamer
Conference will take place on Monday-Tuesday 25-26 September 2023
in A Coruña, Spain, followed by a hackfest.
</p><p>
You can find more details about the conference on the
<a href="https://gstreamer.freedesktop.org/conference/2023/">GStreamer
Conference 2023 web site</a>.
</p><p><b>
The call for papers is now open for talk proposals and lightning talks.
Please submit your talk now!
</b></p><p>
The initial submission needs to be only a couple of sentences or paragraphs
describing the talk you want to give and the desired length of your talk
(please allow at least 5 minutes for questions at the end as well).
</p><p>
You will still be able to edit and refine your talk proposal after the
initial submission, so no need to hold off until the last minute to get
it perfect.
</p><p>
<b>The talk submission deadline is 20 August 2023!</b>
</p><p>
Talks can be on almost anything multimedia related, ranging from talks about
applications to challenges in the lower levels in the stack or hardware.
</p><p>
Talk slots will have one of the following durations: 20 minutes, 30 minutes,
or 45 minutes, including questions. If you need more time than that we might
be able to accommodate that as well, please get in touch to discuss details.
</p><p>
We want to hear from you, what you are working on, what you are using
GStreamer for, what difficulties you have encountered and how you solved them,
what your plans are, what you like, what you dislike, how to improve things!
</p><p>
Please submit all proposals through the <b><a href="https://indico.freedesktop.org/event/5/">GStreamer Conference 2023 Paper Submission Portal</a></b>.
</p><p>
You can log in using your existing freedesktop.org GitLab account if you have one or register a new account with the system.
</p><p>
Considering becoming a sponsor? Please check out our
<a href="https://gstreamer.freedesktop.org/conference/2023/gstreamer-conference-sponsor-brief-2023.pdf">sponsor brief</a>
or <a href="mailto:sponsors@gstreamer-foundation.org">get in touch</a>.
</p><p>
We hope to see you in A Coruña!
</p><p>
Please spread the word!
</p>2023-07-10T22:00:00+00:00Phil Normand: GNOME Web Canary is back
https://base-art.net/Articles/gnome-web-canary-is-back/
<p>This is a short <span class="caps">PSA</span> post announcing the return of the <span class="caps">GNOME</span> Web Canary builds.
Read on for the crunchy details.</p>
<p>A couple years ago I was blogging about the <a href="https://base-art.net/Articles/introducing-the-gnome-web-canary-flavor/"><span class="caps">GNOME</span> Web Canary
flavor</a>.
In summary this special build of <span class="caps">GNOME</span> Web provides a preview of the upcoming
version of …</p>2023-07-08T08:00:00+00:00GStreamer: GStreamer 1.22.4 stable bug fix release
https://gstreamer.freedesktop.org/news/#2023-06-20T18:00:00Z
<p>
The GStreamer team is pleased to announce another bug fix release
in the stable 1.22 release series of your favourite cross-platform
multimedia framework!
</p><p>
This release only contains bugfixes and security fixes and it should be safe
to update from 1.22.x.
</p><p>
<b>Highlighted bugfixes:</b>
<ul>
<li><a href="https://gstreamer.freedesktop.org/security/">Security fixes</a> for flacparse, dvdspu, and subparse</li>
<li>d3d11videosink: Fix error on pause and play</li>
<li>decklink: Correctly handle SDK strings on macOS and free strings after usage on Linux</li>
<li>filesink: Fix buffered mode writing of buffer lists and buffers with multiple memories</li>
<li>gldownload: handle passthrough without a critical</li>
<li>h265parse: Fix framerate handling regression</li>
<li>oggdemux: vp8 fixes</li>
<li>mp4mux, qtmux, qtdemux: Opus audio mapping fixes</li>
<li>pngdec: Fix wrong colours output from 16bit RGB images</li>
<li>ptp clock: Work around ptpd bug in default configuration</li>
<li>srtpdec: fix critical warnings on shutdown</li>
<li>v4l2src: fix support for bayer format</li>
<li>v4l2videoenc: support force-keyframe event in v4l2 encoder</li>
<li>vtenc: apply DTS offset to ensure DTS <= PTS</li>
<li>gst-python: allow more functions to be called before <tt>gst_init()</tt></li>
<li>cerbero: fix vaapi variant; add qt6 build on windows; ensure errors on unguarded use of new APIs, require macOS 10.13</li>
<li>packages: ship codecalpha, rtponvif, dvbsubenc, switchbin, videosignal plugins; fix pango crash on 32-bit windows</li>
<li>various bug fixes, memory leak fixes, and other stability and reliability improvements</li>
</ul>
</p><p>
See the <a href="https://gstreamer.freedesktop.org/releases/1.22/#1.22.4">GStreamer 1.22.4 release notes</a>
for more details.
</p><p>
Binaries for Android, iOS, Mac OS X and Windows will be available shortly.
</p><p>
Release tarballs can be downloaded directly here:
<ul>
<a href="https://gstreamer.freedesktop.org/src/gstreamer/gstreamer-1.22.4.tar.xz">gstreamer</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-base/gst-plugins-base-1.22.4.tar.xz">gst-plugins-base</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-good/gst-plugins-good-1.22.4.tar.xz">gst-plugins-good</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-ugly/gst-plugins-ugly-1.22.4.tar.xz">gst-plugins-ugly</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-bad/gst-plugins-bad-1.22.4.tar.xz">gst-plugins-bad</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-libav/gst-libav-1.22.4.tar.xz">gst-libav</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-rtsp-server/gst-rtsp-server-1.22.4.tar.xz">gst-rtsp-server</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-python/gst-python-1.22.4.tar.xz">gst-python</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-editing-services/gst-editing-services-1.22.4.tar.xz">gst-editing-services</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-devtools/gst-devtools-1.22.4.tar.xz">gst-devtools</a>,
<a href="https://gstreamer.freedesktop.org/src/gstreamer-vaapi/gstreamer-vaapi-1.22.4.tar.xz">gstreamer-vaapi</a>,
<a href="https://gstreamer.freedesktop.org/src/gstreamer-sharp/gstreamer-sharp-1.22.4.tar.xz">gstreamer-sharp</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-omx/gst-omx-1.22.4.tar.xz">gst-omx</a>, or
<a href="https://gstreamer.freedesktop.org/src/gstreamer-docs/gstreamer-docs-1.22.4.tar.xz">gstreamer-docs</a>.
</ul>
</p>2023-06-20T18:00:00+00:00Andy Wingo: parallel futures in mobile application development
https://wingolog.org/archives/2023/06/15/parallel-futures-in-mobile-application-development
<div><p>Good morning, hackers. Today I’d like to pick up my series on mobile
application development. To recap, we looked at:</p><ul><li><p><a href="https://wingolog.org/archives/2023/04/20/structure-and-interpretation-of-capacitor-programs">Ionic/Capacitor</a>, which makes mobile app development more like web
app development;</p></li><li><p><a href="https://wingolog.org/archives/2023/04/21/structure-and-interpretation-of-react-native">React Native</a>, a flavor of React that renders to
platform-native UI components rather than the Web, with ahead-of-time
compilation of JavaScript;</p></li><li><p><a href="https://wingolog.org/archives/2023/04/24/structure-and-interpretation-of-nativescript">NativeScript</a>, which exposes all platform capabilities directly to
JavaScript and lets users layer their preferred framework on top;</p></li><li><p><a href="https://wingolog.org/archives/2023/04/26/structure-and-interpretation-of-flutter">Flutter</a>, which bypasses the platform’s native UI components to
render directly using the GPU, and uses Dart instead of
JavaScript/TypeScript; and</p></li><li><p><a href="https://wingolog.org/archives/2023/05/02/structure-and-interpretation-of-ark">Ark</a>, which is Flutter-like in its rendering, but programmed via a
dialect of TypeScript, with its own multi-tier compilation
and distribution pipeline.</p></li></ul><p>Taking a step back, with the exception of Ark which has a special
relationship to HarmonyOS and Huawei, these frameworks are all layers on
top of what is provided by Android or iOS. Why would you do that?
Presumably there are benefits to these interstitial layers; what are
they?</p><p>Probably the most basic answer is that an app framework layer offers the
promise of abstracting over the different platforms. This way you can
just have one mobile application development team instead of two or
more. In practice you still need to test on iOS and Android at least,
but this is cheaper than having fully separate Android and iOS teams.</p><p>Given that we are abstracting over platforms, it is natural also to
abandon platform-specific languages like Swift or Kotlin. This is the
moment in the strategic planning process that unleashes chaos: there is
a fundamental element of randomness and risk when choosing a programming
language and its community. Languages exist on a hype and adoption
cycle; ideally you want to catch one on its way up, and you want it to
remain popular over the life of your platform (10 years or so). This is
not an easy thing to do and it’s quite possible to bet on the wrong
horse. However the communities around popular languages also bring
their own risks, in that they have fashions that change over time, and
you might have to adapt your platform to the language as fashions come and
go, whether or not these fashions actually make better apps.</p><p>Choosing JavaScript as your language places more emphasis on the
benefits of popularity, and is in turn a promise to adapt to ongoing
fads. Choosing a more niche language like Dart places more emphasis on
predictability of where the language will go, and ability to shape the
language’s future; Flutter is a big fish in a small pond.</p><p>There are other language choices, though; if you are building your own
thing, you can choose any direction you like. What if you used Rust?
What if you doubled down on WebAssembly, somehow? In some ways we’ll
never know unless we go down one of these paths; one has to pick a
direction and stick to it for long enough to ship something, and endless
tergiversations on such basic questions as language are not helpful.
But in the early phases of platform design, all is open, and it would be
prudent to spend some time thinking about what it might look like in one
of these alternate worlds. In that spirit, let us explore these futures
to see how they might be.</p><h3>alternate world: rust</h3><p>The arc of history bends away from C and C++ and towards Rust. Given
that a mobile development platform has to have some low-level code,
there are arguments in favor of writing it in Rust already instead of
choosing to migrate in the future.</p><p>One advantage of Rust is that programs written in it generally have
fewer memory-safety bugs than their C and C++ counterparts, which is
important in the context of smart phones that handle untrusted
third-party data and programs, i.e., web sites.</p><p>Also, Rust makes it easy to write parallel programs. For the same
implementation effort, we can expect Rust programs to make more
efficient use of the hardware than C++ programs.</p><p>And relative to JavaScript et al, Rust also has the advantage of
predictable performance: it requires quite a good ahead-of-time
compiler, but no adaptive optimization at run-time.</p><p>These observations are just conversation-starters, though, and when it
comes to imagining what a real mobile device would look like with a Rust
application development framework, things get more complicated.
Firstly, there is the approach to UI: how do you get pixels on the
screen and events from the user? The three general solutions are to use
a web browser engine, to use platform-native widgets, or to build
everything in Rust using low-level graphics primitives.</p><p>The first approach is taken by the
<a href="https://github.com/tauri-apps/tauri/blob/dev/ARCHITECTURE.md">Tauri</a>
framework: an app is broken into two pieces, a Rust server and an
HTML/JS/CSS front-end. Running a Tauri app creates a WebView in which
to run the front-end, and establishes a bridge between the web client
and the Rust server. In many ways the resulting system ends up looking
a lot like Ionic/Capacitor, and many of the UI questions are left open
to the user: what UI framework to use, all of the JavaScript
programming, and so on.</p><p>Instead of using a platform’s WebView library, a Rust app could instead
ship a WebView. This would of course make the application binary size
larger, but tighter coupling between the app and the WebView may allow
you to run the UI logic from Rust itself instead of having a large JS
component. Notably this would be an interesting opportunity to adopt
the <a href="https://servo.org/">Servo</a> web engine, which is itself written in
Rust. Servo is a project that in many ways exists <i>in potentia</i>; with
more investment it could become a viable alternative to Gecko, Blink, or
WebKit, and whoever does the investment would then be in a position of
influence in the web platform.</p><p>If we look towards the platform-native side, though there are quite a
number of <a href="https://www.areweguiyet.com/">Rust libraries that provide wrappers to native
widgets</a>, practically all of these
primarily target the desktop. Only
<a href="https://github.com/ryanmcgrath/cacao">cacao</a> supports iOS widgets, and
there is no equivalent binding for Android, so any
<a href="https://wingolog.org/archives/2023/04/24/structure-and-interpretation-of-nativescript">NativeScript</a>-like
solution in Rust would require a significant amount of work.</p><p>In contrast, the ecosystem of Rust UI libraries that are implemented on
top of OpenGL and other low-level graphics facilities is much more
active and interesting. Probably the best recent overview of this
landscape is by <a href="https://raphlinus.github.io/rust/gui/2022/05/07/ui-architecture.html">Raph
Levien</a>,
(see the “quick tour of existing architectures” subsection). In
summary, everything is still in motion and there is no established
consensus as to how to approach the problem of UI development, but there
are many interesting experiments in progress. With my engineer hat on,
exploring these directions looks like fun. As Raph notes, some degree
of exploration seems necessary as well: we will only know if a given
approach is a good idea if we spend some time with it.</p><p>However if instead we consider the situation from the perspective of
someone building a mobile application development framework, Rust seems
more of a mid/long-term strategy than a concrete short-term option.
Sure, build low-level libraries in Rust, to the extent possible, but
there is no compelling-in-and-of-itself story yet that you can sell to
potential UI developers, because everything is still so undecided.</p><p>Finally, let us consider the question of scripting: sometimes you need
to add logic to a program at run-time. It could be because actually
most of your app is dynamic and comes from the network; in that case
your app is like a little virtual machine. If your app development
framework is written in JavaScript, like Ionic/Capacitor, then you have
a natural solution: just serve JavaScript. But if your app is written
in Rust, what do you do? Waiting until the app store pushes a new
version of the app to the user is not an option.</p><p>There would appear to be three common solutions to this problem. One is
to use JavaScript – that’s what Servo does, for example. As a web
engine, Servo doesn’t have much of a choice, but the point stands.
Currently Servo embeds a copy of SpiderMonkey, the JS engine from
Firefox, and it does make sense for Servo to take advantage of an
industrial, complete JS engine. Of course, SpiderMonkey is written in
C++; if there were a JS engine written in Rust, probably Rust
programmers would prefer it. Also it would be fun to write, or rather,
fun to <i>start</i> writing; reaching the level of ECMA-262 conformance of
SpiderMonkey is at least a hundred-million-dollar project. Anyway what
I am saying is that I understand why <a href="https://boajs.dev/">Boa</a> was
started, and I wish them the many millions of dollars needed to see it
through to completion.</p><p>You are not obliged to script your app via JavaScript, of course; there
are many languages out there that have “extending a low-level core” as
one of their core use cases. I think the mitigated success that this
approach has had over the years—who embeds Python into an iPhone
app?—should probably rule out this strategy as a core part of an
application development framework. Still, I should mention one
Rust-specific option, <a href="https://rhai.rs">Rhai</a>; the pitch is that by
being Rust-specific, you get more expressive interoperation between Rhai
and Rust than you would between Rust and any other dynamic language.
Still, it is not a solution that I would bet on: Rhai internalizes so
many Rust concepts (notably around borrowing and lifetimes) that I think
you have to know Rust to write effective Rhai, and knowing both is quite
rare. Anyone who writes Rhai would probably rather be writing Rust, and
that’s not a good equilibrium.</p><p>The third option for scripting Rust is WebAssembly. We’ll get to that
in a minute.</p><h3>alternate world: the web of pixels</h3><p>Let’s return to Flutter for a moment, if you will. Like the more active
Rust GUI development projects, Flutter is an all-in-one rendering
framework based on low-level primitives; all it needs is Vulkan or Metal
or (soon) WebGPU, and it handles the rest, layering on opinionated
patterns for how to build user interfaces. It didn’t arrive to this
state in a day, though. To hear <a href="https://www.youtube.com/watch?v=h7HOt3Jb1Ts">Eric Seidel tell the
story</a>, Flutter began as a
kind of “reset” for the Web, a conscious attempt to determine from the
pieces that compose the Web rendering stack, which ones enable smooth
user interfaces and which ones get in the way. After taking away all of
the parts they didn’t need, Flutter wasn’t left with much: just GPU
texture layers, a low-level drawing toolkit, and the necessary bindings
to input events. Of course what the application programmer sees is much
more high-level, but underneath, these are the platform primitives that
Flutter uses.</p><p>So, imagine you work at Google. You used to work on the web—maybe on
WebKit and then Chrome like Eric, maybe on web standards—but you broke
with this past to see what Flutter might become. Flutter works: great
job everybody! The set of graphical and input primitives that you use
is minimal enough that it is abstract by nature; it doesn’t much matter
whether you target iOS or Android, because the primitives will be there.
But the web is still the web, and it is annoying, aesthetically
speaking. Could we Flutter-ize the web? What would that mean?</p><p>That’s exactly what former HTML specification editor and now Flutter
team member Ian Hixie proposed this January in a brief manifesto,
<a href="https://docs.google.com/document/d/1peUSMsvFGvqD5yKh3GprskLC3KVdAlLGOsK6gFoEOD0/edit?pli=1&resourcekey=0-bPajpoo9IBZpG__-uCBE6w">Towards a modern Web
stack</a>.
The basic idea is that the web and thus the browser is, well, a bit
much. Hixie proposed to start over, rebuilding the web on top of
<a href="https://webassembly.org">WebAssembly</a> (for code),
<a href="https://developer.chrome.com/blog/webgpu-release/">WebGPU</a> (for
graphics),
<a href="https://developer.mozilla.org/en-US/docs/Web/API/WebHID_API">WebHID</a>
(for input), and <a href="https://w3c.github.io/aria/">ARIA</a> (for
accessibility). Technically it’s a very interesting proposition! After
all, people that build complex web apps end up having to fight with the
platform to get the results they want; if we can reorient them to focus
on these primitives, perhaps web apps can compete better with native
apps.</p><p>However if you game out what is being proposed, I have doubts. The
existing web is largely HTML, with JavaScript and CSS as add-ons: a web
of structured text. Hixie’s flutterized web proposal, on the other
hand, is a web of pixels. This has a number of implications. One is
that each app has to ship its own text renderer and internationalization
tables, which is a bit silly to say the least. And whereas we take it
for granted that we can mouse over a web page and select its text, with
a web of pixels it is much less obvious how that would happen. Hixie’s
proposal is that apps expose structure via
<a href="https://w3c.github.io/aria/">ARIA</a>, but as far as I understand there is
no association between pixels and ARIA properties: the pixels themselves
really have no built-in structure to speak of.</p><p>And of course unlike in the web of structured text, in a web of pixels
it would be up each app to actually describe its structure via ARIA:
it’s not a built-in part of the system. But if you combine this with
the rendering story (here’s WebGPU, now <a href="https://i.kym-cdn.com/photos/images/newsfeed/000/572/078/d6d.jpg">draw the rest of the
owl</a>),
Hixie’s proposal leaves a void for frameworks to fill between what the
app developer wants to write (e.g. Flutter/Dart) and the platform
(WebGPU/ARIA/etc).</p><p>I said before that I had doubts and indeed I have doubts about my
doubts. I am old enough to remember when <a href="https://www.x.org/wiki/guide/fonts/">X11 apps on Unix
desktops</a> changed from having fonts
rendered on the server (i.e. by the operating system) to having them
rendered on the client (i.e. the app), which was associated with a
similar kind of anxiety. There were similar factors at play:
slow-moving standards (X11) and not knowing at build-time what the
platform would actually provide (which X server would be in use, etc).
But instead of using the server, you could just ship pixels, and that’s
how <a href="https://www.gnome.org/">GNOME</a> got good text rendering, with
<a href="https://en.wikipedia.org/wiki/Pango">Pango</a> and
<a href="https://freetype.org/">FreeType</a> and
<a href="https://www.freedesktop.org/wiki/Software/fontconfig/">fontconfig</a>, and
eventually <a href="https://github.com/harfbuzz/harfbuzz">HarfBuzz</a>, the text
shaper used in Chromium and Flutter and many other places. Client-side
fonts not only enabled more complex text shaping but also eliminated
some round-trips for text measurement during UI layout, which is a bit
of a theme in this article series. So could it be that pixels instead
of text does not represent an apocalypse for the web? I don’t know.</p><p>Incidentally I cannot move on from this point without pointing out
another narrative thread, which is that of continued human effort over
time. <a href="https://levien.com/">Raph Levien</a>, who I mentioned above as a
Rust UI toolkit developer, actually spent quite some time doing graphics
for GNOME in the early 2000s; I remember working with his <tt>libart_lgpl</tt>.
<a href="https://behdad.org/">Behdad Esfahbod</a>, author of HarfBuzz, built many
parts of the free software text rendering stack before moving on to
Chrome and many other things. I think that if you work on this low
level where you are constantly translating text to textures, the
accessibility and interaction benefits of using a platform-provided text
library start to fade: you are the boss of text around here and you can
implement the needed functionality yourself. From this perspective,
pixels don’t represent risk at all. In the old days of GNOME 2,
client-side font rendering didn’t lead to bad UI or poor accessibility.
To be fair, there were other factors pushing to keep work in a commons,
as the actual text rendering libraries still tended to be shipped with
the operating system as shared libraries. Would similar factors prevail
in a statically-linked web of pixels?</p><p>In a way it’s a moot question for us, because in this series we are
focussing on native app development. So, if you ship a platform, should
your app development framework look like the web-of-pixels proposal, or
something else? To me it is clear that as a platform, you need more.
You need a common development story for how to build user-facing apps:
something that looks more like Flutter and less like the primitives that
Flutter uses. Though you surely will include a web-of-pixels-like
low-level layer, because you need it yourself, probably you should also
ship shared text rendering libraries, to reduce the install size for
each individual app.</p><p>And of course, having text as part of the system has the side benefit of
making it easier to get users to install OS-level security patches: it
is well-known in the industry that users will make time for the update
if they get a new goose emoji in exchange.</p><h3>alternate world: webassembly</h3><p>Hark! Have you heard the good word? Have you accepted your Lord and
savior, <a href="https://webassembly.org/">WebAssembly</a>, into your heart? I
jest; it does sometime feel like messianic narratives surrounding
WebAssembly prevent us from considering its concrete aspects. But
despite the hype, WebAssembly is clearly a technology that will be a
part of the future of computing. So let’s dive in: what would it mean
for a mobile app development platform to embrace WebAssembly?</p><p>Before answering that question, a brief summary of what WebAssembly is.
WebAssembly 1.0 is portable bytecode format that is a good compilation
target for C, C++, and Rust. These languages have good compiler
toolchains that can produce WebAssembly. The nice thing is that when
you instantiate a WebAssembly module, it is completely isolated from its
host: it can’t harm the host (approximately speaking). All points of
interoperation with the host are via copying data into memory owned by
the WebAssembly guest; the compiler toolchains abstract over these
copies, allowing a Rust-compiled-to-native host to call into a
Rust-compiled-to-WebAssembly module using idiomatic Rust code.</p><p>So, WebAssembly 1.0 can be used as a way to script a Rust application.
The guest script can be
<a href="https://dl.acm.org/doi/abs/10.1145/3563311">interpreted</a>, compiled just
in time, or compiled ahead of time for peak throughput.</p><p>Of course, people that would want to script an application probably want
a higher-level language than Rust. In a way, WebAssembly is in a
similar situation as WebGPU in the web-of-pixels proposal: it is a
low-level tool that needs higher-level toolchains and patterns to bridge
the gap between developers and primitives.</p><p>Indeed, the web-of-pixels proposal specifies WebAssembly as the compute
primitive. The idea is that you ship your application as a WebAssembly
module, and give that module WebGPU, WebHID, and ARIA capabilities via
<a href="https://webassembly.github.io/spec/core/syntax/modules.html#syntax-import">imports</a>.
Such a WebAssembly module doesn’t script an existing application: it
<i>is</i> the app. So another way for an app development platform to use
WebAssembly would be like how the web-of-pixels proposes to do it: as an
interchange format and as a low-level abstraction. As in the scripting
case, you can interpret or compile the module. Perhaps an
infrequently-run app would just be interpreted, to save on disk space,
whereas a more heavily-used app would be optimized ahead of time, or
something.</p><p>We should mention another interesting benefit of WebAssembly as a
distribution format, which is that it abstracts over the specific
chipset on the user’s device; it’s the device itself that is responsible
for efficiently executing the program, possibly via compilation to
specialized machine code. I understand for example that RISC-V people
are quite happy about this property because it lowers the barrier to
entry for them relative to an ARM monoculture.</p><p>WebAssembly does have some limitations, though. One is that if the
throughput of data transfer between guest and host is high, performance
can be bad due to copying overhead. The nascent <a href="https://github.com/WebAssembly/memory-control/blob/main/proposals/memory-control/Overview.md">memory-control
proposal</a>
aims to provide an <tt>mmap</tt> capability, but it is still early days. The
need to copy would be a limitation for using WebGPU primitives.</p><p>More generally, as an abstraction, WebAssembly may not be able to
express programs in the most efficient way for a given host platform.
For example, its SIMD operations work on 128-bit vectors, whereas host
platforms may have much wider vectors. Any current limitation will
recede with time, as WebAssembly gains new features, but every year brings new hardware capabilities (tensor operation accelerator,
anyone?), so there will be some impedance-matching to do for the
foreseeable future.</p><p>The more fundamental limitation of the 1.0 version of WebAssembly is
that it’s only a good compilation target for some languages. This is
because some of the fundamental parts of WebAssembly that enable
isolation between host and guest (structured control flow, opaque stack,
no instruction pointer) make it difficult to efficiently implement
languages that need garbage collection, such as Java or Go. The coming
WebAssembly 2.0 starts to address this need by including low-level
managed arrays and records, allowing for reasonable ahead-of-time
compilation of languages like Java. Getting a dynamic language like
JavaScript to compile to efficient WebAssembly can still be a challenge,
though, because many of the just-in-time techniques needed to
efficiently implement these languages will still be missing in
WebAssembly 2.0.</p><p>Before moving on to WebAssembly as part of an app development framework,
one other note: currently WebAssembly modules do not compose very well
with each other and with the host, requiring extensive toolchain support
to enable e.g. the use of any data type that’s not a scalar integer or
floating-point value. The <a href="https://github.com/WebAssembly/component-model">component model working
group</a> is trying to
establish some abstractions and associated tooling, but (again!) it is
still early days. Anyone wading into this space needs to be prepared to
get their hands dirty.</p><p>To return to the question at hand, an app development framework can use
WebAssembly for scripting, though the problem of how to compose a host
application with a guest script requires good tooling. Or, an app
development framework that exposes a web-of-pixels primitive layer can
support running WebAssembly apps directly, though again, the set of
imports remains to be defined. Either of these two patterns can stick
with WebAssembly 1.0 or also allow for garbage collection in WebAssembly
2.0, aiming to capture mindshare among a broader community of potential
developers, potentially in a wide range of languages.</p><p>As a final observation: WebAssembly is ecumenical, in the sense that it
favors no specific church of how to write programs. As a platform,
though, you might prefer a state religion, to avoid wasting internal and
external efforts on redundant or ill-advised development. After all, if
it’s your platform, presumably you know best.</p><h3>summary</h3><p>What is to be done?</p><p>Probably there are as many answers as people, but since this is my blog,
here are mine:</p><ol><li><p>On the shortest time-scale I think that it is entirely reasonable to
base a mobile application development framework on JavaScript. I
would particularly focus on TypeScript, as late error detection is
more annoying in native applications.</p></li><li><p>I would to build something that looks like Flutter underneath:
reactive, based on low-level primitives, with a multithreaded
rendering pipeline. Perhaps it makes sense to take some inspiration
from <a href="https://github.com/openwebf/webf">WebF</a>.</p></li><li><p>In the medium-term I am sympathetic to Ark’s desire to extend the
language in a more <a href="https://github.com/apple/swift-evolution/blob/main/proposals/0289-result-builders.md">ResultBuilder</a>-like
direction, though this is not without risk.</p></li><li><p>Also in the medium-term I think that modifications to TypeScript to
allow for sound typing could provide some of the advantages of
Dart’s ahead-of-time compiler to JavaScript developers.</p></li><li><p>In the long term... well we can do all things with unlimited
resources, right? So after solving climate change and homelessness,
it makes sense to invest in frameworks that might be usable 3 or 5
years from now. WebAssembly in particular has a chance of sweeping
across all platforms, and the primitives for the web-of-pixels will
be present everywhere, so if you manage to produce a compelling
application development story targetting those primitives, you could
eat your competitors’ lunch.</p></li></ol><p>Well, friends, that brings this article series to an end; it has been
interesting for me to dive into this space, and if you have read down to
here, I can only think that you are a masochist or that you have also
found it interesting. In either case, you are very welcome. Until next
time, happy hacking.</p></div>2023-06-15T14:02:16+00:00Andy WingoGStreamer: GStreamer Conference 2023 announced to take place 25-26 Sept 2023 in A Coruña, Spain
https://gstreamer.freedesktop.org/news/#2023-06-14T18:00:00Z
<p>
The GStreamer project is thrilled to announce that this year's GStreamer
Conference will take place on Monday-Tuesday 25-26 September 2023
in A Coruña, Spain, followed by a hackfest.
</p><p>
You can find more details about the conference on the
<a href="https://gstreamer.freedesktop.org/conference/2023/">GStreamer
Conference 2023 web site</a>.
</p><p>
A call for papers will be sent out in due course.
</p><p>
Registration will open at a later time, in late June / early July.
</p><p>
We will announce those and any further updates on the
<a href="https://lists.freedesktop.org/mailman/listinfo/gstreamer-announce">gstreamer-announce mailing list</a>,
the <a href="https://gstreamer.freedesktop.org/news/">website</a>,
<a href="https://twitter.com/GStreamer">on Twitter</a>.
and <a href="https://floss.social/@GStreamer">on Mastodon</a>.
</p><p>
Talk slots will be available in varying durations from 20 minutes up to
45 minutes. Whatever you're doing or planning to do with GStreamer,
we'd like to hear from you!
</p><p>
We also plan to have sessions with short lightning talks / demos /
showcase talks for those who just want to show what they've been
working on or do a mini-talk instead of a full-length talk. Lightning
talk slots will be allocated on a first-come-first-serve basis, so make
sure to reserve your slot if you plan on giving a lightning talk.
</p><p>
Our main social event will be on Monday evening, and there'll also be a welcome drinks/snacks get-together on Sunday evening.
</p><p>
A GStreamer hackfest will take place right after the conference, on 27-29 September 2023.
</p><p>
Considering becoming a sponsor? Please check out our
<a href="https://gstreamer.freedesktop.org/conference/2023/gstreamer-conference-sponsor-brief-2023.pdf">sponsor brief</a>
or <a href="mailto:sponsors@gstreamer-foundation.org">get in touch</a>.
</p><p>
We hope to see you in A Coruña!
</p><p>
Please spread the word!
</p>2023-06-14T18:00:00+00:00Sebastian Pölsterl: scikit-survival 0.21.0 released
https://k-d-w.org/blog/2023/06/scikit-survival-0.21.0-released/
<p>Today marks the release of <a href="https://scikit-survival.readthedocs.io/" target="_blank">scikit-survival</a> 0.21.0.
This release features some exciting new features and significant performance improvements:</p>
<ul>
<li>Pointwise confidence intervals for the Kaplan-Meier estimator.</li>
<li>Early stopping in GradientBoostingSurvivalAnalysis.</li>
<li>Improved performance of fitting SurvivalTree and RandomSurvivalForest.</li>
<li>Reduced memory footprint of concordance_index_censored.</li>
</ul>
<h2 id="pointwise-confidence-intervals-for-the-kaplan-meier-estimator">Pointwise Confidence Intervals for the Kaplan-Meier Estimator</h2>
<p><a href="https://scikit-survival.readthedocs.io/en/v0.21.0/api/generated/sksurv.nonparametric.kaplan_meier_estimator.html#sksurv.nonparametric.kaplan_meier_estimator" target="_blank">kaplan_meier_estimator()</a>
can now estimate pointwise confidence intervals by specifying the <code>conf_type</code> parameter.</p>
<pre><code class="language-python">import matplotlib.pyplot as plt
from sksurv.datasets import load_veterans_lung_cancer
from sksurv.nonparametric import kaplan_meier_estimator
_, y = load_veterans_lung_cancer()
time, survival_prob, conf_int = kaplan_meier_estimator(
y["Status"], y["Survival_in_days"], conf_type="log-log"
)
plt.step(time, survival_prob, where="post")
plt.fill_between(time, conf_int[0], conf_int[1], alpha=0.25, step="post")
plt.ylim(0, 1)
plt.ylabel("est. probability of survival $\hat{S}(t)$")
plt.xlabel("time $t$")
</code></pre>
<img src="https://k-d-w.org/blog/2023/06/scikit-survival-0.21.0-released/img/km-with-ci.svg" alt="Kaplan-Meier curve with pointwise confidence intervals." />
<p>Kaplan-Meier curve with pointwise confidence intervals.</p>
<h2 id="early-stopping-in-gradientboostingsurvivalanalysis">Early Stopping in GradientBoostingSurvivalAnalysis</h2>
<p>Early stopping enables us to determine when the model is sufficiently complex.
This is usually done by continuously evaluating the model on held-out data.
For <a href="https://scikit-survival.readthedocs.io/en/v0.21.0/api/generated/sksurv.ensemble.GradientBoostingSurvivalAnalysis.html#sksurv.ensemble.GradientBoostingSurvivalAnalysis" target="_blank">GradientBoostingSurvivalAnalysis</a>,
the easiest way to achieve this is by setting <code>n_iter_no_change</code> and
optionally <code>validation_fraction</code> (defaults to 0.1).</p>
<pre><code class="language-python">from sksurv.datasets import load_whas500
from sksurv.ensemble import GradientBoostingSurvivalAnalysis
X, y = load_whas500()
model = GradientBoostingSurvivalAnalysis(
n_estimators=1000, max_depth=2, subsample=0.8, n_iter_no_change=10, random_state=0,
)
model.fit(X, y)
print(model.n_estimators_)
</code></pre>
<p>In this example, <code>model.n_estimators_</code> indicates that fitting stopped after 73 iterations,
instead of the maximum 1000 iterations.</p>
<p>Alternatively, one can provide a custom callback function to the
<a href="https://scikit-survival.readthedocs.io/en/v0.21.0/api/generated/sksurv.ensemble.GradientBoostingSurvivalAnalysis.html#sksurv.ensemble.GradientBoostingSurvivalAnalysis.fit" target="_blank">fit</a>
method. If the callback returns <code>True</code>, training is stopped.</p>
<pre><code class="language-python">model = GradientBoostingSurvivalAnalysis(
n_estimators=1000, max_depth=2, subsample=0.8, random_state=0,
)
def early_stopping_monitor(iteration, model, args):
"""Stop training if there was no improvement in the last 10 iterations"""
start = max(0, iteration - 10)
end = iteration + 1
oob_improvement = model.oob_improvement_[start:end]
return all(oob_improvement < 0)
model.fit(X, y, monitor=early_stopping_monitor)
print(model.n_estimators_)
</code></pre>
<p>In the example above, early stopping is determined by checking
the last 10 entries of the <code>oob_improvement_</code> attribute.
It contains the improvement in loss on the out-of-bag samples
relative to the previous iteration.
This requires setting <code>subsample</code> to a value smaller 1, here 0.8.
Using this approach, training stopped after 114 iterations.</p>
<h2 id="improved-performance-of-survivaltree-and-randomsurvivalforest">Improved Performance of SurvivalTree and RandomSurvivalForest</h2>
<p>Another exciting feature of scikit-survival 0.21.0 is due to a re-write of
the training routine of <a href="https://scikit-survival.readthedocs.io/en/v0.21.0/api/generated/sksurv.tree.SurvivalTree.html#sksurv.tree.SurvivalTree" target="_blank">SurvivalTree</a>.
This results in roughly <strong>3x faster training times</strong>.</p>
<img src="https://k-d-w.org/blog/2023/06/scikit-survival-0.21.0-released/img/fit-times.svg" alt="Runtime comparison of fitting SurvivalTree." />
<p>Runtime comparison of fitting SurvivalTree.</p>
<p>The plot above compares the time required to fit a single SurvivalTree on data with
25 features and varying number of samples.
The performance difference becomes notable for data with 1000 samples and above.</p>
<p>Note that this improvement also speeds-up fitting
<a href="https://scikit-survival.readthedocs.io/en/v0.21.0/api/generated/sksurv.ensemble.RandomSurvivalForest.html#sksurv.ensemble.RandomSurvivalForest" target="_blank">RandomSurvivalForest</a>
and <a href="https://scikit-survival.readthedocs.io/en/v0.21.0/api/generated/sksurv.ensemble.ExtraSurvivalTrees.html#sksurv.ensemble.ExtraSurvivalTrees" target="_blank">ExtraSurvivalTrees</a>.</p>
<h2 id="improved-concordance-index">Improved concordance index</h2>
<p>Another performance improvement is due to <a href="https://github.com/cpoerschke" target="_blank">Christine Poerschke</a>
who significantly reduced the memory footprint of
<a href="https://scikit-survival.readthedocs.io/en/v0.21.0/api/generated/sksurv.metrics.concordance_index_censored.html#sksurv.metrics.concordance_index_censored" target="_blank">concordance_index_censored()</a>.
With scikit-survival 0.21.0, memory usage <strong>scales linear, instead of quadratic</strong>, in the number of samples, making performance evaluation on large datasets much more manageable.</p>
<p>For a full list of changes in scikit-survival 0.21.0, please see the
<a href="https://scikit-survival.readthedocs.io/en/v0.21.0/release_notes.html#scikit-survival-0-21-0-2023-06-11" target="_blank">release notes</a>.</p>
<h2 id="install">Install</h2>
<p>Pre-built conda packages are available for Linux, macOS (Intel), and Windows, either</p>
<p>via pip:</p>
<pre><code class="language-bash">pip install scikit-survival
</code></pre>
<p>or via conda</p>
<pre><code class="language-bash"> conda install -c sebp scikit-survival
</code></pre>2023-06-11T16:43:30+00:00GStreamer: Orc 0.4.34 release
https://gstreamer.freedesktop.org/news/#2023-05-30T01:00:00Z
<p>
The GStreamer team is pleased to announce another release of liborc,
the Optimized Inner Loop Runtime Compiler.
</p><p>
<b>Highlights:</b>
<ul>
<li>Thread-safety improvements around orc codemem allocation/freeing</li>
<li>Add orc_parse_code() with more detailed error reporting</li>
<li>Implement Orc function lazy initialization correctly via atomic operations</li>
<li>orc program parser fixes and improvements</li>
<li>build fixes and compiler warning fixes</li>
<li>coverity and clang scan-build static code analysis fixes</li>
<li>meson: Do not always generate static library for test library</li>
<li>ci improvements</li>
</ul>
</p><p>
Direct tarball download: <a href="https://gstreamer.freedesktop.org/src/orc/orc-0.4.34.tar.xz">orc-0.4.34</a>.
</p>2023-05-30T01:00:00+00:00Andy Wingo: approaching cps soup
https://wingolog.org/archives/2023/05/20/approaching-cps-soup
<div><p>Good evening, hackers. Today's missive is more of a massive, in the
sense that it's another presentation transcript-alike; these things always
translate to many vertical pixels.</p><p>In my defense, I hardly ever give a
presentation twice, so not only do I miss out on the usual
per-presentation cost amortization and on the incremental improvements
of repetition, the more dire error is that whatever message I might have
can only ever reach a subset of those that it might interest; here at
least I can be more or less sure that if the presentation would interest
someone, that they will find it.</p><p>So for the time being I will try to
share presentations here, in the spirit of, well, why the hell not.</p><div>
<h1>CPS Soup</h1>
<p>A functional intermediate language</p>
<p>10 May 2023 – Spritely</p>
<p>Andy Wingo</p>
<p>Igalia, S.L.</p>
</div><p>Last week I gave a training talk to <a href="https://spritely.instutute">Spritely
Institute</a> collaborators on the intermediate
representation used by <a href="https://gnu.org/s/guile">Guile</a>'s compiler.</p><div>
<h2>CPS Soup</h2>
<p>Compiler: Front-end to Middle-end to Back-end</p>
<p>Middle-end spans gap between high-level source code (AST) and low-level machine code</p>
<p>Programs in middle-end expressed in intermediate language</p>
<p>CPS Soup is the language of Guile’s middle-end</p>
</div><p>An <i>intermediate representation</i> (IR) (or <i>intermediate language</i>, IL)
is just another way to express a computer program. Specifically it's
the kind of language that is appropriate for the middle-end of a
compiler, and by "appropriate" I meant that an IR serves a purpose:
there has to be a straightforward transformation to the IR from
high-level abstract syntax trees (ASTs) from the front-end, and there
has to be a straightforward translation from IR to machine code.</p><p>There are also usually a set of necessary source-to-source
transformations on IR to "lower" it, meaning to make it closer to the
back-end than to the front-end. There are usually a set of optional
transformations to the IR to make the program run faster or allocate
less memory or be more simple: these are the <i>optimizations</i>.</p><p>"CPS soup" is Guile's IR. This talk presents the essentials of CPS soup
in the context of more traditional IRs.</p><div>
<h2>How to lower?</h2>
<p>High-level:</p>
<pre>(+ 1 (if x 42 69))</pre>
<p>Low-level:</p>
<pre> cmpi $x, #f
je L1
movi $t, 42
j L2
L1:
movi $t, 69
L2:
addi $t, 1</pre>
<p>How to get from here to there?</p>
</div><p>Before we dive in, consider what we might call the dynamic range of an
intermediate representation: we start with what is usually an algebraic
formulation of a program and we need to get down to a specific sequence
of instructions operating on registers (unlimited in number, at this
stage; allocating to a fixed set of registers is a back-end concern),
with explicit control flow between them. What kind of a language might
be good for this? Let's attempt to answer the question by looking into
what the standard solutions are for this problem domain.</p><div>
<h2>1970s</h2>
<p>Control-flow graph (CFG)</p>
<pre>graph := array<block>
block := tuple<preds, succs, insts>
inst := goto B
| if x then BT else BF
| z = const C
| z = add x, y
...
BB0: if x then BB1 else BB2
BB1: t = const 42; goto BB3
BB2: t = const 69; goto BB3
BB3: t2 = addi t, 1; ret t2</pre>
<p>Assignment, not definition</p>
</div><p>Of course in the early days, there was no intermediate language;
compilers translated ASTs directly to machine code. It's been a while
since I dove into all this but the milestone I have in my head is that
it's the 70s when compiler middle-ends come into their own right, with
Fran Allen's work on flow analysis and optimization.</p><p>In those days the intermediate representation for a compiler was a graph
of basic blocks, but unlike today the paradigm was assignment to
locations rather than definition of values. By that I mean that in our
example program, we get <tt>t</tt> assigned to in two places (BB1 and BB2); the
actual definition of <tt>t</tt> is implicit, as a storage location, and our
graph consists of assignments to the set of storage locations in the
program.</p><div>
<h2>1980s</h2>
<p>Static single assignment (SSA) CFG</p>
<pre>graph := array<block>
block := tuple<preds, succs, phis, insts>
phi := z := φ(x, y, ...)
inst := z := const C
| z := add x, y
...
BB0: if x then BB1 else BB2
BB1: v0 := const 42; goto BB3
BB2: v1 := const 69; goto BB3
BB3: v2 := φ(v0,v1); v3:=addi t,1; ret v3</pre>
<p>
Phi is phony function: <tt>v2</tt> is <tt>v0</tt> if coming from first predecessor, or <tt>v1</tt> from second predecessor
</p>
</div><p>These days we still live in Fran Allen's world, but with a twist: we no
longer model programs as graphs of assignments, but rather graphs of
definitions. The introduction in the mid-80s of so-called "static
single-assignment" (SSA) form graphs mean that instead of having two
assignments to <tt>t</tt>, we would define two different values <tt>v0</tt> and <tt>v1</tt>.
Then later instead of reading the value of the storage location
associated with <tt>t</tt>, we define <tt>v2</tt> to be <i>either</i> <tt>v0</tt> or <tt>v1</tt>: the
former if we reach the use of <tt>t</tt> in BB3 from BB1, the latter if we are
coming from BB2.</p><p>If you think on the machine level, in terms of what the resulting
machine code will be, this <i>either</i> function isn't a real operation;
probably register allocation will put <tt>v0</tt>, <tt>v1</tt>, and <tt>v2</tt> in the same
place, say <tt>$rax</tt>. The function linking the definition of <tt>v2</tt> to the
inputs <tt>v0</tt> and <tt>v1</tt> is purely notational; in a way, you could say that
it is <i>phony</i>, or not real. But when the creators of SSA went to submit
this notation for publication they knew that they would need something
that sounded more rigorous than "phony function", so they instead called it a "phi" (φ)
function. Really.</p><div>
<h2>2003: MLton</h2>
<p>Refinement: phi variables are basic block args</p>
<pre>graph := array<block>
block := tuple<preds, succs, args, insts></pre>
<p>Inputs of phis implicitly computed from preds</p>
<pre>BB0(a0): if a0 then BB1() else BB2()
BB1(): v0 := const 42; BB3(v0)
BB2(): v1 := const 69; BB3(v1)
BB3(v2): v3 := addi v2, 1; ret v3</pre>
</div><p>SSA is still where it's at, as a conventional solution to the IR
problem. There have been some refinements, though. I learned of one of
them from <a href="https://mlton.org/">MLton</a>; I don't know if they were first
but they had the idea of interpreting phi variables as <i>arguments</i> to
basic blocks. In this formulation, you don't have explicit phi
instructions; rather the "<tt>v2</tt> is either <tt>v1</tt> or <tt>v0</tt>" property is
expressed by <tt>v2</tt> being a parameter of a block which is "called" with
either <tt>v0</tt> or <tt>v1</tt> as an argument. It's the same semantics, but an
interesting notational change.</p><div>
<h2>Refinement: Control tail</h2>
<p>Often nice to know how a block ends (e.g. to compute phi input vars)</p>
<pre>graph := array<block>
block := tuple<preds, succs, args, insts,
control>
control := if v then L1 else L2
| L(v, ...)
| switch(v, L1, L2, ...)
| ret v</pre>
</div><p>One other refinement to SSA is to note that basic blocks consist of some
number of instructions that can define values or have side effects but
which otherwise exhibit fall-through control flow, followed by a single
instruction that transfers control to another block. We might as well
store that control instruction separately; this would let us easily know
how a block ends, and in the case of phi block arguments, easily say
what values are the inputs of a phi variable. So let's do that.</p><div>
<h2>Refinement: DRY</h2>
<p>Block successors directly computable from control</p>
<p>Predecessors graph is inverse of successors graph</p>
<pre>graph := array<block>
block := tuple<args, insts, control></pre>
<p>Can we simplify further?</p>
</div><p>At this point we notice that we are repeating ourselves; the successors
of a block can be computed directly from the block's terminal control
instruction. Let's drop those as a distinct part of a block, because
when you transform a program it's unpleasant to have to needlessly
update something in two places.</p><p>While we're doing that, we note that the predecessors array is also
redundant, as it can be computed from the graph of block successors.
Here we start to wonder: am I simpliying or am I removing something that
is fundamental to the algorithmic complexity of the various graph
transformations that I need to do? We press on, though, hoping we will
get somewhere interesting.</p><div>
<h2>Basic blocks are annoying</h2>
<p>Ceremony about managing insts; array or doubly-linked list?</p>
<p>Nonuniformity: “local” vs ‘`global’' transformations</p>
<p>Optimizations transform graph A to graph B; mutability complicates this task</p>
<ul>
<li>Desire to keep A in mind while making B</li>
<li>Bugs because of spooky action at a distance</li>
</ul>
</div><p>Recall that the context for this meander is Guile's compiler, which is written in Scheme. Scheme doesn't have expandable arrays built-in. You
can build them, of course, but it is annoying. Also, in Scheme-land,
functions with side-effects are conventionally suffixed with an
exclamation mark; after too many of them, both the writer and the
reader get fatigued. I know it's a silly argument but it's one of the
things that made me grumpy about basic blocks.</p><p>If you permit me to continue with this introspection, I find there is an
uneasy relationship between instructions and locations in an IR that is
structured around basic blocks. Do instructions live in a
function-level array and a basic block is an array of instruction
indices? How do you get from instruction to basic block? How would you
hoist an instruction to another basic block, might you need to
reallocate the block itself?</p><p>And when you go to transform a graph of blocks... well how do you do
that? Is it in-place? That would be efficient; but what if you need to
refer to the original program during the transformation? Might you risk
reading a stale graph?</p><p>It seems to me that there are too many concepts, that in the same way
that SSA itself moved away from assignment to a more declarative
language, that perhaps there is something else here that might be more
appropriate to the task of a middle-end.</p><div>
<h2>Basic blocks, phi vars redundant</h2>
<p>Blocks: label with args sufficient; “containing” multiple instructions is superfluous</p>
<p>Unify the two ways of naming values: every var is a phi</p>
<pre>graph := array<block>
block := tuple<args, inst>
inst := L(expr)
| if v then L1() else L2()
...
expr := const C
| add x, y
...</pre>
</div><p>I took a number of tacks here, but the one I ended up on was to declare
that basic blocks themselves are redundant. Instead of containing an
array of instructions with fallthrough control-flow, why not just make
every instruction a control instruction? (Yes, there are arguments
against this, but do come along for the ride, we get to a funny place.)</p><p>While you are doing that, you might as well unify the two ways in which
values are named in a MLton-style compiler: instead of distinguishing
between basic block arguments and values defined within a basic block,
we might as well make all names into basic block arguments.</p><div>
<h2>Arrays annoying</h2>
<p>Array of blocks implicitly associates a label with each block</p>
<p>Optimizations add and remove blocks; annoying to have dead array entries</p>
<p>Keep labels as small integers, but use a map instead of an array</p>
<pre>graph := map<label, block></pre>
</div><p>In the traditional SSA CFG IR, a graph transformation would often not
touch the structure of the graph of blocks. But now having given each
instruction its own basic block, we find that transformations of the
program necessarily change the graph. Consider an instruction that we
elide; before, we would just remove it from its basic block, or replace
it with a no-op. Now, we have to find its predecessor(s), and forward
them to the instruction's successor. It would be useful to have a more
capable data structure to represent this graph. We might as well keep
labels as being small integers, but allow for sparse maps and growth by
using an integer-specialized map instead of an array.</p><div>
<h2>This is CPS soup</h2>
<pre>
graph := map<label, <b>cont</b>>
cont := tuple<args, <b>term</b>>
term := <b>continue</b> to L
with values from expr
| if v then L1() else L2()
...
expr := const C
| add x, y
...
</pre>
<p>SSA is CPS</p>
</div><p>This is exactly what CPS soup is! We came at it "from below", so to
speak; instead of the heady fumes of the lambda calculus, we get here
from down-to-earth basic blocks. (If you prefer the other way around,
you might enjoy <a href="https://wingolog.org/archives/2011/07/12/static-single-assignment-for-functional-programmers">this article from a long time
ago</a>.)
The remainder of this presentation goes deeper into what it is like to
work with CPS soup in practice.</p><div>
<h2>Scope and dominators</h2>
<pre>BB0(a0): if a0 then BB1() else BB2()
BB1(): v0 := const 42; BB3(v0)
BB2(): v1 := const 69; BB3(v1)
BB3(v2): v3 := addi v2, 1; ret v3</pre>
<p>
What vars are “in scope” at BB3? <tt>a0</tt> and <tt>v2</tt>.
</p>
<p>
Not <tt>v0</tt>; not all paths from BB0 to BB3 define <tt>v0</tt>.
</p>
<p>
<tt>a0</tt> always defined: its definition <i>dominates</i> all uses.
</p>
<p>BB0 dominates BB3: All paths to BB3 go through BB0.</p>
</div><p>Before moving on, though, we should discuss what it means in an
SSA-style IR that variables are defined rather than assigned. If you
consider variables as locations to which values can be assigned and
which initially hold garbage, you can read them at any point in your
program. You might get garbage, though, if the variable wasn't assigned
something sensible on the path that led to reading the location's value.
It sounds bonkers but it is still the C and C++ semantic model.</p><p>If we switch instead to a definition-oriented IR, then a variable never
has garbage; the single definition always precedes any uses of the
variable. That is to say that all paths from the function entry to the
use of a variable must pass through the variable's definition, or, in
the jargon, that <i>definitions dominate uses</i>. This is an invariant of
an SSA-style IR, that all variable uses be dominated by their associated
definition.</p><p>You can flip the question around to ask what variables are available for
use at a given program point, which might be read equivalently as which
variables are in scope; the answer is, all definitions from all program
points that dominate the use site. The "CPS" in "CPS soup" stands for
<i>continuation-passing style</i>, a dialect of the lambda calculus, which
has also has a history of use as a compiler intermediate representation.
But it turns out that if we use the lambda calculus in its conventional
form, we end up needing to maintain a lexical scope nesting at the same
time that we maintain the control-flow graph, and the lexical scope tree
can fail to reflect the dominator tree. I go into this topic in more
detail in <a href="https://wingolog.org/archives/2015/07/27/cps-soup">an old
article</a>, and if it
interests you, please do go deep.</p><div>
<h2>CPS soup in Guile</h2>
<p>Compilation unit is intmap of label to cont</p>
<pre>cont := $kargs names vars term
| ...
term := $continue k src expr
| ...
expr := $const C
| $primcall ’add #f (a b)
| ...</pre>
<p>Conventionally, entry point is lowest-numbered label</p>
</div><p>Anyway! In Guile, the concrete form that CPS soup takes is that a
program is an intmap of <i>label</i> to <i>cont</i>. A cont is the smallest
labellable unit of code. You can call them blocks if that makes you
feel better. One kind of cont, <tt>$kargs</tt>, binds incoming values to
variables. It has a list of variables, <i>vars</i>, and also has an
associated list of human-readable names, <i>names</i>, for debugging
purposes.</p><p>A <tt>$kargs</tt> contains a <i>term</i>, which is like a control instruction. One
kind of term is <tt>$continue</tt>, which passes control to a continuation <i>k</i>.
Using our earlier language, this is just <tt>goto *k*</tt>, with values, as in
MLton. (The <i>src</i> is a source location for the term.) The values come
from the term's <i>expr</i>, of which there are a dozen kinds or so, for
example <tt>$const</tt> which passes a literal constant, or <tt>$primcall</tt>, which
invokes some kind of primitive operation, which above is <tt>add</tt>. The
primcall may have an immediate operand, in this case <tt>#f</tt>, and some
variables that it uses, in this case <tt>a</tt> and <tt>b</tt>. The number and type
of the produced values is a property of the primcall; some are just for
effect, some produce one value, some more.</p><div>
<h2>CPS soup</h2>
<pre>term := $continue k src expr
| $branch kf kt src op param args
| $switch kf kt* src arg
| $prompt k kh src escape? tag
| $throw src op param args</pre>
<p>Expressions can have effects, produce values</p>
<pre>expr := $const val
| $primcall name param args
| $values args
| $call proc args
| ...</pre>
</div><p>There are other kinds of terms besides <tt>$continue</tt>: there is <tt>$branch</tt>,
which proceeds either to the false continuation <i>kf</i> or the true
continuation <i>kt</i> depending on the result of performing <i>op</i> on the
variables <i>args</i>, with immediate operand <i>param</i>. In our running
example, we might have made the initial term via:</p><pre>(build-term
($branch BB1 BB2 'false? #f (a0)))
</pre><p>The definition of <tt>build-term</tt> (and <tt>build-cont</tt> and <tt>build-exp</tt>) is in
the <a href="https://www.gnu.org/software/guile/manual/html_node/Building-CPS.html"><tt>(language cps)</tt></a>
module.</p><p>There is also <tt>$switch</tt>, which takes an unboxed unsigned integer <i>arg</i>
and performs an array dispatch to the continuations in the list <i>kt</i>,
or <i>kf</i> otherwise.</p><p>There is <tt>$prompt</tt> which continues to its <i>k</i>, having pushed on a new
continuation delimiter associated with the var <i>tag</i>; if code aborts to
<i>tag</i> before the prompt exits via an <tt>unwind</tt> primcall, the stack will
be unwound and control passed to the handler continuation <i>kh</i>. If
<i>escape?</i> is true, the continuation is escape-only and aborting to the
prompt doesn't need to capture the suspended continuation.</p><p>Finally there is <tt>$throw</tt>, which doesn't continue at all, because it
causes a non-resumable exception to be thrown. And that's it; it's just
a handful of kinds of term, determined by the different shapes of
control-flow (how many continuations the term has).</p><p>When it comes to values, we have about a dozen expression kinds. We saw
<tt>$const</tt> and <tt>$primcall</tt>, but I want to explicitly mention <tt>$values</tt>,
which simply passes on some number of values. Often a <tt>$values</tt>
expression corresponds to passing an input to a phi variable, though
<tt>$kargs</tt> vars can get their definitions from any expression that
produces the right number of values.</p><div>
<h2>Kinds of continuations</h2>
<p>Guile functions untyped, can multiple return values</p>
<p>Error if too few values, possibly truncate too many values, possibly cons as rest arg...</p>
<p>Calling convention: contract between val producer & consumer</p>
<ul>
<li>both on call and return side</li>
</ul>
<p>
Continuation of <tt>$call</tt> unlike that of <tt>$const</tt>
</p>
</div><p>When a <tt>$continue</tt> term continues to a <tt>$kargs</tt> with a <tt>$const 42</tt>
expression, there are a number of invariants that the compiler can
ensure: that the <tt>$kargs</tt> continuation is always passed the expected
number of values, that the <i>vars</i> that it binds can be allocated to
specific locations (e.g. registers), and that because all predecessors
of the <tt>$kargs</tt> are known, that those predecessors can place their
values directly into the variable's storage locations. Effectively, the
compiler determines a <i>custom calling convention</i> between each <tt>$kargs</tt>
and its predecessors.</p><p>Consider the <tt>$call</tt> expression, though; in general you don't know what
the callee will do to produce its values. You don't even generally know
that it will produce the right number of values. Therefore <tt>$call</tt>
can't (in general) continue to <tt>$kargs</tt>; instead it continues to
<tt>$kreceive</tt>, which expects the return values in well-known places. <tt>$kreceive</tt> will
check that it is getting the right number of values and then continue to
a <tt>$kargs</tt>, shuffling those values into place. A <i>standard calling
convention</i> defines how functions return values to callers.</p><div>
<h2>The conts</h2>
<pre>cont := $kfun src meta self ktail kentry
| $kclause arity kbody kalternate
| $kargs names syms term
| $kreceive arity kbody
| $ktail</pre>
<p>$kclause, $kreceive very similar</p>
<p>Continue to $ktail: return</p>
<p>$call and return (and $throw, $prompt) exit first-order flow graph</p>
</div><p>Of course, a <tt>$call</tt> expression could be a tail-call, in which case it
would continue instead to <tt>$ktail</tt>, indicating an exit from the
first-order function-local control-flow graph.</p><p>The calling convention also specifies how to pass arguments to callees,
and likewise those continuations have a fixed calling convention; in
Guile we start functions with <tt>$kfun</tt>, which has some metadata attached,
and then proceed to <tt>$kclause</tt> which bridges the boundary between the
standard calling convention and the specialized graph of <tt>$kargs</tt>
continuations. (Many details of this could be tweaked, for example that
the <tt>case-lambda</tt> dispatch built-in to <tt>$kclause</tt> could instead dispatch
to distinct functions instead of to different places in the same
function; historical accidents abound.)</p><p>As a detail, if a function is <i>well-known</i>, in that all its callers are
known, then we can lighten the calling convention, moving the
argument-count check to callees. In that case <tt>$kfun</tt> continues
directly to <tt>$kargs</tt>. Similarly for return values, optimizations can
make <tt>$call</tt> continue to <tt>$kargs</tt>, though there is still some
value-shuffling to do.</p><div>
<h2>High and low</h2>
<p>CPS bridges AST (Tree-IL) and target code</p>
<p>High-level: vars in outer functions in scope</p>
<p>Closure conversion between high and low</p>
<p>Low-level: Explicit closure representations; access free vars through closure</p>
</div><p>CPS soup is the bridge between parsed Scheme and machine code. It
starts out quite high-level, notably allowing for nested scope, in which
expressions can directly refer to free variables. Variables are small
integers, and for high-level CPS, variable indices have to be unique
across all functions in a program. CPS gets lowered via
<a href="https://wingolog.org/archives/2016/02/08/a-lambda-is-not-necessarily-a-closure">closure
conversion</a>,
which chooses specific representations for each closure that remains
after optimization. After closure conversion, all variable access is
local to the function; free variables are accessed via explicit loads
from a function's closure.</p><div>
<h2>Optimizations at all levels</h2>
<p>Optimizations before and after lowering</p>
<p>Some exprs only present in one level</p>
<p>Some high-level optimizations can merge functions (higher-order to first-order)</p>
</div><p>Because of the broad remit of CPS, the language itself has two dialects,
high and low. The high level dialect has cross-function variable
references, first-class abstract functions (whose representation hasn't
been chosen), and recursive function binding. The low-level dialect has
only specific ways to refer to functions: labels and specific closure
representations. It also includes calls to function labels instead of
just function values. But these are minor variations; some optimization
and transformation passes can work on either dialect.</p><div>
<h2>Practicalities</h2>
<p>Intmap, intset: Clojure-style persistent functional data structures</p>
<p>
Program: <tt>intmap<label,cont></tt>
</p>
<p>
Optimization: <tt>program→program</tt>
</p>
<p>
Identify functions: <tt>(program,label)→intset<label></tt>
</p>
<p>
Edges: <tt>intmap<label,intset<label>></tt>
</p>
<p>
Compute succs: <tt>(program,label)→edges</tt>
</p>
<p>
Compute preds: <tt>edges→edges</tt>
</p>
</div><p>I mentioned that programs were intmaps, and specifically in Guile they
are Clojure/Bagwell-style persistent functional data structures. By
<i>functional</i> I mean that intmaps (and intsets) are values that can't be
mutated in place (though we do have the <a href="https://clojure.org/reference/transients">transient
optimization</a>).</p><p>I find that immutability has the effect of deploying a sense of calm to
the compiler hacker -- I don't need to worry about data structures
changing out from under me; instead I just structure all the
transformations that you need to do as functions. An optimization is
just a function that takes an intmap and produces another intmap. An
analysis associating some data with each program label is just a
function that computes an intmap, given a program; that analysis will
never be invalidated by subsequent transformations, because the program
to which it applies will never be mutated.</p><p>This pervasive feeling of calm allows me to tackle problems that I
wouldn't have otherwise been able to fit into my head. One example is
the novel <a href="https://github.com/wingo/online-cse/blob/main/online-cse.md">online CSE
pass</a>; one
day I'll either wrap that up as a paper or just capitulate and blog it
instead.</p><div>
<h2>Flow analysis</h2>
<pre>A[k] = meet(A[p] for p in preds[k])
- kill[k] + gen[k]</pre>
<p>Compute available values at labels:</p>
<ul>
<li>
A: <tt>intmap<label,intset<val>></tt>
</li>
<li>
meet: <tt>intmap-intersect<intset-intersect></tt>
</li>
<li>
-, +: <tt>intset-subtract</tt>, <tt>intset-union</tt>
</li>
<li>kill[k]: values invalidated by cont because of side effects</li>
<li>gen[k]: values defined at k</li>
</ul>
</div><p>But to keep it concrete, let's take the example of flow analysis. For
example, you might want to compute "available values" at a given label:
these are the values that are candidates for common subexpression
elimination. For example if a term is dominated by a <tt>car x</tt> primcall
whose value is bound to <tt>v</tt>, and there is no path from the definition of
V to a subsequent <tt>car x</tt> primcall, we can replace that second duplicate
operation with <tt>$values (v)</tt> instead.</p><p>There is a standard solution for this problem, which is to solve the
flow equation above. I wrote about this at length <a href="https://wingolog.org/archives/2014/07/01/flow-analysis-in-guile">ages
ago</a>,
but looking back on it, the thing that pleases me is how easy it is to
decompose the task of flow analysis into manageable parts, and how the
types tell you exactly what you need to do. It's easy to compute an
initial analysis A, easy to define your meet function when your maps and
sets have built-in intersect and union operators, easy to define what
addition and subtraction mean over sets, and so on.</p><div>
<h2>Persistent data structures FTW</h2>
<ul>
<li>
meet: <tt>intmap-intersect<intset-intersect></tt>
</li>
<li>
-, +: <tt>intset-subtract</tt>, <tt>intset-union</tt>
</li>
</ul>
<p>Naïve: O(nconts * nvals)</p>
<p>Structure-sharing: O(nconts * log(nvals))</p>
</div><p>Computing an analysis isn't free, but it is manageable in cost: the
structure-sharing means that <tt>meet</tt> is usually trivial (for fallthrough
control flow) and the cost of <tt>+</tt> and <tt>-</tt> is proportional to the log of
the problem size.</p><div>
<h2>CPS soup: strengths</h2>
<p>Relatively uniform, orthogonal</p>
<p>Facilitates functional transformations and analyses, lowering mental load: “I just have to write a function from foo to bar; I can do that”</p>
<p>Encourages global optimizations</p>
<p>Some kinds of bugs prevented by construction (unintended shared mutable state)</p>
<p>We get the SSA optimization literature</p>
</div><p>Well, we're getting to the end here, and I want to take a step back.
Guile has used CPS soup as its middle-end IR for about 8 years now,
enough time to appreciate its fine points while also understanding its
weaknesses.</p><p>On the plus side, it has what to me is a kind of low cognitive overhead,
and I say that not just because I came up with it: Guile's development
team is small and not particularly well-resourced, and we can't afford
complicated things. The simplicity of CPS soup works well for our
development process (flawed though that process may be!).</p><p>I also like how by having every variable be potentially a phi, that any
optimization that we implement will be global (i.e. not local to a basic
block) by default.</p><p>Perhaps best of all, we get these benefits while also being able to use
the existing SSA transformation literature. Because CPS is SSA, the
lessons learned in SSA (e.g. loop peeling) apply directly.</p><div>
<h2>CPS soup: weaknesses</h2>
<p>Pointer-chasing, indirection through intmaps</p>
<p>Heavier than basic blocks: more control-flow edges</p>
<p>Names bound at continuation only; phi predecessors share a name</p>
<p>Over-linearizes control, relative to sea-of-nodes</p>
<p>Overhead of re-computation of analyses</p>
</div><p>CPS soup is not without its drawbacks, though. It's not suitable for
JIT compilers, because it imposes some significant constant-factor (and
sometimes algorithmic) overheads. You are always indirecting through
intmaps and intsets, and these data structures involve significant
pointer-chasing.</p><p>Also, there are some forms of lightweight flow analysis that can be
performed naturally on a graph of basic blocks without looking too much
at the contents of the blocks; for example in our available variables
analysis you could run it over blocks instead of individual
instructions. In these cases, basic blocks themselves are an
optimization, as they can reduce the size of the problem space, with
corresponding reductions in time and memory use for analyses and
transformations. Of course you could overlay a basic block graph on top
of CPS soup, but it's not a well-worn path.</p><p>There is a little detail that not all phi predecessor values have names,
since names are bound at successors (continuations). But this is a
detail; if these names are important, little <tt>$values</tt> trampolines can
be inserted.</p><p>Probably the main drawback as an IR is that the graph of conts in CPS
soup over-linearizes the program. There are <a href="https://dl.acm.org/doi/pdf/10.1145/207110.207154">other intermediate
representations</a> that
don't encode ordering constraints where there are none; perhaps it would
be useful to marry CPS soup with sea-of-nodes, at least during some
transformations.</p><p>Finally, CPS soup does not encourage a style of programming where an
analysis is incrementally kept up to date as a program is transformed in
small ways. The result is that we end up performing much redundant
computation within each individual optimization pass.</p><div>
<h2>Recap</h2>
<p>CPS soup is SSA, distilled</p>
<p>Labels and vars are small integers</p>
<p>Programs map labels to conts</p>
<p>Conts are the smallest labellable unit of code</p>
<p>Conts can have terms that continue to other conts</p>
<p>Compilation simplifies and lowers programs</p>
<p>Wasm vs VM backend: a question for another day :)</p>
</div><p>But all in all, CPS soup has been good for Guile. It's just SSA by
another name, in a simpler form, with a functional flavor. Or, it's
just CPS, but first-order only, without lambda.</p><p>In the near future, I am interested in seeing what a <a href="https://wingolog.org/archives/2023/02/07/whippet-towards-a-new-local-maximum">new
GC</a>
will do for CPS soup; will bump-pointer allocation palliate some of the
costs of pointer-chasing? We'll see. A tricky thing about CPS soup is
that I don't think that anyone else has tried it in other languages, so
it's hard to objectively understand its characteristics independent of
Guile itself.</p><p>Finally, it would be nice to engage in the academic conversation by
publishing a paper somewhere; I would like to see interesting criticism,
and blog posts don't really participate in the citation graph. But in
the <a href="https://wingolog.org/archives/2023/04/18/sticking-point">limited time available to
me</a>, faced with
the choice between hacking on something and writing a paper, it's always
been hacking, so far :)</p><p>Speaking of limited time, I probably need to hit publish on this one and
move on. Happy hacking to all, and until next time.</p></div>2023-05-20T07:10:03+00:00Andy WingoGStreamer: GStreamer 1.22.3 stable bug fix release
https://gstreamer.freedesktop.org/news/#2023-05-19T10:00:00Z
<p>
The GStreamer team is pleased to announce the second bug fix release
in the stable 1.22 release series of your favourite cross-platform
multimedia framework!
</p><p>
This release only contains bugfixes and it should be safe to update from
1.22.x.
</p><p>
<b>Highlighted bugfixes:</b>
<ul>
<li>avdec: fix occasional video decoder deadlock on seeking with FFmpeg 6.0</li>
<li>decodebin3: fix regression handling input streams without CAPS or TIME segment such as e.g. <tt>udpsrc</tt> or <tt>pushfilesrc</tt></li>
<li>bluez: a2dpsink: fix Bluetooth SIG Certification test failures</li>
<li>osxvideosink: fix deadlock upon closing output window</li>
<li>qtdemux: fix edit list handling regression and AV1 codec box parsing</li>
<li>qtmux: fix extraction of CEA608 closed caption data from S334-1A packets</li>
<li>rtspsrc: Fix handling of <tt>*</tt> control path</li>
<li>splitmux: timestamp handling improvements</li>
<li>v4l2videodec: Rework dynamic resolution change handling (needed for IMX6 mainline codec)</li>
<li>videoflip: fix regression with automatically rotating video based on tags</li>
<li>d3d11: many d3d11videosink and d3d11compositor fixes</li>
<li>webrtc, rtp: numerous data race fixes and stability fixes</li>
<li>cerbero: Add support for RHEL9 and Rocky Linux; build timecodestamper plugin with libltc support</li>
<li>various bug fixes, memory leak fixes, and other stability and reliability improvements</li>
</ul>
</p><p>
See the <a href="https://gstreamer.freedesktop.org/releases/1.22/#1.22.3">GStreamer 1.22.3 release notes</a>
for more details.
</p><p>
Binaries for Android, iOS, Mac OS X and Windows will be available shortly.
</p><p>
Release tarballs can be downloaded directly here:
<ul>
<a href="https://gstreamer.freedesktop.org/src/gstreamer/gstreamer-1.22.3.tar.xz">gstreamer</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-base/gst-plugins-base-1.22.3.tar.xz">gst-plugins-base</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-good/gst-plugins-good-1.22.3.tar.xz">gst-plugins-good</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-ugly/gst-plugins-ugly-1.22.3.tar.xz">gst-plugins-ugly</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-bad/gst-plugins-bad-1.22.3.tar.xz">gst-plugins-bad</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-libav/gst-libav-1.22.3.tar.xz">gst-libav</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-rtsp-server/gst-rtsp-server-1.22.3.tar.xz">gst-rtsp-server</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-python/gst-python-1.22.3.tar.xz">gst-python</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-editing-services/gst-editing-services-1.22.3.tar.xz">gst-editing-services</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-devtools/gst-devtools-1.22.3.tar.xz">gst-devtools</a>,
<a href="https://gstreamer.freedesktop.org/src/gstreamer-vaapi/gstreamer-vaapi-1.22.3.tar.xz">gstreamer-vaapi</a>,
<a href="https://gstreamer.freedesktop.org/src/gstreamer-sharp/gstreamer-sharp-1.22.3.tar.xz">gstreamer-sharp</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-omx/gst-omx-1.22.3.tar.xz">gst-omx</a>, or
<a href="https://gstreamer.freedesktop.org/src/gstreamer-docs/gstreamer-docs-1.22.3.tar.xz">gstreamer-docs</a>.
</ul>
</p>2023-05-19T10:00:00+00:00Andy Wingo: structure and interpretation of ark
https://wingolog.org/archives/2023/05/02/structure-and-interpretation-of-ark
<div><p>Hello, dear readers! Today's article describes Ark, a new
JavaScript-based mobile development platform. If you haven't read them
yet, you might want to start by having a look at my past articles on
<a href="https://wingolog.org/archives/2023/04/20/structure-and-interpretation-of-capacitor-programs">Capacitor</a>,
<a href="https://wingolog.org/archives/2023/04/21/structure-and-interpretation-of-react-native">React
Native</a>,
<a href="https://wingolog.org/archives/2023/04/24/structure-and-interpretation-of-nativescript">NativeScript</a>,
and
<a href="https://wingolog.org/archives/2023/04/26/structure-and-interpretation-of-flutter">Flutter</a>;
having a common understanding of the design space will help us
understand where Ark is similar and where it differs.</p><h3>Ark, what it is</h3><p>If I had to bet, I would guess that you have not heard of Ark. (I
certainly hadn't either, when commissioned to do this research series.)
To a first approximation, Ark—or rather, what I am calling Ark; I don't
actually know the name for the whole architecture—is a loosely
Flutter-like UI library implemented on top of a dialect of JavaScript,
with build-time compilation to bytecode (like Hermes) but also with
support for just-in-time and ahead-of-time compilation of bytecode to
native code. It is made by Huawei.</p><p>At this point if you are already interested in this research series, I
am sure this description raises more questions than it answers.
Flutter-like? A dialect? Native compilation? Targetting what
platforms? <i>From Huawei?</i> We'll get to all of these, but I think we
need to start with the last question.</p><h3>How did we get here?</h3><p>In my last article on Flutter, I told a kind of just-so history of how
Dart and Flutter came to their point in the design space. Thanks to
corrections from a kind reader, it happened to also be more or less
correct. In this article, though, I haven't talked with Ark developers
at all; I don't have the benefit of a true claim on history. And yet,
the only way I can understand Ark is by inventing a narrative, so here
we go. It might even be true!</p><p>Recall that in 2018, Huawei was a dominant presence in the smartphone
market. They were shipping excellent hardware at good prices both to
the Chinese and to the global markets. Like most non-Apple, non-Google
manufacturers, they shipped Android, and like most Android OEMs, they
shipped <a href="https://en.wikipedia.org/wiki/Google_Mobile_Services">Google's proprietary apps (mail, maps,
etc.)</a>.</p><p>But then, over the next couple years, the US decided that allowing
Huawei to continue on as before was, like, against national security
interests or something. Huawei was barred from American markets, a
number of suppliers were forbidden from selling hardware components to
Huawei, and even Google was prohibited from shipping its mobile apps on
Huawei devices. The effect on Huawei's market share for mobile devices
was enormous: its revenue was cut in half over a period of a couple
years.</p><p>In this position, as Huawei, what do you do? I can't even imagine, but
specifically looking at smartphones, I think I would probably do about
what they did. I'd fork Android, for starters, because that's what you
already know and ship, and Android is mostly open source. I'd probably
plan on continuing to use its lower-level operating system pieces
indefinitely (kernel and so on) because that's not a value
differentiator. I'd probably ship the same apps on top at first,
because otherwise you slip all the release schedules and lose revenue
entirely.</p><p>But, gosh, there is the risk that your product will be perceived as just
a worse version of Android: that's not a good position to be in. You
need to be different, and ideally better. That will take time. In the
meantime, you <i>claim</i> that you're different, without actually being
different yet. It's a somewhat ridiculous position to be in, but I can
understand how you get here; <a href="https://arstechnica.com/gadgets/2021/02/harmonyos-hands-on-huaweis-android-killer-is-just-android/">Ars Technica published a scathing
review</a>
poking fun at the situation. But, you are big enough to ride it out,
knowing that somehow eventually you will be different.</p><p>Up to now, this part of the story is relatively well-known; the part
that follows is more speculative on my part. Firstly, I would note that
Huawei had been working for a while on a compiler and language run-time
called <a href="https://www.huaweicentral.com/ark-compiler-huaweis-self-developed-android-application-compiler-explained/">Ark
Compiler</a>,
with the goal of getting better performance out of Android applications.
If I understand correctly, this compiler took the Java / Dalvik /
Android Run Time bytecodes as its input, and outputted native binaries
along with a new run-time implementation.</p><p>As I can attest from personal experience, having a compiler leads to
hubris: you start to consider source languages like a hungry person
looks at a restaurant menu. "Wouldn't it be nice to ingest that?"
That's what we say at restaurants, right, fellow humans? So in 2019 and
2020 when the Android rug was pulled out from underneath Huawei, I think
having in-house compiler expertise allowed them to consider whether they
wanted to stick with Java at all, or whether it might be better to
choose a more fashionable language.</p><p>Like black, JavaScript is always in fashion. What would it mean,
then, to retool Huawei's operating system -- by then known by the name
"HarmonyOS" -- to expose a JavaScript-based API as its primary app
development framework? You could use your Ark compiler somehow to
implement JavaScript (hubris!) and then you need a UI framework. Having
ditched Java, it is now thinkable to ditch all the other Android
standard libraries, including the UI toolkit: you start anew, in a way.
So are you going to build a Capacitor, a React Native, a NativeScript, a
Flutter? Surely not precisely any of these, but what will it be like,
and how will it differ?</p><p>Incidentally, I don't know the origin story for the name Ark, but to me
it brings to mind tragedy and rebuilding: in the midst of being cut off
from your rich Android ecosystem, you launch a boat into the sea,
holding a promise of a new future built differently. Hope and hubris in
one vessel.</p><h3>Two programming interfaces</h3><p>In the end, Huawei builds two things: something web-like and something
like Flutter. (I don't mean to suggest copying or degeneracy here; it's
rather that I can only understand things in relation to other things,
and these are my closest points of comparison for what they built.)</p><p>The web-like programming interface specifies UIs using an XML dialect,
<a href="https://developer.harmonyos.com/en/docs/documentation/doc-guides-V3/js-framework-syntax-hml-0000001477981005-V3">HML</a>,
and styles the resulting node tree with CSS. You augment these nodes
with JavaScript behavior; the main app is a set of <a href="https://developer.harmonyos.com/en/docs/documentation/doc-guides-V3/js-framework-syntax-js-0000001428061552-V3">DOM-like event
handlers</a>.
There is an API to <a href="https://developer.harmonyos.com/en/docs/documentation/doc-references-V3/js-components-create-elements-0000001478181509-V3?catalogVersion=V3">dynamically create DOM
nodes</a>,
but unlike the other systems we have examined, the HarmonyOS
documentation doesn't really sell you on using a high-level framework
like Angular.</p><p>If this were it, I think Ark would not be so compelling: the programming
model is more like what was available back in the <a href="https://en.wikipedia.org/wiki/Dynamic_HTML">DHTML
days</a>. I wouldn't expect
people to be able to make rich applications that delight users, given
these primitives, though CSS animation and the HML <a href="https://developer.harmonyos.com/en/docs/documentation/doc-references-V3/js-components-container-list-0000001427584924-V3#EN-US_TOPIC_0000001544695117__example">loop and conditional
rendering</a>
from the template system might be just expressive enough for simple
applications.</p><p>The more interesting side is the so-called "declarative" UI programming
model which exposes a Flutter/React-like interface. The programmer
describes the "what" of the UI by providing a tree of UI nodes in its
<tt>build</tt> function, and the framework takes care of calling <tt>build</tt> when
necessary and of rendering that tree to the screen.</p><p>Here I need to show some example code, because it is... weird. Well, I
find it weird, but it's not too far from
<a href="https://developer.apple.com/xcode/swiftui/">SwiftUI</a> in flavor. A
small example from <a href="https://developer.harmonyos.com/en/docs/documentation/doc-guides-V3/ui-ts-creating-simple-page-0000001493575800-V3">the fine
manual</a>:</p><pre class="pre-js">@Entry
@Component
struct MyComponent {
build() {
Stack() {
Image($rawfile('Tomato.png'))
Text('Tomato')
.fontSize(26)
.fontWeight(500)
}
}
}
</pre><p>The <tt>@Entry</tt> decorator (*) marks this <tt>struct</tt> (**) as being the main
entry point for the app. <tt>@Component</tt> marks it as being a component,
like a React <i>functional component</i>. Components conform to an interface
(***) which defines them as having a <tt>build</tt> method which takes no
arguments and returns no values: it creates the tree in a somewhat
imperative way.</p><p>But as you see the flavor is somewhat declarative, so how does that
work? Also, <tt>build() { ... }</tt> looks syntactically a lot like <tt>Stack() { ... }</tt>; what's the deal, are they the same?</p><p>Before going on to answer this, note my asterisks above: these are
concepts that aren't in JavaScript. Indeed, programs written for
HarmonyOS's declarative framework aren't JavaScript; they are in a
dialect of TypeScript that Huawei calls ArkTS. In this case, an
<tt>interface</tt> is <a href="https://www.typescriptlang.org/docs/handbook/interfaces.html">a TypeScript
concept</a>.
Decorators would appear to correspond to <a href="https://www.typescriptlang.org/docs/handbook/decorators.html">an experimental TypeScript
feature</a>,
looking at the source code.</p><p>But <tt>struct</tt> is an <a href="https://gitee.com/openharmony/third_party_typescript#changes">ArkTS-specific
extension</a>,
and Huawei has actually extended the TypeScript compiler to specifically
recognize the <tt>@Component</tt> decorator, such that when you "call" a
struct, for example as above in <tt>Stack() { ... }</tt>, TypeScript will parse
that as a new expression type
<a href="https://gitee.com/openharmony/third_party_typescript/blob/master/src/compiler/types.ts#L2065"><tt>EtsComponentExpression</tt></a>,
which may optionally be followed by a block. When <tt>Stack()</tt> is invoked,
its children (instances of <tt>Image</tt> and <tt>Text</tt>, in this case) will be
populated via running the block.</p><p>Now, though TypeScript isn't everyone's bag, it's quite normalized in the
JavaScript community and not a hard sell. Language extensions like the handling of <tt>@Component</tt>
pose a more challenging problem. Still, Facebook managed to sell people
on <a href="https://legacy.reactjs.org/docs/introducing-jsx.html">JSX</a>, so perhaps Huawei can do the same for their dialect. More on
that later.</p><p>Under the hood, it would seem that we have a similar architecture to
Flutter: invoking the <a href="https://gitee.com/openharmony/arkui_ace_engine/blob/master/frameworks/core/pipeline/base/component.h">components</a> creates a corresponding tree of
<a href="https://gitee.com/openharmony/arkui_ace_engine/blob/master/frameworks/core/pipeline/base/element.h"><i>elements</i></a> (as with React Native's shadow tree), which then are lowered
to <a href="https://gitee.com/openharmony/arkui_ace_engine/blob/master/frameworks/core/pipeline/base/render_node.h"><i>render nodes</i></a>, which draw themselves onto layers using Skia, in a
multi-threaded <a href="https://gitee.com/openharmony/arkui_ace_engine/blob/master/frameworks/core/pipeline/pipeline_context.h">rendering pipeline</a>. Underneath, the UI code actually
re-uses some parts of Flutter, though from what I can tell
HarmonyOS developers are replacing those over time.</p><h3>Restrictions and extensions</h3><p>So we see that the source language for the declarative UI framework is
TypeScript, but with some extensions. It also has its restrictions, and
to explain these, we have to talk about implementation.</p><p>Of the JavaScript mobile application development frameworks we
discussed, Capacitor and NativeScript used "normal" JS engines from web browsers, while
React Native built their own Hermes implementation. Hermes is also
restricted, in a way, but mostly inasmuch as it lags the browser JS
implementations; it relies on source-to-source transpilers to get access
to new language features. ArkTS—that's the name of HarmonyOS's
"extended TypeScript" implementation—has more fundamental restrictions.</p><p>Recall that the Ark compiler was originally built for Android apps.
There you don't really have the ability to load new Java or Kotlin
source code at run-time; in Java you have class loaders, but those load
bytecode. On an Android device, you don't have to deal with the Java
source language. If we use a similar architecture for JavaScript,
though, what do we do about <tt>eval</tt>?</p><p>ArkTS's answer is: don't. As in, <tt>eval</tt> is not supported on HarmonyOS.
In this way the implementation of ArkTS can be divided into two parts, a
frontend that produces bytecode and a runtime that runs the bytecode,
and you never have to deal with the source language on the device where
the runtime is running. Like Hermes, the developer produces bytecode
when building the application and ships it to the device for the runtime
to handle.</p><p>Incidentally, before we move on to discuss the runtime, there are
actually two front-ends that generate ArkTS bytecode: <a href="https://gitee.com/openharmony/arkcompiler_ets_frontend/tree/master/es2panda">one written in
C++ that seems to only handle standard TypeScript and
JavaScript</a>,
and <a href="https://gitee.com/openharmony/arkcompiler_ets_frontend/tree/master/ts2panda">one written in TypeScript that also handles "extended
TypeScript"</a>.
The former has a <a href="https://gitee.com/openharmony/arkcompiler_ets_frontend/blob/master/es2panda/test/test262skiplist.txt">test262 runner with about 10k skipped
tests</a>,
and the latter doesn't appear to have a test262 runner. Note, I haven't
actually built either one of these (or any of the other frameworks, for
that matter).</p><p>The <a href="https://gitee.com/openharmony/arkcompiler_ets_runtime">ArkTS
runtime</a> is
itself built on a non-language-specific <a href="https://gitee.com/openharmony/arkcompiler_runtime_core">common Ark
runtime</a>, and
the set of supported instructions is the union of the <a href="https://gitee.com/openharmony/arkcompiler_runtime_core/blob/master/isa/isa.yaml">core
ISA</a>
and the <a href="https://gitee.com/openharmony/arkcompiler_ets_runtime/blob/master/ecmascript/ecma_isa.yaml">JavaScript-specific
instructions</a>.
Bytecode can be
<a href="https://gitee.com/openharmony/arkcompiler_ets_runtime/blob/master/ecmascript/interpreter/interpreter-inl.h">interpreted</a>,
JIT-compiled, or AOT-compiled.</p><p>On the side of design documentation, it's somewhat sparse. There are
some <a href="https://gitee.com/openharmony/arkcompiler_runtime_core/tree/master/compiler/docs">core design
docs</a>;
readers may be interested in the <a href="https://gitee.com/openharmony/arkcompiler_runtime_core/blob/master/docs/rationale-for-bytecode.md">rationale to use a bytecode
interface</a>
for Ark as a whole, or the <a href="https://gitee.com/openharmony/arkcompiler_runtime_core/blob/master/docs/ir_format.md">optimization
overview</a>.</p><p>Indeed ArkTS as a whole has a surfeit of optimizations, to an extent
that makes me wonder which ones are actually needed. There are
<a href="https://gitee.com/openharmony/arkcompiler_runtime_core/tree/master/bytecode_optimizer">source-to-source optimizations on
bytecode</a>,
which I expect are useful if you are generating ArkTS bytecode from
JavaScript, where you probably don't have a full compiler
implementation. There is a <a href="https://gitee.com/openharmony/arkcompiler_ets_runtime/blob/master/ecmascript/compiler/pass.h">completely separate
optimizer</a>
in the eTS part of the run-time, based on what would appear to be a
novel <a href="https://gitee.com/openharmony/arkcompiler_ets_runtime/blob/master/ecmascript/compiler/circuit_ir_specification.md">"circuit-based"
IR</a>
that bears some similarity to sea-of-nodes. Finally <a href="https://gitee.com/openharmony/arkcompiler_ets_runtime/blob/master/ecmascript/compiler/llvm_ir_builder.h">the whole thing
appears to bottom out in
LLVM</a>,
which of course has its own optimizer. I can only assume that this
situation is somewhat transitory. Also, ArkTS does appear to generate
its own native code sometimes, notably for inline cache stubs.</p><p>Of course, when it comes to JavaScript, there are some fundamental
language semantics and there is also a large and growing standard
library. In the case of ArkTS, this standard library is <a href="https://gitee.com/openharmony/arkcompiler_ets_runtime/tree/master/ecmascript">part of the
run-time</a>,
like the interpreter, compilers, and the garbage collector
(<a href="https://gitee.com/openharmony/arkcompiler_ets_runtime/tree/master/ecmascript/mem">generational concurrent mark-sweep with optional
compaction</a>).</p><p>All in all, when I step back from it, it's a huge undertaking.
Implementing JavaScript is no joke. It appears that ArkTS has done the
first 90% though; the proverbial second 90% should only take a few more
years :)</p><h3>Evaluation</h3><p>If you told a younger me that a major smartphone vendor switched from
Java to JavaScript for their UI, you would probably hear me react in
terms of the relative virtues of the programming languages in question.
At this point in my career, though, the only thing that comes to mind is
what an <i>expensive</i> proposition it is to change everything about an
application development framework. 200 people over 5 years would be my
estimate, though of course teams are variable. So what is it that we
can imagine that Huawei bought with a thousand person-years of
investment? Towards what other local maximum are we heading?</p><h3>Startup latency</h3><p>I didn't mention it before, but it would seem that one of the goals of
HarmonyOS is in the name: Huawei wants to harmonize development across
the different range of deployment targets. To the extent possible, it
would be nice to be able to write the same kinds of programs for IoT
devices as you do for feature-rich smartphones and tablets and the like.
In that regard one can see through all the source code how there is a
culture of doing work ahead-of-time and preventing work at run-time; for
example see the <a href="https://gitee.com/openharmony/arkcompiler_runtime_core/blob/master/docs/design-of-interpreter.md">design doc for the
interpreter</a>,
or for the <a href="https://gitee.com/openharmony/arkcompiler_runtime_core/blob/master/docs/file_format.md">file
format</a>,
or indeed the lack of JavaScript <tt>eval</tt>.</p><p>Of course, this wide range of targets also means that the HarmonyOS
platform bears the burden of a high degree of abstraction; not only can
you change the kernel, but also the JavaScript engine (using
<a href="https://jerryscript.net/">JerryScript</a> on "lite" targets).</p><p>I mention this background because sometimes in news articles and indeed
official communication from recent years there would seem to be some
confusion that HarmonyOS is just for IoT, or aimed to be super-small, or
something. In this evaluation I am mostly focussed on the feature-rich
side of things, and there my understanding is that the developer will
generate bytecode ahead-of-time. When an app is installed on-device,
the AOT compiler will turn it into a single ELF image. This should
generally lead to fast start-up.</p><p>However it would seem that <a href="https://gitee.com/openharmony/arkui_ace_engine/tree/master/frameworks/core/pipeline/base">the rendering
library</a>
that paints UI nodes into layers and then composits those layers uses
Skia in the way that Flutter did pre-Impeller, which to be fair is a
quite recent change to Flutter. I expect therefore that Ark (ArkTS +
ArkUI) applications also experience shader compilation jank at startup,
and that they may be well-served by tesellating their shapes into
primitives like Impeller does so that they can precompile a fixed,
smaller set of shaders.</p><h3>Jank</h3><p>Maybe it's just that apparently I think Flutter is great, but ArkUI's
fundamental architectural similarity to Flutter makes me think that jank
will not be a big issue. There is a render thread that is separate from
the ArkTS thread, so like with Flutter, async communication with
main-thread interfaces is the main jank worry. And on the ArkTS side,
ArkTS even has a number of extensions to be able to share objects
between threads without copying, should that be needed. I am not sure
how well-developed and well-supported these extensions are, though.</p><p>I am hedging my words, of course, because I am missing a bit of social
proof; HarmonyOS is still in infant days, and it doesn't have much in
the way of a user base outside China, from what I can tell, and my
ability to read Chinese is limited to what Google Translate can do for
me :) Unlike other frameworks, therefore, I haven't been as able to
catch a feel of the pulse of the ArkUI user community: what people are
happy about, what the pain points are.</p><p>It's also interesting that unlike iOS or Android, HarmonyOS is <i>only</i>
exposing these "web-like" and "declarative" UI frameworks for app
development. This makes it so that the same organization is responsible
for the software from top to bottom, which can lead to interesting
cross-cutting optimizations: functional reactive programming isn't just
a developer-experience narrative, but it can directly affect the shape
of the rendering pipeline. If there is jank, someone in the building is
responsible for it and should be able to fix it, whether it is in the
GPU driver, the kernel, the ArkTS compiler, or the application itself.</p><h3>Peak performance</h3><p>I don't know how to evaluate ArkTS for peak performance. Although there
is a JIT compiler, I don't have the feeling that it is as tuned for
adaptive optimization as V8 is.</p><p>At the same time, I find it interesting that HarmonyOS has chosen to
modify JavaScript. While it is doing that, could they switch to a sound
type system, to allow the kinds of AOT optimizations that Dart can do?
It would be an interesting experiment.</p><p>As it is, though, if I had to guess, I would say that ArkTS is
well-positioned for predictably good performance with AOT compilation,
although I would be interested in seeing the results of actually running
it.</p><h3>Aside: On the importance of storytelling</h3><p>In this series I have tried to be charitable towards the frameworks that
I review, to give credit to what they are trying to do, even while
noting where they aren't currently there yet. That's part of why I need
a plausible narrative for how the frameworks got where they are, because
that lets me have an idea of where they are going.</p><p>In that sense I think that Ark is at an interesting inflection point.
When I started reading documentation about ArkUI and HarmonyOS and all
that, I bounced out—there were too many architectural box
diagrams, too many generic descriptions of components, too many promises
with buzzwords. It felt to me like the project was trying to justify
itself to a kind of clueless management chain. Was there actually
anything here?</p><p>But now when I see the adoption of a modern rendering architecture and a
bold new implementation of JavaScript, along with the willingness to
experiment with the language, I think that there is an interesting story
to be told, but this time not to management but to app developers.</p><p>Of course you wouldn't want to market to app developers when your
system is still a mess because you haven't finished rebuilding an MVP
yet. Retaking my charitable approach, then, I can only think that all
the architectural box diagrams were a clever blind to avoid piquing outside
interest while the app development kit wasn't ready
yet :) As and when the system starts working well, presumably over the
next year or so, I would expect HarmonyOS to invest much more heavily in
marketing and developer advocacy; the story is interesting, but you have
to actually tell it.</p><h3>Aside: O platform, my platform</h3><p>All of the previous app development frameworks that we looked at were
cross-platform; Ark is not. It could be, of course: it does appear to
be thoroughly open source. But HarmonyOS devices are the main target.
What implications does this have?</p><p>A similar question arises in perhaps a more concrete way if we start
with the mature Flutter framework: what would it mean to make a Flutter
phone?</p><p>The first thought that comes to mind is that having a Flutter OS would
allow for the potential for more cross-cutting optimizations that cross
abstraction layers. But then I think, what does Flutter really need?
It has the GPU drivers, and we aren't going to re-implement those. It
has the bridge to the platform-native SDK, which is not such a large and
important part of the app. You get input from the platform, but that's
also not so specific. So maybe optimization is not the answer.</p><p>On the other hand, a Flutter OS would not have to solve the
make-it-look-native problem; because there would be no other "native"
toolkit, your apps won't look out of place. That's nice. It's not
something that would make the <i>platform</i> compelling, though.</p><p>HarmonyOS does have this embryonic concept of app mobility, where like
you could put an app from your phone on your fridge, or something.
Clearly I am not doing it justice here, but let's assume it's a
compelling use case. In that situation it would be nice for all devices
to present similar abstractions, so you could somehow install the same
app on two different kinds of devices, and they could communicate to
transfer data. As you can see here though, I am straying far from my
domain of expertise.</p><p>One reasonable way to "move" an app is to have it stay running on your
phone and the phone just communicates pixels with your fridge (or
whatever); this is the low-level solution. I think HarmonyOS appears to
be going for the higher-level solution where the app actually runs logic
on the device. In that case it would make sense to ship UI assets and
JavaScript / extended TypeScript bytecode to the device, which would run
the app with an interpreter (for low-powered devices) or use JIT/AOT
compilation. The Ark runtime itself would live on all devices,
specialized to their capabilities.</p><p>In a way this is the Apple WatchOS solution (as I understand it);
developers publish their apps as LLVM bitcode, and Apple compiles it for
the specific devices. A FlutterOS with a Flutter run-time on all
devices could do something similar. As with WatchOS you wouldn't have
to ship the framework itself in the app bundle; it would be on the
device already.</p><p>Finally, publishing apps as some kind of intermediate representation
also has security benefits: as the OS developer, you can ensure some
invariants via the toolchain that you control. Of course, you would have to ensure
that the Flutter API is sufficiently expressive for high-performance
applications, while also not having arbitrary machine code execution
vulnerabilities; there is a question of language and framework design as
well as toolchain and runtime quality of implementation. HarmonyOS
could be headed in this direction.</p><h3>Conclusion</h3><p>Ark is a fascinating effort that holds much promise. It's also still in
motion; where will it be when it anneals to its own local optimum? It
would appear that the system is approaching usability, but I expect a
degree of churn in the near-term as Ark designers decide which language
abstractions work for them and how to, well, <i>harmonize</i> them with the
rest of JavaScript.</p><p>For me, the biggest open question is whether developers will love Ark in
the way they love, say, React. In a market where Huawei is still a
dominant vendor, I think the material conditions are there for a good
developer experience: people tend to like Flutter and React, and Ark is
similar. Huawei "just" needs to explain their framework well (and where
it's hard to explain, to go back and change it so that it is
explainable).</p><p>But in a more heterogeneous market, to succeed Ark would need to make a
cross-platform runtime like the one Flutter has and engage in some
serious marketing efforts, so that developers don't have to limit
themselves to targetting the currently-marginal HarmonyOS. Selling
extensions to JavaScript will be much more difficult in a context where
the competition is already established, but perhaps Ark will be able to
productively engage with TypeScript maintainers to move the language so it
captures some of the benefits of Dart that facilitate ahead-of-time
compilation.</p><p>Well, that's it for my review round-up; hope you have enjoyed the
series. I have one more pending article, speculating about some future
technologies. Until then, happy hacking, and see you next time.</p></div>2023-05-02T09:13:41+00:00Andy WingoStéphane Cerveau: ESExtractor: how to integrate a dependency-free library to the Khronos CTS
https://dabrain34.github.io/jekyll/update/2023/04/27/esextractor.html
<h1 id="esextractor-how-to-integrate-a-dependency-free-library-to-the-khronos-cts">ESExtractor, how to integrate a dependency-free library to the Khronos CTS</h1>
<p>Since the <a href="https://github.com/KhronosGroup/VK-GL-CTS">Vulkan CTS</a> is now able to test and check <a href="https://www.khronos.org/news/press/vulkan-sdk-is-vulkan-video-ready">Vulkan Video support</a>
including video decoding, it was necessary to define the kind of media container to be used inside the test cases and the library
to extract the necessary encoded data.</p>
<p>In a first attempt, the <a href="https://ffmpeg.org/">FFMpeg media toolkit</a> had been chosen to extract the video packets from the A/V ISO base media
format chosen as a container reference. This library was provided as a binary package and loaded dynamically at each
test run.</p>
<p>As Vulkan video aims to test only <em>video</em> contents, it was not necessary to choose a complex media container,
so first all the videos were converted to the elementary stream format for
H264 and H265 contents.
This is a very elementary format based on MPEG start codes and NAL unit identification.</p>
<p>To avoid an extra multimedia solution integrable only with binaries, a first attempt to replace FFmpeg was,
to use GStreamer and an in-house helper library called <a href="https://github.com/Igalia/GstVkVideoParser/tree/main/lib/demuxeres">demuxeres</a>.
It was smaller but needed to be a binary still to avoid the glib/gstreamer system dependencies (self contained library).
it was a no-go still because the binary package would be awkward to support on the various platforms targetted by the the Khronos CTS.</p>
<p>So at <a href="https://www.igalia.com/">Igalia</a>, we decided to implement a minimal, dependency-free, custom library, written in C++
to be compliant with the Khronos CTS and simple to integrate into any build system.</p>
<p>This library is called <a href="https://github.com/Igalia/ESExtractor">ESExtractor</a></p>
<h2 id="what-is-esextractor-">What is ESExtractor ?</h2>
<p>ESExtractor aims to be a simple elementary stream extractor. For the first revision it was able to extract video data from
a file in the <a href="https://en.wikipedia.org/wiki/Network_Abstraction_Layer">NAL standard</a>.
The first official release was <a href="https://github.com/Igalia/ESExtractor/releases/tag/release-v0.2.4">0.2.4</a>. In this release,
only the NAL was supported with both the H264 and H265 streams supported.</p>
<p>As Vulkan Video aims to support more than H264 and H265 including format such as AV1 or VP9, the ESExtractor had to support multiple format.
A redesign has been started to support multiple format and is now available in the version 0.3.2.</p>
<h2 id="how-esextractor-works">How ESExtractor works</h2>
<p>A simple C interface is provided to maximise portability and use by other languages.</p>
<h3 id="es_extractor_new">es_extractor_new</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ESExtractor extractor = es_extractor_new(filePath, "options"));
</code></pre></div></div>
<p>This interface returns the main object which will give you access to the packets according to the file path and the options given in the arguments.
This interface returns a ready to use object where the stream has been initially parsed to determine the kind of video during
the object creation.</p>
<p>Then you can check the video format with:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ESEVideoFormat eVideoFormat = es_extractor_video_format(extractor);
</code></pre></div></div>
<p>or the video codec with:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ESEVideoCodec eVideoCodec = es_extractor_video_codec(extractor);
</code></pre></div></div>
<p>It supports H264, H265, AV1 and VP9 for now.</p>
<h3 id="es_extractor_read_packet">es_extractor_read_packet</h3>
<p>This API is the main function to retrieve the available packets from the file. Each time this API is called,
the library will return the next available packet according to the format and the specific alignment (ie NAL) and
a status to let the application decide what to do next. The packet should be freed using <code class="language-plaintext highlighter-rouge">es_extractor_clear_packet</code>.</p>
<h2 id="has-ci-powered-by-github">Has CI powered by github</h2>
<p>To test the library usage, we have implementing a testing framework in addition to a CI infrastructure
As github offers a very powerful worklow, we decided to use this platform to test the library on various architectures and platforms.
The <a href="https://github.com/Igalia/ESExtractor/actions">CI</a> is now configured to release packages for 64 and 32 bits on Linux and Windows.</p>
<p>As usual, if you would like to learn more about Vulkan Video, ESExtractor or any other open multimedia framework, please contact <a href="https://www.igalia.com/">us</a>!</p>2023-04-27T10:10:45+00:00Andy Wingo: structure and interpretation of flutter
https://wingolog.org/archives/2023/04/26/structure-and-interpretation-of-flutter
<div><p>Good day, gentle hackfolk. Like an
<a href="https://en.wikipedia.org/wiki/Old-time_music">old-time</a> fiddler I would
appear to be deep in the groove, playing endless variations on a theme,
in this case mobile application frameworks. But one can only recognize
novelty in relation to the familiar, and today's note is a departure: we
are going to look at <a href="https://flutter.dev/">Flutter</a>, a UI toolkit based
not on JavaScript but on the <a href="https://dart.dev/">Dart</a> language.</p><h3>The present, from the past</h3><p>Where to start, even? The problem is big enough that I'll
approach it from three different angles: from the past, from the top,
and from the bottom.</p><p>With the other frameworks we looked at, we didn't have to say much about
their use of JavaScript. JavaScript is an obvious choice, in 2023 at
least: it is ubiquitous, has high quality implementations, and as a
language it is quite OK and progressively getting better. Up to now,
"always bet on JS" has had an uninterrupted winning streak.</p><p>But winning is not the same as unanimity, and Flutter and Dart represent
an interesting pole of contestation. To understand how we got here, we
have to go back in time. Ten years ago, JavaScript just wasn't a great
language: there were no modules, no async functions, no destructuring,
no classes, no extensible iteration, no optional arguments to functions.
In addition it was hobbled with a significant degree of what can only be
called accidental sloppiness: <tt>with</tt> which can dynamically alter a
lexical scope, direct <tt>eval</tt> that can define new local variables,
<tt>Function.caller</tt>, and so on. Finally, larger teams were starting to
feel the need for more rigorous language tooling that could use types to
prohibit some classes of invalid programs.</p><p>All of these problems in JavaScript have been addressed over the last
decade, mostly successfully. But in 2010 or so if you were a virtual
machine engineer, you might look at JavaScript and think that in some
other world, things could be a lot better. That's effectively what
happened: the team that originally built V8 broke off and started to
work on what became Dart.</p><p>Initially, Dart was targetted for inclusion in the Chrome web browser as
an alternate "native" browser language. This didn't work, for various
reasons, but since then Dart grew the Flutter UI toolkit, which has
breathed new life into the language. And this is a review of Flutter,
not a review of Dart, not really anyway; to my eyes, Dart is spiritually
another JavaScript, different but in the same family. Dart's
implementation has many interesting aspects as well that we'll get into
later on, but all of these differences are incidental: they could just
as well be implemented on top of JavaScript, TypeScript, or another
source language in that family. Even if Flutter isn't strictly part of
the JavaScript-based mobile application development frameworks that we
are comparing, it is valuable to the extent that it shows what is
possible, and in that regard there is much to say.</p><h3>Flutter, from the top</h3><p>At its most basic, Flutter is a UI toolkit for Dart. In many ways it is
like React. Like React, its interface follows the functional-reactive
paradigm: programmers describe the "what", and Flutter takes care of the
"how". Also, like the phenomenon in which new developers can learn
React without really knowing JavaScript, Flutter is the killer app for
Dart: Flutter developers mostly learn Dart at the same time that they
pick up Flutter.</p><p>In some other ways, Flutter is the logical progression of React, going
in the same direction but farther along. Whereas React-on-the-web takes
the user's declarative specifications of what the UI should look like
and lowers them into DOM trees, and <a href="https://wingolog.org/archives/2023/04/21/structure-and-interpretation-of-react-native">React Native lowers them to
platform-native UI
widgets</a>,
Flutter has its own built-in layout, rasterization, and compositing
engine: Flutter draws all the pixels.</p><p>This has the predictable challenge that Flutter has to make significant
investments so that its applications don't feel out-of-place on their
platform, but on the other hand it opens up a huge space for
experimentation and potential optimization: Flutter has the potential to
beat native at its own game. Recall that with React Native, the result
of the <a href="https://reactnative.dev/architecture/render-pipeline">render-commit-mount
process</a> is a tree
of native widgets. The native platform will surely then perform a kind
of layout on those widgets, divide them into layers that correspond to
GPU textures, paint those layers, then composite them to the screen --
basically, <a href="https://developer.chrome.com/articles/layoutng/">what a web engine will
do</a>.</p><p>What if we could instead skip the native tree and go directly to the
lower GPU layer? That is the promise of Flutter. Flutter has the
potential to create much more smooth and complex animations than the
other application development frameworks we have mentioned, with lower
overhead and energy consumption.</p><p>In practice... that's always the question, isn't it? Again, please
accept my usual caveat that I am a compilers guy moonlighting in the
user interface domain, but my understanding is that Flutter mostly lives
up to its promise, but with one significant qualification which we'll
get to in a minute. But before that, let's traverse Flutter from the
other direction, coming up from Dart.</p><h3>Dart, from the bottom</h3><p>To explain some aspects of Dart I'm going to tell a just-so story that
may or may not be true. I know and like many of the Dart developers,
and we have similar instincts, so it's probably not too far from the
truth.</p><p>Let's say you are the team that originally developed V8, and you decide
to create a new language. You write a new virtual machine that looks
like V8, taking Dart source code as input and applying advanced adaptive
compilation techniques to get good performance. You can even be faster
than JS because your language is just a bit more rigid than JavaScript
is: you have traded off expressivity for performance. (Recall from <a href="https://wingolog.org/archives/2023/04/24/structure-and-interpretation-of-nativescript">our
discussion of
NativeScript</a>
that expressivity isn't a value judgment: there can be reasons to pay
for more "mathematically appealing operational equivalences", in
Felleisen's language, in exchange for applying more constraints on a
language.)</p><p>But, you fail to <a href="https://lists.webkit.org/pipermail/webkit-dev/2011-December/018775.html">ship the VM in a
browser</a>;
what do you do? The project could die; that would be annoying, but you
work for Google, so it happens all the time. However, a few interesting
things happen around the same time that will cause you to pivot. One is
a concurrent experiment by Chrome developers to pare the web platform
down to its foundations and rebuild it. This effort will eventually
become Flutter; while it was originally <a href="https://github.com/flutter/flutter/commit/00882d626a478a3ce391b736234a768b762c853a">based on
JS</a>,
eventually they will choose to <a href="https://bugs.chromium.org/p/chromium/issues/detail?id=454613">switch to
Dart</a>.</p><p>The second thing that happens is that recently-invented smart phones
become ubiquitous. Most people have one, and the two platforms are iOS
and Android. Flutter wants to target them. You are looking for your
niche, and you see that mobile application development might be it. As
the Flutter people continue to experiment, you start to think about what
it would mean to target mobile devices with Dart.</p><p><a href="https://mrale.ph/talks/vmil2020/#/3">The initial Dart VM was made to
JIT</a>, but as we know, Apple doesn't
let people do this on iOS. So instead you look to write a
quick-and-dirty ahead-of-time compiler, based on your JIT compiler that
takes your program as input, parses and baseline-compiles it, and
generates an image that can be loaded at runtime. It ships on iOS.
Funnily enough, it ships on Android too, because AOT compilation allows
you to avoid some startup costs; forced to choose between peak
performance via JIT and fast startup via AOT, you choose fast startup.</p><p>It's a success, you hit your product objectives, and you start to look
further to a proper ahead-of-time compiler native code that can stand
alone without the full Dart run-time. After all, if you have to compile
at build-time, you might as well <a href="https://github.com/dart-lang/sdk/issues/30480">take the time to do some proper
optimizations</a>. You
actually <a href="https://medium.com/dartlang/dart-and-the-performance-benefits-of-sound-types-6ceedd5b6cdc">change the language to have a sound typing
system</a>
so that the compiler can make program transformations that are valid as
long as it can rely on the program's types.</p><p>Fun fact: I am told that the shift to a sound type system actually
started before Flutter and thus before AOT, because of a
Dart-to-JavaScript compiler that you inherited from the time in which
you thought the web would be the main target. The Dart-to-JS compiler
used to be a whole-program compiler; this enabled it to do
flow-sensitive type inference, resulting in faster and smaller emitted
JavaScript. But whole-program compilation doesn't scale well in terms
of compilation time, so Dart-to-JS switched to separate per-module
compilation. But then you lose lots of types! The way to recover the
fast-and-small-emitted-JS property was through a stronger, sound type
system for Dart.</p><p>At this point, you still have your virtual machine, plus your
ahead-of-time compiler, plus your Dart-to-JS compiler. Such riches,
such bounty! It is not a bad situation to be in, in 2023: you can offer
a good development experience via the just-in-time compiled virtual
machine. Apparently you can even use the JIT on iOS in developer mode,
because attaching <tt>ptrace</tt> to a binary allows for native code
generation. Then when you go to deploy, you make a native binary that
includes everything.</p><p>For the web, you also have your nice story, even nicer than with
JavaScript in some ways: because the type checker and ahead-of-time
compiler are integrated in Dart, you don't have to worry about WebPack
or Vite or minifiers or uglifiers or TypeScript or JSX or Babel or any
of the other things that JavaScript people are used to. Granted, the
tradeoff is that innovation is mostly centralized with the Dart
maintainers, but currently Google seems to be investing enough so that's
OK.</p><p>Stepping back, this story is not unique to Dart; many of its scenes also
played out in the world of JavaScript over the last 5 or 10 years as
well. <a href="https://hermesengine.dev/">Hermes</a> (and
<a href="https://bellard.org/quickjs/">QuickJS</a>, for that matter) does
ahead-of-time compilation, albeit only to bytecode, and V8's snapshot
facility is a form of native AOT compilation. But the tooling in the
JavaScript world is more diffuse than with Dart. With the perspective
of developing a new JavaScript-based mobile operating system in mind,
the advantages that Dart (and thus Flutter) has won over the years are
also on the table for JavaScript to win. Perhaps even TypeScript could
eventually migrate to have a sound type system, over time; it would take
a significant investment but the JS ecosystem does evolve, if slowly.</p><p>(Full disclosure: while the other articles in this series were written
without input from the authors of the frameworks under review, through
what I can only think was good URL guesswork, a draft copy of this
article leaked to Flutter developers. Dart hacker Slava Egorov kindly
sent me a mail correcting a number of misconceptions I had about Dart's
history. Fair play on whoever guessed the URL, and many thanks to Slava
for the corrections; any remaining errors are wholly mine, of course!)</p><h3>Evaluation</h3><p>So how do we expect Flutter applications to perform? If we were writing a new mobile OS based on JavaScript, what would it mean in terms of performance to adopt a Flutter-like architecture?</p><h3>Startup latency</h3><p>Flutter applications are well-positioned to start fast, with
ahead-of-time compilation. However they have had <a href="https://github.com/flutter/flutter/projects/188">problems realizing
this potential</a>, with
many users seeing a big stutter when they launch a Flutter app.</p><p>To explain this situation, consider the structure of a typical low-end
Android mobile device: you have a small number of not-terribly-powerful
CPU cores, but attached to the same memory you also have a decent GPU
with many cores. For example, the <a href="https://www.cpu-monkey.com/en/cpu-qualcomm_snapdragon_460">SoC in the low-end Moto E7
Plus</a> has 8
CPU cores and 128 GPU cores (texture shader units). You could paint
widget pixels into memory from either the CPU or the GPU, but you'd
rather do it in the GPU because it has so many more cores: in the time
it takes to compute the color of a single pixel on the CPU, on the GPU
you could do, like, 128 times as many, given that the comparison is
often between multi-threaded rasterization on the GPU versus
single-threaded rasterization on the CPU.</p><p>Flutter has always tried to paint on the GPU. Historically it has done
so via a GPU back-end to the Skia graphics library, notably used by
Chrome among other projects. But, Skia's API is a drawing API, not a
GPU API; Skia is the one responsible for configuring the GPU to draw
what we want. And here's the problem: configuring the GPU takes time.
Skia generates shader code at run-time for rasterizing the specific
widgets used by the Flutter programmer. That shader code then needs to
be compiled to the language the GPU driver wants, which looks more like
<a href="https://www.vulkan.org/">Vulkan</a> or
<a href="https://developer.apple.com/metal/">Metal</a>. The process of compilation
and linking takes time, potentially seconds, even.</p><p>The solution to "too much startup shader compilation" is much like the
solution to "too much startup JavaScript compilation": move this phase
to build time. The new
<a href="https://docs.flutter.dev/perf/impeller">Impeller</a> rendering library
does just that. However to do that, it had to change the way that
Flutter renders: instead of having Skia generate specialized shaders at
run-time, Impeller instead lowers the shapes that it draws to a fixed
set of primitives, and then renders those primitives using a <a href="https://github.com/flutter/engine/tree/main/impeller/entity/shaders">smaller,
fixed set of
shaders</a>.
These primitive shaders are pre-compiled at build time and included in
the binary. By switching to this new renderer, Flutter should be able
to avoid startup jank.</p><h3>Jank</h3><p>Of all the application development frameworks we have considered, to my
mind Flutter is the best positioned to avoid jank. It has the
React-like asynchronous functional layout model, but "closer to the
metal"; by skipping the tree of native UI widgets, it can potentially
spend less time for each frame render.</p><p>When you start up a Flutter app on iOS, the shell of the application is
actually written in Objective C++. On Android it's the same, except
that it's Java. That shell then creates a FlutterView widget and spawns
a new thread to actually run Flutter (and the user's Dart code).
Mostly, Flutter runs on its own, rendering frames to the GPU resources
backing the FlutterView directly.</p><p>If a Flutter app needs to communicate with the platform, it <a href="https://docs.flutter.dev/development/platform-integration/platform-channels">passes
messages across an asynchronous channel back to the main
thread</a>.
Although these messages are asynchronous, this is probably the largest
potential source of jank in a Flutter app, outside the initial frame
paint: any graphical update which depends on the answer to an
asynchronous call may lag.</p><h3>Peak performance</h3><p>Dart's type system and ahead-of-time compiler optimize for predictable
good performance rather than the more variable but potentially higher
peak performance that could be provided by just-in-time compilation.</p><p>This story should probably serve as a lesson to any future platform.
The people that developed the original Dart virtual machine had a
built-in bias towards just-in-time compilation, because it allows the VM
to generate code that is specialized not just to the program but also to
the problem at hand. A given system with ahead-of-time compilation can
always be made to perform better via the addition of a just-in-time
compiler, so the initial focus was on JIT compilation. On iOS of course
this was not possible, but on Android and other platforms where this was
available it was the default deployment model.</p><p>However, even Android switched to ahead-of-time compilation instead of
the JIT model in order to reduce startup latency: doing any machine code
generation at all at program startup was more work than was needed to
get to the first frame. One could add JIT back again on top of AOT but
it does not appear to be a high priority.</p><p>I would expect that Capacitor could beat Dart in some raw throughput
benchmarks, given that <a href="https://wingolog.org/archives/2023/04/20/structure-and-interpretation-of-capacitor-programs">Capacitor's JavaScript implementation can take
advantage of the platform's native JIT capability</a>. Does it matter,
though, as long as you are hitting your frame budget? I do not know.</p><h3>Aside: An escape hatch to the platform</h3><p>What happens if you want to embed a web view into a Flutter app?</p><p>If you think on the problem for a moment I suspect you will arrive at
the unsatisfactory answer, which is that for better or for worse, at
this point it is too expensive even for Google to make a new web engine.
Therefore Flutter will have to embed the native WebView. However
Flutter runs on its own threads; the native WebView has its own process
and threads but its interface to the app is tied to the main UI thread.</p><p>Therefore either you need to make the native WebView (or indeed any
other native widget) render itself to (a region of) Flutter's GPU
backing buffer, or you need to copy the native widget's pixels into
their own texture and then composite them in Flutter-land. It's not so
nice! The
<a href="https://docs.flutter.dev/development/platform-integration/android/platform-views">Android</a>
and
<a href="https://docs.flutter.dev/development/platform-integration/ios/platform-views">iOS</a>
platform view documentation discuss some of the tradeoffs and
mitigations.</p><h3>Aside: For want of a canvas</h3><p>There is a very funny situation in the React Native world in which, if
the application programmer wants to draw to a canvas, they have to
<a href="https://github.com/react-native-webview/react-native-webview/blob/master/docs/Guide.md#communicating-between-js-and-native">embed a whole WebView into the React Native
app</a>
and then <a href="https://github.com/iddan/react-native-canvas#readme">proxy the canvas calls into the
WebView</a>. Flutter
is happily able to avoid this problem, because it includes its own
drawing library with a canvas-like API. Of course, Flutter also has the
luxury of defining its own set of standard libraries instead of
necessarily inheriting them from the web, so when and if they want to
provide equivalent but differently-shaped interfaces, they can do so.</p><p>Flutter manages to be more expressive than React Native in this case,
without losing much in the way of understandability. Few people will
have to reach to the canvas layer, but it is nice to know it is there.</p><h3>Conclusion</h3><p>Dart and Flutter are terribly attractive from an engineering
perspective. They offer a delightful API and a high-performance,
flexible runtime with a built-in toolchain. Could this experience be
brought to a new mobile operating system as its primary programming
interface, based on JavaScript? React Native is giving it a try, but I
think there may be room to take things further to own the application
from the program all the way down to the pixels.</p><p>Well, that's all from me on Flutter and Dart for the time being. Next
up, a mystery guest; see you then!</p></div>2023-04-26T13:50:01+00:00Andy WingoAndy Wingo: structure and interpretation of nativescript
https://wingolog.org/archives/2023/04/24/structure-and-interpretation-of-nativescript
<div><p>Greetings, hackers tall and hackers small!</p><p>We're only a few articles in to this series on mobile application development frameworks, but I feel like we are already well into our journey. We started our trip through the design space with a look at
<a href="https://wingolog.org/archives/2023/04/20/structure-and-interpretation-of-capacitor-programs">Ionic /
Capacitor</a>,
which defines its user interface in terms of the web platform, and only
calls out to iOS or Android native features as needed. We proceeded
on to <a href="https://wingolog.org/archives/2023/04/21/structure-and-interpretation-of-react-native">React
Native</a>,
which moves closer to native by rendering to platform-provided UI
widgets, layering a cross-platform development interface on top.</p><p>Today's article takes an in-depth look at <a href="https://nativescript.org/">NativeScript</a>, whose point in the design space is further on the road towards
the platform, unabashedly embracing the specificities of the API
available on iOS and Android, exposing these interfaces directly to the
application programmer.</p><p>In practice what this looks like is that a NativeScript app is a native
app which simply happens to call JavaScript on the main UI thread. That
JavaScript has access to <i>all</i> native APIs, directly, without the
mediation of serialization or message-passing over a bridge or message
queue.</p><p>The first time I heard this I thought that it couldn't actually be <i>all</i>
native APIs. After all, new versions of iOS and Android come out quite
frequently, and surely it would take some effort on the part of
NativeScript developers to expose the new APIs to JavaScript. But no,
it really includes all of the various native APIs: the NativeScript
developers wrote a build-time inspector that uses the platform's native
reflection capabilities to grovel through all available APIs and to
automatically generate JavaScript bindings, with associated TypeScript
type definitions so that the developer knows what is available.</p><p>Some of these generated files are checked into source, so you can get an
idea of the range of interfaces that are accessible to programmers; for
example, see <a href="https://github.com/NativeScript/NativeScript/tree/main/packages/types-ios/src/lib/ios/objc-x86_64">the iOS type definitions for
x86-64</a>.
There are bindings for, like, everything.</p><p>Given access to all the native APIs, how do you go about making an app?
You could write the same kind of programs that you would in Swift or
Kotlin, but in JavaScript. But this would require more than just the
<i>ability</i> to access native capabilities when needed: it needs a thorough
knowledge of the platform interfaces, plus NativeScript itself on top.
Most people don't have this knowledge, and those that do are probably
programming directly in Swift or Kotlin already.</p><p>On one level, NativeScript's approach is to take refuge in that most
ecumenical of adjectives, "unopinionated". Whereas Ionic / Capacitor
encourages use of web platform interfaces, and React Native only really
supports React as a programming paradigm, NativeScript provides a
low-level platform onto which you can layer a number of different
high-level frameworks.</p><p>Now, most high-level JavaScript application development frameworks are
oriented to targetting the web: they take descriptions of user
interfaces and translate them to the DOM. When targetting NativeScript,
you could make it so that they target native UI widgets instead.
However given the baked-in assumptions of how widgets should be laid out
(notably via CSS), there is some impedance-matching to do between
DOM-like APIs and native toolkits.</p><p>NativeScript's answer to this problem is a middle layer: a
<a href="https://github.com/NativeScript/NativeScript/tree/main/packages/core/ui">cross-platform UI
library</a>
that provides DOM-like abstractions and CSS layout in a way that bridges
the gap between web-like and native. You can even define parts of the
UI using a <a href="https://github.com/NativeScript/NativeScript/blob/main/apps/automated/src/xml-declaration/tns.xsd">NativeScript-specific XML
vocabulary</a>,
which NativeScript compiles to native UI widget calls <a href="https://github.com/NativeScript/NativeScript/tree/main/packages/core/ui/builder">at
run-time</a>.
Of course, there is no CSS engine in UIKit or Android's UI toolkit, so
NativeScript includes its own, <a href="https://github.com/NativeScript/NativeScript/blob/main/packages/core/ui/styling/css-selector.ts">implemented in JavaScript of
course</a>.</p><p>You could program directly to this middle layer, but I suspect that its
real purpose is in enabling Angular, Vue, Svelte, or the like. The
pitch would be that NativeScript lets app developers use pleasant
high-level abstractions, but while remaining close to the native APIs;
you can always drop down for more power and expressiveness if needed.</p><p>Diving back down to the low level, as we mentioned all of the
interactions between JavaScript and the native platform APIs happen on
the main application UI thread. NativeScript does also allow
programmers to create background threads, using an <a href="https://docs.nativescript.org/advanced-concepts.html#multithreading-model">implementation of
the Web Worker
API</a>.
One could even in theory run a React-based UI in a worker thread and
proxy native UI updates to the main thread; as an unopinionated
platform, NativeScript can support many different frameworks and
paradigms.</p><p>Finally, there is the question of how NativeScript runs the JavaScript
in an application. Recall that Ionic / Capacitor uses the native JS
engine, by virtue of using the native WebView, and that React Native
used to use JavaScriptCore on both platforms but now uses its own Hermes
implementation. NativeScript is another point in the design space,
using V8 on both platforms. (They used to use JavaScriptCore on iOS but
<a href="https://blog.nativescript.org/the-new-ios-runtime-powered-by-v8/">switched to
V8</a>
once V8 was able to run on iOS in "jitless" mode.) Besides the reduced
maintenance burden of using a single implementation on all platforms,
this also has the advantage of being able to use <a href="https://v8.dev/blog/custom-startup-snapshots">V8
snapshots</a> to move
JavaScript parse-and-compile work to build-time, even on iOS.</p><h3>Evaluation</h3><p>NativeScript is fundamentally simple: it's V8 running in an
application's main UI thread, with access to all platform native APIs.
So how do we expect it to perform?</p><h3>Startup latency</h3><p>In theory, applications with a NativeScript-like architecture should
have no problem with startup time, because they can pre-compile all of
their JavaScript into V8 snapshots. Snapshots are cheap to load up
because they are already in a format that V8 is ready to consume.</p><p>In practice, it would seem that <a href="https://github.com/NativeScript/NativeScript/issues/8926">V8 snapshots do not perform as expected
for
NativeScript</a>.
There are a number of aspects about this situation that I don't
understand, which I suspect relate to the state of the tooling around V8
rather than to the fundamental approach of ahead-of-time compilation.
V8 is really made for Chrome, and it could be that not enough
maintenance resources have been devoted to this snapshot facility.</p><p>In the meantime, NativeScript instead uses V8's <a href="https://v8.dev/blog/code-caching">code cache
feature</a>, which caches the result of
parsing and compiling JavaScript files on the device. In this way the
first time an app is installed or updated, it might start up slowly, but
subsequent runs are faster. If you were designing a new operating
system, you'd probably want to move this work to app install-time.</p><p>As we mentioned above, NativeScript apps have access to all native APIs.
That is a lot of APIs, and only some of those interfaces will actually
be used by any given app. In an ideal world, we would expect the build
process to only include JavaScript code for those APIs that are needed
by the application. However in the presence of <tt>eval</tt> and dynamic
property lookup, pruning the native API surface to the precise minimum
is a hard problem for a bundler to perform on its own. The solution for
the time being is to <a href="https://docs.nativescript.org/advanced-concepts.html#metadata-filtering">manually allow and deny subsets of the platform
native
API</a>.
It's not an automatic process though, so it can be error-prone.</p><p>Besides the work that the JavaScript engine has to do to load an
application's code, the other startup overhead involves whatever work
that JavaScript might need to perform before the first frame is shown.
In the case of NativeScript, more work is done before the initial layout
than one would think: the main UI XML file is parsed by an XML parser
written in JavaScript, any needed CSS files are parsed and loaded (again
by JavaScript), and the tree of XML elements is translated to a <a href="https://docs.nativescript.org/advanced-concepts.html#the-layout-process">tree of
UI
elements</a>.
The layout of the items in the view tree is then computed (in
JavaScript, but calling into native code to measure text and so on), and
then the app is ready.</p><p>At this point, I am again going to wave my "I am just a compiler
engineer" flag: I am not a UI specialist, much less a NativeScript
specialist. As in compilers, performance measurement and monitoring are
key to UI development, but I suspect that also as in compilers there is
a role for gut instinct. Incremental improvements are best driven by
metrics, but qualitative leaps are often the result of somewhat
ineffable hunches or even guesswork. In that spirit I can only surmise
that React Native has an advantage over NativeScript in
time-to-first-frame, because its layout is performed in C++ and because
its element tree is computed directly from JavaScript instead of having
JavaScript interpret XML and CSS files. In any case, I look forward to
the forthcoming part 2 of the <a href="https://blog.nativescript.org/perf-metrics-universal-javascript-part1/">NativeScript and React Native performance
investigations</a>
that were started in November 2022.</p><p>If I were NativeScript and using NativeScript's UI framework, and if
startup latency proves to actually be a problem, I would lean into
something in the shape of <a href="https://angular.io/guide/aot-compiler">Angular's ahead-of-time compilation
mode</a>, but for the middle
NativeScript UI layer.</p><h3>Jank</h3><p>On the face of it, NativeScript is the most jank-prone of the three
frameworks we have examined, because it runs JavaScript on the main
application UI thread, interleaved with UI event handling and painting
and all of that. If an app's JavaScript takes too long to run, the app
might miss frames or fail to promptly handle an event.</p><p>On the other hand, relative to React Native, the user's code is much
closer to the application's behavior. There's no asynchrony between the
application's logic and its main loop: in NativeScript it is easy to
identify the code causing jank and eventually fix it.</p><p>The other classic JavaScript-on-the-main-thread worry relates to garbage
collection pauses. V8's garbage collector does try to minimize the
stop-the-world phase by <a href="https://v8.dev/blog/trash-talk">tracing the heap concurrently and leveraging
parallelism during pauses</a>. Also, the
user interface of a mobile app runs in an event loop, and typically
spends most of its time idle; V8 exposes some API that can take
advantage of this idle time to perform housekeeping tasks instead of
needing to do them when handling high-priority events.</p><p>That said, having looked into the code of both the iOS and Android
run-times, NativeScript does not currently take advantage of this
facility. I dug deeper and it would seem that V8 itself is in flux, as
the <a href="https://chromium.googlesource.com/v8/v8/+/refs/heads/main/include/v8-isolate.h#1281">IdleNotificationDeadline
API</a>
is on its way out; is the thought that concurrent tracing is largely
sufficient? I would expect that if <a href="https://groups.google.com/g/v8-reviews/c/5ptLig88soA?pli=1">conservative stack
scanning</a>
lands, we will see a re-introduction of this kind of API, as it does
make sense to synchronize with the event loop when scanning the main
thread stack.</p><h3>Peak performance</h3><p>As we have seen in our previous evaluations, this question boils down to
"is the JavaScript engine state-of-the-art, and can it perform
just-in-time compilation". In the case of NativeScript, the answers are
yes and maybe, respectively: V8 is state-of-the-art, and it can JIT on
Android, but not on iOS.</p><p>Perhaps the mitigation here is that the hardware that iOS runs on tends
to be significantly more powerful than median Android devices; if you
had to pick a subset of users to penalize with an interpreter-only
run-time, people with iPhones are the obvious choice, because they can
afford it.</p><h3>Aside: Are markets wise?</h3><p>Recall that our perspective in this series is that of the designer of a
new JavaScript-based mobile development platform. We are trying to
answer the question of what would it look like if a new platform offered
a NativeScript-like experience. In this regard, only the structure of
NativeScript is of interest, and notably its "market success" is not
relevant, except perhaps in some Hayekian conception of the world in which
markets are necessarily smarter than, well, me, or you, or any one of
us.</p><p>It must be said, though, that React Native is the 800-pound gorilla of
JavaScript mobile application development. The <a href="https://2022.stateofjs.com/en-US/libraries/mobile-desktop/">2022 State of JS
survey</a>
shows that among survey respondents, more people are aware of React
Native than any other mobile framework, and people are generally more
positive about React Native than other frameworks. Does NativeScript's
mitigated market share indicate something about its architecture, or
does it speak speak more to the size of Facebook's budget, both on the
developer experience side and on marketing?</p><h3>Aside: On the expressive power of application framworks</h3><p>Oddly, I think the answer to the market wisdom question might be found
in a 35-year-old computer science paper, <a href="https://dl.acm.org/doi/10.1016/0167-6423%2891%2990036-W">"On the expressive power of
programming
languages"</a>
(<a href="https://www.sciencedirect.com/science/article/pii/016764239190036W/pdf?md5=b7dedd960214d9191929e6f41f5fd5be&pid=1-s2.0-016764239190036W-main.pdf&_valck=1">PDF</a>).</p><p>In this paper, Matthias Felleisen considers the notion of what it means
for one programming language to be more expressive than another. For
example, is a language with just <tt>for</tt> less expressive than a language
with both <tt>for</tt> and <tt>while</tt>? Intuitively we would say no, these are
similar things; you can make a simple local transformation of <tt>while (x) {...}</tt> to <tt>for (;x;) {...}</tt> and you have exactly the same program
semantics. On the other hand a language with just <tt>for</tt> is less
expressive than one which also has <tt>goto</tt>; there is no simple local
rewrite that can turn <tt>goto</tt> into <tt>for</tt>.</p><p>In the same way, we can consider the question of what it would mean for
one library to be more expressive than another. After all, the API of a
library exposes a language in which its user can write programs; we
should be able to reason about these languages. So between React Native
and NativeScript, which one is more expressive?</p><p>By Felleisen's definitions, NativeScript is clearly the more expressive
language: there is no simple local transformation that can turn
imperative operations on native UI widgets into equivalent
functional-reactive programs. Yes, with enough glue code React Native
can reach directly to native APIs in a similar way as NativeScript, but
everything that touches the native UI tree is expressly under React
Native's control: there is no sanctioned escape hatch.</p><p>You might think that "more expressive" is always better, but Felleisen's
take is more nuanced than that. Yes, he says, more expressive languages
do allow programmers to make more concise programs, because they allow
programmers to define abstractions that encapsulate patterns, and this
is a good thing. However he also concludes that "an increase in
expressive power is related to a decrease of the set of 'natural'
(mathematically appealing) operational equivalences." Less expressive
programming languages are easier to reason about, in general, and indeed
that is one of the recognized strengths of React's programming model: it
is easy to compose components and have confidence that the result will
work.</p><h3>Summary</h3><p>A NativeScript-like architecture offers the possibility of performance:
the developer has all the capabilities needed for writing
pleasant-to-use applications that blend in with the platform-native
experience. It is up to the developers to choose how to use the power
at their disposal. In the wild, I expect that the low-level layer of
NativeScript's API is used mainly by expert developers, who know how to
assemble well-functioning machines from the parts on offer.</p><p>As a primary programming interface for a new JavaScript-based mobile
platform, though, just providing a low-level API would seem to be not
enough. NativeScript rightly promotes the use of more well-known
high-level frameworks on top: Angular, Vue, Svelte, and such. Less
experienced developers should use an opinionated high-level UI
framework; these developers don't have good opinions yet and the API
should lead them in the right direction.</p><p>That's it for today. Thanks for reading these articles, by the way; I
have enjoyed diving into this space. Next up, we'll take a look beyond
JavaScript, to <a href="https://wingolog.org/archives/2023/04/26/structure-and-interpretation-of-flutter">Flutter and Dart</a>. Until then, happy hacking!</p></div>2023-04-24T09:10:58+00:00Andy WingoAndy Wingo: structure and interpretation of react native
https://wingolog.org/archives/2023/04/21/structure-and-interpretation-of-react-native
<div><p>Hey hey! Today's missive continues exploring the space of JavaScript
and mobile application development.</p><p>Yesterday we looked into <a href="https://wingolog.org/archives/2023/04/20/structure-and-interpretation-of-capacitor-programs">Ionic / Capacitor</a>, giving a brief
structural overview of what Capacitor apps look like under the hood and
how this translates to three aspects of performance: startup latency, jank,
and peak performance. Today we'll apply that same approach to another
popular development framework, <a href="https://reactnative.dev/">React Native</a>.</p><h3>Background: React</h3><p>I don't know about you, but I find that there is so much marketing smoke
and lights around the whole phenomenon that is React and React Native
that sometimes it's hard to see what's actually there. This is
compounded by the fact that the programming paradigm espoused by React
(and its "native" cousin that we are looking at here) is so effective at
enabling JavaScript UI programmers to focus on the "what" and not the
"how" that the machinery supporting React recedes into the background.</p><p>At its most basic, React is what they call a <i>functional reactive</i>
programming model. It is <i>functional</i> in the sense that the user
interface elements render as a <i>function</i> of the global application
state. The <i>reactive</i> comes into how user input is handled, but I'm not
going to focus on that here.</p><p>React's rendering process starts with a root <i>element tree</i>, describing
the root node of the user interface. An <i>element</i> is a JavaScript
object with a <tt>type</tt> property. To render an element tree, if the value
of the <tt>type</tt> property is a string, then the element is <i>terminal</i> and
doesn't need further lowering, though React will visit any node in the
<tt>children</tt> property of the element to render them as needed.</p><p>Otherwise if the <tt>type</tt> property of an element is a function, then the
element node is <i>functional</i>. In that case React invokes the node's
render function (the <tt>type</tt> property), passing the JavaScript element
object as the argument. React will then recursively re-render the
element tree produced as a result of rendering the component until all
nodes are terminal. (Functional element nodes can instead have a class
as their <tt>type</tt> property, but the concerns are pretty much the same.)</p><p>(In the <a href="https://reactnative.dev/architecture/glossary">language of React
Native</a>, a terminal node
is a <i>React Host Component</i>, and a functional node is a <i>React Composite
Component</i>, and both are <i>React Elements</i>. There are many
imprecisely-used terms in React and I will continue this tradition by
using the terms I mention above.)</p><p>The rendering phase of a React application is thus a function from an
element tree to a terminal element tree. Nodes of element trees can be
either functional or terminal. Terminal element trees are composed only
of terminal elements. Rendering lowers all functional nodes to terminal
nodes. This description applies both to React (targetting the web) and
React Native (which we are reviewing here).</p><p>It's probably useful to go deeper into what React does with a terminal
element tree, before building to the more complex pipeline used in React
Native, so here we go. The basic idea is that React-on-the-web does
impedance matching between the functional description of what the UI
should have, as described by a terminal element tree, and the stateful
tree of DOM nodes that a web browser uses to actually paint and display
the UI. When rendering yields a new terminal element tree, React will
compute the difference between the new and old trees. From that
difference React then computes the set of imperative actions needed to
mutate the DOM tree to correspond to what the new terminal element tree
describes, and finally applies those changes.</p><p>In this way, small changes to the leaves of a React element tree should
correspond to small changes in the DOM. Additionally, since rendering
is a pure function of the global application state, we can avoid
rendering at all when the application state hasn't changed. We'll dive
into performance more deeply later on in the article.</p><h3>React Native doesn't use a WebView</h3><p>React Native is similar to React-on-the-web in intent but different in
structure. Instead of using a WebView on native platforms, as <a href="https://wingolog.org/archives/2023/04/20/structure-and-interpretation-of-capacitor-programs">Ionic /
Capacitor</a> does, React Native renders the terminal element tree to
platform-native UI widgets.</p><p>When a React Native functional element renders to a terminal element, it
will create not just a JS object for the terminal node as
React-on-the-web does, but also a corresponding C++ <a href="https://reactnative.dev/architecture/render-pipeline"><i>shadow
object</i></a>. The
fully lowered tree of terminal elements will thus have a corresponding
tree of C++ shadow objects. React Native will then calculate the layout
for each node in the shadow tree, and then <i>commit</i> the shadow tree: as
on the web, React Native computes the set of imperative actions needed
to change the current UI so that it corresponds to what the shadow tree
describes. These changes are then applied on the main thread of the
application.</p><h3>The twisty path that leads one to implement JavaScript</h3><p>The description above of React Native's rendering pipeline applies to
the so-called <a href="https://reactnative.dev/architecture/overview">"new
architecture"</a>, which has
been in the works for some years and is only now (April 2023) starting
to be deployed. The key development that has allowed React Native to
move over to this architecture is tighter integration and control over
its JavaScript implementation. Instead of using the platform's
JavaScript engine (JavaScriptCore on iOS or V8 on Android), Facebook
went and made their own whole new JavaScript implementation,
<a href="https://hermesengine.dev/">Hermes</a>. Let's step back a bit to see if we
can imagine why anyone in their right mind would make a new JS
implementation.</p><p>In the last article, I mentioned that the only way to get peak JS
performance on iOS is to use the platform's WkWebView, which enables JIT
compilation of JavaScript code. React Native doesn't want a WebView,
though. I guess you could create an invisible WebView and just run your
JavaScript in it, but the real issue is that the interface to the
JavaScript engine is so narrow as to be insufficiently expressive. You
can't cheaply synchronously create a shadow tree of layout objects, for
example, because every interaction with JavaScript has to cross a
process boundary.</p><p>So, it may be that JIT is just not worth paying for, if it means having
to keep JavaScript at arm's distance from other parts of the
application. How do you do JavaScript without a browser on mobile,
though? Either you use the platform's JavaScript engine, or you ship
your own. It would be nice to use the same engine on iOS and Android,
though. When React Native was first made, V8 wasn't able to operate in
a mode that didn't JIT, so React Native went with JavaScriptCore on both
platforms.</p><p>Bundling your own JavaScript engine has the nice effect that you can
easily augment it with native extensions, for example to talk to the
Swift or Java app that actually runs the main UI. That's what I
describe above with the creation of the shadow tree, but that's not
quite what the original React Native did; I can only speculate but I
suspect that there was a fear that JavaScript rendering work (or garbage
collection!) could be heavy enough to cause the main UI to drop frames.
Phones were less powerful in 2016, and JavaScript engines were less
good. So the original React Native instead ran JavaScript in a separate
thread. When a render would complete, the resulting terminal element
tree would be serialized as JSON and shipped over to the "native" side
of the application, which would actually apply the changes.</p><p>This arrangement did work, but it ran into problems whenever the system
needed synchronous communication between native and JavaScript
subsystems. As I understand it, this was notably the case when React
layout would need the dimensions of a native UI widget; to avoid a
stall, React would assume something about the dimensions of the native
UI, and then asynchronously re-layout once the actual dimensions were
known. This was particularly gnarly with regards to text measurements,
which depend on low-level platform-specific rendering details.</p><p>To recap: React Native had to interpret its JS on iOS and was using a
"foreign" JS engine on Android, so they weren't gaining anything by
using a platform JS interpreter. They would sometimes have some
annoying layout jank when measuring native components. And what's more,
React Native apps would still experience the same problem as Ionic /
Capacitor apps, in that application startup time was dominated by
parsing and compiling the JavaScript source files.</p><p>The solution to this problem was partly to switch to the so-called "new
architecture", which doesn't serialize and parse so much data in the
course of rendering. But the other side of it was to find a way to move
parsing and compiling JavaScript to the build phase, instead of having
to parse and compile JS every time the app was run. On V8, you would do
this by generating a
<a href="https://v8.dev/blog/custom-startup-snapshots">snapshot</a>. On
JavaScriptCore, which React Native used, there was no such facility.
Faced with this problem and armed with Facebook's bank account, the
React Native developers decided that the best solution would be to make
a new JavaScript implementation optimized for ahead-of-time compilation.</p><p>The result is <a href="https://hermesengine.dev/">Hermes</a>. If you are familiar
with JavaScript engines, it is what you might expect: a JavaScript
parser, originally built to match the behavior of
<a href="https://esprima.org/">Esprima</a>; an <a href="https://github.com/facebook/hermes/blob/main/include/hermes/IR/IR.h">SSA-based intermediate
representation</a>;
a <a href="https://github.com/facebook/hermes/blob/main/include/hermes/Optimizer/PassManager/Passes.def">set of basic
optimizations</a>;
a <a href="https://github.com/facebook/hermes/blob/main/include/hermes/BCGen/HBC/BytecodeList.def">custom bytecode
format</a>;
an <a href="https://github.com/facebook/hermes/blob/main/lib/VM/Interpreter.cpp">interpreter to run that
bytecode</a>;
a <a href="https://hermesengine.dev/docs/hades/">GC to manage JS objects</a>; and
so on. Of course, given the presence of <tt>eval</tt>, Hermes needs to include
the parser and compiler as part of the virtual machine, but the hope is
that most user code will be parsed and compiled ahead-of-time.</p><p>If this were it, I would say that Hermes seems to me to be a dead end.
V8 is complete; Hermes is not. For example, Hermes doesn't have <tt>with</tt>,
async function implementation has been lagging, and so on. Why Hermes
when you can V8 (with snapshots), now that <a href="https://v8.dev/blog/jitless">V8 doesn't require JIT code
generation</a>?</p><p>I thought about this for a while and in the end, given that V8's main
target isn't as an embedded library in a mobile app, perhaps the binary
size question is the one differentiating factor (in theory) for Hermes.
By focussing on lowering distribution size, perhaps Hermes will be a
compelling JS engine in its own right. In any case, Facebook can afford
to keep Hermes running for a while, regardless of whether it has a
competitive advantage or not.</p><p>It sounds like I'm criticising Hermes here but that's not really the
point. If you can afford it, it's good to have code you control. For
example one benefit that I see React Native getting from Hermes is that
they control the <a href="https://reactnative.dev/architecture/threading-model">threading
model</a>; they can
mostly execute JS in its own thread, but interrupt that thread and
switch to synchronous main-thread execution in response to high-priority
events coming from the user. You might be able to do that with V8 at
some point but the mobile-apps-with-JS domain is still in flux, so it's
nice to have a sandbox that React Native developers can use to explore
the system design space.</p><h3>Evaluation</h3><p>With that long overview out of the way, let's take a look to what kinds
of performance we can expect out of a React Native system.</p><h3>Startup latency</h3><p>Because React Native apps have their JavaScript code pre-compiled to
Hermes bytecode, we can expect that the latency imposed by JavaScript
during application startup is lower than is the case with Ionic /
Capacitor, which needs to parse and compile the JavaScript at run-time.</p><p>However, it must be said that as a framework, <a href="https://timkadlec.com/remembers/2020-04-21-the-cost-of-javascript-frameworks/#javascript-bytes">React tends to result in
large application
sizes</a>
and incurs <a href="https://timkadlec.com/remembers/2020-04-21-the-cost-of-javascript-frameworks/#javascript-main-thread-time">significant work at startup
time</a>.
One of React's strengths is that it allows development teams inside an
organization to compose well: because rendering is a pure function, it's
easy to break down the task of making an app into subtasks to be handled
by separate groups of people. Could this strength lead to a kind of
weakness, in that there is less of a need for overall coordination on
the project management level, such that in the end nobody feels
responsible for overall application performance? I don't know. I think
the concrete differences between React Native and React (the C++ shadow
object tree, the multithreading design, precompilation) could mean that
React Native is closer to an optimum in the design space than React. It
does seem to me though that whether a platform's primary development
toolkit shold be React-like remains an open question.</p><h3>Jank</h3><p>In theory React Native is well-positioned to avoid jank. JavaScript
execution is mostly off the main UI thread. The <a href="https://reactnative.dev/architecture/threading-model">threading
model</a> changes to
allow JavaScript rendering to be pre-empted onto the main thread do make
me wonder, though: what if that work takes too much time, or what if
there is a GC pause during that pre-emption? I would not be surprised
to see an article in the next year or two from the Hermes team about
efforts to avoid GC during high-priority event processing.</p><p>Another question I would have about jank relates to interactivity. Say
the user is dragging around a UI element on the screen, and the UI needs
to re-layout itself. If rendering is slow, then we might expect to see
a lag between UI updates and the dragging motion; the app technically
isn't dropping frames, but the render can't complete in the 16
milliseconds needed for a 60 frames-per-second update frequency.</p><h3>Peak perf</h3><p>But why might rendering be slow? On the one side, there is the fact
that Hermes is not a high-performance JavaScript implementation. It
uses a simple bytecode interpreter, and will never be able to meet the
performance of V8 with JIT compilation.</p><p>However the other side of this is the design of the application
framework. In the limit, React suffers from the O(n) problem: any
change to the application state requires the whole element tree to be
recomputed. Rendering and layout work is proportional to the size of
the application, which may have thousands of nodes.</p><p>Of course, React tries to minimize this work, by detecting subtrees
whose layout does not change, by avoiding re-renders when state doesn't
change, by minimizing the set of mutations to the native widget tree.
But the native widgets aren't the problem: the programming model is, or
it can be anyway.</p><h3>Aside: As good as native?</h3><p>Again in theory, React Native can used to write apps that are as good as
if they were written directly against platform-native APIs in Kotlin or
Swift, because it uses the same platform UI toolkits as native
applications. React Native can also do this at the same time as being
cross-platform, targetting iOS and Android with the same code. In
practice, besides the challenge of designing suitable cross-platform
abstractions, React Native has to grapple with potential performance and
memory use overheads of JavaScript, but the result has the potential to
be quite satisfactory.</p><h3>Aside: Haven't I seen that rendering model somewhere?</h3><p>As I mentioned in the last article, I am a compiler engineer, not a UI
specialist. In the course of my work I do interact with a number of
colleagues working on graphics and user interfaces, notably in the
context of browser engines. I was struck when reading about React
Native's rendering pipeline about how much it resembled what <a href="https://developer.chrome.com/articles/renderingng-data-structures/#the-immutable-fragment-tree">a browser
itself will
do</a>
as part of the layout, paint, and render pipeline: translate a tree of
objects to a tree of immutable layout objects, clip those to the
viewport, paint the ones that are dirty, and composite the resulting
textures to the screen.</p><p>It's funny to think about how many levels we have here: the element
tree, the recursively expanded terminal element tree, the shadow object
tree, the platform-native widget tree, surely a corresponding
platform-native layout tree, and then the GPU backing buffers that are
eventually composited together for the user to see. Could we do better?
I could certainly imagine any of these mobile application development
frameworks switching to their own Metal/Vulkan-based rendering
architecture at some point, to flatten out these layers.</p><h3>Summary</h3><p>By all accounts, React Native is a real delight to program for; it makes
developers happy. The challenge is to make it perform well for users.
With its new rendering architecture based on Hermes, React Native may
well be on the path to addressing many of these problems. Bytecode
pre-compilation should go a long way towards solving startup latency,
provided that React's expands-to-fit-all-available-space tendency is
kept in check.</p><p>If you were designing a new mobile operating system from the ground up,
though, I am not sure that you would necessarily end up with React
Native as it is. At the very least, you would include Hermes and the
base run-time as part of your standard library, so that every app
doesn't have to incur the space costs of shipping the run-time. Also,
in the same way that Android <a href="https://source.android.com/docs/core/runtime/jit-compiler">can ahead-of-time and just-in-time compile
its
bytecode</a>, I
would expect that a mobile operating system based on React Native would
extend its compiler with on-device post-install compilation and possibly
JIT compilation as well. And at that point, why not switch back to V8?</p><p>Well, that's food for thought. Next up, <a href="https://wingolog.org/archives/2023/04/24/structure-and-interpretation-of-nativescript">NativeScript</a>. Until then,
happy hacking!</p></div>2023-04-21T08:20:17+00:00Andy WingoAndy Wingo: structure and interpretation of capacitor programs
https://wingolog.org/archives/2023/04/20/structure-and-interpretation-of-capacitor-programs
<div><p>Good day, hackers! Today's note is a bit of a departure from compilers
internals. A client at work recently asked me to look into
cross-platform mobile application development and is happy for the
results to be shared publically. This, then, is the first in a series
of articles.</p><h3>Mobile apps and JavaScript: how does it work?</h3><p>I'll be starting by taking a look at
<a href="https://ionicframework.com/">Ionic</a>/<a href="https://capacitorjs.com/">Capacitor</a>,
<a href="https://reactnative.dev/">React Native</a>,
<a href="https://nativescript.org/">NativeScript</a>,
<a href="https://flutter.dev/">Flutter</a>/<a href="https://dart.dev/">Dart</a>, and then a
mystery guest. This article will set the stage and then look into
Ionic/Capacitor.</p><p>The angle I am taking is, if you were designing a new mobile operating
system that uses JavaScript as its native application development
language, what would it mean to adopt one of these as your primary app
development toolkit? It's a broad question but I hope we can come up
with some useful conclusions.</p><p>I'm going to approach the problem from the perspective of a compiler
engineer. Compilers translate high-level languages that people like to
write into low-level languages that machines like to run. Compilation
is a bridge between developers and users: developers experience the
source language interface to the compiler, while users experience the
result of compiling those source languages. Note, though, that my
expertise is mostly on the language and compiler side of things; though
I have worked on a few user-interface libraries in my time, I am
certainly not a graphics specialist, nor a user-experience specialist.</p><p>I'm explicitly not including the native application development toolkits
from Android and iOS, because these are most useful to me as a kind of
"control" to the experiment. A cross-platform toolkit that compiles
down to use native widgets can only be as good as the official SDK, but
it might be worse; and a toolkit that routes around the official widgets
by using alternate drawing primitives has the possibility to be better,
but at the risk of being inconsistent with other platform apps, along
with the more general risk of not being as good as the native UI
libraries.</p><p>And, of course, there is the irritation that SwiftUI / UIKit / AppKit
aren't not open source, and that neither iOS nor Android's native UI
library is cross platform. If we are designing a new JavaScript-based
mobile operating system—that's the conceit of the original problem
statement—then we might as well focus on the existing JS-based
toolkits. (Flutter of course is the odd one out, but it's interesting
enough to include in the investigation.)</p><h3>Ionic / Capacitor</h3><p>Let's begin with the <a href="https://ionicframework.com/">Ionic Framework</a> UI
toolkit, based on the <a href="https://capacitorjs.com/">Capacitor</a> web
run-time.</p><h4>Overview</h4><p>Capacitor is like an alternate web browser development kit for phones.
It uses the platform's native WebView:
<a href="https://developer.apple.com/documentation/webkit/wkwebview">WKWebView</a>
on iOS or
<a href="https://developer.android.com/reference/android/webkit/WebView">WebView</a>
on Android. The JavaScript APIs available within that WebView are
extended with <a href="https://capacitorjs.com/docs/apis">Capacitor plugins</a>
which can access native APIs; these plugins let JavaScript do the sort
of things you wouldn't be able to do directly in a web browser.</p><p>(Over time, most of the features provided by Capacitor end up in the web
platform in some way. For example, there are web standards to allow
JavaScript to detect or lock screen rotation. The <a href="https://fugu-tracker.web.app/">Fugu
project</a> is particularly avant-garde in
pushing these capabilities to the web. But, the wheel of web standards
grinds very finely, and for an app to have access to, say, the user's
contact list now, it is pragmatic to use an extended-webview solution
like Capacitor.)</p><p>I call Capacitor a "web browser <i>development kit</i>" because creating a
Capacitor project effectively copies the shell of a specialized browser
into your source tree, usually with an iOS and Android version.
Building the app makes a web browser with your extensions that runs your
JS, CSS, image, and other assets. Running the app creates a WebView and
invokes your JavaScript; your JS should then create the user interface
using the DOM APIs that are standard in a WebView.</p><p>As you can see, Capacitor is quite full-featured when it comes to native
platform capabilities but is a bit bare-bones when it comes to UI. To
provide a richer development environment, the <a href="https://ionicframework.com/">Ionic
Framework</a> adds a set of widgets and some
application life-cycle abstractions on top of the Capacitor WebView.
Ionic is not like a "normal" app framework like
<a href="https://vuejs.org/">Vue</a> or <a href="https://angular.io/">Angular</a> because it
takes advantage of Capacitor APIs, and because it tries to mimic the
presentation and behavior of the native plaform (iOS or Android,
mainly), so that apps built with it don't look out-of-place.</p><p>The Ionic Framework itself is built using <a href="https://developer.mozilla.org/en-US/docs/Web/API/Web_components">Web
Components</a>,
which end up creating styled DOM nodes with JavaScript behavior. These
components can compose with other app frameworks such as Vue or Angular,
allowing developers to share some of their app code between web and
mobile apps---you write the main app in Vue, and just implement the
components one way on the web and using something like Ionic Framework
on mobile.</p><p>To recap, an Ionic app uses Ionic Framework libraries on top of a
Capacitor run-time. You could use other libraries on top of Capacitor
instead of Ionic. In any case, I'm most interested in Capacitor, so
let's continue our focus there.</p><h4>Evaluation</h4><p>Performance has a direct effect on the experience that users have when
interacting with an application, and here I would like to break down the
performance in three ways: one, startup latency; two, jank; and three,
peak throughput. As Capacitor is structurally much like a web browser,
many of these concerns, pitfalls, and mitigation techniques are similar
to those on the web.</p><h4>Startup latency</h4><p>Startup latency is how long the app makes you wait after you run it
before you can interact with it. The app may hide some of this work
behind a splash screen, but if there is too much work to do at
startup-time, the risk is the user might get bored or frustrated and
switch away. The goal, therefore, it to minimize startup latency. Most
people consider time-to-interactive of less than than a second to be
sufficient; there's room to improve but for better or for worse, user
expectations here are lower than they could be.</p><p>In the case of Capacitor, when a user launches an application, the app
loads a minimal skeleton program that loads a WebView. On iOS and most
Android systems, the rendering and JS parts of the WebView run in a
separate process that has access to the app's bundled assets: JavaScript
files, images, CSS, and so on. The WebView then executes the JavaScript
<tt>main</tt> function to create the UI.</p><p>Capacitor application startup time will be dominated by <a href="https://infrequently.org/2021/03/the-performance-inequality-gap/">parsing and
compiling the JavaScript source
files</a>,
as well as any initial work needed to boot the app framework (which
could require running a significant amount of JS), and will depend on
how fast the user's device is. The most effective performance
mitigation here is to reduce the amount of JavaScript loaded, but this
can be difficult for complex apps. The main techniques to reduce app
size are toolchain-based.
<a href="https://rollupjs.org/faqs/#what-is-tree-shaking">Tree-shaking</a>,
<a href="https://esbuild.github.io/api/#bundle">bundling</a>, and
<a href="https://esbuild.github.io/api/#minify">minification</a> reduce the number
of JS bytes that the engine needs to parse without changing application
logic. <a href="https://developer.mozilla.org/en-US/docs/Glossary/Code_splitting">Code
splitting</a>
can defer some work until it is needed, but this might just result in
jank later on.</p><p>Common Ionic application sizes would seem to be between 500 kB and 5 MB
of JavaScript. In 2021, <a href="https://infrequently.org/2021/03/the-performance-inequality-gap/">Alex Russell suggested that the soon-to-be
standard Android performance baseline should be the Moto E7
Plus</a>
which, if its <a href="https://browser.geekbench.com/v4/cpu/compare/15987586?baseline=15667410">performance relative to the mid-range Moto G4
phone</a>
that he measured in 2019 translates to JavaScript engine speed, should
be able to parse and compile uncompressed JavaScript at a rate of about
1 MB/s. That's not very much, if you want to get startup latency under
a second, and a JS payload size of 5 MB would lead to 5-second startup
delays for many users. To my mind, this is the biggest challenge for a
Capacitor-based app.</p><p>(Calculation detail: The Moto G4 JavaScript parse-and-compile throughput
is measured at 170 kB/s, for compressed JS. Assuming a compression
ratio of 4 and that the Moto E7 Plus is 50% faster than the Moto G4,
that gets us to 1 MB/s for uncompressed JS, for what was projected to be
a performance baseline in 2023.)</p><h4>Jank</h4><p>Jank is when an application's animation and interaction aren't smooth,
because the application somehow missed rendering one or more frames.
Technically startup time is a form of jank, but it is useful to treat
startup separately.</p><p>Generally speaking, the question of whether a Capacitor application will
show jank depends on who is doing the rendering: if application
JavaScript is driving an animation, then this could cause jank, in the
same way as it would on the web. The mitigation is to lean on the web
platform so that the WebView is the one in charge of ensuring smooth
interaction, for example by using CSS animations or the native scrolling
capability.</p><p>There may always be an impetus to do some animation in JavaScript,
though, for example if the design guidelines require a specific behavior
that can't be created using raw CSS.</p><h4>Peak perf</h4><p>Peak throughput is how much work an app can do per unit time. For an
application written in JavaScript, this is a question of how good can
the JavaScript engine make a user's code run fast.</p><p>Here we need to make an important aside on the strategies that a
JavaScript engine uses to run a user's code. A standard technique that
JS engines use to make JavaScript run fast is just-in-time (JIT)
compilation, in which the engine emits specialized machine code at
run-time and then runs that code. However, on iOS the platform
prohibits JIT code generation for most applications. The usual
justification for this restriction is that the <a href="https://wingolog.org/archives/2011/06/21/security-implications-of-jit-compilation">low-level capability
granted to a program that allows it to JIT-compile increases the
likelihood of security
exploits</a>.
The only current exception to the no-JIT policy on iOS is for WebView
(including the one in the system web browser), which iOS maintainers
consider to be sufficiently sandboxed, so that any security exploit that
uses JIT code generation won't be able to access other capabilities
possessed by the application process.</p><p>So much for the motivation, as I understand it. The upshot is, there is
no JIT on iOS outside web browsers or web views. If a cross-platform
application development toolkit wants peak JavaScript performance on
iOS, it has to run that JS in a WebView. Capacitor does this, so it has
fast JS on iOS. Note that these restrictions are not in place on
Android; you can have fast JS on Android without using a WebView, by
using the system's JavaScript libraries.</p><p>Of course, actually getting peak throughput out of JavaScript is an art
but the well-known techniques for JavaScript-on-the-web apply here; we
won't cover them in this series.</p><h4>The bridge</h4><p>All of these performance observations are common to all web browsers,
but with Capacitor there is the additional wrinkle that a Capacitor
application can access native capabilities, for example access to a
user's contacts. When application JavaScript goes to access the
contacts API, Capacitor will send a message to the native side of the
application over a <i>bridge</i>. The message will be serialized to JSON.
Capacitor will include an implementation for the native side of the
bridge into your app's source code, written in Swift for iOS or Java for
Android. The native end of the bridge parses the JSON message,
performs the requested action, and sends a message back with the result.
Some messages may need to be proxied to the main application thread,
because some native APIs can only be processed there.</p><p>As you can imagine, the bridge has an overhead. Apps that have
high-bandwidth access to native capabilities will also have JSON
encoding overhead, as well general asynchronous coordination overhead.
It may even be possible that encoding or decoding a large JSON message
causes the WebView to miss a frame, for example when accessing a large
file in local storage.</p><p>The bridge is a necessary component to the design, though; an iOS
WebView can't have direct in-process access to native capabilities. For
Android, the WebView APIs do not appear to allow this either, though it
is theoretically possible to ship your own WebView that could access
native APIs directly. In any case, the Capacitor multi-process solution
does allow for some parallelism, and the enforced asynchronous nature of
the APIs should lead to less modal application programming.</p><h3>Aside: Can WebView act like native widgets?</h3><p>Besides performance, one way that users experience application behavior
is by way of expectations: users expect (but don't require) a degree of
uniformity between different apps on the same platform, for example the
same scrolling behavior or the same kinds of widgets. This aspect isn't
the most important one to my investigation, because I'm more concerned
with a new operating system that might come with its own design language
with its own standard JavaScript-based component library, but it's one
to note; an Ionic app is starting at a disadvantage relative to a
platform-native app. Sometimes ensuring a platform-native UI is just a
question of CSS styling and using the right library (like Ionic
Framework), but sometimes it might take significant engineering or
possibly even new additions to the web platform.</p><h3>Aside: Is a WebView as good as native widgets?</h3><p>As a final observation on user experience, it's worth asking the
question of whether you can actually make a user interface using the web
platform that is "as good" as native widgets. After all, the API and
widget set (DOM) available to a WebView is very different from that
available to a native application; is a WebView able to use CPU and GPU
resources efficiently to create a smooth interface? Are web standards,
CSS, and the DOM API somehow a constraint that prevents efficient user
interface construction? Here, somewhat embarrassingly, I can't actually
say. I am instead going to just play the "compiler engineer" card, that
I'm not a UI specialist; to me it is an open question.</p><h3>Summary</h3><p>Capacitor, with or without the Ionic Framework on top, is the
always-bet-on-the-web solution to the cross-platform mobile application
development problem. It uses stock system WebViews and JavaScript,
augmented with native capabilities as needed. Consistency with the UX
of the specific platform (iOS or Android) may require significant
investment, though.</p><p>Performance-wise, my evaluation is that Capacitor apps are
well-positioned for peak JS performance due to JIT code generation and
low jank due to offloading animations to the web platform, but may have
problems with startup latency, as Capacitor needs to parse and compile a
possibly significant amount of JavaScript each time it is run.</p><p>Next time, a different beast: <a href="https://wingolog.org/archives/2023/04/21/structure-and-interpretation-of-react-native">React Native</a>. Until then, happy hacking!</p></div>2023-04-20T10:20:32+00:00Andy WingoSebastian Dröge: Building a GStreamer plugin in Rust with meson instead of cargo
https://coaxion.net/blog/2023/04/building-a-gstreamer-plugin-in-rust-with-meson-instead-of-cargo/
<p>Over the Easter holidays I spent some time on a little experiment: How hard is it to build a GStreamer plugin in <a href="https://www.rust-lang.org">Rust</a> with <a href="https://mesonbuild.com">meson</a> instead of using the default Rust build system <a href="https://doc.rust-lang.org/cargo/">cargo</a>?</p>
<p><a href="https://mesonbuild.com">meson</a> is a community-developed, open-source, cross-platform (including Windows), multi-language build system that is supposed to be fast and easy to use. It’s nowadays used as a build system by a lot of components of the <a href="https://www.gnome.org">GNOME</a> desktop, by <a href="https://gstreamer.freedesktop.org">GStreamer</a>, <a href="https://systemd.io">systemd</a>, <a href="https://mesa3d.org">Mesa</a>, <a href="https://x.org">X.org</a>, <a href="https://www.qemu.org">QEMU</a> and many other pieces of Free Software. Mostly for building C or C++ code, but also for configuring and installing Python or JavaScript code, Vala, and other languages.</p>
<p>Wouldn’t it be nice if we could also build software in Rust with it, build it together with existing code in other languages and have a common build system for it all? What would be the advantages and disadvantages of that, and what’s the current status of Rust support in meson? How much effort is it to make use of <a href="https://crates.io">all the existing 100k software packages</a> (“crates”) that are available for Rust?</p>
<p>Especially as most the projects mentioned before are looking into adopting Rust more or less seriously as a safer and more modern successor of C, these seem like useful questions to investigate. Anecdotally, I also heard that a maintainer of one of these projects said that being able to use the same build system as the rest of the codebase would be a requirement to even consider the language. Another project is starting to write some components in Rust and building them with meson, but without depending on any external Rust dependencies for now.</p>
<p>Another reason for looking into this was that there seems to be the opinion that you can’t really use any build system apart from cargo for building Rust code, and using meson would be very hard to impossible and involve a lot of disadvantages. This has lead to currently all GNOME applications written in Rust having a chimera of a build system using both meson and cargo because neither of the two does everything these applications need. Such a setup is hard to maintain, debug and probably almost nobody really understands it. cargo’s design does not make embedding into another build system easy, and both cargo and meson have very specific, and to some degree incompatible, opinions about how things have to be done. Let’s see if that’s actually necessary and what’s missing to move away from that. As Facebook is apparently using <a href="https://buck.build">buck</a> to build part of their Rust code, and Google <a href="https://bazel.build">bazel</a>, this shouldn’t really be impossible.</p>
<p>As I’m a <a href="https://gstreamer.freedesktop.org">GStreamer</a> developer, the maintainer of its <a href="https://gitlab.freedesktop.org/gstreamer/gstreamer-rs">Rust bindings</a> and the one who started writing plugins for it in <a href="https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs">Rust</a>, trying to do build a GStreamer plugin in Rust with meson instead of cargo seemed like the obvious choice for this experiment.</p>
<p>However, everything here applies in the same way to building <a href="https://gtk.org">GTK</a> applications with its <a href="https://gtk-rs.org">Rust bindings</a> or similarly to any of the software mentioned before for writing components of them in Rust.</p>
<p><strong>EDIT</strong>: After publishing the first version I was told that meson actually supports a couple of things that I missed before. For example, running Rust tests is already supported just fine (but more verbose in the build definition than cargo), and meson can already install executables setuid root by itself. Also there is an equivalent to <code>cargo init</code>, <code>meson init</code>, which makes starting new projects a bit more convenient. Also I wrote that <code class="EnlighterJSRAW">cargo clippy</code> is not really supported yet. While true, you can get around that by telling meson (or any other build system) to use <code class="EnlighterJSRAW">clippy-driver</code> as compiler instead of using <code>rustc</code>. I’ve updated the text below accordingly.</p>
<h2>Summary</h2>
<p>The code for my experiment can be found <a href="https://gitlab.freedesktop.org/slomo/gst-plugin-meson-test">here</a>, but at the point of writing it needs some changes to meson that are not merged yet. A list of those changes with some explanation can be found further below. The git repository also includes a cargo build system that gives the same build results for comparison purposes.</p>
<p>I should also make clear that this is only an experiment at this point and while it works fine, it is more manual work than necessary and if you depend on a lot of existing crates from <a href="https://crates.io">crates.io</a> then you probably want to wait a bit more before considering meson. Also more on that later. However, if you won’t have to depend on a lot of crates, your codebase is relatively self-contained and maybe even has to be built together with C code, then meson is already a viable alternative and has some advantages to offer compared to cargo. But also some disadvantages.</p>
<p>Almost all of the manual work I did as part of this experiment can be automated, and a big part of that is not going to be a lot of work either. I didn’t do that here to get an idea of the problems that would actually be encountered in practice when implementing such an automated system. I’ll get to that in more detail at the very end.</p>
<p>In summary I would say that</p>
<ul>
<li>meson is less convenient and less declarative than cargo, but in exchange more powerful and flexible</li>
<li>meson’s Rust support is not very mature yet and there’s very little tooling integration</li>
<li>cargo is great and easy to use if your project falls into the exact pattern it handles but easily becomes annoying for you and your users otherwise</li>
<li>the developer experience is much better with cargo currently but the build and deployment experience is better with meson</li>
</ul>
<p>More on each of these items below.</p>
<h2>Procedure</h2>
<p>As a goal I wanted to build one of the parts of the <a href="https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/tree/main/tutorial/src/identity">gst-plugins-rs</a> tutorial, specifically an <code>identity</code>-kind of element that simply passes through its input, and build that into a GStreamer plugin that can be loaded into any GStreamer process. For that it has to be built into <a href="https://doc.rust-lang.org/reference/linkage.html"><code>cdylib</code></a> in Rust terms: a shared library that offers a C ABI. For this purpose, meson already has a big advantage over cargo in that it can actually install the build result in its correct place while cargo can only install executables right now. But more on that part later too.</p>
<p>The main task for this was translating all the Rust crate dependencies from cargo to meson, manually one by one. Overall 44 dependencies were needed, and I translated them into so-called meson <a href="https://mesonbuild.com/Wrap-dependency-system-manual.html">wrap</a> dependencies. The wrap dependency system of meson, similar to cargo, allows to download source code of dependencies from another location (in this case crates.io) and to extend it with a patch, in this case to add the meson-based build system for each.</p>
<p>In practice this meant creating a lot of <code class="EnlighterJSRAW">meson.build</code> files based on the <code class="EnlighterJSRAW">Cargo.toml</code> of each crate. The following following is the <code class="EnlighterJSRAW">meson.build</code> for the <a href="https://crates.io/crates/toml_edit"><code>toml_edit</code></a> crate.</p>
<pre class="EnlighterJSRAW">
project('toml_edit-rs', 'rust',
version : '0.19.8',
meson_version : '>= 1.0.0',
default_options : ['buildtype=debugoptimized',
'rust_std=2021'
]
)
rustc = meson.get_compiler('rust')
toml_datetime_dep = dependency('toml_datetime-rs', version : ['>= 0.6', '< 0.7'])
winnow_dep = dependency('winnow-rs', version : ['>= 0.4', '< 0.5'])
indexmap_dep = dependency('indexmap-rs', version : ['>= 1.9.1', '< 2.0'])
features = []
features += ['--cfg', 'feature="default"']
toml_edit = static_library('toml_edit', 'src/lib.rs',
rust_args : features,
rust_crate_type : 'rlib',
dependencies : [toml_datetime_dep, winnow_dep, indexmap_dep],
pic : true,
)
toml_edit_dep = declare_dependency(link_with : toml_edit)
</pre>
<p>and the corresponding wrap file</p>
<pre class="EnlighterJSRAW">
[wrap-file]
directory = toml_edit-0.19.8
source_url = https://crates.io/api/v1/crates/toml_edit/0.19.8/download
source_filename = toml_edit-0.19.8.tar.gz
source_hash = 239410c8609e8125456927e6707163a3b1fdb40561e4b803bc041f466ccfdc13
diff_files = toml_edit-rs.meson.patch
[provide]
toml_edit-rs = toml_edit_dep
</pre>
<p>As can be seen from the above, this could all be autogenerated from the corresponding <a href="https://docs.rs/crate/toml_edit/0.19.8/source/Cargo.toml"><code>Cargo.toml</code></a> and that’s the case for a lot of crates. There are also some plans and ideas since quite a while in the meson community to actually develop such a tool, but so far this hasn’t materialized yet. Maybe my experiment can provide some motivation to actually start with that work.</p>
<p>For simplicity, when translating these by hand I didn’t consider including</p>
<ul>
<li>any optional dependencies unless needed for my tasks</li>
<li>any tests, examples, executables as part of the crate</li>
<li>any cargo <a href="https://doc.rust-lang.org/cargo/reference/features.html">feature</a> configuration instead of exactly what I needed</li>
</ul>
<p>All of this can be easily done with meson though, and an automated tool for translating <code class="EnlighterJSRAW">Cargo.toml</code> into meson wraps should easily be able to handle that. For cargo features there are multiple ways of mapping them to meson, so some conventions have to be defined first of all. Similarly for naming Rust crates as meson dependencies it will be necessary to define some conventions to allow sharing between projects.</p>
<p>The hard part for translating some of the crates were the cargo <a href="https://doc.rust-lang.org/cargo/reference/build-scripts.html">build.rs</a> build scripts. These build scripts allow running arbitrary Rust code as part of the build, which will also make automatic translation challenging. More on that later.</p>
<p>Once I had translated all the 44 dependencies, including the GStreamer Rust bindings, various <a href="https://doc.rust-lang.org/reference/procedural-macros.html">procedural Rust macros</a> and a lot of basic crates of the Rust ecosystem, I copied the plugin code into the new repository, changed some minor things and could then write a <code class="EnlighterJSRAW">meson.build</code> file for it.</p>
<pre class="EnlighterJSRAW">
project('gst-plugin-meson-test', 'rust',
version : '0.0.1',
meson_version : '>= 1.0.0',
default_options : ['buildtype=debugoptimized',
'rust_std=2021'
]
)
plugins_install_dir = '@0@/gstreamer-1.0'.format(get_option('libdir'))
rustc = meson.get_compiler('rust')
add_global_arguments(
'-C', 'embed-bitcode=no',
language: 'rust'
)
gst_dep = dependency('gstreamer-rs', version : ['>= 0.20', '< 0.21'])
once_cell_dep = dependency('once_cell-rs', version : ['>= 1.0', '< 2.0'])
gst_plugin_meson_test = dynamic_library('gstmesontest', 'src/lib.rs',
rust_crate_type : 'cdylib',
rust_dependency_map : {
'gstreamer' : 'gst',
},
dependencies : [gst_dep, once_cell_dep],
install : true,
install_dir : plugins_install_dir,
)
</pre>
<p>To get everything building like this with the latest meson version (1.1.0), at this point some additional changes are needed: <a href="https://github.com/mesonbuild/meson/pull/11674">1</a>, <a href="https://github.com/mesonbuild/meson/pull/11676">2</a>, <a href="https://github.com/mesonbuild/meson/pull/11681">3</a> and <a href="https://github.com/mesonbuild/meson/pull/11693">4</a>. Also currently all of this only supports native compilation. Cross-compilation of <code class="EnlighterJSRAW">proc-macro</code> in meson is currently <a href="https://github.com/mesonbuild/meson/issues/11702">broken</a> but should be not too hard to fix in the future.</p>
<p>A careful reader will also notice that currently all crates with a dash (<code>-</code>) in their name are named like that for the dependency name, but the library they’re building has the dash replaced by an underscore in its name. This is due to a <a href="https://github.com/rust-lang/rust/issues/110460">rustc</a> bug (or undocumented behaviour?), and meson <a href="https://github.com/mesonbuild/meson/pull/11698">will likely require</a> this translation of forbidden characters in crate names for the foreseeable future.</p>
<p>With all that in place, building the project is a matter of running</p>
<pre class="EnlighterJSRAW">
$ meson builddir
$ ninja -C builddir
</pre>
<p>compared to a single command with cargo</p>
<pre class="EnlighterJSRAW">
$ cargo build
</pre>
<p>Note that the meson build will be slower because by default meson already builds with optimizations (<code>-C opt-level=2</code>) while cargo does not.</p>
<h2>Comparison between cargo and meson</h2>
<p>In the following section I’m going to compare some aspects of cargo and meson, and give my general impression of both. To take the conclusion ahead, my perfect build system would include aspects of both cargo and meson and currently both are lacking in their own way. Like with any other tool (or programming language, for that matter): if you don’t have anything to complain about it you don’t know it well enough yet or don’t know any of the alternatives.</p>
<p>To avoid that people focus on the negative aspects, I’m also going to try to describe all the shortcomings of one build system as a good feature of the other. This is no competition but both meson and cargo can learn a lot from each other.</p>
<h3>Build Times</h3>
<p>Above I shortly mentioned build times. And because that’s something everybody is interested in and a common complaint about Rust, let’s start with that topic even if it’s in my opinion the most boring one.</p>
<p>Overall you would expect both build systems to behave the same if they do the same work as they are both basically just sophisticated Rust compiler process spawners. If you want to improve build times then your time is probably spent better on the Rust compiler itself and <a href="https://llvm.org">LLVM</a>.</p>
<p>All times below are measured on my system very unscientifically with <code>time</code>. This is all only to give an idea of the general behaviour and to check if there are conceptual inefficiencies or problems. Also, when reproducing these results make sure to pass <code class="EnlighterJSRAW">-Dbuildtype=debug</code> to meson for comparable results between meson and cargo.</p>
<ul>
<li>Running meson (build configuration): 1.4s (1.0s user, 0.4s system)</li>
<li>Running ninja (actual build): 10.4s (23.5s user, 3.2s system)</li>
<li>Running cargo: 11.0s (34.1s user, 6.6s system)</li>
</ul>
<p>One thing that shows here immediately is that both need approximately the same time. The build alone takes slightly less than cargo, the configure and build steps together slightly more. So far what would be expected. However, cargo uses almost 45% more CPU time. I didn’t investigate this in great detail but the main two reasons for that are likely</p>
<ul>
<li>cargo is building and running a <code class="EnlighterJSRAW">build.rs</code> build script in 23 of the 44 dependencies, which takes a while and also pulls in some more dependencies that are not needed for the meson build</li>
<li>meson is currently <a href="https://github.com/mesonbuild/meson/issues/11695">parallelizing less well</a> compared to cargo when building Rust code, or otherwise the build step would likely be 10% or more faster</li>
</ul>
<h3>cargo build scripts</h3>
<p>Interestingly, the main pain point when translating the crate build systems from cargo to meson also seems to be the main factor for cargo being more inefficient than it could be. This also seems to be a known fact in the <a href="https://toot.cat/@imperio/110179549937549435">wider Rust community</a> by now.</p>
<p>But in addition to being inefficient and painful to translate (especially automatically), it is in my opinion also a maintenance nightmare and literally the worst part of cargo. It’s not declarative in what the build script is actually trying to do, it’s very easy to write build scripts that don’t work correctly in other environments and two crates doing the same things in a build script are generally going to behave differently in non-obvious ways.</p>
<p>For all the crates that I translated, the reasons why the build scripts existed in 23 of 44 crates should really be features that cargo should provide directly and that meson can do directly:</p>
<ul>
<li>Checking which version of the Rust compiler is used and based on that enabling/disabling various features</li>
<li>Checking features or versions of the underlying platform or operating system</li>
<li>Checking for existence of native, external libraries or even building them</li>
<li>Not in these cases: code generation</li>
</ul>
<p>Especially the last two points are painful for build and deployment of Rust code, and that every crate has its own special way of solving it in a build script doesn’t make it better. And in the end both tasks are exactly what you have a build system for: building things, tracking dependencies between them and generating an ideal build plan or schedule.</p>
<p>The <a href="https://crates.io/crates/system-deps"><code>system-deps</code></a> crate is providing a way for expressing external dependencies in a declarative way as part of <code>Cargo.toml</code>, and a similar system built into cargo and integrated with the platform’s mechanism for finding libraries would be a huge improvement. Similar approaches for the other aspects would also be helpful. And not just for making a different build system for crates, but also for developers using cargo.</p>
<p>I’m sure that once cargo gained features for handling the four items above there will still be a need for custom build scripts in some situations, but these four items should cover more than 95% of the currently existing build scripts. It’s a mystery to me why there are no apparent efforts being made to improve cargo in this regard.</p>
<h3>Good Parts of meson</h3>
<p>Now let’s focus on some of the good parts of meson in comparison to cargo.</p>
<h4>More flexible and expressive</h4>
<p>Generally, meson has a much more flexible and expressive build definition language. It looks more like a scripting language than the toml files used by cargo, and as such appears more complex.</p>
<p>However thanks to this approach, almost anything for which cargo requires custom build scripts, with the disadvantages listed above, can be written directly as part of the meson build definitions. Expressing many of these things in toml for cargo would likely become convoluted and not very straightforward, as can already be seen nowadays with e.g. <a href="https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#platform-specific-dependencies">platform-specific dependencies</a>.</p>
<p>meson provides built-in <a href="https://mesonbuild.com/Modules.html">modules</a> with support for e.g.</p>
<ul>
<li>finding and checking versions of compilers of different languages and other build tools, e.g. for code generation, including the Rust <a href="https://github.com/rust-lang/rust-bindgen">bindgen</a> tool</li>
<li>testing compilation of code snippets</li>
<li>finding external library dependencies and checking their versions</li>
<li>defining shared/static library, excutable and code generation build targets together with how (and if) they should be installed later</li>
<li>installing (and generating) data files, including specific support for e.g. library metadata (<a href="https://www.freedesktop.org/wiki/Software/pkg-config/">pkg-config</a>), documentation, <a href="https://gi.readthedocs.io/en/latest/">gobject-introspection</a>, …</li>
</ul>
<p>As an escape hatch, whenever something can’t be expressed by just meson, it is also possible to run external configuration/build/install scripts that could be written as a Python/shell script, C or Rust executable, or anything else really. This is rarely needed though and the meson project seems to add support for anything that people actually need in practice.</p>
<h4>Build configuration system</h4>
<p>meson provides an extensible configuration mechanism for the build, which allow the user building the software to customize the build process and its results.</p>
<ol>
<li>The <a href="https://mesonbuild.com/Builtin-options.html">built-in options</a> allow for configuring things like switching between debug and release builds, defining toolchains and defining where to install build results.</li></ol>
<li>
<p>The <a href="https://mesonbuild.com/Build-options.html">build options</a> allow each project to extend the above with custom configuration, e.g. for enabling/disabling optional features of the project or generally selecting between different configurations. Apart from simple boolean flags this also allows for other types of configuration, including integers and strings.</p>
</li>
<p>cargo only provides one of the two in the form of fixed, non-extensible <a href="https://doc.rust-lang.org/cargo/reference/profiles.html">compiler profiles</a> and <a href="https://doc.rust-lang.org/cargo/reference/config.html">configuration</a>.</p>
<p>The second is not to be confused with cargo’s <a href="https://doc.rust-lang.org/cargo/reference/features.html">feature</a> flags, which are more a mechanism for selecting configuration of dependencies by the developer of the software. The meson build configuration however is for the builder of the software to select different configurations. While for executables cargo feature flags are used in a similar way for boolean configurations, cargo does not provide anything for this in general.</p>
<p>As a workaround for this, a couple of Rust crates are using environment variables together with the <a href="https://doc.rust-lang.org/stable/std/macro.env.html"><code>env!</code></a> macro for build configuration but this is fragile and not discoverable, and more of a hack than a real solution.</p>
<h4>Support for Rust and non-Rust code in the same project</h4>
<p>meson supports mixing Rust and non-Rust code in the same project, and allows tracking dependencies between targets using different languages in the same way.</p>
<p>While it is not supported to mix Rust and e.g. C code in the same build target due to Rust’s compilation model, it is possible to e.g. build a static Rust library, a static C library and link both together into e.g. a <a href="https://dlang.org">D</a> application or Python module. An example for this would be <a href="https://github.com/jpakkane/polysnake">this Python module</a> that combines C, C++, Rust and Fortran code.</p>
<p>Code generation can be handled in a similar way as in the end code generation is just another transformation from one format into another.</p>
<p>cargo doesn’t directly support anything but Rust code. As usual, build scripts provide a mechanism to get around this limitation. The <a href="https://crates.io/crates/cc">cc</a> crate for example is widely used to build C code and there are also crates for building <a href="https://crates.io/crates/meson-next">meson</a> or <a href="https://crates.io/crates/cmake">cmake</a> based software as part of the build process.</p>
<p>All of this is completely opaque to cargo though and can’t be taken into account for defining an optimal build schedule, can’t be configured from the outside and regularly fails in non-standard build situations (e.g. cross-compilation).</p>
<h4>Installation of other files than executables</h4>
<p>meson allows every build result to be installed in a configurable location. This is especially useful for more complex applications that might have to provide various data files or come as an executable plus multiple libraries or plugins, or simply for projects that only provide a shared library. If any of the built-in installation mechanisms are not sufficient (e.g. the executable should be get specific process capabilities set via <a href="https://linux.die.net/man/8/setcap"><code>setcap</code></a>), meson also allows to customize the install process via scripts.</p>
<p>cargo only allows to install executables right now. There are cargo extensions that also allow for more complex tasks, e.g. <a href="https://github.com/matklad/cargo-xtask">cargo xtask</a> but there is no standard mechanism. There once was an <a href="https://github.com/rust-lang/rfcs/pull/2376">RFC</a> to make cargo’s installation process extensible, but the way how this was proposed would also suffer from the same problems as the cargo build scripts.</p>
<h4>External dependency and library support</h4>
<p>In addition to mixing build targets with multiple languages in the same project, meson also has a mechanism to find external dependencies in different ways. If an external dependency is not found, it can be provided and built as part of the project via the wrap mechanism mentioned before. The latter is similar to how cargo handles dependencies, but the former is missing completely and currently implemented via build scripts instead, e.g. by using the <a href="https://crates.io/crates/pkg-config">pkg-config</a> crate.</p>
<p>As Rust does not currently provide a stable ABI and provides no standard mechanism to locate library crates on the system, this mostly applies to library dependencies written in other languages. meson does support building Rust shared/static libraries and installing them too, but because of the lack of a stable ABI this has to be made use of very carefully.</p>
<p>On the other hand, Rust allows building shared/static libraries that provide the stable C ABI of the platform (<code>cdylib</code>, <code class="EnlighterJSRAW">staticlib</code> crate types). meson allows building these correctly too, and also offers mechanisms for installing them together with their (potentially <a href="https://github.com/eqrion/cbindgen/">autogenerated</a> header files) and locating them again later in other projects via e.g. pkg-config.</p>
<p>For cargo this job can be taken care of by <a href="https://github.com/lu-zero/cargo-c">cargo-c</a>, including actually building shared libraries correctly by setting e.g. the <a href="https://en.wikipedia.org/wiki/Soname">soname</a> <a href="https://github.com/rust-lang/cargo/issues/5045">correctly</a> and setting other kinds of versioning information.</p>
<h3>Good Parts of cargo</h3>
<p>After writing so much about meson and how great it is, let’s now look at some aspects of cargo that are better than what meson provides. Like I said before, both have their good sides.</p>
<h4>Simpler and more declarative build definitions</h4>
<p>The cargo <a href="https://doc.rust-lang.org/cargo/reference/manifest.html">manifest</a> format is clearly a lot simpler and more declarative than the meson build definition format.</p>
<p>For a simple project it looks more like a project description than something written in a scripting language.</p>
<pre class="EnlighterJSRAW">
[package]
name = "hello_world"
version = "0.1.0"
edition = "2021"
[dependencies]
anyhow = "1.0"
</pre>
<p>As long as a project stays in the boundaries of what cargo makes easy to express, which should be the case for the majority of existing Rust projects, it is going to be simpler than meson. The lack of various features in cargo that require the use of a build script prevent this currently for many crates, but that seems like something that could be easily improved.</p>
<p>meson on the other hand feels more like writing actual build scripts in some kind of scripting language, and information like “what dependencies does this have” are not as easily visible as from something like a <code>Cargo.toml</code>.</p>
<h4>Tooling integration</h4>
<p>cargo provides a lot development tools that make development with it a very convenient and smooth experience. There are also dozens of cargo extension commands that provide additional features on top of cargo.</p>
<p><code class="EnlighterJSRAW">cargo init</code> creates a new project, <code class="EnlighterJSRAW">cargo add</code> adds new dependencies, <code class="EnlighterJSRAW">cargo check</code> type-checks code without full compilation, <code class="EnlighterJSRAW">cargo clippy</code> runs a powerful linter, <code class="EnlighterJSRAW">cargo doc</code> builds the documentation (incl. dependencies), <code class="EnlighterJSRAW">cargo bench</code> and <code class="EnlighterJSRAW">cargo test</code> allow running tests and benchmarks, <a href="https://crates.io/crates/cargo-show-asm"><code>cargo show-asm</code></a> shows the generated, annotated assembly for the code, <a href="https://crates.io/crates/cargo-udeps"><code>cargo udeps</code></a> finds unused dependencies, …</p>
<p>All of this makes development of Rust project a smooth and well-integrated experience.</p>
<p>In addition <a href="https://github.com/rust-lang/rust-analyzer/">rust-analyzer</a> provides a lot of IDE features via the LSP protocol for various editors and IDEs.</p>
<p>Right now, IDEs and editors are assuming Rust projects to make use of cargo and offer integration with its features.</p>
<p>On the other hand, when using meson almost none of this is currently provided and development feels less well integrated. Right now the only features provided by meson for making Rust development easier is generation of a <a href="https://rust-analyzer.github.io/manual.html#non-cargo-based-projects"><code>rust-project.json</code></a> to make use of rust-analyzer and being able to run tests in a similar way to cargo, and of course actually building the code. Building of documentation could be easily added to meson and is supported for other languages, something like <code class="EnlighterJSRAW">cargo add</code> for wrap dependencies exists already and adding crates.io support to it would be possible, but it’s going to take a while and a bit of effort to handle crates with cargo build scripts. Making use of all the cargo extension commands without actually using cargo seems unrealistic.</p>
<p>In the end, cargo is the default build system for Rust and everything currently assumes usage of cargo so using cargo offers the best developer experience.</p>
<h4>Rust dependencies</h4>
<p>As shortly mentioned above, <code class="EnlighterJSRAW">cargo add</code> makes it extremely easy to add new Rust dependencies to a project and build them as part of the project. This helps a lot with code reuse and modularity. That an average Rust project has dozens of direct dependencies and maybe a hundred or more indirect dependencies shows this quite clearly, as does the encapsulation of very small tasks in separate dependencies compared to huge multi-purpose libraries that are common with other languages.</p>
<p>cargo also directly handles updating of dependencies via <code>cargo update</code>, including making sure that only <a href="https://semver.org">semver</a> compatible versions are automatically updated and allows including multiple incompatible versions of dependencies in the same build if necessary.</p>
<p>In addition to just adding dependencies, <a href="https://doc.rust-lang.org/cargo/reference/features.html">cargo features</a> allow for conditional compilation and for defining which parts and features of a dependency should be enabled or not.</p>
<p>meson has some explicit handling of dependencies and the wrap system also allows building external dependencies as part of the project, but adding or updating dependencies is more manual process unless they are in the meson <a href="https://mesonbuild.com/Wrapdb-projects.html">wrapdb</a>. For now it is completely manual with regards to Rust crates. It is no different from adding dependencies in non-Rust languages though.</p>
<p>There is also no direct equivalent of the cargo feature flags, which potentially seems like a useful addition to meson.</p>
<h2>Next steps</h2>
<p>Considering all of the above, there is not really a simple answer to which of the two choices is the best for your project. As of now I would personally always use cargo for Rust projects if I can, especially if they will have other Rust dependencies. It’s a lot more convenient to develop with cargo.</p>
<p>However various features of meson might make it a better choice for some projects, maybe already now or at least in the future. For example the ability to handle multiple languages, to handle dependencies and shared libraries correctly or the ability to install data files together with the application. Or simply because the remainder of the project already uses meson for compiling e.g. C code, meson might be the more natural choice for adding Rust code to the mix.</p>
<p>As outlined above there are many areas of meson that could be improved and where Rust support is not very mature yet, and to make meson a more viable alternative to cargo these will have to happen sooner or later. Similarly, there are various areas where cargo also could be improved and learn from meson, and where improvements to cargo, in addition to making cargo more flexible and easier to use, would also make it easier to handle other build systems.</p>
<p>From a meson point of view, I would consider the following the next steps.</p>
<h3>Various bugs and missing features</h3>
<h4>rustc and cargo</h4>
<p>On the Rust compiler side, there are currently two minor issues that would need looking into.</p>
<ul>
<li><a href="https://github.com/rust-lang/rust/issues/80792">rustc#80792</a>: Adding support for passing environment variables via the command-line instead of using actual environment variables. Environment variables are currently used for build configuration by many crates. Allowing to provide them via the command-line would allow for more clean and reliable build rules without accidentally leaking actual environment variables into the build.</li>
<li><a href="https://github.com/rust-lang/rust/issues/110460">rustc#110460</a>: Undocumented and maybe unintentional library filename requirement based on the crate name. This currently requires <a href="https://github.com/mesonbuild/meson/pull/11698">meson to disallow dashes</a> in crate names and to have them explicitly replaced by underscore, which is something cargo does implicitly. Implicit conversion inside meson would also be feasible but probably not desireable because there would then be a mismatch between the name of the build target in the build definition and the actual name of the build target when building it. Similar to the mild confusion that some people ran into when noticing that a crate with a dash in the name can only be referenced with underscores from Rust code.</li>
</ul>
<p>In addition it would be useful to look into moving various features from build scripts into proper cargo, as outlined above. This will probably need a lot of design and discussion effort, and will likely also take years after implementation until the important crates are moved to it due to very conservative Rust toolchain version requirement policies in various of these crates.</p>
<p>So on the cargo side that’s more something for the long run but also something that would greatly benefit users of cargo.</p>
<h4>meson</h4>
<p>On the meson side there are a couple of bugs of different severity that will have to be solved, and probably more that will show up once more people are starting to use meson to build Rust projects.</p>
<p>In addition to the ones I mentioned above and that are merged already into the meson git repository, at the time of this writing there were for example the following outstanding issues</p>
<ul>
<li><a href="https://github.com/mesonbuild/meson/pull/11681">meson#11681</a>: Add a feature to meson to allow renaming crates when using as a dependency. This is used throughout the Rust ecosystem for handling multiple versions of the same crate at once, or also simply for having a more convenient local name of a dependency.</li>
<li><a href="https://github.com/mesonbuild/meson/issues/11695">meson#11695</a>: Parallelize the Rust build better by already starting to build the next Rust targets when the metadata of the dependencies is available. This should bring build time improvements of 10% or more on machines with enough cores, and the lack of it is the main reason why the meson build in my experiment was “only” as fast as the cargo build and not faster.</li>
<li><a href="https://github.com/mesonbuild/meson/issues/11702">meson#11702</a>: Cross-compilation of <code class="EnlighterJSRAW">proc-macro</code> crates is currently using the wrong toolchain and simply doesn’t work.</li>
<li><a href="https://github.com/mesonbuild/meson/issues/11694">meson#11694</a>: Indirect dependencies of all dependencies are passed onwards on the later compiler invocations, which brings the risk of unnecessary name conflicts and simply causes more work for the compiler than necessary.</li>
<li><a href="https://github.com/mesonbuild/meson/pull/10030">meson#10030</a>: Add support for passing environment variables to the Rust compiler. As mentioned above, many crates are currently using this for build configuration so this would have to be supported by meson in one way or another.</li>
</ul>
<p>Apart from the second one in this list these should all be doable relatively fast and generally getting fixes and improvements required for Rust merged into meson was a fast and pleasant experience so far. I didn’t encounter any unnecessary bikeshedding or stop energy.</p>
<h3>Tooling for managing cargo dependencies</h3>
<p>During my experiment I wrote all the meson wrap files manually. This does not really scale, is inconvenient, and also makes it harder to update dependencies later.</p>
<p>The goal here would be to provide a tool in the shape of <code class="EnlighterJSRAW">cargo add</code> and <code class="EnlighterJSRAW">cargo update</code> that allows to automatically add cargo-based Rust dependencies to a meson project. This is something that was discussed a lot in the past and <a href="https://github.com/dcbaker">various</a> <a href="https://github.com/nirbheek">people</a> in the meson community have ideas and plans around this. meson already has something similar for the <a href="https://mesonbuild.com/Wrapdb-projects.html">wrapdb</a>, <code class="EnlighterJSRAW">meson wrap add</code> and <code>meson wrap update</code>, but the idea would be to have something similar (or integration into that) to directly support crates from crates.io so Rust dependencies can be added with as little effort to a meson project as they can currently to a cargo project.</p>
<p>Apart from the cargo build scripts this shouldn’t be a lot of effort and a project for an afternoon at most, so maybe I’ll give that a try one of these days if nobody beats me to it.</p>
<p>As part of such a tool, it will also be necessary to define conventions about naming, mapping of cargo features, versioning, etc. of Rust crates inside meson and this should ideally be done from early on to avoid unnecessary churn. The way how I did it as part of my experiment has various drawbacks with regards to versioning and needs improvements.</p>
<p>Handling cargo build scripts is a bigger issue though. As my experiment showed, about half of the crates had build scripts. While all of them were more or less trivial, automatically translating this into meson build definitions seems unrealistic to me.</p>
<p>It might be possible to have meson use the cargo build scripts directly in one way or another, or they would have to be translated manually to meson build definitions. The latter would considerably improve build times so seems like a better approach for common crates at least. And for those the meson build definitions could be stored in a central place, like the meson <a href="https://mesonbuild.com/Wrapdb-projects.html">wrapdb</a> or maybe even be included in the crates on crates.io if their maintainers feel like dealing with two build systems.</p>
<p>Together with all this, some thought will also have to be put into how to locate such Rust dependencies, similar to how pkg-config allows to locate shared libraries. For example, Linux distributions will want to package such dependencies and make sure that a project built for such a distribution is making use of the packaged dependencies instead of using any other version, or worse downloading some version from the Internet at build time. The way how this is currently handled by cargo is also not optimal for Linux distributions and a couple of other build and deployment scenarios.</p>
<p>Because of the lack of a stable Rust ABI this would mean locating Rust source code.</p>
<h3>Tooling integration</h3>
<p>And last, as mentioned above there is basically no tooling integration right now apart from being able to build Rust code and using rust-analyzer. meson should at least support the most basic tasks that cargo supports, and that meson already supports for other languages: running tests and benchmarks, running linters and building documentation.</p>
<p>Once those basic tasks are done, it might be worth investigating other tooling integration like the various cargo extension commands offer or extending those commands to handle other build systems, e.g. via the <code class="EnlighterJSRAW">rust-project.json</code> that rust-analyzer uses for that purpose.</p>2023-04-19T14:26:36+00:00Andy Wingo: sticking point
https://wingolog.org/archives/2023/04/18/sticking-point
<div><p>Good evening, gentle readers. A brief note tonight, on a sticky place.</p><p>See, I have too many projects right now.</p><p>In and of itself this is not so much of a problem as a condition. I know
my limits; I keep myself from burning out by shedding load, and there is
a kind of priority list of which projects keep adequate service levels.</p><p>First come the tiny humans that are in my care who need their butts
wiped and bodies translated to and from school or daycare and who --
well you know the old Hegelian trope, that the dialectic crank of
history doesn't turn itself, that it takes actions from people to
synthesize the thesis and the antithesis, and that even History itself
isn't always monotonic; in the same way, bedtime is a reality,
there are the material conditions of sleepiness and
you're-gonna-be-so-tired-tomorrow but without the World-Historical Actor
which whose name is Dada ain't nobody getting into pyjamas, not to
mention getting in bed and going to sleep.</p><p>Lower in the priority queue there is work, and work is precious: a whole
day, a day of just thinking about things and solving problems; yes, I
wish I could choose all of the problems that I work on but the simple
fact of being able to think and solve is a gift, or rather a reward,
time stolen from the arbitrary chaotic preliterate interruptors. I love
my kids -- and here I have to choose my conjunction wisely -- <i>and</i> also
they love me. Which is nice! They express this through wanting to
sit on my lap and demand that I read to them when I am thinking about things. We all have our love
languages, I suppose.</p><p>And the house? There are 5 of us now, and that's, you know, more than 5
times the work it was before because the 3 wee ones are more chaotic
than the adults. There is so much laundry. We are now a
cook-the-whole-500g-bag-of-pasta family, and we're far from the teenager
stage.</p><p>Finally there are the weird side projects, of which there are a few. I
have some sporty things I do that I can't avoid because I am socially
coerced into doing them; all in all a fine configuration. I have some
volunteer work, which has parts I like that take time, which I am fine
with, and parts I don't that I neglect. There's the garden, also
neglected but nice to muck about in sometimes. Sometimes I see friends?
Rarely, so rarely, but sometimes.</p><p>But, netfriends, if I find a free couple hours, I hack. Not less than a
couple hours, because otherwise I can't get into things. More? Yes please! But
usually no, because priorities.</p><p>I write this note because I realized that in the last month or so,
hain't been no hacktime; there's barely 30 minutes between the end of
the evening kitchen-clean and the onset of zzzz. That's fine, really;
ultimately I need to carve out work time for this sort of thing, somehow
or other.</p><p>But I was also realizing that I painted myself into a corner with my <a href="https://wingolog.org/archives/2023/02/07/whippet-towards-a-new-local-maximum">GC
work</a>.
See, I need some new benchmarks, and I can't afford to use Guile itself
as my benchmark yet, because I can't afford to fix deep bugs right now.
I need something small. Specifically I need a little language
implementation with efficient GC integration. I was looking at porting
the <a href="https://wingolog.org/archives/2022/08/18/just-in-time-code-generation-within-webassembly">little Scheme
implementation</a>
from my Wasm JIT work to C -- because the rest of whippet is in C -- but
is that a good enough reason? I got mired in the swamps of precise GC
roots in C and let me tell you, it is disgusting. You don't have RAII
like you do in C++, without GCC extensions. You don't have stack maps,
like you would like. I might be able to push through but it's the sort
of project you need a day for, and those days have not been forthcoming
recently.</p><p>Anyway, just a little hack log tonight, not detailing a success, but
rather a condition. Perhaps observing the position of this logjam will
change its velocity. Here's hoping, and until next time, happy hacking!</p></div>2023-04-18T20:23:49+00:00Andy WingoGStreamer: GStreamer 1.22.2 stable bug fix release
https://gstreamer.freedesktop.org/news/#2023-04-11T20:00:00Z
<p>
The GStreamer team is pleased to announce the second bug fix release
in the stable 1.22 release series of your favourite cross-platform
multimedia framework!
</p><p>
This release only contains bugfixes and it should be safe to update from
1.22.x.
</p><p>
<b>Highlighted bugfixes:</b>
<ul>
<li>avdec_h264: fix decoder deadlocks with FFmpeg 6.0</li>
<li>rtspsrc: fix regression with URI protocols in OPTIONS requests for RTSP over TLS</li>
<li>rtspsrc: improved control url handling compatibility for broken servers</li>
<li>decklink: fix 10 bit RGB (r210) format auto detection for capture and fix playout if video caps are configured before audio caps</li>
<li>d3d11videosink: Fix tearing in case of fullscreen mode</li>
<li>playbin: fix deadlock when stopping stream with subtitles visible (even more)</li>
<li>typefinding: fix regression not detecting application/dash+xml in some corner cases</li>
<li>osxvideosink: fix broken aspect ratio and frame drawing region</li>
<li>decodebin3, parsebin: Improve elementary stream handling when decoders are not present and fix hang when removing a failing stream</li>
<li>urisourcebin: Propagate sticky events from parsebin, so that the <tt>STREAM_START</tt> event with the <tt>GstStream</tt> info is always available when pads get exposed</li>
<li>v4l2: Add support for YVU420M format; mark JPEG content as parsed</li>
<li>h264decoder, h265decoder: DPB bumping process and latency reporting fixes</li>
<li>Opus: Fix reading of extended channel config in MPEG-TS and fix missing sample rate when remuxing from RTP to Matroska</li>
<li>zxing: add support for building against zxing-c++ 2.0</li>
<li>cerbero: Fix packaging of Rust plugins on Android; fix modern Gentoo distro detection</li>
<li>various bug fixes, memory leak fixes, and other stability and reliability improvements</li>
</ul>
</p><p>
See the <a href="https://gstreamer.freedesktop.org/releases/1.22/#1.22.2">GStreamer 1.22.2 release notes</a>
for more details.
</p><p>
Binaries for Android, iOS, Mac OS X and Windows will be available shortly.
</p><p>
Release tarballs can be downloaded directly here:
<ul>
<a href="https://gstreamer.freedesktop.org/src/gstreamer/gstreamer-1.22.2.tar.xz">gstreamer</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-base/gst-plugins-base-1.22.2.tar.xz">gst-plugins-base</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-good/gst-plugins-good-1.22.2.tar.xz">gst-plugins-good</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-ugly/gst-plugins-ugly-1.22.2.tar.xz">gst-plugins-ugly</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-bad/gst-plugins-bad-1.22.2.tar.xz">gst-plugins-bad</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-libav/gst-libav-1.22.2.tar.xz">gst-libav</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-rtsp-server/gst-rtsp-server-1.22.2.tar.xz">gst-rtsp-server</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-python/gst-python-1.22.2.tar.xz">gst-python</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-editing-services/gst-editing-services-1.22.2.tar.xz">gst-editing-services</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-devtools/gst-devtools-1.22.2.tar.xz">gst-devtools</a>,
<a href="https://gstreamer.freedesktop.org/src/gstreamer-vaapi/gstreamer-vaapi-1.22.2.tar.xz">gstreamer-vaapi</a>,
<a href="https://gstreamer.freedesktop.org/src/gstreamer-sharp/gstreamer-sharp-1.22.2.tar.xz">gstreamer-sharp</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-omx/gst-omx-1.22.2.tar.xz">gst-omx</a>, or
<a href="https://gstreamer.freedesktop.org/src/gstreamer-docs/gstreamer-docs-1.22.2.tar.xz">gstreamer-docs</a>.
</ul>
</p>2023-04-11T20:00:00+00:00Andy Wingo: a world to win: webassembly for the rest of us
https://wingolog.org/archives/2023/03/20/a-world-to-win-webassembly-for-the-rest-of-us
<div><p>Good day, comrades!</p><p>Today I'd like to share the good news that WebAssembly is finally coming
for the rest of us weirdos.</p><div>
<h1>A world to win</h1>
<p>WebAssembly for the rest of us</p>
<p>17 Mar 2023 – BOB 2023</p>
<p>Andy Wingo</p>
<p>Igalia, S.L.</p>
</div><p>This is a transcript-alike of a talk that I gave last week at <a href="https://bobkonf.de/2023/en/program.html">BOB
2023</a>, a gathering in Berlin of
people that are using "technologies beyond the mainstream" to get things
done: Haskell, Clojure, Elixir, and so on. PDF slides <a href="https://wingolog.org/pub/2023-bobkonf-wasm-for-the-rest-of-us-slides.pdf">here</a>, and I'll link the video too when it becomes available.</p><div>
<h2>WebAssembly, the story</h2>
<p>WebAssembly is an exciting new universal compute platform</p>
</div><p>WebAssembly: what even is it? Not a programming language that
you would write software in, but rather a compilation target: a sort of
assembly language, if you will.</p><div>
<h2>WebAssembly, the pitch</h2>
<p>Predictable portable performance</p>
<ul>
<li>Low-level</li>
<li>Within 10% of native</li>
</ul>
<p>Reliable composition via isolation</p>
<ul>
<li>Modules share nothing by default</li>
<li>No nasal demons</li>
<li>Memory sandboxing</li>
</ul>
<p>Compile your code to WebAssembly for easier distribution and composition</p>
</div><p>If you look at what the characteristics of WebAssembly are as an
abstract machine, to me there are two main areas in which it is an
advance over the alternatives.</p><p>Firstly it's "close to the metal" -- if you compile for example an
image-processing library to WebAssembly and run it, you'll get similar
performance when compared to compiling it to x86-64 or ARMv8 or what
have you. (For image processing in particular, native still generally
wins because the SIMD primitives in WebAssembly are more narrow and
because getting the image into and out of WebAssembly may imply a copy,
but the general point remains.) WebAssembly's instruction set covers a
broad range of low-level operations that allows compilers to produce
efficient code.</p><p>The novelty here is that WebAssembly is both portable while also being
successful. We language weirdos know that it's not enough to do
something technically better: you have to also succeed in getting
traction for your alternative.</p><p>The second interesting characteristic is that WebAssembly is (generally
speaking) a principle-of-least-authority architecture: a WebAssembly
module starts with access to nothing but itself. Any capabilities that
an instance of a module has must be explicitly shared with it by the
host at instantiation-time. This is unlike DLLs which have access to
all of main memory, or JavaScript libraries which can mutate global
objects. This characteristic allows WebAssembly modules to be reliably
composed into larger systems.</p><div>
<h2>WebAssembly, the hype</h2>
<p>It’s in all browsers! Serve your code to anyone in the world!</p>
<p>It’s on the edge! Run code from your web site close to your users!</p>
<p>Compose a library (eg: Expat) into your program (eg: Firefox), without risk!</p>
<p>It’s the new lightweight virtualization: Wasm is what containers were to VMs! Give me that Kubernetes cash!!!</p>
</div><p>Again, the remarkable thing about WebAssembly is that it is succeeding!
It's on all of your phones, all your desktop web browsers, all of the
content distribution networks, and in some cases it seems set to replace
containers in the cloud. Launch the rocket emojis!</p><div>
<h2>WebAssembly, the reality</h2>
<p>WebAssembly is a weird backend for a C compiler</p>
<p>Only some source languages are having success on WebAssembly</p>
<p>
What about Haskell, Ocaml, Scheme, F#, and so on – what about <i>us</i>?
</p>
<p>Are we just lazy? (Well...)</p>
</div><p>So why aren't we there? Where is Clojure-on-WebAssembly? Where are the
F#, the Elixir, the Haskell compilers? Some early efforts exist, but
they aren't really succeeding. Why is that? Are we just not putting in
the effort? Why is it that Rust gets to ride on the rocket ship but Scheme does not?</p><div>
<h2>WebAssembly, the reality (2)</h2>
<p>WebAssembly (1.0, 2.0) is not well-suited to garbage-collected languages</p>
<p>Let’s look into why</p>
</div><p>As it turns out, there is a reason that there is no good Scheme
implementation on WebAssembly: the initial version of WebAssembly is a
terrible target if your language relies on the presence of a garbage
collector. There have been some advances but this observation still
applies to the current standardized and deployed versions of
WebAssembly. To better understand this issue, let's dig into the guts
of the system to see what the limitations are.</p><div>
<h2>GC and WebAssembly 1.0</h2>
<p>Where do garbage-collected values live?</p>
<p>For WebAssembly 1.0, only possible answer: linear memory</p>
<pre>(module
(global $hp (mut i32) (i32.const 0))
(memory $mem 10)) ;; 640 kB</pre>
</div><p>The primitive that WebAssembly 1.0 gives you to represent your data is
what is called <i>linear memory</i>: just a buffer of bytes to which you can
read and write. It's pretty much like what you get when compiling
natively, except that the memory layout is more simple. You can obtain
this memory in units of 64-kilobyte pages. In the example above we're
going to request 10 pages, for 640 kB. Should be enough, right? We'll
just use it all for the garbage collector, with a bump-pointer
allocator. The heap pointer / allocation pointer is kept in the mutable
global variable <tt>$hp</tt>.</p><div>
<pre>
(func $alloc (param $size i32) (result i32)
(local $ret i32)
(loop $retry
(local.set $ret (global.get $hp))
(global.set $hp
(i32.add (local.get $size) (local.get $ret)))
(br_if 1
(i32.lt_u (i32.shr_u (global.get $hp) 16)
(memory.size))
(local.get $ret))
<b>(call $gc)</b>
(br $retry)))
</pre>
</div><p>Here's what an allocation function might look like. The allocation
function <tt>$alloc</tt> is like malloc: it takes a number of bytes and returns
a pointer. In WebAssembly, a pointer to memory is just an offset, which
is a 32-bit integer (<tt>i32</tt>). (Having the option of a 64-bit address
space is planned but not yet standard.)</p><p>If this is your first time seeing the text representation of a
WebAssembly function, you're in for a treat, but that's not the point of
the presentation :) What I'd like to focus on is the <tt>(call $gc)</tt> --
what happens when the allocation pointer reaches the end of the region?</p><div>
<h2>GC and WebAssembly 1.0 (2)</h2>
<p>
What hides behind <tt>(call $gc)</tt> ?
</p>
<p>Ship a GC over linear memory</p>
<p>Stop-the-world, not parallel, not concurrent</p>
<p>
But... <b>roots</b>.
</p>
</div><p>The first thing to note is that you have to provide the <tt>$gc</tt> yourself.
Of course, this is doable -- this is what we do when compiling to a
native target.</p><p>Unfortunately though the multithreading support in WebAssembly is
somewhat underpowered; it lets you share memory and use atomic
operations but you have to create the threads outside WebAssembly. In
practice probably the GC that you ship will not take advantage of
threads and so it will be rather primitive, deferring all collection
work to a stop-the-world phase.</p><div>
<h2>GC and WebAssembly 1.0 (3)</h2>
<p>Live objects are</p>
<ul>
<li>the roots</li>
<li>any object referenced by a live object</li>
</ul>
<p>Roots are globals and locals in active stack frames</p>
<p>
<b>No way to visit active stack frames</b>
</p>
</div><p>What's worse though is that you have no access to roots on the stack. A
GC has to keep live objects, as defined circularly as any object
referenced by a root, or any object referenced by a live object. It
starts with the roots: global variables and any GC-managed object
referenced by an active stack frame.</p><p>But there we run into problems, because in WebAssembly (any version, not
just 1.0) you can't iterate over the stack, so you can't find active
stack frames, so you can't find the stack roots. (Sometimes people want
to support this as a <a href="https://github.com/WebAssembly/exception-handling/issues/105">low-level
capability</a>
but generally speaking the consensus would appear to be that overall
performance will be better if the engine is the one that is responsible
for implementing the GC; but that is foreshadowing!)</p><div>
<h2>GC and WebAssembly 1.0 (3)</h2>
<p>Workarounds</p>
<ul>
<li>handle stack for precise roots</li>
<li>spill all possibly-pointer values to linear memory and collect conservatively</li>
</ul>
<p>Handle book-keeping a drag for compiled code</p>
</div><p>Given the noniterability of the stack, there are basically two
work-arounds. One is to have the compiler and run-time maintain an
explicit stack of object roots, which the garbage collector can know for
sure are pointers. This is nice because it lets you move objects. But,
maintaining the stack is overhead; the state of the art solution is
rather to create a side table (a "stack map") associating each potential
point at which GC can be called with instructions on how to find the
roots.</p><p>The other workaround is to spill the whole stack to memory. Or,
possibly just pointer-like values; anyway, you conservatively scan all
words for things that might be roots. But instead of having access to
the memory to which the WebAssembly implementation would spill your
stack, you have to do it yourself. This can be OK but it's sub-optimal;
see <a href="https://wingolog.org/archives/2023/02/07/whippet-towards-a-new-local-maximum">my recent post on the Whippet garbage
collector</a>
for a deeper discussion of the implications of conservative
root-finding.</p><div>
<h2>GC and WebAssembly 1.0 (4)</h2>
<p>Cycles with external objects (e.g. JavaScript) uncollectable</p>
<p>A pointer to a GC-managed object is an offset to linear memory, need capability over linear memory to read/write object from outside world</p>
<p>No way to give back memory to the OS</p>
<p>Gut check: gut says no</p>
</div><p>If that were all, it would already be not so great, but it gets worse!
Another problem with linear-memory GC is that it limits the potential
for composing a number of modules and the host together, because the
garbage collector that manages JavaScript objects in a web browser knows
nothing about your garbage collector over your linear memory. You can
easily create memory leaks in a system like that.</p><p>Also, it's pretty gross that a reference to an object in linear memory
requires arbitrary read-write access over all of linear memory in order
to read or write object fields. How do you build a reliable system
without invariants?</p><p>Finally, once you collect garbage, and maybe you manage to compact
memory, you can't give anything back to the OS. There are proposals in
the works but they are not there yet.</p><p>If the BOB audience had to choose between <a href="https://www.dreamsongs.com/WorseIsBetter.html">Worse is Better and The Right
Thing</a>, I think the BOB
audience is much closer to the Right Thing. People like that feel
instinctual revulsion to ugly systems and I think GC over linear memory
describes an ugly system.</p><div>
<h2>GC and WebAssembly 1.0 (5)</h2>
<p>
<i>There is already a high-performance concurrent parallel compacting GC in the browser</i>
</p>
<p>Halftime: C++ N – Altlangs 0</p>
</div><p>The kicker is that WebAssembly 1.0 requires you to write and deliver a
terrible GC when there is already probably a great GC just sitting there
in the host, one that has hundreds of person-years of effort invested in
it, one that will surely do a better job than you could ever do.
WebAssembly as hosted in a web browser should have access to the
browser's garbage collector!</p><p>I have the feeling that while those of us with a soft spot for languages
with garbage collection have been standing on the sidelines, Rust and
C++ people have been busy on the playing field scoring goals. Tripping
over the ball, yes, but eventually they do manage to make within
striking distance.</p><div>
<h2>Change is coming!</h2>
<p>Support for built-in GC set to ship in Q4 2023</p>
<p>With GC, the material conditions are now in place</p>
<p>
Let’s compile <i>our</i> languages to WebAssembly
</p>
</div><p>But to continue the sportsball metaphor, I think in the second half our
players will finally be able to get out on the pitch and give it the
proverbial 110%. Support for garbage collection is coming to
WebAssembly users, and I think even by the end of the year it will be
shipping in major browsers. This is going to be big! We have a chance
and we need to sieze it.</p><div>
<h2>Scheme to Wasm</h2>
<p>Spritely + Igalia working on Scheme to WebAssembly</p>
<p>Avoid truncating language to platform; bring whole self</p>
<ul>
<li>
<b>Value representation</b>
</li>
<li>Varargs</li>
<li>Tail calls</li>
<li>Delimited continuations</li>
<li>Numeric tower</li>
</ul>
</div><p>Even with GC, though, WebAssembly is still a weird machine. It would
help to see the concrete approaches that some languages of interest
manage to take when compiling to WebAssembly.</p><p>In that spirit, the rest of this article/presentation is a walkthough of
the approach that I am taking as I work on a WebAssembly compiler for
Scheme. (Thanks to <a href="https://spritely.institute">Spritely</a> for supporting
this work!)</p><p>Before diving in, a meta-note: when you go to compile a language to,
say, JavaScript, you are mightily tempted to cut corners. For example
you might implement numbers as JavaScript numbers, or you might omit
implementing continuations. In this work I am trying to not cut
corners, and instead to implement the language faithfully. Sometimes
this means I have to work around weirdness in WebAssembly, and that's
OK.</p><p>When thinking about Scheme, I'd like to highlight a few specific areas
that have interesting translations. We'll start with value
representation, which stays in the GC theme from the introduction.</p><div>
<h2>Scheme to Wasm: Values</h2>
<pre>
;; any extern func
;; |
;; eq
;; / | \
;; <b>i31</b> <b>struct</b> array
</pre>
<p>
The unitype: <tt>(ref eq)</tt>
</p>
<p>
Immediate values in <tt>(ref i31)</tt>
</p>
<ul>
<li>fixnums with 30-bit range</li>
<li>chars, bools, etc</li>
</ul>
<p>
Explicit nullability: <tt>(ref null eq)</tt> vs <tt>(ref eq)</tt>
</p>
</div><p>The GC extensions for WebAssembly are phrased in terms of a type system.
Oddly, there are three top types; as far as I understand it, this is the
result of a compromise about how WebAssembly engines might want to
represent these different kinds of values. For example, an opaque
JavaScript value flowing into a WebAssembly program would have type
<tt>(ref extern)</tt>. On a system with <a href="https://wingolog.org/archives/2011/05/18/value-representation-in-javascript-implementations">NaN
boxing</a>,
you would need 64 bits to represent a JS value. On the other hand a
native WebAssembly object would be a subtype of <tt>(ref any)</tt>, and might
be representable in 32 bits, either because it's a 32-bit system or
because of pointer compression.</p><p>Anyway, three top types. The user can define subtypes of <tt>struct</tt> and
<tt>array</tt>, instantiate values of those types, and access their fields.
The life cycle of reference-typed objects is automatically managed by
the run-time, which is just another way of saying they are
garbage-collected.</p><p>For Scheme, we need a common supertype for all values: <a href="https://medium.com/@samth/on-typed-untyped-and-uni-typed-languages-8a3b4bedf68c">the unitype, in
Bob Harper's memorable
formulation</a>.
We can use <tt>(ref any)</tt>, but actually we'll use <tt>(ref eq)</tt> -- this is the
supertype of values that can be compared by (pointer) identity. So now
we can code up <tt>eq?</tt>:</p><pre class="pre-scheme">(func $eq? (param (ref eq) (ref eq))
(result i32)
(ref.eq (local.get a) (local.get b)))
</pre><p>Generally speaking in a Scheme implementation there are <i>immediates</i> and
<i>heap objects</i>. Immediates can be encoded in the bits of a value,
whereas for heap object the bits of a value encode a reference (pointer)
to an object on the garbage-collected heap. We usually represent small
integers as immediates, as well as booleans and other oddball values.</p><p>Happily, WebAssembly gives us an immediate value type, <tt>i31</tt>. We'll
encode our immediates there, and otherwise represent heap objects as
instances of <tt>struct</tt> subtypes.</p><div>
<h2>Scheme to Wasm: Values (2)</h2>
<p>
Heap objects subtypes of <tt>struct</tt>; concretely:
</p>
<pre>
(struct $heap-object
(struct (field $tag-and-hash i32)))
(struct $pair
(<b>sub $heap-object</b>
(struct <b>i32</b> (ref eq) (ref eq))))
</pre>
<p>GC proposal allows subtyping on structs, functions, arrays</p>
<p>Structural type equivalance: explicit tag useful</p>
</div><p>We actually need to have a common struct supertype as well, for two
reasons. One is that we need to be able to hash Scheme values by
identity, but for this we need an embedded lazily-initialized hash
code. It's a bit annoying to take the per-object memory hit but it's a
reality, and the JVM does it this way, so it must not be so terrible.</p><p>The other reason is more subtle: WebAssembly's type system is built in
such a way that types that are "structurally" equivalent are
indistinguishable. So a pair has two fields, besides the hash, but
there might be a number of other fundamental object types that have the
same shape; you can't fully rely on WebAssembly's dynamic type checks
(<tt>ref.test</tt> et al) to be able to query the type of a value. Instead we
re-use the low bits of the hash word to include a type tag, which might
be 1 for pairs, 2 for vectors, 3 for closures, and so on.</p><div>
<h2>Scheme to Wasm: Values (3)</h2>
<pre>
(func $cons (param (ref eq)
(ref eq))
(result (ref $pair))
(<b>struct.new_canon</b> $pair
;; Assume heap tag for pairs is 1.
(i32.const 1)
;; Car and cdr.
(local.get 0)
(local.get 1)))
(func $%car (param <b>(ref $pair)</b>)
(result (ref eq))
(struct.get $pair 1 (local.get 0)))
</pre>
</div><p>With this knowledge we can define <tt>cons</tt>, as a simple call to
<tt>struct.new_canon pair</tt>.</p><p>I didn't have time for this in the talk, but there is a ghost haunting
this code: the ghost of nominal typing. See, in a web browser at least,
every heap object will have its first word point to its "hidden class" /
"structure" / "map" word. If the engine ever needs to check that a
value is of a specific shape, it can do a quick check on the map word's
value; if it needs to do deeper introspection, it can dereference that
word to get more details.</p><p>Under the hood, testing whether a <tt>(ref eq)</tt> is a pair or not should be
a simple check that it's a <tt>(ref struct)</tt> (and not a fixnum), and then a
comparison of its map word to the run-time type corresponding to
<tt>$pair</tt>. If subtyping of <tt>$pair</tt> is allowed, we start to want inline
caches to handle polymorphism, but the checking the map word is still
the basic mechanism.</p><p>However, as I mentioned, we only have structural equality of types; two
<tt>(struct (ref eq))</tt> type definitions will define the same type and have
the same map word (run-time type / RTT). Hence the <tt>_canon</tt> in the name of
<tt>struct.new_canon $pair</tt>: we create an instance of <tt>$pair</tt>, with the
<i>canonical</i> run-time-type for objects having <tt>$pair</tt>-shape.</p><p>In earlier drafts of the WebAssembly GC extensions, users could define
their own RTTs, which effectively amounts to nominal typing: not only
does this object have the right structure, but was it created with
respect to this particular RTT. But, this facility was cut from the
first release, and it left ghosts in the form of these <tt>_canon</tt> suffixes
on type constructor instructions.</p><p>For the Scheme-to-WebAssembly effort, we effectively add back in a
degree of nominal typing via type tags. For better or for worse this
results in a so-called "open-world" system: you can instantiate a
separately-compiled WebAssembly module that happens to define the same
types and use the same type tags and it will be able to happily access
the contents of Scheme values from another module. If you were to use
nominal types, you would't be able to do so, unless there were some
common base module that defined and exported the types of interests, and
which any extension module would need to
<a href="https://github.com/WebAssembly/proposal-type-imports/blob/main/proposals/type-imports/Overview.md">import</a>.</p><div>
<pre>
(func $car (param <b>(ref eq)</b>) (result (ref eq))
(local (ref $pair))
(block $not-pair
(br_if $not-pair
<b>(i32.eqz (ref.test $pair (local.get 0)))</b>)
(local.set 1 (ref.cast $pair) (local.get 0))
(br_if $not-pair
<b>(i32.ne
(i32.const 1)
(i32.and
(i32.const 0xff)
(struct.get $heap-object 0 (local.get 1))))</b>)
(return_call $%car (local.get 1)))
(call $type-error)
(unreachable))
</pre>
</div><p>In the previous example we had <tt>$%car</tt>, with a funny <tt>%</tt> in the name,
taking a <tt>(ref $pair)</tt> as an argument. But in the general case (barring
compiler heroics) <tt>car</tt> will take an instance of the unitype <tt>(ref eq)</tt>.
To know that it's actually a pair we have to make two checks: one, that
it is a struct and has the <tt>$pair</tt> shape, and two, that it has the right
tag. Oh well!</p><div>
<h2>Scheme to Wasm</h2>
<ul>
<li>
<i>Value representation</i>
</li>
<li>
<b>Varargs</b>
</li>
<li>Tail calls</li>
<li>Delimited continuations</li>
<li>Numeric tower</li>
</ul>
</div><p>But with all of that I think we have a solid story on how to represent
values. I went through all of the basic value types in Guile and
checked that they could <a href="https://gitlab.com/spritely/guile-hoot-updates/-/blob/main/examples/basic-types.wat">all be represented using GC
types</a>,
and it seems that all is good. Now on to the next point: varargs.</p><div>
<h2>Scheme to Wasm: Varargs</h2>
<pre>(list 'hey) ;; => (hey)
(list 'hey 'bob) ;; => (hey bob)</pre>
<p>Problem: Wasm functions strongly typed</p>
<pre>(func $list (param ???) (result (ref eq))
???)</pre>
<p>Solution: Virtualize calling convention</p>
</div><p>In WebAssembly, you define functions with a type, and it is impossible
to call them in an unsound way. You must call <tt>$car</tt> exactly 2
arguments or it will not compile, and those arguments have to be of
specific types, and so on. But Scheme doesn't enforce these
restrictions on the language level, bless its little miscreant heart.
You can call <tt>car</tt> with 5 arguments, and you'll get a run-time error.
There are some functions that can take a variable number of arguments,
doing different things depending on incoming argument count.</p><p>How do we square these two approaches to function types?</p><div>
<pre>;; "Registers" for args 0 to 3
(global $arg0 (mut (ref eq)) (i31.new (i32.const 0)))
(global $arg1 (mut (ref eq)) (i31.new (i32.const 0)))
(global $arg2 (mut (ref eq)) (i31.new (i32.const 0)))
(global $arg3 (mut (ref eq)) (i31.new (i32.const 0)))
;; "Memory" for the rest
(type $argv (array (ref eq)))
(global $argN (ref $argv)
(array.new_canon_default
$argv (i31.const 42) (i31.new (i32.const 0))))</pre>
<p>Uniform function type: argument count as sole parameter</p>
<p>Callee moves args to locals, possibly clearing roots</p>
</div><p>The approach we are taking is to virtualize the calling convention. In
the same way that when calling an x86-64 function, you pass the first
argument in <tt>$rdi</tt>, then <tt>$rsi</tt>, and eventually if you run out of
registers you put arguments in memory, in the same way we'll pass the
first argument in the <tt>$arg0</tt> global, then <tt>$arg1</tt>, and eventually in
memory if needed. The function will receive the number of incoming
arguments as its sole parameter; in fact, all functions will be of type
<tt>(func (param i32))</tt>.</p><p>The expectation is that after checking argument count, the callee will
load its arguments from globals / memory to locals, which the compiler
can do a better job on than globals. We might not even emit code to
null out the argument globals; might leak a little memory but probably
would be a win.</p><p>You can imagine a world in which <tt>$arg0</tt> actually gets globally
allocated to <tt>$rdi</tt>, because it is only live during the call sequence;
but I don't think that world is this one :)</p><div>
<h2>Scheme to Wasm</h2>
<ul>
<li>
<i>Value representation</i>
</li>
<li>
<i>Varargs</i>
</li>
<li>
<b>Tail calls</b>
</li>
<li>Delimited continuations</li>
<li>Numeric tower</li>
</ul>
</div><p>Great, two points out of the way! Next up, tail calls.</p><div>
<h2>Scheme to Wasm: Tail calls</h2>
<pre>;; Call known function
(return_call $f arg ...)
;; Call function by value
(return_call_ref $type callee arg ...)</pre>
</div><p>Friends -- I almost cried making this slide. We Schemers are used to
working around the lack of tail calls, and I could have done so here,
but it's just such a relief that these functions are just going to be
there and I don't have to think much more about them. Technically
speaking the <a href="https://github.com/WebAssembly/tail-call">proposal</a> isn't
merged yet; checking the <a href="https://github.com/WebAssembly/proposals">phases
document</a> it's at the last
station before headed to the great depot in the sky. But, soon soon it
will be present and enabled in all WebAssembly implementations, and we
should build systems now that rely on it.</p><div>
<h2>Scheme to Wasm</h2>
<ul>
<li>
<i>Value representation</i>
</li>
<li>
<i>Varargs</i>
</li>
<li>
<i>Tail calls</i>
</li>
<li>
<b>Delimited continuations</b>
</li>
<li>Numeric tower</li>
</ul>
</div><p>Next up, my favorite favorite topic: delimited continuations.</p><div>
<h2>Scheme to Wasm: Prompts (1)</h2>
<p>Problem: Lightweight threads/fibers, exceptions</p>
<p>Possible solutions</p>
<ul>
<li>Eventually, built-in coroutines</li>
<li>
<a href="https://github.com/WebAssembly/binaryen">binaryen</a>’s asyncify (not yet ready for GC); see Julia
</li>
<li>
<b>Delimited continuations</b>
</li>
</ul>
<p>“Bring your whole self”</p>
</div><p>Before diving in though, one might wonder why bother. Delimited
continuations are a building-block that one can use to build other, more
useful things, notably exceptions and light-weight threading / fibers.
Could there be another way of achieving these end goals without having
to implement this relatively uncommon primitive?</p><p>For fibers, it is possible to <a href="https://wingolog.org/archives/2018/05/16/lightweight-concurrency-in-lua">implement them in terms of a built-in
coroutine
facility</a>.
The standards body seems willing to include a coroutine primitive, but
it seems far off to me; not within the next 3-4 years I would say. So
let's put that to one side.</p><p>There is a more near-term solution, to use <a href="https://github.com/WebAssembly/design/issues/1294#issuecomment-520231415">asyncify to implement
coroutines</a>
somehow; but my understanding is that asyncify is not ready for GC yet.</p><p>For the Guile flavor of Scheme at least, delimited continuations are
table stakes of their own right, so given that we will have them on
WebAssembly, we might as well use them to implement fibers and
exceptions in the same way as we do on native targets. Why compromise
if you don't have to?</p><div>
<h2>Scheme to Wasm: Prompts (2)</h2>
<p>Prompts delimit continuations</p>
<pre>(define k
(call-with-prompt ’foo
; body
(lambda ()
(+ 34 (abort-to-prompt 'foo)))
; handler
(lambda (continuation)
continuation)))
(k 10) ;; ⇒ 44
(- (k 10) 2) ;; ⇒ 42</pre>
<p>
<tt>k</tt> is the <tt>_</tt> in <tt>(lambda () (+ 34 _))</tt>
</p>
</div><p>There are a few ways to implement <a href="https://wingolog.org/archives/2010/02/26/guile-and-delimited-continuations">delimited
continuations</a>,
but my usual way of thinking about them is that a delimited continuation
is a slice of the stack. One end of the slice is the <i>prompt</i>
established by <tt>call-with-prompt</tt>, and the other by the continuation of
the call to <tt>abort-to-prompt</tt>. Capturing a slice pops it off the stack,
copying it out to the heap as a callable function. Calling that
function splats the captured slice back on the stack and resumes it
where it left off.</p><div>
<h2>Scheme to Wasm: Prompts (3)</h2>
<p>Delimited continuations are stack slices</p>
<p>Make stack explicit via minimal continuation-passing-style conversion</p>
<ul>
<li>Turn all calls into tail calls</li>
<li>Allocate return continuations on explicit stack</li>
<li>Breaks functions into pieces at non-tail calls</li>
</ul>
</div><p>This low-level intuition of what a delimited continuation is leads
naturally to an implementation; the only problem is that we can't slice
the WebAssembly call stack. The workaround here is similar to the
varargs case: we virtualize the stack.</p><p>The mechanism to do so is a continuation-passing-style (CPS)
transformation of each function. Functions that make no calls, such as
leaf functions, don't need to change at all. The same goes for
functions that make only tail calls. For functions that make non-tail
calls, we split them into pieces that preserve the only-tail-calls
property.</p><div>
<h2>Scheme to Wasm: Prompts (4)</h2>
<p>Before a non-tail-call:</p>
<ul>
<li>Push live-out vars on stacks (one stack per top type)</li>
<li>Push continuation as funcref</li>
<li>Tail-call callee</li>
</ul>
<p>Return from call via pop and tail call:</p>
<pre>(return_call_ref (call $pop-return)
(i32.const 0))</pre>
<p>After return, continuation pops state from stacks</p>
</div><p>Consider a simple function:</p><pre class="pre-scheme">(define (f x y)
(+ x (g y))
</pre><p>Before making a non-tail call, a "tailified" function will instead push
all live data onto an explicitly-managed stack and tail-call the
callee. It also pushes on the return continuation. Returning from the
callee pops the return continuation and tail-calls it. The return
continuation pops the previously-saved live data and continues.</p><p>In this concrete case, tailification would split <tt>f</tt> into two pieces:</p><pre class="pre-scheme">(define (f x y)
(push! x)
(push-return! f-return-continuation-0)
(g y))
(define (f-return-continuation-0 g-of-y)
(define k (pop-return!))
(define x (pop! x))
(k (+ x g-of-y)))
</pre><p>Now there are no non-tail calls, besides calls to run-time routines like
<tt>push!</tt> and <tt>+</tt> and so on. This transformation is implemented by
<a href="https://git.savannah.gnu.org/gitweb/?p=guile.git;a=blob;f=module/language/cps/tailify.scm;h=f9ebb63d20b7ad736e74c39327d854de0fb798af;hb=refs/heads/wip-tailify">tailify.scm</a>.</p><div>
<h2>Scheme to Wasm: Prompts (5)</h2>
<p>
<tt>abort-to-prompt</tt>:
</p>
<ul>
<li>Pop stack slice to reified continuation object</li>
<li>Tail-call new top of stack: prompt handler</li>
</ul>
<p>Calling a reified continuation:</p>
<ul>
<li>Push stack slice</li>
<li>Tail-call new top of stack</li>
</ul>
<p>No need to wait for effect handlers proposal; you can have it all now!</p>
</div><p>The salient point is that the stack on which <tt>push!</tt> operates (in
reality, probably four or five stacks: one in linear memory or an array
for types like <tt>i32</tt> or <tt>f64</tt>, three for each of the managed top types
<tt>any</tt>, <tt>extern</tt>, and <tt>func</tt>, and one for the stack of return
continuations) are managed by us, so we can slice them.</p><p>Someone asked in the talk about whether the explicit memory traffic and
avoiding the return-address-buffer branch prediction is a source of
inefficiency in the transformation and I have to say, yes, but I don't
know by how much. I guess we'll find out soon.</p><div>
<h2>Scheme to Wasm</h2>
<ul>
<li>
<i>Value representation</i>
</li>
<li>
<i>Varargs</i>
</li>
<li>
<i>Tail calls</i>
</li>
<li>
<i>Delimited continuations</i>
</li>
<li>
<b>Numeric tower</b>
</li>
</ul>
</div><p>Okeydokes, last point!</p><div>
<h2>Scheme to Wasm: Numbers</h2>
<p>Numbers can be immediate: fixnums</p>
<p>Or on the heap: bignums, fractions, flonums, complex</p>
<p>
Supertype is still <tt>ref eq</tt>
</p>
<p>Consider imports to implement bignums</p>
<ul>
<li>
On web: <tt>BigInt</tt>
</li>
<li>On edge: Wasm support module (mini-gmp?)</li>
</ul>
<p>Dynamic dispatch for polymorphic ops, as usual</p>
</div><p>First, I would note that sometimes the compiler can unbox numeric
operations. For example if it infers that a result will be an inexact
real, it can use unboxed <tt>f64</tt> instead of library routines working on
heap flonums (<tt>(struct i32 f64)</tt>; the initial i32 is for the hash and
tag). But we still need a story for the general case that involves
dynamic type checks.</p><p>The basic idea is that we get to have fixnums and heap numbers. Fixnums
will handle most of the integer arithmetic that we need, and will avoid
allocation. We'll inline most fixnum operations as a fast path and call
out to library routines otherwise. Of course fixnum inputs may produce
a bignum output as well, so the fast path sometimes includes another
slow-path callout.</p><p>We want to minimize binary module size. In an ideal
compile-to-WebAssembly situation, a small program will have a small
module size, down to a minimum of a kilobyte or so; larger programs can
be megabytes, if the user experience allows for the download delay.
Binary module size will be dominated by code, so that means we need to
plan for aggressive dead-code elimination, minimize the size of fast
paths, and also minimize the size of the standard library.</p><p>For numbers, we try to keep module size down by leaning on the platform.
In the case of bignums, we can punt some of this work to the host; on a
JavaScript host, we would use <tt>BigInt</tt>, and on a WASI host we'd compile
an external bignum library. So that's the general story: inlined fixnum
fast paths with dynamic checks, and otherwise library routine callouts,
combined with aggressive whole-program dead-code elimination.</p><div>
<h2>Scheme to Wasm</h2>
<ul>
<li>
<i>Value representation</i>
</li>
<li>
<i>Varargs</i>
</li>
<li>
<i>Tail calls</i>
</li>
<li>
<i>Delimited continuations</i>
</li>
<li>
<i>Numeric tower</i>
</li>
</ul>
</div><p>Hey I think we did it! Always before when I thought about compiling
Scheme or Guile to the web, I got stuck on some point or another, was
tempted down the corner-cutting alleys, and eventually gave up before
starting. But finally it would seem that the stars are aligned: we get
to have our Scheme and run it too.</p><div>
<h2>Miscellenea</h2>
<p>Debugging: The wild west of DWARF; prompts</p>
<p>
Strings: <tt>stringref</tt> host strings spark joy
</p>
<p>
JS interop: Export accessors; Wasm objects opaque to JS. <tt>externref</tt>.
</p>
<p>
JIT: <a href="https://wingolog.org/archives/2022/08/18/just-in-time-code-generation-within-webassembly">A whole ’nother talk!</a>
</p>
<p>AOT: wasm2c</p>
</div><p>Of course, like I said, WebAssembly is still a weird machine: as a
compilation target but also at run-time. Debugging is a right proper
mess; perhaps some other article on that some time.</p><p>How to represent strings is a surprisingly gnarly question; there is
tension within the WebAssembly standards community between those that
think that <a href="https://github.com/WebAssembly/stringref/blob/main/proposals/stringref/Overview.md">it's possible for JavaScript and WebAssembly to share an
underlying string
representation</a>,
and those that think that it's a fool's errand and that copying is the
only way to go. I don't know which side will prevail; perhaps more on
that as well later on.</p><p>Similarly the whole interoperation with JavaScript question is very much
in its early stages, with the current situation choosing to <a href="https://github.com/WebAssembly/gc/issues/279">err on the
side of nothing rather than the wrong
thing</a>. You can pass a
WebAssembly <tt>(ref eq)</tt> to JavaScript, but JavaScript can't do anything
with it: it has no prototype. The state of the art is to also ship a JS
run-time that wraps each wasm object, proxying exported functions from
the wasm module as object methods.</p><p>Finally, some language implementations really need JIT support, like
PyPy. There, that's a whole 'nother talk!</p><div>
<h2>WebAssembly for the rest of us</h2>
<p>With GC, WebAssembly is now ready for us</p>
<p>Getting our languages on WebAssembly now a S.M.O.P.</p>
<p>Let’s score some goals in the second half!</p>
<pre>(visit-links
"<a href="https://gitlab.com/spritely/guile-hoot-updates">gitlab.com/spritely/guile-hoot-updates</a>"
"<a href="https://wingolog.org/wingolog.org">wingolog.org</a>"
"wingo@igalia.com"
"<a href="https://igalia.com/">igalia.com</a>"
"<a href="https://mastodon.social/@wingo">mastodon.social/@wingo</a>")</pre>
</div><p>WebAssembly has proven to have some great wins for C, C++, Rust, and so
on -- but now it's our turn to get in the game. GC is coming and we as
a community need to be getting our compilers and language run-times
ready. Let's put on the coffee and bang some bytes together; it's still
early days and there's a world to win out there for the language
community with the best WebAssembly experience. The game is afoot: happy
consing!</p></div>2023-03-20T09:06:42+00:00Andy WingoVíctor Jáquez: Review of Igalia Multimedia activities (2022)
https://blogs.igalia.com/vjaquez/2023/03/14/review-of-igalia-multimedia-activities-2022/
<p>We, <a href="https://www.igalia.com/technology/multimedia">Igalia’s multimedia team</a>, would like to share with you our list of achievements along the past 2022.</p>
<h2>WebKit Multimedia</h2>
<p><strong>WebRTC</strong></p>
<p>Phil already wrote a first blog post, of a series, on this regard: <a href="https://base-art.net/Articles/webrtc-in-webkitgtk-and-wpe-status-updates-part-i/">WebRTC in WebKitGTK and WPE, status updates, part I</a>. Please, be sure to give it a glance, it has nice videos.</p>
<p>Long story short, last year we started to support <a href="https://www.w3.org/TR/mediacapture-streams/">Media Capture and Streams</a> in WebKitGTK and WPE using GStreamer, either for input devices (camera and microphone), desktop sharing, <em>webaudio</em>, and web canvas. But this is just the first step. We are currently working on <a href="https://developer.mozilla.org/en/US/docs/Web/API/RTCPeerConnection">RTCPeerConnection</a>, also <a href="https://github.com/WebKit/WebKit/commit/a530847a7ce739e1b8b7a55cf f5ae41a0938a11">using GStreamer</a>, to share all these captured streams with other web peers. Meanwhile, we’ll wait for the second episode of Phil’s series <img src="https://s.w.org/images/core/emoji/12.0.0-1/72x72/1f642.png" alt="🙂" class="wp-smiley" /></p>
<p><strong>MediaRecorder</strong></p>
<p>We worked in an initial implementation of <a href="https://w3c.github.io/mediacapture record/MediaRecorder.html">MediaRecorder</a> with <a href="https://github.com/WebKit/WebKit/commit/56ec50c054f07cc92f7c331dc2402b36bb10bb0f">GStreamer</a> (1.20 or superior). The specification goes about allowing a web browser to record a selected stream. For example, a voice-memo or video application which could encode and upload a capture of your microphone / camera.</p>
<p><strong>Gamepad</strong></p>
<p>While WebKitGTK already has <a href="https://w3c.github.io/gamepad/">Gamepad</a> support, WPE lacked it. We did the implementation last year, and there’s a blog post about it: <a href="https://blogs.igalia.com/vjaquez/2022/07/20/gamepad-in wpewebkit/">Gamepad in WPEWebkit</a>, with video showing a demo of it.</p>
<p><strong>Capture encoded video streams from webcams</strong></p>
<p>Some webcams only provide high resolution frames encoded in H.264 or so. In order to support these resolutions with those webcams we added the <a href="https://github.com/WebKit/WebKit/commit/4e6c9f2028df849574df76dab51e7b94bf34a44f">support for negotiate of those formats</a> and decode them internally to handle the streams. Though we are just at the beginning of more efficient support.</p>
<p><strong>Flatpak SDK maintenance</strong></p>
<p>A lot of effort went to maintain the Flatpak SDK for WebKit. It is a set of runtimes that allows to have a reproducible build of WebKit, independently of the used Linux distribution. Nowadays the Flatpak SDK is used in <a href="https://trac.webkit.org/wiki/EarlyWarningSystem">Webkit’s EWS</a>, and by many developers.</p>
<p>Among all the features added during the year we can highlight added Rust support, a full integrity check before upgrading, and offer a way to override dependencies as <a href="https://blogs.igalia.com/vjaquez/2022/05/">local projects</a>.</p>
<p><strong>MSE/EME enhancements</strong></p>
<p>As every year, massive work was done in WebKit ports using GStreamer for <a href="https://www.w3.org/TR/media-source-2/">Media Source Extensions</a> and <a href="https://www.w3.org/TR/encrypted-media/">Encrypted Media Extensions</a>, improving user experience with different streaming services in the Web, such as Odysee, Amazon, DAZN, etc.</p>
<p>In the case of encrypted media, GStreamer-based WebKit ports provide the stubs to communicate with an external Content Decryption Module (<a href="https://www.w3.org/TR/encrypted-media/#cdm">CDM</a>). If you’re willing to support this in your platform, <a href="https://www.igalia.com/contact/">you can reach us</a>.</p>
<p>Also we worked in a video demo showing how MSE/EME works in a Raspberry Pi 3 using WPE:</p>
<div class="wp-video"><!--[if lt IE 9]><script>document.createElement('video');</script><![endif]-->
<a href="http://blogs.igalia.com/vjaquez/files/2023/03/EME-demo.webm">http://blogs.igalia.com/vjaquez/files/2023/03/EME-demo.webm</a></div>
<p><strong>WebAudio demo</strong></p>
<p>We also spent time recording video demos, such as this one, showing WebAudio using WPE on a desktop computer.</p>
<div class="wp-video"><a href="http://blogs.igalia.com/vjaquez/files/2023/03/WPE-WebAudio-Demo.webm">http://blogs.igalia.com/vjaquez/files/2023/03/WPE-WebAudio-Demo.webm</a></div>
<h2>GStreamer</h2>
<p>We managed to merge a lot of bug fixes in GStreamer, which in many cases can be harder to solve rather than implementing new features, though former are more interesting to tell, such as those related with making Rust the main developing language for GStreamer besides C.</p>
<p><strong>Rust bindings and GStreamer elements for Vonage Video API / OpenTok</strong></p>
<p>OpenTok is the legacy name of Vonage Video API, and is a <a href="https://en.wikipedia.org/wiki/Platform_as_a_service">PaaS</a> (Platform As a Service) to ease the development and deployment of WebRTC services and applications.</p>
<p>We published <a href="https://github.com/opentok-rust">our work in Github</a> of Rust bindings both for the <a href="https://tokbox.com/developer/sdks/linux/">Client SDK for Linux</a> and the <a href="https://tokbox.com/developer/rest/">Server SDK using REST API</a>, along with a GStreamer plugin to publish and subscribe to video and audio streams.</p>
<p><strong>GstWebRTCSrc</strong></p>
<p>In the beginning there was <a href="https://docs.gstreamer.com/documentation/webrtc/index.html">webrtcbin</a>, an element that implements the majority of W3C <a href="https://www.w3.org/TR/webrtc/#dom-rtcpeerconnection">RTCPeerConnection</a> API. It’s so flexible and powerful that it’s rather hard to use for the most common cases. Then appeared <a href="https://docs.gstreamer.com/documentation/rswebrtc/webrtcsink.html">webrtcsink</a>, a wrapper of <code>webrtcbin</code>, written in Rust, which receives GStreamer streams which will be offered and streamed to web peers. Later on, we developed <a href="https://docs.gstreamer.com/documentation/rswebrtc/webrtcsrc.html">webrtcsrc</a>, the <code>webrtcsink</code> counterpart: an element which source pads push streams from web peers, such as another browser, and forward those Web streams as GStreamer ones in a pipeline. Both <code>webrtcsink</code> and <code>webrtcsrc</code> are written in Rust.</p>
<p><strong>Behavior-Driven Development test framework for GStreamer</strong></p>
<p><a href="https://en.wikipedia.org/wiki/Behavior-driven_development">Behavior-Driven Development</a> is gaining relevance with tools like <a href="https://cucumber.io/">Cucumber</a> for Java and its domain specific language, <a href="https://cucumber.io/docs/gherkin/">Gherkin</a> to define software behaviors. <a href="https://www.rustaceans.org/">Rustaceans</a> have picked up these ideas and developed <a href="https://github.com/cucumber-rs/cucumber">cucumber-rs</a>. The logical consequence was obvious: Why not GStreamer?</p>
<p>Last year we tinkered with <a href="https://github.com/philn/gstreamer-cucumber">GStreamer-Cucumber</a>, a BDD to define behavior tests for GStreamer pipelines.</p>
<p><strong>GstValidate Rust bindings</strong></p>
<p>There have been some discussion if BDD is the best way to test GStreamer pipelines, and there’s <a href="https://gstreamer.freedesktop.org/documentation/gst-devtools/gst-validate.html">GstValidate</a>, and also, last year, we added <a href="https://gitlab.freedesktop.org/gstreamer/gstreamer-rs/-/tree/main/gstreamer-validate">its Rust bindings</a>.</p>
<p><strong>GStreamer Editing Services</strong></p>
<p>Though not everything was Rust. We work hard on GStreamer’s nuts and bolts.</p>
<p>Last year, we gathered the team to hack <a href="https://gstreamer.freedesktop.org/documentation/gst-editing-service">GStreamer Editing Services</a>, particularly to explore adding OpenGL and DMABuf support, such as downloading or uploading a texture before processing, and selecting a proper filter to avoid those transfers.</p>
<p><strong>GstVA and GStreamer-VAAPI</strong></p>
<p>We helped in the maintenance of <a href="https://gstreamer.freedesktop.org/documentation/vaapi/index.html">GStreamer-VAAPI</a> and the development of its near replacement: <a href="https://gstreamer.freedesktop.org/documentation/va/index.html">GstVA</a>, adding new elements such as the H.264 encoder, the compositor and the JPEG decoder. Along with participation on the debate and code reviewing of negotiating DMABuf streams in the pipeline.</p>
<p><strong>Vulkan decoder and parser library for CTS</strong></p>
<p>You might have heard about <a href="https://blogs.igalia.com/vjaquez/2022/">Vulkan has now integrated in its API video decoding</a>, while encoding is currently work-in-progress. We devoted time on helping Khronos with the Vulkan Video Conformance Tests (CTS), particularly with a parser based on GStreamer and developing a H.264 decoder in GStreamer using Vulkan Video API.</p>
<p>You can check the presentation we did last Vulkanised.</p>
<p></p>
<p><strong>WPE Android Experiment</strong></p>
<p>In a joint adventure with Igalia’s Webkit team we did some experiments to port WPE to Android. This is just an internal proof of concept so far, but we are looking forward to see how this will evolve in the future, and what new possibilities this might open up.</p>
<div class="wp-video"><a href="http://blogs.igalia.com/vjaquez/files/2023/03/wpeview_demo.webm">http://blogs.igalia.com/vjaquez/files/2023/03/wpeview_demo.webm</a></div>
<p>If you have any questions about WebKit, GStreamer, Linux video stack, compilers, <a href="https://www.igalia.com/technologies/">etc</a>., please <a href="https://www.igalia.com/contact/">contact us</a>.</p>2023-03-14T10:45:11+00:00Andy Wingo: pre-initialization of garbage-collected webassembly heaps
https://wingolog.org/archives/2023/03/10/pre-initialization-of-garbage-collected-webassembly-heaps
<div><p>Hey comrades, I just had an idea that I won't be able to work on in the
next couple months and wanted to release it into the wild. They say if
you love your ideas, you should let them go and see if they come back to
you, right? In that spirit I abandon this idea to the woods.</p><p>Basically the idea is <a href="https://github.com/bytecodealliance/wizer">Wizer-like pre-initialization of WebAssembly
modules</a>, but for modules
that store their data on the GC-managed heap instead of just in linear
memory.</p><p>Say you have a WebAssembly module with <a href="https://github.com/WebAssembly/gc/blob/master/proposals/gc/MVP.md">GC
types</a>.
It might look like this:</p><pre class="pre-wat">(module
(type $t0 (struct (ref eq)))
(type $t1 (struct (ref $t0) i32))
(type $t2 (array (mut (ref $t1))))
...
(global $g0 (ref null eq)
(ref.null eq))
(global $g1 (ref $t1)
(array.new_canon $t0 (i31.new (i32.const 42))))
...
(function $f0 ...)
...)
</pre><p>You define some struct and array types, there are some global variables,
and some functions to actually do the work. (There are probably also
tables and other things but I am simplifying.)</p><p>If you consider the object graph of an instantiated module, you will
have some set of <i>roots</i> R that point to GC-managed objects. The live
objects in the heap are the roots and any object referenced by a live
object.</p><p>Let us assume a standalone WebAssembly module. In that case the set of
types T of all objects in the heap is closed: it can only be one of the
types <i><tt>$t0</tt></i>, <i><tt>$t1</tt></i>, and so on that are defined in the module. These
types have a partial order and can thus be sorted from most to least
specific. Let's assume that this sort order is just the reverse of the
definition order, for now. Therefore we can write a general type
introspection function for any object in the graph:</p><pre>(func $introspect (param $obj anyref)
(block $L2 (ref $t2)
(block $L1 (ref $t1)
(block $L0 (ref $t0)
(br_on_cast $L2 $t2 (local.get $obj))
(br_on_cast $L1 $t1 (local.get $obj))
(br_on_cast $L0 $t0 (local.get $obj))
(unreachable))
;; Do $t0 things...
(return))
;; Do $t1 things...
(return))
;; Do $t2 things...
(return))
</pre><p>In particular, given a WebAssembly module, we can generate a function to
trace edges in an object graph of its types. Using this, we can
identify all live objects, and what's more, we can take a snapshot of
those objects:</p><pre class="pre-wat">(func $snapshot (result (ref (array (mut anyref))))
;; Start from roots, use introspect to find concrete types
;; and trace edges, use a worklist, return an array of
;; all live objects in topological sort order
)
</pre><p>Having a heap snapshot is interesting for introspection purposes, but my
interest is in having fast start-up. Many programs have a kind of
"initialization" phase where they get the system up and running, and
only then proceed to actually work on the problem at hand. For example,
when you run <tt>python3 foo.py</tt>, Python will first spend some time parsing
and byte-compiling <tt>foo.py</tt>, importing the modules it uses and so on,
and then will actually run <tt>foo.py</tt>'s code. Wizer lets you snapshot the
state of a module after initialization but before the real work begins,
which can save on startup time.</p><p>For a GC heap, we actually have similar possibilities, but the mechanism
is different. Instead of generating an array of all live objects, we
could generate a serialized state of the heap as bytecode, and another
function to read the bytecode and reload the heap:</p><pre>(func $pickle (result (ref (array (mut i8))))
;; Return an array of bytecode which, when interpreted,
;; can reconstruct the object graph and set the roots
)
(func $unpickle (param (ref (array (mut i8))))
;; Interpret the bytecode, building object graph in
;; topological order
)
</pre><p>The unpickler is module-dependent: it will need one case to construct
each concrete type <i><tt>$tN</tt></i> in the module. Therefore the bytecode
grammar would be module-dependent too.</p><p>What you would get with a bytecode-based <tt>$pickle</tt>/<tt>$unpickle</tt> pair
would be the ability to serialize and reload heap state many times. But
for the pre-initialization case, probably that's not precisely what you
want: you want to residualize a new WebAssembly module that, when
loaded, will rehydrate the heap. In that case you want a function like:</p><pre>(func $make-init (result (ref (array (mut i8))))
;; Return an array of WebAssembly code which, when
;; added to the module as a function and invoked,
;; can reconstruct the object graph and set the roots.
)
</pre><p>Then you would use binary tools to add that newly generated function to
the module.</p><p>In short, there is a space open for a tool which takes a WebAssembly+GC
module M and produces M', a module which contains a <tt>$make-init</tt>
function. Then you use a WebAssembly+GC host to load the module and
call the <tt>$make-init</tt> function, resulting in a WebAssembly function
<tt>$init</tt> which you then patch in to the original M to make M'', which is
M pre-initialized for a given task.</p><h3>Optimizations</h3><p>Some of the object graph is constant; for example, an instance of a
<tt>struct</tt> type that has no mutable fields. These objects don't have to
be created in the init function; they can be declared as new constant
global variables, which an engine may be able to initialize more
efficiently.</p><p>The pre-initialized module will still have an initialization phase in
which it builds the heap. This is a constant function and it would be
nice to avoid it. Some WebAssembly hosts will be able to run
pre-initialization and then snapshot the GC heap using lower-level facilities (copy-on-write mappings, pointer compression and relocatable cages, pre-initialization on an internal level...). This would potentially decrease latency and may allow for cross-instance memory sharing.</p><h3>Limitations</h3><p>There are five preconditions to be able to pickle and unpickle the GC
heap:</p><ol><li><p>The set of concrete types in a module must be closed.</p></li><li><p>The roots of the GC graph must be enumerable.</p></li><li><p>The object-graph edges from each live object must be enumerable.</p></li><li><p>To prevent cycles, we have to know when an object has been visited:
objects must have identity.</p></li><li><p>We must be able to create each type in a module.</p></li></ol><p>I think there are three limitations to this pre-initialization idea in
practice.</p><p>One is <tt>externref</tt>; these values come from the host and are by
definition not introspectable by WebAssembly. Let's keep the
closed-world assumption and consider the case where the set of external
reference types is closed also. In that case if a module allows for
external references, we can perhaps make its pickling routines call out
to the host to (2) provide any external roots (3) identify edges on
<tt>externref</tt> values (4) compare <tt>externref</tt> values for identity and (5)
indicate some imported functions which can be called to re-create
exernal objects.</p><p>Another limitation is <tt>funcref</tt>. In practice in the current state of
WebAssembly and GC, you will only have a <tt>funcref</tt> which is created by
<tt>ref.func</tt>, and which (3) therefore has no edges and (5) can be
re-created by <tt>ref.func</tt>. However neither WebAssembly nor the JS API
has no way of knowing which function index corresponds to a given
funcref. Including function references in the graph would therefore
require some sort of host-specific API. Relatedly, function references
are not comparable for equality (<tt>func</tt> is not a subtype of <tt>eq</tt>), which
is a little annoying but not so bad considering that function references
can't participate in a cycle. Perhaps a solution though would be to
assume (!) that the host representation of a funcref is constant: the
JavaScript (e.g.) representations of <tt>(ref.func 0)</tt> and <tt>(ref.func 0)</tt>
are the same value (in terms of <tt>===</tt>). Then you could compare a given
function reference against a set of known values to determine its index.
Note, when function references are expanded to include closures, we will
have more problems in this area.</p><p>Finally, there is the question of roots. Given a module, we can
generate a function to read the values of all reference-typed globals
and of all entries in all tables. What we can't get at are any
references from the stack, so our object graph may be incomplete.
Perhaps this is not a problem though, because when we unpickle the graph
we won't be able to re-create the stack anyway.</p><p>OK, that's my idea. Have at it, hackers!</p></div>2023-03-10T09:20:33+00:00Andy WingoRobert McQueen: Flathub in 2023
https://ramcq.net/2023/03/07/flathub-in-2023/
<p>It’s been quite a few months since the most recent updates about Flathub last year. We’ve been busy behind the scenes, so I’d like to share what we’ve been up to at Flathub and why—and what’s coming up from us this year. I want to focus on:</p>
<ul>
<li>Where Flathub is today as a <strong>strong ecosystem with 2,000 apps</strong></li>
<li>Our progress on <strong>evolving Flathub from a build service to an app store</strong></li>
<li>The <strong>economic barrier to growing the ecosystem</strong>, and its consequences</li>
<li>What’s next to <strong>overcome our challenges with focused initiatives</strong></li>
</ul>
<h1 class="wp-block-heading"><strong>Today</strong></h1>
<p>Flathub is going strong: we offer <strong>2,000 apps</strong> from <strong>over 1,500 collaborators</strong> on GitHub. We’re averaging <strong>700,000 app downloads a day</strong>, with 898 million HTTP requests totalling <strong>88.3 TB served by our CDN each day</strong> (thank you Fastly!). Flatpak has, in my opinion, solved the largest technical issue which has held back the mainstream growth and acceptance of Linux on the desktop (or other personal computing devices) for the past 25 years: namely, the difficulty for app developers to publish their work in a way that makes it easy for people to discover, download (or sideload, for people in challenging connectivity environments), install and use. Flathub builds on that to help users discover the work of app developers and helps that work reach users in a timely manner.</p>
<p>Initial results of this disintermediation are promising: even with its modest size so far, Flathub has hundreds of apps that I have never, ever heard of before—and that’s even considering I’ve been working in the Linux desktop space for nearly 20 years and spent many of those staring at the contents of <kbd>dselect</kbd> (showing my age a little) or GNOME Software, attending conferences, and reading blog posts, news articles, and forums. I am also heartened to see that many of our OS distributor partners have recognised that this model is hugely complementary and additive to the indispensable work they are doing to bring the Linux desktop to end users, and that <strong>“having more apps available to your users” is a value-add</strong> allowing you to focus on your core offering and not a zero-sum game that should motivate infighting.</p>
<h1 class="wp-block-heading"><strong>Ongoing Progress</strong></h1>
<p>Getting Flathub into its current state has been a long ongoing process. Here’s what we’ve been up to behind the scenes:</p>
<h2 class="wp-block-heading">Development</h2>
<p>Last year, we concluded our first engagement with Codethink to build features into the Flathub web app to <strong>move from a build service to an app store.</strong> That includes accounts for users and developers, payment processing via Stripe, and the ability for developers to manage upload tokens for the apps they control. In parallel, James Westman has been working on app verification and the corresponding features in flat-manager to ensure app metadata accurately reflects verification and pricing, and to provide authentication for paying users for app downloads when the developer enables it. Only verified developers will be able to make direct uploads or access payment settings for their apps.</p>
<h2 class="wp-block-heading">Legal</h2>
<p>So far, the GNOME Foundation has acted as an incubator and legal host for Flathub even though it’s not purely a GNOME product or initiative. Distributing software to end users along with processing and forwarding payments and donations also has a different legal profile in terms of risk exposure and nonprofit compliance than the current activities of the GNOME Foundation. Consequently, <strong>we plan to establish an independent legal entity to own and operate Flathub</strong> which reduces risk for the GNOME Foundation, better reflects the independent and cross-desktop interests of Flathub, and provides flexibility in the future should we need to change the structure.</p>
<p>We’re currently in the process of reviewing legal advice to ensure we have the right structure in place before moving forward.</p>
<h2 class="wp-block-heading">Governance</h2>
<p>As Flathub is something we want to set outside of the existing Linux desktop and distribution space—and ensure we represent and serve the widest community of Linux users and developers—we’ve been working on <strong>a governance model that ensures that there is transparency and trust in who is making decisions, and why</strong>. We have set up a working group with myself and Martín Abente Lahaye from GNOME, Aleix Pol Gonzalez, Neofytos Kolokotronis, and Timothée Ravier from KDE, and Jorge Castro flying the flag for the Flathub community. Thanks also to Neil McGovern and Nick Richards who were also more involved in the process earlier on.</p>
<p>We don’t want to get held up here creating something complex with memberships and elections, so at first we’re going to come up with a simple/balanced way to appoint people into a board that makes key decisions about Flathub and iterate from there.</p>
<h2 class="wp-block-heading">Funding</h2>
<p><strong>We have received one grant for 2023 of $100K</strong> from <a href="https://www.endlessnetwork.com/">Endless Network</a> which will go towards the infrastructure, legal, and operations costs of running Flathub and setting up the structure described above. (Full disclosure: Endless Network is the umbrella organisation which also funds my employer, Endless OS Foundation.) <strong>I am hoping to grow the available funding to $250K for this year</strong> in order to cover the next round of development on the software, prepare for higher operations costs (e.g., accounting gets more complex), and bring in a second full-time staff member in addition to Bartłomiej Piotrowski to handle enquiries, reviews, documentation, and partner outreach.</p>
<p>We’re currently in discussions with <a href="https://nlnet.nl/">NLnet</a> about funding further software development, but have been unfortunately turned down for a grant from the <a href="https://www.plaintextgroup.com/">Plaintext Group</a> for this year; this Schmidt Futures project around OSS sustainability is not currently issuing grants in 2023. However, we continue to work on other funding opportunities.</p>
<h1 class="wp-block-heading"><strong>Remaining Barriers</strong></h1>
<p>My personal hypothesis is that <strong>our largest remaining barrier to Linux desktop scale and impact is economic</strong>. On competing platforms—mobile or desktop—a developer can offer their work for sale via an app store or direct download with payment or subscription within hours of making a release. While we have taken the “time to first download” time down from months to days with Flathub, as a community we continue to have a challenging relationship with money. Some creators are lucky enough to have a full-time job within the FLOSS space, while a few “superstar” developers are able to nurture some level of financial support by investing time in building a following through streaming, Patreon, Kickstarter, or similar. However, a large proportion of us have to make do with the main payback from our labours being a stream of bug reports on GitHub interspersed with occasional conciliatory beers at FOSDEM (other beverages and events are available).</p>
<p>The first and most obvious consequence is that <strong>if there is no financial payback for participating in developing apps for the free and open source desktop, we will lose many people in the process</strong>—despite the amazing achievements of those who have brought us to where we are today. As a result, we’ll have far fewer developers and apps. If we can’t offer access to a growing base of users or the opportunity to offer something of monetary value to them, the reward in terms of adoption and possible payment will be very small. Developers would be forgiven for taking their time and attention elsewhere. With fewer apps, our platform has less to entice and retain prospective users.</p>
<p>The second consequence is that this also represents <strong>a significant hurdle for diverse and inclusive participation</strong>. We essentially require that somebody is in a position of privilege and comfort that they have internet, power, time, and income—not to mention childcare, etc.—to spare so that they can take part. If that’s not the case for somebody, we are leaving them shut out from our community before they even have a chance to start. My belief is that <strong>free and open source software represents a better way for people to access computing</strong>, and there are billions of people in the world we should hope to reach with our work. But if the mechanism for participation ensures their voices and needs are never represented in our community of creators, we are significantly less likely to understand and meet those needs.</p>
<p>While these are <em>my</em> thoughts, you’ll notice a strong theme to this year will be leading a consultation process to <strong>ensure that we are including, understanding and reflecting the needs of our different communities</strong>—app creators, OS distributors and Linux users—as I don’t believe that our initiative will be successful without ensuring mutual benefit and shared success. Ultimately, no matter how beautiful, performant, or featureful the latest versions of the Plasma or GNOME desktops are, or how slick the newly rewritten installer is from your favourite distribution, all of the projects making up the Linux desktop ecosystem are subdividing between ourselves an absolutely tiny market share of the global market of personal computers. To make a bigger mark on the world, as a community, we need to get out more.</p>
<h1 class="wp-block-heading"><strong>What’s Next?</strong></h1>
<p>After identifying our major barriers to overcome, we’ve planned a number of focused initiatives and restructuring this year:</p>
<h2 class="wp-block-heading">Phased Deployment</h2>
<p>We’re working on deploying the work we have been doing over the past year, starting first with launching the <a href="https://beta.flathub.org/">new Flathub web experience</a> as well as <a href="https://blog.jimmac.eu/2023/flathub-brand-refresh/">the rebrand that Jakub has been talking about on his blog</a>. This also will finally launch the <strong>verification features so we can distinguish those apps which are uploaded by their developers</strong>.</p>
<p>In parallel, we’ll also be able to turn on the <strong>Flatpak repo subsets that enable users to select only verified and/or FLOSS apps</strong> in the Flatpak CLI or their desktop’s app center UI.</p>
<h2 class="wp-block-heading">Consultation</h2>
<p>We would like to make sure that the voices of app creators, OS distributors, and Linux users are reflected in our plans for 2023 and beyond. <strong>We will be launching this in the form of Flathub Focus Groups</strong> at the Linux App Summit in Brno in May 2023, followed up with surveys and other opportunities for online participation. We see our role as interconnecting communities and want to be sure that we remain transparent and accountable to those we are seeking to empower with our work.</p>
<p>Whilst we are being bold and ambitious with what we are trying to create for the Linux desktop community, we also want to make sure we provide the right forums to listen to the FLOSS community and prioritise our work accordingly.</p>
<h2 class="wp-block-heading">Advisory Board</h2>
<p>As we build the Flathub organisation up in 2023, we’re also planning to <strong>expand its governance by creating an Advisory Board</strong>. We will establish an ongoing forum with different stakeholders around Flathub: OS vendors, hardware integrators, app developers and user representatives to help us create the Flathub that supports and promotes our mutually shared interests in a strong and healthy Linux desktop community.</p>
<h2 class="wp-block-heading">Direct Uploads</h2>
<p><strong>Direct app uploads are close to ready</strong>, and they enable exciting stuff like allowing Electron apps to be built outside of flatpak-builder, or driving automatic Flathub uploads from GitHub actions or GitLab CI flows; however, we need to think a little about how we encourage these to be used. Even with its frustrations, our current Buildbot ensures that the build logs and source versions of each app on Flathub are captured, and that the apps are built on all supported architectures. (Is 2023 when we add RISC-V? Reach out if you’d like to help!). If we hand upload tokens out to any developer, even if the majority of apps are open source, we will go from this relatively structured situation to something a lot more unstructured—and we fear many apps will be available on only 64-bit Intel/AMD machines.</p>
<p>My sketch here is that we need to establish some best practices around how to integrate Flathub uploads into popular CI systems, <strong>encouraging best practices so that we promote the properties of transparency and reproducibility that we don’t want to lose</strong>. If anyone is a CI wizard and would like to work with us as a thought partner about how we can achieve this—make it more flexible where and how build tasks can be hosted, but not lose these cross-platform and inspectability properties—we’d love to hear from you.</p>
<h2 class="wp-block-heading">Donations and Payments</h2>
<p>Once the work around legal and governance reaches a decent point, we will be in the position to move ahead with our Stripe setup and switch on the third big new feature in the Flathub web app. At present, <strong>we have already implemented support for one-off payments</strong> either as donations or a required purchase. We would like to go further than that, in line with what we were describing earlier about helping developers sustainably work on apps for our ecosystem: <strong>we would also like to enable developers to offer subscriptions.</strong> This will allow us to create a relationship between users and creators that funds <em>ongoing work</em> rather than <em>what we already have</em>.</p>
<h2 class="wp-block-heading">Security</h2>
<p>For Flathub to succeed, we need to make sure that as we grow, we continue to be a platform that can give users confidence in the quality and security of the apps we offer. To that end, we are planning to set up infrastructure to help ensure developers are shipping the best products they possibly can to users. For example, we’d like to set up <strong>automated linting and security scanning</strong> on the Flathub back-end to help developers avoid bad practices, unnecessary sandbox permissions, outdated dependencies, etc. and to <strong>keep users informed and as secure as possible</strong>.</p>
<h2 class="wp-block-heading">Sponsorship</h2>
<p>Fundraising is a forever task—as is running such a big and growing service. We hope that one day, we can cover our costs through some modest fees built into our payments—but until we reach that point, <strong>we’re going to be seeking a combination of grant funding and sponsorship to keep our roadmap moving.</strong> Our hope is very much that we can encourage different organisations that buy into our vision and will benefit from Flathub to help us support it and ensure we can deliver on our goals. If you have any suggestions of who might like to support Flathub, we would be very appreciative if you could reach out and get us in touch.</p>
<h1 class="wp-block-heading"><strong>Finally, Thank You!</strong></h1>
<p>Thanks to you all for reading this far and supporting the work of Flathub, and also to our major sponsors and donors without whom Flathub could not exist: <a href="https://foundation.gnome.org/">GNOME Foundation</a>, <a href="https://ev.kde.org/">KDE e.V.</a>, <a href="https://www.mythic-beasts.com/">Mythic Beasts</a>, <a href="https://www.endlessnetwork.com/">Endless Network</a>, <a href="https://www.fastly.com/">Fastly</a>, and <a href="https://www.equinix.com/products/digital-infrastructure-services/equinix-metal">Equinix Metal</a> via the <a href="https://github.com/cncf/cluster">CNCF Community Cluster</a>. Thanks also to the tireless work of the <a href="https://freedesktop-sdk.io/">Freedesktop SDK</a> community to give us the runtime platform most Flatpaks depend on, particularly Seppo Yli-Olli, <a href="https://www.codethink.co.uk/">Codethink</a> and others.</p>
<p>I wanted to also give my personal thanks to a handful of dedicated people who keep Flathub working as a service and as a community: Bartłomiej Piotrowski is keeping the infrastructure working essentially single-handedly (in his spare time from keeping everything running at GNOME); Kolja Lampe and Bart built the new web app and backend API for Flathub which all of the new functionality has been built on, and Filippe LeMarchand maintains the checker bot which helps keeps all of the Flatpaks up to date.</p>
<p>And finally, all of the submissions to Flathub are reviewed to ensure quality, consistency and security by a small dedicated team of reviewers, with a huge amount of work from Hubert Figuière and Bart to keep the submissions flowing. Thanks to everyone—named or unnamed—for building this vision of the future of the Linux desktop together with us.</p>
<p><em>(originally posted to <a href="https://discourse.flathub.org/t/flathub-in-2023/3808">Flathub Discourse</a>, head there if you have any questions or comments)</em></p>2023-03-07T11:00:00+00:00GStreamer: GStreamer 1.22.1 stable bug fix release
https://gstreamer.freedesktop.org/news/#2023-03-04T14:00:00Z
<p>
The GStreamer team is pleased to announce the first bug fix release in the
stable 1.22 release series of your favourite cross-platform
multimedia framework!
</p><p>
This release only contains bugfixes and it should be safe to update from
1.22.0.
</p><p>
<b>Highlighted bugfixes:</b>
<ul>
<li>audio channel-mix: allow up to 64 channels (instead of up to 63 channels)</li>
<li>avfvideosrc: Don't wait on main thread for permissions request</li>
<li>avvidenc: avoid generating inaccurate output timestamps, especially with variable framerate streams</li>
<li>AV1 video codec caps signalling improvements in various elements</li>
<li>codectimestamper: Fix timestamping on sequence update</li>
<li>d3d11overlaycompositor: fix texture width and height</li>
<li>d3d11videosink: Fix rendering on external handle</li>
<li>dashdemux2: fix seek operation taking a log time to finish for some streams</li>
<li>nvencoder: Fix B-frame encoding on Linux and min buffers in auto GPU mode</li>
<li>playbin3: fixing buffering for live pipelines</li>
<li>playbin: fix potential deadlock when stopping stream with subtitles visible</li>
<li>redenc: fix setting of extension ID for twcc</li>
<li>rtspsrc: improved compatibility with more broken RTSP servers</li>
<li>v4l2h264dec: Fix Raspberry Pi4 will not play video in application</li>
<li>vtdec: fix jittery playback of H.264 Level 4.1 movies in macOS</li>
<li>vtdec: Fix non-deterministic frame output after flushing seeks</li>
<li>vtenc: fix handling of interlaced ProRes on Apple M1 hardware</li>
<li>vtenc: don't advertise ARGB/RGBA64 input caps on M1 Pro/Max with macOS <13</li>
<li>wasapi2src: Fix loopback capture on Windows 10 Anniversary Update</li>
<li>tools: better handling of non-ASCII command line arguments on Windows</li>
<li>gst-libav: fix build against newer ffmpeg versions</li>
<li>gst-python: Use arch-specific install dir for gi overrides</li>
<li>cerbero: Fix setuptools site.py breakage in Python 3.11</li>
<li>macOS packages: Fix broken binaries on macos < 11.0</li>
<li>various bug fixes, memory leak fixes, and other stability and reliability improvements</li>
</ul>
</p><p>
See the <a href="https://gstreamer.freedesktop.org/releases/1.22/#1.22.1">GStreamer 1.22.1 release notes</a>
for more details.
</p><p>
Binaries for Android, iOS, Mac OS X and Windows will be available shortly.
</p><p>
Release tarballs can be downloaded directly here:
<ul>
<a href="https://gstreamer.freedesktop.org/src/gstreamer/gstreamer-1.22.1.tar.xz">gstreamer</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-base/gst-plugins-base-1.22.1.tar.xz">gst-plugins-base</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-good/gst-plugins-good-1.22.1.tar.xz">gst-plugins-good</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-ugly/gst-plugins-ugly-1.22.1.tar.xz">gst-plugins-ugly</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-plugins-bad/gst-plugins-bad-1.22.1.tar.xz">gst-plugins-bad</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-libav/gst-libav-1.22.1.tar.xz">gst-libav</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-rtsp-server/gst-rtsp-server-1.22.1.tar.xz">gst-rtsp-server</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-python/gst-python-1.22.1.tar.xz">gst-python</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-editing-services/gst-editing-services-1.22.1.tar.xz">gst-editing-services</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-devtools/gst-devtools-1.22.1.tar.xz">gst-devtools</a>,
<a href="https://gstreamer.freedesktop.org/src/gstreamer-vaapi/gstreamer-vaapi-1.22.1.tar.xz">gstreamer-vaapi</a>,
<a href="https://gstreamer.freedesktop.org/src/gstreamer-sharp/gstreamer-sharp-1.22.1.tar.xz">gstreamer-sharp</a>,
<a href="https://gstreamer.freedesktop.org/src/gst-omx/gst-omx-1.22.1.tar.xz">gst-omx</a>, or
<a href="https://gstreamer.freedesktop.org/src/gstreamer-docs/gstreamer-docs-1.22.1.tar.xz">gstreamer-docs</a>.
</ul>
</p>2023-03-04T14:00:00+00:00