April 20, 2021

Enrique Ocaña GonzálezGStreamer WebKit debugging tricks using GDB (2/2)

This post is a continuation of a series of blog posts about the most interesting debugging tricks I’ve found while working on GStreamer WebKit on embedded devices. These are the other posts of the series published so far:

Print corrupt stacktraces

In some circumstances you may get stacktraces that eventually stop because of missing symbols or corruption (?? entries).

#3  0x01b8733c in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

However, you can print the stack in a useful way that gives you leads about what was next in the stack:

  • For i386: x/256wa $esp
  • For x86_64: x/256ga $rsp
  • For ARM 32 bit: x/256wa $sp

You may want to enable asm-demangle: set print asm-demangle

Example output, the 3 last lines give interesting info:

0x7ef85550:     0x1b87400       0x2     0x0     0x1b87400
0x7ef85560:     0x0     0x1b87140       0x1b87140       0x759e88a4
0x7ef85570:     0x1b87330       0x759c71a9 <gst_base_sink_change_state+956>     0x140c  0x1b87330
0x7ef85580:     0x759e88a4      0x7ef855b4      0x0     0x7ef855b4
...
0x7ef85830:     0x76dbd6c4 <WebCore::AppendPipeline::resetPipeline()::__PRETTY_FUNCTION__>        0x4     0x3     0x1bfeb50
0x7ef85840:     0x0     0x76d59268      0x75135374      0x75135374
0x7ef85850:     0x76dbd6c4 <WebCore::AppendPipeline::resetPipeline()::__PRETTY_FUNCTION__>        0x1b7e300       0x1d651d0       0x75151b74

More info: 1

Sometimes the symbol names aren’t printed in the stack memdump. You can do this trick to iterate the stack and print the symbols found there (take with a grain of salt!):

(gdb) set $i = 0
(gdb) p/a *((void**)($sp + 4*$i++))

[Press ENTER multiple times to repeat the command]

$46 = 0xb6f9fb17 <_dl_lookup_symbol_x+250>
$58 = 0xb40a9001 <g_log_writer_standard_streams+128>
$142 = 0xb40a877b <g_return_if_fail_warning+22>
$154 = 0xb65a93d5 <WebCore::MediaPlayerPrivateGStreamer::changePipelineState(GstState)+180>
$164 = 0xb65ab4e5 <WebCore::MediaPlayerPrivateGStreamer::playbackPosition() const+420>
...

Many times it’s just a matter of gdb not having loaded the unstripped version of the library. /proc/<PID>/smaps and info proc mappings can help to locate the library providing the missing symbol. Then we can load it by hand.

For instance, for this backtrace:

#0  0x740ad3fc in syscall () from /home/enrique/buildroot-wpe/output/staging/lib/libc.so.6 
#1  0x74375c44 in g_cond_wait () from /home/enrique/buildroot-wpe/output/staging/usr/lib/libglib-2.0.so.0 
#2  0x6cfd0d60 in ?? ()

In a shell, we examine smaps and find out that the unknown piece of code comes from libgstomx:

$ cat /proc/715/smaps
...
6cfc1000-6cff8000 r-xp 00000000 b3:02 785380     /usr/lib/gstreamer-1.0/libgstomx.so
...

Now we load the unstripped .so in gdb and we’re able to see the new symbol afterwards:

(gdb) add-symbol-file /home/enrique/buildroot-wpe/output/build/gst-omx-custom/omx/.libs/libgstomx.so 0x6cfc1000
(gdb) bt
#0  0x740ad3fc in syscall () from /home/enrique/buildroot-wpe/output/staging/lib/libc.so.6
#1  0x74375c44 in g_cond_wait () from /home/enrique/buildroot-wpe/output/staging/usr/lib/libglib-2.0.so.0
#2  0x6cfd0d60 in gst_omx_video_dec_loop (self=0x6e0c8130) at gstomxvideodec.c:1311
#3  0x6e0c8130 in ?? ()

Useful script to prepare the add-symbol-file:

cat /proc/715/smaps | grep '[.]so' | sed -e 's/-[0-9a-f]*//' | { while read ADDR _ _ _ _ LIB; do echo "add-symbol-file $LIB 0x$ADDR"; done; }

More info: 1

The “figuring out corrupt ARM stacktraces” post has some additional info about how to use addr2line to translate memory addresses to function names on systems with a hostile debugging environment.

Debugging a binary without debug symbols

There are times when there’s just no way to get debug symbols working, or where we’re simply debugging on a release version of the software. In those cases, we must directly debug the assembly code. The gdb text user interface (TUI) can be used to examine the disassebled code and the CPU registers. It can be enabled with these commands:

layout asm
layout regs
set print asm-demangle

Some useful keybindings in this mode:

  • Arrows: scroll the disassemble window
  • CTRL+p/n: Navigate history (previously done with up/down arrows)
  • CTRL+b/f: Go backward/forward one character (previously left/right arrows)
  • CTRL+d: Delete character (previously “Del” key)
  • CTRL+a/e: Go to the start/end of the line

This screenshot shows how we can infer that an empty RefPtr is causing a crash in some WebKit code.

Wake up an unresponsive gdb on ARM

Sometimes, when you continue (‘c’) execution on ARM there’s no way to stop it again unless a breakpoint is hit. But there’s a trick to retake the control: just send a harmless signal to the process.

kill -SIGCONT 1234

Know which GStreamer thread id matches with each gdb thread

Sometimes you need to match threads in the GStreamer logs with threads in a running gdb session. The simplest way is to ask it to GThread for each gdb thread:

(gdb) set output-radix 16
(gdb) thread apply all call g_thread_self()

This will print a list of gdb threads and GThread*. We only need to find the one we’re looking for.

Generate a pipeline dump from gdb

If we have a pointer to the pipeline object, we can call the function that dumps the pipeline:

(gdb) call gst_debug_bin_to_dot_file_with_ts((GstBin*)0x15f0078, GST_DEBUG_GRAPH_SHOW_ALL, "debug")

by eocanha at April 20, 2021 06:00 AM

April 13, 2021

Enrique Ocaña GonzálezGStreamer WebKit debugging tricks using GDB (1/2)

I’ve been developing and debugging desktop and mobile applications on embedded devices over the last decade or so. The main part of this period I’ve been focused on the multimedia side of the WebKit ports using GStreamer, an area that is a mix of C (glib, GObject and GStreamer) and C++ (WebKit).

Over these years I’ve had to work on ARM embedded devices (mobile phones, set-top-boxes, Raspberry Pi using buildroot) where most of the environment aids and tools we take for granted on a regular x86 Linux desktop just aren’t available. In these situations you have to be imaginative and find your own way to get the work done and debug the issues you find in along the way.

I’ve been writing down the most interesting tricks I’ve found in this journey and I’m sharing them with you in a series of 7 blog posts, one per week. Most of them aren’t mine, and the ones I learnt in the begining of my career can even seem a bit naive, but I find them worth to share anyway. I hope you find them as useful as I do.

Breakpoints with command

You can break on a place, run some command and continue execution. Useful to get logs:

break getenv
command
 # This disables scroll continue messages
 # and supresses output
 silent
 set pagination off
 p (char*)$r0
continue
end

break grl-xml-factory.c:2720 if (data != 0)
command
 call grl_source_get_id(data->source)
 # $ is the last value in the history, the result of
 # the previous call
 call grl_media_set_source (send_item->media, $)
 call grl_media_serialize_extended (send_item->media, 
  GRL_MEDIA_SERIALIZE_FULL)
 continue
end

This idea can be combined with watchpoints and applied to trace reference counting in GObjects and know from which places the refcount is increased and decreased.

Force execution of an if branch

Just wait until the if chooses a branch and then jump to the other one:

6 if (i > 3) {
(gdb) next
7 printf("%d > 3\n", i);
(gdb) break 9
(gdb) jump 9
9 printf("%d <= 3\n", i);
(gdb) next
5 <= 3

Debug glib warnings

If you get a warning message like this:

W/GLib-GObject(18414): g_object_unref: assertion `G_IS_OBJECT (object)' failed

the functions involved are: g_return_if_fail_warning(), which calls to g_log(). It’s good to set a breakpoint in any of the two:

break g_log

Another method is to export G_DEBUG=fatal_criticals, which will convert all the criticals in crashes, which will stop the debugger.

Debug GObjects

If you want to inspect the contents of a GObjects that you have in a reference…

(gdb) print web_settings 
$1 = (WebKitWebSettings *) 0x7fffffffd020

you can dereference it…

(gdb) print *web_settings
$2 = {parent_instance = {g_type_instance = {g_class = 0x18}, ref_count = 0, qdata = 0x0}, priv = 0x0}

even if it’s an untyped gpointer…

(gdb) print user_data
(void *) 0x7fffffffd020
(gdb) print *((WebKitWebSettings *)(user_data))
{parent_instance = {g_type_instance = {g_class = 0x18}, ref_count = 0, qdata = 0x0}, priv = 0x0}

To find the type, you can use GType:

(gdb) call (char*)g_type_name( ((GTypeInstance*)0x70d1b038)->g_class->g_type )
$86 = 0x2d7e14 "GstOMXH264Dec-omxh264dec"

Instantiate C++ object from gdb

(gdb) call malloc(sizeof(std::string))
$1 = (void *) 0x91a6a0
(gdb) call ((std::string*)0x91a6a0)->basic_string()
(gdb) call ((std::string*)0x91a6a0)->assign("Hello, World")
$2 = (std::basic_string<char, std::char_traits<char>, std::allocator<char> > &) @0x91a6a0: {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x91a6f8 "Hello, World"}}
(gdb) call SomeFunctionThatTakesAConstStringRef(*(const std::string*)0x91a6a0)

See: 1 and 2

by eocanha at April 13, 2021 10:49 AM

April 12, 2021

Jan SchmidtRift CV1 – Getting close now…

(Jan Schmidt)

It’s been a while since my last post about tracking support for the Oculus Rift in February. There’s been big improvements since then – working really well a lot of the time. It’s gone from “If I don’t make any sudden moves, I can finish an easy Beat Saber level” to “You can’t hide from me!” quality.

Equally, there are still enough glitches and corner cases that I think I’ll still be at this a while.

Here’s a video from 3 weeks ago of (not me) playing Beat Saber on Expert+ setting showing just how good things can be now:

Beat Saber – Skunkynator playing Expert+, Mar 16 2021

Strap in. Here’s what I’ve worked on in the last 6 weeks:

Pose Matching improvements

Most of the biggest improvements have come from improving the computer vision algorithm that’s matching the observed LEDs (blobs) in the camera frames to the 3D models of the devices.

I split the brute-force search algorithm into 2 phases. It now does a first pass looking for ‘obvious’ matches. In that pass, it does a shallow graph search of blobs and their nearest few neighbours against LEDs and their nearest neighbours, looking for a match using a “Strong” match metric. A match is considered strong if expected LEDs match observed blobs to within 1.5 pixels.

Coupled with checks on the expected orientation (matching the Gravity vector detected by the IMU) and the pose prior (expected position and orientation are within predicted error bounds) this short-circuit on the search is hit a lot of the time, and often completes within 1 frame duration.

In the remaining tricky cases, where a deeper graph search is required in order to recover the pose, the initial search reduces the number of LEDs and blobs under consideration, speeding up the remaining search.

I also added an LED size model to the mix – for a candidate pose, it tries to work out how large (in pixels) each LED should appear, and use that as a bound on matching blobs to LEDs. This helps reduce mismatches as devices move further from the camera.

LED labelling

When a brute-force search for pose recovery completes, the system now knows the identity of various blobs in the camera image. One way it avoids a search next time is to transfer the labels into future camera observations using optical-flow tracking on the visible blobs.

The problem is that even sped-up the search can still take a few frame-durations to complete. Previously LED labels would be transferred from frame to frame as they arrived, but there’s now a unique ID associated with each blob that allows the labels to be transferred even several frames later once their identity is known.

IMU Gyro scale

One of the problems with reverse engineering is the guesswork around exactly what different values mean. I was looking into why the controller movement felt “swimmy” under fast motions, and one thing I found was that the interpretation of the gyroscope readings from the IMU was incorrect.

The touch controllers report IMU angular velocity readings directly as a 16-bit signed integer. Previously the code would take the reading and divide by 1024 and use the value as radians/second.

From teardowns of the controller, I know the IMU is an Invensense MPU-6500. From the datasheet, the reported value is actually in degrees per second and appears to be configured for the +/- 2000 °/s range. That yields a calculation of Gyro-rad/s = Gyro-°/s * (2000 / 32768) * (?/180) – or a divisor of 938.734.

The 1024 divisor was under-estimating rotation speed by about 10% – close enough to work until you start moving quickly.

Limited interpolation

If we don’t find a device in the camera views, the fusion filter predicts motion using the IMU readings – but that quickly becomes inaccurate. In the worst case, the controllers fly off into the distance. To avoid that, I added a limit of 500ms for ‘coasting’. If we haven’t recovered the device pose by then, the position is frozen in place and only rotation is updated until the cameras find it again.

Exponential filtering

I implemented a 1-Euro exponential smoothing filter on the output poses for each device. This is an idea from the Project Esky driver for Project North Star/Deck-X AR headsets, and almost completely eliminates jitter in the headset view and hand controllers shown to the user. The tradeoff is against introducing lag when the user moves quickly – but there are some tunables in the exponential filter to play with for minimising that. For now I’ve picked some values that seem to work reasonably.

Non-blocking radio

Communications with the touch controllers happens through USB radio command packets sent to the headset. The main use of radio commands in OpenHMD is to read the JSON configuration block for each controller that is programmed in at the factory. The configuration block provides the 3D model of LED positions as well as initial IMU bias values.

Unfortunately, reading the configuration block takes a couple of seconds on startup, and blocks everything while it’s happening. Oculus saw that problem and added a checksum in the controller firmware. You can read the checksum first and if it hasn’t changed use a local cache of the configuration block. Eventually, I’ll implement that caching mechanism for OpenHMD but in the meantime it still reads the configuration blocks on each startup.

As an interim improvement I rewrote the radio communication logic to use a state machine that is checked in the update loop – allowing radio communications to be interleaved without blocking the regularly processing of events. It still interferes a bit, but no longer causes a full multi-second stall as each hand controller turns on.

Haptic feedback

The hand controllers have haptic feedback ‘rumble’ motors that really add to the immersiveness of VR by letting you sense collisions with objects. Until now, OpenHMD hasn’t had any support for applications to trigger haptic events. I spent a bit of time looking at USB packet traces with Philipp Zabel and we figured out the radio commands to turn the rumble motors on and off.

In the Rift CV1, the haptic motors have a mode where you schedule feedback events into a ringbuffer – effectively they operate like a low frequency audio device. However, that mode was removed for the Rift S (and presumably in the Quest devices) – and deprecated for the CV1.

With that in mind, I aimed for implementing the unbuffered mode, with explicit ‘motor on + frequency + amplitude’ and ‘motor off’ commands sent as needed. Thanks to already having rewritten the radio communications to use a state machine, adding haptic commands was fairly easy.

The big question mark is around what API OpenHMD should provide for haptic feedback. I’ve implemented something simple for now, to get some discussion going. It works really well and adds hugely to the experience. That code is in the https://github.com/thaytan/OpenHMD/tree/rift-haptics branch, with a SteamVR-OpenHMD branch that uses it in https://github.com/thaytan/SteamVR-OpenHMD/tree/controller-haptics-wip

Problem areas

Unexpected tracking losses

I’d say the biggest problem right now is unexpected tracking loss and incorrect pose extractions when I’m not expecting them. Especially my right controller will suddenly glitch and start jumping around. Looking at a video of the debug feed, it’s not obvious why that’s happening:

To fix cases like those, I plan to add code to log the raw video feed and the IMU information together so that I can replay the video analysis frame-by-frame and investigate glitches systematically. Those recordings will also work as a regression suite to test future changes.

Sensor fusion efficiency

The Kalman filter I have implemented works really nicely – it does the latency compensation, predicts motion and extracts sensor biases all in one place… but it has a big downside of being quite expensive in CPU. The Unscented Kalman Filter CPU cost grows at O(n^3) with the size of the state, and the state in this case is 43 dimensional – 22 base dimensions, and 7 per latency-compensation slot. Running 1000 updates per second for the HMD and 500 for each of the hand controllers adds up quickly.

At some point, I want to find a better / cheaper approach to the problem that still provides low-latency motion predictions for the user while still providing the same benefits around latency compensation and bias extraction.

Lens Distortion

To generate a convincing illusion of objects at a distance in a headset that’s only a few centimetres deep, VR headsets use some interesting optics. The LCD/OLED panels displaying the output get distorted heavily before they hit the users eyes. What the software generates needs to compensate by applying the right inverse distortion to the output video.

Everyone that tests the CV1 notices that the distortion is not quite correct. As you look around, the world warps and shifts annoyingly. Sooner or later that needs fixing. That’s done by taking photos of calibration patterns through the headset lenses and generating a distortion model.

Camera / USB failures

The camera feeds are captured using a custom user-space UVC driver implementation that knows how to set up the special synchronisation settings of the CV1 and DK2 cameras, and then repeatedly schedules isochronous USB packet transfers to receive the video.

Occasionally, some people experience failure to re-schedule those transfers. The kernel rejects them with an out-of-memory error failing to set aside DMA memory (even though it may have been running fine for quite some time). It’s not clear why that happens – but the end result at the moment is that the USB traffic for that camera dies completely and there’ll be no more tracking from that camera until the application is restarted.

Often once it starts happening, it will keep happening until the PC is rebooted and the kernel memory state is reset.

Occluded cases

Tracking generally works well when the cameras get a clear shot of each device, but there are cases like sighting down the barrel of a gun where we expect that the user will line up the controllers in front of one another, and in front of the headset. In that case, even though we probably have a good idea where each device is, it can be hard to figure out which LEDs belong to which device.

If we already have a good tracking lock on the devices, I think it should be possible to keep tracking even down to 1 or 2 LEDs being visible – but the pose assessment code will have to be aware that’s what is happening.

Upstreaming

April 14th marks 2 years since I first branched off OpenHMD master to start working on CV1 tracking. How hard can it be, I thought? I’ll knock this over in a few months.

Since then I’ve accumulated over 300 commits on top of OpenHMD master that eventually all need upstreaming in some way.

One thing people have expressed as a prerequisite for upstreaming is to try and remove the OpenCV dependency. The tracking relies on OpenCV to do camera distortion calculations, and for their PnP implementation. It should be possible to reimplement both of those directly in OpenHMD with a bit of work – possibly using the fast LambdaTwist P3P algorithm that Philipp Zabel wrote, that I’m already using for pose extraction in the brute-force search.

Others

I’ve picked the top issues to highlight here. https://github.com/thaytan/OpenHMD/issues has a list of all the other things that are still on the radar for fixing eventually.

Other Headsets

At some point soon, I plan to put a pin in the CV1 tracking and look at adapting it to more recent inside-out headsets like the Rift S and WMR headsets. I implemented 3DOF support for the Rift S last year, but getting to full positional tracking for that and other inside-out headsets means implementing a SLAM/VIO tracking algorithm to track the headset position.

Once the headset is tracking, the code I’m developing here for CV1 to find and track controllers will hopefully transfer across – the difference with inside-out tracking is that the cameras move around with the headset. Finding the controllers in the actual video feed should work much the same.

Sponsorship

This development happens mostly in my spare time and partly as open source contribution time at work at Centricular. I am accepting funding through Github Sponsorships to help me spend more time on it – I’d really like to keep helping Linux have top-notch support for VR/AR applications. Big thanks to the people that have helped get this far.

by thaytan at April 12, 2021 08:29 AM

April 11, 2021

Andy Wingoguile's reader, in guile

(Andy Wingo)

Good evening! A brief(ish?) note today about some Guile nargery.

the arc of history

Like many language implementations that started life when you could turn on the radio and expect to hear Def Leppard, Guile has a bottom half and a top half. The bottom half is written in C and exposes a shared library and an executable, and the top half is written in the language itself (Scheme, in the case of Guile) and somehow loaded by the C code when the language implementation starts.

Since 2010 or so we have been working at replacing bits written in C with bits written in Scheme. Last week's missive was about replacing the implementation of dynamic-link from using the libltdl library to using Scheme on top of a low-level dlopen wrapper. I've written about rewriting eval in Scheme, and more recently about how the road to getting the performance of C implementations in Scheme has been sometimes long.

These rewrites have a quixotic aspect to them. I feel something in my gut about rightness and wrongness and I know at a base level that moving from C to Scheme is the right thing. Much of it is completely irrational and can be out of place in a lot of contexts -- like if you have a task to get done for a customer, you need to sit and think about minimal steps from here to the goal and the gut doesn't have much of a role to play in how you get there. But it's nice to have a project where you can do a thing in the way you'd like, and if it takes 10 years, that's fine.

But besides the ineffable motivations, there are concrete advantages to rewriting something in Scheme. I find Scheme code to be more maintainable, yes, and more secure relative to the common pitfalls of C, obviously. It decreases the amount of work I will have when one day I rewrite Guile's garbage collector. But also, Scheme code gets things that C can't have: tail calls, resumable delimited continuations, run-time instrumentation, and so on.

Taking delimited continuations as an example, five years ago or so I wrote a lightweight concurrency facility for Guile, modelled on Parallel Concurrent ML. It lets millions of fibers to exist on a system. When a fiber would need to block on an I/O operation (read or write), instead it suspends its continuation, and arranges to restart it when the operation becomes possible.

A lot had to change in Guile for this to become a reality. Firstly, delimited continuations themselves. Later, a complete rewrite of the top half of the ports facility in Scheme, to allow port operations to suspend and resume. Many of the barriers to resumable fibers were removed, but the Fibers manual still names quite a few.

Scheme read, in Scheme

Which brings us to today's note: I just rewrote Guile's reader in Scheme too! The reader is the bit that takes a stream of characters and parses it into S-expressions. It was in C, and now is in Scheme.

One of the primary motivators for this was to allow read to be suspendable. With this change, read-eval-print loops are now implementable on fibers.

Another motivation was to finally fix a bug in which Guile couldn't record source locations for some kinds of datums. It used to be that Guile would use a weak-key hash table to associate datums returned from read with source locations. But this only works for fresh values, not for immediate values like small integers or characters, nor does it work for globally unique non-immediates like keywords and symbols. So for these, we just wouldn't have any source locations.

A robust solution to that problem is to return annotated objects rather than using a side table. Since Scheme's macro expander is already set to work with annotated objects (syntax objects), a new read-syntax interface would do us a treat.

With read in C, this was hard to do. But with read in Scheme, it was no problem to implement. Adapting the expander to expect source locations inside syntax objects was a bit fiddly, though, and the resulting increase in source location information makes the output files bigger by a few percent -- due somewhat to the increased size of the .debug_lines DWARF data, but also due to serialized source locations for syntax objects in macros.

Speed-wise, switching to read in Scheme is a regression, currently. The old reader could parse around 15 or 16 megabytes per second when recording source locations on this laptop, or around 22 or 23 MB/s with source locations off. The new one parses more like 10.5 MB/s, or 13.5 MB/s with positions off, when in the old mode where it uses a weak-key side table to record source locations. The new read-syntax runs at around 12 MB/s. We'll be noodling at these in the coming months, but unlike when the original reader was written, at least now the reader is mainly used only at compile time. (It still has a role when reading s-expressions as data, so there is still a reason to make it fast.)

As is the case with eval, we still have a C version of the reader available for bootstrapping purposes, before the Scheme version is loaded. Happily, with this rewrite I was able to remove all of the cruft from the C reader related to non-default lexical syntax, which simplifies maintenance going forward.

An interesting aspect of attempting to make a bug-for-bug rewrite is that you find bugs and unexpected behavior. For example, it turns out that since the dawn of time, Guile always read #t and #f without requiring a terminating delimiter, so reading "(#t1)" would result in the list (#t 1). Weird, right? Weirder still, when the #true and #false aliases were added to the language, Guile decided to support them by default, but in an oddly backwards-compatible way... so "(#false1)" reads as (#f 1) but "(#falsa1)" reads as (#f alsa1). Quite a few more things like that.

All in all it would seem to be a successful rewrite, introducing no new behavior, even producing the same errors. However, this is not the case for backtraces, which can expose the guts of read in cases where that previously wouldn't happen because the C stack was opaque to Scheme. Probably we will simply need to add more sensible error handling around callers to read, as a backtrace isn't a good user-facing error anyway.

OK enough rambling for this evening. Happy hacking to all and to all a good night!

by Andy Wingo at April 11, 2021 07:51 PM

April 08, 2021

Andy Wingosign of the times

(Andy Wingo)

Hello all! There is a mounting backlog of things that landed in Guile recently and to avoid having to eat the whole plate in one bite, I'm going to try to send some shorter missives over the next weeks.

Today's is about a silly thing, dynamic-link. This interface is dlopen, but "portable". See, back in the day -- like, 1998 -- there were lots of kinds of systems and how to make and load a shared library portably was hard. You'd have people with AIX and Solaris and all kinds of weird compilers and linkers filing bugs on your project if you hard-coded a GNU toolchain invocation when creating loadable extensions, or hard-coded dlopen or similar to use them.

Libtool provided a solution to create portable loadable libraries, which involved installing .la files alongside the .so files. You could use libtool to link them to a library or an executable, or you could load them at run-time via the libtool-provided libltdl library.

But, the .la files were a second source of truth, and thus a source of bugs. If a .la file is present, so is an .so file, and you could always just use the .so file directly. For linking against an installed shared library on modern toolchains, the .la files are strictly redundant. Therefore, all GNU/Linux distributions just delete installed .la files -- Fedora, Debian, and even Guix do so.

Fast-forward to today: there has been a winnowing of platforms, and a widening of the GNU toolchain (in which I include LLVM as well as it has a mostly-compatible interface). The only remaining toolchain flavors are GNU and Windows, from the point of view of creating loadable shared libraries. Whether you use libtool or not to create shared libraries, the result can be loaded either way. And from the user side, dlopen is the universally supported interface, outside of Windows; even Mac OS fixed their implementation a few years back.

So in Guile we have been in an unstable equilibrium: creating shared libraries by including a probably-useless libtool into the toolchain, and loading them by using a probably-useless libtool-provided libltdl.

But the use of libltdl has not been without its costs. Because libltdl intends to abstract over different platforms, it encourages you to leave off the extension when loading a library, instead promising to try a platform-specific set such as .so, .dll, .dylib etc as appropriate. In practice the abstraction layer was under-maintained and we always had some problems on Mac OS, for example.

Worse, as ltdl would search through the path for candidates, it would only report the last error it saw from the underlying dlopen interface. It was almost always the case that if A and B were in the search path, and A/foo.so failed to load because of a missing dependency, the error you would get as a user would instead be "file not found", because ltdl swallowed the first error and kept trucking to try to load B/foo.so which didn't exist.

In summary, this is a case where the benefits of an abstraction layer decline over time. For a few years now, libltdl hasn't been paying for itself. Libtool is dead, for all intents and purposes (last release in 2015); best to make plans to migrate away, somehow.

In the case of the dlopen replacement, in Guile we ended up rewriting the functionality in Scheme. The underlying facility is now just plain dlopen, for which we shim a version of dlopen on Windows, inspired by the implementation in cygwin. There are still platform-specific library extensions, but that is handled easily on the Scheme layer.

Looking forward, I think it's probably time to replace Guile's use of libtool to create its libraries and executables. I loathe the fact that libtool puts shell scripts in the place of executables in build directories and stashes the actual executables elsewhere -- like, visceral revulsion. There is no need for that nowadays. Not sure what to replace it with, nor on what timeline.

And what about autotools? That, my friends, would be a whole nother blog post. Until then, & probably sooner, happy hacking!

by Andy Wingo at April 08, 2021 07:09 PM

April 06, 2021

Jean-François Fortin Tam“Getting Things GNOME” 0.5 released!

It is time to welcome a new release of the Rebuild of EvanGTGelion: 0.5, “You Can (Not) Improve Performance”!

This release of GTG has been 9 months in the making after the groundbreaking 0.4 release. While 0.4 was a major “perfect storm” overhaul, 0.5 is also a very technology-intensive release, even though it was done in a relatively short timeframe comparatively.

Getting Things GNOME 0.5 brings a truckload of user experience refinements, bugfixes, a completely revamped file format and task editor, and a couple of notable performance improvements. It doesn’t solve every performance problem yet (some remain), but it certainly improves a bunch of them for workaholics like me. If 0.4 felt a bit like a turtle, 0.5 is a definitely a much faster turtle.

If that was not enough already, it has some killer new features too. It’s the bee’s knees!

To benefit from one performance improvement in particular, it requires the new version of liblarch, 3.1, that we have released this month. GTG with the latest liblarch is available all-in-one in a Flatpak update near you. 📦

This release announcement and all that led up to it was, as you can imagine, planned using GTG:

“Who’s laughing now eh, señor Ruiz?”

As you can see, I came prepared. So continue reading below for the summary of improvements, I guarantee it’ll be worth it.


Some brief statistics

Beyond what is featured in these summarized release notes below, GTG itself has undergone roughly 400 changes affecting over 230 files, and received hundreds of bug fixes, as can be seen here. In fact, through this development cycle, we have crossed the 6,000 commits mark!

If you need another indication of how active the project has been in the last 9 months, I have received over 950 GTG-related emails since the 0.4 release last July.

We are thankful to all of our supporters and contributors, past and present, who made GTG 0.5 possible. Check out the “About” dialog for a list of contributors for this release.


New features

Recurring Tasks

Thanks to Mohieddine Drissi and Youssef Toukabri, tasks can now be made recurring (repeating). This allows you to keep reminding yourself that you need to clean your room every Saturday, and then not do it because it’s more fun to spend your whole Saturday fixing GTG bugs!

  • In the task editor, recurrence can be toggled with the dedicated menubutton that shows a nice user interface to manage the recurrence (daily, every other day, weekly, monthly and yearly).
  • Recurring tasks can also be added from the main window, using the “quick add” entry’s syntax.

Emojis as tag emblems 🦄

The old tag icons have been replaced by emojis. I decided it would be best for us to switch to emojis instead of icon-based emblems, because:

  • The availability of emblem icons varies a lot from one system to another, and from one icon theme to another;
  • In recent years, emblem icons on Linux have become quite a mismatch of styles, and their numbers in standard themes have also decreased;
  • Emojis are scalable by default, which makes them better for HiDPI displays (while traditional pixmap icons still are a hit-or-miss);
  • There is a much, much, much wider choice of emojis (for every conceivable situation!) than there are emblem icons on typical Linux systems;
  • It’s simply easier to deal with, in the long run. 😌

Unfortunately, this means that when you upgrade from 0.4 to 0.5, the old emblem icons will disappear and you will need to choose new emblems for your tasks.

Undead Hamster

The GTG plugin to interface with Hamster has been brought back to life by Francisco Lavin (issue #114 and pull request #465). I have not personally tested this. I’m still a bit worried about the ferrets rising from their graves.

Performance improvements
for power users

An elegant search trigger
for a more… civilized age

The filter-as-you-type global live search in GTG now uses a timeout approach (it waits a fraction of a second until after you stopped typing). This changes many things:

  • No more spamming the database and the UI’s treeview
  • The UI stays much more responsive, and you can see your characters being typed smoothly instead of appearing 15 seconds later
  • It won’t hog your CPU nearly as much
  • You get search results much faster: the results now tend to show up within a second after you stop typing, rather than a linear or exponential amount of time. For example, with 1000 tasks, it is now over 5 times faster (ex: bringing down the search time from 17+ seconds to 3 seconds), and the difference would be even more drastic if you have more tasks than that.

Optimizations to avoid processing all views all the time

As a result of the changes proposed in pull request #530 and follow-up PR #547, for issue #402, most operations switching between various representations of your tasks will now be 20% to 200% faster (depending on how your tasks are structured):

  • Faster startup
  • Faster switching between tags (particularly when inside the “Actionable” view)
  • Faster mid-night refreshing
  • It most likely makes the global search feature faster, too

However, changing between global view modes (Open/Actionable/Closed) is now a bit slower the first time around (but is instantaneous afterwards, until a filtering change occurs).

It’s a performance tradeoff, but it seems to be worth it. Especially when you have lots of tasks:

Note that to benefit from these changes, GTG 0.5 depends on the newly released version of liblarch (3.1) we have published this month.

Faster read/write operations on the data file 🚀

The switch to LXML as our parser ensures any operations on the local file format now happen instantly (less than a millisecond for a normal/lightweight task lists, and up to 25 milliseconds for heavy-duty 1000-tasks lists like mine). This was originally such a big deal that we thought we’d nickname this release “¿Que parser, amigo?” … but other major improvements kept happening since then, so this release is not just about the pasa!

Less micro-freezes when focusing task windows

Mr. Bean waitingPictured: me, waiting for the task editor windows to un-freeze when I focus them.

Users with a thousand tasks will probably have noticed that GTG 0.4 had a tendency to painfully lock up for a couple of seconds when focusing/defocusing individual task editor windows.

Sure, “It doesn’t catastrophically lock up in your face multiple times per day anymore” doesn’t sound like a feature in theory, but it is a usability feature in practice.

Now, even when typing inside the new Task Editor and making lots of changes, it remains butter-smooth.

Part of it might be because it now uses a timer-based approach to parsing, and part of it is probably due to the code having been refactored heavily and being of higher quality overall. It just tends to be better all around now, and I’ve tried fairly hard to break it during my smoke-testing, especially since a lot of the improvements have also been tied to the major file format redesign that Diego implemented.

All in all, I’m not sure exactly how it accomplishes better reliability, to be honest, so we’ll just assume Diego secretly is a warlock.

50% more awesome task editing

Oh yeah, speaking of the task editor…

Rewriting the Task Editor’s content area and its natural language parser brought many user-visible improvements in addition to the technological improvements:

  • The new timeout-based parser is faster (because we’re not spamming it like barbarians), more reliable, and now gains the ability to recognize subtasks (lines starting with “-“) live, instead of “after pressing Enter.” This also makes copy-pasting from a dashed list easier, as it will be converted into many subtasks in one go
  • Clicking on a tag inside the task editor’s text selects the tag in the main window
  • Tags’ text highlight color now uses the tag’s chosen emblem color
  • Subtasks now have GTK checkboxes!
  • Completed subtasks show up in normal strikethrough text style
  • Opening a subtask takes care of closing the parent task, and the other way around (opening a parent task closes the subtask window), minimizing the amount of utility windows cluttering your view (and reducing the chances of making conflicting changes)
  • Now supports subheadings, in Markdown format (type # title). Now that it supports some basic rich-text formatting features, even Grumpy Sri will be happy with this release:

Here is a short demonstration video that lets you see the improved “parsing as you type” engine in action:

Other user interface improvements

General bug fixes

Technological changes

  • Eliminate the pyxdg dependency in favor of directly using GLib
  • dbus-python dependency dropped in favor of Gio (except in the Hamster plugin)
  • New and improved file format (see issue 279, “Better XML format [RFC]“, and my previous spooky blog post):
    • Automatically converts upon first startup, nothing to do manually
    • More consistent
    • Uses a single file instead of three separate ones
    • Uses valid XML
    • Fully documented (see docs/contributors/file format.md)
    • Adds file format versioning to make it future-proof
    • Uses LXML as the parser, which is reportedly 40 times faster (see issue #285)
    • Fundamentally allows the task editor to be a lot more reliable for parsing, as we ditch the XML-in-XML matryoshka approach
    • Solves a few other bugs related to (de)serializing
  • Rewrote the Task Editor’s “TaskView” (content area) and its natural language parser, paving the way for more improvements and increased reliability:
    • The timeout-based approach mentioned earlier means that parsing happens less often, “when it needs to”, and is also less error-prone as a result.
    • With the implementation of subheadings, the code now has the foundations for more Markdown features in the future.
    • The taskview supports invisible characters that turn visible when the cursor passes over them, which could be useful for more text tags in the future.
    • Easier to maintain improved performance in the future: When running in debug mode (launch.sh -d), the process function now prints the time it took to process the buffer. This will help us stay conscious of the performance.
    • Support for linking between tasks with gtg://[TASK_ID] (no UI for this yet, though)
    • Better code overall, a lot more extensible and easier to work on
  • Rewrote the Quick Add entry’s parsing and added unit tests
  • Command line parsing ported to GLib, so GLib/GTK-specific options such as --display are now available
  • GTG is now dbus-activatable. Although no API that other applications can use has been implemented yet (outside of “standard” GTK ones), this opens up the possibility in the future, such as the GNOME Shell search integration.

Developer Experience improvements

Want to be part of a great team? Join BAHRAM GTG!

Here are some changes to primarily benefit users crazy enough to be called developers.

  • New plugin: “Developer Console”.
    Play with GTG’s internals from the comfort of GTG itself! Remember: when exiting your Orbital Frame, wear… a helmet.
  • New parameter in launch.sh: -w to enable warnings
    (in case you still are not wearing a helmet, or want to catch more bugs)
  • Better help message for launch.sh, documenting the available options
  • Include Git commit version in the about window
  • Use the proper exit code when the application quits cleanly
  • We closed all the tickets on LaunchPad (over 300 of them) so there is no confusion when it comes to reporting bugs and the current state of the app’s quality and priorities, and to ensure that issues are reported in the currently used bug tracker where development actually happens.

Help out Diego!

It is no secret that Diego has been spending a lot of time and effort making this release happen, just like the 0.4 release. The amount and depth of his changes are quite amazing, so it’s no surprise he’s the top contributor here:

If you would like to show your appreciation and support his maintainership work with some direct monthly donations to him, please consider giving a couple of dollars per month on his gumroad page or his liberapay page!

More coming in the next release

Many new features, plugins, and big code changes have been held back from merging so that we could release 0.5 faster, so a bunch of other changes will land in the development version on the way to 0.6.

“This blog post is ridiculously long! And we still have 10 pull requests almost ready to merge!”
— Diego, five minutes before I published this blog post

Spreading this announcement

We have made some social postings on Twitter, on Mastodon and on LinkedIn that you can re-share/retweet/boost. Please feel free to link to this announcement on forums and blogs as well!

by Jeff at April 06, 2021 02:00 PM

March 25, 2021

Andy Wingohere we go again

(Andy Wingo)

Around 18 months ago, Richard Stallman was forced to resign from the Free Software Foundation board of directors and as president. It could have been anything -- at that point he already had a history of behaving in a way that was particularly alienating to women -- but in the end it was his insinuation that it was somehow OK if his recently-deceased mentor Marvin Minsky, then in his 70s or 80s, had sex with a 17-year-old on Jeffrey Epstein's private island. A weird pick of hill to stake one's reputation on, to say the least.

At the time I was relieved that we would finally be getting some leadership renewal at the FSF, and hopeful that we could get some mission renewal as well. I was also looking forward to the practical implications would be for the GNU project, as more people agreed that GNU was about software freedom and not about its founder.

But now we're back! Not only has RMS continued through this whole time to insist that he runs the GNU project -- something that is simply not the case, in my estimation -- but this week, a majority of a small self-selected group of people, essentially a subset of current and former members of the FSF board of directors and including RMS himself, elected to reinstate RMS to the board of the Free Software Foundation. Um... read the room, FSF voting members? What kind of message are you sending?

In this context I can only agree with the calls for the entire FSF board to resign. The board is clearly not fit for purpose, if it can make choices like this.

dissociation?

I haven't (yet?) signed the open letter because I would be in an inconsistent position if I did so. The letter enjoins people to "refuse to contribute to projects related to the FSF and RMS"; as a co-maintainer of GNU Guile, which has its origins in the heady 1990s of the FSF but has nothing to do any more with RMS, but whose copyrights are entirely held by the FSF, is hosted on FSF-run servers, and is even obliged (GPLv3 §5d, as referenced by LGPLv3) to print out Copyright (C) 1995-2021 Free Software Foundation, Inc. when it starts, I must admit that I contribute to a project that is "related to the FSF". But I don't see how Guile could continue this association, if the FSF board continues as it is. It's bad for contributors and for the future of the project.

It would be very tricky to disentangle Guile from the FSF -- consider hosting, for example -- so it's not the work of a day, but it's something to think about.

Of course I would rather that the FSF wouldn't allow itself to be seen as an essentially misogynist organization. So clean house, FSF!

on the nature of fire

Reflecting on how specifically we could have gotten here -- I don't know. I don't know the set of voting members at the FSF, what discussions were made, who voted what. But, having worked as a volunteer on GNU projects for almost two decades now, I have a guess. RMS and his closest supporters see themselves as guardians of the flame of free software -- a lost world of the late 70s MIT AI lab, reborn in a flurry of mid-80s hack, but since 25 years or so, slipping further and further away. These are dark times, in their view, and having the principled founder in a leadership role can only be a good thing.

(Of course, the environment in the AI lab was only good for some. The treatment of Margaret Hamilton as recounted in Levy's Hackers shows that not all were welcome. If this were just one story, I would discount it, but looking back, it does seem to be part of a pattern.)

But is that what the FSF is for today? If so, Guile should certainly leave. I'm not here for software as perfomative nostalgia -- I'm here to have fun with friends and start a fire. The FSF should look to do the same -- look at the world we are in, look where the energy is now, and engage in real conversations about success and failure and tactics. There is a world to win and doubling down on RMS won't get us there from here.

by Andy Wingo at March 25, 2021 12:22 PM

March 20, 2021

Sebastian Pölsterlscikit-survival 0.15 Released

I am proud to announce the release if version 0.15.0 of scikit-survival, which brings support for scikit-learn 0.24 and Python 3.9. Moreover, if you fit a gradient boosting model with loss='coxph', you can now predict the survival and cumulative hazard function using the predict_cumulative_hazard_function and predict_survival_function methods.

The other enhancement is that cumulative_dynamic_auc now supports evaluating time-dependent predictions. For instance, you can now evaluate the predicted time-dependent risk of a RandomSurvivalForest rather than just evaluating the predicted total number of events per instance, which is what RandomSurvivalForest.predict returns.

All you have to do is create an array where the columns are the predictions at the time points you want to evaluate. The snippet below summarizes the idea:

from sksurv.ensemble import RandomSurvivalForest
from sksurv.metrics import cumulative_dynamic_auc
rsf = RandomSurvivalForest()
rsf.fit(X_train, y_train)
chf_funcs = rsf.predict_cumulative_hazard_function(X_test)
time_points = np.array([30, 60, …])
risk_scores = np.row_stack([
chf(time_points) for chf in chf_funcs
])
aucs, mean_auc = cumulative_dynamic_auc(
y_train, y_test, risk_scores, time_points
)

For a complete example, please have a look at the User Guide.

If you want to know about all changes in scikit-survival 0.15.0, please see the release notes.

As usual, pre-built conda packages are available for Linux, macOS, and Windows via

 conda install -c sebp scikit-survival

Alternatively, scikit-survival can be installed from source following these instructions.

March 20, 2021 07:58 PM

March 15, 2021

Christian SchallerWhat to look for in Fedora Workstation 34

(Christian Schaller)

As we are heading towards April and the release of Fedora Workstation 34 I wanted to post an update on what we are working on for this release and what we are looking at going forward. 2020 was a year where we focused a lot on polishing what we had and getting things past the finish line and Fedora Workstation 34 is going to be the culmination of that effort in many ways.

Wayland:
The big ticket item we have wanted to close off on was Wayland, because while Wayland has been production ready for most of us for a while, there was still some cases it didn’t cover as well as X.org. The biggest of this was of course the lack of accelerated XWayland support with the binary NVidia driver. Fixing that issue of course wasn’t something we could do ourselves, but we have been working diligently with our friends at NVidia to help ensure everything was in place for them to enable that support in their driver, so I have been very happy to see the public reports confirming that NVidia will have accelerated 3D in the summer release of their driver. The other Wayland area we have put a lot of effort into has been the work undertaken by Jonas Ådahl to get headless display support working with Wayland. This is a critical feature for people who for instance want a desktop instance on their servers or in the cloud, who want a desktop they access through things like VNC or RDP to use for sysadmin related tasks. Jonas spent a lot of time laying the groundwork for this over the course of last year and we are now in the final stages of merging the patches to enable this feature in GNOME and Wayland in preparation for Fedora Workstation 34. Once those two items are out we consider our Wayland rampup/rollout to be complete, so while there of course will continue to be bugfixes and new features implemented, that will be part of a natural evolution of Wayland and not part of a ‘close gaps with X11’ effort like now.

PipeWire
Another big ticket item we are hoping to release fully in Fedora Workstation 34 is PipeWire. PipeWire as most of you know is the engine we use to deal with handling video streams in a secure and shareable away in Fedora Workstation, so when you interact with your web camera(s) or do screen casting or make screenshots it is all routed and handled by PipeWire. But in Fedora Workstation 34 we are aiming to also switch to using PipeWire for audio, to replace both PulseAudio and Jack. For those of you who had read any previous blog post from me you will know what an important step forward this will be as we would finally be making the pro-audio community first class citizens in Fedora Workstation and Linux in general. When we decided to try to switch to PipeWire in Fedora Workstation 34 I have to admit I was a little skeptical about if we would be able to get all things ready in time as there are so many things that needs to be tested and fixed when you switch out such a critical component. Up to that point we had a lot of people interested in PipeWire, but only limited community involvement, but I feel the announcement of bringing in PipeWire for Fedora Workstation 34 galvanized the community around the project and we now have a very active community around PipeWire in #pipewire on the freenode IRC network. Not only is Wim Taymans getting a ton of help with testing and verification, but we also see a stead stream of patches coming in, with for instance improved Bluetooth audio support being contributed, in fact I believe that PipeWire will be able to usher in better bluetooth audio support in Fedora than we ever had before, with great support for high quality Bluetooth audio codecs like LDAC.

I am especially happy to see so many of the key members of the pro-audio Linux community taking part in this effort and is of course also happy to see many pro-audio folks testing Fedora Workstation for the first time due to this effort. The community is working closely with Wim to test and verify as many important ProAudio applications as possible and work to update Fedora packaging as needed to ensure they can transition from Jack to PipeWire without dependency conflicts or issues. One last item to mention here is that you might have seen that Red Hat is getting into the automotive space, I can’t share a lot of details about that effort, but one thing I can say is that PipeWire will be a core part of it and thus we will soon be looking to hire more engineers to work on PipeWire, so if that is of interest to you be sure to track my twitter feed or blog as I will announce our job openings there as they become available. For the community at large this should be great too as it means that we can get a lot of synergy between automotive and the desktop around audio and video handling.

It is still somewhat of an open question if we end up actually switching to PipeWire in Fedora Workstation 34, but things are looking good at this point in time and worst case scenario it will be in place for Fedora Workstation 35.

Toolbox
Toolbox is another effort that is in a great spot now. Toolbox is our tool for making working with pet containers a breeze for developers. The initial version was prototyped quickly by writing it as a shell script, but we spent time last year getting it rewritten in Go in order to make it possible to keep expanding the project and allow us to implement all the things we envision for it. With that done feature work is now in focus again and Ondřej Michal has done some great work making it possible to set up RHEL containers in Toolbox. This means that you can run Fedora on your laptop and get the latest and greatest features that way, but you can do your development in a RHEL pet container, so you get an environment identical to what you applications will see once they are deployed into the cloud or onto company hardware. This gives you the best of both worlds in my opinion, the fast moving Fedora Workstation that brings the most out of our laptop and desktop hardware, but still easy access to the RHEL platform for development and testing. You can even test this today on Fedora Workstation 33, just open a terminal and type ‘toolbox create --distro rhel --release 8.3‘. The resulting toolbox will then be based on RHEL and not Fedora and thus perfect for doing RHEL targeted development. You will need to use the subscription-manager tool to register it (be sure to register on developer.redhat.com for your free RHEL developer subscription. Over time we hope to integrate this into GNOME Online accounts like we do for the RHEL virtual machines you can set up with GNOME Boxes, so that once you set up your RHEL account you can create RHEL virtual machines and RHEL containers easily on Fedora Workstation.

Toolbox with RHEL

Toolbox pet container with RHEL UBI

Flatpak
Owen Taylor has been doing some incredible work behind the scenes for the last year trying to ensure the infrastructure we have in RHEL and Fedora provides a great integrated Flatpak experience. As we move forward we expect Flatpaks to be the primary packaging format that Fedora users consume their applications in, but to make that a reality we needed to ensure the experience is good both for Fedora maintainers and for the end users. So one of the big ticket items Owen been working on is getting incremental updates working in Fedora. If you have used applications from Flathub you probably noticed that their updates are small and nimble despite being packaged as Flatpak containers, while the Fedora flatpaks causes big updates each time. The reason for this is that the Fedora flatpaks are shipping as OCI (Open Container Initiative) images, while the Flatpaks on Flathub are shipping as OStree repositories (if you don’t know OStree, think of it as git for binaries). So shipping the Flatpaks as OCI images has advantages in the form of being the same format we at Red Hat use for our kubernetes/docker/openshift containers and thus it allows us to reuse a lot of the work that Red Hat as put into ensuring we can provide and keep such containers up to date and secure, but the downside until now has been that these containers where shipped in a way which cause each update, no matter how small the change, to be a full re-download of the whole image. Well Owen Taylor and Alex Larsson worked together to resolve this and came up with a method to allow incremental updates of such containers and thus bring the update sizes in line with what you see on Flathub for Flatpaks. This should be deployed in time for Fedora Workstation 34 and we also hope to  eventually deploy it for  kubernetes/docker containers too. Finally to make even more applications available we are doing work to enable people to get access to Flathub.org out of the box in Fedora when you enable 3rd party repositories in initial setup, so that your our of the box application selection will be even bigger.

Flathub frontpage

Flathub webpage

GNOME 40
Another major change in Fedora Workstation 34 is GNOME40 which contains a revamp of the GNOME 3 user interface. This was a collaborative effort between a lot of GNOME 3 stakeholders with Allan Day representing Red Hat. This was also an effort by the GNOME design community to up their game and thus as part of the development process the GNOME Foundation paid a professional company to do user testing on the proposed changes and some of the alternatives. This means that the changes where verified to actually be experienced as an improvement for the experienced GNOME user participants and that it felt intuitive for new users of GNOME. One advantage we have in Fedora is that since we don’t do major tweaking of the GNOME user interface which means once Fedora Workstation 34 ships you are set to enjoy the new GNOME experience from day one. For long time GNOME users I hope and expect that the updates will be a welcome refresh and at the same time that the changes provide a more easy onramp for new GNOME and Fedora Workstation users. Some of the early versions did lead some long term fans of how multimonitor support in GNOME3 worked to be a bit concerned, but be assured that multi monitor is a critical usecase in our opinion and something we have been looking at and will be looking at keep improving. In fact Allan Day wrote a great blog post about GNOME40 multimonitor support recently to explain what we are doing and how we see it evolving going forward.

Input
Another area where we keep putting in a lot of effort is input. Thanks to Peter Hutterer and Benjamin Tissoires we keep making sure Fedora Workstation and the world of Linux keeps having access to the newest and best in input. The latest effort they are working on has been to enable haptic touchpads. Haptics touchpads should be familiar among people who tried Apple hardware, but they are expected to appear in force on laptops in general this year, so we have been putting in the effort to ensure that we can support this new type of device as they come out. So if you see laptops you want with haptic touchpads then Fedora Workstation should be ready for it, but of course until these devices are commonplace and we had a chance to test and verify I can make no guarantees.

Another major effort that we undertook in relation to input was move the GNOME input to a separate thread. Carlos Garnacho worked on this patch to make that happen. This should provide a smoother experience with Fedora Workstation 34 as it means the mouse should not stall due to the main thread running Wayland being busy. This was done as part of the overall performance work we been continuously doing over the last years to ensure to address performance issues and make Fedora and GNOME have the best performance and performance related behaviour possible.

Lenovo Laptops
So one of the major announcements of last year was Lenovo Laptops with Fedora Linux pre-installed. There are currently two models available with Linux, the X1 Carbon and the Lenovo P1. Between them they cover the two most commons requests we see, a ultralight weight laptop with the X1 and a more powerful ‘portable workstation’ model with the P1. We are working on a couple of more models and also to get them sold globally, which was key goal of the effort. Be aware that both models are on sale as I am writing this (hopefully still true when you read this), so it is a good time to grab a great laptop with a great OS.

Lenovo P1

Lenovo P1

Vision
So one thing I wanted to do to is tie the work we do in Fedora Workstation together by articulating what we are trying to achieve. Fedora has for the longest time been the place where Linux as an operating system is evolving and being developed. There are very few major innovations that has come to Linux that hasn’t been developed and incubated in Fedora by Fedora contributors, including of course Red Hat. This include things like the Linux Vendor Firmware Service, Wayland, Flatpak, SilverBlue, PipeWire, SystemD, flicker free boot, HiDPI support, gaming mouse support and so much more. We have always done this work in close cooperation with the upstreams we are collaborating with, which is why the patch delta in any given Fedora release is small. We work hard to get improvements into the upstream kernel and into GNOME and so on right away, to avoid needing to ship downstream patches in Fedora. That of course saves us from having to maintain temporary forks, but more importantly it is the right way to collaborate with an open source community.

So looking back to when we launched Fedora Workstation we realized that being at the front like that had come at the cost of not being stable and user friendly. So the big question we tried to ask ourselves when launching Fedora Workstation and the question that still drives a lot of our decision making and focus is : how can we preserve being the heart and center of Linux OS development, but at the same time provide end users with a stable and well functioning system? To achieve that we have done lot of changes over the last years, ranging from some policy changes in terms of how and when we brought changes into Fedora, but maybe even more importantly we focused on a lot on figuring out ways to reduce the challenges caused by a rapidly evolving OS, like the introduction of Flatpaks to allow applications to be developed and released without strong ties to the host system libraries and with the concepts we are maturing in Silverblue around image based operating systems or how we are looking at pet container development with Toolbox. All of these things combined remove a lot of the fragility we seen in Linux up to this point and instead let us treat the rapidly evolving linux landscape as a strength.

So where we are today is that I think we are very close to realizing the vision of being able to let Fedora be the place where exiting new stuff happens, yet at the same time provide the robustness and polish that end users need to be able to use it as their daily driver, it has been my daily driver for many years now and by the rapid growth of users we seen in Fedora over the last 5 years I think that is true for a lot of other people too. The goal is to allow the wider community around Linux, especially the developers, sysadmins and creators relying on Linux to do their work, to come to Fedora and interact and collaborate with the developers working on the OS itself to the benefit of all. You all are probably better judges than me to if we are succeeding with that, but I do take the increased chatter and adoption of Fedora by a lot of people doing Linux related podcasts, news sites and so on as a sign that we are succeeding. And PipeWire for me is a perfect example of how this can look, where we want to bring in the pro-audio creators to Fedora Workstation and let them interact and work closely with Wim Taymans and the PipeWire development team to make the experience even better for themselves and their fellow creators and at the same time give them a great stable platform to create their music on.

by uraeus at March 15, 2021 06:20 PM

GStreamerGStreamer 1.18.4 stable bug fix release

(GStreamer)

The GStreamer team is pleased to announce another bug fix release in the stable 1.18 release series of your favourite cross-platform multimedia framework!

This release only contains bugfixes and important security fixes, and it should be safe to update from 1.18.x.

Highlighted bugfixes:

  • important security fixes for ID3 tag reading, matroska and realmedia parsing, and gst-libav audio decoding
  • audiomixer, audioaggregator: input buffer handling fixes
  • decodebin3: improve stream-selection message handling
  • uridecodebin3: make "caps" property work
  • wavenc: fix writing of INFO chunks in some cases
  • v4l2: bt601 colorimetry, allow encoder resolution changes, fix decoder frame rate negotiation
  • decklinkvideosink: fix auto format detection, and fixes for 29.97fps framerate output
  • mpeg-2 video handling fixes when seeking
  • avviddec: fix bufferpool negotiation and possible memory corruption when changing resolution
  • various stability, performance and reliability improvements
  • memory leak fixes
  • build fixes: rpicamsrc, qt overlay example, d3d11videosink on UWP

See the GStreamer 1.18.4 release notes for more details.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

Download tarballs directly here: gstreamer, gst-plugins-base, gst-plugins-good, gst-plugins-ugly, gst-plugins-bad, gst-libav, gst-rtsp-server, gst-python, gst-editing-services, gst-devtools, gstreamer-vaapi, gstreamer-sharp, gst-omx, or gstreamer-docs.

March 15, 2021 04:00 PM

March 08, 2021

Andy Wingo99% spam

(Andy Wingo)

Hey all, happy new year apparently! A quick service update on the old wingolog. For some time the site has been drowning in spam comments, despite my best efforts to point a bayesian classifier at the problem.

I don't keep logs of the number of attempts at posting comments that don't pass the classifier. But what I can say is that since I put in the classifier around 4 years ago, about 2500 comments a year made it through -- enough to turn the comment section into a bit of a dump. Icky, right??

At the same time of course, that's too many comments to triage manually, so I never got around to fixing the problem. So in fact I had two problems: lots 'o spam, and lots 'o incoming spam.

With regards to the existing spam, I took a heavyhanded approach. I took a look at all good emails and URLs that people had submitted for comments prior to 2017, assuming they were triaged. Then I made a goodlist of comments since 2017 that had those comments or emails. There were very few of those -- maybe 50 or 70 or so.

Then I took a look at when comments were made relative to the posts. Turns out, 98.3% of comments were made more than 21 days after their parent post was published -- and sometimes years afterwards. I used that as a first filter, that if a post wasn't from a known poster, and was made 3 weeks or more after the post, I just classified it as spam.

The list of comments made within 3 weeks of the parent post was small enough for me to triage manually, and there I was able to save a bit of wheat from the chaff. In the end, though, the result of winnowing was less than 1% of what went in.

As I don't really want to babysit this wobsite, I'll keep this policy in place in the future -- comments will be open for a while after articles are posted. Hopefully that should keep the ol' wingolog in tidy shape going forward, while still permitting people to comment -- something I have really appreciated in the past.

So happy 2021 to everybody, may the vaccine gods shine upon your shoulders, and happy hacking :)

by Andy Wingo at March 08, 2021 02:36 PM

February 27, 2021

Jean-François Fortin TamA new data format has landed in the upcoming GTG 0.5

Here’s a general call for testing from your favorite pythonic native Linux desktop personal productivity app, GTG.

In recent months, Diego tackled the epic task of redesigning the XML file format from a new specification devised with the help of Brent Saner (proposal episodes 1, 2 and 3), and then implementing the new file format in GTG. This work has now been merged to the main development branch on GTG’s git repository:

Diego’s changes are major, invasive technological changes, and they would benefit from extensive testing by everybody with “real data” before 0.5 happens (very soon). I’ve done some pretty extensive testing & bug reporting in the last few months; Diego fixed all the issues I’ve reported so far, so I’ve pretty much run out of serious bugs now, as only a few remain targetted to the 0.5 milestone… But I’m only human, and it is possible that issues might remain, even after my troll-testing.

Grab GTG’s git version ASAP, with a copy of your real data (for extra caution, and also because we want you to test with real data); see the instructions in the README, including the “Where is my user data and config stored?” section.

Please torture-test it to make sure everything is working properly, and report issues you may find (if any). Look for anything that might seem broken “compared to 0.4”, incorrect task parenting/associations, incorrect tagging, broken content, etc.

If you’ve tried to break it and still couldn’t find any problems, maybe one way to indicate that would be a “👍” on the merge request—I’m not sure we really have another way to know if it turns out that “everything is OK” 🙂

Your help in testing this (or spreading the word) will help ensure a smooth transition for users getting an upgrade from 0.4 to 0.5, letting us release 0.5 with confidence. Thanks!

by Jeff at February 27, 2021 11:53 PM

February 04, 2021

Jan SchmidtRift CV1 – Testing SteamVR

(Jan Schmidt)

I’ve had a few people ask how to test my OpenHMD development branch of Rift CV1 positional tracking in SteamVR. Here’s what I do:

  • Make sure Steam + SteamVR are already installed.
  • Clone the SteamVR-OpenHMD repository:
git clone --recursive https://github.com/ChristophHaag/SteamVR-OpenHMD.git
  • Switch the internal copy of OpenHMD to the right branch:
cd subprojects/openhmd
git remote add thaytan-github https://github.com/thaytan/OpenHMD.git
git fetch thaytan-github
git checkout -b rift-kalman-filter thaytan-github/rift-kalman-filter
cd ../../
  • Use meson to build and register the SteamVR-OpenHMD binaries. You may need to install meson first (see below):
meson -Dbuildtype=release build
ninja -C build
./install_files_to_build.sh
./register.sh
  • It is important to configure in release mode, as the kalman filtering code is generally too slow for real-time in debug mode (it has to run 2000 times per second)
  • Make sure your USB devices are accessible to your user account by configuring udev. See the OpenHMD guide here: https://github.com/OpenHMD/OpenHMD/wiki/Udev-rules-list
  • Please note – only Rift sensors on USB 3.0 ports will work right now. Supporting cameras on USB 2.0 requires someone implementing JPEG format streaming and decoding.
  • It can be helpful to test OpenHMD is working by running the simple example. Check that it’s finding camera sensors at startup, and that the position seems to change when you move the headset:
./build/subprojects/openhmd/openhmd_simple_example
  • Calibrate your expectations for how well tracking is working right now! Hint: It’s very experimental 🙂
  • Start SteamVR. Hopefully it should detect your headset and the light(s) on your Rift Sensor(s) should power on.

Meson

I prefer the Meson build system here. There’s also a cmake build for SteamVR-OpenHMD you can use instead, but I haven’t tested it in a while and it sometimes breaks as I work on my development branch.

If you need to install meson, there are instructions here – https://mesonbuild.com/Getting-meson.html summarising the various methods.

I use a copy in my home directory, but you need to make sure ~/.local/bin is in your PATH

pip3 install --user meson

by thaytan at February 04, 2021 01:58 AM

January 31, 2021

Jan SchmidtRift CV1 – Pose rejection

(Jan Schmidt)

I spent some time this weekend implementing a couple of my ideas for improving the way the tracking code in OpenHMD filters and rejects (or accepts) possible poses when trying to match visible LEDs to the 3D models for each device.

In general, the tracking proceeds in several steps (in parallel for each of the 3 devices being tracked):

  1. Do a brute-force search to match LEDs to 3D models, then (if matched)
    1. Assign labels to each LED blob in the video frame saying what LED they are.
    2. Send an update to the fusion filter about the position / orientation of the device
  2. Then, as each video frame arrives:
    1. Use motion flow between video frames to track the movement of each visible LED
    2. Use the IMU + vision fusion filter to predict the position/orientation (pose) of each device, and calculate which LEDs are expected to be visible and where.
  3. Try and match up and refine the poses using the predicted pose prior and labelled LEDs. In the best case, the LEDs are exactly where the fusion predicts they’ll be. More often, the orientation is mostly correct, but the position has drifted and needs correcting. In the worst case, we send the frame back to step 1 and do a brute-force search to reacquire an object.

The goal is to always assign the correct LEDs to the correct device (so you don’t end up with the right controller in your left hand), and to avoid going back to the expensive brute-force search to re-acquire devices as much as possible

What I’ve been working on this week is steps 1 and 3 – initial acquisition of correct poses, and fast validation / refinement of the pose in each video frame, and I’ve implemented two new strategies for that.

Gravity Vector matching

The first new strategy is to reject candidate poses that don’t closely match the known direction of gravity for each device. I had a previous implementation of that idea which turned out to be wrong, so I’ve re-worked it and it helps a lot with device acquisition.

The IMU accelerometer and gyro can usually tell us which way up the device is (roll and pitch) but not which way they are facing (yaw). The measure for ‘known gravity’ comes from the fusion Kalman filter covariance matrix – how certain the filter is about the orientation of the device. If that variance is small this new strategy is used to reject possible poses that don’t have the same idea of gravity (while permitting rotations around the Y axis), with the filter variance as a tolerance.

Partial tracking matches

The 2nd strategy is based around tracking with fewer LED correspondences once a tracking lock is acquired. Initial acquisition of the device pose relies on some heuristics for how many LEDs must match the 3D model. The general heuristic threshold I settled on for now is that 2/3rds of the expected LEDs must be visible to acquire a cold lock.

With the new strategy, if the pose prior has a good idea where the device is and which way it’s facing, it allows matching on far fewer LED correspondences. The idea is to keep tracking a device even down to just a couple of LEDs, and hope that more become visible soon.

While this definitely seems to help, I think the approach can use more work.

Status

With these two new approaches, tracking is improved but still quite erratic. Tracking of the headset itself is quite good now and for me rarely loses tracking lock. The controllers are better, but have a tendency to “fly off my hands” unexpectedly, especially after fast motions.

I have ideas for more tracking heuristics to implement, and I expect a continuous cycle of refinement on the existing strategies and new ones for some time to come.

For now, here’s a video of me playing Beat Saber using tonight’s code. The video shows the debug stream that OpenHMD can generate via Pipewire, showing the camera feed plus overlays of device predictions, LED device assignments and tracked device positions. Red is the headset, Green is the right controller, Blue is the left controller.

Initial tracking is completely wrong – I see some things to fix there. When the controllers go offline due to inactivity, the code keeps trying to match LEDs to them for example, and then there are some things wrong with how it’s relabelling LEDs when they get incorrect assignments.

After that, there are periods of good tracking with random tracking losses on the controllers – those show the problem cases to concentrate on.

by thaytan at January 31, 2021 04:37 PM

January 26, 2021

Jan SchmidtHitting a milestone – Beat Saber!

(Jan Schmidt)

I hit an important OpenHMD milestone tonight – I completed a Beat Saber level using my Oculus Rift CV1!

I’ve been continuing to work on integrating Kalman filtering into OpenHMD, and on improving the computer vision that matches and tracks device LEDs. While I suspect noone will be completing Expert levels just yet, it’s working well enough that I was able to play through a complete level of Beat Saber. For a long time this has been my mental benchmark for tracking performance, and I’m really happy 🙂

Check it out:

I should admit at this point that completing this level took me multiple attempts. The tracking still has quite a tendency to lose track of controllers, or to get them confused and swap hands suddenly.

I have a list of more things to work on. See you at the next update!

by thaytan at January 26, 2021 03:33 PM

January 22, 2021

Robert McQueenLaunching Endless OS Foundation

(Robert McQueen) Passion Led Us Here

How our for-profit company became a nonprofit, to better tackle the digital divide.

Originally posted on the Endless OS Foundation blog.

An 8-year journey to a nonprofit

On the 1st of April 2020, our for-profit Endless Mobile officially became a nonprofit as the Endless OS Foundation. Our launch as a nonprofit just as the global pandemic took hold was, predictably, hardly noticed, but for us the timing was incredible: as the world collectively asked “What can we do to help others in need?”, we framed our mission statement and launched our .org with the same very important question in mind. Endless always had a social impact mission at its heart, and the challenges related to students, families, and communities falling further into the digital divide during COVID-19 brought new urgency and purpose to our team’s decision to officially step in the social welfare space.

On April 1st 2020, our for-profit Endless Mobile officially became a nonprofit as the Endless OS Foundation, focused on the #DigitalDivide.

Our updated status was a long time coming: we began our transformation to a nonprofit organization in late 2019 with the realization that the true charter and passions of our team would be greatly accelerated without the constraints of for-profit goals, investors and sales strategies standing in the way of our mission of digital access and equity for all. 

But for 8 years we made a go of it commercially, headquartered in Silicon Valley and framing ourselves as a tech startup with access to the venture capital and partnerships on our doorstep. We believed that a successful commercial channel would be the most efficient way to scale the impact of bringing computer devices and access to communities in need. We still believe this – we’ve just learned through our experience that we don’t have the funding to enter the computer and OS marketplace head-on. With the social impact goal first, and the hope of any revenue a secondary goal, we have had many successes in those 8 years bridging the digital divide throughout the world, from Brazil, to Kenya, and the USA. We’ve learned a huge amount which will go on to inform our strategy as a nonprofit.

Endless always had a social impact mission at its heart. COVID-19 brought new urgency and purpose to our team’s decision to officially step in the social welfare space.

Our unique perspective

One thing we learned as a for-profit is that the OS and technology we’ve built has some unique properties which are hugely impactful as a working solution to digital equity barriers. And our experience deploying in the field around the world for 8 years has left us uniquely informed via many iterations and incremental improvements.

Endless OS designer in discussion with prospective user

With this knowledge in-hand, we’ve been refining our strategy throughout 2020 and now starting to focus on what it really means to become an effective nonprofit and make that impact. In many ways it is liberating to abandon the goals and constraints of being a for-profit entity, and in other ways it’s been a challenging journey for me and the team to adjust our way of thinking and let these for-profit notions and models go. Previously we exclusively built and sold a product that defined our success; and any impact we achieved was a secondary consequence of that success and seen through that lens. Now our success is defined purely in terms of social impact, and through our actions, those positive impacts can be made with or without our “product”. That means that we may develop and introduce technology to solve a problem, but it is equally as valid to find another organization’s existing offering and design a way to increase that positive impact and scale.

We develop technology to solve access equity issues, but it’s equally as valid to find another organization’s offering and partner in a way that increases their positive impact.

The analogy to Free and Open Source Software is very strong – while Endless has always used and contributed to a wide variety of FOSS projects, we’ve also had a tension where we’ve been trying to hold some pieces back and capture value – such as our own application or content ecosystem, our own hardware platform – necessarily making us competitors to other organisations even though they were hoping to achieve the same things as us. As a nonprofit we can let these ideas go and just pick the best partners and technologies to help the people we’re trying to reach.

School kids writing on paper

Digital equity … 4 barriers we need to overcome

In future, our decisions around which projects to build or engage with will revolve around 4 barriers to digital equity, and how our Endless OS, Endless projects, or our partners’ offerings can help to solve them. We define these 4 equity barriers as: barriers to devices, barriers to connectivity, barriers to literacy in terms of your ability to use the technology, and barriers to engagement in terms of whether using the system is rewarding and worthwhile.

We define the 4 digital equity barriers we exist to impact as:
1. barriers to devices
2. barriers to connectivity
3. barriers to literacy
4. barriers to engagement

It doesn’t matter who makes the solutions that break these barriers; what matters is how we assist in enabling people to use technology to gain access to the education and opportunities these barriers block. Our goal therefore is to simply ensure that solutions exist – building them ourselves and with partners such as the FOSS community and other nonprofits – proving them with real-world deployments, and sharing our results as widely as possible to allow for better adoption globally.

If we define our goal purely in terms of whether people are using Endless OS, we are effectively restricting the reach and scale of our solutions to the audience we can reach directly with Endless OS downloads, installs and propagation. Conversely, partnerships that scale impact are a win-win-win for us, our partners, and the communities we all serve. 

Engineering impact

Our Endless engineering roots and capabilities feed our unique ability to build and deploy all of our solutions, and the practical experience of deploying them gives us evidence and credibility as we advocate for their use. Either activity would be weaker without the other.

Our engineering roots and capabilities feed our unique ability to build and deploy digital divide solutions.

Our partners in various engineering communities will have already seen our change in approach. Particularly, with GNOME we are working hard to invest in upstream and reconcile the long-standing differences between our experience and GNOME. If successful, many more people can benefit from our work than just users of Endless OS. We’re working with Learning Equality on Kolibri to build a better app experience for Linux desktop users and bring content publishers into our ecosystem for the first time, and we’ve also taken our very own Hack, the immersive and fun destination for kids learning to code, released it for non-Endless systems on Flathub, and made it fully open-source.

Planning tasks with sticky notes on a whiteboard

What’s next for our OS?

What then is in store for the future of Endless OS, the place where we have invested so much time and planning through years of iterations? For the immediate future, we need the capacity to deploy everything we’ve built – all at once, to our partners. We built an OS that we feel is very unique and valuable, containing a number of world-firsts: first production OS shipped with OSTree, first Flatpak-only desktop, built-in support for updating OS and apps from USBs, while still providing a great deal of reliability and convenience for deployments in offline and educational-safe environments with great apps and content loaded on every system.

However, we need to find a way to deliver this Linux-based experience in a more efficient way, and we’d love to talk if you have ideas about how we can do this, perhaps as partners. Can the idea of “Endless OS” evolve to become a spec that is provided by different platforms in the future, maybe remixes of Debian, Fedora, openSUSE or Ubuntu? 

Build, Validate, Advocate

Beyond the OS, the Endless OS Foundation has identified multiple programs to help underserved communities, and in each case we are adopting our “build, validate, advocate” strategy. This approach underpins all of our projects: can we build the technology (or assist in the making), will a community in-need validate it by adoption, and can we inspire others by telling the story and advocating for its wider use?

We are adopting a “build, validate, advocate” strategy.
1. build the technology (or assist in the making)
2. validate by community adoption
3. advocate for its wider use

As examples, we have just launched the Endless Key (link) as an offline solution for students during the COVID-19 at-home distance learning challenges. This project is also establishing a first-ever partnership of well-known online educational brands to reach an underserved offline audience with valuable learning resources. We are developing a pay-as-you-go platform and new partnerships that will allow families to own laptops via micro-payments that are built directly into the operating system, even if they cannot qualify for standard retail financing. And during the pandemic, we’ve partnered with Teach For America to focus on very practical digital equity needs in the USA’s urban and rural communities.

One part of the world-wide digital divide solution

We are one solution provider for the complex matrix of issues known collectively as the #DigitalDivide, and these issues will not disappear after the pandemic. Digital equity was an issue long before COVID-19, and we are not so naive to think it can be solved by any single institution, or by the time the pandemic recedes. It will take time and a coalition of partnerships to win. We are in for the long-haul and we are always looking for partners, especially now as we are finding our feet in the nonprofit world. We’d love to hear from you, so please feel free to reach out to me – I’m ramcq on IRC, RocketChat, Twitter, LinkedIn or rob@endlessos.org.

by ramcq at January 22, 2021 05:00 PM

January 17, 2021

Jean-François Fortin TamThe origins of the Flow Game 🎥

Let’s kickstart the new year with a short & simple blog post, as a way to get me back on the blogging treadmill, and as a way to ensure my blog still works fine (I have just finished a very heavy-handed migration and database encoding surgery for my blog, which took months to solve… that’ll be a story for another blog post, if anyone is interested? 🤔 and yes, I’m totally using emojis and exotic languages in this post just to see if it still breaks Planet GNOME. わたしは にほんごがすこししかはなせません!)…


Sometime in the flurry of events that was 2020, my friend Hélène asked me a favor: to edit and publish some interview footage she had previously recorded—and that had been sitting on her hard drive since. The goal was to salvage this material and do a simple edit—not aim for perfection—and publish it so that the public can benefit from the shared knowledge and life experiences.

And so I did. The resulting video (17 mins) is here for your enjoyment:

What is this about?

The video is an informal speech given by Monica Nissén and Toke Paludan Møller, where they share their experience as business founders in the nineties, and what led them to create the Flow Game, a serious game that serves as a tool for creating “an interactive reflection, dialogue and action space for groups, teams and individuals.”

Indeed, while they were working on their entrepreneurial project (with at least one other collaborator, I believe), they felt the need to create a system to facilitate their brainstorming sessions and guide the evolution of their projects.

“What a strange edit! Were you drunk?”

A friend of mine who worked in Hollywood keeps telling me, “Editing is the pits!”, probably because you have to work with what you’re given, and have to cut and rearrange thousands of little pieces to build a coherent story.

In this case, the video was recorded with only one camera angle, so I did not have a lot of material to work with. I added some footage from other sessions to hide various handheld camera angle relocation moves; this is why you sometimes see visuals of other people (such as Toke) “silently talking” while Monica’s voice narration continues. There are scenes in Toke’s speech where I had only audio (it seems the video recording started late) so I had no choice but to fill the gaps with footage of other people talking (in that case I did find it amusing to have synched lips on some occasions), and I tried to not reuse the same footage too much to provide some variety.


Note: this video is a pro-bono production for educational and historical purposes, and is not a business endorsement of the Flow Game by idéemarque/atypica. I’m not selling anything, there is no Danish Conspiracy (unlike the GNOME Swedish Conspiracy), etc. I just hope someone finds this content insightful. Feel free to share it around (or retweet it) or to leave a comment if you liked it.

by Jeff at January 17, 2021 12:55 PM

January 13, 2021

GStreamerGStreamer 1.18.3 stable bug fix release

(GStreamer)

The GStreamer team is pleased to announce another bug fix release in the stable 1.18 release series of your favourite cross-platform multimedia framework!

This release only contains bugfixes and it should be safe to update from 1.18.x.

Highlighted bugfixes:

  • Fix ogg playback regression for ogg files that also have ID3 or APE tags
  • compositor: fix artefacts and invalid memory access when blending subsampled formats
  • exported mini object ref/unref/copy functions for use in bindings such as gstreamer-sharp
  • Add support for Apple silicon (M1) to cerbero package builder
  • Ship RIST plugin in binary packages
  • various stability, performance and reliability improvements
  • memory leak fixes
  • build fixes

See the GStreamer 1.18.3 release notes for more details.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

Download tarballs directly here: gstreamer, gst-plugins-base, gst-plugins-good, gst-plugins-ugly, gst-plugins-bad, gst-libav, gst-rtsp-server, gst-python, gst-editing-services, gst-devtools, gstreamer-vaapi, gstreamer-sharp, gst-omx, or gstreamer-docs.

January 13, 2021 11:00 PM

January 07, 2021

Jan SchmidtRift CV1 – Adventures in Kalman filtering Part 2

(Jan Schmidt)

In the last post I had started implementing an Unscented Kalman Filter for position and orientation tracking in OpenHMD. Over the Christmas break, I continued that work.

A Quick Recap

When reading below, keep in mind that the goal of the filtering code I’m writing is to combine 2 sources of information for tracking the headset and controllers.

The first piece of information is acceleration and rotation data from the IMU on each device, and the second is observations of the device position and orientation from 1 or more camera sensors.

The IMU motion data drifts quickly (at least for position tracking) and can’t tell which way the device is facing (yaw, but can detect gravity and get pitch/roll).

The camera observations can tell exactly where each device is, but arrive at a much lower rate (52Hz vs 500/1000Hz) and can take a long time to process (hundreds of milliseconds) to analyse to acquire or re-acquire a lock on the tracked device(s).

The goal is to acquire tracking lock, then use the motion data to predict the motion closely enough that we always hit the ‘fast path’ of vision analysis. The key here is closely enough – the more closely the filter can track and predict the motion of devices between camera frames, the better.

Integration in OpenHMD

When I wrote the last post, I had the filter running as a standalone application, processing motion trace data collected by instrumenting a running OpenHMD app and moving my headset and controllers around. That’s a really good way to work, because it lets me run modifications on the same data set and see what changed.

However, the motion traces were captured using the current fusion/prediction code, which frequently loses tracking lock when the devices move – leading to big gaps in the camera observations and more interpolation for the filter.

By integrating the Kalman filter into OpenHMD, the predictions are improved leading to generally much better results. Here’s one trace of me moving the headset around reasonably vigourously with no tracking loss at all.

Headset motion capture trace

If it worked this well all the time, I’d be ecstatic! The predicted position matched the observed position closely enough for every frame for the computer vision to match poses and track perfectly. Unfortunately, this doesn’t happen every time yet, and definitely not with the controllers – although I think the latter largely comes down to the current computer vision having more troubler matching controller poses. They have fewer LEDs to match against compared to the headset, and the LEDs are generally more side-on to a front-facing camera.

Taking a closer look at a portion of that trace, the drift between camera frames when the position is interpolated using the IMU readings is clear.

Headset motion capture – zoomed in view

This is really good. Most of the time, the drift between frames is within 1-2mm. The computer vision can only match the pose of the devices to within a pixel or two – so the observed jitter can also come from the pose extraction, not the filtering.

The worst tracking is again on the Z axis – distance from the camera in this case. Again, that makes sense – with a single camera matching LED blobs, distance is the most uncertain part of the extracted pose.

Losing Track

The trace above is good – the computer vision spots the headset and then the filtering + computer vision track it at all times. That isn’t always the case – the prediction goes wrong, or the computer vision fails to match (it’s definitely still far from perfect). When that happens, it needs to do a full pose search to reacquire the device, and there’s a big gap until the next pose report is available.

That looks more like this

Headset motion capture trace with tracking errors

This trace has 2 kinds of errors – gaps in the observed position timeline during full pose searches and erroneous position reports where the computer vision matched things incorrectly.

Fixing the errors in position reports will require improving the computer vision algorithm and would fix most of the plot above. Outlier rejection is one approach to investigate on that front.

Latency Compensation

There is inherent delay involved in processing of the camera observations. Every 19.2ms, the headset emits a radio signal that triggers each camera to capture a frame. At the same time, the headset and controller IR LEDS light up brightly to create the light constellation being tracked. After the frame is captured, it is delivered over USB over the next 18ms or so and then submitted for vision analysis. In the fast case where we’re already tracking the device the computer vision is complete in a millisecond or so. In the slow case, it’s much longer.

Overall, that means that there’s at least a 20ms offset between when the devices are observed and when the position information is available for use. In the plot above, this delay is ignored and position reports are fed into the filter when they are available. In the worst case, that means the filter is being told where the headset was hundreds of milliseconds earlier.

To compensate for that delay, I implemented a mechanism in the filter where it keeps extra position and orientation entries in the state that can be used to retroactively apply the position observations.

The way that works is to make a prediction of the position and orientation of the device at the moment the camera frame is captured and copy that prediction into the extra state variable. After that, it continues integrating IMU data as it becomes available while keeping the auxilliary state constant.

When a the camera frame analysis is complete, that delayed measurement is matched against the stored position and orientation prediction in the state and the error used to correct the overall filter. The cool thing is that in the intervening time, the filter covariance matrix has been building up the right correction terms to adjust the current position and orientation.

Here’s a good example of the difference:

Before: Position filtering with no latency compensation After: Latency-compensated position reports

Notice how most of the disconnected segments have now slotted back into position in the timeline. The ones that haven’t can either be attributed to incorrect pose extraction in the compute vision, or to not having enough auxilliary state slots for all the concurrent frames.

At any given moment, there can be a camera frame being analysed, one arriving over USB, and one awaiting “long term” analysis. The filter needs to track an auxilliary state variable for each frame that we expect to get pose information from later, so I implemented a slot allocation system and multiple slots.

The downside is that each slot adds 6 variables (3 position and 3 orientation) to the covariance matrix on top of the 18 base variables. Because the covariance matrix is square, the size grows quadratically with new variables. 5 new slots means 30 new variables – leading to a 48 x 48 covariance matrix instead of 18 x 18. That is a 7-fold increase in the size of the matrix (48 x 48 = 2304 vs 18 x 18 = 324) and unfortunately about a 10x slow-down in the filter run-time.

At that point, even after some optimisation and vectorisation on the matrix operations, the filter can only run about 3x real-time, which is too slow. Using fewer slots is quicker, but allows for fewer outstanding frames. With 3 slots, the slow-down is only about 2x.

There are some other possible approaches to this problem:

  • Running the filtering delayed, only integrating IMU reports once the camera report is available. This has the disadvantage of not reporting the most up-to-date estimate of the user pose, which isn’t great for an interactive VR system.
  • Keeping around IMU reports and rewinding / replaying the filter for late camera observations. This limits the overall increase in filter CPU usage to double (since we at most replay every observation twice), but potentially with large bursts when hundreds of IMU readings need replaying.
  • It might be possible to only keep 2 “full” delayed measurement slots with both position and orientation, and to keep some position-only slots for others. The orientation of the headset tends to drift much more slowly than position does, so when there’s a big gap in the tracking it would be more important to be able to correct the position estimate. Orientation is likely to still be close to correct.
  • Further optimisation in the filter implementation. I was hoping to keep everything dependency-free, so the filter implementation uses my own naive 2D matrix code, which only implements the features needed for the filter. A more sophisticated matrix library might perform better – but it’s hard to say without doing some testing on that front.

Controllers

So far in this post, I’ve only talked about the headset tracking and not mentioned controllers. The controllers are considerably harder to track right now, but most of the blame for that is in the computer vision part. Each controller has fewer LEDs than the headset, fewer are visible at any given moment, and they often aren’t pointing at the camera front-on.

Oculus Camera view of headset and left controller.

This screenshot is a prime example. The controller is the cluster of lights at the top of the image, and the headset is lower left. The computer vision has gotten confused and thinks the controller is the ring of random blue crosses near the headset. It corrected itself a moment later, but those false readings make life very hard for the filtering.

Position tracking of left controller with lots of tracking loss.

Here’s a typical example of the controller tracking right now. There are some very promising portions of good tracking, but they are interspersed with bursts of tracking losses, and wild drifting from the computer vision giving wrong poses – leading to the filter predicting incorrect acceleration and hence cascaded tracking losses. Particularly (again) on the Z axis.

Timing Improvements

One of the problems I was looking at in my last post is variability in the arrival timing of the various USB streams (Headset reports, Controller reports, camera frames). I improved things in OpenHMD on that front, to use timestamps from the devices everywhere (removing USB timing jitter from the inter-sample time).

There are still potential problems in when IMU reports from controllers get updated in the filters vs the camera frames. That can be on the order of 2-4ms jitter. Time will tell how big a problem that will be – after the other bigger tracking problems are resolved.

Sponsorships

All the work that I’m doing implementing this positional tracking is a combination of my free time, hours contributed by my employer Centricular and contributions from people via Github Sponsorships. If you’d like to help me spend more hours on this and fewer on other paying work, I appreciate any contributions immensely!

Next Steps

The next things on my todo list are:

  • Integrate the delayed-observation processing into OpenHMD (at the moment it is only in my standalone simulator).
  • Improve the filter code structure – this is my first kalman filter and there are some implementation decisions I’d like to revisit.
  • Publish the UKF branch for other people to try.
  • Circle back to the computer vision and look at ways to improve the pose extraction and better reject outlying / erroneous poses, especially for the controllers.
  • Think more about how to best handle / schedule analysis of frames from multiple cameras. At the moment each camera operates as a separate entity, capturing frames and analysing them in threads without considering what is happening in other cameras. That means any camera that can’t see a particular device starts doing full pose searches – which might be unnecessary if another camera still has a good view of the device. Coordinating those analyses across cameras could yield better CPU consumption, and let the filter retain fewer delayed observation slots.

by thaytan at January 07, 2021 05:36 PM

December 30, 2020

Jean-François Fortin TamBlogging about Python desktop apps improvements on Planet Python

Hi, fellow pythonistas! Before I start publishing future Python-related posts to this aggregator, I would like to shortly introduce myself and the reason for this blog’s presence on the planet.

I am a business management consultant, but I am also, in my spare time, an independent free & open-source software developer, designer and maintainer who happens to use Python as his sole programming language. I created Specto (all Python) many years ago, I co-maintained the Pitivi video editor (also written in Python) for many years, and nowadays I co-maintain GTG which is, you guessed it, another “pure Python” free & open-source desktop application. It looks like this:

Here I blog mainly about new releases and improvements in my Python software apps (which means GTG lately, but I also have a couple of pythonic utility apps I’ve been meaning to publish sometime soon), and sometimes write about performance optimization in software applications in general, or how a particular bug was solved. As such, my blog posts tend to be “applied Python” type of content rather than theoretical tutorial-style blog posts.

I believe that successful Python application improvements & releases serve as tangible proof of Python’s power as a desktop GUI application development technology that can attract healthy contributor communities. It is not every day you see a project come back to life like what we did with GTG, and I hope this well-designed, practical, cornerstone desktop application can serve as a success story for Python on the desktop.

In recent days, I have also started working for the Montréal Polytechnique university’s neuroimaging laboratory/research group, where I help them structure their documentation and scientific communication efforts to the public. Their neuroimaging software projects are typically written in Python, so if there is some interest in that (?), and should I find topics that seem relevant to talk about on my blog, I may write about that too (but I’m no neuroscientist!)

I look forward to contributing to Planet Python, and if you’re curious about the broader topics I blog about, I also have a posts notification mailing list, and on Twitter you can also find me, GTG, Pitivi, the PolyMTL Neuro lab account, and the accounts of the lab’s biggest pythonic apps as well.

by Jeff at December 30, 2020 12:25 PM

December 21, 2020

Jan SchmidtRift CV1 – Adventures in Kalman filtering

(Jan Schmidt)

In my last post I wrote about changes in my OpenHMD positional tracking branch to split analysis of the tracking frames from the camera sensors across multiple threads. In the 2 months since then, the only real change in the repository was to add some filtering to the pose search that rejects bad poses by checking if they align with the gravity vector observed by the IMU. That is in itself a nice improvement, but there is other work I’ve been doing that isn’t published yet.

The remaining big challenge (I think) to a usable positional tracking solution is fusing together the motion information that comes from the inertial tracking sensors (IMU) in the devices (headset, controllers) with the observations that come from the camera sensors (video frames). There are some high level goals for that fusion, and lots of fiddly details about why it’s hard.

At the high level, the IMUs provide partial information about the motion of each device at a high rate, while the cameras provide observations about the actual position in the room – but at a lower rate, and with sometimes large delays.

In the Oculus CV1, the IMU provides Accelerometer and Gyroscope readings at 1000Hz (500Hz for controllers), and from those it’s possible to compute the orientation of the device relative to the Earth (but not the compass direction it’s facing), and also to integrate acceleration readings to get velocity and position – but the position tracking from an IMU is only useful in the short term (a few seconds) as it drifts rapidly due to that double integration.

The accelerometers measure (surprise) the acceleration of the device, but are always also sensing the Earth’s gravity field. If a device is at rest, it will ideally report 9.81 m/s2, give or take noise and bias errors. When the device is in motion, the acceleration measured is the sum of the gravity field, bias errors and actual linear acceleration. To interpolate the position with any accuracy at all, you need to separate those 3 components with tight tolerance.

That’s about the point where the position observations from the cameras come into play. You can use those snapshots of the device position to determine the real direction that the devices are facing, and to correct for any errors in the tracked position and device orientation from the IMU integration – by teasing out the bias errors and gravity offset.

The current code uses some simple hacks to do the positional tracking – using the existing OpenHMD 3DOF complementary filter to compute the orientation, and some hacks to update the position when a camera finds the pose of a device.

The simple hacks work surprisingly well when devices don’t move too fast. The reason is (as previously discussed) that the video analysis takes a variable amount of time – if we can predict where a device is with a high accuracy and maintain “tracking lock”, then the video analysis is fast and runs in a few milliseconds. If tracking lock is lost, then a full search is needed to recover the tracking, and that can take hundreds of milliseconds to complete… by which time the device has likely moved a long way and requires another full pose search, which takes hundreds of milliseconds..

So, the goal of my current development is to write a single unified fusion filter that combines IMU and camera observations to better track and predict the motion of devices between camera frames. Better motion prediction means hitting the ‘fast analysis’ path more often, which leads to more frequent corrections of the unknowns in the IMU data, and (circularly) better motion predictions.

To do that, I am working on an Unscented Kalman Filter that tracks the position, velocity, acceleration, orientation and IMU accelerometer and gyroscope biases – with promising initial results.

Graph of position error (m) between predicted position and position from camera observations Graph of orientation error (degrees) between predicted orientation and camera observed pose.

In the above graphs, the filter is predicting the position of the headset at each camera frame to within 1cm most of the time and the pose to within a few degrees, but with some significant spikes that still need fixing. The explanation for the spikes lies in the data sets that I’m testing against, and points to the next work that needs doing.

To develop the filter, I’ve modifed OpenHMD to record traces as I move devices around. It saves out a JSON file for each device with a log of each IMU reading and each camera frame. The idea is to have a baseline data set that can be used to test each change in the filter – but there is a catch. The current data was captured using the upstream positional tracking code – complete with tracking losses and long analysis delays.

The spikes in the filter graph correspond with when the OpenHMD traces have big delays between when a camera frame was captured and when the analysis completes.

Delay (ms) between camera frame and analysis results.

What this means is that when the filter makes bad predictions, it’s because it’s trying to predict the position of the device at the time the sensor result became available, instead of when the camera frame was captured – hundreds of milliseconds earlier.

So, my next step is to integrate the Kalman filter code into OpenHMD itself, and hopefully capture a new set of motion data with fewer tracking losses to prove the filter’s accuracy more clearly.

Second – I need to extend the filter to compensate for that delay between when a camera frame is captured and when the results are available for use, by using an augmented state matrix and lagged covariances. More on that next time.

To finish up, here’s a taste of another challenge hidden in the data – variability in the arrival time of IMU updates. The IMU updates at 1000Hz – ideally we’d receive those IMU updates 1 per millisecond, but transfer across the USB and variability in scheduling on the host computer make that much noisier. Sometimes further apart, sometimes bunched together – and in one part there’s a 1.2 second gap.

IMU reading timing variability (nanoseconds)

by thaytan at December 21, 2020 12:25 PM

December 06, 2020

GStreamerGStreamer 1.18.2 stable bug fix release

(GStreamer)

The GStreamer team is pleased to announce another bug fix release in the stable 1.18 release series of your favourite cross-platform multimedia framework!

This release only contains bugfixes and it should be safe to update from 1.18.x.

Highlighted bugfixes:

  • Fix MPEG-TS timestamping regression when playing DVB streams
  • compositor: fix artefacts in certain input scaling/conversion situations and make sure that the output format is actually supported, plus renegotiation fixes
  • Fix sftp:// URI playback in decodebin/playbin via giosrc
  • adaptivedemux/dashdemux/hlsdemux fixes
  • rtsp-server fixes
  • android media: fix crash when encoding AVC
  • fix races in various unit tests
  • lots of other bug fixes and memory leak fixes
  • various stability, performance and reliability improvements
  • g-i annotation fixes
  • build fixes

See the GStreamer 1.18.2 release notes for more details.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

Download tarballs directly here: gstreamer, gst-plugins-base, gst-plugins-good, gst-plugins-ugly, gst-plugins-bad, gst-libav, gst-rtsp-server, gst-python, gst-editing-services, gst-devtools, gstreamer-vaapi, gstreamer-sharp, gst-omx, or gstreamer-docs.

December 06, 2020 03:00 PM

November 29, 2020

Phil NormandCatching up on WebKit GStreamer WebAudio backends maintenance

(Phil Normand)

Over the past few months the WebKit development team has been working on modernizing support for the WebAudio specification. This post highlights some of the changes that were recently merged, focusing on the GStreamer ports.

My fellow WebKit colleague, Chris Dumez, has been very active lately, updating the WebAudio implementation …

by Philippe Normand at November 29, 2020 12:45 PM

November 26, 2020

Jean-François Fortin TamLes cônes oranges du libre: du ramollissement de l'enthousiasme face aux chantiers technologiques

Dans ce billet, l’ami “Antistress” (Thibaut) fait un état des lieux des grands changements technologiques du libre autour de la plateforme GNU+Linux depuis une quinzaine d’années, lorsque plusieurs d’entre nous ont commencé à utiliser cette plateforme plus intensivement. Il y laisse également transpirer un certain épuisement, ou “manque d’excitation”, qui est probablement ressenti par plusieurs d’entre nous.

Pour ma part j’ai commencé à bidouiller avec Linux en 2003, mais ce n’est qu’avec la première version d’Ubuntu, 4.10, que j’ai pu y coller à plein temps (note: je suis passé à Fedora depuis 2010). Et depuis, ça a été une histoire de patience. Beaucoup, beaucoup de patience.

Je vois souvent des parallèles entre le monde du logiciel libre et la ville de Montréal. Comme les chantiers dans les rues de Montréal qui semblent éternels (au point où nous avons un cône orange qui nous sert de mascotte informelle), on a l’impression de ne pas voir le bout de nos chantiers technologiques.

Les cônes oranges typiques du paysage montréalais, une photo du Devoir (pour mes besoins de parodie)

La tâche de modernisation technologique de notre système d’opération favori est colossale, a impliqué nombre d’individus et de compagnies (qui sont venues et parties à divers moments des “desktop wars” et “mobile wars”), et nombres de rebondissements et apprentissages en cours de route. On peut penser notamment:

  • à Xorg et ses multiples tentatives d’avoir du compositing qui tienne la route (incluant XGL, AIGLX, Compiz/Beryl, Metacity vs Mutter, GNOME Shell, KWin) et l’éventuel ras-le-bol-on-recommence-avec-une-architecture-qui-ne-date-pas-des-années-70 qu’est Wayland (visionnement obligatoire: cette merveilleuse présentation de Daniel Stone);
  • au chantier du multimédia avec GStreamer et PulseAudio (et maintenant/prochainement: PipeWire);
  • au long combat de modernisation de la technologie de Firefox pour survivre face à l’épidémie d’obésité du web et rattraper les performances de Chrome/Chromium;
  • à l’attente de pilotes graphiques libres, performants et stables pour les cartes graphiques AMD (depuis leur changement de stratégie circa 2010 où AMD s’est mise à publier de la documentation pour soutenir les pilotes libres, et quelques années plus tard aller plus loin et rationaliser leurs pilotes propriétaires avec les pilotes libres);
  • à l’interminable migration de l’écosystème d’applications GTK2 vers GTK3 (juste à temps pour GTK4, tiens!), idem pour Python 2 vs Python 3;
  • à l’interminable attente, de 2010 à 2019, pour que des gens compétents se penchent de manière concertée sur la question de l’optimisation de performances de GNOME Shell (depuis 3.36 et 3.38, c’est le pur bonheur);
  • etc.

Dans le monde du libre, nous sommes tous un peu sado-masos. Masochistes pour ce qui est de la patience dont nous devons faire preuve, et sadiques dans notre relation amour-haine envers nos logiciels favoris (je ne me suis pas gêné, par le passé, pour être assez critique sur mon blog concernant les performances de Firefox; critiques qui sont, une décennie plus tard, enfin résolues), de la même façon que les Montréalais(e)s ont une relation amour-haine avec les cônes oranges (même sans être fans de VLC). Et c’est sans compter la déprimante hégémonie de Google et Apple dans le monde du mobile au sein de la population, ce qui confine notre impact au monde du “desktop”.

Alors effectivement, après toutes ces années, on devient un peu moins excités que dans notre jeunesse où on se faisait fervents évangélistes du libre. Thibaut est assez clair lorsqu’il écrit ceci:

[…] je dirais qu’à part pour le premier item (la finalisation des chantiers en cours de Firefox sous GNU/Linux), je manque un peu d’excitation.

Pour expliquer ce phénomène, je crois que je dirais ceci: du côté “desktop Linux” il y a eu beaucoup de progrès un peu partout au fil des années, et une des raisons pour laquelle nous sommes peut-être moins excités par rapport aux nouveautés sur nos ordinateurs, c’est aussi parce que “globalement, ça marche maintenant” (vous souvenez-vous de l’époque pré-Xorg où on devait s’attendre à ce que l’interface graphique ne fonctionne pas au premier démarrage? et l’époque 2004-2012 où on devait se battre pour avoir du WiFi qui fonctionne? Heureusement je n’ai plus croisé de Broadcom dans les dernières années…).

Donc, il ne reste plus tant de choses criantes que ça à régler. Et ce qui reste à régler… remplace souvent des choses qui “fonctionnent”. De 2003 à 2013 environ, j’ai eu plusieurs situations où mon système de fichiers ext4 s’est spontanément corrompu “par lui-même”, sans explication (alors que ext4 était supposément stable et largement testé depuis des années). Depuis 2013-2014, ça ne m’est plus arrivé. Alors, pour reprendre le constat d’Antistress concernant l’adoption modeste de Btrfs, excusez-moi, mais je n’ai aucun intérêt ni enthousiasme à reformatter mon disque dur 2 teraoctets (détenant mon /home) de ext4 vers Btrfs. Ni l’empressement de passer à Fedora Silverblue au lieu de Fedora Workstation. Je vais laisser les nouveaux jeunes s’amuser avec ça!

by Jeff at November 26, 2020 05:25 PM

November 05, 2020

Jean-François Fortin TamActiver la réduction de bruit en temps réel et l'annulation acoustique d'écho pour un microphone sous Linux avec PulseAudio (et faire croire aux gens que vous roulez en Porsche)

On se souviendra que j’avais témoigné de mon admiration pour le acoustic echo cancelling dans Empathy/Telepathy via PulseAudio il y a plusieurs années.

Or, Empathy et Telepathy sont morts et enterrés, et WebRTC a largement pris leur place (ça, on le voyait venir) avec des applications comme Jitsi Meet, BigBlueButton, et plein de sites web SaaS propriétaires.

Ce que j’avais oublié de dire en 2012, c’est que l’AEC et débruiteur peuvent également être activés de façon permanente pour toutes les applications utilisant le micro, sans que ce soit forcément Empathy/Telepathy. Cette fonctionnalité est toujours disponible dans PulseAudio en 2020*.

Cela peut être utile si vous ne voulez pas traiter vos enregistrements dans Audacity à chaque fois, ou pour les quelques cas d’utilisation de visioconférence où l’annulation acoustique d’écho n’est pas prise en charge par votre application (n’oublions pas que echo concealment n’est pas la même chose que echo cancellation). Ce n’est pas tellement mon cas (ayant de l’équipement de studio de production avec des microphones bien amplifiés et sans bruit de fond), mais ça peut être pratique pour les libristes moins bien équipé(e)s qui doivent tout de même participer aux appels conférence de plus en plus fréquents mais ne veulent pas infliger aux autres le bruit de leur microphone analogique pourri collé sur le ventilateur d’un ordinateur en surchauffe. Enfin, je suppose que toutes sortes de gens veulent se débarrasser de leur echo en 2020:

Pour activer la fonctionnalité “par utilisateur” (survit aux réinstallations du système si vous préservez votre partition home), éditez le fichier ~/.config/pulse/default.pa ; Si, au contraire, vous voulez l’activer system-wide pour tous les utilisateurs, éditer /etc/pulse/default.pa, et y insérer le contenu suivant:

load-module module-echo-cancel source_name=noechosource sink_name=noechosink
set-default-source noechosource
set-default-sink noechosink

On aurait pu écrire “Fiji” au lieu de “noechosource” et “lavabo” au lieu de “noechosink” je suppose. Il y a aussi une pléthore d’options possibles pour le module echo cancel, mais on va garder ça simple. Comme une Toyota.

Dans le cas de ~/.config/pulse/default.pa, n’oubliez pas que votre fichier doit hériter du fichier système, et donc commencer avec ces lignes au début du fichier:

.include /etc/pulse/default.pa

Puis vous pouvez redémarrer l’instance PulseAudio de votre session d’usager en le stoppant (puisqu’il redémarrera tout seul, du moins si vous êtes sous GNOME): pulseaudio --kill

Des nouveaux périphériques virtuels d’enregistrement seront alors à votre disposition dans pavucontrol (ou le mixeur audio de GNOME Control Center). Mais avant de vous lancer dans toute cette aventure, peut-être voudrez-vous renommer vos périphériques matériels aux yeux de PulseAudio… ainsi tout le monde dans la visio croira que vous roulez en Porsche.


*: du moins, jusqu’à ce qu’on en ait marre de PulseAudio et qu’on le remplace par un autre système qui fait le café dans les années à venir…

by Jeff at November 05, 2020 12:00 PM

October 31, 2020

Jean-François Fortin TamSpooky GTG features to try out for Halloween 2020

Are you an irresistible creature with an insatiable love for the dead… bugs? Well, grab your bug hunter crossbow, because we need you to test some big technological changes in GTG so that we can confidently release version 0.5 sometime soon (way before the year end, ideally).

Synchronizing your tasks across devices using CalDAV

A new synch backend/plugin was developed by Mildred during the summer. I think that’s super cool and it will probably be useful to many of you! I will now let this thematic tweet speak for itself (so that you can retweet it too today):

It would be great to have many adventuring early-adopters try out this feature. See the details (and report issues) in this ticket. This has not yet been merged into the main branch of the Git version of GTG, as I would like to have more people testing it before it is included by default in what will become a stable GTG release. Please indicate your success or failures by voting on this comment.

Changes related to the local file format data storage

In an attempt to rule out the possibility of our XML file format being the cause of GTG’s performance problems that occur when you have multiple hundreds of tasks (I have a thousand), Diego ported GTG’s XML storage backend to LXML, which is reportedly 40 times faster.

  • Unfortunately, that was not the silver bullet; the performance issues remained, but at least now we can rule out XML (and the task editor, see below) as suspects, and we can move on to a different part of the problem.
  • We suspect our performance woes have something to do with sorting in liblarch, or perhaps deeper: maybe it’s all GtkTreeView/GtkTreeModel’s fault? Who knows.
  • Do you take pride in optimizing the sh!t out of GTK applications? Do you have experience in duck-taping a jetpack onto a turtle? Join the party! Your help is more than welcome in investigating our performance problems and devising potential fixes. We will owe you beer and candy.

In addition to the LXML port, another area where Diego brought down the might of his code refactoring skills is the new “task editor” backend and view renderer, which includes a bunch of new features and refinements such as support for subheadings (with the “#” markdown syntax), the use of GTK checkboxes to represent subtasks, tag highlight colors that match the tag’s color, and a bunch of bugfixes. It was also envisioned as a way to address one of the performance issues I theorized about. This is what it looks like:

New taskview screenshot

And as if that wasn’t enough, Diego set out to redesign the XML file format from a new specification devised with the help of Brent Saner (proposal episodes 1, 2 and 3). The new file format is not yet merged into master, and we consider it quite experimental. If you’re feeling courageous and are looking for a bug hunting challenge, back up your data (really) before trying it out with the “file_format_2” branch (which you can see in the current list of branches).

As you can imagine, Diego’s changes are all major, invasive technological changes (particularly the proposed file format change), and they would benefit from extensive testing by everybody before 0.5 happens. I’ve done some pretty exhaustive testing before he merged his first two branches of course, but I’m only human, and it is possible that issues might remain. To make testing easier, a snapshot of the current git version has been packaged as a Flatpak package, available here. As for the third (file format) branch, we’d like to have more people testing it with copies of “real data” before we feel confident about merging it, so you’ll have to build the git version yourself instead of using the flatpak. So please grab GTG’s git version ASAP, with a copy of your data (see the instructions in the README, including the “Where is my user data and config stored?” section), and torture-test it to make sure everything is working properly, and report issues you may find (if any).

Other ongoing improvements

There have been countless bug fixes and improvements going on in recent months, here are a few that I can remember from the top of my head:

  • The main window’s previously active tab/mode is now restored when you launch the application.
  • Task windows no longer all reopen on launch when you shutdown the PC without closing GTG first.
  • There is a setting in GTG’s preference now to specify whether you are using a dark theme or not, so that GTG can adapt its colors accordingly. From what I am told, it seems GNOME still doesn’t have a mechanism for applications to know for certain if the user’s system is using a dark theme, or to connect to a signal when this changes…
  • Izidor Matusov made a surprise appearance to nuke the pyxdg dependency from orbit
  • Diego landed the new icon, along with a nighty version of it. I mourn the pixel-perfectness of the oldschool bitmap icon, but there are worse things going on in 2020 😉
  • The about dialog now properly shows the application icon
  • Parent tasks automatically close their window when you click to open a subtask, which makes that consistent with closing the subtask’s window when you click “Open parent”
  • Neui started by working on the German translation, then quickly got addicted and started fixing a bunch of papercuts in the code! Among other things, Neui removed dbus-python in favor of gdbus, made the plugins’ “about” text translatable, documented the contribution process for translators, and various other things.
  • New Swedish and Hungarian translation updates have been contributed!
  • Francisco Lavin fixed the Hamster plugin! Productivity metric rodents rejoice! Please test this thing.
  • Recurring tasks. Holy shit.

More great things to come.


P.s.: I realized today that I totally forgot to mention this in the context of GTG: I have a personal news announcement mailing list where I send out notifications about my new blog posts (with a short hand-crafted summary), as courtesy to my readers. Over the past 12 months, I have sent 5 such emails (so, not exactly a deluge), and they have so far all had something to do with GTG. If you’re afraid of missing some news about GTG, or don’t have time to be following every ticket and every tweet, feel free to subscribe to my mailing list. Details and subscription form can be found here (that page sucks, I know, I need to write something simpler).

by Jeff at October 31, 2020 09:29 PM

Jean-François Fortin TamL'indicible horreur des fichiers SQLite (et les mandelbugs de corruption de profil Firefox)

En cet Halloween, je vous partage aujourd’hui une petite histoire d’horreur et quelques astuces, fruit de mes observations sur un mystérieux phénomène survenu en 2017. En fait, ce billet est resté dans mes brouillons depuis trois ans et trois semaines (essayez de battre ce record, bande de flemmards), et je me suis dit qu’il vallait mieux que j’en finisse avant la réélection de Trump, et publie ce billet enfin… et puis hein, ho, ça fait une éternité que j’ai bloggé en Français et sur Planet Libre, alors coucou Planet Libre, ça fait un bail. Regardez, j’ai ressuscité GTG cette année, c’est bien, ça fait le café. D’ailleurs, gardez l’oeil ouvert sur mon blog car il y a du nouveau qui s’en vient, même si c’est pas forcément en Français (et donc pas sur Planet Libre)…


L’indiscible horreur des fichiers SQLite

Je n’ai pas beaucoup d’amour pour les fichiers SQLite, format répandu de stockage de données pour les applications logicielles. Je trouve que:

  1. cela obscurcit les données (pas inspectables et éditables avec un éditeur de texte comme c’est le cas du XML, par exemple);
  2. ce format a tendance à créer des problèmes de performance où on doit régulièrement passer l’aspirateur (fait amusant: c’est quatre ans plus tard que Jim, mainteneur de Geary à l’époque, en vint à découvrir le problème de performance que peut représenter SQLite… en étudiant mon billet de blog!);
  3. ça a la fâcheuse habitude, dans le cas de Firefox (et Tracker), à se corrompre beaucoup trop facilement (et rendu là, la seule solution est d’exploser le fichier au complet, à cause du point #1).

Le bizarre incident du panda rouge pendant dans la nuit

Alors, pour ce qui est de la petite anecdote “récente”…

Au début de l’automne 2017, j’ai constaté que certains sites web dynamiques (comme LinkedIn, appear.in, rocketchat) refusaient de plus en plus souvent de charger dans Firefox 55, puis, avec Firefox 56, ont déclaré la grève générale illimitée et cessé entièrement de fonctionner. Perplexe, à ce moment là j’ai tenté de désactiver toutes mes extensions, vider le cache web, vider le cache applicatif à partir de la page de préférences de Firefox, emprisonner le fou du village, rien n’y faisait.

Or, selon mes expériences précédentes, je me suis dit “Si le problème se produit sur un ordinateur particulier et non un autre… la cause est probablement un fichier SQLite corrompu”. Et comme de fait, en utilisant la console de debug web de Firefox, je découvris que ce dernier rhâlait avec des erreurs “NS_ERROR_FILE_CORRUPTED” en tentant de charger LinkedIn, ce qui m’a mené vers ce sujet de discussion dans StackOverflow, entre autres.

Il se trouve que, cette fois-là, mon fichier ~/.mozilla/firefox/profil_normal.default/webappsstore.sqlite avait muté hors de contrôle. Il pesait 86 Mo, ce qui est très louche déjà, et même si je l’ouvrais avec sqlite3 et lançais “VACUUM;”, la taille ne descendait qu’à 84 Mo. Il était clair que c’était devenu…

Alors, il ne restait qu’une solution…

J’ai donc supprimé complètement le fichier, qui s’est recréé par lui même ensuite. Sa nouvelle taille: 622 ko. Et du coup, tous mes sites applicatifs se sont mis à fonctionner. CQFD.

J’en ai alors profité pour inspecter le profil Firefox dans mon autre ordinateur, simplement en triant par taille—j’ai toujours une certaine méfiance envers les “gros fichiers” dans ce dossier (l’expérience oblige). “Tiens donc,” m’exclamai-je, “ce webappstore.sqlite me semble également bien en chair!”

Alors, cette fois là, il s’agissait de webappsstore.sqlite, au lieu de l’habituel coupable “urlclassifier3.sqlite” (qui ne semble plus exister depuis l’été 2016 environ, comme la date de modification l’indique) ou, occasionnellement, “places.sqlite” (attention, concernant ce dernier: il semble qu’il contienne également vos favoris, que vous feriez bien d’exporter au préalable d’une frappe nucléaire; voyez le glossaire des divers fichiers présents dans le dossier de profil de Firefox).

Heureusement, je n’ai pas rencontré de nouveaux problèmes depuis cet incident de 2017. On dirait qu’avec les années, ces mandelbugs se font un peu plus rares, ou se font plus discrets. Mais parfois, seul dans la nuit froide d’octobre, je me demande s’il n’y a pas encore une bestiole mutante qui grouille dans le pelage de mon Firefox. Fichu concept de stockage par fichiers SQLite corruptibles. Fichu char d’assault bipède nucléaire.


P.s.: Malgré tout, j’aime quand même Firefox; c’est le navigateur que j’utilise et préconise (en même temps, c’est le seul navigateur un peu digne de confiance qu’il nous reste, avec un moteur indépendant…). Mais ce billet est néanmoins une anecdote amusante à raconter et je me dis qu’il pourrait être, potentiellement, utile à d’autres personnes.

by Jeff at October 31, 2020 06:22 AM

October 28, 2020

Bastien NoceraSandboxing inside the sandbox: No rogue thumbnailers inside Flatpak

(Bastien Nocera)

 A couple of years ago, we sandboxed thumbnailers using bubblewrap to avoid drive-by downloads taking advantage of thumbnailers with security issues.

 It's a great tool, and it's a tool that Flatpak relies upon to create its own sandboxes. But that also meant that we couldn't use it inside the Flatpak sandboxes themselves, and those aren't always as closed as they could be, to support legacy applications.

 We've finally implemented support for sandboxing thumbnailers within Flatpak, using the Spawn D-Bus interface (indirectly).

This should all land in GNOME 40, though it should already be possible to integrate it into your Flatpaks. Make sure to use the latest gnome-desktop development version, and that the flatpak-spawn utility is new enough in the runtime you're targeting (it's been updated in the freedesktop.org runtimes #1, #2, #3, but it takes time to trickle down to GNOME versions). Example JSON snippets:

        {
"name": "flatpak-xdg-utils",
"buildsystem": "meson",
"sources": [
{
"type": "git",
"url": "https://github.com/flatpak/flatpak-xdg-utils.git",
"tag": "1.0.4"
}
]
},
{
"name": "gnome-desktop",
"buildsystem": "meson",
"config-opts": ["-Ddebug_tools=true", "-Dudev=disabled"],
"sources": [
{
"type": "git",
"url": "https://gitlab.gnome.org/GNOME/gnome-desktop.git"
}
]
}  

(We also sped up GStreamer-based thumbnailers by allowing them to use a cache, and added profiling information to the thumbnail test tools, which could prove useful if you want to investigate performance or bugs in that area)

Edit: correct a link, thanks to the commenters for the notice

by Bastien Nocera (noreply@blogger.com) at October 28, 2020 11:20 AM

October 26, 2020

GStreamerGStreamer 1.18.1 stable bug fix release

(GStreamer)

The GStreamer team is pleased to announce the first bug fix release in the stable 1.18 release series of your favourite cross-platform multimedia framework!

This release contains important security fixes. We suggest you upgrade at your earliest convenience.

This release only contains bugfixes and it should be safe to update from 1.18.0.

See the GStreamer 1.18.1 release notes for more details.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

Download tarballs directly here: gstreamer, gst-plugins-base, gst-plugins-good, gst-plugins-ugly, gst-plugins-bad, gst-libav, gst-rtsp-server, gst-python, gst-editing-services, gst-devtools, gstreamer-vaapi, gstreamer-sharp, gst-omx, or gstreamer-docs.

October 26, 2020 11:00 AM

October 21, 2020

GStreamerGStreamer 1.16.3 old-stable bug fix release

(GStreamer)

The GStreamer team is pleased to announce the third and likely last bug fix release in the stable 1.16 release series of your favourite cross-platform multimedia framework!

This release contains important security fixes. We suggest you upgrade at your earliest convenience, either to 1.16.3 or 1.18.

This release only contains bugfixes and it should be safe to update from 1.16.x.

See the GStreamer 1.16.3 release notes/ for the details.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

Download tarballs directly here: gstreamer, gst-plugins-base, gst-plugins-good, gst-plugins-ugly, gst-plugins-bad, gst-libav, gst-rtsp-server, gst-python, gst-editing-services, gst-validate, gstreamer-vaapi, or gst-omx.

October 21, 2020 08:00 PM