July 09, 2021

Víctor JáquezVideo decoding in GStreamer with Vulkan

Warning: Vulkan video is still work in progress, from specification to available drivers and applications. Do not use it for production software just yet.

Introduction

Vulkan is a cross-platform Application Programming Interface (API), backed by the Khronos Group, aimed at graphics developers for a wide range of different tasks. The interface is described by a common specification, and it is implemented by different drivers, usually provided by GPU vendors and Mesa.

One way to visualize Vulkan, at first glance, is like a low-level OpenGL API, but better described and easier to extend. Even more, it is possible to implement OpenGL on top of Vulkan. And, as far as I am told by my peers in Igalia, Vulkan drivers are easier and cleaner to implement than OpenGL ones.

A couple years ago, a technical specification group (TSG), inside the Vulkan Working Group, proposed the integration of hardware accelerated video compression and decompression into the Vulkan API. In April 2021 the formed Vulkan Video TSG published an introduction to the
specification
. Please, do not hesitate to read it. It’s quite good.

Matthew Waters worked on a GStreamer plugin using Vulkan, mainly for uploading, composing and rendering frames. Later, he developed a library mapping Vulkan objects to GStreamer. This work was key for what I am presenting here. In 2019, during the last GStreamer Conference, Matthew delivered a talk about his work. Make sure to watch it, it’s worth it.

Other key components for this effort were the base classes for decoders and the bitstream parsing libraries in GStreamer, jointly developed by Intel, Centricular, Collabora and Igalia. Both libraries allow using APIs for stateless video decoding and encoding within the GStreamer framework, such as Vulkan Video, VAAPI, D3D11, and so on.

When the graphics team in Igalia told us about the Vulkan Video TSG, we decided to explore the specification. Therefore, Igalia decided to sponsor part of my time to craft a GStreamer element to decode H.264 streams using these new Vulkan extensions.

Assumptions

As stated at the beginning of this text, this development has to be considered unstable and the APIs may change without further notice.

Right now, the only Vulkan driver that offers these extensions is the beta NVIDIA driver. You would need, at least, version 455.50.12 for Linux, but it would be better to grab the latest one. And, of course, I only tested this on Linux. I would like to thank NVIDIA for their Vk Video samples. Their test application drove my work.

Finally, this work assumes the use of the main development branch of GStreamer, because the base classes for decoders are quite recent. Naturally, you can use gst-build for an efficient upstream workflow.

Work done

This work basically consists of two new objects inside the GstVulkan code:

  • GstVulkanDeviceDecoder: a GStreamer object in GstVulkan library, inherited from GstVulkanDevice, which enables VK_KHR_video_queue and VK_KHR_video_decode_queue extensions. Its purpose is to handle codec-agnostic operations.
  • vulkanh264dec: a GStreamer element, inherited from GstH264Decoder, which tries to instantiate a GstVulkanDeviceDecoder to composite it and is in charge of handling codec-specific operations later, such as matching the parsed structures. It outputs, in the source pad, memory:VulkanImage featured frames, with NV12 color format.

  • So far this pipeline works without errors:

    $ gst-launch-1.0 filesrc location=big_buck_bunny_1080p_h264.mov ! parsebin ! vulkanh264dec ! fakesink
    

    As you might see, the pipeline does not use vulkansink to render frames. This is because the Vulkan format output by the driver’s decoder device is VK_FORMAT_G8_B8R8_2PLANE_420_UNORM, which is NV12 crammed in a single image, while for GstVulkan a NV12 frame is a buffer with two images, one per component. So the current color conversion in GstVulkan does not support this Vulkan format. That is future work, among other things.

    You can find the merge request for this work in GStreamer’s Gitlab.

    Future work

    As was mentioned before, it is required to fully support VK_FORMAT_G8_B8R8_2PLANE_420_UNORM format in GstVulkan. That requires thinking about how to keep backwards compatibility. Later, an implementation of the sampler to convert this format to RGB will be needed, so that decoded frames can be rendered by vulkansink.

    Also, before implementing any new feature, the code and its abstractions will need to be cleaned up, since currently the division between codec-specific and codec-agnostic code is not strict, and it must be fixed.

    Another important cleanup task is to enhance the way the Vulkan headers are handled. Since the required headers files for video extensions are beta, they are not expected to be available in the system, so temporally I had to add the those headers as part of the GstVulkan library.

    Then it will be possible to implement the H.265 decoder, since the NVIDIA driver also supports it.

    Later on, it will be nice to start thinking about encoders. But this requires extending support for stateless encoders in GStreamer, something I want do to for the new VAAPI plugin too.

    Thanks for bearing with me, and thanks to Igalia for sponsoring this work.

    by vjaquez at July 09, 2021 05:38 PM

    June 28, 2021

    GStreamerGStreamer Rust bindings 0.17.0 release

    (GStreamer)

    A new version of the GStreamer Rust bindings, 0.17.0, was released.

    As usual this release follows the latest gtk-rs release.

    This is the first version that includes optional support for new GStreamer 1.20 APIs. As GStreamer 1.20 was not released yet, these new APIs might still change. The minimum supported version of the bindings is still GStreamer 1.8 and the targetted GStreamer API version can be selected by applications via feature flags.

    Apart from this, the new version features a lot of API cleanup, especially of the subclassing APIs, and the addition of a few missing bindings. As usual, the focus of this release was to make usage of GStreamer from Rust as convenient and complete as possible.

    The new release also brings a lot of bugfixes, most of which were already part of the 0.16.x bugfix releases.

    A new release of the GStreamer Rust plugins will follow in the next days.

    Details can be found in the release notes for gstreamer-rs.

    The code and documentation for the bindings is available on the freedesktop.org GitLab

    as well as on crates.io.

    If you find any bugs, notice any missing features or other issues please report them in GitLab.

    June 28, 2021 11:00 PM

    June 24, 2021

    Seungha YangGStreamer Media Foundation Video Encoder Is Now Faster — Direct3D11 Awareness

    GStreamer Media Foundation Video Encoder Is Now Faster — Direct3D11 Awareness

    TL;DR

    GStreamer MediaFoundation video encoders (H.264, HEVC, and VP9 if supported by GPU) gained the ability to accept Direct3D11 textures, which will bring noticeable performance improvements

    As of the GStreamer 1.18 release, hardware accelerated Direct3D11/DXVA video decoding and MediaFoundation based video encoding features were landed.

    Those native Windows video APIs can be very helpful for application development/deployment, since they are hardware platform-agnostic APIs for the Windows platform. The questions is if they are sufficiently competitive with hardware-specific APIs such as NVIDIA NVCODEC SDK or Intel Media SDK?

    Probably the answer is … “NO”

    How much faster than before are things?

    One simple way to compare performance would be to measure the time spent for transcoding. Of course, encoded file size and visual quality are also very important factors. However, as per my experiments, resulting video file size and visual quality (in terms of PSNR) were very close to each other. Then our remaining interest is speed!

    Let’s take a look at my measurement. I performed the measurement by using one 4K H.264 video content with an NVIDIA RTX 3060 GPU and an Intel Core i7–1065G7 integrated GPU. For reference, NVCODEC and Intel Media SDK plugins were tested by using GStreamer 1.19.1 as well. Each test used performance (speed) oriented encoding options to be a fair comparison.

    - NVIDA RTX 3060

    GStreamer 1.18 — 2 min 1 sec
    GStreamer 1.19.1 1 min 9 sec
    NVCODEC plugin (nvh264dec/nvh264enc pair) — 1 min 19 sec

    - Intel Core i7–1065G7 integrated GPU

    GStreamer 1.18 — 3 min 8 sec
    GStreamer 1.19.1 — 2 min 45 sec
    Intel Media SDK plugin (msdkh264dec/msdkh264enc pair)3 min 10 sec

    So, is it true that the Direct3D11/DXVA and MediaFoundation combination can be faster than hardware-specific APIs? Yes, as you can see

    Note that such results would be very system environment and encoding option dependent, so, you’d likely see different numbers

    Why MediaFoundation plugin got faster

    GStreamer 1.18 — The story was, because of the lack of Direct3D11 integration at MediaFoundation plugin side, each decoded frame (Direct3D11 texture) must be downloaded into system memory first, which is usually very slow path. And then, the memory was copied to another system memory allocated by MediaFoundation. Moreover, likely GPU driver would upload to GPU memory again. Well, twice visible redundant copies and another potential copy per frame!?!? hrm…

    In GStreamer 1.19.1, thanks to the Direct3D11 integration, MediaFoundation can accept Direct3D11 texture, which means we don’t need to download GPU texture and re-upload it any more.

    More details

    Since all Direct3D11/DXVA, MediaFoundation, NVCODEC, and Intel Media SDK APIs work with underlying GPU hardware, the performance should not be much different in theory, unless there are visible overhead around GPU vendor’s driver implementation.

    Then, remaining factor would be API consumer-side optimization.
    And yes, from GStreamer plugin implementation point of view, Direct3D11/DXVA and MediaFoundation plugins
    are more optimized than NVCODEC and MSDK plugins in terms of GPU memory transfer on Windows.

    It doesn’t mean Direct3D11/DXVA and MediaFoundation themselves are superior APIs than hardware-specific APIs at all. The difference is just result of more or less optimized plugin implementations

    You can try this enhancement right now!

    Install the official GStreamer 1.19.1 release, and just run this command.

    gst-launch-1.0.exe filesrc location=where-your-h264-file-located ! parsebin ! d3d11h264dec ! queue ! mfh264enc ! h264parse ! mp4mux ! filesink location=my-output.mp4

    You will be likely able to see the improvement by yourself :)

    There are still a lot of interesting topics for better Windows support in GStreamer. Specifically, nicer text support via DirectWrite and fine tuned GPU scheduling via Direct3D12 are on our radar. Not only for video features, we will keep improving various Windows specific features, including audio capture/render device support.

    If you’ve see any bugs, please contact me, and even better would be a bug report at GitLab. I’m watching it most of time 😊

    by Seungha Yang at June 24, 2021 04:01 PM

    June 15, 2021

    GStreamerIRC Channel has moved from Freenode to OFTC

    (GStreamer)

    Due to the widely reported issues at the Freenode IRC network, the official GStreamer discussion IRC channel has moved to #gstreamer on the OFTC IRC network alongside other Freedesktop projects.

    You can connect to it with your existing IRC client, or using Matrix which has a browser client and native apps for all platforms.

    For more information, please see the mailing list announcement.

    June 15, 2021 08:30 AM

    June 03, 2021

    Thomas Vander SticheleAmazing Marvin and KeyCombiner

    (Thomas Vander Stichele)

    I recently came across an excellent tool called KeyCombiner that helps you practice keyboard shortcuts (3 sets for free, $29/6 months for more sets). I spent some time to create a set for Amazing Marvin, my current todo manager of choice.

    The shareable URL to use in KeyCombiner is https://keycombiner.com/collecting/collections/shared/f1f78977-0920-4888-a86d-d00a7201502e

    I generated it from the printed PDF version of Marvin’s keyboard guide and a bunch of manual editing, in a google sheet.

    Keyboard shortcuts are great timesavers and help reduce friction, but it’s getting harder to learn them properly, and this tool has been a great help for some other apps, and for figuring out common shortcuts across apps, and for picking custom shortcuts (in other apps) that don’t conflict. If this is a problem you recognize, give KeyCombiner a try.

    Flattr this!

    by Thomas at June 03, 2021 02:27 AM

    June 01, 2021

    GStreamerGStreamer 1.19.1 unstable development release

    (GStreamer)

    The GStreamer team is pleased to announce the first development release in the unstable 1.19 release series.

    The unstable 1.19 release series adds new features on top of the current stable 1.18 series and is part of the API and ABI-stable 1.x release series of the GStreamer multimedia framework.

    The unstable 1.19 release series is for testing and development purposes in the lead-up to the stable 1.20 series which is scheduled for release in a few weeks time. Any newly-added API can still change until that point, although it is rare for that to happen.

    Full release notes will be provided in the near future, highlighting all the new features, bugfixes, performance optimizations and other important changes.

    This development release is primarily for distributors and early adaptors and anyone who still needs to update their build/packaging setup for Meson.

    Packagers: please note that plugins may have moved between modules, so please take extra care and make sure inter-module version dependencies are such that users can only upgrade all modules in one go, instead of seeing a mix of 1.19 and 1.18 on their system.

    Binaries for Android, iOS, Mac OS X and Windows are also available at the usual location.

    Release tarballs can be downloaded directly here:

    As always, please let us know of any issues you run into by filing an issue in Gitlab.

    June 01, 2021 04:00 PM

    Robert McQueenNext steps for the GNOME Foundation

    (Robert McQueen)

    As the President of the GNOME Foundation Board of Directors, I’m really pleased to see the number and breadth of candidates we have for this year’s election. Thank you to everyone who has submitted their candidacy and volunteered their time to support the Foundation. Allan has recently blogged about how the board has been evolving, and I wanted to follow that post by talking about where the GNOME Foundation is in terms of its strategy. This may be helpful as people consider which candidates might bring the best skills to shape the Foundation’s next steps.

    Around three years ago, the Foundation received a number of generous donations, and Rosanna (Director of Operations) gave a presentation at GUADEC about her and Neil’s (Executive Director, essentially the CEO of the Foundation) plans to use these funds to transform the Foundation. We would grow our activities, increasing the pace of events, outreach, development and infrastructure that supported the GNOME project and the wider desktop ecosystem – and, crucially, would grow our funding to match this increased level of activity.

    I think it’s fair to say that half of this has been a great success – we’ve got a larger staff team than GNOME has ever had before. We’ve widened the GNOME software ecosystem to include related apps and projects under the GNOME Circle banner, we’ve helped get GTK 4 out of the door, run a wider-reaching program in the Community Engagement Challenge, and consistently supported better infrastructure for both GNOME and the Linux app community in Flathub.

    Aside from another grant from Endless (note: my employer), our fundraising hasn’t caught up with this pace of activities. As a result, the Board recently approved a budget for this financial year which will spend more funds from our reserves than we expect to raise in income. Due to our reserves policy, this is essentially the last time we can do this: over the next 6-12 months we need to either raise more money, or start spending less.

    For clarity – the Foundation is fit and well from a financial perspective – we have a very healthy bank balance, and a very conservative “12 month run rate” reserve policy to handle fluctuations in income. If we do have to slow down some of our activities, we will return to a “steady state” where our regular individual donations and corporate contributions can support a smaller staff team that supports the events and infrastructure we’ve come to rely on.

    However, this isn’t what the Board wants to do – the previous and current boards were unanimous in their support of the idea that we should be ambitious: try to do more in the world and bring the benefits of GNOME to more people. We want to take our message of trusted, affordable and accessible computing to the wider world.

    Typically, a lot of the activities of the Foundation have been very inwards-facing – supporting and engaging with either the existing GNOME or Open Source communities. This is a very restricted audience in terms of fundraising – many corporate actors in our community already support GNOME hugely in terms of both financial and in-kind contributions, and many OSS users are already supporters either through volunteer contributions or donating to those nonprofits that they feel are most relevant and important to them.

    To raise funds from new sources, the Foundation needs to take the message and ideals of GNOME and Open Source software to new, wider audiences that we can help. We’ve been developing themes such as affordability, privacy/trust and education as promising areas for new programs that broaden our impact. The goal is to find projects and funding that allow us to both invest in the GNOME community and find new ways for FOSS to benefit people who aren’t already in our community.

    Bringing it back to the election, I’d like to make clear that I see this – reaching the outside world, and finding funding to support that – as the main priority and responsibility of the Board for the next term. GNOME Foundation elections are a slightly unusual process that “filters” our board nominees by being existing Foundation members, which means that candidates already work inside our community when they stand for election. If you’re a candidate and are already active in the community – THANK YOU – you’re doing great work, keep doing it! That said, you don’t need to be a Director to achieve things within our community or gain the support of the Foundation: being a community leader is already a fantastic and important role.

    The Foundation really needs support from the Board to make a success of the next 12-18 months. We need to understand our financial situation and the trade-offs we have to make, and help to define the strategy with the Executive Director so that we can launch some new programs that will broaden our impact – and funding – for the future. As people cast their votes, I’d like people to think about what kind of skills – building partnerships, commercial background, familiarity with finances, experience in nonprofit / impact spaces, etc – will help the Board make the Foundation as successful as it can be during the next term.

    by ramcq at June 01, 2021 11:45 AM

    May 25, 2021

    Enrique Ocaña GonzálezGStreamer WebKit debugging by using external tools (2/2)

    This is the last post of the series showing interesting debugging tools, I hope you have found it useful. Don’t miss the custom scripts at the bottom to process GStreamer logs, help you highlight the interesting parts and find the root cause of difficult bugs. Here are also the previous posts of the series:

    How to debug pkgconfig

    When pkg-config finds the PKG_CONFIG_DEBUG_SPEW env var, it explains all the steps used to resolve the packages:

    PKG_CONFIG_DEBUG_SPEW=1 /usr/bin/pkg-config --libs x11

    This is useful to know why a particular package isn’t found and what are the default values for PKG_CONFIG_PATH when it’s not defined. For example:

    Adding directory '/usr/local/lib/x86_64-linux-gnu/pkgconfig' from PKG_CONFIG_PATH
    Adding directory '/usr/local/lib/pkgconfig' from PKG_CONFIG_PATH
    Adding directory '/usr/local/share/pkgconfig' from PKG_CONFIG_PATH
    Adding directory '/usr/lib/x86_64-linux-gnu/pkgconfig' from PKG_CONFIG_PATH
    Adding directory '/usr/lib/pkgconfig' from PKG_CONFIG_PATH
    Adding directory '/usr/share/pkgconfig' from PKG_CONFIG_PATH

    If we have tuned PKG_CONFIG_PATH, maybe we also want to add the default paths. For example:

    SYSROOT=~/sysroot-x86-64
    export PKG_CONFIG_PATH=${SYSROOT}/usr/local/lib/pkgconfig:${SYSROOT}/usr/lib/pkgconfig
    # Add also the standard pkg-config paths to find libraries in the system
    export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:/usr/local/lib/x86_64-linux-gnu/pkgconfig:\
    /usr/local/lib/pkgconfig:/usr/local/share/pkgconfig:/usr/lib/x86_64-linux-gnu/pkgconfig:\
    /usr/lib/pkgconfig:/usr/share/pkgconfig
    # This tells pkg-config where the "system" pkg-config dir is. This is useful when cross-compiling for other
    # architecture, to avoid pkg-config using the system .pc files and mixing host and target libraries
    export PKG_CONFIG_LIBDIR=${SYSROOT}/usr/lib
    # This could have been used for cross compiling:
    #export PKG_CONFIG_SYSROOT_DIR=${SYSROOT}

    Man in the middle proxy for WebKit

    Sometimes it’s useful to use our own modified/unminified files with a 3rd party service we don’t control. Mitmproxy can be used as a man-in-the-middle proxy, but I haven’t tried it personally yet. What I have tried (with WPE) is this:

    1. Add an /etc/hosts entry to point the host serving the files we want to change to an IP address controlled by us.
    2. Configure a web server to provide the files in the expected path.
    3. Modify the ResourceRequestBase constructor to change the HTTPS requests to HTTP when the hostname matches the target:
    ResourceRequestBase(const URL& url, ResourceRequestCachePolicy policy)
        : m_url(url)
        , m_timeoutInterval(s_defaultTimeoutInterval)
        ...
        , m_isAppBound(false)
    {
        if (m_url.host().toStringWithoutCopying().containsIgnoringASCIICase(String("out-of-control-service.com"))
            && m_url.protocol().containsIgnoringASCIICase(String("https"))) {
            printf("### %s: URL %s detected, changing from https to http\n",
                __PRETTY_FUNCTION__, m_url.string().utf8().data()); 
            fflush(stdout);
            m_url.setProtocol(String("http"));
        }
    }

    :bulb: Pro tip: If you have to debug minified/obfuscated JavaScript code and don’t have a deobfuscated version to use in a man-in-the-middle fashion, use http://www.jsnice.org/ to deobfuscate it and get meaningful variable names.

    Bandwidth control for a dependent device

    If your computer has a “shared internet connection” enabled in Network Manager and provides access to a dependent device , you can control the bandwidth offered to that device. This is useful to trigger quality changes on adaptive streaming videos from services out of your control.

    This can be done using tc, the Traffic Control tool from the Linux kernel. You can use this script to automate the process (edit it to suit to your needs).

    Useful scripts to process GStreamer logs

    I use these scripts in my daily job to look for strange patterns in GStreamer logs that help me to find the cause of the bugs I’m debugging:

    • h: Highlights each expression in the command line in a different color.
    • mgrep: Greps (only) for the lines with the expressions in the command line and highlights each expression in a different color.
    • filter-time: Gets a subset of the log lines between a start and (optionally) an end GStreamer log timestamp.
    • highlight-threads: Highlights each thread in a GStreamer log with a different color. That way it’s easier to follow a thread with the naked eye.
    • remove-ansi-colors: Removes the color codes from a colored GStreamer log.
    • aha: ANSI-HTML-Adapter converts plain text with color codes to HTML, so you can share your GStreamer logs from a web server (eg: for bug discussion). Available in most distros.
    • gstbuffer-leak-analyzer: Analyzes a GStreamer log and shows unbalances in the creation/destruction of GstBuffer and GstMemory objects.

    by eocanha at May 25, 2021 06:00 AM

    May 20, 2021

    Reynaldo VerdejoShort tutorial: Digital Television with GStreamer (ATSC setup)

    GStreamer support for Digital Television (DTV) is still working. If you follow a few very simple steps you can get DTV signals tuned-to, captured, decoded and displayed with your computer and a few accessories. 

    This short article presents basic instructions for a working DTV setup. I use ATSC in the US as an example, but the short process should work equally well for other terrestrial delivery systems and countries.

    What hardware do you need?

    Beyond a computer, you will need an antenna and a capture device. I recently tested a Hauppauge WinTV-dualHD (ATSC/QAM) USB dongle with a 5-years old Amazon basics flat indoor antenna (that looks pretty much like this one) and it worked quite well at roughly 60km from the repeater station.

    Installation

    The hardware setup is simple. Install the antenna pointing to your repeater station and as high as you can. Your local telecommunications and/or broadcasting association should have data about repeaters near you. You can get this information by ZIP code from the Federal Communications Commission (FCC) if you are in the US.

    Don't stress about the "as high as you can" part of the antenna installation. It's great to have a uninterrupted and direct line of sight to the repeating station's antenna, but this is hard to accomplish in urban areas. My current test system is indoors, on a first floor, and it works fine for a few channels.

    Software

    You will need GStreamer. I'm building it from the master branch with gst-build but the DTV code hasn't changed much in the last few years so any recently-packaged version should work for you. You will also need dvbv5-scan from DVBv5 tools.

    TL;DR

    Build a channels.conf configuration file by scanning an initial set of frequencies for your area with dvbv5-scan. You can get initial-frequencies files from the the dtv-scan-tables repository. Here I'm using my Mountain View, California file as an example:

    $ dvbv5-scan -o channels.conf ./dtv-scan-tables/atsc/us-CA-Mountain-View

    The scanning can take several minutes.

    If you grep the resulting file you can see what channels were detected:

    $ grep "\\[" channels.conf

    The output should look something like this:


    [KNTV-HD]
    [Hai Le]
    [KPIX-TV]
    [KQED-HD]
    [KTVU-HD]
    [ION]
    [KFSF-HD]
    [KICU-HD]


    If you get no channels, and you have reasons to believe your hardware is working correctly, try repositioning your antenna and rescanning till you get some. Small direction and position changes can have big effects on VHF/UHF reception with semi-directional antennas like the flat one I'm using for this example.

    To playback a channel with GStreamer you can use the generated channels.conf configuration file and any of the scanned-channel names to let playbin/dvbbasebin figure out all necessary parameters at runtime

    $ GST_DVB_CHANNELS_CONF=channels.conf gst-play-1.0 dvb://KNTV-HD

    And that's it.

    There are lots of details, pitfalls and options I don't write about in this short tutorial but if there's some interest I will try to discuss the subject in more depth in future ones.

    I leave you with two screenshot of the HD feed by KNTV in the bay area

     



     

     

     


     






    by reynaldo (noreply@blogger.com) at May 20, 2021 06:36 PM

    Christian SchallerNew opportunities in the Red Hat Desktop team

    (Christian Schaller)

    So we are looking to hire quite a few people into the Desktop team currently. First of all we are looking to hire two graphics engineers to help us work on Linux Graphics drivers. The first of those two jobs is now online on the Red Hat jobs site. This is a job in our core graphics team focusing on RHEL, Fedora and upstream around the Intel, AMD and NVidia open source drivers. This is an opportunity to join a team of incredibly talented engineers working on everything from the graphics system of the Linux kernel and on the userspace bits like Vulkan, OpenGL and Wayland.  The job is listed as Senior Engineer, but for the right candidate we have flexibility there. We also have flexibility for people who want to work remotely, so as long as there is a Red Hat office in your home country you can work remotely for us.  The second job, which we hope to have up soon, will be looking more at ARM graphics and be tied to our automotive effort, but we will be looking at the applications for either position in combination so feel free to apply for the already listed job even if you are more interested in the second one as we will discuss both jobs with potential candidates.

    The second job we have up is for – Software Engineer – GPU, Input and Multimedia which is also for joining our Graphics team. This job is targetted at our  office in Brno, Czechia and is a great entry level position if you are interested in the field of graphics. The job listing can be found here and outlines the kind of things we want you to look at, but do expect initially your job will be focused on helping the rest of the team manage their backlog and then grow from there.

    The last job we have online now is for the automotive team, where we are looking for someone at the Senior/Principal level to join our Infotainment team, working with car makers around issues related to multimedia and help identifying needs and gaps and then work with upstream communities to figure out how we can resolve those issues. The job is targeted at Madrid, Spain as it is where we hope to center some of the infotainment effort and it makes things easier in terms of hardware access and similar, but for the right candidate we might be open to looking for candidates wanting to work remote or in another Red Hat office. You can find this job listing here.

    We expect to be posting further jobs for the infotainment team within a week or two, so I will update once they are up.

    by uraeus at May 20, 2021 03:56 PM

    May 18, 2021

    Enrique Ocaña GonzálezGStreamer WebKit debugging by using external tools (1/2)

    In this new post series, I’ll show you how both existing and ad-hoc tools can be helpful to find the root cause of some problems. Here are also the older posts of this series in case you find them useful:

    Use strace to know which config/library files are used by a program

    If you’re becoming crazy supposing that the program should use some config and it seems to ignore it, just use strace to check what config files, libraries or other kind of files is the program actually using. Use the grep rules you need to refine the search:

    $ strace -f -e trace=%file nano 2> >(grep 'nanorc')
    access("/etc/nanorc", R_OK)             = 0
    access("/usr/share/nano/javascript.nanorc", R_OK) = 0
    access("/usr/share/nano/gentoo.nanorc", R_OK) = 0
    ...

    Know which process is killing another one

    First, try to strace -e trace=signal -p 1234 the killed process.

    If that doesn’t work (eg: because it’s being killed with the uncatchable SIGKILL signal), then you can resort to modifying the kernel source code (signal.c) to log the calls to kill():

    SYSCALL_DEFINE2(kill, pid_t, pid, int, sig)
    {
        struct task_struct *tsk_p;
        ...
        /* Log SIGKILL */
        if (sig & 0x1F == 9) {
            tsk_p = find_task_by_vpid(pid);
    
            if (tsk_p) {
                printk(KERN_DEBUG "Sig: %d from pid: %d (%s) to pid: %d (%s)\n",
                    sig, current->pid, current->comm, pid, tsk_p->comm);
            } else {
                printk(KERN_DEBUG "Sig: %d from pid: %d (%s) to pid: %d\n",
                    sig, current->pid, current->comm, pid);
            }
        }
        ...
    }

    Wrap gcc/ld/make to tweak build parameters

    If you ever find yourself with little time in front of a stubborn build system and, no matter what you try, you can’t get the right flags to the compiler, think about putting something (a wrapper) between the build system and the compiler. Example for g++:

    #!/bin/bash
    main() {
        # Build up arg[] array with all options to be passed
        # to subcommand.
        i=0
        for opt in "$@"; do
            case "$opt" in
            -O2) ;; # Removes this option
            *)
                arg[i]="$opt" # Keeps the others
                i=$((i+1))
                ;;
            esac
        done
        EXTRA_FLAGS="-O0" # Adds extra option
        echo "g++ ${EXTRA_FLAGS} ${arg[@]}" # >> /tmp/build.log # Logs the command
        /usr/bin/ccache g++ ${EXTRA_FLAGS} "${arg[@]}" # Runs the command
    }
    main "$@"

    Make sure that the wrappers appear earlier than the real commands in your PATH.

    The make wrapper can also call remake instead. Remake is fully compatible with make but has features to help debugging compilation and makefile errors.

    Analyze the structure of MP4 data

    The ISOBMFF Box Structure Viewer online tool allows you to upload an MP4 file and explore its structure.

    by eocanha at May 18, 2021 06:00 AM

    May 13, 2021

    Andy Wingocross-module inlining in guile

    (Andy Wingo)

    Greetings, hackers of spaceship Earth! Today's missive is about cross-module inlining in Guile.

    a bit of history

    Back in the day... what am I saying? I always start these posts with loads of context. Probably you know it all already. 10 years ago, Guile's partial evaluation pass extended the macro-writer's bill of rights to Schemers of the Guile persuasion. This pass makes local function definitions free in many cases: if they should be inlined and constant-folded, you are confident that they will be. peval lets you write clear programs with well-factored code and still have good optimization.

    The peval pass did have a limitation, though, which wasn't its fault. In Guile, modules have historically been a first-order concept: modules are a kind of object with a hash table inside, which you build by mutating. I speak crassly but that's how it is. In such a world, it's hard to reason about top-level bindings: what module do they belong to? Could they ever be overridden? When you have a free reference to a, and there's a top-level definition of a in the current compilation unit, is that the a that's being referenced, or could it be something else? Could the binding be mutated in the future?

    During the Guile 2.0 and 2.2 cycles, we punted on this question. But for 3.0, we added the notion of declarative modules. For these modules, bindings which are defined once in a module and which are not mutated in the compilation unit are declarative bindings, which can be reasoned about lexically. We actually translate them to a form of letrec*, which then enables inlining via peval, contification, and closure optimization -- in descending order of preference.

    The upshot is that with Guile 3.0, top-level bindings are no longer optimization barriers, in the case of declarative modules, which are compatible enough with historic semantics and usage that they are on by default.

    However, module boundaries have still been an optimization barrier. Take (srfi srfi-1), a little utility library on lists. One definition in the library is xcons, which is cons with arguments reversed. It's literally (lambda (cdr car) (cons car cdr)). But does the compiler know that? Would it know that (car (xcons x y)) is the same as y? Until now, no, because no part of the optimizer will look into bindings from outside the compilation unit.

    mr compiler, tear down this wall

    But no longer! Guile can now inline across module boundaries -- in some circumstances. This feature will be part of a future Guile 3.0.8.

    There are actually two parts of this. One is the compiler can identify a set of "inlinable" values from a declarative module. An inlinable value is a small copyable expression. A copyable expression has no identity (it isn't a fresh allocation), and doesn't reference any module-private binding. Notably, lambda expressions can be copyable, depending on what they reference. The compiler then extends the module definition that's residualized in the compiled file to include a little procedure that, when passed a name, will return the Tree-IL representation of that binding. The design of that was a little tricky; we want to avoid overhead when using the module outside of the compiler, even relocations. See compute-encoding in that module for details.

    With all of that, we can call ((module-inlinable-exports (resolve-interface '(srfi srfi-1))) 'xcons) and get back the Tree-IL equivalent of (lambda (cdr car) (cons car cdr)). Neat!

    The other half of the facility is the actual inlining. Here we lean on peval again, causing <module-ref> forms to trigger an attempt to copy the term from the imported module to the residual expression, limited by the same effort counter as the rest of peval.

    The end result is that we can be absolutely sure that constants in imported declarative modules will inline into their uses, and fairly sure that "small" procedures will inline too.

    caveat: compiled imported modules

    There are a few caveats about this facility, and they are sufficiently sharp that I should probably fix them some day. The first one is that for an imported module to expose inlinable definitions, the imported module needs to have been compiled already, not loaded from source. When you load a module from source using the interpreter instead of compiling it first, the pipeline is optimized for minimizing the latency between when you ask for the module and when it is available. There's no time to analyze the module to determine which exports are inlinable and so the module exposes no inlinable exports.

    This caveat is mitigated by automatic compilation, enabled by default, which will compile imported modules as needed.

    It could also be fixed for modules by topologically ordering the module compilation sequence; this would allow some parallelism in the build but less than before, though for module graphs with cycles (these exist!) you'd still have some weirdness.

    caveat: abi fragility

    Before Guile supported cross-module inlining, there was only explicit inlining across modules in Guile, facilitated by macros. If you write a module that has a define-inlinable export and you think about its ABI, then you know to consider any definition referenced by the inlinable export, and you know by experience that its code may be copied into other compilation units. Guile doesn't automatically recompile a dependent module when a macro that it uses changes, currently anyway. Admittedly this situation leans more on culture than on tools, which could be improved.

    However, with automatically inlinable exports, this changes. Any definition in a module could be inlined into its uses in other modules. This may alter the ABI of a module in unexpected ways: you think that module C depends on module B, but after inlining it may depend on module A as well. Updating module B might not update the inlined copies of values from B into C -- as in the case of define-inlinable, but less lexically apparent.

    At higher optimization levels, even private definitions in a module can be inlinable; these may be referenced if an exported macro from the module expands to a term that references a module-private variable, or if an inlinable exported binding references the private binding. But these optimization levels are off by default precisely because I fear the bugs.

    Probably what this cries out for is some more sensible dependency tracking in build systems, but that is a topic for another day.

    caveat: reproducibility

    When you make a fresh checkout of Guile from git and build it, the build proceeds in the following way.

    Firstly, we build libguile, the run-time library implemented in C.

    Then we compile a "core" subset of Scheme files at optimization level -O1. This subset should include the evaluator, reader, macro expander, basic run-time, and compilers. (There is a bootstrap evaluator, reader, and macro expander in C, to start this process.) Say we have source files S0, S1, S2 and so on; generally speaking, these files correspond to Guile modules M0, M1, M2 etc. This first build produces compiled files C0, C1, C2, and so on. When compiling a file S2 implementing module M2, which happens to import M1 and M0, it may be M1 and M0 are provided by compiled files C1 and C0, or possibly they are loaded from the source files S1 and S0, or C1 and S0, or S1 and C0.

    The bootstrap build uses make for parallelism, with each compile process starts afresh, importing all the modules that comprise the compiler and then using them to compile the target file. As the build proceeds, more and more module imports can be "serviced" by compiled files instead of source files, making the build go faster and faster. However this introduces system-specific nondeterminism as to the set of compiled files available when compiling any other file. This strategy works because it doesn't really matter whether module M1 is provided by compiled file C1 or source file S1; the compiler and the interpreter implement the same language.

    Once the compiler is compiled at optimization level -O1, Guile then uses that freshly built compiler to build everything at -O2. We do it in this way because building some files at -O1 then all files at -O2 takes less time than going straight to -O2. If this sounds weird, that's because it is.

    The resulting build is reproducible... mostly. There is a bug in which some unique identifiers generated as part of the implementation of macros can be non-reproducible in some cases, and that disabling parallel builds seems to solve the problem. The issue being that gensym (or equivalent) might be called a different number of times depending on whether you are loading a compiled module, or whether you need to read and macro-expand it. The resulting compiled files are equivalent under alpha-renaming but not bit-identical. This is a bug to fix.

    Anyway, at optimization level -O1, Guile will record inlinable definitions. At -O2, Guile will actually try to do cross-module inlining. We run into two issues when compiling Guile; one is if we are in the -O2 phase, and we compile a module M which uses module N, and N is not in the set of "core" modules. In that case depending on parallelism and compile order, N may be loaded from source, in which case it has no inlinable exports, or from a compiled file, in which case it does. This is not a great situation for the reliability of this optimization. I think probably in Guile we will switch so that all modules are compiled at -O1 before compiling at -O2.

    The second issue is more subtle: inlinable bindings are recorded after optimization of the Tree-IL. This is more optimal than recording inlinable bindings before optimization, as a term that is not inlinable due to its size in its initial form may become small enough to inline after optimization. However, at -O2, optimization includes cross-module inlining! A term that is inlinable at -O1 may become not inlinable at -O2 because it gets slightly larger, or vice-versa: terms that are too large at -O1 could shrink at -O2. We don't even have a guarantee that we will reach a fixed point even if we repeatedly recompile all inputs at -O2, because we allow non-shrinking optimizations.

    I think this probably calls for a topological ordering of module compilation inside Guile and perhaps in other modules. That would at least give us reproducibility, provided we avoid the feedback loop of keeping around -O2 files compiled from a previous round, even if they are "up to date" (their corresponding source file didn't change).

    and for what?

    People who have worked on inliners will know what I mean that a good inliner is like a combine harvester: ruthlessly efficient, a qualitative improvement compared to not having one, but there is a pointy end with lots of whirling blades and it's important to stop at the end of the row. You do develop a sense of what will and won't inline, and I think Dybvig's "Macro writer's bill of rights" encompasses this sense. Luckily people don't lose fingers or limbs to inliners, but inliners can maim expectations, and cross-module inlining more so.

    Still, what it buys us is the freedom to be abstract. I can define a module like:

    (define-module (elf)
      #:export (ET_NONE ET_REL ET_EXEC ET_DYN ET_CORE))
    
    (define ET_NONE		0)		; No file type
    (define ET_REL		1)		; Relocatable file
    (define ET_EXEC		2)		; Executable file
    (define ET_DYN		3)		; Shared object file
    (define ET_CORE		4)		; Core file
    

    And if a module uses my (elf) module and references ET_DYN, I know that the module boundary doesn't prevent the value from being inlined as a constant (and possibly unboxed, etc).

    I took a look and on our usual microbenchmark suite, cross-module inlining doesn't make a difference. But that's both a historical oddity and a bug: firstly that the benchmark suite comes from an old Scheme world that didn't have modules, and so won't benefit from cross-module inlining. Secondly, Scheme definitions from the "default" environment that aren't explicitly recognized as primitives aren't inlined, as the (guile) module isn't declarative. (Probably we should fix the latter at some point.)

    But still, I'm really excited about this change! Guile developers use modules heavily and have been stepping around this optimization boundary for years. I count 100 direct uses of define-inlinable in Guile, a number of them inside macros, and many of these are to explicitly hack around the optimization barrier. I really look forward to seeing if we can remove some of these over time, to go back to plain old define and just trust the compiler to do what's needed.

    by the numbers

    I ran a quick analysis of the modules include in Guile to see what the impact was. Of the 321 files that define modules, 318 of them are declarative, and 88 contain inlinable exports (27% of total). Of the 6519 total bindings exported by declarative modules, 566 of those are inlinable (8.7%). Of the inlinable exports, 388 (69%) are functions (lambda expressions), 156 (28%) are constants, and 22 (4%) are "primitives" referenced by value and not by name, meaning definitions like (define head car) (instead of re-exporting car as head).

    On the use side, 90 declarative modules import inlinable bindings (29%), resulting in about 1178 total attempts to copy inlinable bindings. 902 of those attempts are to copy a lambda expressions in operator position, which means that peval will attempt to inline their code. 46 of these attempts fail, perhaps due to size or effort constraints. 191 other attempts end up inlining constant values. 20 inlining attempts fail, perhaps because a lambda is used for a value. Somehow, 19 copied inlinable values get elided because they are evaluated only for their side effects, probably to clean up let-bound values that become unused due to copy propagation.

    All in all, an interesting endeavor, and one to improve on going forward. Thanks for reading, and catch you next time!

    by Andy Wingo at May 13, 2021 11:25 AM

    May 11, 2021

    Enrique Ocaña GonzálezGStreamer WebKit debugging by instrumenting source code (3/3)

    This is the last post on the instrumenting source code series. I hope you to find the tricks below as useful as the previous ones.

    In this post I show some more useful debugging tricks. Don’t forget to have a look at the other posts of the series:

    Finding memory leaks in a RefCounted subclass

    The source code shown below must be placed in the .h where the class to be debugged is defined. It’s written in a way that doesn’t need to rebuild RefCounted.h, so it saves a lot of build time. It logs all refs, unrefs and adoptPtrs, so that any anomaly in the refcounting can be traced and investigated later. To use it, just make your class inherit from LoggedRefCounted instead of RefCounted.

    Example output:

    void WTF::adopted(WTF::LoggedRefCounted<T>*) [with T = WebCore::MediaSourceClientGStreamerMSE]: this=0x673c07a4, refCount 1
    void WTF::adopted(WTF::LoggedRefCounted<T>*) [with T = WebCore::MediaSourceClientGStreamerMSE]: this=0x673c07a4, refCount 1
    ^^^ Two adopts, this is not good.
    void WTF::LoggedRefCounted<T>::ref() [with T = WebCore::MediaSourceClientGStreamerMSE]: this=0x673c07a4, refCount 1 --> ...
    void WTF::LoggedRefCounted<T>::ref() [with T = WebCore::MediaSourceClientGStreamerMSE]: this=0x673c07a4, refCount ... --> 2
    void WTF::LoggedRefCounted<T>::deref() [with T = WebCore::MediaSourceClientGStreamerMSE]: this=0x673c07a4, refCount 2 --> ...
    void WTF::LoggedRefCounted<T>::deref() [with T = WebCore::MediaSourceClientGStreamerMSE]: this=0x673c07a4, refCount ... --> 1
    void WTF::adopted(WTF::LoggedRefCounted<T>*) [with T = WebCore::MediaSourceClientGStreamerMSE]: this=0x673c07a4, refCount 1
    void WTF::LoggedRefCounted<T>::deref() [with T = WebCore::MediaSourceClientGStreamerMSE]: this=0x673c07a4, refCount 1 --> ...
    void WTF::LoggedRefCounted<T>::deref() [with T = WebCore::MediaSourceClientGStreamerMSE]: this=0x673c07a4, refCount 1 --> ...
    ^^^ Two recursive derefs, not good either.
    #include "Logging.h"
    
    namespace WTF {
    
    template<typename T> class LoggedRefCounted : public WTF::RefCounted<T> {
        WTF_MAKE_NONCOPYABLE(LoggedRefCounted); WTF_MAKE_FAST_ALLOCATED;
    public:
        void ref() {
            printf("%s: this=%p, refCount %d --> ...\n", __PRETTY_FUNCTION__, this, WTF::RefCounted<T>::refCount()); fflush(stdout);
            WTF::RefCounted<T>::ref();
            printf("%s: this=%p, refCount ... --> %d\n", __PRETTY_FUNCTION__, this, WTF::RefCounted<T>::refCount()); fflush(stdout);
        }
    
        void deref() {
            printf("%s: this=%p, refCount %d --> ...\n", __PRETTY_FUNCTION__, this, WTF::RefCounted<T>::refCount()); fflush(stdout);
            WTF::RefCounted<T>::deref();
            printf("%s: this=%p, refCount ... --> %d\n", __PRETTY_FUNCTION__, this, WTF::RefCounted<T>::refCount()); fflush(stdout);
        }
    
    protected:
        LoggedRefCounted() { }
        ~LoggedRefCounted() { }
    };
    
    template<typename T> inline void adopted(WTF::LoggedRefCounted<T>* object)
    {
        printf("%s: this=%p, refCount %d\n", __PRETTY_FUNCTION__, object, (object)?object->refCount():0); fflush(stdout);
        adopted(static_cast<RefCountedBase*>(object));
    }
    
    } // Namespace WTF

    Pause WebProcess on launch

    WebProcessMainGtk and WebProcessMainWPE will sleep for 30 seconds if a special environment variable is defined:

    export WEBKIT2_PAUSE_WEB_PROCESS_ON_LAUNCH=1

    It only works #if ENABLE(DEVELOPER_MODE), so you might want to remove those ifdefs if you’re building in Release mode.

    Log tracers

    In big pipelines (e.g. playbin) it can be very hard to find what element is replying to a query or handling an event. Even using gdb can be extremely tedious due to the very high level of recursion. My coworker Alicia commented that using log tracers is more helpful in this case.

    GST_TRACERS=log enables additional GST_TRACE() calls all accross GStreamer. The following example logs entries and exits into the query function.

    GST_TRACERS=log GST_DEBUG='query:TRACE'

    The names of the logging categories are somewhat inconsistent:

    • log (the log tracer itself)
    • GST_BUFFER
    • GST_BUFFER_LIST
    • GST_EVENT
    • GST_MESSAGE
    • GST_STATES
    • GST_PADS
    • GST_ELEMENT_PADS
    • GST_ELEMENT_FACTORY
    • query
    • bin

    The log tracer code is in subprojects/gstreamer/plugins/tracers/gstlog.c.

    by eocanha at May 11, 2021 06:00 AM

    May 04, 2021

    Enrique Ocaña GonzálezGStreamer WebKit debugging by instrumenting source code (2/3)

    In this post I show some more useful debugging tricks. Check also the other posts of the series:

    Print current thread id

    The thread id is generated by Linux and can take values higher than 1-9, just like PIDs. This thread number is useful to know which function calls are issued by the same thread, avoiding confusion between threads.

    #include <stdio.h>
    #include <unistd.h>
    #include <sys/syscall.h>
    
    printf("%s [%d]\n", __PRETTY_FUNCTION__, syscall(SYS_gettid));
    fflush(stdout);

    Debug GStreamer thread locks

    We redefine the GST_OBJECT_LOCK/UNLOCK/TRYLOCK macros to print the calls, compare locks against unlocks, and see who’s not releasing its lock:

    #include "wtf/Threading.h"
    #define GST_OBJECT_LOCK(obj) do { \
      printf("### [LOCK] %s [%p]\n", __PRETTY_FUNCTION__, &Thread::current()); fflush(stdout); \
      g_mutex_lock(GST_OBJECT_GET_LOCK(obj)); \
    } while (0)
    #define GST_OBJECT_UNLOCK(obj) do { \
      printf("### [UNLOCK] %s [%p]\n", __PRETTY_FUNCTION__, &Thread::current()); fflush(stdout); \
      g_mutex_unlock(GST_OBJECT_GET_LOCK(obj)); \
    } while (0)
    #define GST_OBJECT_TRYLOCK(obj) ({ \
      gboolean result = g_mutex_trylock(GST_OBJECT_GET_LOCK(obj)); \
      if (result) { \
       printf("### [LOCK] %s [%p]\n", __PRETTY_FUNCTION__, &Thread::current()); fflush(stdout); \
      } \
      result; \
    })

    Warning: The statement expression that allows the TRYLOCK macro to return a value will only work on GCC.

    There’s a way to know which thread has taken a lock in glib/GStreamer using gdb. First locate the stalled thread:

    (gdb) thread 
    (gdb) bt
    #2  0x74f07416 in pthread_mutex_lock ()
    #3  0x7488aec6 in gst_pad_query ()
    #4  0x6debebf2 in autoplug_query_allocation ()
    
    (gdb) frame 3
    #3  0x7488aec6 in gst_pad_query (pad=pad@entry=0x54a9b8, ...)
    4058        GST_PAD_STREAM_LOCK (pad);

    Now get the process id (PID) and use the pthread_mutex_t structure to print the Linux thread id that has acquired the lock:

    (gdb) call getpid()
    $30 = 6321
    (gdb) p ((pthread_mutex_t*)pad.stream_rec_lock.p)->__data.__owner
    $31 = 6368
    (gdb) thread find 6321.6368
    Thread 21 has target id 'Thread 6321.6368'

    Trace function calls (poor developer version)

    If you’re using C++, you can define a tracer class. This is for webkit, but you get the idea:

    #define MYTRACER() MyTracer(__PRETTY_FUNCTION__);
    class MyTracer {
    public:
        MyTracer(const gchar* functionName)
          : m_functionName(functionName) {
          printf("### %s : begin %d\n", m_functionName.utf8().data(), currentThread()); fflush(stdout);
        }
        virtual ~MyTracer() {
            printf("### %s : end %d\n", m_functionName.utf8().data(), currentThread()); fflush(stdout);
        }
    private:
        String m_functionName;
    };

    And use it like this in all the functions you want to trace:

    void somefunction() {
      MYTRACER();
      // Some other code...
    }

    The constructor will log when the execution flow enters into the function and the destructor will log when the flow exits.

    Setting breakpoints from C

    In the C code, just call raise(SIGINT) (simulate CTRL+C, normally the program would finish).

    And then, in a previously attached gdb, after breaking and having debugging all you needed, just continue the execution by ignoring the signal or just plainly continuing:

    (gdb) signal 0
    (gdb) continue

    There’s a way to do the same but attaching gdb after the raise. Use raise(SIGSTOP) instead (simulate CTRL+Z). Then attach gdb, locate the thread calling raise and switch to it:

    (gdb) thread apply all bt
    [now search for "raise" in the terminal log]
    Thread 36 (Thread 1977.2033): #1 0x74f5b3f2 in raise () from /home/enrique/buildroot/output2/staging/lib/libpthread.so.0
    (gdb) thread 36

    Now, from a terminal, send a continuation signal: kill -SIGCONT 1977. Finally instruct gdb to single-step only the current thread (IMPORTANT!) and run some steps until all the raises have been processed:

    (gdb) set scheduler-locking on
    (gdb) next    // Repeat several times...

    Know the name of a GStreamer function stored in a pointer at runtime

    Just use this macro:

    GST_DEBUG_FUNCPTR_NAME(func)

    Detecting memory leaks in WebKit

    RefCountedLeakCounter is a tool class that can help to debug reference leaks by printing this kind of messages when WebKit exits:

      LEAK: 2 XMLHttpRequest
      LEAK: 25 CachedResource
      LEAK: 3820 WebCoreNode

    To use it you have to modify the particular class you want to debug:

    • Include wtf/RefCountedLeakCounter.h
    • DEFINE_DEBUG_ONLY_GLOBAL(WTF::RefCountedLeakCounter, myClassCounter, ("MyClass"));
    • In the constructor: myClassCounter.increment()
    • In the destructor: myClassCounter.decrement()

    by eocanha at May 04, 2021 06:00 AM

    May 03, 2021

    Jean-François Fortin TamIntroducing Regento, marketing for FLOSS-centric companies and transitioning industries

    In this blog post, I’m taking a quick break from my GTG blogging frenzy to talk about another one of my non-software projects from the past few months years (estimated reading time: 3 ½ minutes).


    Some may remember I previously introduced the Atypica collective as a dedicated place where I could showcase some of my video production work (instead of having everything and the kitchen sink into my own website).

    Launching Atypica was a long-standing project of mine that had been put on the back-burner because of some of my intensive CMO work in recent years (such as this, for instance). Awakening from the lucid dream allowed me to re-enter a long R&D phase where I could not only shave a ton of “infrastructure and productivity” yaks (that’s a story for another blog post, including the ridiculous tale of the months of work it took to fix my personal blog), but also realign my business objectives and pick up where I had left off “the last time”, hence the delayed announcement of Atypica’s website launch.

    But Atypica, besides being the kind of initiative that works much better outside a pandemic, is merely one of my creative interests, and therefore part of my broader business orientations.


    Today I’m revealing part two (of potentially many): Regento, the fractional CMO agency regency. You could say we are Riskbreakers, but in the business sense. Because in business, it’s all fun and games until an expensive wyvern loses an eye.

    We break down silos and reduce risk. Although we can dream of being digital nomads some day, this is no Vagrant Story.

    “Why another brand, don’t you already do idéemarque,” you ask? Because, in addition to various practical and technical considerations:

    • This area of work is less about creative services and more about deeply involved strategic marketing and strategic business consulting, rather than what you’d find in a typical tactical agency service offering.
    • It allows me to articulate my specific “CMO for hire” offering in a simplified way, instead of overloading again the scope of my personal homepage
      • Until now, the feedback I received from some of the people who looked at my personal homepage was, “You do too many things, it’s unbelievable… therefore we don’t believe you, or don’t understand what you do”.
      • The thing is, my personal home page is a reflection of who I am as a human, laid bare for anyone who cares to dig… yet visitors landing on that page may all have different interests from each other (for example, some will only care about my open-source work, some will only care about my curriculum, some only care about my businesses, and some weird people might actually be interested in seeing my old illustration works), so it can’t really niche down that much without watering down some of its purpose and personality. As major Kusanagi once said, overspecialization is slow death.
    • Separating Regento from the rest lets me build different kinds of partnerships, the kind that are otherwise not available in a typical agency setting. Different team, different clientèle. And indeed, I am not alone in this endeavour (more on that below).

    Double Regento all the way?
    But what does it mean?! 🌈

    One of the main reasons behind this name is that “regento” is the esperanto word for regent, which matches how I see my services as an “interim CMO for hire”, if you will pardon the royal metaphor here: I take care of your kingdom (business) by structuring and strengthening it until the heir (my replacement) comes of age and takes over at some point in time. This is, I posit, one of the most interesting ways for me to make an impact on the world of business: by helping multiple businesses flourish instead of through one endless reign in Leá Monde one industry.

    When it comes to strategic work, I am very picky about who I surround myself with, so I am super excited to highlight the fact that—as can be seen in the Regento about/team pageI have partnered up with Halle Baksh, a good friend of mine whose business experience and wisdom I have great admiration for; our personalities also have great affinity. She is a wonderful human being (not a rogue A.I. like me) and a kindred spirit, and I look forward to continue working with her on making the world a slightly better place, one client’s business at a time.

    Watch that space!

    Although the Regento initiative has been going on since June 2019, it is only now that I’m announcing its existence (again: yak shavings). The Regento website is pretty embryonic, you could almost say it’s a placeholder, but it’s better to have something than to wait for perfection. The design is not where I would want it to be, the contents are “bare minimum” (there are only two articles published for now, and no case studies and testimonials have been written yet), but “perfect” is the enemy of “good”, and all will come in due time.


    If you are interested in insights about business management & business development best practices, high-tech startups (the good, the bad, the ugly), sustainable businesses, or any other topic you may suggest (send me an email!), you may decide to subscribe to the contents email notification list on Regento’s website. As usual: no spam, no data capitalism, just the convenience of not having to remember to check periodically “manually” for updates. And we always enjoy knowing that whatever we write gets read.

    by Jeff at May 03, 2021 02:36 AM

    April 28, 2021

    Nirbheek ChauhanGStreamer has grown a WebRTC implementation

    In other news, GStreamer is now almost buzzword-compliant! The next blog post on our list: blockchains and smart contracts in GStreamer.

    Late last year, we at Centricular announced a new implementation of WebRTC in GStreamer.  Today we're happy to announce that after community review, that work has been merged into GStreamer itself! The plugin is called webrtcbin, and the library is, naturally, called gstwebrtc.

    The implementation has all the basic features, is transparently compatible with other WebRTC stacks (particularly in browsers), and has been well-tested with both Firefox and Chrome.

    Some of the more advanced features such as FEC are already a work in progress, and others will be too—if you want them to be! Hop onto IRC on #gstreamer @ Freenode.net or join the mailing list.

    How do I use it?


    Currently, the easiest way to use webrtcbin is to build GStreamer using either gst-uninstalled (Linux and macOS) or Cerbero (Windows, iOS, Android). If you're a patient person, you can follow @gstreamer and wait for GStreamer 1.14 to be released which will include Windows, macOS, iOS, and Android binaries.

    The API currently lacks documentation, so the best way to learn it is to dive into the source-tree examples. Help on this will be most appreciated! To see how to use GStreamer to do WebRTC with a browser, checkout the bidirectional audio-video demos.

    Show me the code! [skip]


    Here's a quick highlight of the important bits that should get you started if you already know how GStreamer works. This example is in C, but GStreamer also has bindings for Rust, Python, Java, C#, Vala, and so on.

    Let's say you want to capture video from V4L2, stream it to a webrtc peer, and receive video back from it. The first step is the streaming pipeline, which will look something like this:

    v4l2src ! queue ! vp8enc ! rtpvp8pay !
    application/x-rtp,media=video,encoding-name=VP8,payload=96 !
    webrtcbin name=sendrecv

    As a short-cut, let's parse the string description to create the pipeline.

    1
    2
    3
    4
    5
    GstElement *pipe;

    pipe = gst_parse_launch ("v4l2src ! queue ! vp8enc ! rtpvp8pay ! "
    "application/x-rtp,media=video,encoding-name=VP8,payload=96 !"
    " webrtcbin name=sendrecv", NULL);

    Next, we get a reference to the webrtcbin element and attach some callbacks to it.

     1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    GstElement *webrtc;

    webrtc = gst_bin_get_by_name (GST_BIN (pipe), "sendrecv");
    g_assert (webrtc != NULL);

    /* This is the gstwebrtc entry point where we create the offer.
    * It will be called when the pipeline goes to PLAYING. */
    g_signal_connect (webrtc, "on-negotiation-needed",
    G_CALLBACK (on_negotiation_needed), NULL);
    /* We will transmit this ICE candidate to the remote using some
    * signalling. Incoming ICE candidates from the remote need to be
    * added by us too. */
    g_signal_connect (webrtc, "on-ice-candidate",
    G_CALLBACK (send_ice_candidate_message), NULL);
    /* Incoming streams will be exposed via this signal */
    g_signal_connect (webrtc, "pad-added",
    G_CALLBACK (on_incoming_stream), pipe);
    /* Lifetime is the same as the pipeline itself */
    gst_object_unref (webrtc);

    When the pipeline goes to PLAYING, the on_negotiation_needed() callback will be called, and we will ask webrtcbin to create an offer which will match the pipeline above.

     1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    static void
    on_negotiation_needed (GstElement * webrtc, gpointer user_data)
    {
    GstPromise *promise;

    promise = gst_promise_new_with_change_func (on_offer_created,
    user_data, NULL);
    g_signal_emit_by_name (webrtc, "create-offer", NULL,
    promise);
    }

    When webrtcbin has created the offer, it will call on_offer_created()

     1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    static void
    on_offer_created (GstPromise * promise, GstElement * webrtc)
    {
    GstWebRTCSessionDescription *offer = NULL;
    const GstStructure *reply;
    gchar *desc;

    reply = gst_promise_get_reply (promise);
    gst_structure_get (reply, "offer",
    GST_TYPE_WEBRTC_SESSION_DESCRIPTION,
    &offer, NULL);
    gst_promise_unref (promise);

    /* We can edit this offer before setting and sending */
    g_signal_emit_by_name (webrtc,
    "set-local-description", offer, NULL);

    /* Implement this and send offer to peer using signalling */
    send_sdp_offer (offer);
    gst_webrtc_session_description_free (offer);
    }

    Similarly, when we have the SDP answer from the remote, we must call "set-remote-description" on webrtcbin.

    1
    2
    3
    4
    5
    6
    7
    answer = gst_webrtc_session_description_new (
    GST_WEBRTC_SDP_TYPE_ANSWER, sdp);
    g_assert (answer);

    /* Set remote description on our pipeline */
    g_signal_emit_by_name (webrtc, "set-remote-description",
    answer, NULL);

    ICE handling is very similar; when the "on-ice-candidate" signal is emitted, we get a local ICE candidate which we must send to the remote. When we have an ICE candidate from the remote, we must call "add-ice-candidate" on webrtcbin.

    There's just one piece left now; handling incoming streams that are sent by the remote. For that, we have on_incoming_stream() attached to the "pad-added" signal on webrtcbin.

     1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    static void
    on_incoming_stream (GstElement * webrtc, GstPad * pad,
    GstElement * pipe)
    {
    GstElement *play;

    play = gst_parse_bin_from_description (
    "queue ! vp8dec ! videoconvert ! autovideosink",
    TRUE, NULL);
    gst_bin_add (GST_BIN (pipe), play);

    /* Start displaying video */
    gst_element_sync_state_with_parent (play);
    gst_element_link (webrtc, play);
    }

    That's it! This is what a basic webrtc workflow looks like. Those of you that have used the PeerConnection API before will be happy to see that this maps to that quite closely.

    The aforementioned demos also include a Websocket signalling server and JS browser components, and I will be doing an in-depth application newbie developer's guide at a later time, so you can follow me @nirbheek to hear when it comes out!

    Tell me more!


    The code is already being used in production in a number of places, such as EasyMile's autonomous vehicles, and we're excited to see where else the community can take it.

    If you're wondering why we decided a new implementation was needed, read on! For a more detailed discussion into that, you should watch Matthew Waters' talk from the GStreamer conference last year. It's a great companion for this article!

    But before we can dig into details, we need to lay some foundations first.

    What is GStreamer, and what is WebRTC? [skip]


    GStreamer is a cross-platform open-source multimedia framework that is, in my opinion, the easiest and most flexible way to implement any application that needs to play, record, or transform media-like data across an extremely versatile scale of devices and products. Embedded (IoT, IVI, phones, TVs, …), desktop (video/music players, video recording, non-linear editing, videoconferencing and VoIP clients, browsers …), to servers (encode/transcode farms, video/voice conferencing servers, …) and more.

    But what I like the most about GStreamer is the pipeline-based model which solves one of the hardest problems in API design: catering to applications of varying complexity; from the simplest one-liners and quick solutions to those that need several hundreds of thousands of lines of code to implement their full featureset. 

    If you want to learn more about GStreamer, Jan Schmidt's tutorial from Linux.conf.au is a good start.

    WebRTC is a set of draft specifications that build upon existing RTP, RTCP, SDP, DTLS, ICE (and many other) real-time communication specifications and defines an API for making RTC accessible using browser JS APIs.

    People have been doing real-time communication over IP for decades with the previously-listed protocols that WebRTC builds upon. The real innovation of WebRTC was creating a bridge between native applications and webapps by defining a standard, yet flexible, API that browsers can expose to untrusted JavaScript code.

    These specifications are constantly being improved upon, which combined with the ubiquitous nature of browsers means WebRTC is fast becoming the standard choice for videoconferencing on all platforms and for most applications.

    Everything is great, let's build amazing apps! [skip]


    Not so fast, there's more to the story! For WebApps, the PeerConnection API is everywhere. There are some browser-specific quirks as usual, and the API itself keeps changing, but the WebRTC JS adapter handles most of that. Overall the WebApp experience is mostly 👍.

    Sadly, for native code or applications that need more flexibility than a sandboxed JS app can achieve, there haven't been a lot of great options.

    libwebrtc (Chrome's implementation), Janus, Kurento, and OpenWebRTC have traditionally been the main contenders, but after having worked with all of these, we found that each implementation has its own inflexibilities, shortcomings, and constraints.

    libwebrtc is still the most mature implementation, but it is also the most difficult to work with. Since it's embedded inside Chrome, it's a moving target, the API can be hard to work with, and the project is quite difficult to build and integrate, all of which are obstacles in the way of native or server app developers trying to quickly prototype and try out things.

    It was also not built for multimedia use-cases, so while the webrtc bits are great, the lower layers get in the way of non-browser use-cases and applications. It is quite painful to do anything other than the default "set raw media, transmit" and "receive from remote, get raw media". This means that if you want to use your own filters, or hardware-specific codecs or sinks/sources, you end up having to fork libwebrtc.

    In contrast, as shown above, our implementation gives you full control over this as with any other GStreamer pipeline.

    OpenWebRTC by Ericsson was the first attempt to rectify this situation, and it was built on top of GStreamer. The target audience was app developers, and it fit the bill quite well as a proof-of-concept—even though it used a custom API and some of the architectural decisions made it quite inflexible for most other use-cases.

    However, after an initial flurry of activity around the project, momentum petered out, the project failed to gather a community around itself, and is now effectively dead.

    Full disclosure: we worked with Ericsson to polish some of the rough edges around the project immediately prior to its public release.

    WebRTC in GStreamer — webrtcbin and gstwebrtc


    Remember how I said the WebRTC standards build upon existing standards and protocols? As it so happens, GStreamer has supported almost all of them for a while now because they were being used for real-time communication, live streaming, and in many other IP-based applications. Indeed, that's partly why Ericsson chose it as the base for OWRTC.

    This combined with the SRTP and DTLS plugins that were written during OWRTC's development meant that our implementation is built upon a solid and well-tested base, and that implementing WebRTC features is not as difficult as one might presume. However, WebRTC is a large collection of standards, and reaching feature-parity with libwebrtc is an ongoing task.

    Lucky for us, Matthew made some excellent decisions while architecting the internals of webrtcbin, and we follow the PeerConnection specification quite closely, so almost all the missing features involve writing code that would plug into clearly-defined sockets.

    We believe what we've been building here is the most flexible, versatile, and easy to use WebRTC implementation out there, and it can only get better as time goes by. Bringing the power of pipeline-based multimedia manipulation to WebRTC opens new doors for interesting, unique, and highly efficient applications.

    To demonstrate this, in the near future we will be publishing articles that dive into how to use the PeerConnection-inspired API exposed by webrtcbin to build various kinds of applications—starting with a CPU-efficient multi-party bidirectional conferencing solution with a mesh topology that can work with any webrtc stack.

    Until next time!

    by Nirbheek (noreply@blogger.com) at April 28, 2021 03:09 PM

    April 27, 2021

    Enrique Ocaña GonzálezGStreamer WebKit debugging by instrumenting source code (1/3)

    This is the continuation of the GStreamer WebKit debugging tricks post series. In the next three posts, I’ll focus on what we can get by doing some little changes to the source code for debugging purposes (known as “instrumenting”), but before, you might want to check the previous posts of the series:

    Know all the env vars read by a program by using LD_PRELOAD to intercept libc calls

    // File getenv.c
    // To compile: gcc -shared -Wall -fPIC -o getenv.so getenv.c -ldl
    // To use: export LD_PRELOAD="./getenv.so", then run any program you want
    // See http://www.catonmat.net/blog/simple-ld-preload-tutorial-part-2/
    
    #define _GNU_SOURCE
    
    #include <stdio.h>
    #include <dlfcn.h>
    
    // This function will take the place of the original getenv() in libc
    char *getenv(const char *name) {
     printf("Calling getenv(\"%s\")\n", name);
    
     char *(*original_getenv)(const char*);
     original_getenv = dlsym(RTLD_NEXT, "getenv");
    
     return (*original_getenv)(name);
    }

    See the breakpoints with command example to know how to get the same using gdb. Check also Zan’s libpine for more features.

    Track lifetime of GObjects by LD_PRELOADing gobject-list

    The gobject-list project, written by Thibault Saunier, is a simple LD_PRELOAD library for tracking the lifetime of GObjects. When loaded into an application, it prints a list of living GObjects on exiting the application (unless the application crashes), and also prints reference count data when it changes. SIGUSR1 or SIGUSR2 can be sent to the application to trigger printing of more information.

    Overriding the behaviour of a debugging macro

    The usual debugging macros aren’t printing messages? Redefine them to make what you want:

    #undef LOG_MEDIA_MESSAGE
    #define LOG_MEDIA_MESSAGE(...) do { \
      printf("LOG %s: ", __PRETTY_FUNCTION__); \
      printf(__VA_ARGS__); \
      printf("\n"); \
      fflush(stdout); \
    } while(0)

    This can be done to enable asserts on demand in WebKit too:

    #undef ASSERT
    #define ASSERT(assertion) \
      (!(assertion) ? \
          (WTFReportAssertionFailure(__FILE__, __LINE__, WTF_PRETTY_FUNCTION, #assertion), \
           CRASH()) : \
          (void)0)
    
    #undef ASSERT_NOT_REACHED
    #define ASSERT_NOT_REACHED() do { \
      WTFReportAssertionFailure(__FILE__, __LINE__, WTF_PRETTY_FUNCTION, 0); \
      CRASH(); \
    } while (0)

    It may be interesting to enable WebKit LOG() and GStreamer GST_DEBUG() macros only on selected files:

    #define LOG(channel, msg, ...) do { \
      printf("%s: ", #channel); \
      printf(msg, ## __VA_ARGS__); \
      printf("\n"); \
      fflush(stdout); \
    } while (false)
    
    #define _GST_DEBUG(msg, ...) do { \
      printf("### %s: ", __PRETTY_FUNCTION__); \
      printf(msg, ## __VA_ARGS__); \
      printf("\n"); \
      fflush(stdout); \
    } while (false)

    Note all the preprocessor trickery used here:

    • First arguments (channel, msg) are captured intependently
    • The remaining args are captured in __VA_ARGS__
    • do while(false) is a trick to avoid {braces} and make the code block work when used in if/then/else one-liners
    • #channel expands LOG(MyChannel,....) as printf("%s: ", "MyChannel"). It’s called “stringification”.
    • ## __VA_ARGS__ expands the variable argument list as a comma-separated list of items, but if the list is empty, it eats the comma after “msg”, preventing syntax errors

    Print the compile-time type of an expression

    Use typeid(<expression>).name(). Filter the ouput through c++filt -t:

    std::vector<char *> v; 
    printf("Type: %s\n", typeid(v.begin()).name());

    Abusing the compiler to know all the places where a function is called

    If you want to know all the places from where the GstClockTime toGstClockTime(float time) function is called, you can convert it to a template function and use static_assert on a wrong datatype like this (in the .h):

    template <typename T = float> GstClockTime toGstClockTime(float time) { 
      static_assert(std::is_integral<T>::value,
        "Don't call toGstClockTime(float)!");
      return 0;
    }

    Note that T=float is different to integer (is_integral). It has nothing to do with the float time parameter declaration.

    You will get compile-time errors like this on every place the function is used:

    WebKitMediaSourceGStreamer.cpp:474:87:   required from here
    GStreamerUtilities.h:84:43: error: static assertion failed: Don't call toGstClockTime(float)!

    Use pragma message to print values at compile time

    Sometimes is useful to know if a particular define is enabled:

    #include <limits.h>
    
    #define _STR(x) #x
    #define STR(x) _STR(x)
    
    #pragma message "Int max is " STR(INT_MAX)
    
    #ifdef WHATEVER
    #pragma message "Compilation goes by here"
    #else
    #pragma message "Compilation goes by there"
    #endif
    
    ...

    The code above would generate this output:

    test.c:6:9: note: #pragma message: Int max is 0x7fffffff
     #pragma message "Int max is " STR(INT_MAX)
             ^~~~~~~
    test.c:11:9: note: #pragma message: Compilation goes by there
     #pragma message "Compilation goes by there"
             ^~~~~~~

    by eocanha at April 27, 2021 06:00 AM

    April 21, 2021

    Víctor JáquezReview of Igalia Multimedia activities (2020/H2)

    As the first quarter of 2021 has aready come to a close, we reckon it’s time to recap our achievements from the second half of 2020, and update you on the improvements we have been making to the multimedia experience on the Web and Linux in general.

    Our previous reports:

    WPE / WebKitGTK

    We have closed ~100 issues related with multimedia in WebKitGTK/WPE, such as fixed seek issues while playback, plugged memory leaks, gardening tests, improved Flatpak-based developing work-flow, enabled new codecs, etc.. Overall, we improved a bit the multimedia’s user experience on these Webkit engine ports.

    To highlight a couple tasks, we did some maintenance work on WebAudio backends, and we upstreamed an internal audio mixer, keeping only one connection to the audio server, such as PulseAudio, instead of multiple connections, one for every audio resource. The mixer combines all streams into a single audio server connection.

    Adaptive media streaming for the Web (MSE)

    We have been working on a new MSE backend for a while, but along the way many related bugs have appeared and they were squashed. Also many code cleanups has been carried out. Though it has been like yak shaving, we are confident that we will reach the end of this long and winding road soonish.

    DRM media playback for the Web (EME)

    Regarding digital protected media playback, we worked to upstream OpenCDM, support with Widevine, through RDK’s Thunder framework, while continued with the usual maintenance of the others key systems, such as Clear Key, Widevine and PlayReady.

    For more details we published a blog post: Serious Encrypted Media Extensions on GStreamer based WebKit ports.

    Realtime communications for the Web (WebRTC)

    Just as EME, WebRTC is not currently enabled by default in browsers such as Epiphany because license problems, but they are available for custom adopters, and we are maintaining it. For example, we collaborated to upgrade LibWebRTC to M87 and fixed the expected regressions and gardening.

    Along the way we experimented a bit with the new GPUProcess for capture devices, but we decided to stop the experimentation while waiting for a broader adoption of the process, for example in graphics rendering, in WPE/WebKitGTK.

    GPUProcess work will be retaken at some point, because it’s not, currently, a hard requirement, since we already have moved capture devices handling from the UIProcess to the WebProcess, isolating all GStreamer operations in the latter.

    GStreamer

    GStreamer is one of our core multimedia technologies, and we contribute on it on a daily basis. We pushed ~400 commits, with similar number of code reviews, along the second half of 2020. Among of those contributions let us highlight the following list:

    • A lot of bug fixing aiming for release 1.18.
    • Reworked and enhanced decodebin3, the GstTranscoder
      API
      and encodebin.
    • Merged av1parse in video parsers plugin.
    • Merged qroverlay plugin.
    • Iterated on the mono-repo
      proposal, which requires consensus and coordination among the whole community.
    • gstwpe element has been greatly improved from new user requests.
    • Contributed on the new libgstcodecs library, which enables stateless video decoders through different platforms (for example, v4l2, d3d11, va, etc.).
    • Developed a new plugin for VA-API using this library, exposing H.264, H.265, VP9, VP8, MPEG2 decoders and a full featured postprocessor, with better performance, according our measurements, than GStreamer-VAAPI.

    Conferences

    Despite 2020 was not a year for conferences, many of them went virtual. We attended one, the Mile high video conference, and participated in the Slack workspace.

    Thank you for reading this report and stay tuned with our work.

    by vjaquez at April 21, 2021 04:49 AM

    April 20, 2021

    Enrique Ocaña GonzálezGStreamer WebKit debugging tricks using GDB (2/2)

    This post is a continuation of a series of blog posts about the most interesting debugging tricks I’ve found while working on GStreamer WebKit on embedded devices. These are the other posts of the series published so far:

    Print corrupt stacktraces

    In some circumstances you may get stacktraces that eventually stop because of missing symbols or corruption (?? entries).

    #3  0x01b8733c in ?? ()
    Backtrace stopped: previous frame identical to this frame (corrupt stack?)

    However, you can print the stack in a useful way that gives you leads about what was next in the stack:

    • For i386: x/256wa $esp
    • For x86_64: x/256ga $rsp
    • For ARM 32 bit: x/256wa $sp

    You may want to enable asm-demangle: set print asm-demangle

    Example output, the 3 last lines give interesting info:

    0x7ef85550:     0x1b87400       0x2     0x0     0x1b87400
    0x7ef85560:     0x0     0x1b87140       0x1b87140       0x759e88a4
    0x7ef85570:     0x1b87330       0x759c71a9 <gst_base_sink_change_state+956>     0x140c  0x1b87330
    0x7ef85580:     0x759e88a4      0x7ef855b4      0x0     0x7ef855b4
    ...
    0x7ef85830:     0x76dbd6c4 <WebCore::AppendPipeline::resetPipeline()::__PRETTY_FUNCTION__>        0x4     0x3     0x1bfeb50
    0x7ef85840:     0x0     0x76d59268      0x75135374      0x75135374
    0x7ef85850:     0x76dbd6c4 <WebCore::AppendPipeline::resetPipeline()::__PRETTY_FUNCTION__>        0x1b7e300       0x1d651d0       0x75151b74

    More info: 1

    Sometimes the symbol names aren’t printed in the stack memdump. You can do this trick to iterate the stack and print the symbols found there (take with a grain of salt!):

    (gdb) set $i = 0
    (gdb) p/a *((void**)($sp + 4*$i++))
    
    [Press ENTER multiple times to repeat the command]
    
    $46 = 0xb6f9fb17 <_dl_lookup_symbol_x+250>
    $58 = 0xb40a9001 <g_log_writer_standard_streams+128>
    $142 = 0xb40a877b <g_return_if_fail_warning+22>
    $154 = 0xb65a93d5 <WebCore::MediaPlayerPrivateGStreamer::changePipelineState(GstState)+180>
    $164 = 0xb65ab4e5 <WebCore::MediaPlayerPrivateGStreamer::playbackPosition() const+420>
    ...

    Many times it’s just a matter of gdb not having loaded the unstripped version of the library. /proc/<PID>/smaps and info proc mappings can help to locate the library providing the missing symbol. Then we can load it by hand.

    For instance, for this backtrace:

    #0  0x740ad3fc in syscall () from /home/enrique/buildroot-wpe/output/staging/lib/libc.so.6 
    #1  0x74375c44 in g_cond_wait () from /home/enrique/buildroot-wpe/output/staging/usr/lib/libglib-2.0.so.0 
    #2  0x6cfd0d60 in ?? ()

    In a shell, we examine smaps and find out that the unknown piece of code comes from libgstomx:

    $ cat /proc/715/smaps
    ...
    6cfc1000-6cff8000 r-xp 00000000 b3:02 785380     /usr/lib/gstreamer-1.0/libgstomx.so
    ...

    Now we load the unstripped .so in gdb and we’re able to see the new symbol afterwards:

    (gdb) add-symbol-file /home/enrique/buildroot-wpe/output/build/gst-omx-custom/omx/.libs/libgstomx.so 0x6cfc1000
    (gdb) bt
    #0  0x740ad3fc in syscall () from /home/enrique/buildroot-wpe/output/staging/lib/libc.so.6
    #1  0x74375c44 in g_cond_wait () from /home/enrique/buildroot-wpe/output/staging/usr/lib/libglib-2.0.so.0
    #2  0x6cfd0d60 in gst_omx_video_dec_loop (self=0x6e0c8130) at gstomxvideodec.c:1311
    #3  0x6e0c8130 in ?? ()

    Useful script to prepare the add-symbol-file:

    cat /proc/715/smaps | grep '[.]so' | sed -e 's/-[0-9a-f]*//' | { while read ADDR _ _ _ _ LIB; do echo "add-symbol-file $LIB 0x$ADDR"; done; }

    More info: 1

    The “figuring out corrupt ARM stacktraces” post has some additional info about how to use addr2line to translate memory addresses to function names on systems with a hostile debugging environment.

    Debugging a binary without debug symbols

    There are times when there’s just no way to get debug symbols working, or where we’re simply debugging on a release version of the software. In those cases, we must directly debug the assembly code. The gdb text user interface (TUI) can be used to examine the disassebled code and the CPU registers. It can be enabled with these commands:

    layout asm
    layout regs
    set print asm-demangle

    Some useful keybindings in this mode:

    • Arrows: scroll the disassemble window
    • CTRL+p/n: Navigate history (previously done with up/down arrows)
    • CTRL+b/f: Go backward/forward one character (previously left/right arrows)
    • CTRL+d: Delete character (previously “Del” key)
    • CTRL+a/e: Go to the start/end of the line

    This screenshot shows how we can infer that an empty RefPtr is causing a crash in some WebKit code.

    Wake up an unresponsive gdb on ARM

    Sometimes, when you continue (‘c’) execution on ARM there’s no way to stop it again unless a breakpoint is hit. But there’s a trick to retake the control: just send a harmless signal to the process.

    kill -SIGCONT 1234

    Know which GStreamer thread id matches with each gdb thread

    Sometimes you need to match threads in the GStreamer logs with threads in a running gdb session. The simplest way is to ask it to GThread for each gdb thread:

    (gdb) set output-radix 16
    (gdb) thread apply all call g_thread_self()

    This will print a list of gdb threads and GThread*. We only need to find the one we’re looking for.

    Generate a pipeline dump from gdb

    If we have a pointer to the pipeline object, we can call the function that dumps the pipeline:

    (gdb) call gst_debug_bin_to_dot_file_with_ts((GstBin*)0x15f0078, GST_DEBUG_GRAPH_SHOW_ALL, "debug")

    by eocanha at April 20, 2021 06:00 AM

    April 13, 2021

    Enrique Ocaña GonzálezGStreamer WebKit debugging tricks using GDB (1/2)

    I’ve been developing and debugging desktop and mobile applications on embedded devices over the last decade or so. The main part of this period I’ve been focused on the multimedia side of the WebKit ports using GStreamer, an area that is a mix of C (glib, GObject and GStreamer) and C++ (WebKit).

    Over these years I’ve had to work on ARM embedded devices (mobile phones, set-top-boxes, Raspberry Pi using buildroot) where most of the environment aids and tools we take for granted on a regular x86 Linux desktop just aren’t available. In these situations you have to be imaginative and find your own way to get the work done and debug the issues you find in along the way.

    I’ve been writing down the most interesting tricks I’ve found in this journey and I’m sharing them with you in a series of 7 blog posts, one per week. Most of them aren’t mine, and the ones I learnt in the begining of my career can even seem a bit naive, but I find them worth to share anyway. I hope you find them as useful as I do.

    Breakpoints with command

    You can break on a place, run some command and continue execution. Useful to get logs:

    break getenv
    command
     # This disables scroll continue messages
     # and supresses output
     silent
     set pagination off
     p (char*)$r0
    continue
    end
    
    break grl-xml-factory.c:2720 if (data != 0)
    command
     call grl_source_get_id(data->source)
     # $ is the last value in the history, the result of
     # the previous call
     call grl_media_set_source (send_item->media, $)
     call grl_media_serialize_extended (send_item->media, 
      GRL_MEDIA_SERIALIZE_FULL)
     continue
    end

    This idea can be combined with watchpoints and applied to trace reference counting in GObjects and know from which places the refcount is increased and decreased.

    Force execution of an if branch

    Just wait until the if chooses a branch and then jump to the other one:

    6 if (i > 3) {
    (gdb) next
    7 printf("%d > 3\n", i);
    (gdb) break 9
    (gdb) jump 9
    9 printf("%d <= 3\n", i);
    (gdb) next
    5 <= 3

    Debug glib warnings

    If you get a warning message like this:

    W/GLib-GObject(18414): g_object_unref: assertion `G_IS_OBJECT (object)' failed

    the functions involved are: g_return_if_fail_warning(), which calls to g_log(). It’s good to set a breakpoint in any of the two:

    break g_log

    Another method is to export G_DEBUG=fatal_criticals, which will convert all the criticals in crashes, which will stop the debugger.

    Debug GObjects

    If you want to inspect the contents of a GObjects that you have in a reference…

    (gdb) print web_settings 
    $1 = (WebKitWebSettings *) 0x7fffffffd020

    you can dereference it…

    (gdb) print *web_settings
    $2 = {parent_instance = {g_type_instance = {g_class = 0x18}, ref_count = 0, qdata = 0x0}, priv = 0x0}

    even if it’s an untyped gpointer…

    (gdb) print user_data
    (void *) 0x7fffffffd020
    (gdb) print *((WebKitWebSettings *)(user_data))
    {parent_instance = {g_type_instance = {g_class = 0x18}, ref_count = 0, qdata = 0x0}, priv = 0x0}

    To find the type, you can use GType:

    (gdb) call (char*)g_type_name( ((GTypeInstance*)0x70d1b038)->g_class->g_type )
    $86 = 0x2d7e14 "GstOMXH264Dec-omxh264dec"

    Instantiate C++ object from gdb

    (gdb) call malloc(sizeof(std::string))
    $1 = (void *) 0x91a6a0
    (gdb) call ((std::string*)0x91a6a0)->basic_string()
    (gdb) call ((std::string*)0x91a6a0)->assign("Hello, World")
    $2 = (std::basic_string<char, std::char_traits<char>, std::allocator<char> > &) @0x91a6a0: {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x91a6f8 "Hello, World"}}
    (gdb) call SomeFunctionThatTakesAConstStringRef(*(const std::string*)0x91a6a0)

    See: 1 and 2

    by eocanha at April 13, 2021 10:49 AM

    April 12, 2021

    Jan SchmidtRift CV1 – Getting close now…

    (Jan Schmidt)

    It’s been a while since my last post about tracking support for the Oculus Rift in February. There’s been big improvements since then – working really well a lot of the time. It’s gone from “If I don’t make any sudden moves, I can finish an easy Beat Saber level” to “You can’t hide from me!” quality.

    Equally, there are still enough glitches and corner cases that I think I’ll still be at this a while.

    Here’s a video from 3 weeks ago of (not me) playing Beat Saber on Expert+ setting showing just how good things can be now:

    Beat Saber – Skunkynator playing Expert+, Mar 16 2021

    Strap in. Here’s what I’ve worked on in the last 6 weeks:

    Pose Matching improvements

    Most of the biggest improvements have come from improving the computer vision algorithm that’s matching the observed LEDs (blobs) in the camera frames to the 3D models of the devices.

    I split the brute-force search algorithm into 2 phases. It now does a first pass looking for ‘obvious’ matches. In that pass, it does a shallow graph search of blobs and their nearest few neighbours against LEDs and their nearest neighbours, looking for a match using a “Strong” match metric. A match is considered strong if expected LEDs match observed blobs to within 1.5 pixels.

    Coupled with checks on the expected orientation (matching the Gravity vector detected by the IMU) and the pose prior (expected position and orientation are within predicted error bounds) this short-circuit on the search is hit a lot of the time, and often completes within 1 frame duration.

    In the remaining tricky cases, where a deeper graph search is required in order to recover the pose, the initial search reduces the number of LEDs and blobs under consideration, speeding up the remaining search.

    I also added an LED size model to the mix – for a candidate pose, it tries to work out how large (in pixels) each LED should appear, and use that as a bound on matching blobs to LEDs. This helps reduce mismatches as devices move further from the camera.

    LED labelling

    When a brute-force search for pose recovery completes, the system now knows the identity of various blobs in the camera image. One way it avoids a search next time is to transfer the labels into future camera observations using optical-flow tracking on the visible blobs.

    The problem is that even sped-up the search can still take a few frame-durations to complete. Previously LED labels would be transferred from frame to frame as they arrived, but there’s now a unique ID associated with each blob that allows the labels to be transferred even several frames later once their identity is known.

    IMU Gyro scale

    One of the problems with reverse engineering is the guesswork around exactly what different values mean. I was looking into why the controller movement felt “swimmy” under fast motions, and one thing I found was that the interpretation of the gyroscope readings from the IMU was incorrect.

    The touch controllers report IMU angular velocity readings directly as a 16-bit signed integer. Previously the code would take the reading and divide by 1024 and use the value as radians/second.

    From teardowns of the controller, I know the IMU is an Invensense MPU-6500. From the datasheet, the reported value is actually in degrees per second and appears to be configured for the +/- 2000 °/s range. That yields a calculation of Gyro-rad/s = Gyro-°/s * (2000 / 32768) * (?/180) – or a divisor of 938.734.

    The 1024 divisor was under-estimating rotation speed by about 10% – close enough to work until you start moving quickly.

    Limited interpolation

    If we don’t find a device in the camera views, the fusion filter predicts motion using the IMU readings – but that quickly becomes inaccurate. In the worst case, the controllers fly off into the distance. To avoid that, I added a limit of 500ms for ‘coasting’. If we haven’t recovered the device pose by then, the position is frozen in place and only rotation is updated until the cameras find it again.

    Exponential filtering

    I implemented a 1-Euro exponential smoothing filter on the output poses for each device. This is an idea from the Project Esky driver for Project North Star/Deck-X AR headsets, and almost completely eliminates jitter in the headset view and hand controllers shown to the user. The tradeoff is against introducing lag when the user moves quickly – but there are some tunables in the exponential filter to play with for minimising that. For now I’ve picked some values that seem to work reasonably.

    Non-blocking radio

    Communications with the touch controllers happens through USB radio command packets sent to the headset. The main use of radio commands in OpenHMD is to read the JSON configuration block for each controller that is programmed in at the factory. The configuration block provides the 3D model of LED positions as well as initial IMU bias values.

    Unfortunately, reading the configuration block takes a couple of seconds on startup, and blocks everything while it’s happening. Oculus saw that problem and added a checksum in the controller firmware. You can read the checksum first and if it hasn’t changed use a local cache of the configuration block. Eventually, I’ll implement that caching mechanism for OpenHMD but in the meantime it still reads the configuration blocks on each startup.

    As an interim improvement I rewrote the radio communication logic to use a state machine that is checked in the update loop – allowing radio communications to be interleaved without blocking the regularly processing of events. It still interferes a bit, but no longer causes a full multi-second stall as each hand controller turns on.

    Haptic feedback

    The hand controllers have haptic feedback ‘rumble’ motors that really add to the immersiveness of VR by letting you sense collisions with objects. Until now, OpenHMD hasn’t had any support for applications to trigger haptic events. I spent a bit of time looking at USB packet traces with Philipp Zabel and we figured out the radio commands to turn the rumble motors on and off.

    In the Rift CV1, the haptic motors have a mode where you schedule feedback events into a ringbuffer – effectively they operate like a low frequency audio device. However, that mode was removed for the Rift S (and presumably in the Quest devices) – and deprecated for the CV1.

    With that in mind, I aimed for implementing the unbuffered mode, with explicit ‘motor on + frequency + amplitude’ and ‘motor off’ commands sent as needed. Thanks to already having rewritten the radio communications to use a state machine, adding haptic commands was fairly easy.

    The big question mark is around what API OpenHMD should provide for haptic feedback. I’ve implemented something simple for now, to get some discussion going. It works really well and adds hugely to the experience. That code is in the https://github.com/thaytan/OpenHMD/tree/rift-haptics branch, with a SteamVR-OpenHMD branch that uses it in https://github.com/thaytan/SteamVR-OpenHMD/tree/controller-haptics-wip

    Problem areas

    Unexpected tracking losses

    I’d say the biggest problem right now is unexpected tracking loss and incorrect pose extractions when I’m not expecting them. Especially my right controller will suddenly glitch and start jumping around. Looking at a video of the debug feed, it’s not obvious why that’s happening:

    To fix cases like those, I plan to add code to log the raw video feed and the IMU information together so that I can replay the video analysis frame-by-frame and investigate glitches systematically. Those recordings will also work as a regression suite to test future changes.

    Sensor fusion efficiency

    The Kalman filter I have implemented works really nicely – it does the latency compensation, predicts motion and extracts sensor biases all in one place… but it has a big downside of being quite expensive in CPU. The Unscented Kalman Filter CPU cost grows at O(n^3) with the size of the state, and the state in this case is 43 dimensional – 22 base dimensions, and 7 per latency-compensation slot. Running 1000 updates per second for the HMD and 500 for each of the hand controllers adds up quickly.

    At some point, I want to find a better / cheaper approach to the problem that still provides low-latency motion predictions for the user while still providing the same benefits around latency compensation and bias extraction.

    Lens Distortion

    To generate a convincing illusion of objects at a distance in a headset that’s only a few centimetres deep, VR headsets use some interesting optics. The LCD/OLED panels displaying the output get distorted heavily before they hit the users eyes. What the software generates needs to compensate by applying the right inverse distortion to the output video.

    Everyone that tests the CV1 notices that the distortion is not quite correct. As you look around, the world warps and shifts annoyingly. Sooner or later that needs fixing. That’s done by taking photos of calibration patterns through the headset lenses and generating a distortion model.

    Camera / USB failures

    The camera feeds are captured using a custom user-space UVC driver implementation that knows how to set up the special synchronisation settings of the CV1 and DK2 cameras, and then repeatedly schedules isochronous USB packet transfers to receive the video.

    Occasionally, some people experience failure to re-schedule those transfers. The kernel rejects them with an out-of-memory error failing to set aside DMA memory (even though it may have been running fine for quite some time). It’s not clear why that happens – but the end result at the moment is that the USB traffic for that camera dies completely and there’ll be no more tracking from that camera until the application is restarted.

    Often once it starts happening, it will keep happening until the PC is rebooted and the kernel memory state is reset.

    Occluded cases

    Tracking generally works well when the cameras get a clear shot of each device, but there are cases like sighting down the barrel of a gun where we expect that the user will line up the controllers in front of one another, and in front of the headset. In that case, even though we probably have a good idea where each device is, it can be hard to figure out which LEDs belong to which device.

    If we already have a good tracking lock on the devices, I think it should be possible to keep tracking even down to 1 or 2 LEDs being visible – but the pose assessment code will have to be aware that’s what is happening.

    Upstreaming

    April 14th marks 2 years since I first branched off OpenHMD master to start working on CV1 tracking. How hard can it be, I thought? I’ll knock this over in a few months.

    Since then I’ve accumulated over 300 commits on top of OpenHMD master that eventually all need upstreaming in some way.

    One thing people have expressed as a prerequisite for upstreaming is to try and remove the OpenCV dependency. The tracking relies on OpenCV to do camera distortion calculations, and for their PnP implementation. It should be possible to reimplement both of those directly in OpenHMD with a bit of work – possibly using the fast LambdaTwist P3P algorithm that Philipp Zabel wrote, that I’m already using for pose extraction in the brute-force search.

    Others

    I’ve picked the top issues to highlight here. https://github.com/thaytan/OpenHMD/issues has a list of all the other things that are still on the radar for fixing eventually.

    Other Headsets

    At some point soon, I plan to put a pin in the CV1 tracking and look at adapting it to more recent inside-out headsets like the Rift S and WMR headsets. I implemented 3DOF support for the Rift S last year, but getting to full positional tracking for that and other inside-out headsets means implementing a SLAM/VIO tracking algorithm to track the headset position.

    Once the headset is tracking, the code I’m developing here for CV1 to find and track controllers will hopefully transfer across – the difference with inside-out tracking is that the cameras move around with the headset. Finding the controllers in the actual video feed should work much the same.

    Sponsorship

    This development happens mostly in my spare time and partly as open source contribution time at work at Centricular. I am accepting funding through Github Sponsorships to help me spend more time on it – I’d really like to keep helping Linux have top-notch support for VR/AR applications. Big thanks to the people that have helped get this far.

    by thaytan at April 12, 2021 08:29 AM

    April 11, 2021

    Andy Wingoguile's reader, in guile

    (Andy Wingo)

    Good evening! A brief(ish?) note today about some Guile nargery.

    the arc of history

    Like many language implementations that started life when you could turn on the radio and expect to hear Def Leppard, Guile has a bottom half and a top half. The bottom half is written in C and exposes a shared library and an executable, and the top half is written in the language itself (Scheme, in the case of Guile) and somehow loaded by the C code when the language implementation starts.

    Since 2010 or so we have been working at replacing bits written in C with bits written in Scheme. Last week's missive was about replacing the implementation of dynamic-link from using the libltdl library to using Scheme on top of a low-level dlopen wrapper. I've written about rewriting eval in Scheme, and more recently about how the road to getting the performance of C implementations in Scheme has been sometimes long.

    These rewrites have a quixotic aspect to them. I feel something in my gut about rightness and wrongness and I know at a base level that moving from C to Scheme is the right thing. Much of it is completely irrational and can be out of place in a lot of contexts -- like if you have a task to get done for a customer, you need to sit and think about minimal steps from here to the goal and the gut doesn't have much of a role to play in how you get there. But it's nice to have a project where you can do a thing in the way you'd like, and if it takes 10 years, that's fine.

    But besides the ineffable motivations, there are concrete advantages to rewriting something in Scheme. I find Scheme code to be more maintainable, yes, and more secure relative to the common pitfalls of C, obviously. It decreases the amount of work I will have when one day I rewrite Guile's garbage collector. But also, Scheme code gets things that C can't have: tail calls, resumable delimited continuations, run-time instrumentation, and so on.

    Taking delimited continuations as an example, five years ago or so I wrote a lightweight concurrency facility for Guile, modelled on Parallel Concurrent ML. It lets millions of fibers to exist on a system. When a fiber would need to block on an I/O operation (read or write), instead it suspends its continuation, and arranges to restart it when the operation becomes possible.

    A lot had to change in Guile for this to become a reality. Firstly, delimited continuations themselves. Later, a complete rewrite of the top half of the ports facility in Scheme, to allow port operations to suspend and resume. Many of the barriers to resumable fibers were removed, but the Fibers manual still names quite a few.

    Scheme read, in Scheme

    Which brings us to today's note: I just rewrote Guile's reader in Scheme too! The reader is the bit that takes a stream of characters and parses it into S-expressions. It was in C, and now is in Scheme.

    One of the primary motivators for this was to allow read to be suspendable. With this change, read-eval-print loops are now implementable on fibers.

    Another motivation was to finally fix a bug in which Guile couldn't record source locations for some kinds of datums. It used to be that Guile would use a weak-key hash table to associate datums returned from read with source locations. But this only works for fresh values, not for immediate values like small integers or characters, nor does it work for globally unique non-immediates like keywords and symbols. So for these, we just wouldn't have any source locations.

    A robust solution to that problem is to return annotated objects rather than using a side table. Since Scheme's macro expander is already set to work with annotated objects (syntax objects), a new read-syntax interface would do us a treat.

    With read in C, this was hard to do. But with read in Scheme, it was no problem to implement. Adapting the expander to expect source locations inside syntax objects was a bit fiddly, though, and the resulting increase in source location information makes the output files bigger by a few percent -- due somewhat to the increased size of the .debug_lines DWARF data, but also due to serialized source locations for syntax objects in macros.

    Speed-wise, switching to read in Scheme is a regression, currently. The old reader could parse around 15 or 16 megabytes per second when recording source locations on this laptop, or around 22 or 23 MB/s with source locations off. The new one parses more like 10.5 MB/s, or 13.5 MB/s with positions off, when in the old mode where it uses a weak-key side table to record source locations. The new read-syntax runs at around 12 MB/s. We'll be noodling at these in the coming months, but unlike when the original reader was written, at least now the reader is mainly used only at compile time. (It still has a role when reading s-expressions as data, so there is still a reason to make it fast.)

    As is the case with eval, we still have a C version of the reader available for bootstrapping purposes, before the Scheme version is loaded. Happily, with this rewrite I was able to remove all of the cruft from the C reader related to non-default lexical syntax, which simplifies maintenance going forward.

    An interesting aspect of attempting to make a bug-for-bug rewrite is that you find bugs and unexpected behavior. For example, it turns out that since the dawn of time, Guile always read #t and #f without requiring a terminating delimiter, so reading "(#t1)" would result in the list (#t 1). Weird, right? Weirder still, when the #true and #false aliases were added to the language, Guile decided to support them by default, but in an oddly backwards-compatible way... so "(#false1)" reads as (#f 1) but "(#falsa1)" reads as (#f alsa1). Quite a few more things like that.

    All in all it would seem to be a successful rewrite, introducing no new behavior, even producing the same errors. However, this is not the case for backtraces, which can expose the guts of read in cases where that previously wouldn't happen because the C stack was opaque to Scheme. Probably we will simply need to add more sensible error handling around callers to read, as a backtrace isn't a good user-facing error anyway.

    OK enough rambling for this evening. Happy hacking to all and to all a good night!

    by Andy Wingo at April 11, 2021 07:51 PM

    April 08, 2021

    Andy Wingosign of the times

    (Andy Wingo)

    Hello all! There is a mounting backlog of things that landed in Guile recently and to avoid having to eat the whole plate in one bite, I'm going to try to send some shorter missives over the next weeks.

    Today's is about a silly thing, dynamic-link. This interface is dlopen, but "portable". See, back in the day -- like, 1998 -- there were lots of kinds of systems and how to make and load a shared library portably was hard. You'd have people with AIX and Solaris and all kinds of weird compilers and linkers filing bugs on your project if you hard-coded a GNU toolchain invocation when creating loadable extensions, or hard-coded dlopen or similar to use them.

    Libtool provided a solution to create portable loadable libraries, which involved installing .la files alongside the .so files. You could use libtool to link them to a library or an executable, or you could load them at run-time via the libtool-provided libltdl library.

    But, the .la files were a second source of truth, and thus a source of bugs. If a .la file is present, so is an .so file, and you could always just use the .so file directly. For linking against an installed shared library on modern toolchains, the .la files are strictly redundant. Therefore, all GNU/Linux distributions just delete installed .la files -- Fedora, Debian, and even Guix do so.

    Fast-forward to today: there has been a winnowing of platforms, and a widening of the GNU toolchain (in which I include LLVM as well as it has a mostly-compatible interface). The only remaining toolchain flavors are GNU and Windows, from the point of view of creating loadable shared libraries. Whether you use libtool or not to create shared libraries, the result can be loaded either way. And from the user side, dlopen is the universally supported interface, outside of Windows; even Mac OS fixed their implementation a few years back.

    So in Guile we have been in an unstable equilibrium: creating shared libraries by including a probably-useless libtool into the toolchain, and loading them by using a probably-useless libtool-provided libltdl.

    But the use of libltdl has not been without its costs. Because libltdl intends to abstract over different platforms, it encourages you to leave off the extension when loading a library, instead promising to try a platform-specific set such as .so, .dll, .dylib etc as appropriate. In practice the abstraction layer was under-maintained and we always had some problems on Mac OS, for example.

    Worse, as ltdl would search through the path for candidates, it would only report the last error it saw from the underlying dlopen interface. It was almost always the case that if A and B were in the search path, and A/foo.so failed to load because of a missing dependency, the error you would get as a user would instead be "file not found", because ltdl swallowed the first error and kept trucking to try to load B/foo.so which didn't exist.

    In summary, this is a case where the benefits of an abstraction layer decline over time. For a few years now, libltdl hasn't been paying for itself. Libtool is dead, for all intents and purposes (last release in 2015); best to make plans to migrate away, somehow.

    In the case of the dlopen replacement, in Guile we ended up rewriting the functionality in Scheme. The underlying facility is now just plain dlopen, for which we shim a version of dlopen on Windows, inspired by the implementation in cygwin. There are still platform-specific library extensions, but that is handled easily on the Scheme layer.

    Looking forward, I think it's probably time to replace Guile's use of libtool to create its libraries and executables. I loathe the fact that libtool puts shell scripts in the place of executables in build directories and stashes the actual executables elsewhere -- like, visceral revulsion. There is no need for that nowadays. Not sure what to replace it with, nor on what timeline.

    And what about autotools? That, my friends, would be a whole nother blog post. Until then, & probably sooner, happy hacking!

    by Andy Wingo at April 08, 2021 07:09 PM

    April 06, 2021

    Jean-François Fortin Tam“Getting Things GNOME” 0.5 released!

    It is time to welcome a new release of the Rebuild of EvanGTGelion: 0.5, “You Can (Not) Improve Performance”!

    This release of GTG has been 9 months in the making after the groundbreaking 0.4 release. While 0.4 was a major “perfect storm” overhaul, 0.5 is also a very technology-intensive release, even though it was done in a relatively short timeframe comparatively.

    Getting Things GNOME 0.5 brings a truckload of user experience refinements, bugfixes, a completely revamped file format and task editor, and a couple of notable performance improvements. It doesn’t solve every performance problem yet (some remain), but it certainly improves a bunch of them for workaholics like me. If 0.4 felt a bit like a turtle, 0.5 is a definitely a much faster turtle.

    If that was not enough already, it has some killer new features too. It’s the bee’s knees!

    To benefit from one performance improvement in particular, it requires the new version of liblarch, 3.1, that we have released this month. GTG with the latest liblarch is available all-in-one in a Flatpak update near you. 📦

    This release announcement and all that led up to it was, as you can imagine, planned using GTG:

    “Who’s laughing now eh, señor Ruiz?”

    As you can see, I came prepared. So continue reading below for the summary of improvements, I guarantee it’ll be worth it.


    Some brief statistics

    Beyond what is featured in these summarized release notes below, GTG itself has undergone roughly 400 changes affecting over 230 files, and received hundreds of bug fixes, as can be seen here. In fact, through this development cycle, we have crossed the 6,000 commits mark!

    If you need another indication of how active the project has been in the last 9 months, I have received over 950 GTG-related emails since the 0.4 release last July.

    We are thankful to all of our supporters and contributors, past and present, who made GTG 0.5 possible. Check out the “About” dialog for a list of contributors for this release.


    New features

    Recurring Tasks

    Thanks to Mohieddine Drissi and Youssef Toukabri, tasks can now be made recurring (repeating). This allows you to keep reminding yourself that you need to clean your room every Saturday, and then not do it because it’s more fun to spend your whole Saturday fixing GTG bugs!

    • In the task editor, recurrence can be toggled with the dedicated menubutton that shows a nice user interface to manage the recurrence (daily, every other day, weekly, monthly and yearly).
    • Recurring tasks can also be added from the main window, using the “quick add” entry’s syntax.

    Emojis as tag emblems 🦄

    The old tag icons have been replaced by emojis. I decided it would be best for us to switch to emojis instead of icon-based emblems, because:

    • The availability of emblem icons varies a lot from one system to another, and from one icon theme to another;
    • In recent years, emblem icons on Linux have become quite a mismatch of styles, and their numbers in standard themes have also decreased;
    • Emojis are scalable by default, which makes them better for HiDPI displays (while traditional pixmap icons still are a hit-or-miss);
    • There is a much, much, much wider choice of emojis (for every conceivable situation!) than there are emblem icons on typical Linux systems;
    • It’s simply easier to deal with, in the long run. 😌

    Unfortunately, this means that when you upgrade from 0.4 to 0.5, the old emblem icons will disappear and you will need to choose new emblems for your tasks.

    Undead Hamster

    The GTG plugin to interface with Hamster has been brought back to life by Francisco Lavin (issue #114 and pull request #465). I have not personally tested this. I’m still a bit worried about the ferrets rising from their graves.

    Performance improvements
    for power users

    An elegant search trigger
    for a more… civilized age

    The filter-as-you-type global live search in GTG now uses a timeout approach (it waits a fraction of a second until after you stopped typing). This changes many things:

    • No more spamming the database and the UI’s treeview
    • The UI stays much more responsive, and you can see your characters being typed smoothly instead of appearing 15 seconds later
    • It won’t hog your CPU nearly as much
    • You get search results much faster: the results now tend to show up within a second after you stop typing, rather than a linear or exponential amount of time. For example, with 1000 tasks, it is now over 5 times faster (ex: bringing down the search time from 17+ seconds to 3 seconds), and the difference would be even more drastic if you have more tasks than that.

    Optimizations to avoid processing all views all the time

    As a result of the changes proposed in pull request #530 and follow-up PR #547, for issue #402, most operations switching between various representations of your tasks will now be 20% to 200% faster (depending on how your tasks are structured):

    • Faster startup
    • Faster switching between tags (particularly when inside the “Actionable” view)
    • Faster mid-night refreshing
    • It most likely makes the global search feature faster, too

    However, changing between global view modes (Open/Actionable/Closed) is now a bit slower the first time around (but is instantaneous afterwards, until a filtering change occurs).

    It’s a performance tradeoff, but it seems to be worth it. Especially when you have lots of tasks:

    Note that to benefit from these changes, GTG 0.5 depends on the newly released version of liblarch (3.1) we have published this month.

    Faster read/write operations on the data file 🚀

    The switch to LXML as our parser ensures any operations on the local file format now happen instantly (less than a millisecond for a normal/lightweight task lists, and up to 25 milliseconds for heavy-duty 1000-tasks lists like mine). This was originally such a big deal that we thought we’d nickname this release “¿Que parser, amigo?” … but other major improvements kept happening since then, so this release is not just about the pasa!

    Less micro-freezes when focusing task windows

    Mr. Bean waitingPictured: me, waiting for the task editor windows to un-freeze when I focus them.

    Users with a thousand tasks will probably have noticed that GTG 0.4 had a tendency to painfully lock up for a couple of seconds when focusing/defocusing individual task editor windows.

    Sure, “It doesn’t catastrophically lock up in your face multiple times per day anymore” doesn’t sound like a feature in theory, but it is a usability feature in practice.

    Now, even when typing inside the new Task Editor and making lots of changes, it remains butter-smooth.

    Part of it might be because it now uses a timer-based approach to parsing, and part of it is probably due to the code having been refactored heavily and being of higher quality overall. It just tends to be better all around now, and I’ve tried fairly hard to break it during my smoke-testing, especially since a lot of the improvements have also been tied to the major file format redesign that Diego implemented.

    All in all, I’m not sure exactly how it accomplishes better reliability, to be honest, so we’ll just assume Diego secretly is a warlock.

    50% more awesome task editing

    Oh yeah, speaking of the task editor…

    Rewriting the Task Editor’s content area and its natural language parser brought many user-visible improvements in addition to the technological improvements:

    • The new timeout-based parser is faster (because we’re not spamming it like barbarians), more reliable, and now gains the ability to recognize subtasks (lines starting with “-“) live, instead of “after pressing Enter.” This also makes copy-pasting from a dashed list easier, as it will be converted into many subtasks in one go
    • Clicking on a tag inside the task editor’s text selects the tag in the main window
    • Tags’ text highlight color now uses the tag’s chosen emblem color
    • Subtasks now have GTK checkboxes!
    • Completed subtasks show up in normal strikethrough text style
    • Opening a subtask takes care of closing the parent task, and the other way around (opening a parent task closes the subtask window), minimizing the amount of utility windows cluttering your view (and reducing the chances of making conflicting changes)
    • Now supports subheadings, in Markdown format (type # title). Now that it supports some basic rich-text formatting features, even Grumpy Sri will be happy with this release:

    Here is a short demonstration video that lets you see the improved “parsing as you type” engine in action:

    Other user interface improvements

    General bug fixes

    Technological changes

    • Eliminate the pyxdg dependency in favor of directly using GLib
    • dbus-python dependency dropped in favor of Gio (except in the Hamster plugin)
    • New and improved file format (see issue 279, “Better XML format [RFC]“, and my previous spooky blog post):
      • Automatically converts upon first startup, nothing to do manually
      • More consistent
      • Uses a single file instead of three separate ones
      • Uses valid XML
      • Fully documented (see docs/contributors/file format.md)
      • Adds file format versioning to make it future-proof
      • Uses LXML as the parser, which is reportedly 40 times faster (see issue #285)
      • Fundamentally allows the task editor to be a lot more reliable for parsing, as we ditch the XML-in-XML matryoshka approach
      • Solves a few other bugs related to (de)serializing
    • Rewrote the Task Editor’s “TaskView” (content area) and its natural language parser, paving the way for more improvements and increased reliability:
      • The timeout-based approach mentioned earlier means that parsing happens less often, “when it needs to”, and is also less error-prone as a result.
      • With the implementation of subheadings, the code now has the foundations for more Markdown features in the future.
      • The taskview supports invisible characters that turn visible when the cursor passes over them, which could be useful for more text tags in the future.
      • Easier to maintain improved performance in the future: When running in debug mode (launch.sh -d), the process function now prints the time it took to process the buffer. This will help us stay conscious of the performance.
      • Support for linking between tasks with gtg://[TASK_ID] (no UI for this yet, though)
      • Better code overall, a lot more extensible and easier to work on
    • Rewrote the Quick Add entry’s parsing and added unit tests
    • Command line parsing ported to GLib, so GLib/GTK-specific options such as --display are now available
    • GTG is now dbus-activatable. Although no API that other applications can use has been implemented yet (outside of “standard” GTK ones), this opens up the possibility in the future, such as the GNOME Shell search integration.

    Developer Experience improvements

    Want to be part of a great team? Join BAHRAM GTG!

    Here are some changes to primarily benefit users crazy enough to be called developers.

    • New plugin: “Developer Console”.
      Play with GTG’s internals from the comfort of GTG itself! Remember: when exiting your Orbital Frame, wear… a helmet.
    • New parameter in launch.sh: -w to enable warnings
      (in case you still are not wearing a helmet, or want to catch more bugs)
    • Better help message for launch.sh, documenting the available options
    • Include Git commit version in the about window
    • Use the proper exit code when the application quits cleanly
    • We closed all the tickets on LaunchPad (over 300 of them) so there is no confusion when it comes to reporting bugs and the current state of the app’s quality and priorities, and to ensure that issues are reported in the currently used bug tracker where development actually happens.

    Help out Diego!

    It is no secret that Diego has been spending a lot of time and effort making this release happen, just like the 0.4 release. The amount and depth of his changes are quite amazing, so it’s no surprise he’s the top contributor here:

    If you would like to show your appreciation and support his maintainership work with some direct monthly donations to him, please consider giving a couple of dollars per month on his gumroad page or his liberapay page!

    More coming in the next release

    Many new features, plugins, and big code changes have been held back from merging so that we could release 0.5 faster, so a bunch of other changes will land in the development version on the way to 0.6.

    “This blog post is ridiculously long! And we still have 10 pull requests almost ready to merge!”
    — Diego, five minutes before I published this blog post

    Spreading this announcement

    We have made some social postings on Twitter, on Mastodon and on LinkedIn that you can re-share/retweet/boost. Please feel free to link to this announcement on forums and blogs as well!

    by Jeff at April 06, 2021 02:00 PM

    March 25, 2021

    Andy Wingohere we go again

    (Andy Wingo)

    Around 18 months ago, Richard Stallman was forced to resign from the Free Software Foundation board of directors and as president. It could have been anything -- at that point he already had a history of behaving in a way that was particularly alienating to women -- but in the end it was his insinuation that it was somehow OK if his recently-deceased mentor Marvin Minsky, then in his 70s or 80s, had sex with a 17-year-old on Jeffrey Epstein's private island. A weird pick of hill to stake one's reputation on, to say the least.

    At the time I was relieved that we would finally be getting some leadership renewal at the FSF, and hopeful that we could get some mission renewal as well. I was also looking forward to the practical implications would be for the GNU project, as more people agreed that GNU was about software freedom and not about its founder.

    But now we're back! Not only has RMS continued through this whole time to insist that he runs the GNU project -- something that is simply not the case, in my estimation -- but this week, a majority of a small self-selected group of people, essentially a subset of current and former members of the FSF board of directors and including RMS himself, elected to reinstate RMS to the board of the Free Software Foundation. Um... read the room, FSF voting members? What kind of message are you sending?

    In this context I can only agree with the calls for the entire FSF board to resign. The board is clearly not fit for purpose, if it can make choices like this.

    dissociation?

    I haven't (yet?) signed the open letter because I would be in an inconsistent position if I did so. The letter enjoins people to "refuse to contribute to projects related to the FSF and RMS"; as a co-maintainer of GNU Guile, which has its origins in the heady 1990s of the FSF but has nothing to do any more with RMS, but whose copyrights are entirely held by the FSF, is hosted on FSF-run servers, and is even obliged (GPLv3 §5d, as referenced by LGPLv3) to print out Copyright (C) 1995-2021 Free Software Foundation, Inc. when it starts, I must admit that I contribute to a project that is "related to the FSF". But I don't see how Guile could continue this association, if the FSF board continues as it is. It's bad for contributors and for the future of the project.

    It would be very tricky to disentangle Guile from the FSF -- consider hosting, for example -- so it's not the work of a day, but it's something to think about.

    Of course I would rather that the FSF wouldn't allow itself to be seen as an essentially misogynist organization. So clean house, FSF!

    on the nature of fire

    Reflecting on how specifically we could have gotten here -- I don't know. I don't know the set of voting members at the FSF, what discussions were made, who voted what. But, having worked as a volunteer on GNU projects for almost two decades now, I have a guess. RMS and his closest supporters see themselves as guardians of the flame of free software -- a lost world of the late 70s MIT AI lab, reborn in a flurry of mid-80s hack, but since 25 years or so, slipping further and further away. These are dark times, in their view, and having the principled founder in a leadership role can only be a good thing.

    (Of course, the environment in the AI lab was only good for some. The treatment of Margaret Hamilton as recounted in Levy's Hackers shows that not all were welcome. If this were just one story, I would discount it, but looking back, it does seem to be part of a pattern.)

    But is that what the FSF is for today? If so, Guile should certainly leave. I'm not here for software as perfomative nostalgia -- I'm here to have fun with friends and start a fire. The FSF should look to do the same -- look at the world we are in, look where the energy is now, and engage in real conversations about success and failure and tactics. There is a world to win and doubling down on RMS won't get us there from here.

    by Andy Wingo at March 25, 2021 12:22 PM

    March 20, 2021

    Sebastian Pölsterlscikit-survival 0.15 Released

    I am proud to announce the release if version 0.15.0 of scikit-survival, which brings support for scikit-learn 0.24 and Python 3.9. Moreover, if you fit a gradient boosting model with loss='coxph', you can now predict the survival and cumulative hazard function using the predict_cumulative_hazard_function and predict_survival_function methods.

    The other enhancement is that cumulative_dynamic_auc now supports evaluating time-dependent predictions. For instance, you can now evaluate the predicted time-dependent risk of a RandomSurvivalForest rather than just evaluating the predicted total number of events per instance, which is what RandomSurvivalForest.predict returns.

    All you have to do is create an array where the columns are the predictions at the time points you want to evaluate. The snippet below summarizes the idea:

    from sksurv.ensemble import RandomSurvivalForest
    from sksurv.metrics import cumulative_dynamic_auc
    rsf = RandomSurvivalForest()
    rsf.fit(X_train, y_train)
    chf_funcs = rsf.predict_cumulative_hazard_function(X_test)
    time_points = np.array([30, 60, …])
    risk_scores = np.row_stack([
    chf(time_points) for chf in chf_funcs
    ])
    aucs, mean_auc = cumulative_dynamic_auc(
    y_train, y_test, risk_scores, time_points
    )
    

    For a complete example, please have a look at the User Guide.

    If you want to know about all changes in scikit-survival 0.15.0, please see the release notes.

    As usual, pre-built conda packages are available for Linux, macOS, and Windows via

     conda install -c sebp scikit-survival
    

    Alternatively, scikit-survival can be installed from source following these instructions.

    March 20, 2021 07:58 PM

    March 15, 2021

    Christian SchallerWhat to look for in Fedora Workstation 34

    (Christian Schaller)

    As we are heading towards April and the release of Fedora Workstation 34 I wanted to post an update on what we are working on for this release and what we are looking at going forward. 2020 was a year where we focused a lot on polishing what we had and getting things past the finish line and Fedora Workstation 34 is going to be the culmination of that effort in many ways.

    Wayland:
    The big ticket item we have wanted to close off on was Wayland, because while Wayland has been production ready for most of us for a while, there was still some cases it didn’t cover as well as X.org. The biggest of this was of course the lack of accelerated XWayland support with the binary NVidia driver. Fixing that issue of course wasn’t something we could do ourselves, but we have been working diligently with our friends at NVidia to help ensure everything was in place for them to enable that support in their driver, so I have been very happy to see the public reports confirming that NVidia will have accelerated 3D in the summer release of their driver. The other Wayland area we have put a lot of effort into has been the work undertaken by Jonas Ådahl to get headless display support working with Wayland. This is a critical feature for people who for instance want a desktop instance on their servers or in the cloud, who want a desktop they access through things like VNC or RDP to use for sysadmin related tasks. Jonas spent a lot of time laying the groundwork for this over the course of last year and we are now in the final stages of merging the patches to enable this feature in GNOME and Wayland in preparation for Fedora Workstation 34. Once those two items are out we consider our Wayland rampup/rollout to be complete, so while there of course will continue to be bugfixes and new features implemented, that will be part of a natural evolution of Wayland and not part of a ‘close gaps with X11’ effort like now.

    PipeWire
    Another big ticket item we are hoping to release fully in Fedora Workstation 34 is PipeWire. PipeWire as most of you know is the engine we use to deal with handling video streams in a secure and shareable away in Fedora Workstation, so when you interact with your web camera(s) or do screen casting or make screenshots it is all routed and handled by PipeWire. But in Fedora Workstation 34 we are aiming to also switch to using PipeWire for audio, to replace both PulseAudio and Jack. For those of you who had read any previous blog post from me you will know what an important step forward this will be as we would finally be making the pro-audio community first class citizens in Fedora Workstation and Linux in general. When we decided to try to switch to PipeWire in Fedora Workstation 34 I have to admit I was a little skeptical about if we would be able to get all things ready in time as there are so many things that needs to be tested and fixed when you switch out such a critical component. Up to that point we had a lot of people interested in PipeWire, but only limited community involvement, but I feel the announcement of bringing in PipeWire for Fedora Workstation 34 galvanized the community around the project and we now have a very active community around PipeWire in #pipewire on the freenode IRC network. Not only is Wim Taymans getting a ton of help with testing and verification, but we also see a stead stream of patches coming in, with for instance improved Bluetooth audio support being contributed, in fact I believe that PipeWire will be able to usher in better bluetooth audio support in Fedora than we ever had before, with great support for high quality Bluetooth audio codecs like LDAC.

    I am especially happy to see so many of the key members of the pro-audio Linux community taking part in this effort and is of course also happy to see many pro-audio folks testing Fedora Workstation for the first time due to this effort. The community is working closely with Wim to test and verify as many important ProAudio applications as possible and work to update Fedora packaging as needed to ensure they can transition from Jack to PipeWire without dependency conflicts or issues. One last item to mention here is that you might have seen that Red Hat is getting into the automotive space, I can’t share a lot of details about that effort, but one thing I can say is that PipeWire will be a core part of it and thus we will soon be looking to hire more engineers to work on PipeWire, so if that is of interest to you be sure to track my twitter feed or blog as I will announce our job openings there as they become available. For the community at large this should be great too as it means that we can get a lot of synergy between automotive and the desktop around audio and video handling.

    It is still somewhat of an open question if we end up actually switching to PipeWire in Fedora Workstation 34, but things are looking good at this point in time and worst case scenario it will be in place for Fedora Workstation 35.

    Toolbox
    Toolbox is another effort that is in a great spot now. Toolbox is our tool for making working with pet containers a breeze for developers. The initial version was prototyped quickly by writing it as a shell script, but we spent time last year getting it rewritten in Go in order to make it possible to keep expanding the project and allow us to implement all the things we envision for it. With that done feature work is now in focus again and Ondřej Michal has done some great work making it possible to set up RHEL containers in Toolbox. This means that you can run Fedora on your laptop and get the latest and greatest features that way, but you can do your development in a RHEL pet container, so you get an environment identical to what you applications will see once they are deployed into the cloud or onto company hardware. This gives you the best of both worlds in my opinion, the fast moving Fedora Workstation that brings the most out of our laptop and desktop hardware, but still easy access to the RHEL platform for development and testing. You can even test this today on Fedora Workstation 33, just open a terminal and type ‘toolbox create --distro rhel --release 8.3‘. The resulting toolbox will then be based on RHEL and not Fedora and thus perfect for doing RHEL targeted development. You will need to use the subscription-manager tool to register it (be sure to register on developer.redhat.com for your free RHEL developer subscription. Over time we hope to integrate this into GNOME Online accounts like we do for the RHEL virtual machines you can set up with GNOME Boxes, so that once you set up your RHEL account you can create RHEL virtual machines and RHEL containers easily on Fedora Workstation.

    Toolbox with RHEL

    Toolbox pet container with RHEL UBI

    Flatpak
    Owen Taylor has been doing some incredible work behind the scenes for the last year trying to ensure the infrastructure we have in RHEL and Fedora provides a great integrated Flatpak experience. As we move forward we expect Flatpaks to be the primary packaging format that Fedora users consume their applications in, but to make that a reality we needed to ensure the experience is good both for Fedora maintainers and for the end users. So one of the big ticket items Owen been working on is getting incremental updates working in Fedora. If you have used applications from Flathub you probably noticed that their updates are small and nimble despite being packaged as Flatpak containers, while the Fedora flatpaks causes big updates each time. The reason for this is that the Fedora flatpaks are shipping as OCI (Open Container Initiative) images, while the Flatpaks on Flathub are shipping as OStree repositories (if you don’t know OStree, think of it as git for binaries). So shipping the Flatpaks as OCI images has advantages in the form of being the same format we at Red Hat use for our kubernetes/docker/openshift containers and thus it allows us to reuse a lot of the work that Red Hat as put into ensuring we can provide and keep such containers up to date and secure, but the downside until now has been that these containers where shipped in a way which cause each update, no matter how small the change, to be a full re-download of the whole image. Well Owen Taylor and Alex Larsson worked together to resolve this and came up with a method to allow incremental updates of such containers and thus bring the update sizes in line with what you see on Flathub for Flatpaks. This should be deployed in time for Fedora Workstation 34 and we also hope to  eventually deploy it for  kubernetes/docker containers too. Finally to make even more applications available we are doing work to enable people to get access to Flathub.org out of the box in Fedora when you enable 3rd party repositories in initial setup, so that your our of the box application selection will be even bigger.

    Flathub frontpage

    Flathub webpage

    GNOME 40
    Another major change in Fedora Workstation 34 is GNOME40 which contains a revamp of the GNOME 3 user interface. This was a collaborative effort between a lot of GNOME 3 stakeholders with Allan Day representing Red Hat. This was also an effort by the GNOME design community to up their game and thus as part of the development process the GNOME Foundation paid a professional company to do user testing on the proposed changes and some of the alternatives. This means that the changes where verified to actually be experienced as an improvement for the experienced GNOME user participants and that it felt intuitive for new users of GNOME. One advantage we have in Fedora is that since we don’t do major tweaking of the GNOME user interface which means once Fedora Workstation 34 ships you are set to enjoy the new GNOME experience from day one. For long time GNOME users I hope and expect that the updates will be a welcome refresh and at the same time that the changes provide a more easy onramp for new GNOME and Fedora Workstation users. Some of the early versions did lead some long term fans of how multimonitor support in GNOME3 worked to be a bit concerned, but be assured that multi monitor is a critical usecase in our opinion and something we have been looking at and will be looking at keep improving. In fact Allan Day wrote a great blog post about GNOME40 multimonitor support recently to explain what we are doing and how we see it evolving going forward.

    Input
    Another area where we keep putting in a lot of effort is input. Thanks to Peter Hutterer and Benjamin Tissoires we keep making sure Fedora Workstation and the world of Linux keeps having access to the newest and best in input. The latest effort they are working on has been to enable haptic touchpads. Haptics touchpads should be familiar among people who tried Apple hardware, but they are expected to appear in force on laptops in general this year, so we have been putting in the effort to ensure that we can support this new type of device as they come out. So if you see laptops you want with haptic touchpads then Fedora Workstation should be ready for it, but of course until these devices are commonplace and we had a chance to test and verify I can make no guarantees.

    Another major effort that we undertook in relation to input was move the GNOME input to a separate thread. Carlos Garnacho worked on this patch to make that happen. This should provide a smoother experience with Fedora Workstation 34 as it means the mouse should not stall due to the main thread running Wayland being busy. This was done as part of the overall performance work we been continuously doing over the last years to ensure to address performance issues and make Fedora and GNOME have the best performance and performance related behaviour possible.

    Lenovo Laptops
    So one of the major announcements of last year was Lenovo Laptops with Fedora Linux pre-installed. There are currently two models available with Linux, the X1 Carbon and the Lenovo P1. Between them they cover the two most commons requests we see, a ultralight weight laptop with the X1 and a more powerful ‘portable workstation’ model with the P1. We are working on a couple of more models and also to get them sold globally, which was key goal of the effort. Be aware that both models are on sale as I am writing this (hopefully still true when you read this), so it is a good time to grab a great laptop with a great OS.

    Lenovo P1

    Lenovo P1

    Vision
    So one thing I wanted to do to is tie the work we do in Fedora Workstation together by articulating what we are trying to achieve. Fedora has for the longest time been the place where Linux as an operating system is evolving and being developed. There are very few major innovations that has come to Linux that hasn’t been developed and incubated in Fedora by Fedora contributors, including of course Red Hat. This include things like the Linux Vendor Firmware Service, Wayland, Flatpak, SilverBlue, PipeWire, SystemD, flicker free boot, HiDPI support, gaming mouse support and so much more. We have always done this work in close cooperation with the upstreams we are collaborating with, which is why the patch delta in any given Fedora release is small. We work hard to get improvements into the upstream kernel and into GNOME and so on right away, to avoid needing to ship downstream patches in Fedora. That of course saves us from having to maintain temporary forks, but more importantly it is the right way to collaborate with an open source community.

    So looking back to when we launched Fedora Workstation we realized that being at the front like that had come at the cost of not being stable and user friendly. So the big question we tried to ask ourselves when launching Fedora Workstation and the question that still drives a lot of our decision making and focus is : how can we preserve being the heart and center of Linux OS development, but at the same time provide end users with a stable and well functioning system? To achieve that we have done lot of changes over the last years, ranging from some policy changes in terms of how and when we brought changes into Fedora, but maybe even more importantly we focused on a lot on figuring out ways to reduce the challenges caused by a rapidly evolving OS, like the introduction of Flatpaks to allow applications to be developed and released without strong ties to the host system libraries and with the concepts we are maturing in Silverblue around image based operating systems or how we are looking at pet container development with Toolbox. All of these things combined remove a lot of the fragility we seen in Linux up to this point and instead let us treat the rapidly evolving linux landscape as a strength.

    So where we are today is that I think we are very close to realizing the vision of being able to let Fedora be the place where exiting new stuff happens, yet at the same time provide the robustness and polish that end users need to be able to use it as their daily driver, it has been my daily driver for many years now and by the rapid growth of users we seen in Fedora over the last 5 years I think that is true for a lot of other people too. The goal is to allow the wider community around Linux, especially the developers, sysadmins and creators relying on Linux to do their work, to come to Fedora and interact and collaborate with the developers working on the OS itself to the benefit of all. You all are probably better judges than me to if we are succeeding with that, but I do take the increased chatter and adoption of Fedora by a lot of people doing Linux related podcasts, news sites and so on as a sign that we are succeeding. And PipeWire for me is a perfect example of how this can look, where we want to bring in the pro-audio creators to Fedora Workstation and let them interact and work closely with Wim Taymans and the PipeWire development team to make the experience even better for themselves and their fellow creators and at the same time give them a great stable platform to create their music on.

    by uraeus at March 15, 2021 06:20 PM

    GStreamerGStreamer 1.18.4 stable bug fix release

    (GStreamer)

    The GStreamer team is pleased to announce another bug fix release in the stable 1.18 release series of your favourite cross-platform multimedia framework!

    This release only contains bugfixes and important security fixes, and it should be safe to update from 1.18.x.

    Highlighted bugfixes:

    • important security fixes for ID3 tag reading, matroska and realmedia parsing, and gst-libav audio decoding
    • audiomixer, audioaggregator: input buffer handling fixes
    • decodebin3: improve stream-selection message handling
    • uridecodebin3: make "caps" property work
    • wavenc: fix writing of INFO chunks in some cases
    • v4l2: bt601 colorimetry, allow encoder resolution changes, fix decoder frame rate negotiation
    • decklinkvideosink: fix auto format detection, and fixes for 29.97fps framerate output
    • mpeg-2 video handling fixes when seeking
    • avviddec: fix bufferpool negotiation and possible memory corruption when changing resolution
    • various stability, performance and reliability improvements
    • memory leak fixes
    • build fixes: rpicamsrc, qt overlay example, d3d11videosink on UWP

    See the GStreamer 1.18.4 release notes for more details.

    Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

    Download tarballs directly here: gstreamer, gst-plugins-base, gst-plugins-good, gst-plugins-ugly, gst-plugins-bad, gst-libav, gst-rtsp-server, gst-python, gst-editing-services, gst-devtools, gstreamer-vaapi, gstreamer-sharp, gst-omx, or gstreamer-docs.

    March 15, 2021 04:00 PM

    March 08, 2021

    Andy Wingo99% spam

    (Andy Wingo)

    Hey all, happy new year apparently! A quick service update on the old wingolog. For some time the site has been drowning in spam comments, despite my best efforts to point a bayesian classifier at the problem.

    I don't keep logs of the number of attempts at posting comments that don't pass the classifier. But what I can say is that since I put in the classifier around 4 years ago, about 2500 comments a year made it through -- enough to turn the comment section into a bit of a dump. Icky, right??

    At the same time of course, that's too many comments to triage manually, so I never got around to fixing the problem. So in fact I had two problems: lots 'o spam, and lots 'o incoming spam.

    With regards to the existing spam, I took a heavyhanded approach. I took a look at all good emails and URLs that people had submitted for comments prior to 2017, assuming they were triaged. Then I made a goodlist of comments since 2017 that had those comments or emails. There were very few of those -- maybe 50 or 70 or so.

    Then I took a look at when comments were made relative to the posts. Turns out, 98.3% of comments were made more than 21 days after their parent post was published -- and sometimes years afterwards. I used that as a first filter, that if a post wasn't from a known poster, and was made 3 weeks or more after the post, I just classified it as spam.

    The list of comments made within 3 weeks of the parent post was small enough for me to triage manually, and there I was able to save a bit of wheat from the chaff. In the end, though, the result of winnowing was less than 1% of what went in.

    As I don't really want to babysit this wobsite, I'll keep this policy in place in the future -- comments will be open for a while after articles are posted. Hopefully that should keep the ol' wingolog in tidy shape going forward, while still permitting people to comment -- something I have really appreciated in the past.

    So happy 2021 to everybody, may the vaccine gods shine upon your shoulders, and happy hacking :)

    by Andy Wingo at March 08, 2021 02:36 PM

    February 27, 2021

    Jean-François Fortin TamA new data format has landed in the upcoming GTG 0.5

    Here’s a general call for testing from your favorite pythonic native Linux desktop personal productivity app, GTG.

    In recent months, Diego tackled the epic task of redesigning the XML file format from a new specification devised with the help of Brent Saner (proposal episodes 1, 2 and 3), and then implementing the new file format in GTG. This work has now been merged to the main development branch on GTG’s git repository:

    Diego’s changes are major, invasive technological changes, and they would benefit from extensive testing by everybody with “real data” before 0.5 happens (very soon). I’ve done some pretty extensive testing & bug reporting in the last few months; Diego fixed all the issues I’ve reported so far, so I’ve pretty much run out of serious bugs now, as only a few remain targetted to the 0.5 milestone… But I’m only human, and it is possible that issues might remain, even after my troll-testing.

    Grab GTG’s git version ASAP, with a copy of your real data (for extra caution, and also because we want you to test with real data); see the instructions in the README, including the “Where is my user data and config stored?” section.

    Please torture-test it to make sure everything is working properly, and report issues you may find (if any). Look for anything that might seem broken “compared to 0.4”, incorrect task parenting/associations, incorrect tagging, broken content, etc.

    If you’ve tried to break it and still couldn’t find any problems, maybe one way to indicate that would be a “👍” on the merge request—I’m not sure we really have another way to know if it turns out that “everything is OK” 🙂

    Your help in testing this (or spreading the word) will help ensure a smooth transition for users getting an upgrade from 0.4 to 0.5, letting us release 0.5 with confidence. Thanks!

    by Jeff at February 27, 2021 11:53 PM