September 16, 2021

Christian SchallerCool happenings in Fedora Workstation land

(Christian Schaller)

Been some time since my last update, so I felt it was time to flex my blog writing muscles again and provide some updates of some of the things we are working on in Fedora in preparation for Fedora Workstation 35. This is not meant to be a comprehensive whats new article about Fedora Workstation 35, more of a listing of some of the things we are doing as part of the Red Hat desktop team.

NVidia support for Wayland
One thing we spent a lot of effort on for a long time now is getting full support for the NVidia binary driver under Wayland. It has been a recurring topic in our bi-weekly calls with the NVidia engineering team ever since we started looking at moving to Wayland. There has been basic binary driver support for some time, meaning you could run a native Wayland session on top of the binary driver, but the critical missing piece was that you could not get support for accelerated graphics when running applications through XWayland, our X.org compatibility layer. Which basically meant that any application requiring 3D support and which wasn’t a native Wayland application yet wouldn’t work. So over the last Months we been having a great collaboration with NVidia around closing this gap, with them working closely with us in fixing issues in their driver while we have been fixing bugs and missing pieces in the rest of the stack. We been reporting and discussing issues back and forth allowing us a very quickly turnaround on issues as we find them which of course all resulted in the NVidia 470.42.01 driver with XWayland support. I am sure we will find new corner cases that needs to be resolved in the coming Months, but I am equally sure we will be able to quickly resolve them due to the close collaboration we have now established with NVidia. And I know some people will wonder why we spent so much time working with NVidia around their binary driver, but the reality is that NVidia is the market leader, especially in the professional Linux workstation space, and there are lot of people who either would end up not using Linux or using Linux with X without it, including a lot of Red Hat customers and Fedora users. And that is what I and my team are here for at the end of the day, to make sure Red Hat customers are able to get their job done using their Linux systems.

Lightweight kiosk mode
One of the wonderful things about open source is the constant flow of code and innovation between all the different parts of the ecosystem. For instance one thing we on the RHEL side have often been asked about over the last few years is a lightweight and simple to use solution for people wanting to run single application setups, like information boards, ATM machines, cash registers, information kiosks and so on. For many use cases people felt that running a full GNOME 3 desktop underneath their application was either to resource hungry and or created a risk that people accidentally end up in the desktop session. At the same time from our viewpoint as a development team we didn’t want a completely separate stack for this use case as that would just increase our maintenance burden as we would end up having to do a lot of things twice. So to solve this problem Ray Strode spent some time writing what we call GNOME Kiosk mode which makes setting up a simple session running single application easy and without running things like the GNOME shell, tracker, evolution etc. This gives you a window manager with full support for the latest technologies such as compositing, libinput and Wayland, but coming in at about 18MB, which is about 71MB less than a minimal GNOME 3 desktop session. You can read more about the new Kiosk mode and how to use it in this great blog post from our savvy Edge Computing Product Manager Ben Breard. The kiosk mode session described in Ben’s article about RHEL will be available with Fedora Workstation 35.

high-definition mouse wheel support
A major part of what we do is making sure that Red Hat Enterprise Linux customers and Fedora users get hardware support on par with what you find on other operating systems. We try our best to work with our hardware partners, like Lenovo, to ensure that such hardware support comes day and date with when those features are enabled on other systems, but some things ends up taking longer time for various reasons. Support for high-definition mouse wheels was one of those. Peter Hutterer, our resident input expert, put together a great blog post explaining the history and status of high-definition mouse wheel support. As Peter points out in his blog post the feature is not yet fully supported under Wayland, but we hope to close that gap in time for Fedora Workstation 35.

Mouse with hires mouse

Mouse with HiRes scroll wheel

PipeWire
I feel I can’t do one of these posts without talking about latest developments in PipeWire, our unified audio and video server. Wim Taymans keeps working with rapidly growing PipeWire community to fix issues as they are reported and add new features to PipeWire. Most recently Wims focus has been on implementing support for S/PDIF passthrough support over both S/PDIF and HDMI connections. This will allow us to send undecoded data over such connections which is critical for working well with surround sound systems and soundbars. Also the PipeWire community has been working hard on further improving the Bluetooth support with bluetooth battery status support for head-set profile and using Apple extensions. aptX-LL and FastStream codec support was also added. And of course a huge amount of bug fixes, it turns out that when you replace two different sound servers that has been around for close to two decades there are a lot of corner cases to cover :). Make sure to check out two latest release notes for 0.3.35 and for 0.3.36 for details.

Screenshot of Easyeffects

EasyEffects is a great example of a cool new application built with PipeWire

Privacy screen
Another feature that we have been working on as a result of our Lenovo partnership is Privacy screen support. For those not familiar with this technology it is basically to allow you to reduce the readability of your screen when viewed from the side, so that if you are using your laptop at a coffee shop for instance then a person sitting close by will have a lot harder time trying to read what is on your screen. Hans de Goede has been shepherding the kernel side of this forward working with Marco Trevisan from Canonical on the userspace part of it (which also makes this a nice example of cross-company collaboration), allowing you to turn this feature on or off. This feature though is not likely to fully land in time for Fedora Workstation 35 so we are looking at if we will bring this in as an update to Fedora Workstation 35 or if it will be a Fedora Workstation 36 feature.

Penny

zink inside

Zink inside the penny


As most of you know the future of 3D graphics on Linux is the Vulkan API from the Khronos Group. This doesn’t mean that OpenGL is going away anytime soon though, as there is a large host of applications out there using this API and for certain types of 3D graphics development developers might still choose to use OpenGL over Vulkan. Of course for us that creates a little bit of a challenge because maintaining two 3D graphics interfaces is a lot of work, even with the great help and contributions from the hardware makers themselves. So we been eyeing the Zink project for a while, which aims at re-implementing OpenGL on top of Vulkan, as a potential candidate for solving our long term needs to support the OpenGL API, but without drowning us in work while doing so. The big advantage to Zink is that it allows us to support one shared OpenGL implementation across all hardware and then focus our HW support efforts on the Vulkan drivers. As part of this effort Adam Jackson has been working on a project called Penny.

Zink implements OpenGL in terms of Vulkan, as far as the drawing itself is concerned, but presenting that drawing to the rest of the system is currently system-specific (GLX). For hardware that already has a Mesa driver, we use GBM. On NVIDIA’s Vulkan (and probably any other binary stacks on Linux, and probably also like WSL or macOS + MoltenVK) we download the image from the GPU back to the CPU and then use the same software upload/display path as llvmpipe, which as you can imagine is Not Fast.

Penny aims to extend Zink by replacing both of those paths, and instead using the various Vulkan WSI extensions to manage presentation. Even for the GBM case this should enable higher performance since zink will have more information about the rendering pipeline (multisampling in particular is poorly handled atm). Future window system integration work can focus on Vulkan, with EGL and GLX getting features “for free” once they’re enabled in Vulkan.

3rd party software cleanup
Over time we have been working on adding more and more 3rd party software for easy consumption in Fedora Workstation. The problem we discovered though was that due to this being done over time, with changing requirements and expectations, the functionality was not behaving in a very intuitive way and there was also new questions that needed to be answered. So Allan Day and Owen Taylor spent some time this cycle to review all the bits and pieces of this functionality and worked to clean it up. So the goal is that when you enable third-party repositories in Fedora Workstation 35 it behaves in a much more predictable and understandable way and also includes a lot of applications from Flathub. Yes, that is correct you should be able to install a lot of applications from Flathub in Fedora Workstation 35 without having to first visit the Flathub website to enable it, instead they will show up once you turned the knob for general 3rd party application support.

Power profiles
Another item we spent quite a bit of time for Fedora Workstation 35 is making sure we integrate the Power Profiles work that Bastien Nocera has been working on as part of our collaboration with Lenovo. Power Profiles is basically a feature that allows your system to behave in a smarter way when it comes to power consumption and thus prolongs your battery life. So for instance when we notice you are getting low on battery we can offer you to go into a strong power saving mode to prolong how long you can use the system until you can recharge. More in-depth explanation of Power profiles in the official README.

Wayland
I usually also have ended up talking about Wayland in my posts, but I expect to be doing less going forward as we have now covered all the major gaps we saw between Wayland and X.org. Jonas Ådahl got the headless support merged which was one of our big missing pieces and as mentioned above Olivier Fourdan and Jonas and others worked with NVidia on getting the binary driver with XWayland support working with GNOME Shell. Of course this being software we are never truly done, there will of course be new issues discovered, random bugs that needs to be fixed, and of course also new features that needs to be implemented. We already have our next big team focus in place, HDR support, which will need work from the graphics drivers, up through Mesa, into the window manager and the GUI toolkits and in the applications themselves. We been investigating and trying out some things for a while already, but we are now ready to make this a main focus for the team. In fact we will soon be posting a new job listing for a fulltime engineer to work on HDR vertically through the stack so keep an eye out for that if you are interested in working on this. The job will be open to candidates who which to work remotely, so as long as Red Hat has a business presence in the country you live we should be able to offer you the job if you are the right candidate for us. Update:Job listing is now online for our HDR engineer.

BTW, if you want to see future updates and keep on top of other happenings from Fedora and Red Hat in the desktop space, make sure to follow me on twitter.

by uraeus at September 16, 2021 12:58 PM

September 08, 2021

GStreamerGStreamer 1.18.5 stable bug fix release

(GStreamer)

The GStreamer team is pleased to announce another bug fix release in the stable 1.18 release series of your favourite cross-platform multimedia framework!

This release only contains bugfixes and important security fixes, and it should be safe to update from 1.18.x.

Highlighted bugfixes:

  • basesink: fix reverse frame stepping
  • downloadbuffer/sparsefile: several fixes for win32
  • systemclock: Update monotonic reference time when re-scheduling, fixes high CPU usage with gnome-music when pausing playback
  • audioaggregator: fix glitches when resyncing on discont
  • compositor: Fix NV12 blend operation
  • rtspconnection: Add IPv6 support for tunneled mode
  • avidemux: fix playback of some H.264-in-AVI streams
  • jpegdec: Fix crash when interlaced field height is not DCT block size aligned
  • qmlglsink: Keep old buffers around a bit longer if they were bound by QML
  • qml: qtitem: don't potentially leak a large number of buffers
  • rtpjpegpay: fix image corruption when compiled with MSVC on Windows
  • rtspsrc: seeking improvements
  • rtpjitterbuffer: Avoid generation of invalid timestamps
  • rtspsrc: Fix behaviour of select-streams, new-manager, request-rtcp-key and before-send signals with GLib >= 2.62
  • multiudpsink: Fix broken SO_SNDBUF get/set on Windows
  • openh264enc: fix broken sps/pps header generation and some minor leaks
  • mpeg2enc: fix interlace-mode detection and unbound memory usage if encoder can't keep up
  • mfvideosrc: Fix for negative MF stride and for negotiation when interlace-mode is specified
  • tsdemux: fix seek-with-stop regression and decoding errors after seeking with dvdlpcmdec
  • rtsp-server: seek handling improvements
  • gst-libav: fix build (and other issues) with ffmpeg 4.4
  • cerbero: spandsp: Fix build error with Visual Studio 2019
  • win32 packages: Fix hang in GLib when `G_SLICE` environment variable is set
  • various stability, performance and reliability improvements
  • memory leak fixes
  • build fixes

See the GStreamer 1.18.5 release notes for more details.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

Download tarballs directly here: gstreamer, gst-plugins-base, gst-plugins-good, gst-plugins-ugly, gst-plugins-bad, gst-libav, gst-rtsp-server, gst-python, gst-editing-services, gst-devtools, gstreamer-vaapi, gstreamer-sharp, gst-omx, or gstreamer-docs.

September 08, 2021 11:30 PM

August 12, 2021

Jan SchmidtOpenHMD update

(Jan Schmidt)

A while ago, I wrote a post about how to build and test my Oculus CV1 tracking code in SteamVR using the SteamVR-OpenHMD driver. I have updated those instructions and moved them to https://noraisin.net/diary/?page_id=1048 – so use those if you’d like to try things out.

The pandemic continues to sap my time for OpenHMD improvements. Since my last post, I have been working on various refinements. The biggest visible improvements are:

  • Adding velocity and acceleration API to OpenHMD.
  • Rewriting the pose transformation code that maps from the IMU-centric tracking space to the device pose needed by SteamVR / apps.

Adding velocity and acceleration reporting is needed in VR apps that support throwing things. It means that throwing objects and using gravity-grab to fetch objects works in Half-Life: Alyx, making it playable now.

The rewrite to the pose transformation code fixed problems where the rotation of controller models in VR didn’t match the rotation applied in the real world. Controllers would appear attached to the wrong part of the hand, and rotate around the wrong axis. Movements feel more natural now.

Ongoing work – record and replay

My focus going forward is on fixing glitches that are caused by tracking losses or outliers. Those problems happen when the computer vision code either fails to match what the cameras see to the device LED models, or when it matches incorrectly.

Tracking failure leads to the headset view or controllers ‘flying away’ suddenly. Incorrect matching leads to controllers jumping and jittering to the wrong pose, or swapping hands. Either condition is very annoying.

Unfortunately, as the tracking has improved the remaining problems get harder to understand and there is less low-hanging fruit for improvement. Further, when the computer vision runs at 52Hz, it’s impossible to diagnose the reasons for a glitch in real time.

I’ve built a branch of OpenHMD that uses GStreamer to record the CV1 camera video, plus IMU and tracking logs into a video file.

To go with those recordings, I’ve been working on a replay and simulation tool, that uses the Godot game engine to visualise the tracking session. The goal is to show, frame-by-frame, where OpenHMD thought the cameras, headset and controllers were at each point in the session, and to be able to step back and forth through the recording.

Right now, I’m working on the simulation portion of the replay, that will use the tracking logs to recreate all the poses.

by thaytan at August 12, 2021 05:30 PM

August 05, 2021

Bastien Nocerapower-profiles-daemon: Follow-up

(Bastien Nocera)

Just about a year after the original announcement, I think it's time to see the progress on power-profiles-daemon.

Note that I would still recommend you read the up-to-date project README if you have questions about why this project was necessary, and why a new project was started rather than building on an existing one.

 The project was born out of the need to make a firmware feature available to end-users for a number of lines of Lenovo laptops for them to be fully usable on Fedora. For that, I worked with Mark Pearson from Lenovo, who wrote the initial kernel support for the feature and served as our link to the Lenovo firmware team, and Hans de Goede, who worked on making the kernel interfaces more generic.

More generic, but in a good way

 With the initial kernel support written for (select) Lenovo laptops, Hans implemented a more generic interface called platform_profile. This interface is now the one that power-profiles-daemon will integrate with, and means that it also supports a number of Microsoft Surface, HP, Lenovo's own Ideapad laptops, and maybe Razer laptops soon.

 The next item to make more generic is Lenovo's "lap detection" which still relies on a custom driver interface. This should be soon transformed into a generic proximity sensor, which will mean I get to work some more on iio-sensor-proxy.

Working those interactions

 power-profiles-dameon landed in a number of distributions, sometimes enabled by default, sometimes not enabled by default (sigh, the less said about that the better), which fortunately meant that we had some early feedback available.

 The goal was always to have the user in control, but we still needed to think carefully about how the UI would look and how users would interact with it when a profile was temporarily unavailable, or the system started a "power saver" mode because battery was running out.

 The latter is something that David Redondo's work on the "HoldProfile" API made possible. Software can programmatically switch to the power-saver or performance profile for the duration of a command. This is useful to switch to the Performance profile when running a compilation (eg. powerprofilesctl jhbuild --no-interact build gnome-shell), or for gnome-settings-daemon to set the power-saver profile when low on battery.

 The aforementioned David Redondo and Kai Uwe Broulik also worked on the KDE interface to power-profiles-daemon, as Florian Müllner implemented the gnome-shell equivalent.

Promised by me, delivered by somebody else :)

 I took this opportunity to update the Power panel in Settings, which shows off the temporary switch to the performance mode, and the setting to automatically switch to power-saver when low on battery.

Low-Power, everywhere

 Talking of which, while it's important for the system to know that they're targetting a power saving behaviour, it's also pretty useful for applications to try and behave better.
 
 Maybe you've already integrated with "low memory" events using GLib, but thanks to Patrick Griffis you can be an event better ecosystem citizen and monitor whether the system is in "Power Saver" mode and adjust your application's behaviour.
 
 This feature will be available in GLib 2.70 along with documentation of useful steps to take. GNOME Software will already be using this functionality to avoid large automated downloads when energy saving is needed.

Availability

 The majority of the above features are available in the GNOME 41 development branches and should get to your favourite GNOME-friendly distribution for their next release, such as Fedora 35.

by Bastien Nocera (noreply@blogger.com) at August 05, 2021 03:50 PM

August 02, 2021

Phil NormandIntroducing the GNOME Web Canary flavor

(Phil Normand)

Today I am happy to unveil GNOME Web Canary which aims to provide bleeding edge, most likely very unstable builds of Epiphany, depending on daily builds of the WebKitGTK development version. Read on to know more about this.

Until recently the GNOME Web browser was available for end-users in two …

by Philippe Normand at August 02, 2021 12:00 PM

July 09, 2021

Víctor JáquezVideo decoding in GStreamer with Vulkan

Warning: Vulkan video is still work in progress, from specification to available drivers and applications. Do not use it for production software just yet.

Introduction

Vulkan is a cross-platform Application Programming Interface (API), backed by the Khronos Group, aimed at graphics developers for a wide range of different tasks. The interface is described by a common specification, and it is implemented by different drivers, usually provided by GPU vendors and Mesa.

One way to visualize Vulkan, at first glance, is like a low-level OpenGL API, but better described and easier to extend. Even more, it is possible to implement OpenGL on top of Vulkan. And, as far as I am told by my peers in Igalia, Vulkan drivers are easier and cleaner to implement than OpenGL ones.

A couple years ago, a technical specification group (TSG), inside the Vulkan Working Group, proposed the integration of hardware accelerated video compression and decompression into the Vulkan API. In April 2021 the formed Vulkan Video TSG published an introduction to the
specification
. Please, do not hesitate to read it. It’s quite good.

Matthew Waters worked on a GStreamer plugin using Vulkan, mainly for uploading, composing and rendering frames. Later, he developed a library mapping Vulkan objects to GStreamer. This work was key for what I am presenting here. In 2019, during the last GStreamer Conference, Matthew delivered a talk about his work. Make sure to watch it, it’s worth it.

Other key components for this effort were the base classes for decoders and the bitstream parsing libraries in GStreamer, jointly developed by Intel, Centricular, Collabora and Igalia. Both libraries allow using APIs for stateless video decoding and encoding within the GStreamer framework, such as Vulkan Video, VAAPI, D3D11, and so on.

When the graphics team in Igalia told us about the Vulkan Video TSG, we decided to explore the specification. Therefore, Igalia decided to sponsor part of my time to craft a GStreamer element to decode H.264 streams using these new Vulkan extensions.

Assumptions

As stated at the beginning of this text, this development has to be considered unstable and the APIs may change without further notice.

Right now, the only Vulkan driver that offers these extensions is the beta NVIDIA driver. You would need, at least, version 455.50.12 for Linux, but it would be better to grab the latest one. And, of course, I only tested this on Linux. I would like to thank NVIDIA for their Vk Video samples. Their test application drove my work.

Finally, this work assumes the use of the main development branch of GStreamer, because the base classes for decoders are quite recent. Naturally, you can use gst-build for an efficient upstream workflow.

Work done

This work basically consists of two new objects inside the GstVulkan code:

  • GstVulkanDeviceDecoder: a GStreamer object in GstVulkan library, inherited from GstVulkanDevice, which enables VK_KHR_video_queue and VK_KHR_video_decode_queue extensions. Its purpose is to handle codec-agnostic operations.
  • vulkanh264dec: a GStreamer element, inherited from GstH264Decoder, which tries to instantiate a GstVulkanDeviceDecoder to composite it and is in charge of handling codec-specific operations later, such as matching the parsed structures. It outputs, in the source pad, memory:VulkanImage featured frames, with NV12 color format.

  • So far this pipeline works without errors:

    $ gst-launch-1.0 filesrc location=big_buck_bunny_1080p_h264.mov ! parsebin ! vulkanh264dec ! fakesink
    

    As you might see, the pipeline does not use vulkansink to render frames. This is because the Vulkan format output by the driver’s decoder device is VK_FORMAT_G8_B8R8_2PLANE_420_UNORM, which is NV12 crammed in a single image, while for GstVulkan a NV12 frame is a buffer with two images, one per component. So the current color conversion in GstVulkan does not support this Vulkan format. That is future work, among other things.

    You can find the merge request for this work in GStreamer’s Gitlab.

    Future work

    As was mentioned before, it is required to fully support VK_FORMAT_G8_B8R8_2PLANE_420_UNORM format in GstVulkan. That requires thinking about how to keep backwards compatibility. Later, an implementation of the sampler to convert this format to RGB will be needed, so that decoded frames can be rendered by vulkansink.

    Also, before implementing any new feature, the code and its abstractions will need to be cleaned up, since currently the division between codec-specific and codec-agnostic code is not strict, and it must be fixed.

    Another important cleanup task is to enhance the way the Vulkan headers are handled. Since the required headers files for video extensions are beta, they are not expected to be available in the system, so temporally I had to add the those headers as part of the GstVulkan library.

    Then it will be possible to implement the H.265 decoder, since the NVIDIA driver also supports it.

    Later on, it will be nice to start thinking about encoders. But this requires extending support for stateless encoders in GStreamer, something I want do to for the new VAAPI plugin too.

    Thanks for bearing with me, and thanks to Igalia for sponsoring this work.

    by vjaquez at July 09, 2021 05:38 PM

    June 28, 2021

    GStreamerGStreamer Rust bindings 0.17.0 release

    (GStreamer)

    A new version of the GStreamer Rust bindings, 0.17.0, was released.

    As usual this release follows the latest gtk-rs release.

    This is the first version that includes optional support for new GStreamer 1.20 APIs. As GStreamer 1.20 was not released yet, these new APIs might still change. The minimum supported version of the bindings is still GStreamer 1.8 and the targetted GStreamer API version can be selected by applications via feature flags.

    Apart from this, the new version features a lot of API cleanup, especially of the subclassing APIs, and the addition of a few missing bindings. As usual, the focus of this release was to make usage of GStreamer from Rust as convenient and complete as possible.

    The new release also brings a lot of bugfixes, most of which were already part of the 0.16.x bugfix releases.

    A new release of the GStreamer Rust plugins will follow in the next days.

    Details can be found in the release notes for gstreamer-rs.

    The code and documentation for the bindings is available on the freedesktop.org GitLab

    as well as on crates.io.

    If you find any bugs, notice any missing features or other issues please report them in GitLab.

    June 28, 2021 11:00 PM

    June 24, 2021

    Seungha YangGStreamer Media Foundation Video Encoder Is Now Faster — Direct3D11 Awareness

    GStreamer Media Foundation Video Encoder Is Now Faster — Direct3D11 Awareness

    TL;DR

    GStreamer MediaFoundation video encoders (H.264, HEVC, and VP9 if supported by GPU) gained the ability to accept Direct3D11 textures, which will bring noticeable performance improvements

    As of the GStreamer 1.18 release, hardware accelerated Direct3D11/DXVA video decoding and MediaFoundation based video encoding features were landed.

    Those native Windows video APIs can be very helpful for application development/deployment, since they are hardware platform-agnostic APIs for the Windows platform. The questions is if they are sufficiently competitive with hardware-specific APIs such as NVIDIA NVCODEC SDK or Intel Media SDK?

    Probably the answer is … “NO”

    How much faster than before are things?

    One simple way to compare performance would be to measure the time spent for transcoding. Of course, encoded file size and visual quality are also very important factors. However, as per my experiments, resulting video file size and visual quality (in terms of PSNR) were very close to each other. Then our remaining interest is speed!

    Let’s take a look at my measurement. I performed the measurement by using one 4K H.264 video content with an NVIDIA RTX 3060 GPU and an Intel Core i7–1065G7 integrated GPU. For reference, NVCODEC and Intel Media SDK plugins were tested by using GStreamer 1.19.1 as well. Each test used performance (speed) oriented encoding options to be a fair comparison.

    - NVIDA RTX 3060

    GStreamer 1.18 — 2 min 1 sec
    GStreamer 1.19.1 1 min 9 sec
    NVCODEC plugin (nvh264dec/nvh264enc pair) — 1 min 19 sec

    - Intel Core i7–1065G7 integrated GPU

    GStreamer 1.18 — 3 min 8 sec
    GStreamer 1.19.1 — 2 min 45 sec
    Intel Media SDK plugin (msdkh264dec/msdkh264enc pair)3 min 10 sec

    So, is it true that the Direct3D11/DXVA and MediaFoundation combination can be faster than hardware-specific APIs? Yes, as you can see

    Note that such results would be very system environment and encoding option dependent, so, you’d likely see different numbers

    Why MediaFoundation plugin got faster

    GStreamer 1.18 — The story was, because of the lack of Direct3D11 integration at MediaFoundation plugin side, each decoded frame (Direct3D11 texture) must be downloaded into system memory first, which is usually very slow path. And then, the memory was copied to another system memory allocated by MediaFoundation. Moreover, likely GPU driver would upload to GPU memory again. Well, twice visible redundant copies and another potential copy per frame!?!? hrm…

    In GStreamer 1.19.1, thanks to the Direct3D11 integration, MediaFoundation can accept Direct3D11 texture, which means we don’t need to download GPU texture and re-upload it any more.

    More details

    Since all Direct3D11/DXVA, MediaFoundation, NVCODEC, and Intel Media SDK APIs work with underlying GPU hardware, the performance should not be much different in theory, unless there are visible overhead around GPU vendor’s driver implementation.

    Then, remaining factor would be API consumer-side optimization.
    And yes, from GStreamer plugin implementation point of view, Direct3D11/DXVA and MediaFoundation plugins
    are more optimized than NVCODEC and MSDK plugins in terms of GPU memory transfer on Windows.

    It doesn’t mean Direct3D11/DXVA and MediaFoundation themselves are superior APIs than hardware-specific APIs at all. The difference is just result of more or less optimized plugin implementations

    You can try this enhancement right now!

    Install the official GStreamer 1.19.1 release, and just run this command.

    gst-launch-1.0.exe filesrc location=where-your-h264-file-located ! parsebin ! d3d11h264dec ! queue ! mfh264enc ! h264parse ! mp4mux ! filesink location=my-output.mp4

    You will be likely able to see the improvement by yourself :)

    There are still a lot of interesting topics for better Windows support in GStreamer. Specifically, nicer text support via DirectWrite and fine tuned GPU scheduling via Direct3D12 are on our radar. Not only for video features, we will keep improving various Windows specific features, including audio capture/render device support.

    If you’ve see any bugs, please contact me, and even better would be a bug report at GitLab. I’m watching it most of time 😊

    by Seungha Yang at June 24, 2021 04:01 PM

    June 15, 2021

    GStreamerIRC Channel has moved from Freenode to OFTC

    (GStreamer)

    Due to the widely reported issues at the Freenode IRC network, the official GStreamer discussion IRC channel has moved to #gstreamer on the OFTC IRC network alongside other Freedesktop projects.

    You can connect to it with your existing IRC client, or using Matrix which has a browser client and native apps for all platforms.

    For more information, please see the mailing list announcement.

    June 15, 2021 08:30 AM

    June 03, 2021

    Thomas Vander SticheleAmazing Marvin and KeyCombiner

    (Thomas Vander Stichele)

    I recently came across an excellent tool called KeyCombiner that helps you practice keyboard shortcuts (3 sets for free, $29/6 months for more sets). I spent some time to create a set for Amazing Marvin, my current todo manager of choice.

    The shareable URL to use in KeyCombiner is https://keycombiner.com/collecting/collections/shared/f1f78977-0920-4888-a86d-d00a7201502e

    I generated it from the printed PDF version of Marvin’s keyboard guide and a bunch of manual editing, in a google sheet.

    Keyboard shortcuts are great timesavers and help reduce friction, but it’s getting harder to learn them properly, and this tool has been a great help for some other apps, and for figuring out common shortcuts across apps, and for picking custom shortcuts (in other apps) that don’t conflict. If this is a problem you recognize, give KeyCombiner a try.

    Flattr this!

    by Thomas at June 03, 2021 02:27 AM

    June 01, 2021

    GStreamerGStreamer 1.19.1 unstable development release

    (GStreamer)

    The GStreamer team is pleased to announce the first development release in the unstable 1.19 release series.

    The unstable 1.19 release series adds new features on top of the current stable 1.18 series and is part of the API and ABI-stable 1.x release series of the GStreamer multimedia framework.

    The unstable 1.19 release series is for testing and development purposes in the lead-up to the stable 1.20 series which is scheduled for release in a few weeks time. Any newly-added API can still change until that point, although it is rare for that to happen.

    Full release notes will be provided in the near future, highlighting all the new features, bugfixes, performance optimizations and other important changes.

    This development release is primarily for distributors and early adaptors and anyone who still needs to update their build/packaging setup for Meson.

    Packagers: please note that plugins may have moved between modules, so please take extra care and make sure inter-module version dependencies are such that users can only upgrade all modules in one go, instead of seeing a mix of 1.19 and 1.18 on their system.

    Binaries for Android, iOS, Mac OS X and Windows are also available at the usual location.

    Release tarballs can be downloaded directly here:

    As always, please let us know of any issues you run into by filing an issue in Gitlab.

    June 01, 2021 04:00 PM

    Robert McQueenNext steps for the GNOME Foundation

    (Robert McQueen)

    As the President of the GNOME Foundation Board of Directors, I’m really pleased to see the number and breadth of candidates we have for this year’s election. Thank you to everyone who has submitted their candidacy and volunteered their time to support the Foundation. Allan has recently blogged about how the board has been evolving, and I wanted to follow that post by talking about where the GNOME Foundation is in terms of its strategy. This may be helpful as people consider which candidates might bring the best skills to shape the Foundation’s next steps.

    Around three years ago, the Foundation received a number of generous donations, and Rosanna (Director of Operations) gave a presentation at GUADEC about her and Neil’s (Executive Director, essentially the CEO of the Foundation) plans to use these funds to transform the Foundation. We would grow our activities, increasing the pace of events, outreach, development and infrastructure that supported the GNOME project and the wider desktop ecosystem – and, crucially, would grow our funding to match this increased level of activity.

    I think it’s fair to say that half of this has been a great success – we’ve got a larger staff team than GNOME has ever had before. We’ve widened the GNOME software ecosystem to include related apps and projects under the GNOME Circle banner, we’ve helped get GTK 4 out of the door, run a wider-reaching program in the Community Engagement Challenge, and consistently supported better infrastructure for both GNOME and the Linux app community in Flathub.

    Aside from another grant from Endless (note: my employer), our fundraising hasn’t caught up with this pace of activities. As a result, the Board recently approved a budget for this financial year which will spend more funds from our reserves than we expect to raise in income. Due to our reserves policy, this is essentially the last time we can do this: over the next 6-12 months we need to either raise more money, or start spending less.

    For clarity – the Foundation is fit and well from a financial perspective – we have a very healthy bank balance, and a very conservative “12 month run rate” reserve policy to handle fluctuations in income. If we do have to slow down some of our activities, we will return to a “steady state” where our regular individual donations and corporate contributions can support a smaller staff team that supports the events and infrastructure we’ve come to rely on.

    However, this isn’t what the Board wants to do – the previous and current boards were unanimous in their support of the idea that we should be ambitious: try to do more in the world and bring the benefits of GNOME to more people. We want to take our message of trusted, affordable and accessible computing to the wider world.

    Typically, a lot of the activities of the Foundation have been very inwards-facing – supporting and engaging with either the existing GNOME or Open Source communities. This is a very restricted audience in terms of fundraising – many corporate actors in our community already support GNOME hugely in terms of both financial and in-kind contributions, and many OSS users are already supporters either through volunteer contributions or donating to those nonprofits that they feel are most relevant and important to them.

    To raise funds from new sources, the Foundation needs to take the message and ideals of GNOME and Open Source software to new, wider audiences that we can help. We’ve been developing themes such as affordability, privacy/trust and education as promising areas for new programs that broaden our impact. The goal is to find projects and funding that allow us to both invest in the GNOME community and find new ways for FOSS to benefit people who aren’t already in our community.

    Bringing it back to the election, I’d like to make clear that I see this – reaching the outside world, and finding funding to support that – as the main priority and responsibility of the Board for the next term. GNOME Foundation elections are a slightly unusual process that “filters” our board nominees by being existing Foundation members, which means that candidates already work inside our community when they stand for election. If you’re a candidate and are already active in the community – THANK YOU – you’re doing great work, keep doing it! That said, you don’t need to be a Director to achieve things within our community or gain the support of the Foundation: being a community leader is already a fantastic and important role.

    The Foundation really needs support from the Board to make a success of the next 12-18 months. We need to understand our financial situation and the trade-offs we have to make, and help to define the strategy with the Executive Director so that we can launch some new programs that will broaden our impact – and funding – for the future. As people cast their votes, I’d like people to think about what kind of skills – building partnerships, commercial background, familiarity with finances, experience in nonprofit / impact spaces, etc – will help the Board make the Foundation as successful as it can be during the next term.

    by ramcq at June 01, 2021 11:45 AM

    May 25, 2021

    Enrique Ocaña GonzálezGStreamer WebKit debugging by using external tools (2/2)

    This is the last post of the series showing interesting debugging tools, I hope you have found it useful. Don’t miss the custom scripts at the bottom to process GStreamer logs, help you highlight the interesting parts and find the root cause of difficult bugs. Here are also the previous posts of the series:

    How to debug pkgconfig

    When pkg-config finds the PKG_CONFIG_DEBUG_SPEW env var, it explains all the steps used to resolve the packages:

    PKG_CONFIG_DEBUG_SPEW=1 /usr/bin/pkg-config --libs x11

    This is useful to know why a particular package isn’t found and what are the default values for PKG_CONFIG_PATH when it’s not defined. For example:

    Adding directory '/usr/local/lib/x86_64-linux-gnu/pkgconfig' from PKG_CONFIG_PATH
    Adding directory '/usr/local/lib/pkgconfig' from PKG_CONFIG_PATH
    Adding directory '/usr/local/share/pkgconfig' from PKG_CONFIG_PATH
    Adding directory '/usr/lib/x86_64-linux-gnu/pkgconfig' from PKG_CONFIG_PATH
    Adding directory '/usr/lib/pkgconfig' from PKG_CONFIG_PATH
    Adding directory '/usr/share/pkgconfig' from PKG_CONFIG_PATH

    If we have tuned PKG_CONFIG_PATH, maybe we also want to add the default paths. For example:

    SYSROOT=~/sysroot-x86-64
    export PKG_CONFIG_PATH=${SYSROOT}/usr/local/lib/pkgconfig:${SYSROOT}/usr/lib/pkgconfig
    # Add also the standard pkg-config paths to find libraries in the system
    export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:/usr/local/lib/x86_64-linux-gnu/pkgconfig:\
    /usr/local/lib/pkgconfig:/usr/local/share/pkgconfig:/usr/lib/x86_64-linux-gnu/pkgconfig:\
    /usr/lib/pkgconfig:/usr/share/pkgconfig
    # This tells pkg-config where the "system" pkg-config dir is. This is useful when cross-compiling for other
    # architecture, to avoid pkg-config using the system .pc files and mixing host and target libraries
    export PKG_CONFIG_LIBDIR=${SYSROOT}/usr/lib
    # This could have been used for cross compiling:
    #export PKG_CONFIG_SYSROOT_DIR=${SYSROOT}

    Man in the middle proxy for WebKit

    Sometimes it’s useful to use our own modified/unminified files with a 3rd party service we don’t control. Mitmproxy can be used as a man-in-the-middle proxy, but I haven’t tried it personally yet. What I have tried (with WPE) is this:

    1. Add an /etc/hosts entry to point the host serving the files we want to change to an IP address controlled by us.
    2. Configure a web server to provide the files in the expected path.
    3. Modify the ResourceRequestBase constructor to change the HTTPS requests to HTTP when the hostname matches the target:
    ResourceRequestBase(const URL& url, ResourceRequestCachePolicy policy)
        : m_url(url)
        , m_timeoutInterval(s_defaultTimeoutInterval)
        ...
        , m_isAppBound(false)
    {
        if (m_url.host().toStringWithoutCopying().containsIgnoringASCIICase(String("out-of-control-service.com"))
            && m_url.protocol().containsIgnoringASCIICase(String("https"))) {
            printf("### %s: URL %s detected, changing from https to http\n",
                __PRETTY_FUNCTION__, m_url.string().utf8().data()); 
            fflush(stdout);
            m_url.setProtocol(String("http"));
        }
    }

    :bulb: Pro tip: If you have to debug minified/obfuscated JavaScript code and don’t have a deobfuscated version to use in a man-in-the-middle fashion, use http://www.jsnice.org/ to deobfuscate it and get meaningful variable names.

    Bandwidth control for a dependent device

    If your computer has a “shared internet connection” enabled in Network Manager and provides access to a dependent device , you can control the bandwidth offered to that device. This is useful to trigger quality changes on adaptive streaming videos from services out of your control.

    This can be done using tc, the Traffic Control tool from the Linux kernel. You can use this script to automate the process (edit it to suit to your needs).

    Useful scripts to process GStreamer logs

    I use these scripts in my daily job to look for strange patterns in GStreamer logs that help me to find the cause of the bugs I’m debugging:

    • h: Highlights each expression in the command line in a different color.
    • mgrep: Greps (only) for the lines with the expressions in the command line and highlights each expression in a different color.
    • filter-time: Gets a subset of the log lines between a start and (optionally) an end GStreamer log timestamp.
    • highlight-threads: Highlights each thread in a GStreamer log with a different color. That way it’s easier to follow a thread with the naked eye.
    • remove-ansi-colors: Removes the color codes from a colored GStreamer log.
    • aha: ANSI-HTML-Adapter converts plain text with color codes to HTML, so you can share your GStreamer logs from a web server (eg: for bug discussion). Available in most distros.
    • gstbuffer-leak-analyzer: Analyzes a GStreamer log and shows unbalances in the creation/destruction of GstBuffer and GstMemory objects.

    by eocanha at May 25, 2021 06:00 AM

    May 20, 2021

    Reynaldo VerdejoShort tutorial: Digital Television with GStreamer (ATSC setup)

    GStreamer support for Digital Television (DTV) is still working. If you follow a few very simple steps you can get DTV signals tuned-to, captured, decoded and displayed with your computer and a few accessories. 

    This short article presents basic instructions for a working DTV setup. I use ATSC in the US as an example, but the short process should work equally well for other terrestrial delivery systems and countries.

    What hardware do you need?

    Beyond a computer, you will need an antenna and a capture device. I recently tested a Hauppauge WinTV-dualHD (ATSC/QAM) USB dongle with a 5-years old Amazon basics flat indoor antenna (that looks pretty much like this one) and it worked quite well at roughly 60km from the repeater station.

    Installation

    The hardware setup is simple. Install the antenna pointing to your repeater station and as high as you can. Your local telecommunications and/or broadcasting association should have data about repeaters near you. You can get this information by ZIP code from the Federal Communications Commission (FCC) if you are in the US.

    Don't stress about the "as high as you can" part of the antenna installation. It's great to have a uninterrupted and direct line of sight to the repeating station's antenna, but this is hard to accomplish in urban areas. My current test system is indoors, on a first floor, and it works fine for a few channels.

    Software

    You will need GStreamer. I'm building it from the master branch with gst-build but the DTV code hasn't changed much in the last few years so any recently-packaged version should work for you. You will also need dvbv5-scan from DVBv5 tools.

    TL;DR

    Build a channels.conf configuration file by scanning an initial set of frequencies for your area with dvbv5-scan. You can get initial-frequencies files from the the dtv-scan-tables repository. Here I'm using my Mountain View, California file as an example:

    $ dvbv5-scan -o channels.conf ./dtv-scan-tables/atsc/us-CA-Mountain-View

    The scanning can take several minutes.

    If you grep the resulting file you can see what channels were detected:

    $ grep "\\[" channels.conf

    The output should look something like this:


    [KNTV-HD]
    [Hai Le]
    [KPIX-TV]
    [KQED-HD]
    [KTVU-HD]
    [ION]
    [KFSF-HD]
    [KICU-HD]


    If you get no channels, and you have reasons to believe your hardware is working correctly, try repositioning your antenna and rescanning till you get some. Small direction and position changes can have big effects on VHF/UHF reception with semi-directional antennas like the flat one I'm using for this example.

    To playback a channel with GStreamer you can use the generated channels.conf configuration file and any of the scanned-channel names to let playbin/dvbbasebin figure out all necessary parameters at runtime

    $ GST_DVB_CHANNELS_CONF=channels.conf gst-play-1.0 dvb://KNTV-HD

    And that's it.

    There are lots of details, pitfalls and options I don't write about in this short tutorial but if there's some interest I will try to discuss the subject in more depth in future ones.

    I leave you with two screenshot of the HD feed by KNTV in the bay area

     



     

     

     


     






    by reynaldo (noreply@blogger.com) at May 20, 2021 06:36 PM

    Christian SchallerNew opportunities in the Red Hat Desktop team

    (Christian Schaller)

    So we are looking to hire quite a few people into the Desktop team currently. First of all we are looking to hire two graphics engineers to help us work on Linux Graphics drivers. The first of those two jobs is now online on the Red Hat jobs site. This is a job in our core graphics team focusing on RHEL, Fedora and upstream around the Intel, AMD and NVidia open source drivers. This is an opportunity to join a team of incredibly talented engineers working on everything from the graphics system of the Linux kernel and on the userspace bits like Vulkan, OpenGL and Wayland.  The job is listed as Senior Engineer, but for the right candidate we have flexibility there. We also have flexibility for people who want to work remotely, so as long as there is a Red Hat office in your home country you can work remotely for us.  The second job, which we hope to have up soon, will be looking more at ARM graphics and be tied to our automotive effort, but we will be looking at the applications for either position in combination so feel free to apply for the already listed job even if you are more interested in the second one as we will discuss both jobs with potential candidates.

    The second job we have up is for – Software Engineer – GPU, Input and Multimedia which is also for joining our Graphics team. This job is targetted at our  office in Brno, Czechia and is a great entry level position if you are interested in the field of graphics. The job listing can be found here and outlines the kind of things we want you to look at, but do expect initially your job will be focused on helping the rest of the team manage their backlog and then grow from there.

    The last job we have online now is for the automotive team, where we are looking for someone at the Senior/Principal level to join our Infotainment team, working with car makers around issues related to multimedia and help identifying needs and gaps and then work with upstream communities to figure out how we can resolve those issues. The job is targeted at Madrid, Spain as it is where we hope to center some of the infotainment effort and it makes things easier in terms of hardware access and similar, but for the right candidate we might be open to looking for candidates wanting to work remote or in another Red Hat office. You can find this job listing here.

    We expect to be posting further jobs for the infotainment team within a week or two, so I will update once they are up.

    by uraeus at May 20, 2021 03:56 PM

    May 18, 2021

    Enrique Ocaña GonzálezGStreamer WebKit debugging by using external tools (1/2)

    In this new post series, I’ll show you how both existing and ad-hoc tools can be helpful to find the root cause of some problems. Here are also the older posts of this series in case you find them useful:

    Use strace to know which config/library files are used by a program

    If you’re becoming crazy supposing that the program should use some config and it seems to ignore it, just use strace to check what config files, libraries or other kind of files is the program actually using. Use the grep rules you need to refine the search:

    $ strace -f -e trace=%file nano 2> >(grep 'nanorc')
    access("/etc/nanorc", R_OK)             = 0
    access("/usr/share/nano/javascript.nanorc", R_OK) = 0
    access("/usr/share/nano/gentoo.nanorc", R_OK) = 0
    ...

    Know which process is killing another one

    First, try to strace -e trace=signal -p 1234 the killed process.

    If that doesn’t work (eg: because it’s being killed with the uncatchable SIGKILL signal), then you can resort to modifying the kernel source code (signal.c) to log the calls to kill():

    SYSCALL_DEFINE2(kill, pid_t, pid, int, sig)
    {
        struct task_struct *tsk_p;
        ...
        /* Log SIGKILL */
        if (sig & 0x1F == 9) {
            tsk_p = find_task_by_vpid(pid);
    
            if (tsk_p) {
                printk(KERN_DEBUG "Sig: %d from pid: %d (%s) to pid: %d (%s)\n",
                    sig, current->pid, current->comm, pid, tsk_p->comm);
            } else {
                printk(KERN_DEBUG "Sig: %d from pid: %d (%s) to pid: %d\n",
                    sig, current->pid, current->comm, pid);
            }
        }
        ...
    }

    Wrap gcc/ld/make to tweak build parameters

    If you ever find yourself with little time in front of a stubborn build system and, no matter what you try, you can’t get the right flags to the compiler, think about putting something (a wrapper) between the build system and the compiler. Example for g++:

    #!/bin/bash
    main() {
        # Build up arg[] array with all options to be passed
        # to subcommand.
        i=0
        for opt in "$@"; do
            case "$opt" in
            -O2) ;; # Removes this option
            *)
                arg[i]="$opt" # Keeps the others
                i=$((i+1))
                ;;
            esac
        done
        EXTRA_FLAGS="-O0" # Adds extra option
        echo "g++ ${EXTRA_FLAGS} ${arg[@]}" # >> /tmp/build.log # Logs the command
        /usr/bin/ccache g++ ${EXTRA_FLAGS} "${arg[@]}" # Runs the command
    }
    main "$@"

    Make sure that the wrappers appear earlier than the real commands in your PATH.

    The make wrapper can also call remake instead. Remake is fully compatible with make but has features to help debugging compilation and makefile errors.

    Analyze the structure of MP4 data

    The ISOBMFF Box Structure Viewer online tool allows you to upload an MP4 file and explore its structure.

    by eocanha at May 18, 2021 06:00 AM

    May 13, 2021

    Andy Wingocross-module inlining in guile

    (Andy Wingo)

    Greetings, hackers of spaceship Earth! Today's missive is about cross-module inlining in Guile.

    a bit of history

    Back in the day... what am I saying? I always start these posts with loads of context. Probably you know it all already. 10 years ago, Guile's partial evaluation pass extended the macro-writer's bill of rights to Schemers of the Guile persuasion. This pass makes local function definitions free in many cases: if they should be inlined and constant-folded, you are confident that they will be. peval lets you write clear programs with well-factored code and still have good optimization.

    The peval pass did have a limitation, though, which wasn't its fault. In Guile, modules have historically been a first-order concept: modules are a kind of object with a hash table inside, which you build by mutating. I speak crassly but that's how it is. In such a world, it's hard to reason about top-level bindings: what module do they belong to? Could they ever be overridden? When you have a free reference to a, and there's a top-level definition of a in the current compilation unit, is that the a that's being referenced, or could it be something else? Could the binding be mutated in the future?

    During the Guile 2.0 and 2.2 cycles, we punted on this question. But for 3.0, we added the notion of declarative modules. For these modules, bindings which are defined once in a module and which are not mutated in the compilation unit are declarative bindings, which can be reasoned about lexically. We actually translate them to a form of letrec*, which then enables inlining via peval, contification, and closure optimization -- in descending order of preference.

    The upshot is that with Guile 3.0, top-level bindings are no longer optimization barriers, in the case of declarative modules, which are compatible enough with historic semantics and usage that they are on by default.

    However, module boundaries have still been an optimization barrier. Take (srfi srfi-1), a little utility library on lists. One definition in the library is xcons, which is cons with arguments reversed. It's literally (lambda (cdr car) (cons car cdr)). But does the compiler know that? Would it know that (car (xcons x y)) is the same as y? Until now, no, because no part of the optimizer will look into bindings from outside the compilation unit.

    mr compiler, tear down this wall

    But no longer! Guile can now inline across module boundaries -- in some circumstances. This feature will be part of a future Guile 3.0.8.

    There are actually two parts of this. One is the compiler can identify a set of "inlinable" values from a declarative module. An inlinable value is a small copyable expression. A copyable expression has no identity (it isn't a fresh allocation), and doesn't reference any module-private binding. Notably, lambda expressions can be copyable, depending on what they reference. The compiler then extends the module definition that's residualized in the compiled file to include a little procedure that, when passed a name, will return the Tree-IL representation of that binding. The design of that was a little tricky; we want to avoid overhead when using the module outside of the compiler, even relocations. See compute-encoding in that module for details.

    With all of that, we can call ((module-inlinable-exports (resolve-interface '(srfi srfi-1))) 'xcons) and get back the Tree-IL equivalent of (lambda (cdr car) (cons car cdr)). Neat!

    The other half of the facility is the actual inlining. Here we lean on peval again, causing <module-ref> forms to trigger an attempt to copy the term from the imported module to the residual expression, limited by the same effort counter as the rest of peval.

    The end result is that we can be absolutely sure that constants in imported declarative modules will inline into their uses, and fairly sure that "small" procedures will inline too.

    caveat: compiled imported modules

    There are a few caveats about this facility, and they are sufficiently sharp that I should probably fix them some day. The first one is that for an imported module to expose inlinable definitions, the imported module needs to have been compiled already, not loaded from source. When you load a module from source using the interpreter instead of compiling it first, the pipeline is optimized for minimizing the latency between when you ask for the module and when it is available. There's no time to analyze the module to determine which exports are inlinable and so the module exposes no inlinable exports.

    This caveat is mitigated by automatic compilation, enabled by default, which will compile imported modules as needed.

    It could also be fixed for modules by topologically ordering the module compilation sequence; this would allow some parallelism in the build but less than before, though for module graphs with cycles (these exist!) you'd still have some weirdness.

    caveat: abi fragility

    Before Guile supported cross-module inlining, there was only explicit inlining across modules in Guile, facilitated by macros. If you write a module that has a define-inlinable export and you think about its ABI, then you know to consider any definition referenced by the inlinable export, and you know by experience that its code may be copied into other compilation units. Guile doesn't automatically recompile a dependent module when a macro that it uses changes, currently anyway. Admittedly this situation leans more on culture than on tools, which could be improved.

    However, with automatically inlinable exports, this changes. Any definition in a module could be inlined into its uses in other modules. This may alter the ABI of a module in unexpected ways: you think that module C depends on module B, but after inlining it may depend on module A as well. Updating module B might not update the inlined copies of values from B into C -- as in the case of define-inlinable, but less lexically apparent.

    At higher optimization levels, even private definitions in a module can be inlinable; these may be referenced if an exported macro from the module expands to a term that references a module-private variable, or if an inlinable exported binding references the private binding. But these optimization levels are off by default precisely because I fear the bugs.

    Probably what this cries out for is some more sensible dependency tracking in build systems, but that is a topic for another day.

    caveat: reproducibility

    When you make a fresh checkout of Guile from git and build it, the build proceeds in the following way.

    Firstly, we build libguile, the run-time library implemented in C.

    Then we compile a "core" subset of Scheme files at optimization level -O1. This subset should include the evaluator, reader, macro expander, basic run-time, and compilers. (There is a bootstrap evaluator, reader, and macro expander in C, to start this process.) Say we have source files S0, S1, S2 and so on; generally speaking, these files correspond to Guile modules M0, M1, M2 etc. This first build produces compiled files C0, C1, C2, and so on. When compiling a file S2 implementing module M2, which happens to import M1 and M0, it may be M1 and M0 are provided by compiled files C1 and C0, or possibly they are loaded from the source files S1 and S0, or C1 and S0, or S1 and C0.

    The bootstrap build uses make for parallelism, with each compile process starts afresh, importing all the modules that comprise the compiler and then using them to compile the target file. As the build proceeds, more and more module imports can be "serviced" by compiled files instead of source files, making the build go faster and faster. However this introduces system-specific nondeterminism as to the set of compiled files available when compiling any other file. This strategy works because it doesn't really matter whether module M1 is provided by compiled file C1 or source file S1; the compiler and the interpreter implement the same language.

    Once the compiler is compiled at optimization level -O1, Guile then uses that freshly built compiler to build everything at -O2. We do it in this way because building some files at -O1 then all files at -O2 takes less time than going straight to -O2. If this sounds weird, that's because it is.

    The resulting build is reproducible... mostly. There is a bug in which some unique identifiers generated as part of the implementation of macros can be non-reproducible in some cases, and that disabling parallel builds seems to solve the problem. The issue being that gensym (or equivalent) might be called a different number of times depending on whether you are loading a compiled module, or whether you need to read and macro-expand it. The resulting compiled files are equivalent under alpha-renaming but not bit-identical. This is a bug to fix.

    Anyway, at optimization level -O1, Guile will record inlinable definitions. At -O2, Guile will actually try to do cross-module inlining. We run into two issues when compiling Guile; one is if we are in the -O2 phase, and we compile a module M which uses module N, and N is not in the set of "core" modules. In that case depending on parallelism and compile order, N may be loaded from source, in which case it has no inlinable exports, or from a compiled file, in which case it does. This is not a great situation for the reliability of this optimization. I think probably in Guile we will switch so that all modules are compiled at -O1 before compiling at -O2.

    The second issue is more subtle: inlinable bindings are recorded after optimization of the Tree-IL. This is more optimal than recording inlinable bindings before optimization, as a term that is not inlinable due to its size in its initial form may become small enough to inline after optimization. However, at -O2, optimization includes cross-module inlining! A term that is inlinable at -O1 may become not inlinable at -O2 because it gets slightly larger, or vice-versa: terms that are too large at -O1 could shrink at -O2. We don't even have a guarantee that we will reach a fixed point even if we repeatedly recompile all inputs at -O2, because we allow non-shrinking optimizations.

    I think this probably calls for a topological ordering of module compilation inside Guile and perhaps in other modules. That would at least give us reproducibility, provided we avoid the feedback loop of keeping around -O2 files compiled from a previous round, even if they are "up to date" (their corresponding source file didn't change).

    and for what?

    People who have worked on inliners will know what I mean that a good inliner is like a combine harvester: ruthlessly efficient, a qualitative improvement compared to not having one, but there is a pointy end with lots of whirling blades and it's important to stop at the end of the row. You do develop a sense of what will and won't inline, and I think Dybvig's "Macro writer's bill of rights" encompasses this sense. Luckily people don't lose fingers or limbs to inliners, but inliners can maim expectations, and cross-module inlining more so.

    Still, what it buys us is the freedom to be abstract. I can define a module like:

    (define-module (elf)
      #:export (ET_NONE ET_REL ET_EXEC ET_DYN ET_CORE))
    
    (define ET_NONE		0)		; No file type
    (define ET_REL		1)		; Relocatable file
    (define ET_EXEC		2)		; Executable file
    (define ET_DYN		3)		; Shared object file
    (define ET_CORE		4)		; Core file
    

    And if a module uses my (elf) module and references ET_DYN, I know that the module boundary doesn't prevent the value from being inlined as a constant (and possibly unboxed, etc).

    I took a look and on our usual microbenchmark suite, cross-module inlining doesn't make a difference. But that's both a historical oddity and a bug: firstly that the benchmark suite comes from an old Scheme world that didn't have modules, and so won't benefit from cross-module inlining. Secondly, Scheme definitions from the "default" environment that aren't explicitly recognized as primitives aren't inlined, as the (guile) module isn't declarative. (Probably we should fix the latter at some point.)

    But still, I'm really excited about this change! Guile developers use modules heavily and have been stepping around this optimization boundary for years. I count 100 direct uses of define-inlinable in Guile, a number of them inside macros, and many of these are to explicitly hack around the optimization barrier. I really look forward to seeing if we can remove some of these over time, to go back to plain old define and just trust the compiler to do what's needed.

    by the numbers

    I ran a quick analysis of the modules include in Guile to see what the impact was. Of the 321 files that define modules, 318 of them are declarative, and 88 contain inlinable exports (27% of total). Of the 6519 total bindings exported by declarative modules, 566 of those are inlinable (8.7%). Of the inlinable exports, 388 (69%) are functions (lambda expressions), 156 (28%) are constants, and 22 (4%) are "primitives" referenced by value and not by name, meaning definitions like (define head car) (instead of re-exporting car as head).

    On the use side, 90 declarative modules import inlinable bindings (29%), resulting in about 1178 total attempts to copy inlinable bindings. 902 of those attempts are to copy a lambda expressions in operator position, which means that peval will attempt to inline their code. 46 of these attempts fail, perhaps due to size or effort constraints. 191 other attempts end up inlining constant values. 20 inlining attempts fail, perhaps because a lambda is used for a value. Somehow, 19 copied inlinable values get elided because they are evaluated only for their side effects, probably to clean up let-bound values that become unused due to copy propagation.

    All in all, an interesting endeavor, and one to improve on going forward. Thanks for reading, and catch you next time!

    by Andy Wingo at May 13, 2021 11:25 AM

    May 11, 2021

    Enrique Ocaña GonzálezGStreamer WebKit debugging by instrumenting source code (3/3)

    This is the last post on the instrumenting source code series. I hope you to find the tricks below as useful as the previous ones.

    In this post I show some more useful debugging tricks. Don’t forget to have a look at the other posts of the series:

    Finding memory leaks in a RefCounted subclass

    The source code shown below must be placed in the .h where the class to be debugged is defined. It’s written in a way that doesn’t need to rebuild RefCounted.h, so it saves a lot of build time. It logs all refs, unrefs and adoptPtrs, so that any anomaly in the refcounting can be traced and investigated later. To use it, just make your class inherit from LoggedRefCounted instead of RefCounted.

    Example output:

    void WTF::adopted(WTF::LoggedRefCounted<T>*) [with T = WebCore::MediaSourceClientGStreamerMSE]: this=0x673c07a4, refCount 1
    void WTF::adopted(WTF::LoggedRefCounted<T>*) [with T = WebCore::MediaSourceClientGStreamerMSE]: this=0x673c07a4, refCount 1
    ^^^ Two adopts, this is not good.
    void WTF::LoggedRefCounted<T>::ref() [with T = WebCore::MediaSourceClientGStreamerMSE]: this=0x673c07a4, refCount 1 --> ...
    void WTF::LoggedRefCounted<T>::ref() [with T = WebCore::MediaSourceClientGStreamerMSE]: this=0x673c07a4, refCount ... --> 2
    void WTF::LoggedRefCounted<T>::deref() [with T = WebCore::MediaSourceClientGStreamerMSE]: this=0x673c07a4, refCount 2 --> ...
    void WTF::LoggedRefCounted<T>::deref() [with T = WebCore::MediaSourceClientGStreamerMSE]: this=0x673c07a4, refCount ... --> 1
    void WTF::adopted(WTF::LoggedRefCounted<T>*) [with T = WebCore::MediaSourceClientGStreamerMSE]: this=0x673c07a4, refCount 1
    void WTF::LoggedRefCounted<T>::deref() [with T = WebCore::MediaSourceClientGStreamerMSE]: this=0x673c07a4, refCount 1 --> ...
    void WTF::LoggedRefCounted<T>::deref() [with T = WebCore::MediaSourceClientGStreamerMSE]: this=0x673c07a4, refCount 1 --> ...
    ^^^ Two recursive derefs, not good either.
    #include "Logging.h"
    
    namespace WTF {
    
    template<typename T> class LoggedRefCounted : public WTF::RefCounted<T> {
        WTF_MAKE_NONCOPYABLE(LoggedRefCounted); WTF_MAKE_FAST_ALLOCATED;
    public:
        void ref() {
            printf("%s: this=%p, refCount %d --> ...\n", __PRETTY_FUNCTION__, this, WTF::RefCounted<T>::refCount()); fflush(stdout);
            WTF::RefCounted<T>::ref();
            printf("%s: this=%p, refCount ... --> %d\n", __PRETTY_FUNCTION__, this, WTF::RefCounted<T>::refCount()); fflush(stdout);
        }
    
        void deref() {
            printf("%s: this=%p, refCount %d --> ...\n", __PRETTY_FUNCTION__, this, WTF::RefCounted<T>::refCount()); fflush(stdout);
            WTF::RefCounted<T>::deref();
            printf("%s: this=%p, refCount ... --> %d\n", __PRETTY_FUNCTION__, this, WTF::RefCounted<T>::refCount()); fflush(stdout);
        }
    
    protected:
        LoggedRefCounted() { }
        ~LoggedRefCounted() { }
    };
    
    template<typename T> inline void adopted(WTF::LoggedRefCounted<T>* object)
    {
        printf("%s: this=%p, refCount %d\n", __PRETTY_FUNCTION__, object, (object)?object->refCount():0); fflush(stdout);
        adopted(static_cast<RefCountedBase*>(object));
    }
    
    } // Namespace WTF

    Pause WebProcess on launch

    WebProcessMainGtk and WebProcessMainWPE will sleep for 30 seconds if a special environment variable is defined:

    export WEBKIT2_PAUSE_WEB_PROCESS_ON_LAUNCH=1

    It only works #if ENABLE(DEVELOPER_MODE), so you might want to remove those ifdefs if you’re building in Release mode.

    Log tracers

    In big pipelines (e.g. playbin) it can be very hard to find what element is replying to a query or handling an event. Even using gdb can be extremely tedious due to the very high level of recursion. My coworker Alicia commented that using log tracers is more helpful in this case.

    GST_TRACERS=log enables additional GST_TRACE() calls all accross GStreamer. The following example logs entries and exits into the query function.

    GST_TRACERS=log GST_DEBUG='query:TRACE'

    The names of the logging categories are somewhat inconsistent:

    • log (the log tracer itself)
    • GST_BUFFER
    • GST_BUFFER_LIST
    • GST_EVENT
    • GST_MESSAGE
    • GST_STATES
    • GST_PADS
    • GST_ELEMENT_PADS
    • GST_ELEMENT_FACTORY
    • query
    • bin

    The log tracer code is in subprojects/gstreamer/plugins/tracers/gstlog.c.

    by eocanha at May 11, 2021 06:00 AM

    May 04, 2021

    Enrique Ocaña GonzálezGStreamer WebKit debugging by instrumenting source code (2/3)

    In this post I show some more useful debugging tricks. Check also the other posts of the series:

    Print current thread id

    The thread id is generated by Linux and can take values higher than 1-9, just like PIDs. This thread number is useful to know which function calls are issued by the same thread, avoiding confusion between threads.

    #include <stdio.h>
    #include <unistd.h>
    #include <sys/syscall.h>
    
    printf("%s [%d]\n", __PRETTY_FUNCTION__, syscall(SYS_gettid));
    fflush(stdout);

    Debug GStreamer thread locks

    We redefine the GST_OBJECT_LOCK/UNLOCK/TRYLOCK macros to print the calls, compare locks against unlocks, and see who’s not releasing its lock:

    #include "wtf/Threading.h"
    #define GST_OBJECT_LOCK(obj) do { \
      printf("### [LOCK] %s [%p]\n", __PRETTY_FUNCTION__, &Thread::current()); fflush(stdout); \
      g_mutex_lock(GST_OBJECT_GET_LOCK(obj)); \
    } while (0)
    #define GST_OBJECT_UNLOCK(obj) do { \
      printf("### [UNLOCK] %s [%p]\n", __PRETTY_FUNCTION__, &Thread::current()); fflush(stdout); \
      g_mutex_unlock(GST_OBJECT_GET_LOCK(obj)); \
    } while (0)
    #define GST_OBJECT_TRYLOCK(obj) ({ \
      gboolean result = g_mutex_trylock(GST_OBJECT_GET_LOCK(obj)); \
      if (result) { \
       printf("### [LOCK] %s [%p]\n", __PRETTY_FUNCTION__, &Thread::current()); fflush(stdout); \
      } \
      result; \
    })

    Warning: The statement expression that allows the TRYLOCK macro to return a value will only work on GCC.

    There’s a way to know which thread has taken a lock in glib/GStreamer using gdb. First locate the stalled thread:

    (gdb) thread 
    (gdb) bt
    #2  0x74f07416 in pthread_mutex_lock ()
    #3  0x7488aec6 in gst_pad_query ()
    #4  0x6debebf2 in autoplug_query_allocation ()
    
    (gdb) frame 3
    #3  0x7488aec6 in gst_pad_query (pad=pad@entry=0x54a9b8, ...)
    4058        GST_PAD_STREAM_LOCK (pad);

    Now get the process id (PID) and use the pthread_mutex_t structure to print the Linux thread id that has acquired the lock:

    (gdb) call getpid()
    $30 = 6321
    (gdb) p ((pthread_mutex_t*)pad.stream_rec_lock.p)->__data.__owner
    $31 = 6368
    (gdb) thread find 6321.6368
    Thread 21 has target id 'Thread 6321.6368'

    Trace function calls (poor developer version)

    If you’re using C++, you can define a tracer class. This is for webkit, but you get the idea:

    #define MYTRACER() MyTracer(__PRETTY_FUNCTION__);
    class MyTracer {
    public:
        MyTracer(const gchar* functionName)
          : m_functionName(functionName) {
          printf("### %s : begin %d\n", m_functionName.utf8().data(), currentThread()); fflush(stdout);
        }
        virtual ~MyTracer() {
            printf("### %s : end %d\n", m_functionName.utf8().data(), currentThread()); fflush(stdout);
        }
    private:
        String m_functionName;
    };

    And use it like this in all the functions you want to trace:

    void somefunction() {
      MYTRACER();
      // Some other code...
    }

    The constructor will log when the execution flow enters into the function and the destructor will log when the flow exits.

    Setting breakpoints from C

    In the C code, just call raise(SIGINT) (simulate CTRL+C, normally the program would finish).

    And then, in a previously attached gdb, after breaking and having debugging all you needed, just continue the execution by ignoring the signal or just plainly continuing:

    (gdb) signal 0
    (gdb) continue

    There’s a way to do the same but attaching gdb after the raise. Use raise(SIGSTOP) instead (simulate CTRL+Z). Then attach gdb, locate the thread calling raise and switch to it:

    (gdb) thread apply all bt
    [now search for "raise" in the terminal log]
    Thread 36 (Thread 1977.2033): #1 0x74f5b3f2 in raise () from /home/enrique/buildroot/output2/staging/lib/libpthread.so.0
    (gdb) thread 36

    Now, from a terminal, send a continuation signal: kill -SIGCONT 1977. Finally instruct gdb to single-step only the current thread (IMPORTANT!) and run some steps until all the raises have been processed:

    (gdb) set scheduler-locking on
    (gdb) next    // Repeat several times...

    Know the name of a GStreamer function stored in a pointer at runtime

    Just use this macro:

    GST_DEBUG_FUNCPTR_NAME(func)

    Detecting memory leaks in WebKit

    RefCountedLeakCounter is a tool class that can help to debug reference leaks by printing this kind of messages when WebKit exits:

      LEAK: 2 XMLHttpRequest
      LEAK: 25 CachedResource
      LEAK: 3820 WebCoreNode

    To use it you have to modify the particular class you want to debug:

    • Include wtf/RefCountedLeakCounter.h
    • DEFINE_DEBUG_ONLY_GLOBAL(WTF::RefCountedLeakCounter, myClassCounter, ("MyClass"));
    • In the constructor: myClassCounter.increment()
    • In the destructor: myClassCounter.decrement()

    by eocanha at May 04, 2021 06:00 AM

    May 03, 2021

    Jean-François Fortin TamIntroducing Regento, marketing for FLOSS-centric companies and transitioning industries

    In this blog post, I’m taking a quick break from my GTG blogging frenzy to talk about another one of my non-software projects from the past few months years (estimated reading time: 3 ½ minutes).


    Some may remember I previously introduced the Atypica collective as a dedicated place where I could showcase some of my video production work (instead of having everything and the kitchen sink into my own website).

    Launching Atypica was a long-standing project of mine that had been put on the back-burner because of some of my intensive CMO work in recent years (such as this, for instance). Awakening from the lucid dream allowed me to re-enter a long R&D phase where I could not only shave a ton of “infrastructure and productivity” yaks (that’s a story for another blog post, including the ridiculous tale of the months of work it took to fix my personal blog), but also realign my business objectives and pick up where I had left off “the last time”, hence the delayed announcement of Atypica’s website launch.

    But Atypica, besides being the kind of initiative that works much better outside a pandemic, is merely one of my creative interests, and therefore part of my broader business orientations.


    Today I’m revealing part two (of potentially many): Regento, the fractional CMO agency regency. You could say we are Riskbreakers, but in the business sense. Because in business, it’s all fun and games until an expensive wyvern loses an eye.

    We break down silos and reduce risk. Although we can dream of being digital nomads some day, this is no Vagrant Story.

    “Why another brand, don’t you already do idéemarque,” you ask? Because, in addition to various practical and technical considerations:

    • This area of work is less about creative services and more about deeply involved strategic marketing and strategic business consulting, rather than what you’d find in a typical tactical agency service offering.
    • It allows me to articulate my specific “CMO for hire” offering in a simplified way, instead of overloading again the scope of my personal homepage
      • Until now, the feedback I received from some of the people who looked at my personal homepage was, “You do too many things, it’s unbelievable… therefore we don’t believe you, or don’t understand what you do”.
      • The thing is, my personal home page is a reflection of who I am as a human, laid bare for anyone who cares to dig… yet visitors landing on that page may all have different interests from each other (for example, some will only care about my open-source work, some will only care about my curriculum, some only care about my businesses, and some weird people might actually be interested in seeing my old illustration works), so it can’t really niche down that much without watering down some of its purpose and personality. As major Kusanagi once said, overspecialization is slow death.
    • Separating Regento from the rest lets me build different kinds of partnerships, the kind that are otherwise not available in a typical agency setting. Different team, different clientèle. And indeed, I am not alone in this endeavour (more on that below).

    Double Regento all the way?
    But what does it mean?! 🌈

    One of the main reasons behind this name is that “regento” is the esperanto word for regent, which matches how I see my services as an “interim CMO for hire”, if you will pardon the royal metaphor here: I take care of your kingdom (business) by structuring and strengthening it until the heir (my replacement) comes of age and takes over at some point in time. This is, I posit, one of the most interesting ways for me to make an impact on the world of business: by helping multiple businesses flourish instead of through one endless reign in Leá Monde one industry.

    When it comes to strategic work, I am very picky about who I surround myself with, so I am super excited to highlight the fact that—as can be seen in the Regento about/team pageI have partnered up with Halle Baksh, a good friend of mine whose business experience and wisdom I have great admiration for; our personalities also have great affinity. She is a wonderful human being (not a rogue A.I. like me) and a kindred spirit, and I look forward to continue working with her on making the world a slightly better place, one client’s business at a time.

    Watch that space!

    Although the Regento initiative has been going on since June 2019, it is only now that I’m announcing its existence (again: yak shavings). The Regento website is pretty embryonic, you could almost say it’s a placeholder, but it’s better to have something than to wait for perfection. The design is not where I would want it to be, the contents are “bare minimum” (there are only two articles published for now, and no case studies and testimonials have been written yet), but “perfect” is the enemy of “good”, and all will come in due time.


    If you are interested in insights about business management & business development best practices, high-tech startups (the good, the bad, the ugly), sustainable businesses, or any other topic you may suggest (send me an email!), you may decide to subscribe to the contents email notification list on Regento’s website. As usual: no spam, no data capitalism, just the convenience of not having to remember to check periodically “manually” for updates. And we always enjoy knowing that whatever we write gets read.

    by Jeff at May 03, 2021 02:36 AM

    April 28, 2021

    Nirbheek ChauhanGStreamer has grown a WebRTC implementation

    In other news, GStreamer is now almost buzzword-compliant! The next blog post on our list: blockchains and smart contracts in GStreamer.

    Late last year, we at Centricular announced a new implementation of WebRTC in GStreamer.  Today we're happy to announce that after community review, that work has been merged into GStreamer itself! The plugin is called webrtcbin, and the library is, naturally, called gstwebrtc.

    The implementation has all the basic features, is transparently compatible with other WebRTC stacks (particularly in browsers), and has been well-tested with both Firefox and Chrome.

    Some of the more advanced features such as FEC are already a work in progress, and others will be too—if you want them to be! Hop onto IRC on #gstreamer @ Freenode.net or join the mailing list.

    How do I use it?


    Currently, the easiest way to use webrtcbin is to build GStreamer using either gst-uninstalled (Linux and macOS) or Cerbero (Windows, iOS, Android). If you're a patient person, you can follow @gstreamer and wait for GStreamer 1.14 to be released which will include Windows, macOS, iOS, and Android binaries.

    The API currently lacks documentation, so the best way to learn it is to dive into the source-tree examples. Help on this will be most appreciated! To see how to use GStreamer to do WebRTC with a browser, checkout the bidirectional audio-video demos.

    Show me the code! [skip]


    Here's a quick highlight of the important bits that should get you started if you already know how GStreamer works. This example is in C, but GStreamer also has bindings for Rust, Python, Java, C#, Vala, and so on.

    Let's say you want to capture video from V4L2, stream it to a webrtc peer, and receive video back from it. The first step is the streaming pipeline, which will look something like this:

    v4l2src ! queue ! vp8enc ! rtpvp8pay !
    application/x-rtp,media=video,encoding-name=VP8,payload=96 !
    webrtcbin name=sendrecv

    As a short-cut, let's parse the string description to create the pipeline.

    1
    2
    3
    4
    5
    GstElement *pipe;

    pipe = gst_parse_launch ("v4l2src ! queue ! vp8enc ! rtpvp8pay ! "
    "application/x-rtp,media=video,encoding-name=VP8,payload=96 !"
    " webrtcbin name=sendrecv", NULL);

    Next, we get a reference to the webrtcbin element and attach some callbacks to it.

     1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    GstElement *webrtc;

    webrtc = gst_bin_get_by_name (GST_BIN (pipe), "sendrecv");
    g_assert (webrtc != NULL);

    /* This is the gstwebrtc entry point where we create the offer.
    * It will be called when the pipeline goes to PLAYING. */
    g_signal_connect (webrtc, "on-negotiation-needed",
    G_CALLBACK (on_negotiation_needed), NULL);
    /* We will transmit this ICE candidate to the remote using some
    * signalling. Incoming ICE candidates from the remote need to be
    * added by us too. */
    g_signal_connect (webrtc, "on-ice-candidate",
    G_CALLBACK (send_ice_candidate_message), NULL);
    /* Incoming streams will be exposed via this signal */
    g_signal_connect (webrtc, "pad-added",
    G_CALLBACK (on_incoming_stream), pipe);
    /* Lifetime is the same as the pipeline itself */
    gst_object_unref (webrtc);

    When the pipeline goes to PLAYING, the on_negotiation_needed() callback will be called, and we will ask webrtcbin to create an offer which will match the pipeline above.

     1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    static void
    on_negotiation_needed (GstElement * webrtc, gpointer user_data)
    {
    GstPromise *promise;

    promise = gst_promise_new_with_change_func (on_offer_created,
    user_data, NULL);
    g_signal_emit_by_name (webrtc, "create-offer", NULL,
    promise);
    }

    When webrtcbin has created the offer, it will call on_offer_created()

     1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    static void
    on_offer_created (GstPromise * promise, GstElement * webrtc)
    {
    GstWebRTCSessionDescription *offer = NULL;
    const GstStructure *reply;
    gchar *desc;

    reply = gst_promise_get_reply (promise);
    gst_structure_get (reply, "offer",
    GST_TYPE_WEBRTC_SESSION_DESCRIPTION,
    &offer, NULL);
    gst_promise_unref (promise);

    /* We can edit this offer before setting and sending */
    g_signal_emit_by_name (webrtc,
    "set-local-description", offer, NULL);

    /* Implement this and send offer to peer using signalling */
    send_sdp_offer (offer);
    gst_webrtc_session_description_free (offer);
    }

    Similarly, when we have the SDP answer from the remote, we must call "set-remote-description" on webrtcbin.

    1
    2
    3
    4
    5
    6
    7
    answer = gst_webrtc_session_description_new (
    GST_WEBRTC_SDP_TYPE_ANSWER, sdp);
    g_assert (answer);

    /* Set remote description on our pipeline */
    g_signal_emit_by_name (webrtc, "set-remote-description",
    answer, NULL);

    ICE handling is very similar; when the "on-ice-candidate" signal is emitted, we get a local ICE candidate which we must send to the remote. When we have an ICE candidate from the remote, we must call "add-ice-candidate" on webrtcbin.

    There's just one piece left now; handling incoming streams that are sent by the remote. For that, we have on_incoming_stream() attached to the "pad-added" signal on webrtcbin.

     1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    static void
    on_incoming_stream (GstElement * webrtc, GstPad * pad,
    GstElement * pipe)
    {
    GstElement *play;

    play = gst_parse_bin_from_description (
    "queue ! vp8dec ! videoconvert ! autovideosink",
    TRUE, NULL);
    gst_bin_add (GST_BIN (pipe), play);

    /* Start displaying video */
    gst_element_sync_state_with_parent (play);
    gst_element_link (webrtc, play);
    }

    That's it! This is what a basic webrtc workflow looks like. Those of you that have used the PeerConnection API before will be happy to see that this maps to that quite closely.

    The aforementioned demos also include a Websocket signalling server and JS browser components, and I will be doing an in-depth application newbie developer's guide at a later time, so you can follow me @nirbheek to hear when it comes out!

    Tell me more!


    The code is already being used in production in a number of places, such as EasyMile's autonomous vehicles, and we're excited to see where else the community can take it.

    If you're wondering why we decided a new implementation was needed, read on! For a more detailed discussion into that, you should watch Matthew Waters' talk from the GStreamer conference last year. It's a great companion for this article!

    But before we can dig into details, we need to lay some foundations first.

    What is GStreamer, and what is WebRTC? [skip]


    GStreamer is a cross-platform open-source multimedia framework that is, in my opinion, the easiest and most flexible way to implement any application that needs to play, record, or transform media-like data across an extremely versatile scale of devices and products. Embedded (IoT, IVI, phones, TVs, …), desktop (video/music players, video recording, non-linear editing, videoconferencing and VoIP clients, browsers …), to servers (encode/transcode farms, video/voice conferencing servers, …) and more.

    But what I like the most about GStreamer is the pipeline-based model which solves one of the hardest problems in API design: catering to applications of varying complexity; from the simplest one-liners and quick solutions to those that need several hundreds of thousands of lines of code to implement their full featureset. 

    If you want to learn more about GStreamer, Jan Schmidt's tutorial from Linux.conf.au is a good start.

    WebRTC is a set of draft specifications that build upon existing RTP, RTCP, SDP, DTLS, ICE (and many other) real-time communication specifications and defines an API for making RTC accessible using browser JS APIs.

    People have been doing real-time communication over IP for decades with the previously-listed protocols that WebRTC builds upon. The real innovation of WebRTC was creating a bridge between native applications and webapps by defining a standard, yet flexible, API that browsers can expose to untrusted JavaScript code.

    These specifications are constantly being improved upon, which combined with the ubiquitous nature of browsers means WebRTC is fast becoming the standard choice for videoconferencing on all platforms and for most applications.

    Everything is great, let's build amazing apps! [skip]


    Not so fast, there's more to the story! For WebApps, the PeerConnection API is everywhere. There are some browser-specific quirks as usual, and the API itself keeps changing, but the WebRTC JS adapter handles most of that. Overall the WebApp experience is mostly 👍.

    Sadly, for native code or applications that need more flexibility than a sandboxed JS app can achieve, there haven't been a lot of great options.

    libwebrtc (Chrome's implementation), Janus, Kurento, and OpenWebRTC have traditionally been the main contenders, but after having worked with all of these, we found that each implementation has its own inflexibilities, shortcomings, and constraints.

    libwebrtc is still the most mature implementation, but it is also the most difficult to work with. Since it's embedded inside Chrome, it's a moving target, the API can be hard to work with, and the project is quite difficult to build and integrate, all of which are obstacles in the way of native or server app developers trying to quickly prototype and try out things.

    It was also not built for multimedia use-cases, so while the webrtc bits are great, the lower layers get in the way of non-browser use-cases and applications. It is quite painful to do anything other than the default "set raw media, transmit" and "receive from remote, get raw media". This means that if you want to use your own filters, or hardware-specific codecs or sinks/sources, you end up having to fork libwebrtc.

    In contrast, as shown above, our implementation gives you full control over this as with any other GStreamer pipeline.

    OpenWebRTC by Ericsson was the first attempt to rectify this situation, and it was built on top of GStreamer. The target audience was app developers, and it fit the bill quite well as a proof-of-concept—even though it used a custom API and some of the architectural decisions made it quite inflexible for most other use-cases.

    However, after an initial flurry of activity around the project, momentum petered out, the project failed to gather a community around itself, and is now effectively dead.

    Full disclosure: we worked with Ericsson to polish some of the rough edges around the project immediately prior to its public release.

    WebRTC in GStreamer — webrtcbin and gstwebrtc


    Remember how I said the WebRTC standards build upon existing standards and protocols? As it so happens, GStreamer has supported almost all of them for a while now because they were being used for real-time communication, live streaming, and in many other IP-based applications. Indeed, that's partly why Ericsson chose it as the base for OWRTC.

    This combined with the SRTP and DTLS plugins that were written during OWRTC's development meant that our implementation is built upon a solid and well-tested base, and that implementing WebRTC features is not as difficult as one might presume. However, WebRTC is a large collection of standards, and reaching feature-parity with libwebrtc is an ongoing task.

    Lucky for us, Matthew made some excellent decisions while architecting the internals of webrtcbin, and we follow the PeerConnection specification quite closely, so almost all the missing features involve writing code that would plug into clearly-defined sockets.

    We believe what we've been building here is the most flexible, versatile, and easy to use WebRTC implementation out there, and it can only get better as time goes by. Bringing the power of pipeline-based multimedia manipulation to WebRTC opens new doors for interesting, unique, and highly efficient applications.

    To demonstrate this, in the near future we will be publishing articles that dive into how to use the PeerConnection-inspired API exposed by webrtcbin to build various kinds of applications—starting with a CPU-efficient multi-party bidirectional conferencing solution with a mesh topology that can work with any webrtc stack.

    Until next time!

    by Nirbheek (noreply@blogger.com) at April 28, 2021 03:09 PM

    April 27, 2021

    Enrique Ocaña GonzálezGStreamer WebKit debugging by instrumenting source code (1/3)

    This is the continuation of the GStreamer WebKit debugging tricks post series. In the next three posts, I’ll focus on what we can get by doing some little changes to the source code for debugging purposes (known as “instrumenting”), but before, you might want to check the previous posts of the series:

    Know all the env vars read by a program by using LD_PRELOAD to intercept libc calls

    // File getenv.c
    // To compile: gcc -shared -Wall -fPIC -o getenv.so getenv.c -ldl
    // To use: export LD_PRELOAD="./getenv.so", then run any program you want
    // See http://www.catonmat.net/blog/simple-ld-preload-tutorial-part-2/
    
    #define _GNU_SOURCE
    
    #include <stdio.h>
    #include <dlfcn.h>
    
    // This function will take the place of the original getenv() in libc
    char *getenv(const char *name) {
     printf("Calling getenv(\"%s\")\n", name);
    
     char *(*original_getenv)(const char*);
     original_getenv = dlsym(RTLD_NEXT, "getenv");
    
     return (*original_getenv)(name);
    }

    See the breakpoints with command example to know how to get the same using gdb. Check also Zan’s libpine for more features.

    Track lifetime of GObjects by LD_PRELOADing gobject-list

    The gobject-list project, written by Thibault Saunier, is a simple LD_PRELOAD library for tracking the lifetime of GObjects. When loaded into an application, it prints a list of living GObjects on exiting the application (unless the application crashes), and also prints reference count data when it changes. SIGUSR1 or SIGUSR2 can be sent to the application to trigger printing of more information.

    Overriding the behaviour of a debugging macro

    The usual debugging macros aren’t printing messages? Redefine them to make what you want:

    #undef LOG_MEDIA_MESSAGE
    #define LOG_MEDIA_MESSAGE(...) do { \
      printf("LOG %s: ", __PRETTY_FUNCTION__); \
      printf(__VA_ARGS__); \
      printf("\n"); \
      fflush(stdout); \
    } while(0)

    This can be done to enable asserts on demand in WebKit too:

    #undef ASSERT
    #define ASSERT(assertion) \
      (!(assertion) ? \
          (WTFReportAssertionFailure(__FILE__, __LINE__, WTF_PRETTY_FUNCTION, #assertion), \
           CRASH()) : \
          (void)0)
    
    #undef ASSERT_NOT_REACHED
    #define ASSERT_NOT_REACHED() do { \
      WTFReportAssertionFailure(__FILE__, __LINE__, WTF_PRETTY_FUNCTION, 0); \
      CRASH(); \
    } while (0)

    It may be interesting to enable WebKit LOG() and GStreamer GST_DEBUG() macros only on selected files:

    #define LOG(channel, msg, ...) do { \
      printf("%s: ", #channel); \
      printf(msg, ## __VA_ARGS__); \
      printf("\n"); \
      fflush(stdout); \
    } while (false)
    
    #define _GST_DEBUG(msg, ...) do { \
      printf("### %s: ", __PRETTY_FUNCTION__); \
      printf(msg, ## __VA_ARGS__); \
      printf("\n"); \
      fflush(stdout); \
    } while (false)

    Note all the preprocessor trickery used here:

    • First arguments (channel, msg) are captured intependently
    • The remaining args are captured in __VA_ARGS__
    • do while(false) is a trick to avoid {braces} and make the code block work when used in if/then/else one-liners
    • #channel expands LOG(MyChannel,....) as printf("%s: ", "MyChannel"). It’s called “stringification”.
    • ## __VA_ARGS__ expands the variable argument list as a comma-separated list of items, but if the list is empty, it eats the comma after “msg”, preventing syntax errors

    Print the compile-time type of an expression

    Use typeid(<expression>).name(). Filter the ouput through c++filt -t:

    std::vector<char *> v; 
    printf("Type: %s\n", typeid(v.begin()).name());

    Abusing the compiler to know all the places where a function is called

    If you want to know all the places from where the GstClockTime toGstClockTime(float time) function is called, you can convert it to a template function and use static_assert on a wrong datatype like this (in the .h):

    template <typename T = float> GstClockTime toGstClockTime(float time) { 
      static_assert(std::is_integral<T>::value,
        "Don't call toGstClockTime(float)!");
      return 0;
    }

    Note that T=float is different to integer (is_integral). It has nothing to do with the float time parameter declaration.

    You will get compile-time errors like this on every place the function is used:

    WebKitMediaSourceGStreamer.cpp:474:87:   required from here
    GStreamerUtilities.h:84:43: error: static assertion failed: Don't call toGstClockTime(float)!

    Use pragma message to print values at compile time

    Sometimes is useful to know if a particular define is enabled:

    #include <limits.h>
    
    #define _STR(x) #x
    #define STR(x) _STR(x)
    
    #pragma message "Int max is " STR(INT_MAX)
    
    #ifdef WHATEVER
    #pragma message "Compilation goes by here"
    #else
    #pragma message "Compilation goes by there"
    #endif
    
    ...

    The code above would generate this output:

    test.c:6:9: note: #pragma message: Int max is 0x7fffffff
     #pragma message "Int max is " STR(INT_MAX)
             ^~~~~~~
    test.c:11:9: note: #pragma message: Compilation goes by there
     #pragma message "Compilation goes by there"
             ^~~~~~~

    by eocanha at April 27, 2021 06:00 AM

    April 21, 2021

    Víctor JáquezReview of Igalia Multimedia activities (2020/H2)

    As the first quarter of 2021 has aready come to a close, we reckon it’s time to recap our achievements from the second half of 2020, and update you on the improvements we have been making to the multimedia experience on the Web and Linux in general.

    Our previous reports:

    WPE / WebKitGTK

    We have closed ~100 issues related with multimedia in WebKitGTK/WPE, such as fixed seek issues while playback, plugged memory leaks, gardening tests, improved Flatpak-based developing work-flow, enabled new codecs, etc.. Overall, we improved a bit the multimedia’s user experience on these Webkit engine ports.

    To highlight a couple tasks, we did some maintenance work on WebAudio backends, and we upstreamed an internal audio mixer, keeping only one connection to the audio server, such as PulseAudio, instead of multiple connections, one for every audio resource. The mixer combines all streams into a single audio server connection.

    Adaptive media streaming for the Web (MSE)

    We have been working on a new MSE backend for a while, but along the way many related bugs have appeared and they were squashed. Also many code cleanups has been carried out. Though it has been like yak shaving, we are confident that we will reach the end of this long and winding road soonish.

    DRM media playback for the Web (EME)

    Regarding digital protected media playback, we worked to upstream OpenCDM, support with Widevine, through RDK’s Thunder framework, while continued with the usual maintenance of the others key systems, such as Clear Key, Widevine and PlayReady.

    For more details we published a blog post: Serious Encrypted Media Extensions on GStreamer based WebKit ports.

    Realtime communications for the Web (WebRTC)

    Just as EME, WebRTC is not currently enabled by default in browsers such as Epiphany because license problems, but they are available for custom adopters, and we are maintaining it. For example, we collaborated to upgrade LibWebRTC to M87 and fixed the expected regressions and gardening.

    Along the way we experimented a bit with the new GPUProcess for capture devices, but we decided to stop the experimentation while waiting for a broader adoption of the process, for example in graphics rendering, in WPE/WebKitGTK.

    GPUProcess work will be retaken at some point, because it’s not, currently, a hard requirement, since we already have moved capture devices handling from the UIProcess to the WebProcess, isolating all GStreamer operations in the latter.

    GStreamer

    GStreamer is one of our core multimedia technologies, and we contribute on it on a daily basis. We pushed ~400 commits, with similar number of code reviews, along the second half of 2020. Among of those contributions let us highlight the following list:

    • A lot of bug fixing aiming for release 1.18.
    • Reworked and enhanced decodebin3, the GstTranscoder
      API
      and encodebin.
    • Merged av1parse in video parsers plugin.
    • Merged qroverlay plugin.
    • Iterated on the mono-repo
      proposal, which requires consensus and coordination among the whole community.
    • gstwpe element has been greatly improved from new user requests.
    • Contributed on the new libgstcodecs library, which enables stateless video decoders through different platforms (for example, v4l2, d3d11, va, etc.).
    • Developed a new plugin for VA-API using this library, exposing H.264, H.265, VP9, VP8, MPEG2 decoders and a full featured postprocessor, with better performance, according our measurements, than GStreamer-VAAPI.

    Conferences

    Despite 2020 was not a year for conferences, many of them went virtual. We attended one, the Mile high video conference, and participated in the Slack workspace.

    Thank you for reading this report and stay tuned with our work.

    by vjaquez at April 21, 2021 04:49 AM

    April 20, 2021

    Enrique Ocaña GonzálezGStreamer WebKit debugging tricks using GDB (2/2)

    This post is a continuation of a series of blog posts about the most interesting debugging tricks I’ve found while working on GStreamer WebKit on embedded devices. These are the other posts of the series published so far:

    Print corrupt stacktraces

    In some circumstances you may get stacktraces that eventually stop because of missing symbols or corruption (?? entries).

    #3  0x01b8733c in ?? ()
    Backtrace stopped: previous frame identical to this frame (corrupt stack?)

    However, you can print the stack in a useful way that gives you leads about what was next in the stack:

    • For i386: x/256wa $esp
    • For x86_64: x/256ga $rsp
    • For ARM 32 bit: x/256wa $sp

    You may want to enable asm-demangle: set print asm-demangle

    Example output, the 3 last lines give interesting info:

    0x7ef85550:     0x1b87400       0x2     0x0     0x1b87400
    0x7ef85560:     0x0     0x1b87140       0x1b87140       0x759e88a4
    0x7ef85570:     0x1b87330       0x759c71a9 <gst_base_sink_change_state+956>     0x140c  0x1b87330
    0x7ef85580:     0x759e88a4      0x7ef855b4      0x0     0x7ef855b4
    ...
    0x7ef85830:     0x76dbd6c4 <WebCore::AppendPipeline::resetPipeline()::__PRETTY_FUNCTION__>        0x4     0x3     0x1bfeb50
    0x7ef85840:     0x0     0x76d59268      0x75135374      0x75135374
    0x7ef85850:     0x76dbd6c4 <WebCore::AppendPipeline::resetPipeline()::__PRETTY_FUNCTION__>        0x1b7e300       0x1d651d0       0x75151b74

    More info: 1

    Sometimes the symbol names aren’t printed in the stack memdump. You can do this trick to iterate the stack and print the symbols found there (take with a grain of salt!):

    (gdb) set $i = 0
    (gdb) p/a *((void**)($sp + 4*$i++))
    
    [Press ENTER multiple times to repeat the command]
    
    $46 = 0xb6f9fb17 <_dl_lookup_symbol_x+250>
    $58 = 0xb40a9001 <g_log_writer_standard_streams+128>
    $142 = 0xb40a877b <g_return_if_fail_warning+22>
    $154 = 0xb65a93d5 <WebCore::MediaPlayerPrivateGStreamer::changePipelineState(GstState)+180>
    $164 = 0xb65ab4e5 <WebCore::MediaPlayerPrivateGStreamer::playbackPosition() const+420>
    ...

    Many times it’s just a matter of gdb not having loaded the unstripped version of the library. /proc/<PID>/smaps and info proc mappings can help to locate the library providing the missing symbol. Then we can load it by hand.

    For instance, for this backtrace:

    #0  0x740ad3fc in syscall () from /home/enrique/buildroot-wpe/output/staging/lib/libc.so.6 
    #1  0x74375c44 in g_cond_wait () from /home/enrique/buildroot-wpe/output/staging/usr/lib/libglib-2.0.so.0 
    #2  0x6cfd0d60 in ?? ()

    In a shell, we examine smaps and find out that the unknown piece of code comes from libgstomx:

    $ cat /proc/715/smaps
    ...
    6cfc1000-6cff8000 r-xp 00000000 b3:02 785380     /usr/lib/gstreamer-1.0/libgstomx.so
    ...

    Now we load the unstripped .so in gdb and we’re able to see the new symbol afterwards:

    (gdb) add-symbol-file /home/enrique/buildroot-wpe/output/build/gst-omx-custom/omx/.libs/libgstomx.so 0x6cfc1000
    (gdb) bt
    #0  0x740ad3fc in syscall () from /home/enrique/buildroot-wpe/output/staging/lib/libc.so.6
    #1  0x74375c44 in g_cond_wait () from /home/enrique/buildroot-wpe/output/staging/usr/lib/libglib-2.0.so.0
    #2  0x6cfd0d60 in gst_omx_video_dec_loop (self=0x6e0c8130) at gstomxvideodec.c:1311
    #3  0x6e0c8130 in ?? ()

    Useful script to prepare the add-symbol-file:

    cat /proc/715/smaps | grep '[.]so' | sed -e 's/-[0-9a-f]*//' | { while read ADDR _ _ _ _ LIB; do echo "add-symbol-file $LIB 0x$ADDR"; done; }

    More info: 1

    The “figuring out corrupt ARM stacktraces” post has some additional info about how to use addr2line to translate memory addresses to function names on systems with a hostile debugging environment.

    Debugging a binary without debug symbols

    There are times when there’s just no way to get debug symbols working, or where we’re simply debugging on a release version of the software. In those cases, we must directly debug the assembly code. The gdb text user interface (TUI) can be used to examine the disassebled code and the CPU registers. It can be enabled with these commands:

    layout asm
    layout regs
    set print asm-demangle

    Some useful keybindings in this mode:

    • Arrows: scroll the disassemble window
    • CTRL+p/n: Navigate history (previously done with up/down arrows)
    • CTRL+b/f: Go backward/forward one character (previously left/right arrows)
    • CTRL+d: Delete character (previously “Del” key)
    • CTRL+a/e: Go to the start/end of the line

    This screenshot shows how we can infer that an empty RefPtr is causing a crash in some WebKit code.

    Wake up an unresponsive gdb on ARM

    Sometimes, when you continue (‘c’) execution on ARM there’s no way to stop it again unless a breakpoint is hit. But there’s a trick to retake the control: just send a harmless signal to the process.

    kill -SIGCONT 1234

    Know which GStreamer thread id matches with each gdb thread

    Sometimes you need to match threads in the GStreamer logs with threads in a running gdb session. The simplest way is to ask it to GThread for each gdb thread:

    (gdb) set output-radix 16
    (gdb) thread apply all call g_thread_self()

    This will print a list of gdb threads and GThread*. We only need to find the one we’re looking for.

    Generate a pipeline dump from gdb

    If we have a pointer to the pipeline object, we can call the function that dumps the pipeline:

    (gdb) call gst_debug_bin_to_dot_file_with_ts((GstBin*)0x15f0078, GST_DEBUG_GRAPH_SHOW_ALL, "debug")

    by eocanha at April 20, 2021 06:00 AM

    April 13, 2021

    Enrique Ocaña GonzálezGStreamer WebKit debugging tricks using GDB (1/2)

    I’ve been developing and debugging desktop and mobile applications on embedded devices over the last decade or so. The main part of this period I’ve been focused on the multimedia side of the WebKit ports using GStreamer, an area that is a mix of C (glib, GObject and GStreamer) and C++ (WebKit).

    Over these years I’ve had to work on ARM embedded devices (mobile phones, set-top-boxes, Raspberry Pi using buildroot) where most of the environment aids and tools we take for granted on a regular x86 Linux desktop just aren’t available. In these situations you have to be imaginative and find your own way to get the work done and debug the issues you find in along the way.

    I’ve been writing down the most interesting tricks I’ve found in this journey and I’m sharing them with you in a series of 7 blog posts, one per week. Most of them aren’t mine, and the ones I learnt in the begining of my career can even seem a bit naive, but I find them worth to share anyway. I hope you find them as useful as I do.

    Breakpoints with command

    You can break on a place, run some command and continue execution. Useful to get logs:

    break getenv
    command
     # This disables scroll continue messages
     # and supresses output
     silent
     set pagination off
     p (char*)$r0
    continue
    end
    
    break grl-xml-factory.c:2720 if (data != 0)
    command
     call grl_source_get_id(data->source)
     # $ is the last value in the history, the result of
     # the previous call
     call grl_media_set_source (send_item->media, $)
     call grl_media_serialize_extended (send_item->media, 
      GRL_MEDIA_SERIALIZE_FULL)
     continue
    end

    This idea can be combined with watchpoints and applied to trace reference counting in GObjects and know from which places the refcount is increased and decreased.

    Force execution of an if branch

    Just wait until the if chooses a branch and then jump to the other one:

    6 if (i > 3) {
    (gdb) next
    7 printf("%d > 3\n", i);
    (gdb) break 9
    (gdb) jump 9
    9 printf("%d <= 3\n", i);
    (gdb) next
    5 <= 3

    Debug glib warnings

    If you get a warning message like this:

    W/GLib-GObject(18414): g_object_unref: assertion `G_IS_OBJECT (object)' failed

    the functions involved are: g_return_if_fail_warning(), which calls to g_log(). It’s good to set a breakpoint in any of the two:

    break g_log

    Another method is to export G_DEBUG=fatal_criticals, which will convert all the criticals in crashes, which will stop the debugger.

    Debug GObjects

    If you want to inspect the contents of a GObjects that you have in a reference…

    (gdb) print web_settings 
    $1 = (WebKitWebSettings *) 0x7fffffffd020

    you can dereference it…

    (gdb) print *web_settings
    $2 = {parent_instance = {g_type_instance = {g_class = 0x18}, ref_count = 0, qdata = 0x0}, priv = 0x0}

    even if it’s an untyped gpointer…

    (gdb) print user_data
    (void *) 0x7fffffffd020
    (gdb) print *((WebKitWebSettings *)(user_data))
    {parent_instance = {g_type_instance = {g_class = 0x18}, ref_count = 0, qdata = 0x0}, priv = 0x0}

    To find the type, you can use GType:

    (gdb) call (char*)g_type_name( ((GTypeInstance*)0x70d1b038)->g_class->g_type )
    $86 = 0x2d7e14 "GstOMXH264Dec-omxh264dec"

    Instantiate C++ object from gdb

    (gdb) call malloc(sizeof(std::string))
    $1 = (void *) 0x91a6a0
    (gdb) call ((std::string*)0x91a6a0)->basic_string()
    (gdb) call ((std::string*)0x91a6a0)->assign("Hello, World")
    $2 = (std::basic_string<char, std::char_traits<char>, std::allocator<char> > &) @0x91a6a0: {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x91a6f8 "Hello, World"}}
    (gdb) call SomeFunctionThatTakesAConstStringRef(*(const std::string*)0x91a6a0)

    See: 1 and 2

    by eocanha at April 13, 2021 10:49 AM

    April 12, 2021

    Jan SchmidtRift CV1 – Getting close now…

    (Jan Schmidt)

    It’s been a while since my last post about tracking support for the Oculus Rift in February. There’s been big improvements since then – working really well a lot of the time. It’s gone from “If I don’t make any sudden moves, I can finish an easy Beat Saber level” to “You can’t hide from me!” quality.

    Equally, there are still enough glitches and corner cases that I think I’ll still be at this a while.

    Here’s a video from 3 weeks ago of (not me) playing Beat Saber on Expert+ setting showing just how good things can be now:

    Beat Saber – Skunkynator playing Expert+, Mar 16 2021

    Strap in. Here’s what I’ve worked on in the last 6 weeks:

    Pose Matching improvements

    Most of the biggest improvements have come from improving the computer vision algorithm that’s matching the observed LEDs (blobs) in the camera frames to the 3D models of the devices.

    I split the brute-force search algorithm into 2 phases. It now does a first pass looking for ‘obvious’ matches. In that pass, it does a shallow graph search of blobs and their nearest few neighbours against LEDs and their nearest neighbours, looking for a match using a “Strong” match metric. A match is considered strong if expected LEDs match observed blobs to within 1.5 pixels.

    Coupled with checks on the expected orientation (matching the Gravity vector detected by the IMU) and the pose prior (expected position and orientation are within predicted error bounds) this short-circuit on the search is hit a lot of the time, and often completes within 1 frame duration.

    In the remaining tricky cases, where a deeper graph search is required in order to recover the pose, the initial search reduces the number of LEDs and blobs under consideration, speeding up the remaining search.

    I also added an LED size model to the mix – for a candidate pose, it tries to work out how large (in pixels) each LED should appear, and use that as a bound on matching blobs to LEDs. This helps reduce mismatches as devices move further from the camera.

    LED labelling

    When a brute-force search for pose recovery completes, the system now knows the identity of various blobs in the camera image. One way it avoids a search next time is to transfer the labels into future camera observations using optical-flow tracking on the visible blobs.

    The problem is that even sped-up the search can still take a few frame-durations to complete. Previously LED labels would be transferred from frame to frame as they arrived, but there’s now a unique ID associated with each blob that allows the labels to be transferred even several frames later once their identity is known.

    IMU Gyro scale

    One of the problems with reverse engineering is the guesswork around exactly what different values mean. I was looking into why the controller movement felt “swimmy” under fast motions, and one thing I found was that the interpretation of the gyroscope readings from the IMU was incorrect.

    The touch controllers report IMU angular velocity readings directly as a 16-bit signed integer. Previously the code would take the reading and divide by 1024 and use the value as radians/second.

    From teardowns of the controller, I know the IMU is an Invensense MPU-6500. From the datasheet, the reported value is actually in degrees per second and appears to be configured for the +/- 2000 °/s range. That yields a calculation of Gyro-rad/s = Gyro-°/s * (2000 / 32768) * (?/180) – or a divisor of 938.734.

    The 1024 divisor was under-estimating rotation speed by about 10% – close enough to work until you start moving quickly.

    Limited interpolation

    If we don’t find a device in the camera views, the fusion filter predicts motion using the IMU readings – but that quickly becomes inaccurate. In the worst case, the controllers fly off into the distance. To avoid that, I added a limit of 500ms for ‘coasting’. If we haven’t recovered the device pose by then, the position is frozen in place and only rotation is updated until the cameras find it again.

    Exponential filtering

    I implemented a 1-Euro exponential smoothing filter on the output poses for each device. This is an idea from the Project Esky driver for Project North Star/Deck-X AR headsets, and almost completely eliminates jitter in the headset view and hand controllers shown to the user. The tradeoff is against introducing lag when the user moves quickly – but there are some tunables in the exponential filter to play with for minimising that. For now I’ve picked some values that seem to work reasonably.

    Non-blocking radio

    Communications with the touch controllers happens through USB radio command packets sent to the headset. The main use of radio commands in OpenHMD is to read the JSON configuration block for each controller that is programmed in at the factory. The configuration block provides the 3D model of LED positions as well as initial IMU bias values.

    Unfortunately, reading the configuration block takes a couple of seconds on startup, and blocks everything while it’s happening. Oculus saw that problem and added a checksum in the controller firmware. You can read the checksum first and if it hasn’t changed use a local cache of the configuration block. Eventually, I’ll implement that caching mechanism for OpenHMD but in the meantime it still reads the configuration blocks on each startup.

    As an interim improvement I rewrote the radio communication logic to use a state machine that is checked in the update loop – allowing radio communications to be interleaved without blocking the regularly processing of events. It still interferes a bit, but no longer causes a full multi-second stall as each hand controller turns on.

    Haptic feedback

    The hand controllers have haptic feedback ‘rumble’ motors that really add to the immersiveness of VR by letting you sense collisions with objects. Until now, OpenHMD hasn’t had any support for applications to trigger haptic events. I spent a bit of time looking at USB packet traces with Philipp Zabel and we figured out the radio commands to turn the rumble motors on and off.

    In the Rift CV1, the haptic motors have a mode where you schedule feedback events into a ringbuffer – effectively they operate like a low frequency audio device. However, that mode was removed for the Rift S (and presumably in the Quest devices) – and deprecated for the CV1.

    With that in mind, I aimed for implementing the unbuffered mode, with explicit ‘motor on + frequency + amplitude’ and ‘motor off’ commands sent as needed. Thanks to already having rewritten the radio communications to use a state machine, adding haptic commands was fairly easy.

    The big question mark is around what API OpenHMD should provide for haptic feedback. I’ve implemented something simple for now, to get some discussion going. It works really well and adds hugely to the experience. That code is in the https://github.com/thaytan/OpenHMD/tree/rift-haptics branch, with a SteamVR-OpenHMD branch that uses it in https://github.com/thaytan/SteamVR-OpenHMD/tree/controller-haptics-wip

    Problem areas

    Unexpected tracking losses

    I’d say the biggest problem right now is unexpected tracking loss and incorrect pose extractions when I’m not expecting them. Especially my right controller will suddenly glitch and start jumping around. Looking at a video of the debug feed, it’s not obvious why that’s happening:

    To fix cases like those, I plan to add code to log the raw video feed and the IMU information together so that I can replay the video analysis frame-by-frame and investigate glitches systematically. Those recordings will also work as a regression suite to test future changes.

    Sensor fusion efficiency

    The Kalman filter I have implemented works really nicely – it does the latency compensation, predicts motion and extracts sensor biases all in one place… but it has a big downside of being quite expensive in CPU. The Unscented Kalman Filter CPU cost grows at O(n^3) with the size of the state, and the state in this case is 43 dimensional – 22 base dimensions, and 7 per latency-compensation slot. Running 1000 updates per second for the HMD and 500 for each of the hand controllers adds up quickly.

    At some point, I want to find a better / cheaper approach to the problem that still provides low-latency motion predictions for the user while still providing the same benefits around latency compensation and bias extraction.

    Lens Distortion

    To generate a convincing illusion of objects at a distance in a headset that’s only a few centimetres deep, VR headsets use some interesting optics. The LCD/OLED panels displaying the output get distorted heavily before they hit the users eyes. What the software generates needs to compensate by applying the right inverse distortion to the output video.

    Everyone that tests the CV1 notices that the distortion is not quite correct. As you look around, the world warps and shifts annoyingly. Sooner or later that needs fixing. That’s done by taking photos of calibration patterns through the headset lenses and generating a distortion model.

    Camera / USB failures

    The camera feeds are captured using a custom user-space UVC driver implementation that knows how to set up the special synchronisation settings of the CV1 and DK2 cameras, and then repeatedly schedules isochronous USB packet transfers to receive the video.

    Occasionally, some people experience failure to re-schedule those transfers. The kernel rejects them with an out-of-memory error failing to set aside DMA memory (even though it may have been running fine for quite some time). It’s not clear why that happens – but the end result at the moment is that the USB traffic for that camera dies completely and there’ll be no more tracking from that camera until the application is restarted.

    Often once it starts happening, it will keep happening until the PC is rebooted and the kernel memory state is reset.

    Occluded cases

    Tracking generally works well when the cameras get a clear shot of each device, but there are cases like sighting down the barrel of a gun where we expect that the user will line up the controllers in front of one another, and in front of the headset. In that case, even though we probably have a good idea where each device is, it can be hard to figure out which LEDs belong to which device.

    If we already have a good tracking lock on the devices, I think it should be possible to keep tracking even down to 1 or 2 LEDs being visible – but the pose assessment code will have to be aware that’s what is happening.

    Upstreaming

    April 14th marks 2 years since I first branched off OpenHMD master to start working on CV1 tracking. How hard can it be, I thought? I’ll knock this over in a few months.

    Since then I’ve accumulated over 300 commits on top of OpenHMD master that eventually all need upstreaming in some way.

    One thing people have expressed as a prerequisite for upstreaming is to try and remove the OpenCV dependency. The tracking relies on OpenCV to do camera distortion calculations, and for their PnP implementation. It should be possible to reimplement both of those directly in OpenHMD with a bit of work – possibly using the fast LambdaTwist P3P algorithm that Philipp Zabel wrote, that I’m already using for pose extraction in the brute-force search.

    Others

    I’ve picked the top issues to highlight here. https://github.com/thaytan/OpenHMD/issues has a list of all the other things that are still on the radar for fixing eventually.

    Other Headsets

    At some point soon, I plan to put a pin in the CV1 tracking and look at adapting it to more recent inside-out headsets like the Rift S and WMR headsets. I implemented 3DOF support for the Rift S last year, but getting to full positional tracking for that and other inside-out headsets means implementing a SLAM/VIO tracking algorithm to track the headset position.

    Once the headset is tracking, the code I’m developing here for CV1 to find and track controllers will hopefully transfer across – the difference with inside-out tracking is that the cameras move around with the headset. Finding the controllers in the actual video feed should work much the same.

    Sponsorship

    This development happens mostly in my spare time and partly as open source contribution time at work at Centricular. I am accepting funding through Github Sponsorships to help me spend more time on it – I’d really like to keep helping Linux have top-notch support for VR/AR applications. Big thanks to the people that have helped get this far.

    by thaytan at April 12, 2021 08:29 AM

    April 11, 2021

    Andy Wingoguile's reader, in guile

    (Andy Wingo)

    Good evening! A brief(ish?) note today about some Guile nargery.

    the arc of history

    Like many language implementations that started life when you could turn on the radio and expect to hear Def Leppard, Guile has a bottom half and a top half. The bottom half is written in C and exposes a shared library and an executable, and the top half is written in the language itself (Scheme, in the case of Guile) and somehow loaded by the C code when the language implementation starts.

    Since 2010 or so we have been working at replacing bits written in C with bits written in Scheme. Last week's missive was about replacing the implementation of dynamic-link from using the libltdl library to using Scheme on top of a low-level dlopen wrapper. I've written about rewriting eval in Scheme, and more recently about how the road to getting the performance of C implementations in Scheme has been sometimes long.

    These rewrites have a quixotic aspect to them. I feel something in my gut about rightness and wrongness and I know at a base level that moving from C to Scheme is the right thing. Much of it is completely irrational and can be out of place in a lot of contexts -- like if you have a task to get done for a customer, you need to sit and think about minimal steps from here to the goal and the gut doesn't have much of a role to play in how you get there. But it's nice to have a project where you can do a thing in the way you'd like, and if it takes 10 years, that's fine.

    But besides the ineffable motivations, there are concrete advantages to rewriting something in Scheme. I find Scheme code to be more maintainable, yes, and more secure relative to the common pitfalls of C, obviously. It decreases the amount of work I will have when one day I rewrite Guile's garbage collector. But also, Scheme code gets things that C can't have: tail calls, resumable delimited continuations, run-time instrumentation, and so on.

    Taking delimited continuations as an example, five years ago or so I wrote a lightweight concurrency facility for Guile, modelled on Parallel Concurrent ML. It lets millions of fibers to exist on a system. When a fiber would need to block on an I/O operation (read or write), instead it suspends its continuation, and arranges to restart it when the operation becomes possible.

    A lot had to change in Guile for this to become a reality. Firstly, delimited continuations themselves. Later, a complete rewrite of the top half of the ports facility in Scheme, to allow port operations to suspend and resume. Many of the barriers to resumable fibers were removed, but the Fibers manual still names quite a few.

    Scheme read, in Scheme

    Which brings us to today's note: I just rewrote Guile's reader in Scheme too! The reader is the bit that takes a stream of characters and parses it into S-expressions. It was in C, and now is in Scheme.

    One of the primary motivators for this was to allow read to be suspendable. With this change, read-eval-print loops are now implementable on fibers.

    Another motivation was to finally fix a bug in which Guile couldn't record source locations for some kinds of datums. It used to be that Guile would use a weak-key hash table to associate datums returned from read with source locations. But this only works for fresh values, not for immediate values like small integers or characters, nor does it work for globally unique non-immediates like keywords and symbols. So for these, we just wouldn't have any source locations.

    A robust solution to that problem is to return annotated objects rather than using a side table. Since Scheme's macro expander is already set to work with annotated objects (syntax objects), a new read-syntax interface would do us a treat.

    With read in C, this was hard to do. But with read in Scheme, it was no problem to implement. Adapting the expander to expect source locations inside syntax objects was a bit fiddly, though, and the resulting increase in source location information makes the output files bigger by a few percent -- due somewhat to the increased size of the .debug_lines DWARF data, but also due to serialized source locations for syntax objects in macros.

    Speed-wise, switching to read in Scheme is a regression, currently. The old reader could parse around 15 or 16 megabytes per second when recording source locations on this laptop, or around 22 or 23 MB/s with source locations off. The new one parses more like 10.5 MB/s, or 13.5 MB/s with positions off, when in the old mode where it uses a weak-key side table to record source locations. The new read-syntax runs at around 12 MB/s. We'll be noodling at these in the coming months, but unlike when the original reader was written, at least now the reader is mainly used only at compile time. (It still has a role when reading s-expressions as data, so there is still a reason to make it fast.)

    As is the case with eval, we still have a C version of the reader available for bootstrapping purposes, before the Scheme version is loaded. Happily, with this rewrite I was able to remove all of the cruft from the C reader related to non-default lexical syntax, which simplifies maintenance going forward.

    An interesting aspect of attempting to make a bug-for-bug rewrite is that you find bugs and unexpected behavior. For example, it turns out that since the dawn of time, Guile always read #t and #f without requiring a terminating delimiter, so reading "(#t1)" would result in the list (#t 1). Weird, right? Weirder still, when the #true and #false aliases were added to the language, Guile decided to support them by default, but in an oddly backwards-compatible way... so "(#false1)" reads as (#f 1) but "(#falsa1)" reads as (#f alsa1). Quite a few more things like that.

    All in all it would seem to be a successful rewrite, introducing no new behavior, even producing the same errors. However, this is not the case for backtraces, which can expose the guts of read in cases where that previously wouldn't happen because the C stack was opaque to Scheme. Probably we will simply need to add more sensible error handling around callers to read, as a backtrace isn't a good user-facing error anyway.

    OK enough rambling for this evening. Happy hacking to all and to all a good night!

    by Andy Wingo at April 11, 2021 07:51 PM

    April 08, 2021

    Andy Wingosign of the times

    (Andy Wingo)

    Hello all! There is a mounting backlog of things that landed in Guile recently and to avoid having to eat the whole plate in one bite, I'm going to try to send some shorter missives over the next weeks.

    Today's is about a silly thing, dynamic-link. This interface is dlopen, but "portable". See, back in the day -- like, 1998 -- there were lots of kinds of systems and how to make and load a shared library portably was hard. You'd have people with AIX and Solaris and all kinds of weird compilers and linkers filing bugs on your project if you hard-coded a GNU toolchain invocation when creating loadable extensions, or hard-coded dlopen or similar to use them.

    Libtool provided a solution to create portable loadable libraries, which involved installing .la files alongside the .so files. You could use libtool to link them to a library or an executable, or you could load them at run-time via the libtool-provided libltdl library.

    But, the .la files were a second source of truth, and thus a source of bugs. If a .la file is present, so is an .so file, and you could always just use the .so file directly. For linking against an installed shared library on modern toolchains, the .la files are strictly redundant. Therefore, all GNU/Linux distributions just delete installed .la files -- Fedora, Debian, and even Guix do so.

    Fast-forward to today: there has been a winnowing of platforms, and a widening of the GNU toolchain (in which I include LLVM as well as it has a mostly-compatible interface). The only remaining toolchain flavors are GNU and Windows, from the point of view of creating loadable shared libraries. Whether you use libtool or not to create shared libraries, the result can be loaded either way. And from the user side, dlopen is the universally supported interface, outside of Windows; even Mac OS fixed their implementation a few years back.

    So in Guile we have been in an unstable equilibrium: creating shared libraries by including a probably-useless libtool into the toolchain, and loading them by using a probably-useless libtool-provided libltdl.

    But the use of libltdl has not been without its costs. Because libltdl intends to abstract over different platforms, it encourages you to leave off the extension when loading a library, instead promising to try a platform-specific set such as .so, .dll, .dylib etc as appropriate. In practice the abstraction layer was under-maintained and we always had some problems on Mac OS, for example.

    Worse, as ltdl would search through the path for candidates, it would only report the last error it saw from the underlying dlopen interface. It was almost always the case that if A and B were in the search path, and A/foo.so failed to load because of a missing dependency, the error you would get as a user would instead be "file not found", because ltdl swallowed the first error and kept trucking to try to load B/foo.so which didn't exist.

    In summary, this is a case where the benefits of an abstraction layer decline over time. For a few years now, libltdl hasn't been paying for itself. Libtool is dead, for all intents and purposes (last release in 2015); best to make plans to migrate away, somehow.

    In the case of the dlopen replacement, in Guile we ended up rewriting the functionality in Scheme. The underlying facility is now just plain dlopen, for which we shim a version of dlopen on Windows, inspired by the implementation in cygwin. There are still platform-specific library extensions, but that is handled easily on the Scheme layer.

    Looking forward, I think it's probably time to replace Guile's use of libtool to create its libraries and executables. I loathe the fact that libtool puts shell scripts in the place of executables in build directories and stashes the actual executables elsewhere -- like, visceral revulsion. There is no need for that nowadays. Not sure what to replace it with, nor on what timeline.

    And what about autotools? That, my friends, would be a whole nother blog post. Until then, & probably sooner, happy hacking!

    by Andy Wingo at April 08, 2021 07:09 PM

    April 06, 2021

    Jean-François Fortin Tam“Getting Things GNOME” 0.5 released!

    It is time to welcome a new release of the Rebuild of EvanGTGelion: 0.5, “You Can (Not) Improve Performance”!

    This release of GTG has been 9 months in the making after the groundbreaking 0.4 release. While 0.4 was a major “perfect storm” overhaul, 0.5 is also a very technology-intensive release, even though it was done in a relatively short timeframe comparatively.

    Getting Things GNOME 0.5 brings a truckload of user experience refinements, bugfixes, a completely revamped file format and task editor, and a couple of notable performance improvements. It doesn’t solve every performance problem yet (some remain), but it certainly improves a bunch of them for workaholics like me. If 0.4 felt a bit like a turtle, 0.5 is a definitely a much faster turtle.

    If that was not enough already, it has some killer new features too. It’s the bee’s knees!

    To benefit from one performance improvement in particular, it requires the new version of liblarch, 3.1, that we have released this month. GTG with the latest liblarch is available all-in-one in a Flatpak update near you. 📦

    This release announcement and all that led up to it was, as you can imagine, planned using GTG:

    “Who’s laughing now eh, señor Ruiz?”

    As you can see, I came prepared. So continue reading below for the summary of improvements, I guarantee it’ll be worth it.


    Some brief statistics

    Beyond what is featured in these summarized release notes below, GTG itself has undergone roughly 400 changes affecting over 230 files, and received hundreds of bug fixes, as can be seen here. In fact, through this development cycle, we have crossed the 6,000 commits mark!

    If you need another indication of how active the project has been in the last 9 months, I have received over 950 GTG-related emails since the 0.4 release last July.

    We are thankful to all of our supporters and contributors, past and present, who made GTG 0.5 possible. Check out the “About” dialog for a list of contributors for this release.


    New features

    Recurring Tasks

    Thanks to Mohieddine Drissi and Youssef Toukabri, tasks can now be made recurring (repeating). This allows you to keep reminding yourself that you need to clean your room every Saturday, and then not do it because it’s more fun to spend your whole Saturday fixing GTG bugs!

    • In the task editor, recurrence can be toggled with the dedicated menubutton that shows a nice user interface to manage the recurrence (daily, every other day, weekly, monthly and yearly).
    • Recurring tasks can also be added from the main window, using the “quick add” entry’s syntax.

    Emojis as tag emblems 🦄

    The old tag icons have been replaced by emojis. I decided it would be best for us to switch to emojis instead of icon-based emblems, because:

    • The availability of emblem icons varies a lot from one system to another, and from one icon theme to another;
    • In recent years, emblem icons on Linux have become quite a mismatch of styles, and their numbers in standard themes have also decreased;
    • Emojis are scalable by default, which makes them better for HiDPI displays (while traditional pixmap icons still are a hit-or-miss);
    • There is a much, much, much wider choice of emojis (for every conceivable situation!) than there are emblem icons on typical Linux systems;
    • It’s simply easier to deal with, in the long run. 😌

    Unfortunately, this means that when you upgrade from 0.4 to 0.5, the old emblem icons will disappear and you will need to choose new emblems for your tasks.

    Undead Hamster

    The GTG plugin to interface with Hamster has been brought back to life by Francisco Lavin (issue #114 and pull request #465). I have not personally tested this. I’m still a bit worried about the ferrets rising from their graves.

    Performance improvements
    for power users

    An elegant search trigger
    for a more… civilized age

    The filter-as-you-type global live search in GTG now uses a timeout approach (it waits a fraction of a second until after you stopped typing). This changes many things:

    • No more spamming the database and the UI’s treeview
    • The UI stays much more responsive, and you can see your characters being typed smoothly instead of appearing 15 seconds later
    • It won’t hog your CPU nearly as much
    • You get search results much faster: the results now tend to show up within a second after you stop typing, rather than a linear or exponential amount of time. For example, with 1000 tasks, it is now over 5 times faster (ex: bringing down the search time from 17+ seconds to 3 seconds), and the difference would be even more drastic if you have more tasks than that.

    Optimizations to avoid processing all views all the time

    As a result of the changes proposed in pull request #530 and follow-up PR #547, for issue #402, most operations switching between various representations of your tasks will now be 20% to 200% faster (depending on how your tasks are structured):

    • Faster startup
    • Faster switching between tags (particularly when inside the “Actionable” view)
    • Faster mid-night refreshing
    • It most likely makes the global search feature faster, too

    However, changing between global view modes (Open/Actionable/Closed) is now a bit slower the first time around (but is instantaneous afterwards, until a filtering change occurs).

    It’s a performance tradeoff, but it seems to be worth it. Especially when you have lots of tasks:

    Note that to benefit from these changes, GTG 0.5 depends on the newly released version of liblarch (3.1) we have published this month.

    Faster read/write operations on the data file 🚀

    The switch to LXML as our parser ensures any operations on the local file format now happen instantly (less than a millisecond for a normal/lightweight task lists, and up to 25 milliseconds for heavy-duty 1000-tasks lists like mine). This was originally such a big deal that we thought we’d nickname this release “¿Que parser, amigo?” … but other major improvements kept happening since then, so this release is not just about the pasa!

    Less micro-freezes when focusing task windows

    Mr. Bean waitingPictured: me, waiting for the task editor windows to un-freeze when I focus them.

    Users with a thousand tasks will probably have noticed that GTG 0.4 had a tendency to painfully lock up for a couple of seconds when focusing/defocusing individual task editor windows.

    Sure, “It doesn’t catastrophically lock up in your face multiple times per day anymore” doesn’t sound like a feature in theory, but it is a usability feature in practice.

    Now, even when typing inside the new Task Editor and making lots of changes, it remains butter-smooth.

    Part of it might be because it now uses a timer-based approach to parsing, and part of it is probably due to the code having been refactored heavily and being of higher quality overall. It just tends to be better all around now, and I’ve tried fairly hard to break it during my smoke-testing, especially since a lot of the improvements have also been tied to the major file format redesign that Diego implemented.

    All in all, I’m not sure exactly how it accomplishes better reliability, to be honest, so we’ll just assume Diego secretly is a warlock.

    50% more awesome task editing

    Oh yeah, speaking of the task editor…

    Rewriting the Task Editor’s content area and its natural language parser brought many user-visible improvements in addition to the technological improvements:

    • The new timeout-based parser is faster (because we’re not spamming it like barbarians), more reliable, and now gains the ability to recognize subtasks (lines starting with “-“) live, instead of “after pressing Enter.” This also makes copy-pasting from a dashed list easier, as it will be converted into many subtasks in one go
    • Clicking on a tag inside the task editor’s text selects the tag in the main window
    • Tags’ text highlight color now uses the tag’s chosen emblem color
    • Subtasks now have GTK checkboxes!
    • Completed subtasks show up in normal strikethrough text style
    • Opening a subtask takes care of closing the parent task, and the other way around (opening a parent task closes the subtask window), minimizing the amount of utility windows cluttering your view (and reducing the chances of making conflicting changes)
    • Now supports subheadings, in Markdown format (type # title). Now that it supports some basic rich-text formatting features, even Grumpy Sri will be happy with this release:

    Here is a short demonstration video that lets you see the improved “parsing as you type” engine in action:

    Other user interface improvements

    General bug fixes

    Technological changes

    • Eliminate the pyxdg dependency in favor of directly using GLib
    • dbus-python dependency dropped in favor of Gio (except in the Hamster plugin)
    • New and improved file format (see issue 279, “Better XML format [RFC]“, and my previous spooky blog post):
      • Automatically converts upon first startup, nothing to do manually
      • More consistent
      • Uses a single file instead of three separate ones
      • Uses valid XML
      • Fully documented (see docs/contributors/file format.md)
      • Adds file format versioning to make it future-proof
      • Uses LXML as the parser, which is reportedly 40 times faster (see issue #285)
      • Fundamentally allows the task editor to be a lot more reliable for parsing, as we ditch the XML-in-XML matryoshka approach
      • Solves a few other bugs related to (de)serializing
    • Rewrote the Task Editor’s “TaskView” (content area) and its natural language parser, paving the way for more improvements and increased reliability:
      • The timeout-based approach mentioned earlier means that parsing happens less often, “when it needs to”, and is also less error-prone as a result.
      • With the implementation of subheadings, the code now has the foundations for more Markdown features in the future.
      • The taskview supports invisible characters that turn visible when the cursor passes over them, which could be useful for more text tags in the future.
      • Easier to maintain improved performance in the future: When running in debug mode (launch.sh -d), the process function now prints the time it took to process the buffer. This will help us stay conscious of the performance.
      • Support for linking between tasks with gtg://[TASK_ID] (no UI for this yet, though)
      • Better code overall, a lot more extensible and easier to work on
    • Rewrote the Quick Add entry’s parsing and added unit tests
    • Command line parsing ported to GLib, so GLib/GTK-specific options such as --display are now available
    • GTG is now dbus-activatable. Although no API that other applications can use has been implemented yet (outside of “standard” GTK ones), this opens up the possibility in the future, such as the GNOME Shell search integration.

    Developer Experience improvements

    Want to be part of a great team? Join BAHRAM GTG!

    Here are some changes to primarily benefit users crazy enough to be called developers.

    • New plugin: “Developer Console”.
      Play with GTG’s internals from the comfort of GTG itself! Remember: when exiting your Orbital Frame, wear… a helmet.
    • New parameter in launch.sh: -w to enable warnings
      (in case you still are not wearing a helmet, or want to catch more bugs)
    • Better help message for launch.sh, documenting the available options
    • Include Git commit version in the about window
    • Use the proper exit code when the application quits cleanly
    • We closed all the tickets on LaunchPad (over 300 of them) so there is no confusion when it comes to reporting bugs and the current state of the app’s quality and priorities, and to ensure that issues are reported in the currently used bug tracker where development actually happens.

    Help out Diego!

    It is no secret that Diego has been spending a lot of time and effort making this release happen, just like the 0.4 release. The amount and depth of his changes are quite amazing, so it’s no surprise he’s the top contributor here:

    If you would like to show your appreciation and support his maintainership work with some direct monthly donations to him, please consider giving a couple of dollars per month on his gumroad page or his liberapay page!

    More coming in the next release

    Many new features, plugins, and big code changes have been held back from merging so that we could release 0.5 faster, so a bunch of other changes will land in the development version on the way to 0.6.

    “This blog post is ridiculously long! And we still have 10 pull requests almost ready to merge!”
    — Diego, five minutes before I published this blog post

    Spreading this announcement

    We have made some social postings on Twitter, on Mastodon and on LinkedIn that you can re-share/retweet/boost. Please feel free to link to this announcement on forums and blogs as well!

    by Jeff at April 06, 2021 02:00 PM

    March 25, 2021

    Andy Wingohere we go again

    (Andy Wingo)

    Around 18 months ago, Richard Stallman was forced to resign from the Free Software Foundation board of directors and as president. It could have been anything -- at that point he already had a history of behaving in a way that was particularly alienating to women -- but in the end it was his insinuation that it was somehow OK if his recently-deceased mentor Marvin Minsky, then in his 70s or 80s, had sex with a 17-year-old on Jeffrey Epstein's private island. A weird pick of hill to stake one's reputation on, to say the least.

    At the time I was relieved that we would finally be getting some leadership renewal at the FSF, and hopeful that we could get some mission renewal as well. I was also looking forward to the practical implications would be for the GNU project, as more people agreed that GNU was about software freedom and not about its founder.

    But now we're back! Not only has RMS continued through this whole time to insist that he runs the GNU project -- something that is simply not the case, in my estimation -- but this week, a majority of a small self-selected group of people, essentially a subset of current and former members of the FSF board of directors and including RMS himself, elected to reinstate RMS to the board of the Free Software Foundation. Um... read the room, FSF voting members? What kind of message are you sending?

    In this context I can only agree with the calls for the entire FSF board to resign. The board is clearly not fit for purpose, if it can make choices like this.

    dissociation?

    I haven't (yet?) signed the open letter because I would be in an inconsistent position if I did so. The letter enjoins people to "refuse to contribute to projects related to the FSF and RMS"; as a co-maintainer of GNU Guile, which has its origins in the heady 1990s of the FSF but has nothing to do any more with RMS, but whose copyrights are entirely held by the FSF, is hosted on FSF-run servers, and is even obliged (GPLv3 §5d, as referenced by LGPLv3) to print out Copyright (C) 1995-2021 Free Software Foundation, Inc. when it starts, I must admit that I contribute to a project that is "related to the FSF". But I don't see how Guile could continue this association, if the FSF board continues as it is. It's bad for contributors and for the future of the project.

    It would be very tricky to disentangle Guile from the FSF -- consider hosting, for example -- so it's not the work of a day, but it's something to think about.

    Of course I would rather that the FSF wouldn't allow itself to be seen as an essentially misogynist organization. So clean house, FSF!

    on the nature of fire

    Reflecting on how specifically we could have gotten here -- I don't know. I don't know the set of voting members at the FSF, what discussions were made, who voted what. But, having worked as a volunteer on GNU projects for almost two decades now, I have a guess. RMS and his closest supporters see themselves as guardians of the flame of free software -- a lost world of the late 70s MIT AI lab, reborn in a flurry of mid-80s hack, but since 25 years or so, slipping further and further away. These are dark times, in their view, and having the principled founder in a leadership role can only be a good thing.

    (Of course, the environment in the AI lab was only good for some. The treatment of Margaret Hamilton as recounted in Levy's Hackers shows that not all were welcome. If this were just one story, I would discount it, but looking back, it does seem to be part of a pattern.)

    But is that what the FSF is for today? If so, Guile should certainly leave. I'm not here for software as perfomative nostalgia -- I'm here to have fun with friends and start a fire. The FSF should look to do the same -- look at the world we are in, look where the energy is now, and engage in real conversations about success and failure and tactics. There is a world to win and doubling down on RMS won't get us there from here.

    by Andy Wingo at March 25, 2021 12:22 PM