Hardware based encoding on AMD GPUs
I've been asking in this group a few weeks ago on how to use the VCE of AMD graphics chips to do hardware based h.264 encoding. As I've finally gotten it to work (although I still have to use gst-launch...), I thought it might be a good idea to share how I did it.
There are several components required to utilize it, and all of them have to work together correctly:
o) The VCE circuitry on the GPU requires the appropriate firmware file. On most distributions it's part of the firmware-linux-nonfree package, and if your distribution is using initramfs, it'll likely work already. If not, you'll need to make sure that the appropriate file is available at boot time, by either putting it in the initramfs, or packing it directly into your kernel image. If unsure, grep your dmesg for "VCE". If there's an error, the most likely reason is that the firmware file couldn't be loaded. No problem though, as dmesg should also tell you the filename of said firmware.
o) In userspace you'll need a mesa build with the OpenMAX state tracker enabled. On Gentoo this can be done by enabling the "openmax" use flag for mesa. I don't know about other distributions though.
o) An integration layer for the OpenMAX state tracker has to be installed as well. The one I've had success with (and the only one I know of for mesa's OpenMAX state tracker) is omxil-bellagio. But installing it isn't enough, one also has to run "omxregister-bellagio" for every user who wants to use hardware encoding. Best run it with the "-v" option, to see immediately if the mesa OpenMAX state tracker is being detected.
o) The next component one needs is the gstreamer plugin for OpenMAX. The version for gstreamer 1.0 is called gst-omx. If your distribution doesn't have packages, the source code can be found here: https://gstreamer.freedesktop.org/releases/gst-omx/1.0.0.html
This gstreamer plugin again has its own pitfalls that caused me quite some headaches. It expects a configuration file to be present in /etc/xdg/gstomx.conf, but the one it ships with, even if one specifically builds it to support omxil-bellagio, is registering unavailable functions, and doesn't list those of mesa at all. The solution I found is to get the /etc/xdg/gstomx.conf file from gearsongallium (here: https://github.com/pontostroy/gearsongallium/blob/master/cd-image/source/root/etc/xdg/gstomx.conf
). With this file in place, gst-inspect-1.0 should be able to detect your gst-omx plugin successfully without blacklisting. If it gets blacklisted, running gst-inspect-1.0 on the gst-omx plugin shared library with GST_DEBUG being set to 4 or above will help identify the issue.
Now, with a bit of luck, you'll be in principle able to use gst-launch to set up a gstreamer pipeline using omxh264enc to encode your videos using your graphics card's hardware. If you figure out how to properly use gstreamer that is (I'm still struggling to get it to decode my mpeg4 files fast enough to actually benefit from hardware based encoding, and I haven't even tried to mux the result with the audio track...).