Some initial results from deep-testing with +GStreamer

First of all, it seems that doing a content-hash of each video buffer in your input videos, and then doing a content-hash of the video buffers produced by gnonlin, is indeed a viable way to do tests for frame-accuracy.

It's a bit messy still, but here's the one-file experiment I'm currently working on:

http://bazaar.launchpad.net/~jderose/+junk/gst-examples/view/head:/gnonlin-0.10

The idea is that you first do a playthrough of all your input files so you know the content-hash of each video buffer. For some codecs, it might not make sense to store these content-hashes and test against (assumed) good values as the codec doesn't specify exact output. But from talking to +Måns Rullgård, H264 happens to be a codec for which it does make sense to compare the output over time because, in theory, it should stay constant.

So for such files, my idea is to store a human-readable dump of key metadata for each frame in a video, something like this:

http://bazaar.launchpad.net/~jderose/+junk/gst-examples/view/head:/playthrough.json

A surprising (and to me rather concerning) result of this first test is that gnonlin apparently just passes buffers through with the exact timestamp and duration found in the source video. Maybe typical render pipelines rely on videorate to fix the timestamps, or maybe the rendered videos are broken in some key ways... dunno yet.

My biggest problem with this gnonlin behavior is it makes it difficult to test, difficult to reason about.

Although I'm currently just testing via gnonlin, I'm also going to write rigorous tests for seeking in a standard playback pipeline using the same approach. For example, use decodebin to start playing a file, and then abuse GStreamer as much as possible with a series of random seeks, all while randomly flipping between State.PLAYING and State.PAUSED.

Because I'm going to test against the buffer content-hash, we'll have a fast, automated way to know without a doubt whether GStreamer is delivering the perfect frame-accuracy we need.
Photo
Shared publiclyView activity