Shared publicly  - 

I've just pushed an interesting change to SDL, but to explain what it is, it's useful to know the problem we were trying to solve.

- The Steam Runtime has (at least in theory) a really kick-ass build of SDL2, but developers are shipping their own SDL2 with individual Steam games--including me. These games might stop getting updates, but a newer SDL2 (with, say, Wayland support, which we'll be shipping in 2.0.2) might be needed later. Certainly we'll always be fixing bugs in SDL, even if a new video target isn't ever needed, and these fixes won't make it to a game shipping its own SDL.
- Even if we replace the SDL2 in those games with a compatible one, that is to say, edit a developer's Steam depot (yuck!), there are developers that are statically linking SDL2 that we can't do this for. We can't even force the dynamic loader to ignore their SDL2 in this case, of course.
- If you don't ship an SDL2 with the game in some form, people that disabled the Steam Runtime, or just tried to run the game from the command line instead of Steam might find themselves unable to run the game, due to a missing dependency.
- If you want to ship on non-Steam platforms like GOG or Humble Bundle, or target generic Linux boxes that may or may not have SDL2 installed, you have to ship the library or risk a total failure to launch. So now, you might have to have a non-Steam build plus a Steam build (that is, one with and one without SDL2 included), which is inconvenient if you could have had one universal build that works everywhere.
- We like the zlib license, but the biggest complaint from the open source community about the license change is the static linking. The LGPL forced this as a legal, not technical issue, but zlib doesn't care. Even those that aren't concerned about the GNU freedoms found themselves solving the same problems: swapping in a newer SDL to an older game often times can save the day. Static linking stops this dead.

So here's what we did:

SDL now has, internally, a table of function pointers. So, this is what SDL_Init now looks like:

    UInt32 SDL_Init(Uint32 flags)
        return jump_table.SDL_Init(flags);

Except that is all done with a bunch of macro magic so we don't have to maintain every one of these.

What is jump_table.SDL_init()? Eventually, that's a function pointer of the real SDL_Init() that you've been calling all this time. But at startup, it looks more like this:

    Uint32 SDL_Init_DEFAULT(Uint32 flags)
        return jump_table.SDL_Init(flags);

SDL_InitDynamicAPI() fills in jump_table with all the actual SDL function pointers, which means that this _DEFAULT function never gets called again. First call to any SDL function sets the whole thing up.

So you might be asking, what was the value in that? Isn't this what the operating system's dynamic loader was supposed to do for us? Yes, but now we've got this level of indirection, we can do things like this:

    export SDL_DYNAMIC_API=/my/actual/

And now, this game that is staticallly linked to SDL, can still be overridden with a newer, or better, SDL. The statically linked one will only be used as far as calling into the jump table in this case. But in cases where no override is desired, the statically linked version will provide its own jump table, and everyone is happy.

So now:
- Developers can statically link SDL, and users can still replace it. (We'd still rather you ship a shared library, though!)
- Developers can ship an SDL with their game, Valve can override it for, say, new features on SteamOS, or distros can override it for their own needs, but it'll also just work in the default case.
- Developers can ship the same package to everyone (Humble Bundle, GOG, etc), and it'll do the right thing.
- End users (and Valve) can update a game's SDL in almost any case, to keep abandoned games running on newer platforms.
- Everyone develops with SDL exactly as they have been doing all along. Same headers, same ABI. Just get the latest version to enable this magic.

A little more about SDL_InitDynamicAPI():

Internally, InitAPI does some locking to make sure everything waits until a single thread initializes everything (although even SDL_CreateThread() goes through here before spinning a thread, too), and then decides if it should use an external SDL library. If not, it sets up the jump table using the current SDL's function pointers (which might be statically linked into a program, or in a shared library of its own). If so, it loads that library and looks for and calls a single function:

    SInt32 SDL_DYNAPI_entry(Uint32 version, void *table, Uint32 tablesize);

That function takes a version number (more on that in a moment), the address of the jump table, and the size, in bytes, of the table. Now, we've got policy here: this table's layout never changes; new stuff gets added to the end. Therefore SDL_DYNAPI_entry() knows that it can provide all the needed functions if tablesize <= sizeof its own jump table. If tablesize is bigger (say, SDL 2.0.4 is trying to load SDL 2.0.3), then we know to abort, but if it's smaller, we know we can provide the entire API that the caller needs.

The version variable is a failsafe switch. Right now it's always 1. This number changes when there are major API changes (so we know if the tablesize might be smaller, or entries in it have changed). Right now SDL_DYNAPI_entry gives up if the version doesn't match, but it's not inconceivable to have a small dispatch library that only supplies this one function and loads different, otherwise-incompatible SDL libraries and has the right one initialize the jump table based on the version. For something that must generically catch lots of different versions of SDL over time, like the Steam Client, this isn't a bad option.

Finally, I'm sure some people are reading this and thinking "I don't want that overhead in my project!"  To which I would point out that the extra function call through the jump table probably wouldn't even show up in a profile, but lucky you: this can all be disabled. You can build SDL without this if you absolutely must, but we would encourage you not to do that. However, on heavily locked down platforms like iOS, or maybe when debugging,  it makes sense to disable it. The way this is designed in SDL, you just have to change one #define, and the entire system vaporizes out, and SDL functions exactly like it always did. Most of it is macro magic, so the system is contained to one C file and a few headers. However, this is on by default and you have to edit a header file to turn it off. Our hopes is that if we make it easy to disable, but not too easy, everyone will ultimately be able to get what they want, but we've gently nudged everyone towards what we think is the best solution.

Here are most of the changes for the Dynamic API:

CMakeLists.txt · file | annotate | diff | revisions · VisualC/SDL/SDL_VS2008.vcproj · file | annotate | diff | revisions · VisualC/SDL/SDL_VS2010.vcxproj · file | annotate | diff | revisions · VisualC/SDL/SDL_VS2012.vcxproj · file | annotate | diff | revisions · VisualC/SDL/SDL_VS2013.vcxproj ...
Clément DAVID's profile photoC.W. Betts's profile photoSam Watkins's profile photoDaniel Gibson's profile photo
Sounds awesome and great answer to the whole static linking problem. Actually I think this could be used by much more libraries, any time commercial software has their own statically linked version of library that has security fix released later it means problem.

It is great even with the few fixmes :-).
This is super great, thanks Ryan. Now to harass all the old ports for an update! :D 
Would you guys actually change libraries in steam games under others' feet for new features?
Shall we paste this in README-dynapi.txt ?
This will make it a lot easier for anyone to hook arbitrary code into any SDL game. Will this have security/cheating implications?
+Wouter van Oortmerssen We could already hook arbitrary code into any game (not just SDL.) So no, it really doesn't have any security implications.
+Ryan Gordon, I don't see what the advantage is here over simply asking users to not statically link SDL. This change makes the internals of SDL a bit more complicated but seems equivalent to asking folks to please use dynamic linking. Especially considering the escape hatch you've provided with #define.

It will make things slightly simpler for game makers who want a bundled binary, but if that's a problem we want to address it seems like something similar to OSX style App bundles is a much better way to go. Those let you bundle more than just shared objects as you can also bundle assets.

From a software engineering perspective this strikes me as a suboptimal change: It increases the (internal) complexity of SDL and only partially addresses the real problem (easy bundling).

Or have I completely missed something?
+Jason Dagit Our experience has been asking people to not statically link doesn't work, but even if it did, this is entirely solvable as a technical problem, and only largely solvable (with more vigilance and evangelization required) as a social problem.

More to the point, we still needed to overcome problems that exist even when everyone dynamically links:
- We need to update the SDL2 that people might ship with, and hijacking other developers' Steam depots is at a minimum distasteful, and a seriously dangerous precedent.
- If you ship an SDL library, you won't get future fixes and platform support if we don't override it.
- If you don't ship an SDL library, you'll fail to start in some places.

Before this, the best practice was don't ship SDL2 with your Steam games and let the Steam Runtime supply it, but then you needed to have a build that supplies SDL2 for everything else...but in practice, this breaks everyone that doesn't want to use the Steam Client and didn't set up their systems to provide these libraries (and why should they have that burden anyhow?).

The new best practice will be "Use at least SDL 2.0.2 (once we ship that) and always provide it with your game. We prefer if you ship a DLL, but you can statically link it, too."

tl;dr: it's not just about the static linking; dynamic linking is problematic, too.

+Ryan Gordon Thanks for the clarifications. I'm not familiar with the steam platform or the packaging there so I admit I don't know what challenges you face. It seems like LD_PRELOAD (and whatever OSX calls it) should suffice when SDL is dynamically linked, but maybe that still requires tampering with Steam depots?
Ryan Gordon
ifunc ("resolver") on GNU Libc Platforms is a far lower overhead than the jump table solution you are using.   Yes a 1 hit overhead first function usage then zero after that.

Jason Dagit  LD_PRELOAD does not work with SUID bit binaries.  This is a security grounds thing.

Ryan Gordon I am sorry to say your implementation of export SDL_DYNAMIC_API is asking for security issues and human error.

Yes I like the idea Ryan but the change the SDL libraries loaded should be in a configuration file that is unique to each application that requires root or administration privileges to mess with.  SDL contains many libraries.  In fact a configuration file like this could be expanded to many libraries.

ifunc("resolver") on GNU Libc platforms can be used to make very compact compatibility libraries as well.  Yes dlopen all the functions that have not be changed from the newer version.  
+Peter Dolding Great find about the ifunc attribute. It would be interesting to see a particular library interface automatically wrapped with ifunc loader as you are probably right that human error might easily break the scheme devised by Ryan. Solving the problem once for SDL would allow other SDL related libraries to re-use it.
+Peter Dolding ifunc is an interesting piece of code...the existing code would still have to remain for basically everything but Linux, but you can see from the existing code that we eventually want the jump table to be a literal jump table (the "function" it points to should be a single unconditional jump instruction to the appropriate implementation), more or less, this is what ifunc does automatically, so it's worth exploring. Right now, the cost is a little stack pushing overhead, but if a compiler supports tail-call optimization, it should more or less come out the same, too, as-is.

We probably wouldn't go so far with the security concerns, but the setuid thing is a good point.
Ryan Gordon inside windows with PE you have dll injection messing with the PE resolve table as well.  Both ifunc(resover) and dll injections in windows are doing very much same thing.

They are messing with the binary formats own resolve table.  Messing with the PE resolve table works under wine as well as long as its from the running application.

Ryan Gordon basically you are putting a resolve table on top of a resolve table.

OS X, OS i.  Don't know if you can mess with the platforms own resolve tables.   ELF on BSD its possible to mess with resolve table.

Statically linked  ELF without any dynamic parts would not have a resolve table.

Creating your own resolve struct I see as a cause of nasty failures.

Forbiding suid bit binaries is the simplest but I guess some games will want to use suid to get at particular system features.
+Clément DAVID OS X does not use ELF: it uses Mach-0. To my knowledge, OS X is the only modern UNIX or UNIX-like that does not use ELF. Although you can mess with the dynamic linking names, there's always a possibility that SDL2.framework won't be installed in either /Library/Frameworks or ~/Library/Frameworks. And said version might be out-of-date itself. Yeah, OS X doesn't have a good native package management built-into it: developers have to work around it.

Not to mention Windows, which uses something quite archaic in comparison of the other two binary formats (I'm pretty sure it supports Pascal symbol linking. Not even OS X/Darwin has that). C.F. "DLL Hell."
Seems not ideal but a good practical solution!
This is cool, but should be in the SDL wiki!
Add a comment...