Shared publicly  - 
I'm really really tired of people saying that there's no documentation for the Linux kernel all the time. Always it comes from people who never even looked for it in the first place.

The kernel comes with loads of documentation in the (duh) Documentation/ directory of the kernel source tree.

Just today, I ran into a place where the running kernel provided a README for me to use, just do:
cat /sys/kernel/debug/tracing/README
to see it.
+Steven Rostedt you rock.

- Edit -
People, I'm talking about "basic" documentation questions here. Here's a sample of what I was asked in just the past 2 weeks:
- where is the documentation on how to build the kernel
- where is the documentation on how the kernel development process works
- where is the documentation on how to get involved in the kernel
- where is the documentation on how to send a patch
- where is the documentation on how to get a patch into the stable kernel
- where is the documentation on what kinds of patches go into the stable kernel
- where is the documentation on how to fix my email client.
All of these are "obvious" if only they had actually looked.

I'm reminded by something someone once said to me when I was working at IBM, "I could go look this up, but it's easier for me to ask you..."
Diego Call.'s profile photoDarren Hart's profile photoSteven Rostedt's profile photoNick Alcock's profile photo
So we are to infer that you are on board with runtime documentation? I totally don't see any way that could be abused </sarcasm>
I had never read as much kernel documentation as during the EX442 exam...
Greg, the documentation is very bad. Most of the things are very badly documented. I am not complaining, but do not downplay the criticism at documentation.

The important thing you did not mention is that there are books (I have read some of them) that document the functionality of Linux kernel.

Just face it, Linux is badly documented, but so what?
Why can't google know exactly what kernel documentation I'm looking for even before I start typing?? jeez
+Martin Seidler Fair point, but it's not really about comparison to other systems. Looking at subsets of Linux kernel it is badly documented. I'm not complaining about the lack of documentation, but Greg implies otherwise which is clearly false.

Anyway, Microsoft documents their systems better. Of course, Linux has the superioir advantage of having the source code available.
The documentation for the kernel is poor, and the books are out of date as the kernel is a fast moving target. That said, the kernel source is right there. For end-users, this is a non-issue. For developers, there's more than enough. Oh, and Google will answer almost any question if you learn how to use search effectively.
+Greg Kroah-Hartman actually it was +Ingo Molnar that added that. The best part of the /sys/kernel/debug/README is:

tracing mini-HOWTO:

# mount -t debugfs nodev /sys/kernel/debug

It tells you how to mount debugfs. But if you haven't mounted debugfs, you can't see the README!
Perhaps we should pay +Michael Kerrisk to work on this. He's done an incredible job on the man pages that document the user-space API calls. And the printed version of the kernel docs could be thick enough to stop a charging elephant...};-)
As for kernel documentation. The problem with that is it has a lot more to do with who writes the code. Just like code quality throughout the kernel, there's also documentation (comment) quality too. I've made it a point to write several design documents and placed them into the Documentation directory. I've even written a few LWN articles about my code. I plan on writing more Documentation design files in the near future too. As the code has become more complex, I've seen a change in kernel developers with respect to commenting. I've seen more "comment this better" remarks from maintainers to other developers. The attitude is less "code is documentation" and more "comment better because I wont remember what this does by next year".
I found plenty as I worked my way through all of the earlier kernels that I ran on my machines.
//"I could go look this up, but it's easier for me to ask you..."// always happens to me :-(
It is him. Almost funny that one.
well, the Kerneldocs are just this bad, because on Linux you compare them to userland manpages, which are giving a far superiour api-documentation, then any document for kernel intern apis does.
Sometimes the stuff in Documentation/ is just awful, if you don't have enough prior knowledge. For example, I've got a particular itch to scratch (stupid Marvell 88SE91xx breaks DMA remapping) and I'm trying to get into the iommu code, but Documentation/Intel-iommu.txt is not helpful at all.

"Intel VT for Direct IO" ( has been helpful and so is googling "RFC" | "PATCH 0/*" -Re: whateveryouarelookingfor
I'd also like to find a way to trace a git commit back to a LKML message. cscope and seascope have also been useful.

The way I see it is that you can either keep trying and be blissfully unaware of your own incompetence, or you can post stupid questions and confirm it.
From my point of view, the kernel parameters document is very helpful. Maybe a standard on how kernel documents are written and maintained my help. (Like the man pages.) Just my 2c worth opinion.
Hear precisely this IBM quote jut yesterday :-/
R 2 tha TFM Man!! But seriously, it should be quicker to access and process the documentation than to ask someone, however (useful/usable) documentation isn't always consistent and quality/usability can vary wildly, although I can't really comment on the kernel specifically, but I'd imagine it's similar...Moar standards in docu please!!
+Greg Kroah-Hartman This "I could go look this up, but it's easier for me to ask you..." is certainly not limited to IBM. I have seen it everywhere.

Its esp. bad at environments where you see each other or people walk to your desk. In written conversation it is actually more simple to wait at least 30min and then answer with an RTFM link or LMGTFY link for worst offenders.

So I established a rule for my colleagues to actually research up to 30min for an answer and show me all intermediate results they had, so we can both continue from there.

That cuts these kind of questions down and leaves you with smarter colleagues and more interesting problems.

Only problem left is: Ignorant people now think you ignore them, but I can live with this as they usually get what ignorance means this way.
Actually ever since git the kernel has been the best documented code I've ever worked on; the quality of the revision control history is miles ahead of any other project I've worked on and I've never really found a project where the formal documentation didn't at least need heavy supplementing from the revision control history to try to understand what the author was actually thinking. Even when it's there and fairly complete formal documentation has far too much of a tendency to be about the what not the why.

Not quite the sort of documentation you were talking about but still.
Documentation/ isn't easily google-able. git uses asciidoc and uploads html docs to their site, which is certainly much nicer.
+Greg Kroah-Hartman did say he was talking about simple documentation, specifically not APIs and such. So while I agree with some of the comments above, they don't argue against Greg's point. "Use the Documentation, that's what it's there for!"
I'll extend on +Darren Hart's comment, specifically towards +Andrew Cooks's comment. If you look at the Documentation directory and find that it's crap. That's fair to complain on LKML about it, and Cc the authors of both the code and the documentation. Basically, say "I read this document and I'm still confused. What does foo mean? Why is this done in the code?" They can't bitch back at you with RTFM, because you did and the FM sucks ;-) That may encourage someone to fix it. Or they may still just flame you. But that's their problem.
I'll add that a lot of my own documentation is out of date, and I've had people complain to me about it. My response is 'oh shit, I need to update that'. Then I put it on my TODO list, and maybe in 10 years I'll get to it :-p
Yeah, that's the software engineer's lament: "I understand it now, I'll come back and document it later -- when I can't remember so much about how it worked, oops".

The manpages are, after all, only this good because Michael Kerrisk in particular (Rob Landley and others also, but mostly Michael) did such a good job of maintaining them, shepherding people into actually submitting changes to manpages, and the like. Developer documentation is unlikely to ever develop that sort of impetus, alas.
Add a comment...